'CUDA Exception: out of memory' when running examples on CUDA

[athas]

Hi, I have installed accelerate-examples and its dependencies (including the CUDA backend) using the stack-8.0.yaml file in HEAD of the accelerate-examples repository. Specifically, my compilation command line was:

$ STACK_YAML=stack-8.0.yaml stack install --flag accelerate-examples:-llvm-cpu --flag accelerate-examples:-llvm-gpu --ghc-options -fasm

This system does not have a compatible LLVM version, so I tried to disable all of that.

When I run any of the examples, I get a CUDA error. For example:

$ accelerate-nbody --benchmark
accelerate-nbody (c) [2012..2013] The Accelerate Team

Usage: accelerate-nbody [OPTIONS]

Available backends:
     interpreter  reference implementation (sequential)
  *  cuda         implementation for NVIDIA GPUs (parallel)

running tests...
t=0: [Failed]
ERROR: CUDA Exception: out of memory
t=10: [Failed]
ERROR: CUDA Exception: out of memory
t=20: [Failed]
ERROR: CUDA Exception: out of memory
t=30: [Failed]
ERROR: CUDA Exception: out of memory
t=40: [Failed]
ERROR: CUDA Exception: out of memory
t=50: [Failed]
ERROR: CUDA Exception: out of memory

         Test Cases  Total      
 Passed  0           0          
 Failed  6           6          
 Total   6           6

This system is headless, so I have not tried running without --benchmark.

I have previously been able to run older versions of these benchmarks on the system, and it is capable of running other CUDA programs. The system configuration is an NVIDIA GTX 780 Ti (actually, two of them), CUDA 7.5, and running RHEL 7.2.

I have tried monitoring the memory usage of the GPUs with nvidia-smi, and it doesn't look like it's running out of memory. In fact, memory usage does not go up at all when running the example.

[tmcdonell]

I think you are being bitten by tmcdonell/cuda#39.

I have a partial fix for this so that compiled programs work, but it is still broken under ghci. I'll bump the SHA in the stack.yaml right now so that it includes those commits and hopefully that is at least a temporary solution for you.

[athas]

Thank you, it seems to work now, except for a weird error at the end:

$ accelerate-nbody --benchmark
accelerate-nbody (c) [2012..2013] The Accelerate Team

Usage: accelerate-nbody [OPTIONS]

Available backends:
     interpreter  reference implementation (sequential)
  *  cuda         implementation for NVIDIA GPUs (parallel)

running tests...
t=0: [OK]
t=10: [OK]
t=20: [OK]
t=30: [OK]
t=40: [OK]
t=50: [OK]

         Test Cases  Total      
 Passed  6           6          
 Failed  0           0          
 Total   6           6          

benchmarking n-body
time                 1.761 ms   (1.752 ms .. 1.771 ms)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 1.777 ms   (1.770 ms .. 1.784 ms)
std dev              24.04 μs   (20.44 μs .. 28.20 μs)

accelerate-nbody: user error (Not a valid Unicode code point!)

That error seems unrelated, though, so I think you can close this issue.

[tmcdonell]

Er, I would have said that your terminal locale is not to UTF-8 or similar, but it appears to be working fine, so... ¯_(ツ)_/¯

Glad the CUDA problem is solved at least!