Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic: CURAND_STATUS_LAUNCH_FAILURE on GTX1080 #77

Closed
Billthered opened this issue Oct 13, 2016 · 18 comments
Closed

panic: CURAND_STATUS_LAUNCH_FAILURE on GTX1080 #77

Billthered opened this issue Oct 13, 2016 · 18 comments

Comments

@Billthered
Copy link

Hi all,

I've encountered the following error after setting up a system with the configuration details as below, using the precompiled version of mumax3:

mumax 3.9.1c windows_amd64 go1.3.3 (gc)
CUDA 8000 GeForce GTX 1080(8192MB) cc6.1, using CC53 PTX

During the simulation, the generation of the mesh and even the relax() phase works properly. However at the end of relax(), before the simulation runs it gives:

panic: CURAND_STATUS_LAUNCH_FAILURE

goroutine 16 [running]:
runtime.panic(0x8af1a0, 0xc9)
        c:/go/src/pkg/runtime/panic.c:279 +0x11f
github.com/mumax/3/cuda/curand.Generator.GenerateNormal(0x14d61f0, 0x70b000000, 0x80000, 0x3f80000000000000)
        C:/Users/Arne/src/github.com/mumax/3/cuda/curand/generator.go:41 +0x7a

If I set the temperature to 0K it works fine. It seems that the error with CURAND comes from compatibility with the GTX1080 architecture and may need update to CUDA8.0?

BVLC/caffe#4324

Would building Mumax3 with CUDA8.0 solve this issue? I've tried, however I could not build. I got an error related to the CUDA header files, exactly as in issue #18, despite I changed the //windows lines in the different cgoflags.go to be

//#cgo windows LDFLAGS:-LC:/cuda/v5.0/lib/x64 -LC:/cuda/v5.5/lib/x64 -LC:/cuda/v6.0/lib/x64 -LC:/cuda/v6.5/lib/x64 -LC:/cuda/v7.0/lib/x64 -LC:/cuda/v8.0/lib/x64
//#cgo windows CFLAGS: -IC:/cuda/v5.0/include -IC:/cuda/v5.5/include -IC:/cuda/v6.0/include -IC:/cuda/v6.5/include -IC:/cuda/v7.0/include -IC:/cuda/v8.0/include

Actually I also tried different ways and paths to include the headers but only manually adding the full path of a header file seems to work, not any path in the //#cgo commands. What could cause this?

Thank you for your help and comments!

@xhuvom
Copy link

xhuvom commented Oct 20, 2016

Caffe version is 0.15.14 with Digits 5.1 DetectNet training error for CUDA 8.0 on Ubuntu 14.04 backed by GTX-1080
Terminal console output:


2016-10-20 18:40:59 [20161020-184058-da4a] [INFO ] Task subprocess args: "/usr/bin/caffe train --solver=/home/xhuv/digits/digits/jobs/20161020-184058-da4a/solver.prototxt --gpu=0 --weights=/home/xhuv/digits/googlenet.caffemodel"
2016-10-20 18:41:31 [20161020-184058-da4a] [ERROR] Train Caffe Model: Check failed: status == CURAND_STATUS_SUCCESS (201 vs. 0)  CURAND_STATUS_LAUNCH_FAILURE
2016-10-20 18:43:26 [20161020-184058-da4a] [ERROR] Train Caffe Model task failed with error code -6

Plz help!!

see issue #1186

@syockit
Copy link
Contributor

syockit commented Oct 23, 2016

Seeing you closed the issue in DIGITS, is this one still an issue here?

@vandrearczyk
Copy link

vandrearczyk commented Nov 11, 2016

I had the same CURAND_STATUS_SUCCESS (201 vs. 0) check fail when running runtest with ubuntu 16.04, GTX1080 and cuda 8.
I realised that I was still using some cuda 7.5 libraries with:
dpkg -l | egrep 'digits|caffe|libcudnn|libnccl|cudart|nvidia'
Some cuda7.5 libraries were still in /usr/lib/x86_64-linux-gnu/
I purged cuda and nvidia and reinstalled cuda 8. It seems to be working now.

@barnex barnex closed this as completed Jan 11, 2017
@MartenArthers
Copy link

I have the same CURAND failure as Billthered had when running with Temp on my workstation with quadro p600, latest driver. When running om my laptop geforce gt720m the temp fuction works fine.

Any solution?

Thanks.

@godsic
Copy link
Contributor

godsic commented May 17, 2018

Please try the windows build provided in #175 . It is linked against CUDA 9.1.

@MartenArthers
Copy link

MartenArthers commented May 17, 2018

Thanks for your help. The version I had running was 3.9.1, as there was no pre-compiled version of 3.9.3. When i run the mumax3 application in the 'mumax3.10beta.cuda91.win64.zip' it says it it missing the dll file. The only thing I could think of was to copy the dll files of the 3.9.1 to the 3.10 folder and change the names to xxxxx64_91.dll. This didn't work. I'm very new to programming and go language so what I did probably was pretty ignorant. Any clue as to what i should do?

NB: I followed the link:
I started a draft release: https://github.com/mumax/3/releases/tag/untagged-a550dd2b59d233cef40d @ALL: please add/modify bullet points in the draft release notes.

As I thought there might be new dll files there, but this page can't be found...

@godsic
Copy link
Contributor

godsic commented May 17, 2018 via email

@MartenArthers
Copy link

MartenArthers commented May 17, 2018 via email

@godsic
Copy link
Contributor

godsic commented May 17, 2018 via email

@MartenArthers
Copy link

MartenArthers commented May 18, 2018 via email

@godsic
Copy link
Contributor

godsic commented May 18, 2018 via email

@MartenArthers
Copy link

MartenArthers commented May 18, 2018 via email

@godsic
Copy link
Contributor

godsic commented May 18, 2018 via email

@MartenArthers
Copy link

MartenArthers commented May 18, 2018 via email

@godsic
Copy link
Contributor

godsic commented May 18, 2018

@Billthered Does 3.10 work for you?

@godsic
Copy link
Contributor

godsic commented May 18, 2018

@MartenArthers
We included DLLs in previous versions of mumax3 for the sake of convenience, as there is no need to install CUDA SDK then. It is quite likely that such approach won't work in the future as Microsoft is hardening the way DLLs are linked and preloaded to avoid their spoofing.

AFAIK, if CUDA SDK is configured properly, then there is no need to copy any DLLs to mumax3 folder as system will automatically pick them up and preload for you. BTW, on the machine where 3.10 was compiled, CUDA SDK is installed in C:\cuda\ (the path could be configure in Advanced Mode of the CUDA SDK installer).

Anyway, if mumax3.10 fails to launch for you, then please make a screenshot of the error and attach it to this conversation.

@MartenArthers
Copy link

MartenArthers commented May 18, 2018 via email

@godsic
Copy link
Contributor

godsic commented May 18, 2018

@MartenArthers

Good to hear mumax3.10 works for you now.

Please note that here we discuss mumax3 issues / bugs / etc.

Any other question should be asked via mumax2 mailing list https://groups.google.com/forum/#!forum/mumax2 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants