Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic: CUDA_ERROR_LAUNCH_FAILED #255

Closed
tuberosa opened this issue May 5, 2020 · 4 comments
Closed

panic: CUDA_ERROR_LAUNCH_FAILED #255

tuberosa opened this issue May 5, 2020 · 4 comments

Comments

@tuberosa
Copy link

tuberosa commented May 5, 2020

Hi. I’m using MuMax3 3.10 of this version(https://github.com/mumax/3/tree/jmulkers/prepare3.10) on windows 10.

Until the May 3rd , it was working fine, however though I changed nothing explicitly it started to show the error below when I ran commands like mumax3 test.mx3 on commandline.

I’m wondering GPU driver’s auto-update did something wrong and did downgrading the GPU driver but things didn’t change at all.

Information that may be useful and the whole error messages are at the end of this report.

Any help or comment would be appreciated.
Thanks in advance.

・ CUDA toolkit ：ver. 10.0
・graphics driver ：ver. 441.12 (although the default version in my device was 442.05, I did custom clean install via 411.12 installer.).

・the result I got when I executed ‘nvidia-smi’.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 442.05 Driver Version: 441.12 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 208... WDDM | 00000000:01:00.0 Off | N/A |
| 30% 46C P8 36W / 250W | 310MiB / 11264MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

・the error I got
This error didn’t appear when I set the smaller sample size like gridsize 16x16x2, cellsize 8x8x1.

panic: CUDA_ERROR_LAUNCH_FAILED goroutine 1 [running, locked to thread]: github.com/mumax/3/cuda/cu.Stream.Synchronize(0x0) C:/Users/Usename/go/src/github.com/mumax/3/cuda/cu/stream.go:41 +0x5e github.com/mumax/3/cuda.Sync(...) C:/Users/Usename/go/src/github.com/mumax/3/cuda/init.go:72 github.com/mumax/3/cuda.MemCpyDtoH(0xc01e762000, 0xb09200000, 0x4000000) C:/Users/Usename/go/src/github.com/mumax/3/cuda/slice.go:40 +0x35 github.com/mumax/3/data.Copy(0xc0000647c0, 0xc000096f00) C:/Users/Usename/go/src/github.com/mumax/3/data/slice.go:222 +0x316 github.com/mumax/3/data.(*Slice).HostCopy(0xc000096f00, 0x10a8cc0) C:/Users/Usename/go/src/github.com/mumax/3/data/slice.go:203 +0x93 github.com/mumax/3/engine.(*magnetization).SetInShape(0x10a64c8, 0xc0001ad110, 0xc0001ad350) C:/Users/Usename/go/src/github.com/mumax/3/engine/magnetization.go:99 +0xe0 reflect.callMethod(0xc000065e40, 0xc0000ada10, 0xc0000ad9f8) C:/Users/Usename/scoop/apps/go/current/src/reflect/value.go:710 +0x1f3 reflect.methodValueCall(0xc0001ad110, 0xc0001ad350, 0xc0001ad350, 0xc0000a4470, 0xc0000adc48, 0x499072, 0xc002388090, 0xc000065e40, 0xc0000a4470, 0x1000000010, ...) C:/Users/Usename/scoop/apps/go/current/src/reflect/asm_amd64.s:35 +0x49 reflect.Value.call(0x933fe0, 0xc000065e40, 0x13, 0x9db454, 0x4, 0xc0001acb70, 0x2, 0x2, 0x3, 0xc00123a2d0, ...) C:/Users/Usename/scoop/apps/go/current/src/reflect/value.go:460 +0x8b2 reflect.Value.Call(0x933fe0, 0xc000065e40, 0x13, 0xc0001acb70, 0x2, 0x2, 0x920ee0, 0xc000098120, 0xc000098118) C:/Users/Usename/scoop/apps/go/current/src/reflect/value.go:321 +0xbb github.com/mumax/3/script.(*call).Eval(0xc0001ad740, 0x7a84a9, 0xc000018f70) C:/Users/Usename/go/src/github.com/mumax/3/script/call.go:61 +0x275 github.com/mumax/3/script.(*BlockStmt).Eval(0xc0001ad1a0, 0x9201e0, 0x8a2d41) C:/Users/Usename/go/src/github.com/mumax/3/script/blockstmt.go:35 +0x68 github.com/mumax/3/script.(*forStmt).Eval(0xc000068460, 0x7a84a9, 0xc000018f58) C:/Users/Usename/go/src/github.com/mumax/3/script/for.go:15 +0x57 github.com/mumax/3/script.(*BlockStmt).Eval(0xc0001ad0e0, 0x9201e0, 0x8a2d41) C:/Users/Usename/go/src/github.com/mumax/3/script/blockstmt.go:35 +0x68 github.com/mumax/3/script.(*forStmt).Eval(0xc000068410, 0x1, 0x1) C:/Users/Usename/go/src/github.com/mumax/3/script/for.go:15 +0x57 github.com/mumax/3/engine.EvalFile(0xc0001acba0) C:/Users/Usename/go/src/github.com/mumax/3/engine/script.go:102 +0x52 main.runScript(0xc0000180c0, 0x8) C:/Users/Usename/go/src/github.com/mumax/3/cmd/mumax3/main.go:121 +0x11a main.runFileAndServe(0xc0000180c0, 0x8) C:/Users/Usename/go/src/github.com/mumax/3/cmd/mumax3/main.go:92 +0x6e main.main() C:/Users/Usename/go/src/github.com/mumax/3/cmd/mumax3/main.go:59 +0x180
@JeroenMulkers
Copy link
Collaborator

JeroenMulkers commented May 5, 2020 via email

@tuberosa
Copy link
Author

tuberosa commented May 5, 2020 •
edited

Thanks for your quick response!!

Since I had gotten an error when I tried to use the binary on mumax ver. 3.10 beta some time ago, I stopped to use the binary and decided to recompile by myself.

However, when I downloaded the binary of 3.10 beta for win right now and ran, it didn't show the error which I'd encountered before.

Instead I got errors like below which look essentially the same as the above.

Thanks in advance.

panic: CUDA_ERROR_LAUNCH_FAILED

goroutine 1 [running, locked to thread]:
github.com/mumax/3/cuda/cu.Stream.Synchronize(0x0)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/cuda/cu/stream.go:41 +0x72
github.com/mumax/3/cuda.Sync()
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/cuda/init.go:71 +0x31
github.com/mumax/3/cuda.MemCpyDtoH(0xc00a2cc000, 0xb09200000, 0x4000000)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/cuda/slice.go:40 +0x29
github.com/mumax/3/data.Copy(0xc0022ac370, 0xc002257ea0)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/data/slice.go:226 +0x39c
github.com/mumax/3/data.(*Slice).HostCopy(0xc002257ea0, 0x472b82)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/data/slice.go:207 +0x8e
github.com/mumax/3/engine.(*magnetization).SetInShape(0xf414b8, 0xc0001528a0, 0xc000153170)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/engine/magnetization.go:94 +0xc7
reflect.callMethod(0xc0001531d0, 0xc0011ab990)
        C:/gosdk/src/reflect/value.go:663 +0x187
reflect.methodValueCall(0xc0001528a0, 0xc000153170, 0xc00005c800, 0xc0021bc140, 0xc0011abc78, 0x4c6af0, 0xc000094510, 0xc0001531d0, 0xc00006b6b0, 0x1000000010, ...)
        C:/gosdk/src/reflect/asm_amd64.s:29 +0x3a
reflect.Value.call(0x8db620, 0xc0001531d0, 0x13, 0x97dd76, 0x4, 0xc0001523c0, 0x2, 0x2, 0xc000153140, 0x8c7140, ...)
        C:/gosdk/src/reflect/value.go:447 +0x450
reflect.Value.Call(0x8db620, 0xc0001531d0, 0x13, 0xc0001523c0, 0x2, 0x2, 0x0, 0x0, 0x0)
        C:/gosdk/src/reflect/value.go:308 +0xab
github.com/mumax/3/script.(*call).Eval(0xc0001539b0, 0x0, 0x0)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/script/call.go:61 +0x29b
github.com/mumax/3/script.(*BlockStmt).Eval(0xc0001530b0, 0x8c6380, 0x84be61)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/script/blockstmt.go:35 +0x68
github.com/mumax/3/script.(*forStmt).Eval(0xc0000ae500, 0x757fb9, 0xc00118e620)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/script/for.go:15 +0x57
github.com/mumax/3/script.(*BlockStmt).Eval(0xc000152ff0, 0x8c6380, 0x84be61)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/script/blockstmt.go:35 +0x68
github.com/mumax/3/script.(*forStmt).Eval(0xc0000ae4b0, 0x1, 0x1)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/script/for.go:15 +0x57
github.com/mumax/3/engine.EvalFile(0xc000152ab0)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/engine/script.go:102 +0x52
main.runScript(0xc00008a080, 0x34)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/cmd/mumax3/main.go:118 +0x106
main.runFileAndServe(0xc00008a080, 0x34)
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/cmd/mumax3/main.go:89 +0x6e
main.main()
        C:/Users/Mykola Dvornik/go/src/github.com/mumax/3/cmd/mumax3/main.go:56 +0xe8

@tuberosa
Copy link
Author

tuberosa commented May 5, 2020

(I edited the comment just above because I'd misunderstood the word 'binary' before edition. Sorry!)

@tuberosa
Copy link
Author

tuberosa commented Sep 4, 2020

I'm sorry to say that this was a matter of HARDWARE -- GPU.
Since Mumax once worked fine in my PC and the GPU problem occurred a few weeks after I started using it, I was excluding possibility of a bug in hardware, but surprising for me, it WAS.

I found an error when I ran a benchmark on a PC, and replacing GPU worked fine.

@tuberosa tuberosa closed this as completed Sep 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JeroenMulkers @tuberosa