New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-GPU MNIST example issue with 2 GPUs and FP16 (no CuDNN) #4897

Closed
AlexDBlack opened this Issue Apr 6, 2018 · 5 comments

Comments

Projects
None yet
2 participants
@AlexDBlack
Copy link
Member

AlexDBlack commented Apr 6, 2018

Looks like I'm getting 0 scores, all values predicted as 0 on FP16.
FP32 is fine.

Test machine:
Kubuntu 17.10
GTX970 x2, CUDA 9.1
Verson: 1.0.0-alpha (RC)

Logs: https://gist.github.com/AlexDBlack/4055ed710f82ee1754865ed9a8259902

@raver119

This comment has been minimized.

Copy link
Contributor

raver119 commented Apr 6, 2018

:/

@raver119

This comment has been minimized.

Copy link
Contributor

raver119 commented Apr 6, 2018

Is it reproducible with 1 GPU as well?

@AlexDBlack

This comment has been minimized.

Copy link
Member

AlexDBlack commented Apr 6, 2018

Yeah. 0 scores if I do export CUDA_VISIBLE_DEVICES=0 first.

@raver119 raver119 removed their assignment Apr 26, 2018

@raver119 raver119 added C++ Bug and removed Bug labels Apr 26, 2018

@saudet saudet referenced this issue Jun 6, 2018

Closed

cuDNN & FP16 #1448

@raver119

This comment has been minimized.

Copy link
Contributor

raver119 commented Jun 22, 2018

Fixed quite a while ago.

@raver119 raver119 closed this Jun 22, 2018

@lock

This comment has been minimized.

Copy link

lock bot commented Sep 21, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Sep 21, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.