Pytorchjob dist-mnist no training logs

[Findlazyfriend]

Hello, guys
As a novice, I encountered a seemingly simple problem. When executing examples/pytorch/mnist/mnist.py, I found that there is no log information after downloading the data, but the master and worker are always running Status, the training process can be displayed normally when debugging locally. This question may seem a bit stupid, I hope you can give pointers, thank you very much.
All outputs :

Using CUDA
Using distributed PyTorch with nccl backend
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz
Processing...
Done!

[johnugeorge]

Do you GPUs with in your cluster?

Can you try Gloo backend?

[Findlazyfriend]

Do you GPUs with in your cluster?

Can you try Gloo backend?

Thank you very much for your reply, the problem has been solved, just always stuck in model.cuda(). It is the problem of the version in my base image.😊

[johnugeorge]

Thanks.
Closing this issue

[N-Kingsley]

Hello, guys As a novice, I encountered a seemingly simple problem. When executing examples/pytorch/mnist/mnist.py, I found that there is no log information after downloading the data, but the master and worker are always running Status, the training process can be displayed normally when debugging locally. This question may seem a bit stupid, I hope you can give pointers, thank you very much. All outputs :

Using CUDA Using distributed PyTorch with nccl backend Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz Processing... Done!

I have met the same problem, could you tell me what happened and how to solve?