error in training

[nikky4D]

Hi, I encountered this error during training and I'm not sure what it means:

2022-02-09,21:22:00 | INFO | Rank 0 | Train Epoch: 9 [28800/43670 (66%)]        Loss: 0.493029  Data (t) 0.000  Batch (t) 0.235 LR: 0.000020    logit_scale 2.821
2022-02-09,21:22:24 | INFO | Rank 0 | Train Epoch: 9 [32000/43670 (73%)]        Loss: 0.642597  Data (t) 0.008  Batch (t) 0.274 LR: 0.000012    logit_scale 2.822
2022-02-09,21:22:48 | INFO | Rank 0 | Train Epoch: 9 [35200/43670 (81%)]        Loss: 0.442177  Data (t) 0.002  Batch (t) 0.243 LR: 0.000006    logit_scale 2.822
2022-02-09,21:23:13 | INFO | Rank 0 | Train Epoch: 9 [38400/43670 (88%)]        Loss: 0.435208  Data (t) 0.000  Batch (t) 0.255 LR: 0.000003    logit_scale 2.823
2022-02-09,21:23:37 | INFO | Rank 0 | Train Epoch: 9 [41600/43670 (95%)]        Loss: 0.295687  Data (t) 0.000  Batch (t) 0.240 LR: 0.000000    logit_scale 2.823
2022-02-09,21:24:36 | INFO | Rank 0 | Eval Epoch: 10 image_to_text_mean_rank: 40.2243   image_to_text_median_rank: 22.0000      image_to_text_R@1: 0.0628       image_to_text_R@5: 0.2063       image_to_text_R@10: 0.3273      text_to_image_mean_rank: 44.4849     text_to_image_median_rank: 25.0000      text_to_image_R@1: 0.0477       text_to_image_R@5: 0.1817       text_to_image_R@10: 0.2948      val_loss: 0.3798        epoch: 10.0000  num_elements: 6432.0000
Exception in thread Thread-5:
Traceback (most recent call last):
  File "C:\Users\nuzuegbunam\Anaconda3\envs\open_clip_3_9\lib\multiprocessing\connection.py", line 317, in _recv_bytes

Does anyone have any idea what this means?

[gabrielilharco]

Hi @nikky4D, thanks for opening up this issue. Is this the full stack trace? I haven't encountered anything like this before, so I'm wondering whether this could potentially be a mismatch in python / pytorch / cuda versions. Do your versions match those in environment.yml?

Also, could you provide more details on what hardware you are using?

[nikky4D]

For the stack trace: That is all I was given. I was running 10 epochs so this is at the final epoch.
For python/pytorch/cuda: I'm using python 3.9.10, with pytorch 1.10.2, and cuda 11.3.
For environment.yml: The versions don't match with those in the environment.yml as I'm using a windows machine and I couldn't install many of the packages in environment.yml in my conda environment.

[gabrielilharco]

I see, that could be the issue here. Our codebase was only tested with the versions in environment.yml and on unix machines

[nikky4D]

You may be correct. I'm using 4 workers, for batchsize of 32, on a 2080TI. And everything runs well until the last epoch where the after calculating the evaluation metrics, I get that error. It appears to only be affecting the saving of the checkpoint of the epoch directly precedeing the error as I don't seem able to load that checkpoint. I can load other checkpoints.

Anyway, I'll close this until I find a solution that would work.