Progress bar doesn't show up on Kaggle TPU with num_workers greater than 0.

[RahulBhalley]

🐛 Bug

As the issue title says: the progress bar doesn't show up on Kaggle TPU with num_workers greater than 0.

Disclaimer: I haven't tested this program on Google Colab TPU.

To Reproduce

Set num_workers to any number greater than zero up to max CPU cores. On Kaggle, the following code sets it to 4.

train_dataset_loader = DataLoader(train_dataset, 
                                  batch_size=BATCH_SIZE, 
                                  shuffle=True, 
                                  num_workers=multiprocessing.cpu_count(),
                                  drop_last=True)

The training is successful but instead of showing progress bar the following output is shown:

2021-10-04 04:35:45.588313: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/callback_hook.py:103: LightningDeprecationWarning: The signature of `Callback.on_train_epoch_end` has changed in v1.3. `outputs` parameter has been removed. Support for the old signature will be removed in v1.5
  "The signature of `Callback.on_train_epoch_end` has changed in v1.3."
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/tpu_spawn.py:192: UserWarning: cleaning up tpu spawn environment...
  rank_zero_warn("cleaning up tpu spawn environment...")

Expected behavior

The progrès bar must show up.

Environment

• PyTorch Lightning Version: 1.4.9
• PyTorch Version: 1.8.0a0+6e9f2c8
• Python version: 3.7.10
• OS (e.g., Linux): Linux
• CUDA/cuDNN version: N/A
• GPU models and configuration: N/A
• How you installed PyTorch (conda, pip, source): pip
• If compiling from source, the output of torch.__config__.show(): N/A
• Any other relevant information: TPU on Kaggle

Additional context

N/A

cc @kaushikb11 @rohitgr7 @tchaton

[Programmer-RD-AI]

ok I will check the issue and check back

[Programmer-RD-AI]

hi can you please send me the "train_dataset" that you are using. thank you

[RahulBhalley]

Here is the data loader function:

def get_dataset_loader():
    transform = transforms.Compose([
        transforms.Resize(IMAGE_SIZE), # the shorter side is resize to match image_size
        transforms.CenterCrop(IMAGE_SIZE),
        transforms.ToTensor(), # to tensor [0,1]
        transforms.Lambda(lambda x: x.mul(255)) # convert back to [0, 255]
    ])
    train_dataset = datasets.ImageFolder(DATASET, 
                                         transform)
    train_loader = DataLoader(train_dataset,
                              batch_size=BATCH_SIZE,
                              shuffle=True,
                              num_workers=multiprocessing.cpu_count(),
                              drop_last=True)
    return train_loader

# Load train dataset
train_dataloader = get_dataset_loader()

Here, DATASET="../input/coco-2017/train2017/". This dataset can be found at this link.

[Programmer-RD-AI]

ok thank you I will check it

[Programmer-RD-AI]

hi I also tried this train loader function with a GPU and Kaggle TPU but even when num_workers is 0 it doesn't give a progress bar https://www.kaggle.com/ranugadisansagamage/notebook353e8aa184 this is the notebook in Kaggle that I used I installed Pytorch 1.8 also is there any differences in this code.

[RahulBhalley]

I can't see the notebook. You haven't saved a version with execution. Btw for me GPU execution is not a problem even if I set CPU cores to max i.e. 4 in case of Kaggle.

[Programmer-RD-AI]

hi I updated the kaggle notebook can you please check it and confirm that the code is correct https://www.kaggle.com/ranugadisansagamage/notebook353e8aa184

[Programmer-RD-AI]

notebook353e8aa184.zip

[Programmer-RD-AI]

notebook353e8aa184.zip

this is the same in https://www.kaggle.com/ranugadisansagamage/notebook353e8aa184

[RahulBhalley]

notebook353e8aa184.zip

There seem to be a training loop there.

My notebooks output when I set num_workers=4 is the following. Looks like whole RAM is consumed whereas in case of GPU this is not an issue at all.

/opt/conda/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/tpu_spawn.py:192: UserWarning: cleaning up tpu spawn environment...
  rank_zero_warn("cleaning up tpu spawn environment...")

[Programmer-RD-AI]

ok I will check this error

[RahulBhalley]

Please try running this notebook on TPU instance. Try to incrementally change values of num_workers. I think around > 2 the RAM starts to get consumed totally and TPU releases the resources and program get's canceled.

[Programmer-RD-AI]

ok I will check it

[Programmer-RD-AI]

hi, I found some similar issues.
Can you check them?
#1112
#207
#Example Repo

Regards

[RahulBhalley]

Hi

1. Can't test #207. The code is incomplete.
2. Modified the example notebook: new one for proper installation of PL, XLA, and other packages. It shows progress bar for NUM_WORKERS from 0 to 2 but not above.
3. #1112 I don't use Google Colab much.

I hope this information is helpful.

Regards,
Rahul Bhalley

[Programmer-RD-AI]

ok I will check the notebook

Regards

[Programmer-RD-AI]

the notebook doesn't exist
it's a 404 error

[RahulBhalley]

the notebook doesn't exist
it's a 404 error

Forgot to save changes after setting visibility to public. Please check it now.

[Programmer-RD-AI]

ok

[RahulBhalley]

@Programmer-RD-AI Btw I am stuck at a problem where I need to implement torch.linalg.svd from scratch in PyTorch. Do you know how to do that? Or are you aware of any correct implementation NumPy or PyTorch (from scratch)?

Sorry, this is not a right place for this question but I feel like I went everywhere on Internet but didn't get anything correct.

[Programmer-RD-AI]

ok I will check If I can find anything

Regards

[Programmer-RD-AI]

hi, I have checked but I couldn't find anything, Sorry
If I find any resource I will add it to this issue

Regards,
Ranuga

[RahulBhalley]

Thank you so much!

[Programmer-RD-AI]

@tchaton

Can you please help with this issue please thank you.

Regards.

[RahulBhalley]

hi, I have checked but I couldn't find anything, Sorry If I find any resource I will add it to this issue

Regards, Ranuga

Btw please look no further if you are trying to. I have found some helpful potential implementations (although not perfect i.e. doesn't give same results in terms of signs but I'll search more).

[Programmer-RD-AI]

ok, I am sorry I couldn't solve the problem.

Regards.

I will try and find a resource for the error.

Regards

[Programmer-RD-AI]

hi can you check the links bellow,

https://www.youtube.com/watch?v=eBZciVDr21o
https://www.kaggle.com/pytorchlightning/pytorch-on-tpu-with-pytorch-lightning
https://pytorch-lightning.readthedocs.io/en/latest/notebooks/lightning_examples/mnist-tpu-training.html
https://pytorch-lightning.readthedocs.io/en/latest/advanced/tpu.html
https://devblog.pytorchlightning.ai/announcing-lightning-1-4-8cd20482aee9
#3649

Regards

[RahulBhalley]

Hi, I've gone through all those references but doesn't look they relate to this issue. 🤔 This issue might have to do something with library itself.

[Programmer-RD-AI]

@tchaton Can you please check this issue?

Thanks.

With best regards,
Ranuga

[Programmer-RD-AI]

Hi, I've gone through all those references but doesn't look they relate to this issue. thinking This issue might have to do something with library itself.

I am new to PyTorch lightning so I am not sure so @tchaton can help.

With best regards,
Ranuga

[kaushikb11]

@RahulBhalley @Programmer-RD-AI I will take a stab at this with GCP TPUs.

[Programmer-RD-AI]

ok, thank you @kaushikb11

With best regards,
Ranuga

[RahulBhalley]

This is issue doesn't really affect my work but just curious if there's been any progress on resolving it.