[CLI] BlockingIOError: [Errno 11] Resource temporarily unavailable on specific Linux distros

[konstantinjdobler]

Describe the bug
I am logging a wandb.Image at the end of every epoch. The docs state that you should log less than 50 at every step, so that should be fine. However I get the following behavior when trying to look at the Images in the wandb web app:

Bildschirmaufnahme.2021-05-08.um.12.34.53.mov

Sometimes the web app also crashes like this after some time

Indeed, I cannot find the images that are said to be "missing" in the video in the logged files. I have the following error in my training run, but the training continues:

So it seems like some error occurs when transferring the files and after that point, no more images are uploaded. However, metrics continue to be logged. Do you have any idea what is going on here and how I could fix this?

[konstantinjdobler]

This issue does not happen when I reduce logging to every 20 epochs

EDIT: No the issue still happens, albeit later on in the training

[ariG23498]

Hey @konstantinjdobler
Could you send in a minimal colab notebook that could help us in reproducing this issue?
Thanks.

[konstantinjdobler]

Hey @ariG23498, I just tried to reproduce the issue and I cannot do it in a Colab notebook. Likely, this is an issue tied to my hardware (Tesla V100 & PowerPC processor arch). I believe the issue happens when the writing of my checkpoint file has not finished when a new checkpoint is being written (and if I just increase the time between checkpointing, the issue goes away). The weird thing is that all uploads of media and checkpoints stop after this BlockingIOError, but I don't know how much this helps you track down the bug.

[cvphelps]

Hi @konstantinjdobler it does look like you're getting rate limited. Could you please try logging predictions with our new Artifacts feature to visualize examples: https://docs.wandb.ai/guides/data-vis

[konstantinjdobler]

I believe to have found the source of the error, as documented in https://bugs.python.org/issue43743

On some Linux Distros (alpine and in my case centOS) there is a bug resulting in BlockingIOError when trying to copy files larger than 65536. The recommended fix is to fall back to another copy procedure by doing

import shuttle
shutil._USE_CP_SENDFILE = False

I'm currently testing if it is sufficient to do this code in my training script or if I need to patch wandb files. I also updated the original issue to reflect this so that (the few) other people with this problem can better find this issue.

[konstantinjdobler]

Since the problem occurs in the python client rather than the Webapps, I've changed the title accordingly. Feel free to also change the label @cvphelps

[konstantinjdobler]

@ariG23498 @cvphelps Unfortunately it is not sufficient to put the shutil fix into my own training script, I have to patch the actual wandb code to get it working, specifically wandb/filesync/step_checksum.py. The fix itself is easy, but I really don't want to have to do it manually on my machines all the time.

Is it possible to insert the fix under an if-statement based on the current operating system (using https://distro.readthedocs.io/en/latest/ package seems to be recommended)?
So something like:

import shutil
import distro
# prevent errors when copying files on certain linux distributions https://bugs.python.org/issue43743
if distro.id() in ["centos", ...]:
  shutil._USE_CP_SENDFILE = False

I can submit a PR if wanted.

Here the full traceback:

Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/site-packages/wandb/filesync/step_checksum.py", line 56, in _thread_body                                                                                                  
    shutil.copy2(req.path, path)
  File "/usr/local/lib/python3.8/shutil.py", line 435, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/lib/python3.8/shutil.py", line 275, in copyfile
    _fastcopy_sendfile(fsrc, fdst)
  File "/usr/local/lib/python3.8/shutil.py", line 172, in _fastcopy_sendfile
    raise err
  File "/usr/local/lib/python3.8/shutil.py", line 152, in _fastcopy_sendfile
    sent = os.sendfile(outfd, infd, offset, blocksize)
BlockingIOError: [Errno 11] Resource temporarily unavailable:

[cvphelps]

@konstantinjdobler great, glad you tracked this down. Could you please put up a PR in this repo and tag @raubitsj and @adrnswanberg to review?

[jinx2018]

I believe to have found the source of the error, as documented in https://bugs.python.org/issue43743

On some Linux Distros (alpine and in my case centOS) there is a bug resulting in BlockingIOError when trying to copy files larger than 65536. The recommended fix is to fall back to another copy procedure by doing

import shuttle
shutil._USE_CP_SENDFILE = False

I'm currently testing if it is sufficient to do this code in my training script or if I need to patch wandb files. I also updated the original issue to reflect this so that (the few) other people with this problem can better find this issue.

This solves my problem.