torchvision data downloader aborts if ipywidgets are not available

[jaintj95]

🐛 Bug

I am testing some PyTorch code in Deepnote and the following call to torchvision aborts since the environment doesn't support ipywidgets

test_data = torchvision.datasets.MNIST('../data', train=False, download=True, transform=local_transforms)

To Reproduce

Check the following notebook - https://deepnote.com/project/0b12c588-a1de-4d66-bb0f-fa6b8f9b1b87

ImportError: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-3-c62d245b04eb> in <module>
      5 local_transforms = torchvision.transforms.Compose([local_transform_1, local_transform_2])
      6 
----> 7 test_data = torchvision.datasets.MNIST('../data', train=False, download=True, transform=local_transforms)

/shared-libs/python3.7/py/lib/python3.7/site-packages/torchvision/datasets/mnist.py in __init__(self, root, train, transform, target_transform, download)
     77 
     78         if download:
---> 79             self.download()
     80 
     81         if not self._check_exists():

/shared-libs/python3.7/py/lib/python3.7/site-packages/torchvision/datasets/mnist.py in download(self)
    144         for url, md5 in self.resources:
    145             filename = url.rpartition('/')[2]
--> 146             download_and_extract_archive(url, download_root=self.raw_folder, filename=filename, md5=md5)
    147 
    148         # process and save as torch files

/shared-libs/python3.7/py/lib/python3.7/site-packages/torchvision/datasets/utils.py in download_and_extract_archive(url, download_root, extract_root, filename, md5, remove_finished)
    254         filename = os.path.basename(url)
    255 
--> 256     download_url(url, download_root, filename, md5)
    257 
    258     archive = os.path.join(download_root, filename)

/shared-libs/python3.7/py/lib/python3.7/site-packages/torchvision/datasets/utils.py in download_url(url, root, filename, md5)
     70             urllib.request.urlretrieve(
     71                 url, fpath,
---> 72                 reporthook=gen_bar_updater()
     73             )
     74         except (urllib.error.URLError, IOError) as e:  # type: ignore[attr-defined]

/shared-libs/python3.7/py/lib/python3.7/site-packages/torchvision/datasets/utils.py in gen_bar_updater()
     13 
     14 def gen_bar_updater() -> Callable[[int, int, int], None]:
---> 15     pbar = tqdm(total=None)
     16 
     17     def bar_update(count, block_size, total_size):

/shared-libs/python3.7/py/lib/python3.7/site-packages/tqdm/notebook.py in __init__(self, *args, **kwargs)
    246         unit_scale = 1 if self.unit_scale is True else self.unit_scale or 1
    247         total = self.total * unit_scale if self.total else self.total
--> 248         self.container = self.status_printer(self.fp, total, self.desc, self.ncols)
    249         self.container.pbar = self
    250         if display_here:

/shared-libs/python3.7/py/lib/python3.7/site-packages/tqdm/notebook.py in status_printer(_, total, desc, ncols)
    113         if IProgress is None:  # #187 #451 #558 #872
    114             raise ImportError(
--> 115                 "IProgress not found. Please update jupyter and ipywidgets."
    116                 " See https://ipywidgets.readthedocs.io/en/stable"
    117                 "/user_install.html")

ImportError: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html

Expected behavior

Maybe a check can be added that skips call to tqdm if it's not available?

cc @pmeier

[vfdev-5]

@jaintj95 which version of tqdm deepnote provides ?

[jaintj95]

which version of tqdm deepnote provides ?

@vfdev-5 From my environment it seems tqdm version 4.55.1

[jaintj95]

CC @jakubzitny

[fmassa]

We normally have a fall-back from PyTorch that if import tqdm fails we use the PyTorch fallback.

The problem here is that tqdm is available in the system. I'm not yet sure what would be the type of check we would need to add to handle this case, as this looks like a problem with the tqdm installation in the notebook. If you uninstall tqdm in that environment the code should work I believe.

[jakubzitny]

Actually, let me clarify it. tqdm itself, or the version is not a problem. The problem is that ipywidgets are not supported in the environment (and there might be many notebook environments without "trusted" flag on, etc.). So disabling tqdm load explicitly from application code, not just if the import fails, would be the solution here.

There are already some ideas in this thread: tqdm/tqdm#619
Can you try those @jaintj95, please?

[vfdev-5]

@jakubzitny If I understand correctly, tqdm does not work with deepnote no matter the package using it, right ? I think disabling/uninstalling tqdm in the environment may be a better solution...

[pmeier]

disabling/uninstalling tqdm in the environment may be a better solution...

If this is possible, I agree that this should be the way forward. Otherwise we might need to add some way (for example an environment variable) to disable the use of tqdm in torchvision. IMO the fallback defined in torch is probably not a good idea in notebooks, since it would clutter the output.

[jaintj95]

Actually, let me clarify it. tqdm itself, or the version is not a problem. The problem is that ipywidgets are not supported in the environment (and there might be many notebook environments without "trusted" flag on, etc.). So disabling tqdm load explicitly from application code, not just if the import fails, would be the solution here.

There are already some ideas in this thread: tqdm/tqdm#619
Can you try those @jaintj95, please?

Based on my limited knowledge I tried most of the solutions listed on the linked thread but they don't seem to work.

[fmassa]

@jaintj95 a solution like tqdm/tqdm#619 (comment) should help , as it disables the tqdm bar IIUC.

If this doesn't work, it might be because PyTorch now defaults to importing tqdm from tqdm.auto, so you might need to patch it from there

[jaintj95]

@jaintj95 a solution like tqdm/tqdm#619 (comment) should help , as it disables the tqdm bar IIUC.

If this doesn't work, it might be because PyTorch now defaults to importing tqdm from tqdm.auto, so you might need to patch it from there

Thanks. I've already tried the solution you linked. It doesn't work. Let me see if I can figure out the patch.