BrokenProcessPool in fastai.vision.utils.download_images() on Windows

[f0k]

Please confirm you have the latest versions of fastai, fastcore, and nbdev prior to reporting a bug (delete one): YES

Describe the bug

When calling `fastai.vision.utils.download_images()` with a URL list and destination Path, some students of mine* who were running on Windows received a BrokenProcessPool exception (click to expand):

~\Documents\hands on ai\Notebook 7\u7_utils.py in download_all_images(path)
    105         dest = path/classname
    106         dest.mkdir(exist_ok=True)
--> 107         download_images(dest, url_file=csv, max_pics=200)
    108     print("Done.")
    109 

~\miniconda3\envs\handsoai\lib\site-packages\fastai\vision\utils.py in download_images(dest, url_file, urls, max_pics, n_workers, timeout, preserve_filename)
     35     dest = Path(dest)
     36     dest.mkdir(exist_ok=True)
---> 37     parallel(partial(_download_image_inner, dest, timeout=timeout, preserve_filename=preserve_filename),
     38              list(enumerate(urls)), n_workers=n_workers)
     39 

~\miniconda3\envs\handsoai\lib\site-packages\fastcore\parallel.py in parallel(f, items, n_workers, total, progress, pause, threadpool, timeout, chunksize, *args, **kwargs)
    104             if total is None: total = len(items)
    105             r = progress_bar(r, total=total, leave=False)
--> 106         return L(r)
    107 
    108 # Cell

~\miniconda3\envs\handsoai\lib\site-packages\fastcore\foundation.py in __call__(cls, x, *args, **kwargs)
     95     def __call__(cls, x=None, *args, **kwargs):
     96         if not args and not kwargs and x is not None and isinstance(x,cls): return x
---> 97         return super().__call__(x, *args, **kwargs)
     98 
     99 # Cell

~\miniconda3\envs\handsoai\lib\site-packages\fastcore\foundation.py in __init__(self, items, use_list, match, *rest)
    103     def __init__(self, items=None, *rest, use_list=False, match=None):
    104         if (use_list is not None) or not is_array(items):
--> 105             items = listify(items, *rest, use_list=use_list, match=match)
    106         super().__init__(items)
    107 

~\miniconda3\envs\handsoai\lib\site-packages\fastcore\basics.py in listify(o, use_list, match, *rest)
     54     elif isinstance(o, list): res = o
     55     elif isinstance(o, str) or is_array(o): res = [o]
---> 56     elif is_iter(o): res = list(o)
     57     else: res = [o]
     58     if match is not None:

~\miniconda3\envs\handsoai\lib\concurrent\futures\process.py in _chain_from_iterable_of_lists(iterable)
    482     careful not to keep references to yielded objects.
    483     """
--> 484     for element in iterable:
    485         element.reverse()
    486         while element:

~\miniconda3\envs\handsoai\lib\concurrent\futures\_base.py in result_iterator()
    609                     # Careful not to keep a reference to the popped future
    610                     if timeout is None:
--> 611                         yield fs.pop().result()
    612                     else:
    613                         yield fs.pop().result(end_time - time.monotonic())

~\miniconda3\envs\handsoai\lib\concurrent\futures\_base.py in result(self, timeout)
    437                 raise CancelledError()
    438             elif self._state == FINISHED:
--> 439                 return self.__get_result()
    440             else:
    441                 raise TimeoutError()

~\miniconda3\envs\handsoai\lib\concurrent\futures\_base.py in __get_result(self)
    386     def __get_result(self):
    387         if self._exception:
--> 388             raise self._exception
    389         else:
    390             return self._result

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

To Reproduce
Steps to reproduce the behavior:

1. Create a list of URLs, such as:

   urls = ['https://www.bk.com/sites/default/files/03156-5%20EggnormousBurrito%20500x540_CR.png',
           'https://s3.eu-central-1.amazonaws.com/food-truck-data-eu-central-1/media/operator/app/a3c936cad578f6e01b6b54809fc10f1e.jpg',
           'http://www.noracooks.com/wp-content/uploads/2018/03/IMG_6057-2.jpg',
           'https://www.nahundfrisch.at/cache/images/recipes/resizecrop_mm_h370_w782_q100/0ae000020b896f9015780ed9fd21b168.jpg']

2. Import download_images:

   from fastai.vision.utils import download_images

3. Run it:

   from pathlib import Path
   download_images(Path('.'), urls=urls)

Expected behavior
The code should download all images, irrespective of the platform it is run on.

Error with full stack trace
At least for some versions of Windows, it fails with the stack trace given on top. As I lack access to Windows machines, I unfortunately don't know exactly what circumstances must be met, but at least 2 in 250 people were affected.

Additional context
Judging from https://stackoverflow.com/questions/43836876/processpoolexecutor-works-on-ubuntu-but-fails-with-brokenprocesspool-when-r and https://bugs.python.org/issue17874, this is a general problem with the ProcessPoolExecutor running from an interactive shell. Using a ThreadPoolExecutor would be an easy fix. As suggested in #978, this would require adding threadpool=True to the call in https://github.com/fastai/fastai/blob/0e01131/fastai/vision/utils.py#L37. (Sorry, I haven't looked into how to submit a PR to fastai, given that all the code seems to be generated from notebooks, so I'm just reporting the problem and a potential fix here.)
Pinging @jph00 since he wanted to be informed.

(* Just if you're wondering, I've used the image downloader and cnn_learner in a university class to demonstrate what's possible with transfer learning and what isn't, and am of course giving proper credit to the fastai library and the corresponding fastbook chapter! Thank you for the great work!)

[hamelsmu]

This appears to be fixed in #3194. This should be fixed on windows, but if by any chance it is not, please let us know and we will reopen this issue.

I have tested this in Colab: see notebook

I have also tested that this works locally on my mac.

[jph00]

Reopening since I believe this is unrelated to #3194

[hamelsmu]

Sorry about this! I was confused.

[hamelsmu]

Ok! This should be resolved with #3197 please let us know if this fixes things for you!

[PandorasStar]

I actually got this problem and for the rest of the dumb coders like me if a standard

download_images(Path('.'), urls=urls) doesnt work then try adding the following parameter:

download_images(Path('.'), urls=urls, n_workers = 0)