Cannot download

[andrewjong]

For some reason there is an error related to threads when attempting to download. Any idea what's up?

$ hdfscli download -a dev /data/my_data my_data

ERROR Error while downloading. Attempting cleanup.
Traceback (most recent call last):
File "/home/user/.miniconda3/lib/python3.8/multiprocessing/pool.py", line 212, in init
self._repopulate_pool()
File "/home/user/.miniconda3/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "/home/user/.miniconda3/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
w.start()
File "/home/user/.miniconda3/lib/python3.8/multiprocessing/dummy/init.py", line 51, in start
threading.Thread.start(self)
File "/home/user/.miniconda3/lib/python3.8/threading.py", line 852, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/user/.miniconda3/lib/python3.8/site-packages/hdfs/client.py", line 814, in download
_map_async(n_threads, _download, fpath_tuples)
File "/home/user/.miniconda3/lib/python3.8/site-packages/hdfs/client.py", line 1179, in _map_async
pool = ThreadPool(pool_size)
File "/home/user/.miniconda3/lib/python3.8/multiprocessing/pool.py", line 922, in init
Pool.init(self, processes, initializer, initargs)
File "/home/user/.miniconda3/lib/python3.8/multiprocessing/pool.py", line 216, in init
p.terminate()
AttributeError: 'DummyProcess' object has no attribute 'terminate'
ERROR Unable to cleanup temporary folder.
ERROR Unexpected exception.
Traceback (most recent call last):
File "/home/user/.miniconda3/lib/python3.8/multiprocessing/pool.py", line 212, in init
self._repopulate_pool()
File "/home/user/.miniconda3/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "/home/user/.miniconda3/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
w.start()
File "/home/user/.miniconda3/lib/python3.8/multiprocessing/dummy/init.py", line 51, in start
threading.Thread.start(self)
File "/home/user/.miniconda3/lib/python3.8/threading.py", line 852, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/user/.miniconda3/lib/python3.8/site-packages/hdfs/config.py", line 195, in wrapper
return func(*args, **kwargs)
File "/home/user/.miniconda3/lib/python3.8/site-packages/hdfs/main.py", line 227, in main
client.download(
File "/home/user/.miniconda3/lib/python3.8/site-packages/hdfs/client.py", line 825, in download
raise err
File "/home/user/.miniconda3/lib/python3.8/site-packages/hdfs/client.py", line 814, in download
_map_async(n_threads, _download, fpath_tuples)
File "/home/user/.miniconda3/lib/python3.8/site-packages/hdfs/client.py", line 1179, in _map_async
pool = ThreadPool(pool_size)
File "/home/user/.miniconda3/lib/python3.8/multiprocessing/pool.py", line 922, in init
Pool.init(self, processes, initializer, initargs)
File "/home/user/.miniconda3/lib/python3.8/multiprocessing/pool.py", line 216, in init
p.terminate()
AttributeError: 'DummyProcess' object has no attribute 'terminate'
0

[andrewjong]

:(
@mtth?

[mtth]

Apologies for the delay. I'm not sure what is going on - it looks like it might be related to Anaconda's multiprocessing implementation. Does it work if you pin the number of threads to 1 (--threads=1)?

[qingdo]

Hi, I also encountered a similar issue when using OpenPAI.
This issue happens when we try to mount an HDFS location to OpenPAI location.

Then openpai will automatically insert the following command into the .yaml file before any other commands.

commands:
- '#CUSTOM_STORAGE_START'
- '#Auto generated code, please do not modify'
- >-
pip install hdfs &>> storage_plugin.log && touch ~/.hdfscli.cfg && echo
'[dev.alias]' >> ~/.hdfscli.cfg && echo 'url = http://some.ip.address.here:50070'
>> ~/.hdfscli.cfg
- >-
if [ ! -d /pai_data/data ]; then mkdir --parents /pai_data/data; fi &>>
storage_plugin.log
- hdfscli download --alias=dev /data /pai_data/data
- '#CUSTOM_STORAGE_END'

I believe the error is in the last command. Here is the full error message.

ERROR Error while downloading. Attempting cleanup.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/hdfs/client.py", line 814, in download
_map_async(n_threads, _download, fpath_tuples)
File "/usr/local/lib/python3.6/dist-packages/hdfs/client.py", line 1179, in _map_async
pool = ThreadPool(pool_size)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 789, in init
Pool.init(self, processes, initializer, initargs)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 174, in init
self._repopulate_pool()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 239, in _repopulate_pool
w.start()
File "/usr/lib/python3.6/multiprocessing/dummy/init.py", line 48, in start
threading.Thread.start(self)
File "/usr/lib/python3.6/threading.py", line 846, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
ERROR Unable to cleanup temporary folder.
ERROR Unexpected exception.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/hdfs/config.py", line 195, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/hdfs/main.py", line 233, in main
progress=progress,
File "/usr/local/lib/python3.6/dist-packages/hdfs/client.py", line 825, in download
raise err
File "/usr/local/lib/python3.6/dist-packages/hdfs/client.py", line 814, in download
_map_async(n_threads, _download, fpath_tuples)
File "/usr/local/lib/python3.6/dist-packages/hdfs/client.py", line 1179, in _map_async
pool = ThreadPool(pool_size)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 789, in init
Pool.init(self, processes, initializer, initargs)
File "/usr/lib/python3.6/multiprocessing/pool.py", line 174, in init
self._repopulate_pool()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 239, in _repopulate_pool
w.start()
File "/usr/lib/python3.6/multiprocessing/dummy/init.py", line 48, in start
threading.Thread.start(self)
File "/usr/lib/python3.6/threading.py", line 846, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread