Deflate64 zip-files not supported by dynamic-sidecar powered services #2843
Description
Long story short
A user created a large zip-file using the windows explorer native capabilities. Appareantly, the algorithm used for the zip-file is ""Deflate64". The file is connected to a jupyter notebook. In the jupyter, the folder-structure of the zip-file is present in the inputs but the actual contents are missing.
The reason is that the unzipping routine does not support the "deflate64" algorithm: https://stackoverflow.com/questions/12809651/opening-zipfile-of-unsupported-compression-type-silently-returns-empty-filestrea
Expected behaviour
Either: The file is successfully unzipped
Or: A verbose and clear error is presented to the user, suggesting she may use a different zipping algorithm
Actual behaviour
No error, folder structure of zip-file is present but contents are missing
Steps to reproduce
Upload a deflate64 zip file to osparc using the filepicker, connect it to a jupyter
Your environment
osparc dalco prod
Logs from the jupyter
transfer completed in 93.63766647316515s
all_dwi.zip: data size: 9791.67063999176MB, transfer rate 104.56978488243162MB/s
completed download [(Port(display_order=1.0, label='input_files_1', description='Any input files. One or serveral files compressed in a zip will be downloaded in an inputs folder.', property_type='data:*/*', file_to_key_map=None, unit=None, key='input_1', widget=None, default_value=None, value=PortLink(node_uuid='a3a528d4-4dcd-46ba-af48-407e29a8175f', output='outFile')), PosixPath('/tmp/simcorefiles/input_1/all_dwi.zip')), (Port(display_order=2.0, label='input_files_2', description='Any input files. One or serveral files compressed in a zip will be downloaded in an inputs folder.', property_type='data:*/*', file_to_key_map=None, unit=None, key='input_2', widget=None, default_value=None, value=None), None), (Port(display_order=3.0, label='input_files_3', description='Any input files. One or serveral files compressed in a zip will be downloaded in an inputs folder.', property_type='data:*/*', file_to_key_map=None, unit=None, key='input_3', widget=None, default_value=None, value=None), None), (Port(display_order=4.0, label='input_files_4', description='Any input files. One or serveral files compressed in a zip will be downloaded in an inputs folder.', property_type='data:*/*', file_to_key_map=None, unit=None, key='input_4', widget=None, default_value=None, value=None), None)]
creating directory /home/jovyan/work/inputs/input_1
unzipping /tmp/simcorefiles/input_1/all_dwi.zip
Unexpected problem when processing retrieve call
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/concurrent/futures/process.py", line 239, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/opt/conda/lib/python3.7/site-packages/servicelib/archiving_utils.py", line 50, in _zipfile_single_file_extract_worker
with zf.open(name=file_in_archive) as zip_fp:
File "/opt/conda/lib/python3.7/zipfile.py", line 1560, in open
return ZipExtFile(zef_file, mode, zinfo, pwd, True)
File "/opt/conda/lib/python3.7/zipfile.py", line 809, in __init__
self._decompressor = _get_decompressor(self._compress_type)
File "/opt/conda/lib/python3.7/zipfile.py", line 722, in _get_decompressor
raise NotImplementedError("compression type %d (%s)" % (compress_type, descr))
NotImplementedError: compression type 9 (deflate64)
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/jupyter_commons/handlers/retrieve.py", line 37, in post
transfered_size = await _input_retriever.download_data(ports)
File "/opt/conda/lib/python3.7/site-packages/jupyter_commons/handlers/_input_retriever.py", line 149, in download_data
archive_to_extract=downloaded_file, destination_folder=dest_path
File "/opt/conda/lib/python3.7/site-packages/servicelib/archiving_utils.py", line 108, in unarchive_dir
extracted_paths: List[Path] = await asyncio.gather(*tasks)
NotImplementedError: compression type 9 (deflate64)