fix(upload-files): prevent crash (DEV-2918)

[jnussbaum]

No description provided.

[linear]

DEV-2918 upload-files: MacPro OS becomes unresponsive

When uploading 35'000 files with the upload-files command, the MacPro OS becomes unresponsive after 14'140 images. This is remarkable, because the batches of 15'000 images could be uploaded without problems.

This is the beginning of the logs:

2023-11-06 13:45:42.412 cli.py              INFO     ****************************************************************************************************************************
2023-11-06 13:45:42.412 cli.py              INFO     *** DSP-TOOLS: Called the action 'upload-files' from the command line
2023-11-06 13:45:42.413 cli.py              INFO     *** DSP-TOOLS version: dsp-tools  git+https://github.com/dasch-swiss/dsp-tools.git40000a85e97d321928a3da1793b2e68d386000fd
2023-11-06 13:45:42.413 cli.py              INFO     *** Location of this installation: /usr/local/lib/python3.11/site-packages/dsp_tools/cli.py
2023-11-06 13:45:42.413 cli.py              INFO     *** CLI arguments:
2023-11-06 13:45:42.413 cli.py              INFO     ***   processed_dir = /Volumes/Thunderbay-1/tmp-real-sgv
2023-11-06 13:45:42.413 cli.py              INFO     ***   nthreads      = 9
2023-11-06 13:45:42.413 cli.py              INFO     ***   server        = dev-02.dasch.swiss
2023-11-06 13:45:42.413 cli.py              INFO     ***   user          = rootexample.com
2023-11-06 13:45:42.413 cli.py              INFO     ***   password      = ********************
2023-11-06 13:45:42.413 cli.py              INFO     ****************************************************************************************************************************
2023-11-06 13:45:42.414 cli.py              INFO     Using DSP server 'https://api.dev-02.dasch.swiss' and SIPI server 'https://iiif.dev-02.dasch.swiss'
2023-11-06 13:45:42.415 upload_files.py     INFO     Found the following pickle files:
2023-11-06 13:45:42.415 upload_files.py     INFO      - processing_result_20230706_221533.pkl
2023-11-06 13:45:42.415 upload_files.py     INFO      - processing_result_20230707_035845.pkl
2023-11-06 13:45:42.415 upload_files.py     INFO      - processing_result_20230707_094017.pkl
2023-11-06 13:45:42.415 upload_files.py     INFO      - processing_result_20230707_153006.pkl
2023-11-06 13:45:42.415 upload_files.py     INFO      - processing_result_20230707_212324.pkl
2023-11-06 13:45:42.415 upload_files.py     INFO      - processing_result_20230708_031244.pkl
2023-11-06 13:45:42.415 upload_files.py     INFO      - processing_result_20230708_090938.pkl
2023-11-06 13:45:42.823 upload_files.py     INFO     Found 35000 files to upload...

And this is what happens after 1 day (see screenshot):

• There are gaps of several hours where nothing happens
• The OS becomes very slow and unresponsive
• The Activity Monitor says that there is enough free CPU and Memory capacity

Image_20231108_083535_448.jpeg