Make multiprocessing Pool safe in quickquasars

[andreicuceu]

The best practice for using multiprocessing.Pool is within a "with as" statement to safely terminate the pool.

We (with Naim) found a bug when nproc > nfiles. This is the error message at the end of running quickquasars:

Exception ignored in: <function Pool.del at 0x1555439bcf70>
Traceback (most recent call last):
File "/global/homes/a/acuceu/.conda/envs/desienv/lib/python3.9/multiprocessing/pool.py", line 268, in del self._change_notifier.put(None)
File "/global/homes/a/acuceu/.conda/envs/desienv/lib/python3.9/multiprocessing/queues.py", line 378, in put self._writer.send_bytes(obj)
File "/global/homes/a/acuceu/.conda/envs/desienv/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes self._send_bytes(m[offset:offset + size])
File "/global/homes/a/acuceu/.conda/envs/desienv/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes self._send(header + buf)
File "/global/homes/a/acuceu/.conda/envs/desienv/lib/python3.9/multiprocessing/connection.py", line 373, in _send n = write(self._handle, buf)
OSError: [Errno 9] Bad file descriptor

This goes away when using "with multiprocessing.Pool(args.nproc) as pool:" to clean the pool after it's used.

[alxogm]

Tests keeps failing, so I wonder if the issue is that the workflow file is referring to the master branch instead of the main?

desisim/.github/workflows/python-package.yml

Line 28 in 8308fc4

DESISIM_TESTDATA_VERSION: master

@sbailey should we modify the python-package.yml file?

[sbailey]

@alxogm @andreicuceu yes, please update master -> main references, both in .github/workflows/python-package.yml and in the top level requirements.txt file.

[alxogm]

Again one of the test failed but is unrelated to this PR or desisim, since is the same case as the last PR by Hiram, I think @andreicuceu you can merge to master.

[andreicuceu]

Hi @alxogm, you have to do the merge, as I don't have permissions.