ENH: spatial: serialize concurrent calls to QHull

[andfoy]

Reference issue

Closes gh-8856

What does this implement/fix?

This PR introduces a Mutex mechanism to serialize QHull calls, thus preventing potential segfaults or undefined behaviour, since it is not thread-safe. See http://www.qhull.org/html/qh-code.htm#reentrant

Additional information

Depends on #20611

[ngoldbaum]

Too bad about the code churn but putting the lock cleanup in a finally block does make sense.

[ngoldbaum]

Probably worth benchmarking singlethreaded performance since acquiring an uncontested PyThread_type_lock does have some overhead.

[rgommers]

Great, looks like this fixes a long-standing crash in LinearNDInterpolator! The one failure is memory allocation in the 32-bit job:

numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.49 GiB for an array with shape (6250000, 32) and data type float64

Could you skip this test on 32-bit platforms and when not enough memory is available @andfoy? Search the code base for _IS_32BIT and check_free_memory to see the patterns for how.

Also, could you add @pytest.mark.slow? This test takes a while to run, so we don't need to do so in every CI job.

[andfoy]

Thanks for the pointers! @ngoldbaum @rgommers @ev-br, I'll make sure to fix the tests and some minor lock acquisition details

[rgommers]

The diff with whitespace hidden looks quite clean now. Probably about ready to go once the merge conflict is resolved?

[rgommers]

Did you really mean to set the limit to 30 GB? That means it won't be run on machine with 32 GB RAM - i.e., almost never.

I seem to be needing about 10 GB total:

$ ulimit -Sv 10000000
$ python dev.py test -t scipy.interpolate.tests.test_interpnd::TestLinearNDInterpolation::test_threading -m full

It crashes with 8 GB on my machine. I guess that makes sense, since the 32-bit test showed that a single array creation needed about 1.5 GB, and there are four threads here. So the test needs 6 GB, and then there's the memory taken by SciPy itself, other tests, and dependencies.

Ideally I think we set the limit to 10 GB or so, and tweak the test a bit so that it requires 0.5 GB per thread. That way it'll run for most contributors.

Could you please check if the test still reproducibly crashes when you slightly lower the array sizes?

[andfoy]

Let me check it again with lower bounds

[andfoy]

Unfortunately, the lowest values for which we can reliably reproduce the crash case imply a memory consumption of ~5GB

[rgommers]

That seems fine - that'll mean it runs on most developer machines. Let's call this good.

[rgommers]

Let's get this in, seems like everything is working now. Thanks @andfoy!