Use a global HDF5 lock (trac: 10464)

[manics]

Opening against develop instead of dev_5_0 in case it deadlocks the demo

See: https://trac.openmicroscopy.org.uk/ome/ticket/10464
Particularly: https://trac.openmicroscopy.org.uk/ome/ticket/10464#comment:11

Testing: Modify the connection details in the following script
https://gist.github.com/manics/b03acf51254b16c30d3b
and run python tables-crash.py 100 100 from 5+ teminals at the same time. This will create 100 tables/script, and for each table will make 100 calls to addData. With a standard HDF5 install (e.g. brew install hdf5 without any custom options) this will probably fail after less than a minute if you connect to a server without this PR. Then try again with this PR merged and it might work. In theory this could be tested against trout-latest and trout-merge if you set the right ports.

If that works then check for deadlocks.

Note this reverts #2621

[joshmoore]

Argh, second full run of (first 5 and then 3) terminals without your patch. From dpkg -l:

ii  libhdf5-7:amd64                                       1.8.11-5ubuntu7                                     amd64
Hierarchical Data Format 5 (HDF5) - runtime files - serial version

I'll keep trying.

[joshmoore]

Another round of 3 shells and one with 5 shells passed. Likely I will have to assume that it's not going to be reproducible on my system. I'm going to add some logging of performance speeds to your script, run before and after your change, and then finish up, since the only thing that concerns me is if there is significant blockage on the single lock. I can certainly see from the code how this change would prevent concurrent access.

[joshmoore]

Just running against the two different versions doesn't show any marked decrease in performance (both roughly 1.7 ms per call to addData). I could get the error to occur, but both with and without your commit my using a multi-threaded version:

...
def runMT(totalopen, totalnew):
    threads = []
    for n in xrange(totalnew):
        t = threading.Thread(target=lambda: run(totalopen))
        threads.append(t)
    for x in threads:
        x.start()
    for x in threads:
        x.join()

At, 100 threads (./tables-crash.py 1 100), 40+ threads will fail with NoneType has no attribute initialize.

[manics]

1. I can reproduce the multi-threaded error (runMT)
2. The original test script works against trout-merge, and fails as expected with trout-latest
3. The unit tests still fail intermittently
   https://travis-ci.org/manics/openmicroscopy/jobs/28783959
   https://s3.amazonaws.com/archive.travis-ci.org/jobs/28783959/log.txt

So I think I've fixed something, just not sure what.

[joshmoore]

Interesting. Guess we may well have some follow-ups...

I'd suggest then that's merge after tomorrow's builds and see if anything arises. If we're worried about the impact on 5.0, it might make sense to put this behind a configuration property so we can ask users to turn in on (or off) depending on errors they're seeing.

[joshmoore]

No Tables-related test failures today. Merging.

[manics]

Il'l hold off rebasing for a few days in case the travis errors reappear.

[joshmoore]

👍

[manics]

--rebased-to #2783