silx.gui.hdf5 and HDF5_USE_FILE_LOCKING

[kklmn]

Hello,

The latest version of silx has broken my application (ParSeq) where I join together QFileSystemModel with silx.gui.hdf5.Hdf5TreeModel. All external links are seen as faulty tree nodes. I see this new behaviour with silx 2.0.0 but not with 1.1.2 on the same Python 3.11.6. I cannot confirm the change on Python 3.12.2 because the older silx 1.1.2 is not installable there.

The reason is in line 184 of silx.gui.hdf5.Hdf5Item.py where parent_obj.get(self.__key) returns None for external links. One may say this is an h5py problem, but its version is the same (3.10.0) when I compare silx 2.0.0 with silx 1.1.2.

The workaround I have found is to do

os.environ["HDF5_USE_FILE_LOCKING"] = "FALSE"  # to work with external links

before the import of silx.gui.hdf5. If I also import hdf5plugin before importing silx, I have to do hdf5 file unlocking before importing hdf5plugin. And if I directly use h5py, I also have to do this beforehand.

Question

What is the silx developers’ view on hdf5 file locking and on its interaction with silx.gui.hdf5, hdf5plugin and h5py? Is it "not silx's business"?

Thank you!

[t20100]

Since this was working with silx 1.1.2, it sounds to be a regression in silx v2.

Did you change any other package when switching from silx v1 to v2?
To be complete, what is the version of libhdf5 you are using (h5py.version.hdf5_version)?

Are the HDF5 files you access being otherwise opened for writing or reading?

What is the silx developers’ view on hdf5 file locking

We try to support HDF5's file locking disabled. We even provide some helpers for accessing HDF5 files being written. We are relying on this for online data access (though my personal opinion is that it is not a good idea since we keep facing issues).

In silx view, we do the same "workaround" as you do: set HDF5_USE_FILE_LOCKING before anything imports h5py

silx/src/silx/app/view/main.py

Line 123 in 0abd43e

os.environ["HDF5_USE_FILE_LOCKING"] = hdf5_file_locking

It's been working fine even with changes in libhdf5 related to file locking. The issue is that you should not write files without locking because they can get corrupted if another application is opening them.

[kklmn]

Hi,

Did you change any other package when switching from silx v1 to v2?

No. I've now checked it again on Python 3.11.6: after uninstalling silx and pip install silx==1.1.2:

> h5py.version.hdf5_version
'1.14.2'

And here after commenting out

# os.environ["HDF5_USE_FILE_LOCKING"] = "FALSE"  # to work with external links

all external links are still alive.

After upgrading silx to 2.0.0, h5py.version.hdf5_version is still '1.14.2'. And all external links are broken.
Doing

os.environ["HDF5_USE_FILE_LOCKING"] = "FALSE"  # to work with external links

restores the links.

Are the HDF5 files you access being otherwise opened for writing or reading?

No, these are offline data files, accessed only by silx.gui.hdf5.Hdf5TreeModel.

[t20100]

I tried to find out the cause, and nothing changed in silx.gui.hdf5 that could explain this...

Could you try to browse the same files with silx view --hdf5-file-locking <main_file.h5>?

The --hdf5-file-locking option enables file locking, so it would be a similar test since silx view also uses Hdf5TreeModel, and would be good for comparison.

If this reproduce the error, could you provide us the file? I couldn't reproduce with dummy files with external links

[kklmn]

Certainly, trying my files with silx view was my first move. They behave normally there.

The difference is that in silx view all hdf5 files are at the topmost level, whereas in my model they are nodes of the file tree, like here:

After a few days of searching I still can't see where is the change in silx that breaks the external links.

[t20100]

This change is related to file locking: https://github.com/silx-kit/silx/pull/3939/files#diff-f7305bec0300313b03a14fece7a1e78cdcb4994f086ee18eb467567d212cfc38
But looking at parseq code I don't see how it can cause this issue, but maybe worth reverting to check.

Could you provide a way to reproduce the issue with parseq so I can investigate?

[kklmn]

This change is related to file locking: ...

Yes! The error is gone after having this change reverted.

[kklmn]

Thank you for agreeing to investigate!

1. Please download the latest GitHub version, I've just updated it. You don't need to install it, simply unpack somewhere.
2. In parseq/gui/fileTreeModelView.py comment line 22 out (os.environ["HDF5_USE_FILE_LOCKING"]); this is a workaround for it to work with silx 2.0.0.
3. run parseq/tests/test_fileTreeModelView.py that in a second will navigate to the hdf5 file: parseq/tests/data/hdf5/20240213s.h5
4. Go into instrument and you will see this when the links are alive (except eiger_xes that was not copied):
   
   and with albaem_2d_01, _02 and pcap links broken otherwise.

[t20100]

Thanks for the easy way to reproduce!

This is indeed related to #3939 and to an issue we recently found in libhdf5/h5py regarding file locking and external links: https://forum.hdfgroup.org/t/external-link-and-file-locking-disabled-issue/12012: External links do not inherit file locking from the file linking them (and file locking is enabled by default) and you cannot open a file twice with different file locking in the same process.

I'm still not sure what's the best way out of this at silx level until this is fixed in libhdf5/h5py.
I feel it's always going to cause trouble one way or another unless either os.environ["HDF5_USE_FILE_LOCKING"] = "FALSE" is set throughout the whole application or file locking is always enabled....

attn @woutdenolf

[t20100]

Simple way to reproduce with silx view:

• Create 2 files with an external link:

  import h5py
  
  with h5py.File("main.h5", "w") as h5f:
      h5f["link"] = h5py.ExternalLink("linked.h5", "/data")
  with h5py.File("linked.h5", "w") as h5f:
      h5f["data"] = 1

• Comment:
  

  silx/src/silx/app/view/main.py

  Line 123 in 0abd43e

  os.environ["HDF5_USE_FILE_LOCKING"] = hdf5_file_locking

• Open the two files before accessing the external link: silx view main.h5 linked.h5

[t20100]

@kklmn , PR #4074 that has been merged to main should fix this issue. Could you check that your issue is fixed with the current tip of the project?

We plan on making a 2.1.0 release in a short time.

[kklmn]

@kklmn , PR #4074 that has been merged to main should fix this issue. Could you check that your issue is fixed with the current tip of the project?

Yes, checked, it's fixed.
Thank you!

[t20100]

Thanks for the testing!