Use soft links instead of the target attribute

[vasole]

It solves nexusformat/definitions#583
It solves nexusformat/definitions#842

Discussion and examples found at nexusformat/definitions#807

As you can appreciate, the use of soft links solves issues in a very clean manner. In fact, those issues would never have existed if instead of using hard links, soft links would have been used.

The target attribute should be limited to backends where the soft link functionality does not exist.

[vasole]

In the file http://ftp.esrf.fr/pub/bliss/lima.h5 you will see soft links to the data at two places. Inside an NXcollection and inside an NXdata. The NXcollection could also be an NXdata but in our most generic case it could not.

The use of soft links allows us to reach the NXdetector group whatever path the user takes to select the detector data.

That file is part of an acquisition and not all of it. That file is accessed from a top master file containing external links and virtual datasets putting together all the parts of the acquisition (motors, counters, different detectors,...) therefore use of attributes and hard links is really dangerous for us.

[rayosborn]

I support this change, although I think the NIAC needs to spend some time in thinking through all the implications. There could be side effects, e.g., for third-party software that utilizes the target attribute for identifying the NXdetector group. That shouldn't be a problem for those using the nexusformat package, because I will make it backwardly-compatible, but without the NAPI, it will be impossible for other languages.

I can make the next release of nexusformat compliant with this request by adding a soft keyword to the NXlink initialization. If NIAC approves this, then I can switch the default value of soft from False to True. It will still be backwardly compatible for files that have the target attribute set.

[benajamin]

I have a vague memory that some people didn't like the way soft links work. From the HDF5 users guide:

HDF5 also has Soft Links similar to Unix soft links. A Soft Link is an object that contains a name and a path name for the target object. The Soft Link can be followed to open the target of the link, just like a regular (hard) link. Unlike hard Links, the target of a Soft Link has no count of the Soft Link to it. The reference count of an object is the number of hard Links (which must be >= 1). A second difference is that the hard link cannot be created if the target object does not exist, and always points to the same object. A Soft Link can be created with any path name, whether or not the object exists. Therefore, it may or may not be possible to follow a Soft Link, or the target object may change from one access to another access of the same Soft Link.

I see two issues:

1. soft links can be broken (go to a non-existent target)
2. a dataset doesn't know what/how many soft links target it

The second issue means you can't modify a dataset without risking an inappropriate change do to an unknown soft link.

[rayosborn]

Both issues are true, but I'm not sure that they invalidate the proposal.

1. Soft links can be broken, but so can external links. Also, there is no guarantee that, if you use a hard link, the target referenced by the hard link exists, so any software following the link will generate an exception.

2. The second issue applies to hard links as well. There is no way of knowing how many hard links point at any particular target without traversing the entire file. There is no reverse-lookup table. The same is true for soft links.

[benajamin]

The way I read the HDF5 users guide excerpt above, you are incorrect on both of your statements. The first is certainly not particularly clear, but the second is pretty explicit refuted by the statement:

The reference count of an object is the number of hard Links
So you can easily know that there are hard links to a dataset. Though you do have a point that we can't easily find where the hard links are coming from.

[rayosborn]

These are important issues to discuss and I don't think we should rush into this change. I will test this in the Python API for a while to see if any issues arise.

[vasole]

@benajamin

Both statements of @rayosborn are correct.

1 - External links can be as broken as soft links. Just create an external link and move one of the two files.
2 - It is also true. You have two independent, unrelated paths to the same buffer.

The only objection for a 100% replacement is also given by @rayosborn. Software exclusively relying on the target attribute would need to be upgraded to contemplate the soft link alternative.

As I see things, having the choice to select soft links or the target attribute allows the data producer to take the decision based on "Is software using the target attribute going to be used?" or "Do I need a solution to nexusformat/definitions#583 and nexusformat/definitions#842?"

I signal that ANY solution to those issues would imply additions or modifications to software in order profit from them. Therefore the main objection of having to modify other software does not apply to people affected by those issues.

[vasole]

Just to add that external links can be created without need for the target to exist. At the ESRF we profit from that. The sequencer can create the external links to the expected files (and targets) created by the different acquisition actors without needing to open the target file thus not blocking the acquisition nor corrupting any file.

The snippet below creates a soft link, a hard link and a broken external link.

import h5py
h5 = h5py.File("broken_external.h5", "w")
h5["entry/detector/data"] = 10.
h5["entry/measurement/soft"] = h5py.SoftLink("/entry/detector/data")
h5["entry/measurement/hard"] = h5["/entry/detector/data"]
h5["entry/measurement/broken_external"] = h5py.ExternalLink("non_existing_file", "/entry/detector/data")
h5.flush()
h5.close()

[mkoennecke]

If my memory serves me right, then both hard links and the target attribute came about for technical reasons:

• Initially, and in HDF4, there were no soft links.
• There was no way to back trace the link to get to the group from which data was linked to with older
  versions of the HDF5 API
• Linking is simulated in XML through the NAPI using the NAPlink and target attributes

Many of these restrictions have gone away with newer versions of the HDF-API's.

I still find that hard links are safer in that the linked to object has to exist. So I would still recommend to use hard links wherever possible but allow the use of soft links as an option.

[prjemian]

To avoid confusing people new to NeXus, is this how we advise about links?

• NeXus recommends to use hard links and the @target attribute whenever possible
• NeXus allows to use soft links as an option

How and why does one (new to NeXus) make this choice?

[vasole]

Whenever one is potentially dealing with external links, soft links is the preferred solution.

[prjemian]

The target attribute is used to resolve ambiguity when copying data from a NeXus file. It avoids needless duplication of data.

[prjemian]

@vasole said:

The target attribute points to the richest location with associated information.

[prjemian]

And that's a good addition for the manual now.

[vasole]

Proposal:

• Any type of link (Soft, Hard and External) is allowed.

• The use of the target attribute is optional when using soft links.