Clarify status of the nexusformat Python package

[rayosborn]

At the NIAC meeting, I made a presentation concerning the nexusformat package, which has been under active development for more than 10 years since its first version by Paul Kienzle. Full documentation is here and an online tutorial is here. The slides I used in my NIAC presentation are here.

I made the following four proposals.

1. Endorse nexusformat as the official Python binding to NeXus.
2. Update the NeXus manual to use nexusformat in all (Python) examples.
3. Use nexusformat to prototype new NeXus features.
4. Recommend use of nexusformat over h5py in new NeXus software.

I don't think we took a vote on any of these, although there was general agreement that we should add nexusformat examples to the manual. In the code camp, on Github, and in an online Jupyter notebook, we are making steps towards doing that, but there has been pushback from a couple of people, who object to proposals 1 and 4, because they think it threatens their own use of h5py. I disagree that the proposals have any implications for how they work. My focus is on new users, who don't want to spend any more time reading the NeXus manual than necessary. I hope we all agree that it is good to minimize any factors that could undermine future NeXus adoption.

I think we should decide how to proceed at the NIAC follow-up meeting. Even if we don't specifically vote on 1 and 4, I think we need to at least discuss the criteria under which such a decision will eventually be made. For example, we could decide to give committee members a specific time frame to test nexusformat before the NIAC makes a decision. If people identify specific issues with the package, I agree that they should be addressed before a final decision, but I don't think a vote should be put off indefinitely.

[vasole]

I thought it was clear in the discussion following your presentation that equivalent h5py and nexusformat examples would be added to the documentation (point 2) but that we would not go beyond that for the time being.

Point 3 is at the very least questionable since it asks to use nexusformat for prototyping while it does not provide access to all the functionality of h5py. As simple example: you are asking that if one wants to prototype the use of soft links one would have to add them to nexusformat instead of simply making use of them.

I have no objection at all to recommend nexusformat for writing NeXus files in interactive sessions but it does not play on the same league as h5py in what concerns acquisition requirements. Therefore points 1 and 4 are a clear no from me.

[prjemian]

My concern with point 1 is consistent with my views that we should not continue support for the NAPI. This support dilutes our effort to establish and maintain a coherent standard. Further, it risks defining the standard by implementation.

I applaud the nexusformat and encourage such efforts to come from the community without requirement for the NIAC to make a formal endorsement.

[rayosborn]

@vasole, as I recall, my presentation persuaded some that there could indeed be value in having a simplified API like nexusformat and even in promoting it, and that it was worth looking into further. I am requesting that we define what the process of 'looking into it' means, and that it isn't open-ended so that no progress has been made by the next NIAC. I think it is fair to say that a large number of NIAC members were not even aware of what nexusformat could do, so I hope they will have a chance to look at it (if they use Python) in the near future. Check the links in my opening comment.

I have worked to promote the NeXus format for over 25 years, and my work on this package over the last 10 years is mainly inspired by my conviction that a simplified API would be of enormous benefit to the user community, e.g., scientists who don't want to have to understand NeXus arcana (I think it would also benefit some large-scale projects, such as Mantid, because I have read many, many files from multiple reputable sources, not naming names, that seriously violate the standard - even software developers don't want to read manuals). However, whenever I mention the idea of recommending the package, I get pushback from people who don't want people to get the idea that the NIAC actually approves of nexusformat in any way.

I think we should make rational decisions based on:

1. Does nexusformat work as advertised?
2. Does it simplify reading and writing NeXus files, while reducing the number of mistakes people make?
3. Is it difficult to install? Could this actively discourage people from using NeXus?
4. Are there serious limitations compared to using h5py directly that make it unwise to recommend it? There is some discussion of this in another issue.
5. Does encouragement to use nexusformat prevent people who want to from using h5py directly?

I am requesting that the NIAC, at least those who use Python, takes active steps to answer these questions. If the answers to the first two are yes and the answers to the last three are no, then why wouldn't we encourage people to use it? Who loses?

[benajamin]

@vasole

... but it does not play on the same league as h5py in what concerns acquisition requirements.

I find this statement curious since nexusformat would be doing the same things with h5py are your self-written code is. Could you please elaborate?

@rayosborn
I agree with @prjemian that we need to be careful that nexusformat doesn't end up "defining the standard by implementation". A comprehensive test suite to ensure that nexusformat correctly implemented the NeXus standard might neutralise this risk. However, the mission of NeXus is to promote a data format and I think that we should try to stay out of the software business as much as possible, with the exception of file validation. I'm happy to be promoting nexusformat by using it in the manual for code examples and in the listing of utilities, but I think that is all the promotion a software package can ask of the NIAC. nexusformat is a very nice tool, but I think you are asking too much (analogous to asking a SC judge to endorse a presidential candidate). We can ask the NIAC to vote to "endorse nexusformat as the official Python binding to NeXus" if you wish, but I think you'll be disappointed in the result.

[vasole]

@benajamin

The following HDF5 features used by our acquisition system everyday are not supported by nexusformat but are supported by h5py:

• Use of virtual datasets
• Use of soft links (easy addition to nexusformat)
• Direct chunk writing

We still have things with higher priority than to further tune performance via dataset chunk cache but sooner or later it will also come. Targeting simplicity, some of those operations are clearly out of the scope of nexusformat . However, acquisition needs to come down to the bare metal and the closer one gets to the actual thing, the better.

NAPI tried to simplify things and it got abandoned in favor of HDF5.
h5py is the python binding that is closest to HDF5.

[benajamin]

@vasole
You don't have to make an exclusive choice between nexusformat and h5py - the two can be mixed together to utilise each of their strengths.

[vasole]

@rayosborn

However, whenever I mention the idea of recommending the package, I get pushback from people who don't want >people to get the idea that the NIAC actually approves of nexusformat in any way

You are here asking to Endorse nexusformat as the official Python binding to NeXus and Recommend use of nexusformat over h5py in new NeXus software

I find it is not the same.

@benajamin

Yes, you can mix them, but that means that in any case you have to learn h5py.

So, why to endorse something that is not enough? Presenting it as a useful alternative is in my opinion very fair. As a matter of fact, nobody is endorsing h5py either.

[benajamin]

@vasole
Although I think that endorsing either h5py or nexusformat as the "official python binding" is something the NIAC should avoid, I do think that we should be presenting both packages as good choices for python users. We seem to agree on this, but you are trying to support this conclusion with reasons that I don't find logical. I am trying to make it clear to @rayosborn that it doesn't matter how "good" a package is, the NIAC should support the principle of an open market rather than endorsing a single package.

[vasole]

We seem to agree on this, but you are trying to support this conclusion with reasons that I don't find logical.

Well, we seem to have different logic. If I have to learn two APIs for just adding some attributes I can prefer to learn the most complete one. If in addition the less complete API makes some choices I consider dangerous in an acquisition environment, my choice is done for the acquisition environment.

[rayosborn]

I accept that "endorse" may be too strong a word because it can mean many different things to different people. One of the reasons I raised this issue is that NeXus has implicitly "endorsed" the older version of the same nexusformat package, which has been sitting in the language bindings of the NAPI for years without any development. I wanted that implicit "endorsement" to be switched to the new package, that is still actively maintained. Since it looks as if the NAPI might be killed off soon, that issue might be moot, although the existence of the older version still causes some confusion.

If the sphinx tab method of showing both nexusformat and h5py versions of the same code side by side in the manual works, and we agree that it is appropriate to list all the examples in multiple versions, then that is good enough for me. @benajamin's way of stating the issue, i.e., that "we should be presenting both packages as good choices for python users", is an excellent way of saying what I was seeking in the first place. If we all agree on that, then I am satisfied.

[benajamin]

Consensus has been reached. nexusformat is listed in the "NeXus Utilities" section of the manual and further work to include nexusformat versions of the code examples is ongoing in issue nexusformat/definitions#807.