Save time-of-flight lookup tables as NeXus files

[nvaytet]

So far, we have been saving the time-of-flight lookup tables to Scipp hdf5 format when storing on disk.
However, this currently contains no metadata indicating which choppers were used etc.

We need a better format, and NeXus is probably the best candidate. It can have NXdisk_chopper directly, and we can save the table values as NXdata. We can also have the source parameters as NXsource.

It makes sense to use the NeXus standard, as it means files could also be read by software other than Scipp.

[celinedurniak]

After discussing with Florence (DREAM instrument scientist), the following metadata should be added to these tof lookup tables:

• date
• frequencies and offsets of all 5 choppers (PSC1, PSC2, OC, BC and T0)

[jokasimr]

I'd like to push back on this a little bit because I think it might be a mistake to make the lookup table format a part of our "API". Why do we need that?

The alternative is to treat it as a cached value internal to our application.
That is, it's only stored to avoid having to recompute it every time it is used, but it's not meant to be persisted as some sort of "source of truth".

Looking at it that way it doesn't make sense to structure the file to "make it readable by other software than scipp". Because why would another software ever read one of our internal cache files?

[jl-wynen]

The alternative is to treat it as a cached value internal to our application.

What do you mean by cached? Eventually, we want the tables to be computed by McStas. So this needs to be done ahead of time, not as part of a reduction workflow.

but it's not meant to be persisted as some sort of "source of truth".

But that is what it will be. There is no easy way to recompute it for a user.

Looking at it that way it doesn't make sense to structure the file to "make it readable by other software than scipp". Because why would another software ever read one of our internal cache files?

We can adjust this statement to be 'by any other software than the current version of scipp'. Our HDF5 format is not meant for long term storage and we may not be able to read files forever.

Also note that we have to include metadata with the file so that we can check that the chopper and source configurations match the data we want to unwrap. This needs some kind of specification that we can rely on long term. And it is useful to be able to look at these configurations without our bespoke reader. E.g., using Silx View.

[nvaytet]

it's only stored to avoid having to recompute it every time it is used

For production when we are in full operations, the plan is to have a library of lookup tables computed with McStas. Computing them could be somewhat expensive and is definitely not something you want to do on-the-fly in your notebook/workflow.

I think it might be a mistake to make the lookup table format a part of our "API"

The NeXus standard is already what we use for reading data, chopper information etc. Not sure why using the same standard would be a bad thing, but maybe I misunderstood?

[jokasimr]

For production when we are in full operations, the plan is to have a library of lookup table computed with McStas. Computing them could be somewhat expensive and is definitely not something you want to do on-the-fly in your notebook/workflow.

That makes a lot of sense, and it begins to explain why we might want to add metadata in the lookup tables.
Is it correct to say that the purpose of the meta data is to be used to determine what lookup table to use in a reduction workflow?

I'm asking because at a glance I don't see why we can't have a library of lookup tables without adding metadata to the tables themselves.

[nvaytet]

Is it correct to say that the purpose of the meta data is to be used to determine what lookup table to use in a reduction workflow?

Yes, that and reproducibility of results.

[jokasimr]

and reproducibility of results.

That's a good point.
But to reproduce the results they still need the exact McStas model, and the script used to run it.
So to achieve reproducibility we need to version control that script.

If we are already version controlling the script, it seems to me we like can enable reproducibility by only storing the version (the commit hash) of the script used to produce the lookup tables.

[jl-wynen]

So to achieve reproducibility we need to version control that script.

And the version of McStas.

[jl-wynen]

I had a go at this. Writing choppers is doable but non-trivial. E.g., it requires turning the axle position back into an NXtransformations group. Ideally, we should implement some helpers in ScippNexus and ScippNeutron.

Also, I think we should turn TimeOfFlightLookupTable from an alias of DataArray to a dedicated class. This would make it easier to store metadata like source and chopper parameters.

But before implementing any of this, what do you think of this idea? And of this NeXus structure: (incomplete, missing source and simulation params)

[jokasimr]

Is the idea that the lookup tables should contain the information needed to recreate them?

In that case I think we should try to make them contain only the minimal information needed to reproduce them.
For example:

version: tof-generation-repo@25.06.01
command: create-tof-lookup-table --instrument dream --chopper-mode high-flux --pulse-skipping 1
tof_lookup_table: ...

Otherwise it's very easy to end up in a situation where the files contain lots of information that is still not sufficient to actually reproduce the table, because it's missing some detail.
Besides, if the chopper information is stored in both the McStas script and in the tof lookup table, what is the source of truth? They should of course agree, but if they do, why store the information twice?

[jl-wynen]

Besides, if the chopper information is stored in both the McStas script and in the tof lookup table,

What about simulations done with the tof package? We also need a way to store those choppers.

what is the source of truth?

The file with the table. It should be self-contained.

But here is my main question: If you are given a NeXus file with a measurement and you select a file with a lookup table, how can you be sure, and I mean sure, that the table can be applied to that data? Are you going to compare the NeXus chopper information to the McStas instrument model and parameter files?
We have two dissociated pieces of data (the measurement file and the table file) and have to make sure that they can work together.