SANS GUI: new user file format

[raquelalvarezbanos]

This issue is to make it possible to save/load all the information in the GUI to a file.

[gemmaguest]

Currently it is possible to save the table, but users want to be able to save all of the information from all tabs in the GUI, including the table, into a single file.

[gemmaguest]

We should look at this together with SANS, who would like something similar - they currently load settings from user files, but can't save state from the GUI. They also want a better format, e.g. json or yaml. There are some notes on this in the Users Workshop minutes (see the the User files and File storage sections)

[edwardb96]

Discussion with @Mantid-Matthew has suggested that SANS are keen to use the same file format as Reflectometry. Saving to a file from the SANS GUI is not however a high priority item. So it is likely that Reflectometry will be the first to implement this. The SANS scientists would like to be kept in the loop and involved in discussions over which format is used.

Matthew has identified the following additional requirements from the SANS scientists which may or may not be requirements for reflectometry as well:

• Ability to open the file in a simple text editor rather than something like hdfview.
• Ability to comment out and change sections of the file.

Bearing these in mind we considered the following formats to start.

XML

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<root>
  <foo>bar</foo>
  <numbers><!-- A comment -->
    <number>1</number>
    <number>2</number>
    <number>3</number>
  </numbers>
  <subobject some_value="true">
    <subsubobject some_other_value="false"></subsubobject>
  </subobject>
</root>

• Hardest to read
• <!-- to open a comment --> to close.
• Largest file size.
• Libraries available in Mantid on C++ and Python side.

JSON

{
  "foo" : "bar",
  "numbers": [ 1, 2, 3 ],
  "subobject": {
     "some_value": true,
     "subsubobject": {
       "other_value": false
     }
  }
}

• Heavily Punctuated
• Cannot comment things out.
• Libraries available in Mantid on C++ and Python side.

YAML

---
foo: "bar"
numbers: # A comment
  - 1
  - 2
  - 3
subobject:
  some_value: true
  subsubobject:
    other_value: false
...

• Less punctuated than JSON or XML.
• Whitespace denotes strucure - good for readability, not so much for reducing user errors.
• Some complex/obscure syntax in places.
• Libraries available in Python but more limited options for C++ which is a problem for reflectometry.

TOML

foo = "bar"
numbers = [ 1, 2, 3 ] # A comment

[subobject]
some_value = true

  [subobject.subsubobject]
  other-value = false

• Extended + formally specified version of INI.
• Comments using #
• Used by Cargo, Rust's package manager.
• Less mature than others on this list.
• Packages available for Python and Libraries (Incl two modern header only libraries) available for C++.

Based on the above, my personal recommendation would be to use TOML, subject to further investigation and discussion with the instrument scientists.

As an additional note, it is possible that when implementing this we could consider adding a solution to the feature:

Jos would like to be able to load them from a nexus file which could be created on ICat or local directory
in mantid's path on the fly and found by the GUI just by specifying its name. Ideally it would then also
populate the text box which could then be edited by hand.

from #21167.

[smk78]

A little historical perspective from the SANS side...

So the way we reduce data in Mantid from a GUI essentially emulates how we used to do things in our previous (VMS based) data reduction programe 'COLETTE' (the name being a corruption of COrrect Laboratoire d'Electronique des Technologies de l'Information, which would have been COLETI; COLETTE just seemed nicer and more memorable. We'd probably get into trouble for it these days!!! LETI made the first LOQ detector). COLETTE actually sat on top of VMS GENIE (the forerunner to OpenGenie used in SECI), which it used for data i/o and plotting. But because GENIE was written by the crystallographers/spectroscopists it knew nothing about 2D detectors, so we (principally Ray Osborn, Richard Heenan & myself) had to create new routines and command sets to augment GENIE to reduce LOQ data. At the same time that gave us the flexibility to employ the VMS screen management library to create what, today, people would recognise as a GUI. But the only human input device back then was a keyboard, so we needed an efficient way of setting up all the reduction parameters. That became a file of COLETTE/GENIE commands we called a 'Mask File'. And it worked very well. So when Mantid came along we had the same concept implemented. The only change has been that for technical reasons we now call the Mask File a 'User File'. But the bottom-line is this is a concept that has been around 30+ years and which our user community understand and are comfortable with.

Essentially the User File is a SANS reduction configuration file for Mantid. I'll attach a LOQ example and a SANS2D example. As you'll see, there are many similarities, but also many differences. And now we have LARMOR and ZOOM too. So the time has definitely come to impart some structure on these files! But they absolutely, most definitely, have to be human-readable, and editable in nothing more sophisticated than Notepad (if really needed; I take the point that a language-sensitive editor like, say, Notepad++ might convey many advantages).

The User Files are generally created on an experiment-by-experiment basis (usually by copying the last one, renaming it, and making some subtle changes), and any single experiment may have several, depending on the sophistication of the reduction. We give the User Files to the users with their data. Equally, many users will copy/edit their own versions. So LOADING these files into the GUI is critical.

Creating a User File by SAVING the state of the GUI is not something which, by its absence, we have ever found to be a problem (after all, how hard is it to copy a file, rename it, and edit it!), but it would be slightly less laborious than the current need to keep reloading an edited User File when you are testing changes. So we would welcome such a feature. It's just not a high priority for SANS.

USER_LOQ_174M_M4_Wang_12mm_Changer_MAIN.txt

USER_SANS2D_174K_2p4_4m_M3_Barlow_8mm_Changer.txt

[smk78]

In terms of a new format for User Files, I had been wondering about something not unlike a a Windows .INI file. But maybe that is too basic?

We use XML for one of our output data formats, and it presents quite nicely, but it is 'wordy'.

I have come across JSON and am not a big fan. I don't find the syntax/structure very user-friendly.

YAML I've heard of but have no experience. TOML I've never heard of!

Does someone want to take one of the User Files I just uploaded and TOML-ise it as a discussion example?

[gemmaguest]

Re-worded to avoid confusion with project save/load which the linked issue was not about:

Re-opened as a SANS issue as this contains relevant information. The SANS requirement is mainly to replace the existing file format which is used for populating the GUI settings from a User File. We should check if creating/updating a User File from the GUI is required (issue #3058 originally requested to do that but I believe it is a low priority compared to loading).

[smk78]

In the case of ISIS SANS much of the information populating the GUI is, of course, loaded in from the User File, which NR do not (yet) have. So, yes, moving the User File format to TOML is the greater priority for SANS.
The question that then arises is, what other information does saving the GUI state record that is not in our User Files? Run numbers in the reduction table are an obvious one, but these would be captured by the proposed changes to Project Save/Load. Is there anything else of particular note?

[gemmaguest]

I've reworded my previous comment as it's about save/load of a User File rather than project save/load. Here's an attempt to summarise the differences, and where we're up to:

Project save/load/recovery
This is concerned with restoring the full state of the GUI, including anything in the table, the rows' processed states, any outputs e.g. from the beam centre finder, etc. etc. It would be for backup/recovery so would not be intended to be user-readable. This is implemented for ISIS Reflectometry and was relatively straightforward to just scrape everything directly off the GUI.

User File save/load
The User File is more concerned with the configuration settings for a particular reduction. Swapping out the parser used so that we can load from a different file format such as TOML should hopefully be relatively straightforward.

Saving is less straightforward due to the requirement to preserve user-friendly formatting and comments. I don't think in the existing SANS workflow we would expect to create a new User File from scratch using the GUI? It sounds like saving might be most useful as a way to update an existing User File to avoid having to re-load it after making changes. If that's the case, maybe a simple Refresh button would be ok, or watching the file for changes so that it automatically updates?

This differs from ISIS Reflectometry workflow where the GUI is the starting point. We have implemented functionality to save the entered information to a "batch" file which contains all of the runs and settings for a specific reduction. This a bit like the SANS User File and Batch file combined, but is not intended to be easily user-readable - we will effectively use the GUI like an editor to load and update it. So, we sort of do have a User File, but it's used in a different way.

[DavidFair]

The majority of this implementation is done, however mapping the TOML syntax to Sans State objects is revealing some unimplemented features / assumptions I'm having to run past

This has lead to concerns from myself and discussed with the instrument scientists about how we might end up releasing something "half-baked". So I'm pushing this off for 5.1 to go into the nightly fairly early on in development

[gemmaguest]

Closing because the new user file is now implemented. There are a couple of issues left in this epic but they are more follow on tasks so I don't think we need the epic any more.