Additional file formats

[NuclearPhoenixx]

Hi,
First of all, great work and it's good to see someone maintaining this great piece of software! I'm really excited to see how this project will advance in the near future, it sure looks like you're working hard on this and have already improved the codebase a lot.

Now to my question, are you currently planning on introducing new file formats to the program? With working on my own MCA software, I've recently had to do a lot with the BecqMoni (2014) XMLs and in principle I really like the aspect of storing all the info with the spectra in a single file. However, I don't really like the XML, because it gets bulky quickly with large file sizes and there is no real documentation (maybe you can help me out on this one?).

So I've come up with a JSON layout that would be well suited for storing the same kinds of data as the XMLs, but can also be validated (i.e. scanned for formatting errors) and is a lot smaller. I've "standardized" the whole layout and created a JSON Schema with the corresponding documentation:

https://github.com/OpenGammaProject/NPES-JSON

I've also started a public thread to discuss this on the GammaSectacular forum. Feel free to also participate/leave feedback on this brain storming thread:
https://www.gammaspectacular.com/phpBB3/viewtopic.php?f=7&t=1042

The goal is to create a file format that can be read from as many gamma spectroscopy programs as possible, while still including meta data like sample info, measurement time, etc. (i.e. more than plain CSVs). What do you think?

[Am6er]

Hi, did you check N42 file format standat? https://www.nist.gov/programs-projects/documentary-standards-ansiieee-n4242-standard

It's quite popular and used across many devices. But it's XML-based too..

[NuclearPhoenixx]

Yes, I know this. I'm not too sure why XML-based formats are this common, probably because it's an old, i.e. tried and tested standard? That's most likely the most important thing for production. Just out of curiosity, why aren't you using these by default?

I just think that these XML files can be pretty hard to read, especially the N42 files (arrays being just inside a single tag). Also, parsing these must be a nightmare. At least with your XMLs, you can read each individual value inside its own tag, especially for arrays.

[Am6er]

N42 standart is quite old and was started since 2006 year - it's an answer, why it is XML-based. It has a very rich collection of attributes for all events, much better than the current BecqMoni format.
BecqMoni has Import/Export N42 files feature.
We are planning to use this standart in last edition by default, it's a matter of time. Many devices (Flir, Canberra) and software (for example InterSpec) use it by default.
You don't need to parse XML - you need to generate transition classes from XSD document and just "cast" incoming XML file to generated classes. Also XSD describes constraints of each attribute (for example attribute1 have MinValue = 0 and MaxValue = 100).

[NuclearPhoenixx]

We are planning to use this standart in last edition by default, it's a matter of time. Many devices (Flir, Canberra) and software (for example InterSpec) use it by default.

I see, that makes sense. Over the years they all had time to implement it.

You don't need to parse XML - you need to generate transition classes from XSD document and just "cast" incoming XML file to generated classes. Also XSD describes constraints of each attribute (for example attribute1 have MinValue = 0 and MaxValue = 100).

Ok, I get it. You can do the exact same thing with JSON Schemas too, while retaining many benefits over the XML files. Especially if you don't need an extremely advanced use case with namespace support or something.

[Am6er]

Also if you are familiar with Python, check this package https://pypi.org/project/becquerel
It allready can work with N42 file format.

[NuclearPhoenixx]

And still, with Python it is much easier to parse JSON as well, as it has the same structure as a standard Python dictionary. You can easily do so without any additional packages. (I know that becquerel is much more than just a file import tool of course)

[Am6er]

I use Python as a backend developer and also like JSON because of its simplicity. The main problem i see, is that we need the major spectrometry equipment manufacturers to support this format. Coding another import/export of the new JSON format in BecqMoni - not a particular problem.

[NuclearPhoenixx]

Yeah, I know, this is a classical chicken and egg problem. And that's also, obivously, the biggest disadvantage of the JSON vs XML format at the moment. However, I think you have to start somewhere, because if you don't then nothing will ever change. What are your thoughts on this? Would you agree or do you think it's a waste of time?

[Am6er]

We may try to JSONify the N42 standard 🤣.
The advantages of this approach are:

• N42 pretty well described in specification datasheet, so we don't need to "reinvent the wheel"
• Everyone knows this format
• It's pretty easy to JSONify existing XML-based N42 file.

[NuclearPhoenixx]

Yes that would also be possible and probably help with adoption. Although, I didn't originally think of this kind of large-scale project 😂

[NuclearPhoenixx]

Oops, I just rediscovered the issue. Looks like I never closed it. Do you want to implement the NPES file format? I can't help you unfortunately, because I have no experience in C#, but I could give you some example files if you want. I'll be honest this is mostly out of self interest to be able to import my own files into BecqMoni. In addition, impulse by GammaSpectacular is now using the file format too, so that would certainly be helpful. As you wish of course, you can definitely close this "issue" now

[Am6er]

@Phoenix1747 of course! Send me NPES files. I don't know exactly time, when i do it, but why not.

[NuclearPhoenixx]

Awesome, thanks! I've attached a bunch of files, had to zip them because GitHub doesn't allow JSON files for whatever reason.

files.zip

There is also an HTML documentation maybe that helps too: https://opengammaproject.github.io/NPES-JSON/

And of course you can have a look at the raw JSON Schema itself: https://github.com/OpenGammaProject/NPES-JSON/blob/main/schema/npes-1.schema.json

[Am6er]

@Phoenix1747 it is necessary to provide for the possibility of storing several spectra in one file (that's why resultDataList exists in BecqMoni)

[NuclearPhoenixx]

Oh okay, I've never actually seen it used that way so I had no idea. Well, thank you for looking into it anyways👍🏻
I'll probably update that in the future to make it possible for multiple spectra to be stored in one file.

[NuclearPhoenixx]

Just revisiting this issue since I created a draft for v2 of the file format and it now includes the possibility to store multiple spectra in one file independently from one another. That implements some suggestions that I received and should resolve this now too. Now you're required to have a data array, which contains an arbitrary number of data packages, which in turn are almost identically formatted like the old standard version. So roughly speaking, you're able to take the data from the old files and just put them in an array inside a single file.

I created the draft branch for this in my repo: https://github.com/OpenGammaProject/NPES-JSON/tree/draft

There are also updated example files if you wanna have a look: https://github.com/OpenGammaProject/NPES-JSON/tree/draft/examples

What do you think?

[Am6er]

Hi @NuclearPhoenixx !
A cursory glance seems to have turned out to be quite a suitable structure. I am currently very busy with other projects and will try to study it more closely over the next two weeks.

Thanks!

[NuclearPhoenixx]

Hi there again! I've lost track of this issue and just found it again in my opened GitHub issues. Is there already support? I haven't checked the countless changelogs, so not sure. Just want to know your status on this so that the issue doesn't get stale.

[Am6er]

Hi! Actually - no :( I was working on RadiaCode support and some stability issues.