New JSON format version

[NuclearPhoenixx]

So, I created a new draft version of the NPES file format that you're using for the JSON files in impulse. It newly adds the possibility of having multiple spectra inside one file in a data array. These entries inside the array are called data packages and are basically almost identical to an entire "old" v1 file. This also means that each data package is completely independent from one another, so you could store totally unrelated data from different devices, etc.

So again, you would now be required to have a data array, which contains an arbitrary number of data packages, which in turn are almost identically formatted like the old standard version. Roughly speaking, you're able to take the data from the old files and just put them in an array inside a single file.

I created a branch in my repo for the draft with all the updated documentation: https://github.com/OpenGammaProject/NPES-JSON/tree/draft
There are also updated example files if you wanna have a look: https://github.com/OpenGammaProject/NPES-JSON/tree/draft/examples

This should hopefully now also comply with the requirements for BecqMoni to be implemented in the software too (fingers crossed :D), see this issue: Am6er/BecqMoni#2

What do you think? Any suggestions or comments? Would you want to implement this in impulse?

It's not a lot of work to implement this, you'd only have to pull out the individual data packages from the array and then parse them just like the old JSON files since the structure inside them is identical. I think the most difficult part would be to decide on how to handle this in the UI, i.e. letting the user decide to save multiple spectra in one file or creating multiple files. The same is true for importing then of course.

[ssesselmann]

Matthias,

I see what you have done. I already did something similar in Impulse for the optional 3D spectrum function, but instead of repeating the whole schema I just appended multiple spectra as an array like this "spectrum": [[0.0.0.0],[0,0,0,0]]

Technically this is not true NPES scheme and it does have some issues running in Impulse, because as the file gets bigger and bigger the program runs slower and slower. I don't think it is the loop appending the data, it is more likely the screen update which is slowing things down.

But when running 3D spectra it would make sense to reduce the number of bins and update intervals.

We can achieve the same result using your NPESv2 albeit with a slightly larger scale.

If not for 3D spectra, what other use cases do you think it is useful for?

I am currently in the middle of updating Impulse for use with ATOM-NANO devices and it is becoming quite complicated because it is a USB serial device with UART communication protocol. Fortunately Am6er (Albert) has done most of the hard work.

Rendering the spectrum from ATOM-NANO is the easy part, most of the changes are about making it user friendly, so all the prompts that relate to sound card devices are hidden when a serial device is connected and new prompts that are required show up.

I will stick with NPESv1 until the current version is published and tested.

Steven

[NuclearPhoenixx]

Technically this is not true NPES scheme and it does have some issues running in Impulse, because as the file gets bigger and bigger the program runs slower and slower. I don't think it is the loop appending the data, it is more likely the screen update which is slowing things down.

Yeah, Plotly unfortunately gets pretty slow for larger and larger data sets. It can already get pretty annyoing at 4096 bins sometimes, and larger than that is just pain. I've played around with 8192 channels, but that's just unbearable to scroll around in general. I'd love to use different renderers and hardware acceleration with Plotly, but there are some bugs that make everything else a lot more complicated.

If not for 3D spectra, what other use cases do you think it is useful for?

• It allows compatability with more programs that require the feature to save multiple spectra in one file, such as BecqMoni for example.
• You can use it in a data logger context, i.e. if you want to save spectra chronologically and/or with different locations for example. This has been discussed here before: Possibility to record of time and or location evolution of measured data OpenGammaProject/NPES-JSON#2

There is practically no downside to this. You can still only save a single spectrum per file if you want and it'll only use a few bytes more of space on your drive.
It's just very useful in general if you want to do any kind of chronological recordings or location-dependent measurements and if you want to join multiple spectra together of course. I didn't think about 3D spectra, but yeah sure that'll work too, it's just a chronological recording after all.

The schema works fine and I'll implement it into Gamma MCA once I got everyone's feedback. I just want to collect everyone's opinion before making it "official" and implementing it myself. The new version will surely help with interoperability, especially if you already tried to get this feature by changing it yourself. The need for it is here, so I'll update it. The new schema will break compatibility with the old one obviously, but I tried to minimize the effect of this by making it like an array of the "old" data files (identical except for the schema version entry) and I think that's the best way to go. Like I said earlier, that way you don't have to change a lot for your file parser. Just some new UI if you want to let users decide to save or load a specific data package from the file.

I could've also done it like you did, with an array for the spectrum entry itself. However, you cannot generally be sure that these arrays are ordered correctly and in the way you expect if you receive a file from someone, so that's not best practice. Also, you might want to have different timestamps or GPS locations for your different spectra. Or even if you wanted to join unrelated spectra, you might need a different calibration or different device info for each one. That's why I think they should be entirely separate instances -- there cannot be any misunderstandings about the data that's contained within the data packages.

[ssesselmann]

Sure, the JSON header doesn't take all that much space, but how do you see the user option for choosing separate NPES files vs. appending multiple NPES files to one master file ?

[NuclearPhoenixx]

I'll just have "Save As" (new file), "Save" (keep file and append to file in the new format) and "Overwrite" (keep file and only save the single spectrum) buttons. Sounds easy enough and the meanings of each button are shouldn't be confusing. And for importing I'll simply create a pop-up that detects if there are multiple packets of data and then lets the user choose what entry they want to select.

[NuclearPhoenixx]

So what do you think about the change? Do you have any suggestions?

[ssesselmann]

Impulse is now NPESv2 compatible as well as backwards compatible with NPESv1 but for the moment only saves one spectrum per file.