Is there a way to add file types to the read_omnic protocol?

[chet-j-ski]

I work with data files that are basically .spg but with a different extension (either .HDR, .DDR, or .SDR). The manufacturer of the instrumentation actually worked with Thermo on making their data files compatible with OMNIC. I am wondering if there is a way to add these extensions to the read_omnic method so they can be read with the same protocol.

[fernandezc]

Hi.
Not yet, but it is not difficult to add these files types to the readers. I will look at this. Is it possible to have some file (small if possible) that we can use for testing?.

[chet-j-ski]

Sure thing! Try this. https://github.com/chet-j-ski/SOC100_example_data/

[fernandezc]

Thanks.
Indeed this file can be read by spectrochempy. I just changed the extension for a test.

I will work on adding these extension soon.

[atravert]

Hello,

There is probably a difference in timestamp format: scpy indicates a very old acquisition date : Jan 22nd 1956 !. Strangely, when opened in omnic (after changing the extension to spg), the date indicated by omnic is April 7th 1982. To my knowledge and to date, the date given by scpy is correct for all spg files generated by omnic.

@chet-j-ski : can you indicate the actual acquisition date of this spectrum ? If you can discliose it, can you also indicate the name of the manufacturer ?

[chet-j-ski]

Yes, I collected this spectrum myself on August 16 2018 using the SOC-100 HDR from Surface Optics Corporation (which is a kind of accessory for the iS50 FTIR from Thermo.).

[fernandezc]

Actually it looks like it is a *.spa not a *.spg. The date is correct using spa extension.

[fernandezc]

It display 2018-01-17 around midnight, so ithe difference with the actual date may be the UTC conversion.
It would be nice to have the exact date (and time of acquisition)!.

[fernandezc]

as well as the timezone!

[atravert]

I understand this strange behaviour now :-)

For the moment the timestamp extracted by the reader is the UTC/ or GMT time because I have not (yet) found where the timezone is indicated in spa/spg files, but I will work on this soon.

[fernandezc]

@atravert, will you work on adding this reader or would you like me to do it?. It seems that it will not take to much time, as it is basically the same as spa?

[atravert]

Yes I can do it.
There might be some supplementaly details to look at such as the data units (these are reflectance spectra), but apparently scp does not recognize them as such, while omnic does.

@chet-j-ski : Thank you very much for the files. Please any other example of distinct type of file you can think of. You mentioned that the file formats were equivalent to .spg, so if the .HDR, .DDR or .SDR also handle multiple files please provide an example. I will look at this asap.

[fernandezc]

@chet-j-ski In addition to these information, you were mentioning also .DDR, or .SDR extension : It could be interesting to have a sample of these files too, if you like us to include reading of them. And of course any information about the kind of experiment and date with timezone seems also necessary.

[chet-j-ski]

Added three files to https://github.com/chet-j-ski/SOC100_example_data/: Fused Silica0004.HDR, *.DDR, *.SDR. All taken on March 3rd 2021. I'm not sure about the timezone. These files are generated during a single data acquisition experiment in which the hemispherical directional reflectance (HDR) is measured. The different file types correspond to the total (HDR), diffuse (DDR), and specular (SDR) components of the reflectance.

[atravert]

@fernandezc , yes I will do it -- there are other details to look at such as the data unit - omnic correctly indentify reflectance spectra but not scpy.

@chet-j-ski, thank you for the sample files. Please add as many sample files as file types you can think of, in particular if these extensions can handle multiple spectra (as spg does). I will look at them asap.

[atravert]

@chet-j-ski, readers have been added en the DEV version and will be included in the next scpy release.

• scp.read_hdr(), scp.read_ddr(), scp.read_sdr() will read only .hdr, .ddr or .sdr, repectively
• scp.read_soc() and the generic reader scp.read() will read any of .hdr, .ddr or .sdr single file
• if several files are present in the same directory the can be read at once with scp.read_dir

For instance, the following are equivalent:

a = scp.read_hdr("Fused Silica0004.HDR")
b = scp.read_soc("Fused Silica0004.HDR")
c = scp.read("Fused Silica0004.HDR")

and if the 3 sample files (hdr, ddr, adn sdr) are present, the following will return a single dataset containing the 3 spectra:

scp.read_dir(".")

Note the files contain text metadata such as, for the sample DDR file:

MeasurementType=2;Polarization=UnPol;PolarAngle=20.0°;AzimuthAngle=0.0°;SampleTemp=300.0°K;Technician=;Notes=;NumScans=32;Reference=SpecAu.ref;FTIRParams=[G8, R16.0, A(Tri), Ph(Power Spectrum [Incoherent]), ZF0] [ST120, Vel0.4747, ADC(24), ZPD=Corr50] [erXyS9.9];WaitTime=0:00;IRSourceTemp=1300.0°K;NumRepeats=1;TotalNumScans=32;

Let us know if you'd like these data being added to the dataset: we could add them in the meta dictionnary.

Finally: we would like to include your 3 sample files in the spectrochempy_data repository in order to facilitate systematic test of theses readers each time scpy is updated. Do you authorize use to do so ?

Thanks,
Arnaud

[fernandezc]

Note the files contain text metadata such as, for the sample DDR file:

MeasurementType=2;Polarization=UnPol;PolarAngle=20.0°;AzimuthAngle=0.0°;SampleTemp=300.0°K;Technician=;Notes=;NumScans=32;Reference=SpecAu.ref;FTIRParams=[G8, R16.0, A(Tri), Ph(Power Spectrum [Incoherent]), ZF0] [ST120, Vel0.4747, ADC(24), ZPD=Corr50] [erXyS9.9];WaitTime=0:00;IRSourceTemp=1300.0°K;NumRepeats=1;TotalNumScans=32;

Let us know if you'd like these data being added to the dataset: we could add them in the meta dictionnary.

@atravert In my opinion, if the metadata is there, I don't see why we shouldn't include it in the NDDataset metadata. It's transparent to the user, but if someone needs it, they can easily find it.

[chet-j-ski]

@atravert Yes, that's fine. Thanks for your help!