Pixel discrepancy between PML HSAS data collected in field (137) and in RADCAL files (180) and consequences for FRM processing

[tjor]

Hi Dirk, Ashley, @dirk, @ARamsay17,

This issue is regarding the HSAS pixel mismatch issue which we discussed online yesterday. In summary:

PML HSAS system records 137 pixels in the field (wavelengths ~ 349 nm – 802 nm) whereas the Tartu RADCAL file assume 180 pixels to be present (~ 308 nm – 903 nm). This pixel mismatch means that PML HSAS cannot be passed through HyperCP in Class and FRM mode. Factory processing is still OK though.

To process Class-based and FRM through HyperCP I set the redundant wavelength bins in the RADCAL to zero. The Class results look reasonable, and close to Factory Results, but the FRM processing results in spurious features (edge effects) at the long and short wavelength limits.

My key query therefore is therefore how we can best pass this data through FRM mode in the processor? Particularly, regarding whether it i best to find a solution in the code which deals with the masking, and whether there is a simple fix for the edge effects in the FRM processing.

I have included sample data in the box in Data/PML_HSAS. The config folder contains 2 versions of the RADCALs - the original files are in a subfolder, and the zero-padded ones are in the main config folder. I also show examples of Factory, Class and FRM output.

No immediate hurry with this.

Thanks
Tom

PS - there is also a `radiance bump' issue impacting on the HSAS data at ~ 600-620 nm. This is a known issue that Tartu have also observed in other Satlantic radiometers. I understand Riho will be discussing this at an upcoming workshop in May. I think it is likely that the Case 1 waters on AMT amplified this feature more than the FICE data, but you can also see it in the Optics Express paper in the 620 nm band.

[ARamsay17]

@tjor I wrote the uncertainty evaluation code to screen out any pixels for which the radcal wavelength is 0. This worked for the PML data from FICE22 which was 180 channels (if memory serves). Thank you for uploading the data, I don't fully know when I will have time but I would be very interested to have a closer look at this issue. I will know better how to answer your questions once I have done that.

Also am I correct in thinking the issues mostly relate to the signal and FRM regime corrections? What do the uncertainties look like?

Thanks,
Ashley

[tjor]

Thanks @ARamsay17 ,

Yes - that's right. The central issue is with the FRM processing results. I think my zero-padding of the RADCAL has worked OK for Class processing. I included some example plots in the FRM folder on the BOX (Unc. breakdown were not generated though).

I appreciate you will have lots of other jobs on the go, so no hurry with this. I'd imagine it would be good to know what is going on by the time of FICE 2025, as the PML HSAS system is being used there.

Thanks again!
tom

[oceancolorcoder]

@tjor When Tartu provided you with the RADCAL from 20230626, did they provide you a SeaBird-style set of .cal files with their calibration values plugged in? The factory cal you have on the Box is from 20191116.

I think what's happening is that HyperCP is initializing on the .cal files (per design), but those old factory cals cover a different range of wavelengths than the Tartu RADCAL. They all use 255 pixels, but the factory cals set many bands to 'NONE' rather than to 'OPTIC3', because they don't have calibrations for those bands.

This mismatch between the bands used in the .cal files included in the Configuration in HyperCP and the bands used in the FidRadDB set of files will not work. What you need is for the .cal file to conform to the actual calibration. That means you need an updated set of .cal files from Tartu that correspond to their RADCAL files. They provided me with those .cal files when they ran full characterization of our HyperSAS in 20222, so they should be able to supply them to you as well.

[oceancolorcoder]

@tjor @ARamsay17

I've placed a slide on the Box under Data/PML_HSAS/FactoryCal_vs_RADCAL.pptx detailing the discrepancy.

Please ask Tartu if they can provide you the appropriate .cal files corresponding to their calibrations.

[tjor]

Hi Dirk, Ashley @oceancolorcoder, @ARamsay17,

Thanks for the pointers - great help! I searched around, and there are at least 2 other versions of the PML HSAS Factory cals around, which I could test. (i) Tartu versions from 2022, (ii) A edited `short format' version, which I think Ashley produced for the purposes of FICE processing

HyperCP is unable to process using case(i)/Tartu files above - error message of form: Unable to convert the following raw message: b'SATHSE2027\x00@\x00\x00\x02x\x02\xbd\x0.............

Using the case (ii)/short format files, data can now be processed without the zero-padding. Class processing looks OK. FRM processing now looks OK at longer wavelengths, but the oscillations at shorter wavelengths still persist. - Example of FRM Rrs is attached:

I've updated the BOX with the different types of factory cal (in sub-folders within config), and with new test figures with the short-format versions. To date, FICE 22 is the only PML HSAS data which has been processed in HyperCP (other than the AMT 31 data I am now looking at), so it is conceivable that the `short-format' versions are the only ones which work without causing an error.

Thanks again! I'm running out of time before the Easter break now, but will return to this after.
Tom

[tjor]

Hi @oceancolorcoder @ARamsay17,

I looked into this issue further, checking what files we have from Tartu. For our most recent HSAS calibration (2023) Tartu sent us 3 calibration files in addition to the radcals. These are named SAT0464.cal, SAT2027cal, SAT2054.cal.

However - my understanding is that HyperCP needs 6 cal files (light/darks) to initialise the config with: HED2027A.cal, HLD0464A.cal
HLD2054A.cal, HSE2027A.cal, HSL0464A.cal, HSL2054A.cal. This is what the previous message refers to, where the `short cals' from 2022 were used, as they are the latest version we have.

Is this interpretation right? (ie. that the 6 files are needed to initialise the HyperCP config), or should I be looking to use the `SAT' cal files?

Thanks,
Tom
PS - all these files are on the box in - PML_HSAS/Config/AMT31_HSAS_Calibration

[oceancolorcoder]

This is a longstanding Satlantic/Sea-Bird idiosyncrasy. For some reason, they provide 6 cal files: 3 for lights, 3 for darks. However, they are exactly the same except for the name. Their content is identical between light and dark. The code for cal file names is as follows:

HED Es dark
HLD Li dark
HLD Lt dark
HSE Es light
HSL Li light
HSL Lt light

So, you can copy your three files and rename them with the appropriate prefix to the serial number, and HyperCP will know what to do with them.

Dirk

[tjor]

Thanks Dirk @oceancolorcoder,

I didn't appreciate that 3 cal files from Tartu were equivalent to the 6 light/darks!

To get HyperCP working OK I also had to change the lines:
'INSTRUMENT SATHSE'' 6 AS 0 NONE' to 'INSTRUMENT SATHED '' 6 AS 0 NONE' in the HED file
and
'INSTRUMENT SATHSL'' 6 AS 0 NONE' to 'INSTRUMENT SATHLD '' 6 AS 0 NONE' in the HLD files.
I think we should request Tartu to provide the 6 files rather than 3, making the above change.

As for the processing results - class looks OK - example here:

Sensor specific still has the oscillations in the blue - example here:

For now, I am happy to work with the data from AMT 31 via the class processing. I am assume we can revisit trouble-shooting the sensor specific processing at time of FICE 25 (the PML HSAS system is being used there).

Thanks again!
Tom

[oceancolorcoder]

Sorry, I forgot that part. Just like I did the first time when I was doing it on my own files years ago. I agree with your suggestion for Tartu to provide both light and dark cals to avoid this confusion in the future.

The sensor-specific results are concerning considering the comparison to class-based. Not only is it noisier after the sensor corrections were applied, but the uncertainties were higher when they should be lower.

I see that you've included all the necessary data and config files to reproduce this. Unfortunately, I'm in the middle of a multi-day batch currently on my local machine and my remote servers can only run scripted, so no good for troubleshooting.

@ARamsay17, @juanchossn, we need to work together to determine what is driving this unexpected behavior in the FRM regime. @tjor I see you uncertainty breakdown plots for 342-1528 class-based (dominated by rho, cosine, and noise) but not for sensor-specific, which is only processed to L1B.

[tjor]

Hi again Dirk @oceancolorcoder,

Thanks for looking into this issue with the PML HSAS sensor further - we really appreciate it. I've now added the 6 renamed cal files to the AMT config folder in box. I also added the test results (using the renamed cal files) from class and FRM as subfolders in PML_HSAS. The previous tests results using Ashley's `short-format' files are also there too.

re: uncertainty breakdowns. I switched this off at some point as there was an error in one of the uncertainty breakdown plots (something to do with one vector being 255 pixels and the other being 180 pixels).

Some additional points to be aware of are:
-We are using ~ 2.5 min rather than usual 5 minute ensemble bins (I implemented this, as it is how AMT Rrs data has previously been processed)
-Our RAW data files are only 10 minute in length (again, this follows past practice on AMT)
-For the AMT 31 test data on the box, you will need to use the anc file 'JamesCookAnc_20241207.sb', which has GPS, windspeed, tilt etc.
-My results that I have shared were processed using the so-rad branch (so basically close to what is dev). I think I also got similar results using the main branch though

Thanks
Tom

[oceancolorcoder]

@tjor, The issue with the higher uncertainties in the FRM-Full branch has been resolved in the latest release. The increased noisiness in the UV is likely due to poor straylight correction in that region and we are working on a resolution.

[tjor]

Hi Dirk, Thanks for the update on this and hope you enjoyed you time out in Italy. I heard there were customs issues with your PySAS, so I am sorry about that. I can't come to the HyperCP meeting today (can also see that you are focused on FICE!). I'm hoping to return to the So-Rad branch soon to get it to stage so that other users (e.g. Vittorio & co. at CNR) are able to use it. The key thing is still working out what that `base file unit' (or databse call file-analog) should be. I think an hourly json file as we discussed last time could work, but I need to dicuss pros/cons properly with Stefan. Cheers Tom

…

------------------------------------------------------------------------------------------------ Tom Jordan, Earth Observation Scientist, Plymouth Marine Laboratory https://pml.ac.uk/profile/dr-thomas-jordan/

________________________________ From: Dirk Aurin ***@***.***> Sent: Tuesday, July 22, 2025 9:30 PM To: nasa/HyperCP ***@***.***> Cc: Tom Jordan ***@***.***>; Mention ***@***.***> Subject: Re: [nasa/HyperCP] Pixel discrepancy between PML HSAS data collected in field (137) and in RADCAL files (180) and consequences for FRM processing (Issue #345) Caution:This is an external email. Please take care when clicking links or opening attachments. When in doubt, contact your IT Department [https://avatars.githubusercontent.com/u/45241177?s=20&v=4]oceancolorcoder left a comment (nasa/HyperCP#345)<#345 (comment)> @tjor<https://github.com/tjor>, The issue with the higher uncertainties in the FRM-Full branch has been resolved in the latest release. The increased noisiness in the UV is likely due to poor straylight correction in that region and we are working on a resolution. — Reply to this email directly, view it on GitHub<#345 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AT4N2R4OFUTRB2ZFUNCXHM33J2NNLAVCNFSM6AAAAAB23FQMHCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMBUG4ZTSNZXGQ>. You are receiving this because you were mentioned.Message ID: ***@***.***> Visit our website www.pml.ac.uk | Subscribe to our newsletter <https://www.pml.ac.uk/newsletter> | Follow us on social media: LinkedIn<https://www.linkedin.com/company/plymouth-marine-laboratory-&-pml-applications-ltd>, X / Twitter<https://twitter.com/PlymouthMarine>, Instagram<https://www.instagram.com/plymouthmarinelab>, Facebook<https://www.facebook.com/PlymouthMarineLaboratory>, YouTube<https://www.youtube.com/channel/UCfsYrcvGOp-JJm5owSJN8OA> This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses. Our Privacy Notice is available at http://www.pml.ac.uk/System-pages/Privacy_Notice . Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK. PML Applications Ltd, www.pml-applications.co.uk , is the wholly owned subsidiary of PML, registered as a company in England & Wales, company number 4232750.

[oceancolorcoder]

@tjor, Are you referring to the So-Rad processing pathway that uses online database extractions, or the new pathway that will use files directly downloaded from the platform (e.g., in NRT in the field, even where internet is unavailable)?

[tjor]

Hi Dirk, I was hoping that we could try to harmonize the two approaches. In the last HyperCP meeting I went to I remember that you saying that it would be advantageous for HyperCP to save hourly units at the `RAW' level (eg. something like a JSON format a base hourly file unit). This has made me realise that, maybe we just want a user to extract the JSON separately (either from the platform in the field, or using our MONDA python package for data that is in database). So-Rad users will already be familiar with doing the geo server requests in MONDA, so it should user friendly. Process L1A would then be the translation of JSON to L1A HDF (appending the relevant calibration info which won't be in the JSON). I need to still properly discuss this with Stefan though. Cheers Tom

…

------------------------------------------------------------------------------------------------ Tom Jordan, Earth Observation Scientist, Plymouth Marine Laboratory https://pml.ac.uk/profile/dr-thomas-jordan/

________________________________ From: Dirk Aurin ***@***.***> Sent: Wednesday, July 23, 2025 2:25 PM To: nasa/HyperCP ***@***.***> Cc: Tom Jordan ***@***.***>; Mention ***@***.***> Subject: Re: [nasa/HyperCP] Pixel discrepancy between PML HSAS data collected in field (137) and in RADCAL files (180) and consequences for FRM processing (Issue #345) Caution:This is an external email. Please take care when clicking links or opening attachments. When in doubt, contact your IT Department [https://avatars.githubusercontent.com/u/45241177?s=20&v=4]oceancolorcoder left a comment (nasa/HyperCP#345)<#345 (comment)> @tjor<https://github.com/tjor>, Are you referring to the So-Rad processing pathway that uses online database extractions, or the new pathway that will use files directly downloaded from the platform (e.g., in NRT in the field, even where internet is unavailable)? — Reply to this email directly, view it on GitHub<#345 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AT4N2R7HVILF4QM7WIU7UDD3J6ELLAVCNFSM6AAAAAB23FQMHCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCMBYGM4DMNJYHA>. You are receiving this because you were mentioned. Visit our website www.pml.ac.uk | Subscribe to our newsletter <https://www.pml.ac.uk/newsletter> | Follow us on social media: LinkedIn<https://www.linkedin.com/company/plymouth-marine-laboratory-&-pml-applications-ltd>, X / Twitter<https://twitter.com/PlymouthMarine>, Instagram<https://www.instagram.com/plymouthmarinelab>, Facebook<https://www.facebook.com/PlymouthMarineLaboratory>, YouTube<https://www.youtube.com/channel/UCfsYrcvGOp-JJm5owSJN8OA> This message is private and confidential. If you have received this message in error, please notify the sender and remove it from your system. You are reminded that e-mail communications are not secure and may contain viruses; PML accepts no liability for any loss or damage which may be caused by viruses. Our Privacy Notice is available at http://www.pml.ac.uk/System-pages/Privacy_Notice . Plymouth Marine Laboratory (PML) is a company limited by guarantee registered in England & Wales, company number 4178503. Registered Charity No. 1091222. Registered Office: Prospect Place, The Hoe, Plymouth PL1 3DH, UK. PML Applications Ltd, www.pml-applications.co.uk , is the wholly owned subsidiary of PML, registered as a company in England & Wales, company number 4232750.

[oceancolorcoder]

Tom,

I do not see a problem with harmonizing the approaches wherein the files retrieved directly from the So-Rad (i.e., offline) are hourly files in the same JSON format as those extracted files from the server, and RAW->L1A in HyperCP performs the same conversion on each.