Can't get spectra to process on latest master branch - outputs constant values, inf error

[tpering]

As above, DOAS just prints out constant values, exactly the same process tried on July 2023 version of pycam and no issues, produces viable results, and processes fine. Going to try a range of different options to see what happens, but for the moment can't process the spectra on latest version.

Don't know where the issue is as no error printed out...

[ubdbra001]

@tpering I haven't had this issue with the Lascar dataset.
If you can reproduce the issue again would you be able to provide info on which dataset you're working with and the config file for the settings?

[ubdbra001]

Current lead is to do with _start_stray_pix and _end_stray_pix.
These are in the 188-224 range in v2023.07 and in the 558-593 range in the latest version

[twVolc]

This sounds reasonable. Have you managed to get those spectra processing on that earlier version? As the GUI hold stray light info in wavelength-space, but the backend needs pixel-space it might be that if you load a new set of data from a different spectrometer the pixel-space isn't being correctly updated.

I wonder if moving the stray and fit window values up/down in the GUI after loading the spectra causes the backend to correctly update its pixel values and then processes properly?

[ubdbra001]

Looks like this was it, the stray_pix variables were not being reset on loading a new spectra directory, and so old/default values were being used. I'm not sure why it wasn't a problem in the pre-config version, possibly because the values aren't initialised until DOAS is being processed.

I wonder if moving the stray and fit window values up/down in the GUI after loading the spectra causes the backend to correctly update its pixel values and then processes properly?

Yep, that worked too!

Have you managed to get those spectra processing on that earlier version?

Yes, but the results were not the same as @tpering 's. CDs in the range of 0-3000 instead of up to 8000. Not sure why to be honest, but I'm getting consistency between the v2023.07 and the fixed version now, which is positive.

[tpering]

Ah yes - I must have given you the light dilution corrected version, so we are all good. Great that it was a relatively simple fix!

…

On Thu, May 23, 2024 at 11:03 PM Dan Brady ***@***.***> wrote: Looks like this was it, the stray_pix variables were not being reset on loading a new spectra directory, and so old/default values were being used. I'm not sure why it wasn't a problem in the pre-config version, possibly because the values aren't initialised until DOAS is being processed. I wonder if moving the stray and fit window values up/down in the GUI after loading the spectra causes the backend to correctly update its pixel values and then processes properly? Yep, that worked too! Have you managed to get those spectra processing on that earlier version? Yes, but the results were not the same as @tpering <https://github.com/tpering> 's. CDs in the range of 0-3000 instead of up to 8000. Not sure why to be honest, but I'm getting consistency between the v2023.07 and the fixed version now, which is positive. — Reply to this email directly, view it on GitHub <#118 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB36N256V6GHZE2NTOPLJ33ZDZRTNAVCNFSM6AAAAABE3OJTEGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRYGA4TENZUGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[twVolc]

This is excellent news! I'll test this on our other colleagues spectra too, which seemed to be hitting a similar thing to what Tom has found.

I'm not sure if you have, but it will be worth double checking that the same isn't happening with the fit_window values too, as these also need to be mapped from wavelength in the GUI to pixels in the backend. Would be good to be certain we're fitting where we're supposed to be before we forget about this issue.

Sidenote - There's always a chance it was a problem pre-config too but we just hadn't picked it up yet.

[ubdbra001]

I'm not sure if you have, but it will be worth double checking that the same isn't happening with the fit_window values too, as these also need to be mapped from wavelength in the GUI to pixels in the backend. Would be good to be certain we're fitting where we're supposed to be before we forget about this issue.

I had a look and I don't think this is a problem. The fit_window values seem to be used directly rather than converted to pixel space, and this was what was being skipped for the stray_window values. Though if you want to have a look to make sure I'm not misunderstanding something before merging the PR then another pair of eyes is always good!