VSO search for STEREO COR2 data does not return all available observations #7439
Comments
|
Thank you for the detailed report. Can you try the web UI for the VSO and see if those return the files? If that does, then we know its a bug in our own VSOClient or if we need to ask the VSO team themselves for advice. |
|
Yes, thanks for the detailed report. I've been looking into this issue, I can say it's an issue with the ingest script that adds SECCHI/COR2 data to the DB that the VSO uses for searching. Also, the issue is broader than just 2020. Match count = 105 DB only = 144 Disc only = 64 Total(DB) = 249/249 Total(disc) = 169/169 So some files are only on disc, others are only in the DB, and 105 are on both the disc and the DB. |
|
No problem and thank you also for looking at this.
Yes, it seems that it is a broader issue. I had time to check 2020 and 2021. Almost 1/3 of the days for 2020 (133/365) are impacted and no TB images are available. For example: 
For 2021 I find a similar situation and numbers. Note that I only searched this for TB images so I am not sure what happens to the PB images, if these files are all availiable or not.
This is definitely an issue. For 7/5/22 (I assumed this is July 5, 2022) what things should be according to the summary files is the following: In sccA202207.img.c2 (the file that provides the TB images) there are: 169 files of which = 97 SSR1 + 72 SW Since VSO provides (I think) both TB and PB images at the same search and (probably) only the synoptic program (SSR1) images (since the SW are low-res and cannot be used scientifically) then for this date the VSO search should give 97 TB SSR1 + 72 PB SSR1 images = 169 images I hope this helps. |
|
I've gotten to the bottom of the missing COR2 files. The first issue, as you've pointed out, is that the VSO only serves out TB, PB and SSR1 images, and filters out the SW (beacon) files. The second is there was an error in the parsing script, most likely due to a change in the format of the log files from NRL that are parsed. The result was files were skipped and marked as MISSING. I've got a fix in for the RE so going forward from 3/6/24 you should see the correct no. of COR2 files returned by the VSO. I'll update here when that job is completed. |
nabobalis
added
the
Remote Server Issue / VSO
Describe the bug
The issue is that the VSO search for STEREO COR2 data does not always return all the Total Brigtnessc (TB) COR2 fits files that are available on the STEREO site. STEREO SECCHI Level 0.5 FITS files for the two spacecraft are provided from https://stereo-ssc.nascom.nasa.gov/data/ins_data/secchi/L0/ and are organized into three directories: the \cal that contains the Calibration files, the \img that contains the “Brightness" images for EUVI, HI, COR instruments and the, \seq that contains the Polarized (PB) images for COR1 or COR2. Additionally, the \summary directory provides summary text files of the available data for each month and instrument. Making some tests it seems that for many days the TB COR2 images are completely missing from the search results.
To Reproduce
A simple test for two selected dates:
A) An example of a date that COR2 data are missing:
The result from the above is the following:
So for this specific date no TB images were found but according to the summary file https://stereo-ssc.nascom.nasa.gov/data/ins_data/secchi/L0/a/summary/sccA202003.img.c2 the availiable TB images for that date are the following:
B) An example of a date that COR2 data are all availiable:
Doing the same for:
time_range = a.Time('2020/03/06 00:00:00', '2020/03/07 00:00:00')we get:
This result maches to all the availiable TB images from https://stereo-ssc.nascom.nasa.gov/data/ins_data/secchi/L0/a/summary/sccA202003.img.c2.
A more advanced test:
In the example below I am trying to find out for which dates during March 2022 there are data missing and how many images every month during 2020 are missing.
Some funcions first:
Then do a VSO search of data for a full month and compare them with the availiable files at SECCHI site:
This gives:
There are days where no TB data returned at all, see for example the 2020/03/08.
Repeating the same for the full 2020 year in a mothly basis:
This gives:
Screenshots
No response
System Details
Installation method
pip
The text was updated successfully, but these errors were encountered: