Low Time-Resolution Alternative Data Sources for AE Index GMAG Stations

[DCarSunDr]

Given the given recent gaps in SNKQ and LRV data, there is a need for a process which detects if main data retrieval fails for given data files and responds by retrieving and processing lower time cadence data from other sources. This primarily affects sites under the THEMIS AE group and Kyoto Proxy AE group. The idea is that if the files become available 6 months down the line, they can overwrite the low time resolution data meant to patch the gaps. The THEMIS AE index is at a 1 minute time resolution, so alternative sources at the 1 minute time resolution should be fine.

For each station in the THEMIS and Kyoto Proxy AE index, we need to design and implement a method to retrieve lower resolution data, process is, and configure the method to run if the initial attempt to download the data fails.

[DCarSunDr]

For any stations which strictly fall under the NRCAN group (SNKQ/SNK), we can retrieve data in the miniseed format, relayed by GOES primary. The data is accessed through a web query in the form of: http://www.earthquakescanada.nrcan.gc.ca/fdsnws/dataselect/1/query/?station=SNK&starttime=2026-01-20T00:00:00Z&endtime=2026-01-21T00:00:00Z

Once that's done, the following script creates 4 text files with the components of data:

import obspy
import numpy as np

# miniseed files from NRCAN are queried via URL in the form of:
# http://www.earthquakescanada.nrcan.gc.ca/fdsnws/dataselect/1/query/?station=SNK&starttime=2026-01-20T00:00:00Z&endtime=2026-01-21T00:00:00Z

saveloc = "C:/Users/DC/Documents/Projects/bleeding_edge_Tasks/task_347/"
station = "snk"
date_str = "20260119"
st = obspy.read(saveloc + station + date_str + ".mseed")

tmp = st.select(network='C2',location='R1',channel='UFX') 
tmp += st.select(network='C2',location='R1',channel='UFY') 
tmp += st.select(network='C2',location='R1',channel='UFZ') 
tmp += st.select(network='C2',location='R1',channel='UFF') 
tmp.plot()
tmp.merge(fill_value=np.nan)
tmp.write(saveloc + station + date_str + "_R1.txt",format = "TSPAIR")

tmp = st.select(network='C2',location='R2',channel='UFX') 
tmp += st.select(network='C2',location='R2',channel='UFY') 
tmp += st.select(network='C2',location='R2',channel='UFZ') 
tmp += st.select(network='C2',location='R2',channel='UFF') 
tmp.plot()
tmp.merge(fill_value=np.nan)
tmp.write(saveloc + station + date_str + "_R2.txt",format = "TSPAIR")

def select_loc_cha(loc,cha,st=st):
    tmp = st.select(network='C2',location=loc,channel=cha) 
    tmp.merge(fill_value=np.nan)
    tmp.write(saveloc + station + date_str + "_" + loc + "_" + cha + ".txt",format = "TSPAIR")
    return

loc_list = ['R1','R2']
cha_list = ['UFX','UFY','UFZ','UFF']

for loc in loc_list:
    for cha in cha_list:
        select_loc_cha(loc=loc,cha=cha,st=st)

This is done over using a single file with all 4 components because the output file simply contains the individual components back to back, instead of alongside one other. The files need to be combined into something which can be processed by one of the l2 processing scripts. It would be good to indicate in the metadata that the time cadence is 1 minute

[DCarSunDr]

Note, this should close a blocked issue: spedas/bleeding_edge#347

[DCarSunDr]

SNKQ is in the old AE list, not the current. If we can regularly retrieve 1 min via GOES relay, we may want to include it in the AE index for THEMIS again. Something to consider

[DCarSunDr]

Notes from SPDF:
We might want to look into an option of using time resolution to select data in helio.data.nasa.gov

If there are some similar cases in the past, it will be great if our mission team can generate such data product https://helio.data.nasa.gov/

Investigate spase records for THEMIS GMAG and variometer data

Treat AE sites as variable time resolution

When the time comes to create SPASE records, you can use https://spdf.gsfc.nasa.gov/istp-metadata-editor/ to create the initial XML files from the ISTP metadata in the CDFs (essentially just open the CDF then use File -> Save SPASE XML File).

[DCarSunDr]

Currently, for variometers, we include the different time resolutions in the same directory, but distinguish them using ISTP compliant file names (adding a 100_ms to the 10 Hz data files)

We could theoretically do the same thing here for the 1 minute auxiliary data

[DCarSunDr]

I've published the branch I'm working on; it isn't operational yet, but I wanted to save progress to the repo before I forget. It currently will only support the minimum number of methods/stations we'd need to service, and we can expand this later.

We may need to add obspy to thmsoc_python environment to handle miniseed files. We also might want to place the url and file handling tools in the variometer retrieval script into separate scripts/submodules. In this case, I've created a url query constructor which is similar to the method in the variometer retrieval script so that it could eventually be shared between the two scripts.

[jameswilburlewis]

That's a good point about including obspy as a thmsoc_python dependency. I thought we were already getting it via pyspedas, but it looks like we only have 'pyspedas' and not 'pyspedas[mth5]' or 'pyspedas[all]' in the project dependencies.

We should probably go with pyspedas[all] to pick up basemap, too. Just in case we ever want to switch to using it for our online ground track summary plots...

But maybe we should still make obspy a direct dependency....it's clearer to look at the pyproject.toml and see an explicit picture of what we're directly depending on, instead of trying to keep track of all the implicit dependencies in our heads..... Same for scipy, numpy, and tomli, which also come along for the ride when you install pyspedas....we should leave them alone to.