GOES 13-15 XRS data out of date

[jwreep]

Description

NOAA recalibrated the historical XRS data for GOES13-15 in May 2020, removing the scaling factors so that the data is now reported in true irradiance values (W/m^2). SunPy still returns the older values with the scaling factors included.

Additional context

This is related to #4173 -- the newly calibrated data was only made available in netcdf format here for example.

[nabobalis]

Hi @jwreep, can I get a bit more context here?

You mean if you read in GOES data into Time Series it will be rescaled?

[jwreep]

NOAA's readme, see Section 3.1.

The older data had a multiplication factor of 0.7 and 0.85 applied to the 1--8 Angstrom and 0.5--4 Angstrom channels, respectively. This was done so that the satellites reported values consistent with the calibration of older satellites. NOAA recently removed these scaling factors shortly after GOES-15 was decommissioned.

When I pull data with FIDO into timeseries, it still returns the older scaled data.

[nabobalis]

Can I get the Fido search you use to download the data please?

[jwreep]

Sure, here's an example. The flare in this case has a peak irradiance of 5.5e-4 W/m^2 with the new calibration, 3.9e-4 with the old.

import numpy as np
from sunpy.net import Fido, attrs as a
from sunpy import timeseries
from datetime import datetime
from datetime import timedelta

result = Fido.search(a.Time('2013/11/04 06:00', '2013/11/06 00:00'), a.Instrument('xrs')) 
print(result)

files = Fido.fetch(result)
goes = timeseries.TimeSeries(files, concatenate=True)
goes = goes.truncate('2013-11-05 22:00','2013-11-05 23:00')
print(np.max(goes.data['xrsb']))

[hayesla]

Yes so the issue is that the Fido GOES client is pulling the FITS files from https://umbra.nascom.nasa.gov/goes/fits/ which was the usual place that the GOES XRS data was hosted (up until GOES 15) and used by SSW etc.

Now NOAA have re-released re-scaled GOES 13, 14 and 15 data here https://satdat.ngdc.noaa.gov/sem/goes/data/science/xrs/goes13/gxrs-l2-irrad_science/ which we do not have a Fido client for yet (i'm actually working on one at the moment)

Also this new data is not in the FITS standard (its NetCDF) so not supported by the GOES TimeSeries object
(also working on this)

[hayesla]

We will need to be careful so that a user knows where the data is being pulled from - so that work prior to this rescaling can be reproduced

[wtbarnes]

Hmm. Yeah, I wonder what is the best way to go about choosing which GOES-15 to deliver to the user when they request XRS data through Fido. We'd probably want to give them the new, recalibrated data (from the netcdf files) by default as this is presumably the most accurate? But of course we'd want to have some way to access the data with the older calibration as well. Maybe we need an attr to specify this?

[jwreep]

For what it's worth, I agree with @wtbarnes that the recalibrated data should be served by default, and the older data could be made available as an option. I absolutely agree that the older data should still be available, and it's extremely frustrating that the recalibration has already caused a number of papers to be out of date!

[pabchg1999]

How often does the XRS images are available to download, for instance to make a semi live timeseries

[kakirastern]

Issue current being dealt with in #4394.

[wafels]

@hayesla thanks for the note about the FITS data at the SDAC. Does anyone know if the SSWIDL client also accesses the SDAC data? It's concerning that the SDAC is serving these data given that more up-to-date exists.

[jwreep]

SSWIDL serves the up to date data, but I'm not sure if it's now direct from NOAA or through SDAC.

[wafels]

which function accesses the GOES data?

[jwreep]

It's hard to tell from the traceback, but I pull the data with a function as follows:

a = ogoes()
a->set, tstart=start_time, tend=end_time, sat=sat
data = a->getdata(/struct)

It looks like it calls a function in there named $SSW/gen/idl/synoptic/goes/ssw_noaa_goes2ssw

Richard Schwartz would be able to answer more definitively.