Functionality to client to download new GOES 16/17 data and reprocessed 13/14/15 #4394

hayesla · 2020-07-27T21:24:14Z

Description

This is adding the functionality to the XRSClient to get access to the GOES XRS 16 and 17 data.

It also now updates the client so that the default GOES13/14/15 data queried will return the new reprocessed data from the NOAA site rather than the old data. The old data can still be accessed through a new attrs a.goes.VersionData. I'm not too sure if this is how we want to go about this - but I think we should still provide an option to be able to download the old data (i.e. the data that this client used to return).

This is for the high time resolution data (2s for 13/14/15 and 1s for 16/17)

Fixes #4173
Fixes #4376

These new data files are in netCDF format so at the moment do not work with sunpy.timeseries (which will be in another PR)

…so the recalbirated 13,14,15 data

pep8speaks · 2020-07-27T21:24:18Z

Hello @hayesla! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file examples/time_series/timeseriesmetadata_example.py:

Line 19:101: E501 line too long (104 > 100 characters)

In the file sunpy/net/dataretriever/sources/goes.py:

Line 63:101: E501 line too long (117 > 100 characters)
Line 69:101: E501 line too long (124 > 100 characters)
Line 71:101: E501 line too long (108 > 100 characters)
Line 74:101: E501 line too long (116 > 100 characters)
Line 75:101: E501 line too long (109 > 100 characters)
Line 77:101: E501 line too long (106 > 100 characters)
Line 134:101: E501 line too long (131 > 100 characters)
Line 141:101: E501 line too long (124 > 100 characters)
Line 143:40: E225 missing whitespace around operator
Line 145:101: E501 line too long (128 > 100 characters)
Line 161:101: E501 line too long (103 > 100 characters)

In the file sunpy/net/dataretriever/sources/tests/test_goes_ud.py:

Line 62:101: E501 line too long (113 > 100 characters)
Line 76:101: E501 line too long (113 > 100 characters)

In the file sunpy/util/scraper.py:

Line 279:101: E501 line too long (120 > 100 characters)

Comment last updated at 2020-11-20 22:52:53 UTC

sunpy/net/dataretriever/attrs/goes.py

sunpy/net/dataretriever/sources/goes.py

sunpy/database/tests/test_tables.py

sunpy/net/dataretriever/sources/goes.py

sunpy/database/tests/test_tables.py

wtbarnes · 2020-07-29T03:09:15Z

This is great! 🎉

dstansby

Marking as needs changes as per #4394 (review)

Cadair · 2020-09-10T11:40:19Z

@hayesla #4321 has been merged so this needs a re-work 😄

hayesla · 2020-09-16T12:08:28Z

@hayesla #4321 has been merged so this needs a re-work 😄

on it 🌞

hayesla · 2020-09-17T12:10:19Z

This is still a work in progress - the main issue being how we deal with the old GOES 13, 14 and 15 data that's on the NASA server. Really people should be using the new re-processed science quality data from the NOAA site (in netcdf), but maybe we should still allow access to the old fits files? The only difference is the scaling factor which is removed in the new netcdf data (which will change the peak flux value/GOES class).

At the moment I've added a a.goes.VersionData attrs - but I'm not married to that idea. Actually its pretty messy.

What are peoples thoughts on this? @wtbarnes @Cadair

wtbarnes · 2020-09-18T18:03:34Z

Seems reasonable to me. Would this be something that the user always has to specify? Or does it take on a default value to always return the latest (netCDF) data?

kakirastern · 2020-11-17T15:22:16Z

Regarding the build_docs check, we have the warnings:


    goes_ts = sunpy.timeseries.TimeSeries(goes_files, source='XRS', concatenate=True)
  File "/home/vsts/work/1/s/.tox/build_docs/lib/python3.8/site-packages/sunpy/timeseries/timeseries_factory.py", line 406, in __call__
    new_ts = self._check_registered_widgets(filepath=filepath, **kwargs)
  File "/home/vsts/work/1/s/.tox/build_docs/lib/python3.8/site-packages/sunpy/timeseries/timeseries_factory.py", line 529, in _check_registered_widgets
    data, meta, units = WidgetType._parse_file(filepath)
  File "/home/vsts/work/1/s/.tox/build_docs/lib/python3.8/site-packages/sunpy/timeseries/sources/goes.py", line 165, in _parse_file
    hdus = sunpy.io.read_file(filepath)
  File "/home/vsts/work/1/s/.tox/build_docs/lib/python3.8/site-packages/sunpy/io/file_tools.py", line 89, in read_file
    readername = _detect_filetype(filepath)
  File "/home/vsts/work/1/s/.tox/build_docs/lib/python3.8/site-packages/sunpy/io/file_tools.py", line 214, in _detect_filetype
    raise UnrecognizedFileTypeError("The requested filetype is not currently "
sunpy.io.file_tools.UnrecognizedFileTypeError: The requested filetype is not currently supported by SunPy.


/home/vsts/work/1/s/examples/time_series/goes_hek_m25.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/vsts/work/1/s/examples/time_series/goes_hek_m25.py", line 24, in <module>
    goes = TimeSeries(files)
  File "/home/vsts/work/1/s/.tox/build_docs/lib/python3.8/site-packages/sunpy/timeseries/timeseries_factory.py", line 406, in __call__
    new_ts = self._check_registered_widgets(filepath=filepath, **kwargs)
  File "/home/vsts/work/1/s/.tox/build_docs/lib/python3.8/site-packages/sunpy/timeseries/timeseries_factory.py", line 529, in _check_registered_widgets
    data, meta, units = WidgetType._parse_file(filepath)
TypeError: cannot unpack non-iterable NotImplementedType object

hayesla · 2020-11-17T15:29:16Z

The build_doc is failing as its waiting on #4592

kakirastern · 2020-11-17T15:30:51Z

The build_doc is failing as its waiting on #4592

I see. Thanks for the heads up!

Cadair · 2020-11-18T13:47:51Z

Oh fun, the goes server is giving us urllib.error.HTTPError: HTTP Error 429: Too Many Requests

Cadair · 2020-11-18T13:50:55Z

Also happened on the online tests, I wonder if this is the scraper making too many calls to the server or similar. The first thing we should check would be to make sure we aren't being inefficient with our use of the scraper or at any other point in the high level code. The next thing we could do is catch a 409 response and implement a delay of a few hundred ms and retry.

Cadair · 2020-11-18T15:10:57Z

sunpy/extern/parse.py

@@ -471,7 +471,7 @@

 __all__ = 'parse search findall with_pattern'.split()

-log = logging.getLogger(__name__)
+log = logging.getLogger("parse")


This is annoying, as we now have to remember to change this on every update.

For context, not doing this was causing an error to be raised when the log level was set to debug. I assume because the name of this logger started with sunpy it was being sent through some of the astropy logging infrastructure in a way that it wasn't supposed to be and throwing a traceback.

Cadair

Looks good, I think there is some deduplication that can happen on the calls to scraper though.

Cadair · 2020-11-18T15:14:11Z

sunpy/net/dataretriever/sources/goes.py

@@ -22,7 +22,22 @@ class XRSClient(GenericClient):
    """
    Provides access to the GOES XRS fits files archive.

-    Searches data hosted by the `Solar Data Analysis Center <https://umbra.nascom.nasa.gov/goes/fits/>`__.
+    Searches data for the GOES XRS data both on NASA servers prior


both? I thought we defaulted to just new servers?

yep - but for the data prior to GOES 13 the NASA server data is correct (NOAA haven't provided re-processed data)

sunpy/net/dataretriever/sources/goes.py

Changes made

Cadair · 2020-11-21T12:08:45Z

Thanks @hayesla this and #4592 was an awesome effort 🎉

kakirastern · 2020-11-21T12:12:25Z

🎉

adding functionality to client to download new GOES 16/17 data and al…

bbfd98d

…so the recalbirated 13,14,15 data

hayesla requested a review from a team as a code owner July 27, 2020 21:24

nabobalis added this to the 2.1 milestone Jul 27, 2020

nabobalis added the net Affects the net submodule label Jul 27, 2020

abhijeetmanhas reviewed Jul 27, 2020

View reviewed changes

sunpy/net/dataretriever/attrs/goes.py Outdated Show resolved Hide resolved

abhijeetmanhas reviewed Jul 27, 2020

View reviewed changes

sunpy/net/dataretriever/sources/goes.py Outdated Show resolved Hide resolved

hayesla requested a review from a team as a code owner July 28, 2020 01:58

hayesla force-pushed the new_goes_fido branch from c7ae6aa to bbfd98d Compare July 28, 2020 02:06

hayesla added 4 commits July 27, 2020 22:10

fixing tests

49fe2c6

adding detail to register_values

e5878b5

fixing source

c7d7cf8

fixing database fido search

883bb96

hayesla requested a review from a team as a code owner July 28, 2020 03:08

abhijeetmanhas reviewed Jul 28, 2020

View reviewed changes

sunpy/net/dataretriever/sources/goes.py Outdated Show resolved Hide resolved

abhijeetmanhas reviewed Jul 28, 2020

View reviewed changes

sunpy/database/tests/test_tables.py Outdated Show resolved Hide resolved

hayesla added 2 commits July 28, 2020 12:51

updating the register_values

1b6a0cb

adding changelog

9af4f20

abhijeetmanhas reviewed Jul 28, 2020

View reviewed changes

sunpy/net/dataretriever/sources/goes.py Outdated Show resolved Hide resolved

wtbarnes reviewed Jul 29, 2020

View reviewed changes

sunpy/database/tests/test_tables.py Outdated Show resolved Hide resolved

wtbarnes mentioned this pull request Jul 29, 2020

Discussion on how we support GOES-XRS Data #4389

Closed

dstansby previously requested changes Aug 18, 2020

View reviewed changes

hayesla added 3 commits September 17, 2020 11:15

merging master

486c155

updating to new generic client

8e1a0bd

pre-commit fixes

c308057

nabobalis requested a review from dstansby November 17, 2020 15:21

Merge branch 'master' into new_goes_fido

af3d45c

Cadair reviewed Nov 18, 2020

View reviewed changes

Cadair requested changes Nov 18, 2020

View reviewed changes

Cadair added 2 commits November 18, 2020 17:23

Add some HTTP 429 backoff code in scraper

df69aa2

Fix extern.parse logging bug

9fc43f6

Cadair force-pushed the new_goes_fido branch from 079d8d4 to 9fc43f6 Compare November 18, 2020 17:23

hayesla added 5 commits November 19, 2020 16:21

updating from comments

640cfed

updating doc string

f8ec6e3

cleaning up with function added

ddc048c

Merge branch 'master' into new_goes_fido

1959192

tidying up

3a5d94e

Cadair approved these changes Nov 20, 2020

View reviewed changes

sunpy/net/dataretriever/sources/goes.py Outdated Show resolved Hide resolved

sunpy/net/dataretriever/sources/goes.py Outdated Show resolved Hide resolved

hayesla added 4 commits November 20, 2020 11:34

fixing format input

cdc8891

incl. sat number keyword

a8a5128

trying shorter timerange to see if time for search/download issue

1aed943

Merge branch 'master' into new_goes_fido

8e813f0

2.1 automation moved this from Review in progress to Reviewer approved Nov 21, 2020

nabobalis removed the request for review from dstansby November 21, 2020 11:13

nabobalis merged commit e3017be into sunpy:master Nov 21, 2020

2.1 automation moved this from Reviewer approved to Done Nov 21, 2020

dstansby removed the Needs Review Needs reviews before merge label Jan 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Functionality to client to download new GOES 16/17 data and reprocessed 13/14/15 #4394

Functionality to client to download new GOES 16/17 data and reprocessed 13/14/15 #4394

hayesla commented Jul 27, 2020 •

edited by nabobalis

pep8speaks commented Jul 27, 2020 •

edited

wtbarnes commented Jul 29, 2020

dstansby left a comment

Cadair commented Sep 10, 2020

hayesla commented Sep 16, 2020

hayesla commented Sep 17, 2020

wtbarnes commented Sep 18, 2020

kakirastern commented Nov 17, 2020

hayesla commented Nov 17, 2020

kakirastern commented Nov 17, 2020

Cadair commented Nov 18, 2020

Cadair commented Nov 18, 2020

Cadair Nov 18, 2020

Cadair Nov 18, 2020

Cadair left a comment

Cadair Nov 18, 2020

hayesla Nov 19, 2020

Cadair commented Nov 21, 2020

kakirastern commented Nov 21, 2020

Functionality to client to download new GOES 16/17 data and reprocessed 13/14/15 #4394

Functionality to client to download new GOES 16/17 data and reprocessed 13/14/15 #4394

Conversation

hayesla commented Jul 27, 2020 • edited by nabobalis

Description

pep8speaks commented Jul 27, 2020 • edited

Comment last updated at 2020-11-20 22:52:53 UTC

wtbarnes commented Jul 29, 2020

dstansby left a comment

Choose a reason for hiding this comment

Cadair commented Sep 10, 2020

hayesla commented Sep 16, 2020

hayesla commented Sep 17, 2020

wtbarnes commented Sep 18, 2020

kakirastern commented Nov 17, 2020

hayesla commented Nov 17, 2020

kakirastern commented Nov 17, 2020

Cadair commented Nov 18, 2020

Cadair commented Nov 18, 2020

Cadair Nov 18, 2020

Choose a reason for hiding this comment

Cadair Nov 18, 2020

Choose a reason for hiding this comment

Cadair left a comment

Choose a reason for hiding this comment

Cadair Nov 18, 2020

Choose a reason for hiding this comment

hayesla Nov 19, 2020

Choose a reason for hiding this comment

Cadair commented Nov 21, 2020

kakirastern commented Nov 21, 2020

hayesla commented Jul 27, 2020 •

edited by nabobalis

pep8speaks commented Jul 27, 2020 •

edited