Redesigning genericClient

changelog/4321.breaking.1.rst

@@ -0,0 +1 @@
+Removed :meth:`~sunpy.net.dataretriever.QueryResponse.create`.

changelog/4321.breaking.rst

@@ -0,0 +1 @@
+Removed `~sunpy.net.dataretriever.QueryResponseBlock` as `~sunpy.net.dataretriever.QueryResponse` now uses a `list` of `~collections.OrderedDict` for building the response table.

changelog/4321.feature.1.rst

@@ -0,0 +1,3 @@
+Additions in `~sunpy.util.scraper` to support the refactoring of `~sunpy.net.dataretriever.GenericClient`:
+- :meth:`~sunpy.util.scraper.Scraper.findDatewith_extractor` that parses the url using extractor to return its start time.
+- A ``matcher`` in :meth:`~sunpy.util.scraper.Scraper._extract_files_meta` which validates the extracted metadata by using the dictionary returned from :meth:`~sunpy.net.dataretriever.GenericClient._get_match_dict`.

changelog/4321.feature.3.rst

@@ -0,0 +1 @@
+Added methods :meth:`~sunpy.net.dataretriever.GenericClient.pre_search_hook` and :meth:`~sunpy.net.dataretriever.GenericClient.post_search_hook` which helps to translate the attrs for scraper before and after the search respectively.

changelog/4321.feature.4.rst

@@ -0,0 +1,4 @@
+New class attributes added to `~sunpy.net.dataretriever.GenericClient`:
+
+- ``baseurl`` and ``pattern`` which are required to define a new simple client.
+- ``optional`` and ``required`` which are a ``set`` of optional and required `~sunpy.net.attrs` respectively; which generalizes :meth:`~sunpy.net.dataretriever.GenericClient._can_handle_query`.

changelog/4321.feature.rst

@@ -0,0 +1,4 @@
+Refactoring of `~sunpy.net.dataretriever` which adds these capabilities to `~sunpy.net.dataretriever.QueryResponse`:
+
+- Any ``attr`` shall not be defaulted to a hard-coded value in all subclasses of `~sunpy.net.dataretriever.GenericClient`; thus records for all possible ``attrs`` shall be returned if it is not specified in the query.
+- `~sunpy.net.dataretriever.QueryResponse` can now show more columns; thus all metadata extractable from matching file URLs shall be shown and for a client, non-spported ``attrs`` shall not be shown in the response tables.

changelog/4321.removal.rst

@@ -0,0 +1,2 @@
+Removed `~sunpy.net.vso.attrs.Source` and `~sunpy.net.vso.attrs.Provider`. They are now `~sunpy.net.attrs.Source` and `~sunpy.net.attrs.Provider` respectively.
+Thus ``attrs``, ``Source`` and ``Provider`` are moved from VSO namespace to `~sunpy.net.attrs`.

docs/guide/acquiring_data/fido.rst

@@ -62,19 +62,6 @@ Using ``Instrument`` as the first example, if you print the object::
 You get a full list of known values, a description and what "Clients" support those values (if you want to use a specific data source).
 This is supported for most attributes including the client specific ones.
 
-For VSO::
-
-    >>> print(a.vso.Provider)
-    sunpy.net.vso.attrs.Provider
-    <BLANKLINE>
-    Specifies the VSO data provider to search for data for.
-    <BLANKLINE>
-    Attribute Name Client Full Name                                   Description
-    -------------- ------ --------- --------------------------------------------------------------------------------
-    hao            VSO    HAO       High Altitude Observatory, NCAR
-    jsoc           VSO    JSOC      SDO Joint Science Operations Center
-    kis            VSO    KIS       Kiepenheuer-Institut für Sonnenphysik
-    ...
 
 For JSOC::
 
@@ -98,7 +85,7 @@ Searching for Data Using Fido
 
 For example::
 
-    >>> result = Fido.search(a.Time('2012/3/4', '2012/3/6'), a.Instrument.lyra) # doctest: +REMOTE_DATA
+    >>> result = Fido.search(a.Time('2012/3/4', '2012/3/6'), a.Instrument.lyra, a.Level.two) # doctest: +REMOTE_DATA
 
 this returns an `~sunpy.net.fido_factory.UnifiedResponse` object containing
 information on the available online files which fit the criteria specified by
@@ -113,11 +100,11 @@ variable set to the Fido search, in this case, result::
     Results from 1 Provider:
     <BLANKLINE>
     3 Results from the LYRAClient:
-         Start Time           End Time      Source Instrument Wavelength
-    ------------------- ------------------- ------ ---------- ----------
-    2012-03-04 00:00:00 2012-03-06 00:00:00 Proba2       lyra        nan
-    2012-03-04 00:00:00 2012-03-06 00:00:00 Proba2       lyra        nan
-    2012-03-04 00:00:00 2012-03-06 00:00:00 Proba2       lyra        nan
+         Start Time           End Time      Instrument ... Source Provider Level
+    ------------------- ------------------- ---------- ... ------ -------- -----
+    2012-03-04 00:00:00 2012-03-04 23:59:59       LYRA ... PROBA2      ESA     2
+    2012-03-05 00:00:00 2012-03-05 23:59:59       LYRA ... PROBA2      ESA     2
+    2012-03-06 00:00:00 2012-03-06 23:59:59       LYRA ... PROBA2      ESA     2
     <BLANKLINE>
     <BLANKLINE>
 
@@ -133,11 +120,11 @@ passbands can be searched for by supplying an `~astropy.units.Quantity` to the
     Results from 1 Provider:
     <BLANKLINE>
     3 Results from the NoRHClient:
-         Start Time           End Time      Source Instrument   Wavelength
-    ------------------- ------------------- ------ ---------- --------------
-    2012-03-04 00:00:00 2012-03-05 00:00:00   NAOJ       NORH 17000000.0 kHz
-    2012-03-05 00:00:00 2012-03-06 00:00:00   NAOJ       NORH 17000000.0 kHz
-    2012-03-06 00:00:00 2012-03-07 00:00:00   NAOJ       NORH 17000000.0 kHz
+         Start Time           End Time      Instrument Source Provider Wavelength
+    ------------------- ------------------- ---------- ------ -------- ----------
+    2012-03-04 00:00:00 2012-03-04 23:59:59       NORH   NAOJ      NRO   17.0 GHz
+    2012-03-05 00:00:00 2012-03-05 23:59:59       NORH   NAOJ      NRO   17.0 GHz
+    2012-03-06 00:00:00 2012-03-06 23:59:59       NORH   NAOJ      NRO   17.0 GHz
     <BLANKLINE>
     <BLANKLINE>
 
@@ -193,10 +180,11 @@ operator would::
     <sunpy.net.fido_factory.UnifiedResponse object at ...>
     Results from 3 Providers:
     <BLANKLINE>
-    1 Results from the LYRAClient:
-         Start Time           End Time      Source Instrument Wavelength
-    ------------------- ------------------- ------ ---------- ----------
-    2012-03-04 00:00:00 2012-03-04 02:00:00 Proba2       lyra        nan
+    2 Results from the LYRAClient:
+         Start Time           End Time      Instrument ... Source Provider Level
+    ------------------- ------------------- ---------- ... ------ -------- -----
+    2012-03-04 00:00:00 2012-03-04 23:59:59       LYRA ... PROBA2      ESA     2
+    2012-03-04 00:00:00 2012-03-04 23:59:59       LYRA ... PROBA2      ESA     3
     <BLANKLINE>
     3 Results from the VSOClient:
        Start Time [1]       End Time [1]    Source ...     Type    Wavelength [2]
@@ -207,9 +195,9 @@ operator would::
     2012-03-04 01:45:40 2012-03-04 02:09:00 RHESSI ... PARTIAL_SUN 3.0 .. 17000.0
     <BLANKLINE>
     1 Results from the RHESSIClient:
-         Start Time           End Time      Source Instrument Wavelength
-    ------------------- ------------------- ------ ---------- ----------
-    2012-03-04 00:00:00 2012-03-04 23:59:59 rhessi     rhessi        nan
+         Start Time           End Time      Instrument ... Source Provider
+    ------------------- ------------------- ---------- ... ------ --------
+    2012-03-04 00:00:00 2012-03-04 23:59:59     RHESSI ... RHESSI     NASA
     <BLANKLINE>
     <BLANKLINE>
 
@@ -227,7 +215,7 @@ the `~sunpy.net.dataretriever.LYRAClient`, and EVE data from the
 `~sunpy.net.vso.VSOClient`::
 
     >>> from sunpy.net import Fido, attrs as a
-    >>> results = Fido.search(a.Time("2012/1/1", "2012/1/2"),
+    >>> results = Fido.search(a.Time("2012/1/1", "2012/1/2"), a.Level.two,
     ...                       a.Instrument.lyra | a.Instrument.eve)  # doctest: +REMOTE_DATA
 
 If you then wanted to inspect just the LYRA data for the whole time range
@@ -239,10 +227,10 @@ results returned by the `~sunpy.net.dataretriever.LYRAClient`::
     Results from 1 Provider:
     <BLANKLINE>
     2 Results from the LYRAClient:
-         Start Time           End Time      Source Instrument Wavelength
-    ------------------- ------------------- ------ ---------- ----------
-    2012-01-01 00:00:00 2012-01-02 00:00:00 Proba2       lyra        nan
-    2012-01-01 00:00:00 2012-01-02 00:00:00 Proba2       lyra        nan
+         Start Time           End Time      Instrument ... Source Provider Level
+    ------------------- ------------------- ---------- ... ------ -------- -----
+    2012-01-01 00:00:00 2012-01-01 23:59:59       LYRA ... PROBA2      ESA     2
+    2012-01-02 00:00:00 2012-02-02 23:59:59       LYRA ... PROBA2      ESA     2
     <BLANKLINE>
     <BLANKLINE>
 
@@ -253,10 +241,10 @@ Or, equivalently::
     Results from 1 Provider:
     <BLANKLINE>
     2 Results from the LYRAClient:
-         Start Time           End Time      Source Instrument Wavelength
-    ------------------- ------------------- ------ ---------- ----------
-    2012-01-01 00:00:00 2012-01-02 00:00:00 Proba2       lyra        nan
-    2012-01-01 00:00:00 2012-01-02 00:00:00 Proba2       lyra        nan
+         Start Time           End Time      Instrument ... Source Provider Level
+    ------------------- ------------------- ---------- ... ------ -------- -----
+    2012-01-01 00:00:00 2012-01-01 23:59:59       LYRA ... PROBA2      ESA     2
+    2012-01-02 00:00:00 2012-02-02 23:59:59       LYRA ... PROBA2      ESA     2
     <BLANKLINE>
     <BLANKLINE>
 

sunpy/database/tables.py

@@ -374,16 +374,16 @@ def _from_fido_search_result_block(cls, sr_block, default_waveunit=None):
         """
         # All attributes of DatabaseEntry that are not in QueryResponseBlock
         # are set as None for now.
-        source = getattr(sr_block, 'source', None)
-        provider = getattr(sr_block, 'provider', None)
-        physobs = getattr(sr_block, 'physobs', None)
+        source = sr_block.get('Source', None)
+        provider = sr_block.get('Provider', None)
+        physobs = sr_block.get('Physobs', None)
         if physobs is not None:
             physobs = str(physobs)
-        instrument = getattr(sr_block, 'instrument', None)
-        time_start = sr_block.time.start.datetime
-        time_end = sr_block.time.end.datetime
+        instrument = sr_block.get('Instrument', None)
+        time_start = sr_block['Time'].start.datetime
+        time_end = sr_block['Time'].end.datetime
 
-        wavelengths = getattr(sr_block, 'wave', None)
+        wavelengths = sr_block.get('Wavelength', np.nan)
         wavelength_temp = {}
         if isinstance(wavelength_temp, tuple):
             # Tuple of values
@@ -409,8 +409,7 @@ def _from_fido_search_result_block(cls, sr_block, default_waveunit=None):
         wavemin = final_values['wavemin']
         wavemax = final_values['wavemax']
 
-        # sr_block.url of a QueryResponseBlock attribute is stored in fileid
-        fileid = str(sr_block.url) if sr_block.url is not None else None
+        fileid = sr_block.get('url', None)
         size = None
         return cls(
             source=source, provider=provider, physobs=physobs, fileid=fileid,
@@ -572,17 +571,17 @@ def entries_from_fido_search_result(sr, default_waveunit=None):
     >>> entries = entries_from_fido_search_result(sr) # doctest: +REMOTE_DATA
     >>> entry = next(entries) # doctest: +REMOTE_DATA
     >>> entry.source # doctest: +REMOTE_DATA
-    'Proba2'
+    'PROBA2'
     >>> entry.provider # doctest: +REMOTE_DATA
-    'esa'
+    'ESA'
     >>> entry.physobs # doctest: +REMOTE_DATA
     'irradiance'
     >>> entry.fileid # doctest: +REMOTE_DATA
     'http://proba2.oma.be/lyra/data/bsd/2012/01/01/lyra_20120101-000000_lev2_std.fits'
     >>> entry.observation_time_start, entry.observation_time_end # doctest: +REMOTE_DATA
-    (datetime.datetime(2012, 1, 1, 0, 0), datetime.datetime(2012, 1, 2, 0, 0))
+    (datetime.datetime(2012, 1, 1, 0, 0),  datetime.datetime(2012, 1, 1, 23, 59, 59, 999000))
     >>> entry.instrument # doctest: +REMOTE_DATA
-    'lyra'
+    'LYRA'
 
     """
     for entry in sr:

sunpy/database/tests/test_database.py

@@ -74,12 +74,11 @@ def fido_search_result():
     # No JSOC query
     return Fido.search(
         net_attrs.Time("2012/1/1", "2012/1/2"),
-        net_attrs.Instrument('lyra') | net_attrs.Instrument('eve') |
+        net_attrs.Instrument('lyra') & net_attrs.Level.two | net_attrs.Instrument('eve') |
         net_attrs.Instrument('XRS') | net_attrs.Instrument('noaa-indices') |
         net_attrs.Instrument('noaa-predict') |
         (net_attrs.Instrument('norh') & net_attrs.Wavelength(17*units.GHz)) |
-        (net_attrs.Instrument('rhessi') & net_attrs.Physobs("summary_lightcurve")) |
-        (net_attrs.Instrument('EVE') & net_attrs.Level(0))
+        (net_attrs.Instrument('rhessi') & net_attrs.Physobs("summary_lightcurve"))
     )
 
 
@@ -1107,8 +1106,8 @@ def test_split_database(split_function_database, database):
         split_function_database, database, net_attrs.Instrument('EIA'))
 
     observed_source_entries = split_function_database.search(
-        vso.attrs.Provider('xyz'), sortby='id')
-    observed_destination_entries = database.search(vso.attrs.Provider('xyz'))
+        net_attrs.Provider('xyz'), sortby='id')
+    observed_destination_entries = database.search(net_attrs.Provider('xyz'))
 
     assert observed_source_entries == [
         DatabaseEntry(id=1, instrument='RHESSI', provider='xyz'),