Skip to content

Latest commit

 

History

History
85 lines (71 loc) · 6.1 KB

GSoC2020-Fido-WorkProduct.md

File metadata and controls

85 lines (71 loc) · 6.1 KB
Raw

GSoC 2020 Work Product | OpenAstronomy | Fido

Project Metadata

Description

SunPy's Fido stands for Federated Internet Data Obtainer. It is a versatile and flexible search interface that allows users to query data for various search attributes and different clients. It is used for fetching solar physics data from various HTTP and FTP archives, and now from web-services too.

The first part of the project was to extend the ability for dataretriever clients to specify what metadata about the results should be displayed in their QueryResponse objects. This was achieved by refactoring in the net submodule and additions to the scraper. Sunpy's scraper is now able to extract metadata from filenames using their URLs. An external module, parse was added to SunPy to ease the retrieval of attrs. GenericClient was redesigned and now show() method can be used to specify the columns to be displayed in search responses. The QueryResponses now can even show SatelliteNumber, Wavelength, Level, Provider, Physobs, etc. in their tables which wasn't possible earlier.

The second part was more closer to the central theme of the project. A base Class, BaseQueryResponseTable was created for clients that return mostly tabular data. JSOCResponse, HEKResponse, and HECResponse now inherit this class and it reduced code duplication. After registering the search "attrs" supported by these metadata clients, they were made compatible with Fido. There were a couple of other issues resolved alongside the big pull request for these changes.

In the last part, I wrote the developer guide for Fido. I added examples of how to use the new functionality. UnifiedResponse know supports string-based indexing. It makes retrieving individual provider responses from a Fido result easy, as opposed to int based indexing. Many reviews on old PRs were resolved in the last phase.

Progress

I have been blogging my progress on this project biweekly on Medium. I highly recommend going through my GSoC blogposts. They contain Github gists, ipython notebooks, and code snippets that will make you understand the project better.

  1. The Start of my summer with SunPy.
  2. The Coding period commences!.
  3. Generalization of Clients.
  4. End of the First Half.
  5. Metadata searches using Fido.
  6. Polishing my code.
  7. Thus Ending.

Links to Work Done

Pull Requests made

  1. Reduced time for Goes Suvi tests sunpy/sunpy#4131
  2. Fixed href in scraper sunpy/sunpy#4132
  3. Fixed wrong sat no. in XRS overlapping URLs sunpy/sunpy#4288
  4. Added a show method for base client sunpy/sunpy#4309
  5. Method to extract metadata from URL in scraper sunpy/sunpy#4313
  6. Post Search filters again for VSO sunpy/sunpy#4011
  7. Gong Synoptic Fido client sunpy/sunpy#4055
  8. VSO fetch fix sunpy/sunpy#4026
  9. Redesign GenericClient sunpy/sunpy#4321
  10. Metadata clients compatible with Fido sunpy/sunpy#4358
  11. Fido dev-guide sunpy/sunpy#4387
  12. Method to know time ranges in scraper sunpy/sunpy#4419

Issues I worked upon

  1. Inconsistent Ordering in VSO Results sunpy/sunpy#3922
  2. VSO fails to filter response sunpy/sunpy#3833
  3. Slow fetch in the test for SUVI Client sunpy/sunpy#4099
  4. Wrong Satellite number in overlapping URLs sunpy/sunpy#4207
  5. QueryResponses can't show desired columns sunpy/sunpy#556
  6. Broken VSO fetch for some providers sunpy/sunpy#3734
  7. Support for Gong Synoptic Maps client dstansby/pfsspy#165
  8. GenericClient not knowing file time ranges sunpy/sunpy#3715
  9. GOES hard-codes dates for available satellites sunpy/sunpy#3337
  10. DR Clients can't specify columns to be shown sunpy/sunpy#3321
  11. Generalize can_handle_query sunpy/sunpy#3306
  12. Inconsistent wavelength tuple sunpy/sunpy#2314
  13. Integrate metadata clients in Fido sunpy/sunpy#2744
  14. Wrong links in WSDL helio parser sunpy/sunpy#2032
  15. JSOC ignores keys attrs sunpy/sunpy#3293
  16. JSOC search_metadata returns pandas data frame sunpy/sunpy#3941
  17. JSOC returns empty IRIS blocks sunpy/sunpy#2002

Reported Issues

  1. Wrong XRS sat_no in overlaps sunpy/sunpy#4207
  2. Skip HELIO tests for not-reachable server sunpy/sunpy#4243
  3. Failing fermi remote doctest sunpy/sunpy#4211
  4. Wrong GOES HEK time-series plot sunpy/sunpy#4429
  5. Client unaware of time ranges sunpy/sunpy#3715
  6. Scraper refactoring sunpy/sunpy#4336