Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Fido client and filetype attr #31

Merged
merged 18 commits into from Nov 3, 2022

Conversation

wtbarnes
Copy link
Contributor

@wtbarnes wtbarnes commented Oct 12, 2021

This pull request adds a new Fido client called EISClient that enables searches of the level 1 HDF5 and FITS files via Fido, the sunpy data retrieval interface. An additional attr is also provided to discriminate between the HDF data and header files and FITS files.

Though I've not deleted any code, I would recommend that this supersede the code that is in download/download_hdf5_data.py.

A few things that could still be done:

  • Tests (there are doctests though I don't actually think these get run at the moment)
  • Remove download/download_hdf5_data.py as this provides duplicate functionality
  • Docs page for net subpackage

At the moment, this is difficult to test because eis.nrl.navy.mil is experiencing SSL certificate issues.

This PR should be squash merged.

@wtbarnes wtbarnes marked this pull request as ready for review October 12, 2021 19:34
eispac/net/attrs.py Outdated Show resolved Hide resolved
eispac/net/client.py Show resolved Hide resolved
@MJWeberg
Copy link
Collaborator

This is great! It looks like the data server issues are finally fixed now, so I will go ahead and merge this in and work on updating the docs to match.

For now, I think it is best to keep the download_hdf5_data() function, since it allows for a quick, direct download from the server (with basic handling of partial files) and, most importantly, it is currently used by the eis_catalog search and download GUI. However, this Fido client is a great tool for users who are looking for multi-spacecraft observations or just prefer to use a Python script instead.

One question for future consideration, is it possible to connect a database of metadata to a Fido client? There is a lot of coordinate and wavelength information in the catalog that I assume would be useful to include as search parametersin Fido (although the official EIS as-run catalog is a bit of a maze, so we might need to generate a reduced database for the just the HDF5 files).

@wtbarnes
Copy link
Contributor Author

I think this is finally ready (only about a year late!)

Please squash this before merging. The commit history is a bit all over the place.

@wtbarnes wtbarnes marked this pull request as ready for review October 25, 2022 21:21
bool
True if this client can handle the given query.
"""
required = {a.Time, a.Instrument, a.Source}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need to make Source required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There could be another future instrument called EIS? 🤷

Copy link

@Cadair Cadair Nov 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't mean you need to support source, if someone didn't provide source then they would get both. They could optionally use source to disambiguate.

(It just needs moving to optional).

@wtbarnes
Copy link
Contributor Author

One question for future consideration, is it possible to connect a database of metadata to a Fido client? There is a lot of coordinate and wavelength information in the catalog that I assume would be useful to include as search parameters in Fido (although the official EIS as-run catalog is a bit of a maze, so we might need to generate a reduced database for the just the HDF5 files).

@MJWeberg we talked about this a bit offline a few months ago, but just wanted to record my thoughts here for posterity. This is definitely possible. The difficulty lies in the fact that the EIS catalogue is not a hosted SQL database, but rather a local file that one needs to download. This means it's not just a matter of translating a Fido query into a query against the EIS catalogue, but also downloading the catalogue and making sure it is up to date, with the assumption that there is always an existing correspondence between this local catalogue and what is being provided on the remote server.

I think this is an interesting direction and would enable much more detailed queries, but is more complicated than what is proposed here. I would recommend spinning this out as a separate issue.

@MJWeberg
Copy link
Collaborator

MJWeberg commented Nov 3, 2022

Finally getting back to this now. This is great, Will! Thanks you very much! I agree that linking in to the SQLite database would be nice, but is out of the scope of this PR. For now, people can still use the eis_catalog GUI to query and download observations based on the metadata.

@MJWeberg MJWeberg merged commit fb02e30 into USNavalResearchLaboratory:main Nov 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants