Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Updated cosmic_gps to use public download option #17

Merged
merged 23 commits into from
May 27, 2021

Conversation

rstoneback
Copy link
Collaborator

@rstoneback rstoneback commented Apr 7, 2021

Addresses: #13, #1 (partial)

Updates cosmic_gps to use the new public data server from CDAAC instead of the option that required authentication. Also turns on tests for cosmic_gps.

The format_str for parsing cosmic file lists was updated to parse out the ground station, version, and satellite ID. The ground station and ID parameters used 'cycle' and 'revision' keyword parameters. I also switched from the delimited filename parsing to fixed width since not all extracted parameters have the appropriate delimiter.

The check for duplicate times was updated to also performs a version check.

The time shift added to files is now constant for a given file, thus if a new version is released and downloaded, it will have the same time shift as the previous version, and will get filtered out by the version check.

@rstoneback rstoneback linked an issue Apr 7, 2021 that may be closed by this pull request
@rstoneback
Copy link
Collaborator Author

@jklenzing - I believe there was some discussion about the sub-directory structure at one point. For this pull, I updated things to be year/doy/file. They wereyear.doy/file. It has been long enough for me that I don't remember if we talked about what to move to. What kind of improvement to directory structure are you talking about for #1?

@rstoneback
Copy link
Collaborator Author

Requires #14 first

@rstoneback rstoneback added this to the 0.1.0 Release milestone Apr 7, 2021
@jklenzing
Copy link
Member

@jklenzing - I believe there was some discussion about the sub-directory structure at one point. For this pull, I updated things to be year/doy/file. They wereyear.doy/file. It has been long enough for me that I don't remember if we talked about what to move to. What kind of improvement to directory structure are you talking about for #1?

I think this should work. As I recall, the . Was giving me trouble

@rstoneback
Copy link
Collaborator Author

I also tried to update the format_str to include parsing the year and day from the leading directories but that didn't work. It doesn't like leading or trailing keywords.

@rstoneback
Copy link
Collaborator Author

Tested new file listing function on my machine with a bunch of GPS files.

In [1]: import pysat                                                                                                                                                      
g
In [2]: gps = pysat.Instrument('cosmic', 'gps', 'ionprf', update_files=True)                                                                                              

In [3]: gps.files.files                                                                                                                                                   
Out[3]: 
2009-01-01 00:12:00.125999872    2009.001/ionPrf_C001.2009.001.00.12.G26_2013.3...
2009-01-01 00:16:00.224000000    2009.001/ionPrf_C002.2009.001.00.16.G24_2013.3...
2009-01-01 00:17:00.209000192    2009.001/ionPrf_C002.2009.001.00.17.G09_2013.3...
2009-01-01 00:27:00.109999872    2009.001/ionPrf_C001.2009.001.00.27.G10_2013.3...
2009-01-01 00:28:00.212999936    2009.001/ionPrf_C002.2009.001.00.28.G13_2013.3...
2009-01-01 00:31:00.530999808    2009.001/ionPrf_C005.2009.001.00.31.G31_2013.3...
In [4]: len(gps.files.files)                                                                                                                                              
Out[4]: 2319390

@jklenzing
Copy link
Member

84.5% coverage! 👀

@rstoneback
Copy link
Collaborator Author

level-2 files are in. Turns out there are a whole host of level-1 files as well. For after lunch.

@rstoneback
Copy link
Collaborator Author

Long lunch but I've expanded support to latest level-2 and level-1 files. I left out files with unknown type or content.

Copy link
Member

@jklenzing jklenzing left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Data loads and downloads as expected. Fixing the "time mangling" of the uts time stamps is outside the scope here, but there should be something in the top-level docstring (either as a warning or a note) that uts is adjusted based on antenna number, etc.

pysatCDAAC/instruments/cosmic_gps.py Outdated Show resolved Hide resolved
pysatCDAAC/instruments/cosmic_gps.py Outdated Show resolved Hide resolved
pysatCDAAC/instruments/cosmic_gps.py Outdated Show resolved Hide resolved
@rstoneback
Copy link
Collaborator Author

Updates to time mangling are going to be handled in #18. I referenced your review note about the docstring over there.

@rstoneback rstoneback requested a review from jklenzing May 26, 2021 23:04
@rstoneback rstoneback merged commit 69014dc into develop May 27, 2021
@rstoneback rstoneback deleted the public_data branch May 27, 2021 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ENH: Updated download address for COSMIC-1 for public access
2 participants