New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update mms.py to download DES and fix FGM #601

Merged
merged 9 commits into from Oct 21, 2018

Conversation

Projects
None yet
2 participants
@JeffreyBroll
Contributor

JeffreyBroll commented Oct 18, 2018

(1) Old FGM version number was hardcoded into fgm_survey. This is not sustainable (survey data in March 2016 alone has 4 distinct versions, because they're absolute maniacs) so instead we ask SDC for a JSON with filenames matching those dates and fetch accordingly. Processing may change down the line for some reason I haven't foreseen, but at least this way data are downloading...

(2) The FPI team is slightly less absolutely maniacal so the version changes can be handled more gracefully - or just left as is. The DES and DIS can be grabbed with a one-letter difference so I've gone and done that.

JeffreyBroll added some commits Oct 18, 2018

Update mms.py to download DES and fix FGM
(1) Old FGM version number was hardcoded into fgm_survey.  This is not sustainable (survey data in March 2016 has *4* distinct versions, because they're absolute maniacs) so instead we ask SDC for a JSON with filenames matching those dates and fetch accordingly.  Processing may change down the line for some reason I haven't foreseen, but at least this way data are downloading...

(2) The FPI team is slightly less absolutely maniacal so the version changes can be handled more gracefully - or just left as is.  The DES and DIS can be grabbed with a one-letter difference so I've gone and done that.
@dstansby

This comment has been minimized.

Collaborator

dstansby commented Oct 18, 2018

Eek, thanks for spotting that. We're trying to move data downloading (where possible) to the CDAS RESTful services, which don't have these kind of problems with version numbers (I think?). It might be worth just converting MMS to these new functions, instead of trying to paper over the cracks in the current code in HelioPy. #600 is an example of the latest function I moved over - if you want you could have a go at moving over some of the MMS functions?

@JeffreyBroll

This comment has been minimized.

Contributor

JeffreyBroll commented Oct 18, 2018

I'll try to take a crack at those when I can, but this is what I already had handy. For now I always go through the SDC, since she's still alive up there and I don't know how quickly CDAWeb gets updated files, but maybe a middle ground is putting the file fetch into its own routine, using that for all MMSey things, and changing that to CDAS when it's all live?

JeffreyBroll added some commits Oct 18, 2018

@JeffreyBroll

This comment has been minimized.

Contributor

JeffreyBroll commented Oct 19, 2018

Well, some thoroughly stupid tests later (mainly I kept syncing the wrong version, sorry) and at least this partial version is going. I think I can take a better look at ai.cdas and get some firm answers on CDAS freshness in the very near future, at least.

If there's a wishlist to data fetching apart from 'get data, open with cdflib' I'll try to implement some of that as well. Much of this can be automated by filename/dataset name (e.g. coordinate systems, units - which are MMS-wise standardized and not necessarily read into units objects) but I don't want to step on any toes.

Show resolved Hide resolved heliopy/data/mms.py Outdated
Show resolved Hide resolved heliopy/data/mms.py Outdated
Show resolved Hide resolved heliopy/data/mms.py

JeffreyBroll added some commits Oct 19, 2018

Update mms.py
I think I'm done mangling those strings now.
Update mms.py
Nope, missed some whitespace apparently.  This should address everything

@dstansby dstansby added this to the 0.6.2 milestone Oct 21, 2018

@dstansby

Thanks for the fix! It looks like it's possible to put together a file query for MMS data in the same way you're getting the filename with requests below, so it would be nice to change over to using this way of downloading in future (so it's not dependent on version numbers)

@dstansby dstansby merged commit ce6399b into heliopython:master Oct 21, 2018

4 of 6 checks passed

codecov/patch 29.54% of diff hit (target 80.26%)
Details
codecov/project 79.31% (-0.95%) compared to 2acbafc
Details
ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
giles Click details to preview the documentation build
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment