Implemented ACE Clients in the current version of SunPy

[abhijeetmanhas]

In continuation with PR #1806 , I have addressed pep8 issues.
The merge can't be done due to previous versions of attrs.py and client.py files. I will fix the mergeability issues in future commits of this PR after getting the changes reviewed.

[pep8speaks]

Hello @abhijeetmanhas! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-02-05 14:15:17 UTC

[abhijeetmanhas]

Line too long errors are because in the ace file, the examples how to use within comments are of Fido queries, like below in a single line
>>> results = Fido.search(a.Time('2016/5/18 00:00:00', '2016/5/20 00:03:00'), a.Instrument('epam'))

it can be ignored or reduced by using
query = a.Time('2016/5/18 00:00:00', '2016/5/20 00:03:00'), a.Instrument('epam')
results = Fido.search(query)
And one doubt, how to fix mergeability issues with attrs and client file?

[nabobalis]

And one doubt, how to fix mergeability issues with attrs and client file?

You are gonna have to workout what parts of those files you need to port into the current version of those files using some sort of merge tool (meld for example).

[abhijeetmanhas]

I will check the master version of all functions imported imported in ace.py file and its test. And will change accordingly to pass tests

[abhijeetmanhas]

This is expected from Fido search, but

>>> from sunpy.net import Fido
>>> from sunpy.net import attrs as a
>>> results = Fido.search(a.Time('2016/5/18 00:00:00', '2016/5/20 00:03:00'), a.Instrument('swepam'))
>>> print(results)
    [<Table length=3>
         Start Time           End Time      Source Instrument
            str19               str19         str3     str6
    ------------------- ------------------- ------ ----------
    2016-05-18 00:00:00 2016-05-19 00:00:00    ACE     swepam
    2016-05-19 00:00:00 2016-05-20 00:00:00    ACE     swepam
    2016-05-20 00:00:00 2016-05-21 00:00:00    ACE     swepam]

But I'm getting this:

Results from 1 Provider:
    <BLANKLINE>
    3 Results from the SWEPAMClient:
         Start Time           End Time      Source Instrument Wavelength
           str19               str19         str3     str6       str3   
    ------------------- ------------------- ------ ---------- ----------
    2016-05-18 00:00:00 2016-05-20 00:03:00    ACE     swepam        nan
    2016-05-18 00:00:00 2016-05-20 00:03:00    ACE     swepam        nan
    2016-05-18 00:00:00 2016-05-20 00:03:00    ACE     swepam        nan

I can't understand why Fido.search is giving same start and end times. It would be great if someone could suggest what changes should I make.

[abhijeetmanhas]

@Cadair, @nabobalis please review this, I have also implemented _get_time_for_url function.

[nabobalis]

Right now just comments/suggestions for docstrings, will need to be repeated for each client.

[nabobalis]

It's good to see these get old PRs brought back.

[abhijeetmanhas]

I have done all changes suggested by @nabobalis in ace.py file, for all clients. Just need small clarifications regarding attrs.py file.

[abhijeetmanhas]

@nabobalis I have done all the suggested changes.

[abhijeetmanhas]

@Cadair after a PR related to attrs.py was merged, mine shows conflicts. Please review the rest of changes of implementing scraper, is it ok to resolve conflicts using github gui in sunpy ?

[dstansby]

As a note, HelioPy currently has functionality for loading this data: https://docs.heliopy.org/en/stable/reference/data/ace.html

[nabobalis]

As a note, HelioPy currently has functionality for loading this data: https://docs.heliopy.org/en/stable/reference/data/ace.html

Would you like a reference to this fact in the client somewhere?

[abhijeetmanhas]

@Cadair all tests passed now!

[wtbarnes]

There was a question on the sunpy call today about the source of the ACE data. The source used here seems to only go back to 2015 despite the fact that the ACE data series goes back much further (into the 1990s). In heliopy, it seems they're using a different data source that presumably covers the entire range of the data: https://github.com/heliopython/heliopy/blob/c55743c00e9c95dc8d317a72730fb0d076560aa7/heliopy/data/ace.py#L4

[samaloney]

Yea I think the canonical data source would be spdf.gsfc.nasa.giov and since they will probably have to shutdown ftp access at some point in the near future https might be the best option

https://spdf.gsfc.nasa.gov/pub/data/ace/

[abhijeetmanhas]

I can implement scraper for https as well, since https has superset of data over ftp data.
https site have 4 or 5 more clients, which are not in ftp I think.

So should this be done after merging this PR in a separate one or should I continue within this PR?

[Cadair]

I think we should swap to the most "official" and complete data source in this PR if that's ok @abhijeetmanhas

[abhijeetmanhas]

So what's the final decision regarding ACE? I will need some guidance for implementing using above mention source, regarding hdf, cdf, h0, h2, k0, k1,etc various types and what "exactly" the scraper should access? For example here in cdaweb: https://spdf.gsfc.nasa.gov/pub/data/ace/swepam/level_2_cdaweb/ ?

All the subfolders or just one of them, if all where I should specify it (filtered through Fido or client query and there is one default) and the format cdf or hdf.

[abhijeetmanhas]

Closing this now. Thanks everyone for reviewing it, this really helped me to learn a lot!