modified search metadata to return Astropy.Table #3950

DShivansh · 2020-03-31T19:08:33Z

nabobalis · 2020-03-31T19:28:31Z

CHANGELOG.rst

@@ -196,6 +196,7 @@ Trivial/Internal Changes
 - Corrected spelling of 'plotting' in timeseries method (changed 'ploting' to 'plotting'). (`#3429 <https://github.com/sunpy/sunpy/pull/3429>`__)
 - Switched to "importlib_metadata" to get package version to speed up import of SunPy. (`#3449 <https://github.com/sunpy/sunpy/pull/3449>`__)
 - Fix tests for `sunpy.data.data_manager` and ensure they are correctly executed with pytest. (`#3550 <https://github.com/sunpy/sunpy/pull/3550>`__)
+- Modified the `search_metadata` function inside `sunpy.net.jsoc` to return the Astropy.Table instead of pandas dataframe. (`#3950 <https://github.com/sunpy/sunpy/pull/3950>`__)


We don't use this changelog directly. You have to add an RST file into the changelogs folder.

@nabobalis should I remove the change in CHANGELOG and only upload a file in changes.rst? or do we have to keep both?

Remove this change and add a changelog file in to the changelog folder.

nabobalis · 2020-03-31T19:29:00Z

What does the output look like now in terminal compared with before?

DShivansh · 2020-03-31T19:32:03Z

New Output

T_REC                  T_OBS          ...       T_REC_epoch      
-------------------- ----------------------- ... -----------------------
2014-01-01T00:00:01Z 2014-01-01T00:00:08.57Z ... 1993.01.01_00:00:04_TAI
2014-01-01T00:00:13Z 2014-01-01T00:00:20.58Z ... 1993.01.01_00:00:04_TAI
2014-01-01T00:00:25Z 2014-01-01T00:00:32.57Z ... 1993.01.01_00:00:04_TAI
2014-01-01T00:00:37Z 2014-01-01T00:00:44.58Z ... 1993.01.01_00:00:04_TAI
2014-01-01T00:00:49Z 2014-01-01T00:00:56.57Z ... 1993.01.01_00:00:04_TAI
2014-01-01T00:01:01Z 2014-01-01T00:01:08.59Z ... 1993.01.01_00:00:04_TAI
2014-01-01T00:01:13Z 2014-01-01T00:01:20.59Z ... 1993.01.01_00:00:04_TAI
2014-01-01T00:01:25Z 2014-01-01T00:01:32.57Z ... 1993.01.01_00:00:04_TAI
2014-01-01T00:01:37Z 2014-01-01T00:01:44.58Z ... 1993.01.01_00:00:04_TAI
2014-01-01T00:01:49Z 2014-01-01T00:01:56.58Z ... 1993.01.01_00:00:04_TAI
2014-01-01T00:02:01Z 2014-01-01T00:02:08.58Z ... 1993.01.01_00:00:04_TAI

Old output

                                                           T_REC  ...              T_REC_epoch
aia.lev1_euv_12s[2014-01-01T00:00:01Z][304]  2014-01-01T00:00:01Z  ...  1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:00:13Z][304]  2014-01-01T00:00:13Z  ...  1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:00:25Z][304]  2014-01-01T00:00:25Z  ...  1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:00:37Z][304]  2014-01-01T00:00:37Z  ...  1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:00:49Z][304]  2014-01-01T00:00:49Z  ...  1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:01:01Z][304]  2014-01-01T00:01:01Z  ...  1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:01:13Z][304]  2014-01-01T00:01:13Z  ...  1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:01:25Z][304]  2014-01-01T00:01:25Z  ...  1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:01:37Z][304]  2014-01-01T00:01:37Z  ...  1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:01:49Z][304]  2014-01-01T00:01:49Z  ...  1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:02:01Z][304]  2014-01-01T00:02:01Z  ...  1993.01.01_00:00:04_TAI

[11 rows x 176 columns]

nabobalis · 2020-03-31T19:33:48Z

Looking at that, I do think we should keep the first column from the old version.

DShivansh · 2020-03-31T19:37:35Z

first column is actually the index value, @nabobalis should I create a column named index in the astropy.Table and put index-values in it and do the indexing according to them?

nabobalis · 2020-03-31T19:39:38Z

Sure.

CHANGELOG.rst

Co-Authored-By: Nabil Freij <nabil.freij@gmail.com>

pep8speaks · 2020-03-31T20:04:06Z

Hello @DShivansh! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file sunpy/net/jsoc/jsoc.py:

Line 870:54: E251 unexpected spaces around keyword / parameter equals
Line 870:56: E251 unexpected spaces around keyword / parameter equals

Comment last updated at 2020-03-31 20:08:33 UTC

changelog/3950.trivial.rst

nabobalis · 2020-03-31T20:04:52Z

What does the output look like now?

… tableChange

DShivansh · 2020-03-31T20:10:58Z

output

 Index                    ...       T_REC_epoch      
------------------------------------------- ... -----------------------
aia.lev1_euv_12s[2014-01-01T00:00:01Z][304] ... 1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:00:13Z][304] ... 1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:00:25Z][304] ... 1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:00:37Z][304] ... 1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:00:49Z][304] ... 1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:01:01Z][304] ... 1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:01:13Z][304] ... 1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:01:25Z][304] ... 1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:01:37Z][304] ... 1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:01:49Z][304] ... 1993.01.01_00:00:04_TAI
aia.lev1_euv_12s[2014-01-01T00:02:01Z][304] ... 1993.01.01_00:00:04_TAI

nabobalis · 2020-04-09T09:56:41Z

As things stand, this PR is good to go but we need to resolve how to deal with the breaking change or even if we need this function anymore.

These two questions are unresolved right now and as such, we can't merge this PR until then.

DShivansh · 2020-04-09T11:19:25Z

@nabobalis should we raise some type of warning that indicates the user that we don't return pandas dataframe anymore instead it is returning the astropy df.

nabobalis · 2020-04-09T11:21:18Z

The warning needs to have a way to turn it off and warning the user as they run the code after a update isn't really an advanced warning.

DShivansh · 2020-04-09T12:07:59Z

can we use some kind of flag in sunpyrc to turn off the warning once user knows about it?

nabobalis · 2020-04-09T12:22:26Z

I don't have a good solution right now.

dstansby · 2020-05-22T10:37:47Z

The way I have seen this sort of deprecation done previously is to add a return_astropy_table keyword argument, which defaults to None. If it is None, the code warns, telling the user can do return_astropy_table=True to silence the error, and if it is False it still uses the old behavior, but raises a warning saying that this functionality will be removed in a future version.

nabobalis · 2020-07-11T11:31:39Z

I think with the current GSoC project, we will probably remove this method as a result I will be closing this.

I want to thank @DShivansh for opening the PR and I am sorry we decided to change the goal posts.

DShivansh added 2 commits March 31, 2020 21:35

changed the return type from pandas dataframe to astropy table

25d2074

modified the search_metadata function to return Astropy.Table

f1e7aa7

DShivansh requested a review from a team as a code owner March 31, 2020 19:08

added to changelog.rst

1c0fb9a

nabobalis reviewed Mar 31, 2020

View reviewed changes

nabobalis added this to the 2.0 milestone Mar 31, 2020

nabobalis added the net Affects the net submodule label Mar 31, 2020

added the index column

42e45c6

nabobalis reviewed Mar 31, 2020

View reviewed changes

CHANGELOG.rst Outdated Show resolved Hide resolved

Update CHANGELOG.rst

3658429

Co-Authored-By: Nabil Freij <nabil.freij@gmail.com>

nabobalis reviewed Mar 31, 2020

View reviewed changes

changelog/3950.trivial.rst Outdated Show resolved Hide resolved

DShivansh added 2 commits April 1, 2020 01:36

renamed from trivial to breaking

2e64638

Merge branch 'tableChange' of https://github.com/DShivansh/sunpy into…

e9e0244

… tableChange

DShivansh requested review from nabobalis and removed request for a team April 5, 2020 14:43

nabobalis removed their request for review April 9, 2020 09:55

Cadair removed this from the 2.0 milestone Apr 15, 2020

Cadair added this to the 2.1 milestone Apr 15, 2020

nabobalis mentioned this pull request Apr 30, 2020

Returning Astropy Table instead of Pandas DataFrame when using search_metadata() in JSOCClient. #3941

Closed

dstansby added the [WIP] label Jul 5, 2020

nabobalis closed this Jul 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

modified search metadata to return Astropy.Table #3950

modified search metadata to return Astropy.Table #3950

DShivansh commented Mar 31, 2020

nabobalis Mar 31, 2020

DShivansh Mar 31, 2020

nabobalis Mar 31, 2020

nabobalis commented Mar 31, 2020

DShivansh commented Mar 31, 2020

nabobalis commented Mar 31, 2020

DShivansh commented Mar 31, 2020 •

edited

nabobalis commented Mar 31, 2020

pep8speaks commented Mar 31, 2020 •

edited

nabobalis commented Mar 31, 2020

DShivansh commented Mar 31, 2020

nabobalis commented Apr 9, 2020

DShivansh commented Apr 9, 2020

nabobalis commented Apr 9, 2020

DShivansh commented Apr 9, 2020 •

edited

nabobalis commented Apr 9, 2020

dstansby commented May 22, 2020

nabobalis commented Jul 11, 2020

modified search metadata to return Astropy.Table #3950

modified search metadata to return Astropy.Table #3950

Conversation

DShivansh commented Mar 31, 2020

nabobalis Mar 31, 2020

Choose a reason for hiding this comment

DShivansh Mar 31, 2020

Choose a reason for hiding this comment

nabobalis Mar 31, 2020

Choose a reason for hiding this comment

nabobalis commented Mar 31, 2020

DShivansh commented Mar 31, 2020

nabobalis commented Mar 31, 2020

DShivansh commented Mar 31, 2020 • edited

nabobalis commented Mar 31, 2020

pep8speaks commented Mar 31, 2020 • edited

Comment last updated at 2020-03-31 20:08:33 UTC

nabobalis commented Mar 31, 2020

DShivansh commented Mar 31, 2020

nabobalis commented Apr 9, 2020

DShivansh commented Apr 9, 2020

nabobalis commented Apr 9, 2020

DShivansh commented Apr 9, 2020 • edited

nabobalis commented Apr 9, 2020

dstansby commented May 22, 2020

nabobalis commented Jul 11, 2020

DShivansh commented Mar 31, 2020 •

edited

pep8speaks commented Mar 31, 2020 •

edited

DShivansh commented Apr 9, 2020 •

edited