Download from fido search result implemented #1877

Punyaslok · 2016-08-11T21:23:04Z

This PR also contains the commits in #1830 and #1785 and #1856

pep8speaks · 2017-08-05T18:55:30Z

Hello @Punyaslok! Thanks for updating the PR.

In the file sunpy/database/database.py, following are the PEP8 issues :

Line 161:101: E501 line too long (104 > 100 characters)
Line 164:101: E501 line too long (103 > 100 characters)
Line 465:101: E501 line too long (101 > 100 characters)
Line 473:21: E128 continuation line under-indented for visual indent
Line 680:101: E501 line too long (101 > 100 characters)

In the file sunpy/database/tables.py, following are the PEP8 issues :

Line 430:9: E265 block comment should start with '# '
Line 432:9: E265 block comment should start with '# '
Line 664:101: E501 line too long (103 > 100 characters)
Line 708:101: E501 line too long (101 > 100 characters)
Line 727:101: E501 line too long (118 > 100 characters)
Line 731:29: E128 continuation line under-indented for visual indent
Line 732:14: E114 indentation is not a multiple of four (comment)
Line 733:14: E114 indentation is not a multiple of four (comment)
Line 736:101: E501 line too long (120 > 100 characters)
Line 740:33: E128 continuation line under-indented for visual indent

In the file sunpy/database/tests/test_database.py, following are the PEP8 issues :

Line 81:5: E265 block comment should start with '# '
Line 85:8: E128 continuation line under-indented for visual indent
Line 88:9: E128 continuation line under-indented for visual indent
Line 91:9: E128 continuation line under-indented for visual indent
Line 94:9: E128 continuation line under-indented for visual indent
Line 97:8: E128 continuation line under-indented for visual indent
Line 100:10: E128 continuation line under-indented for visual indent
Line 103:9: E128 continuation line under-indented for visual indent
Line 106:9: E128 continuation line under-indented for visual indent
Line 580:5: E125 continuation line with same indent as next logical line
Line 580:5: E128 continuation line under-indented for visual indent
Line 581:1: W293 blank line contains whitespace
Line 705:50: E231 missing whitespace after ','
Line 715:9: E128 continuation line under-indented for visual indent
Line 725:9: E128 continuation line under-indented for visual indent
Line 748:50: E231 missing whitespace after ','
Line 758:9: E128 continuation line under-indented for visual indent
Line 768:9: E128 continuation line under-indented for visual indent

In the file sunpy/net/dataretriever/client.py, following are the PEP8 issues :

Line 382:47: E128 continuation line under-indented for visual indent

In the file sunpy/net/download.py, following are the PEP8 issues :

Line 27:8: E221 multiple spaces before operator
Line 29:1: E302 expected 2 blank lines, found 1
Line 150:9: E265 block comment should start with '# '

Comment last updated on May 03, 2018 at 19:58 Hours UTC

Cadair

It would be good to move Database.fetch over to using Fido.

nabobalis · 2017-08-10T11:42:42Z

Here is a non suspicious patch in a zip.
patch.zip

Will not fix the online tests though.

nabobalis · 2017-08-11T09:01:39Z

Looks like a real test failure.

    def test_entries_from_file_time_string_parse_format():
    
        with pytest.raises(ValueError):
            # Error should be  raised because of the date format in GOES_DATA
>           entries = list(entries_from_file(GOES_DATA))
E           Failed: DID NOT RAISE <class 'ValueError'>

…_result_block in tables.py

…ormat dd/mm/yyyy

… created.

…as in path, then fname was not assigned any value as both if and elif were skipped.

Cadair · 2018-05-03T19:36:47Z

docs/guide/acquiring_data/database.rst

+
+    >>> from sunpy.net import Fido, attrs as a   # doctest:  +REMOTE_DATA
+    >>> search_result = Fido.search(a.Time("2012/1/1", "2012/1/2"),
+    ...                              a.Instrument('lyra'))   # doctest:  +REMOTE_DATA


indentation error

Cadair · 2018-05-03T19:38:12Z

docs/guide/acquiring_data/database.rst

+resulting search result determines the number of entries that will be added 
+to the database. The number of entries that will be added depends on the total 
+number of FITS headers. The :meth:`Database.download_from_fido_search_result()` 
+method also accepts an optional keyword argument `path` which determines the 


should be two backticks as it's not a code reference

Cadair · 2018-05-03T19:38:19Z

docs/guide/acquiring_data/database.rst

+
+    >>> database.download_from_fido_search_result(search_result)   # doctest:  +REMOTE_DATA
+    >>> print(display_entries(database, ['id', 'observation_time_start', 
+    ...                     'observation_time_end', 'instrument', 'source']))   # doctest:  +REMOTE_DATA


indentation

Cadair · 2018-05-03T19:41:06Z

sunpy/database/database.py

+        for sr_entry, temp in zip(search_result, entries_list):
+            for database_entry in self:
+                if database_entry.path is not None and sr_entry._compare_attributes(database_entry,
+                    ["source", "provider", "physobs", "fileid", "observation_time_start",


I would indent this a but more so it dosen't align to the following lines.

Cadair · 2018-05-03T19:44:08Z

sunpy/database/database.py

+                        delete_entries.append(database_entry)
+
+        for temp in remove_list:
+            search_result = [x for x in search_result if x != temp]


You could re-write this using sets rather than the for loop.

I remembered the reason not to use set difference for this. It will not maintain the ordering of the elements. Should I leave it as it is ?

Cadair · 2018-05-03T19:55:24Z

sunpy/database/database.py

+        for (path, sr_entry) in zip(paths, entries_list):
+
+            try:
+                read_file_header(path)


As discussed this probably shouldn't be here, as entries_from_file should raise if there is an error.

My commit means you don't need this line anymore, I think

Cadair · 2018-05-03T19:55:52Z

sunpy/database/database.py

+                if os.path.isfile(path):
+                    entries = tables.entries_from_file(path, self.default_waveunit)
+                elif os.path.isdir(path):
+                    entries = tables.entries_from_dir(path, self.default_waveunit)


We should modify entries_from_dir to support returning a bare bones entry for non-sunpy io supported files.

Ok, so path should never actually be a directory, so we can probably just drop this line or replace it with an exception.

Cadair · 2018-05-03T20:04:01Z

sunpy/net/download.py

@@ -147,6 +147,8 @@ def download(self, url, path=None, callback=None, errback=None):
        server = self._get_server(url)

        # Create function to compute the filepath to download to if not set
+        #if path is not None:


are these supposed to be here?

Cadair · 2018-05-03T20:06:47Z

sunpy/database/tables.py

+        try:
+            instrument_name = next(x for x in entry.fits_header_entries if x.key == 'TELESCOP').value
+        except Exception:
+            pass


this should be instrument_name = '' so that it's defined below.

Cadair · 2018-05-03T20:07:38Z

sunpy/database/tables.py

+                    if 'goes' in instrument_name.lower():
+                        entry.observation_time_end = datetime.strptime(value,
+                            '%d/%m/%Y')
+             #       else:


uncomment this

Cadair · 2018-05-03T20:07:45Z

sunpy/database/tables.py

+                        if 'goes' in instrument_name.lower():
+                            entry.observation_time_start = datetime.strptime(value,
+                                '%d/%m/%Y')
+            #            else:


Cadair · 2018-05-03T20:08:10Z

sunpy/net/dataretriever/client.py

            dobj.download(aurl, fname, ncall, error_callback)
+        #    dobj.download(aurl, kwargs.get('path', None), ncall,


stale · 2018-09-30T22:48:19Z

This pull request has been automatically marked as stale because it has not had any activity for the past five months. It will be closed if no further activity occurs. If the ideas in this pull request are still worth implementing, please make sure you open an issue to keep track of that idea!

stale · 2018-10-30T22:54:49Z

This pull request has been automatically closed since there was no activity for a month since it was marked as stale. If the ideas in this pull request are still worth implementing, please make sure you open an issue to keep track of that idea!

Cadair changed the title ~~Download from fido search result implemented~~ [unidown] Download from fido search result implemented Aug 11, 2016

Cadair added the unidown label Sep 15, 2016

dpshelio changed the base branch from unidown to master January 11, 2017 17:17

Cadair changed the title ~~[unidown] Download from fido search result implemented~~ Download from fido search result implemented Jan 11, 2017

stale bot added the Stale The bot will close this PR after 6 months label Jun 11, 2017

stale bot closed this Jul 12, 2017

nabobalis added the Needs Adoption PRs that were abandoned but should be picked up again and merged in label Jul 12, 2017

Cadair self-assigned this Jul 12, 2017

Punyaslok reopened this Aug 5, 2017

Punyaslok removed Needs Adoption PRs that were abandoned but should be picked up again and merged in Stale The bot will close this PR after 6 months labels Aug 5, 2017

Punyaslok assigned Punyaslok and unassigned Cadair Aug 5, 2017

Cadair reviewed Aug 7, 2017

View reviewed changes

Cadair added this to Nice to Haves in SunPy 0.8 Aug 10, 2017

nabobalis added this to Non-Blocking Tasks in SunPy 0.9 Aug 12, 2017

Punyaslok added 11 commits August 15, 2017 14:14

Function to produce Database entries from a Fido Search result written.

e64af01

Tests for add_from_fido_search_result function written

88d8696

Docstring for _from_fido_search_result_block in tables.py added.

ac31163

Tests added for entries_from_fido_search_result and _from_fido_search…

f51ab0f

…_result_block in tables.py

Download from Fido Search Result function added

bfc4507

Fixed time parse error coming from goes files which had time of the f…

119f0c7

…ormat dd/mm/yyyy

Uncommented caching part

b84e54b

Path was changed to path. Also of directory doesnt exist a new one is…

70e0fb8

… created.

Moved the path processing to GenericClient

5332702

Additional file types added.

cd02451

Added an else condition. Previously if path was not none and {file} w…

123ec3a

…as in path, then fname was not assigned any value as both if and elif were skipped.

Punyaslok added 2 commits May 1, 2018 12:02

Test was repeated

2e3b393

Fix conflict

98d169c

Punyaslok reopened this May 2, 2018

Merge branch 'master' of github.com:sunpy/sunpy into pr/1877

8fb9a64

nabobalis removed the Needs Adoption PRs that were abandoned but should be picked up again and merged in label May 2, 2018

sunpy deleted a comment from stale bot May 2, 2018

Punyaslok added 8 commits May 2, 2018 12:09

Changed online to remote_data in tests

6cf48ba

Removed repeated code, and fixed the doctest

8240ae1

Trying to fix database.rst

da6183f

Still trying to fix database.rst

c872b46

Added the print statement

98309b1

Fixed some docs

eae1c13

Fixing some damn spacing

4097c4b

Trying to see if tests pass. Removing SKIP

048702d

Cadair reviewed May 3, 2018

View reviewed changes

Punyaslok and others added 2 commits May 3, 2018 15:56

pep8

945985a

Small changes to tables to use sunpy.io better

417c181

Cadair reviewed May 3, 2018

View reviewed changes

stale bot added the Stale The bot will close this PR after 6 months label Sep 30, 2018

stale bot closed this Oct 30, 2018

nabobalis added Needs Adoption PRs that were abandoned but should be picked up again and merged in and removed Stale The bot will close this PR after 6 months labels Oct 30, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download from fido search result implemented #1877

Download from fido search result implemented #1877

Punyaslok commented Aug 11, 2016 •

edited

pep8speaks commented Aug 5, 2017 •

edited

Cadair left a comment

nabobalis commented Aug 10, 2017 •

edited

nabobalis commented Aug 11, 2017

Cadair May 3, 2018

Cadair May 3, 2018

Cadair May 3, 2018

Cadair May 3, 2018

Cadair May 3, 2018

Punyaslok May 3, 2018

Cadair May 3, 2018

Cadair May 3, 2018

Cadair May 3, 2018

Cadair May 3, 2018

Cadair May 3, 2018

Cadair May 3, 2018

Cadair May 3, 2018

Cadair May 3, 2018

Cadair May 3, 2018

stale bot commented Sep 30, 2018

stale bot commented Oct 30, 2018

		dobj.download(aurl, fname, ncall, error_callback)
		# dobj.download(aurl, kwargs.get('path', None), ncall,

Download from fido search result implemented #1877

Download from fido search result implemented #1877

Conversation

Punyaslok commented Aug 11, 2016 • edited

pep8speaks commented Aug 5, 2017 • edited

Comment last updated on May 03, 2018 at 19:58 Hours UTC

Cadair left a comment

Choose a reason for hiding this comment

nabobalis commented Aug 10, 2017 • edited

nabobalis commented Aug 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stale bot commented Sep 30, 2018

stale bot commented Oct 30, 2018

Punyaslok commented Aug 11, 2016 •

edited

pep8speaks commented Aug 5, 2017 •

edited

nabobalis commented Aug 10, 2017 •

edited