Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added method to extract metadata from url using parse #4313

Merged
merged 4 commits into from Jun 26, 2020

Conversation

abhijeetmanhas
Copy link
Contributor

@abhijeetmanhas abhijeetmanhas commented Jun 23, 2020

Description

It is a split of #4213 in which a new method is added for scraper changes which would possibly be backward compatible.

@pep8speaks
Copy link

pep8speaks commented Jun 23, 2020

Hello @abhijeetmanhas! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 533:11: E114 indentation is not a multiple of four (comment)
Line 534:11: E111 indentation is not a multiple of four
Line 536:11: E114 indentation is not a multiple of four (comment)
Line 537:11: E111 indentation is not a multiple of four
Line 538:15: E111 indentation is not a multiple of four
Line 539:19: E111 indentation is not a multiple of four
Line 540:15: E111 indentation is not a multiple of four
Line 541:19: E111 indentation is not a multiple of four
Line 542:15: E111 indentation is not a multiple of four
Line 543:19: E111 indentation is not a multiple of four
Line 566:13: E128 continuation line under-indented for visual indent
Line 606:9: E128 continuation line under-indented for visual indent
Line 613:10: E225 missing whitespace around operator
Line 614:10: E225 missing whitespace around operator
Line 615:10: E225 missing whitespace around operator
Line 707:5: E128 continuation line under-indented for visual indent
Line 793:17: E128 continuation line under-indented for visual indent
Line 806:25: E128 continuation line under-indented for visual indent
Line 820:25: E128 continuation line under-indented for visual indent
Line 823:21: E128 continuation line under-indented for visual indent
Line 824:21: E128 continuation line under-indented for visual indent
Line 890:39: E231 missing whitespace after ','
Line 924:13: E128 continuation line under-indented for visual indent
Line 986:25: E128 continuation line under-indented for visual indent
Line 987:25: E128 continuation line under-indented for visual indent
Line 1064:101: E501 line too long (105 > 100 characters)
Line 1065:58: E261 at least two spaces before inline comment
Line 1065:101: E501 line too long (115 > 100 characters)
Line 1071:17: E128 continuation line under-indented for visual indent
Line 1078:17: E128 continuation line under-indented for visual indent
Line 1085:17: E128 continuation line under-indented for visual indent
Line 1090:17: E128 continuation line under-indented for visual indent
Line 1093:17: E128 continuation line under-indented for visual indent
Line 1098:17: E128 continuation line under-indented for visual indent
Line 1101:17: E128 continuation line under-indented for visual indent
Line 1108:17: E128 continuation line under-indented for visual indent
Line 1114:17: E128 continuation line under-indented for visual indent
Line 1120:17: E128 continuation line under-indented for visual indent
Line 1204:13: E128 continuation line under-indented for visual indent
Line 1286:9: E128 continuation line under-indented for visual indent
Line 1320:9: E128 continuation line under-indented for visual indent
Line 1347:101: E501 line too long (112 > 100 characters)

Comment last updated at 2020-06-26 11:57:56 UTC

@nabobalis nabobalis added this to the 2.1 milestone Jun 23, 2020
@nabobalis nabobalis added net Affects the net submodule util Issues relating to sunpy.util labels Jun 23, 2020
@nabobalis nabobalis requested a review from Cadair June 23, 2020 19:29
@abhijeetmanhas
Copy link
Contributor Author

@nabobalis
Copy link
Contributor

Need to exclude the entire extern location from pytest in setup.cfg. I would have thought it would have been.

Copy link
Member

@Cadair Cadair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is good, just a few API design questions.

@@ -331,3 +332,42 @@ def _smallerPattern(self, directoryPattern):
return None
except Exception:
raise

def _extractMetaURLs(self, timerange, extractor=None, translator=None):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function name isn't great, python method names are supposed to use snake_case.

I am not sure what a good name for this function is, it's doing a few things (calling filelist as well as running things through parse). Do we want to see this become the main entry point for scraper in a land where you use parse for everything?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, filelist will become internal function for scraper and the client would only be using the extractMetaURLs method.

sunpy/util/scraper.py Outdated Show resolved Hide resolved
@Cadair
Copy link
Member

Cadair commented Jun 24, 2020

if you pull in master the figure test build should pass.

@nabobalis
Copy link
Contributor

Can you rebase this onto master?

@nabobalis
Copy link
Contributor

Windows failures unrelated.

FAILED ..\..\.tox\py37\lib\site-packages\sunpy\coordinates\tests\test_metaframes.py::test_rotatedsun_transforms[HeliocentricInertial] - hypothesis.errors.FailedHealthCheck: Data...
FAILED ..\..\.tox\py37\lib\site-packages\sunpy\coordinates\tests\test_metaframes.py::test_rotatedsun_transforms[HeliographicCarrington] - hypothesis.errors.DeadlineExceeded: Tes...

@nabobalis
Copy link
Contributor

Doc failure is due to

DEBUG: Failed to download SWAP_LEVEL1_IMAGE from http://data.sunpy.org/sunpy/v1/swap_lv1_20110607_063329.fits: Timeout on reading data from socket [sunpy.data._sample]
loading translations [en]... done

which is unrelated to the PR.

@nabobalis nabobalis merged commit 8b78f59 into sunpy:master Jun 26, 2020
@abhijeetmanhas abhijeetmanhas deleted the scraper branch June 29, 2020 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
net Affects the net submodule util Issues relating to sunpy.util
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants