Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ESO: fixes to retrieve_data() and improvements in login() #420

Merged
merged 14 commits into from
Sep 15, 2014
Merged

ESO: fixes to retrieve_data() and improvements in login() #420

merged 14 commits into from
Sep 15, 2014

Conversation

jwoillez
Copy link
Member

Fixes:

  • form activation wasn't properly handled for checkboxes
  • download link needed an update

Improvements:

  • made username optional in login()
  • use configuration item for default username

keflavich
keflavich reviewed Sep 12, 2014
View reviewed changes
astroquery/eso/core.py
@@ -539,8 +541,7 @@ def retrieve_data(self, datasets, cache=True):
" perhaps the requested file could not be found?")
log.info("Downloading files...")
for fileId in root.select('input[name=fileId]'):
fileLink = fileId.attrs['value'].split()[1]
fileLink = fileLink.replace("/api", "").replace("https://", "http://")
fileLink = "http://dataportal.eso.org/dataPortal"+fileId.attrs['value'].split()[1]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this universally correct for all ESO data products (if you know)? Should this be configurable or hard-coded as is?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't know. The full file link is also available but I am having a hard time extracting it intelligently. I would say, for now, it just works... (tm)

@keflavich
Copy link
Contributor

Could you add a remote test that will catch this? i.e., the query you performed where you got unexpected results?

@jwoillez
Copy link
Member Author

Right away... ;)

@jwoillez
Copy link
Member Author

This is going to be difficult as we need to authenticate first. Maybe we should support login() with an empty ID and have login and password be input interactively...

@keflavich
Copy link
Contributor

Testing for the data download will be difficult, I agree - perhaps we can ask the ESO archivists for a special account for testing that would be granted access only to public data, and store that password publicly?

But, testing for the query should not require login

@jwoillez
Copy link
Member Author

But none of the failing issues were in the non-authenticated zone. Marching forward...

@keflavich
Copy link
Contributor

Wouldn't the checkbox issue show up in the non-authenticated zone? If not, I guess we need to find a way to test when authenticated.

@jwoillez
Copy link
Member Author

And it does not work. Running tests and raw_input don't go together. I should use a username configuration item instead...

@jwoillez
Copy link
Member Author

And the above works when having your username configured. But the test suite does not pick-up your configuration... I need help from an astropy ConfigItem guru...

@keflavich
Copy link
Contributor

So you put the line username = 'JWOILLEZ' under [eso] in ~/.astropy/config/astroquery.cfg and that is not getting picked up?

@jwoillez
Copy link
Member Author

Yes, not picked-up when the test suite is run with: python setup.py test -P eso -R

@keflavich
Copy link
Contributor

@astrofrog - can you comment here? Are configuration files ignored during testing?

@jwoillez
Copy link
Member Author

Just in case, does not work either with:

>>> import astroquery
>>> astroquery.test(remote_data=True, package='eso')

keflavich
keflavich reviewed Sep 12, 2014
View reviewed changes
astroquery/eso/tests/test_eso_remote.py
@@ -104,3 +104,10 @@ def test_list_instruments(self):
# data_files = eso.retrieve_data([data_product_id])
# # How do we know if we're going to get .fits or .fits.Z?
# assert 'AMBER.2006-03-14T07:40:03.741.fits' in data_files[0]

def test_retrieve_data(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still fails now, right? Could you mark this as failing; @pytest.mark.xfail I think? Then I'll merge this and we can open a fresh issue for the tests.

@jwoillez
Copy link
Member Author

Rebased against master... should be all good now. Except for the xfail...

@jwoillez jwoillez changed the title Fixes to ESO's retrieve_data() ESO: fixes to retrieve_data() and improvements in login() Sep 12, 2014
@jwoillez
Copy link
Member Author

With change log included!

@jwoillez
Copy link
Member Author

This is the reason why the local configuration file is not used when running tests:
https://github.com/astropy/astropy/blob/master/astropy/tests/helper.py#L252

@keflavich
Copy link
Contributor

Ah, you have one of the XDG variables set?

@jwoillez
Copy link
Member Author

Done by TestRunner(). Do we have a good enough use case to request a default_config=True option to TestRunner.run_tests()?

@keflavich
Copy link
Contributor

Yes. I think it's the only way we'll be able to run login tests.

@jwoillez
Copy link
Member Author

With the above, you can now omit login(): as long as your username is configured, retrieve_data() will log you in if needed.

PS: I need to update CHANGES and documentation.

@jwoillez
Copy link
Member Author

@keflavich - Good for review. Let's address the xfail at a later stage.

@keflavich
Copy link
Contributor

@jwoillez - OK, passes review. Let's use @pytest.mark.skipif('not eso.conf.USERNAME') once we have a solution that allows the config to be used for testing.

keflavich added a commit that referenced this pull request Sep 15, 2014
@keflavich
Merge pull request #420 from jwoillez/form_activation
d059a58 
ESO: fixes to retrieve_data() and improvements in login()
@keflavich keflavich merged commit d059a58 into astropy:master Sep 15, 2014
@jwoillez
Copy link
Member Author

This @pytest.mark.skipif('not eso.conf.USERNAME') seems very powerful!

This was referenced Sep 15, 2014
Closed
Merged
jwoillez added a commit to jwoillez/astroquery that referenced this pull request Sep 15, 2014
@jwoillez
Reference astropy#427 in CHANGES
e16ed86 
Fixes astropy#420
@jwoillez jwoillez deleted the form_activation branch September 16, 2014 19:12
agroener pushed a commit to agroener/astroquery that referenced this pull request Sep 23, 2014
@jwoillez @agroener
Reference astropy#427 in CHANGES
eebad37 
Fixes astropy#420
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jwoillez @keflavich