Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to use astropy's get_readable_fileobj #23

Closed
astrofrog opened this issue Nov 23, 2012 · 5 comments
Closed

Switch to use astropy's get_readable_fileobj #23

astrofrog opened this issue Nov 23, 2012 · 5 comments

Comments

@astrofrog
Copy link
Member

We recently put a lot of effort into developing a robust I/O routine in astropy.utils.data that can read local or remote data, supports caching, and shows a progress bar for downloads, e.g.:

In [5]: with get_readable_fileobj('http://www.eso.org/public/archives/images/original/eso0932a.tif') as f:
            results = f.read()
   ...:     
Downloading http://www.eso.org/public/archives/images/original/eso0932a.tif
|==========================================|  26/ 26M (100.00%)        10s

It supports on-the-fly decompression of gzip and bzip2 too. We should probably make use of this instead of chunk_read.

@keflavich
Copy link
Contributor

This is great! High on the to-do list. Will this be part of the astropy 0.2 release?

@astrofrog
Copy link
Member Author

Yes, it's already in the developer version, so we can already update technically. I would do it, but I'm short on time, so feel free to try it out. The only trick to be aware of is that you have to use it as a context manager, not a normal function.

@keflavich
Copy link
Contributor

@astrofrog - I started translating over some of the downloads. It would be nice if get_readable_fileobj had a verbose option, or some way to turn off the progressbar. I'm going to leave this in my feature branch until the next astropy release to avoid introducing a new dependency on development code.

@eteq
Copy link
Member

eteq commented Dec 23, 2012

@keflavich - why is it that you want to turn off the progress bar? It's supposed to figure that out automatically (e.g. it only appears when you're at a terminal or the like)

@eteq
Copy link
Member

eteq commented Dec 23, 2012

oh, ignore that comment - better to discuss this on #28

keflavich referenced this issue in keflavich/astroquery Feb 24, 2013
… per Issue #23

Conflicts:

	astroquery/ukidss/ukidss.py

minor bugfix in IRSA fileobj useage

added astropy.utils.data import (oops!)
keflavich referenced this issue in keflavich/astroquery Feb 24, 2013
… per Issue #23

Conflicts:

	astroquery/ukidss/ukidss.py

minor bugfix in IRSA fileobj useage

added astropy.utils.data import (oops!)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants