(fits) File list downloader #69
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can you pull keflavich#2 ? And can you change the example to some files that are
That will make it easier for people to try it out ... we could even make it into a unit test? |
Fix imports and add __all__ for download_file_list
|
Good ideas & question. @astrofrog, is this functionality planned for astropy? Should this sort of downloader go into astropy main? Would astropy really accept a thing with so many clunky keywords (they're clunky, but I think this approach is necessary)? For the others, they're on a to-do list. Probably an IRSA search is best for permanence & speed, maybe a few tiny 2MASS cutouts or spectra. |
|
I'm going to merge this as a stop-gap so we can continue development on the Fermi LAT tool. |
keflavich
pushed a commit
to keflavich/astroquery
that referenced
this pull request
Jun 30, 2014
keflavich
pushed a commit
to keflavich/astroquery
that referenced
this pull request
Feb 5, 2015
updated docs Makefile to reflect astropy Move the py.test conftest.py file to the astropy directory so it is installed. Changed the import of plugins accordingly. Also updated the .gitignore file so it ignores the Mac OS X .DS_Store file. fixed conf.py to not insert source dir into sys.path typo fixes in conf.py comment Added CSS file from Astropy core package Updated setup.cfg to match Astropy core package Copy bootstrap-astropy theme from astropy remove default.css import from astropy.css - now not needed update conf.py to use bootstrap-astropy theme, mirror astropy's version use installed astropy theme Added templates that inherit from core package Added `_templates` to list of patterns to exclude added comment directing to base of jinja template Update testing to only import pytest when tests are run Added `coverage` option that was added in the Astropy core Removed comments added kwarg options to test runner removed line setting unused _test_runner variable Updated setup.py to bring it in line with the Astropy core Update distribute_setup.py to 0.6.28 Remove unnecessary instructions from README.rst tweak to instructions from @astrofrog's suggestion Changed check in setup.py to syntax in core package to avoid issues when Sphinx is not installed. Update setup.py to reflect re-factored setup code for 0.2 release If necessary, move configuration file to the right place on first import Added open_files option for test() and implemented fix for caught exception in __init__.py Fix scripts/README.rst bug added commented-out edit_on_github section to conf.py and related instructions in README line wrap fixes Add .travis.yml file Add instructions for travis-ci to README _PACKAGE_SETUP_ should be _ASTROPY_SETUP_ (see astropy/astropy#950) Test with different Numpy versions, only test Sphinx with Python 2.7. Ensure that optional dependencies for Sphinx are installed Updated distribute_setup.py to fix compatibility with setuptools 0.7 README typo fixes Switch to using ez_setup.py, and simplify setup.py, following recent changes in Astropy core. Updates to __init__.py to match Astropy core Minor fix Added backward compatibility with Astropy 0.2 parallel option can only be used with Astropy 0.3 or later Re-organize the setup.py so that Astropy is not required for egg_info Revert "Re-organize the setup.py so that Astropy is not required for egg_info" This reverts commit 9aa5339. Replace distribute_setup.py with ez_setup.py and ``setuptools_bootstrap.py`` Support `setup.py test --coverage` in affiliated packages. Simplify package-template usage and updating by requiring fewer files to be edited. Add information about pulling changes to package template Store stuff in "metadata" section, not "affiliated" section. Insert space before comment. Fix documentation build Elaborate on the need to search for .c files in a comment. Change ``clean`` rule to remove generated docs Made necessary by astropy/astropy@b94fd38 Update to newer ez_setup.py Add the current version of ah_bootstrap--for the tests themselves we will always copy over the latest version of ah_bootstrap (so that we can test ah_bootstrap itself), but there should still be a default copy of it in the package-template repo as well Configure the package-template to use astropy_helpers. By default includes astropy_helpers as a submodule. Yes, astropy_helpers is now a submodule of package_template which is a submodule of astropy_helpers update the astropy_helpers version to include fix from c3f841b9e099f506a751b8a6971b15e7e3d6590a Update the astropy_helpers version used in the package_template Update the astropy_helpers version again; use the auto_use feature of ah_bootstrap. Switch to using astropy_helpers for docs build. Updated astropy_helpers version (included fix for Python 3) Update the git submodule path to use http instead--hopefully this will work then on Travis Updating to the actual astropy-helpers Updated astropy-helpers to latest version Fixing docs links in package test function Added example API docs Updated astropy-helpers to latest version. This version includes fixes for the coverage testing in affiliated packages. Use wheels for travis testing Care is taken to use a numpy wheel whose version matches the scipy/astropy wheels. Previous commit messages: Use development astropy for test --coverage Install coverage when needed Add cython to the travis build environment, controlled by an environmental variable Also added a few more words of advice to the travis part of the README Restore -w option to build_sphinx Change URL for numpy-specific wheels and make scipy an optional install Fix latest stable numpy version Several tweaks to improve performance and reduce unnecessary tests Remove debugging statements and cython install in egg_only Don't override package data after it's been determined from setup_package.py files Include astropy_helpers in the affilated package's tarball. Fix astropy#69: Affiliated packages can opt-in to deprecations-as-exceptions removed setuptools_bootstrap.py as it doesn't exist any more in the new package-template added ah_bootstrap and setup.cfg Update astropy_helpers Use MiniConda for affiliated packages Make sure conda’s astropy is removed before pip installing the latest developer version Added more comments to .travis.yml Pin Numpy version Don’t use --parallel with Python 3 More tidying up of .travis.yml file Move astropy dev builds earlier in .travis.yml Since they take longer, it is better to start them sooner for improved load balancing. Import astropy_helpers from inside the docs/ directory Fix a warning that "Could not determine version of package ccdproc" from configuration system (since the __version__ member doesn't yet exist for the package when the configuration file is copied). More detail in example __init__.py Add numpy import to exmaple_mod This ensures that egg_info builds will break if __init__.py is not set up properly Add coveralls --rcfile option Update README for coveralls rcfile Use the version module correctly in edit_on_github Closes astropy#55. Added files to ignore in affiliated package coverage tests Clarify purpose of cextern directory. Updated astropy-helpers to v0.4 Update ah_bootstrap.py from astropy-helpers Replace development build with stable now that astropy 0.4 is out Closes astropy#65 Add info about a specific Sphinx version check. explain readthedocs add released versions of astropy-helper and astropy to rtd-requirements Add a few entries to .gitignore Remove affiliated package instructions from README page update readme with a bit more info Typo fix pointed out by @cdeil Updated astropy-helpers and ah_bootstrap.py to v0.4.1 Added TEMPLATE_CHANGES.md. Fixes astropy#102. MANIFEST.in should exclude pyc *after* including astropy-helpers prune docs/api gitignore manifest template changes udpate Updated astropy-helpers to v0.4.3 Update TEMPLATE_CHANGES.md Fix typo Tagging v0.4.1 Updated astropy-helpers to patched v0.4.3 Update dependency information for open-files option Add note about adding/removing packages to/from pytest header Ensure that conftest.py change works with all versions of Astropy Make sure to update apt-get listings Add jinja2 in .travis.yml Drop python 3.2 testing Drop numpy 1.5 testing and use numpy 1.9 as baseline Use setuptools entry_points for command line scripts Update MANIFEST.in [skip ci] Update TEMPLATE_CHANGES [skip ci] add powered by astropy badge
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Both UKIDSS and FermiLAT return lists of matching FITS files, and I think in many cases IRSA does the same. So, it makes sense to abstract out that step.
This downloader simply gets the URLs (using
get_readable_fileobj) and opens them asfitsobjects, with an option of saving the file in an optional directory with a lot of optional file name arguments. The intent is to make it easy to create files of the form:UKIDSS_G043.17+0.01_H2_GPS_12312012.fitsor something along that line, rather than the cryptic form like:
L130413170713F15B52BC06_PH01.fits(but still make it possible to keep that cryptic form in the filename)