-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Minutes for AstroPy Meeting 2012
The following minutes were taken by Thomas Robitaille. Please report any issues/mistakes to thomas.robitaille@gmail.com.
The following people were present locally:
- Tom Aldcroft
- Azalee Bostroem
- Erik Bray
- Matt Davis
- Nadia Dencheva
- Michael Droettboom
- Justin Ely
- Henry Ferguson
- Perry Greenfield
- Tim Jenness
- Wolfgang Kerzendorf
- Pey Lian Lim
- Demitri Muna
- Adrian Price-Whelan
- Andy Ptak
- Thomas Robitaille
- Erik Tollerud
The following people were present remotely:
- Andy Casey
- Steve Crawford
These local (STScI) attended part of the meeting:
- Howard Bushouse
- Dave Grumm
- Phil Hodge
- Megan Sosey
The discussion on this day was focused on the current status of Astropy, and plans for the 0.2 release.
The meeting started off with an update of the current status of the Astropy package, which is summarized here.
We have come a long way since the previous coordination meeting. The current components of Astropy that are now implemented are:
- N-dimensional datasets (astropy.nddata)
- Units (astropy.units)
- Time and Dates (astropy.time)
- Data Tables (astropy.table)
- Cosmological Calculations (astropy.cosmology)
- FITS File handling (astropy.io.fits)
- ASCII Tables (astropy.io.ascii)
- VOTable XML handling (astropy.io.vo)
- World Coordinate System (astropy.wcs)
In addition, the astropy core package contains a lot of useful framework code for testing, logging, documentation, etc.
The code is hosted on GitHub, and we've found that this has been great for collaborating on the code. Pull requests provide a nice way to have discussions around code (which are often more productive than just mailing list discussions).
We also have a documentation website set up at http://docs.astropy.org (or http://devdocs.astropy.org for the latest developer version). The docs use Sphinx, and the built documentation is hosted on http://www.readthedocs.org, which provides automated documentation builds.
We use two continuous integration systems (systems that run the Astropy tests continuously).
- Jenkins, which includes builds for many python/numpy configurations, for Linux, MacOS X, and Windows. This includes staging branches that core developers can push to to ensure that tests run on all platforms/environments before merging into master
- Travis, which provides testing for a smaller subset of environments (only Linux), but will automatically test all pull requests, and determine whether these can be merged.
The two are complementary - Travis will catch the majority of issues very early, while Jenkins will catch more intricate platform-dependent issues.
-
astropy.nddata: base
NDDatacontainer class is there, with support for units, WCS, mask, errors, and meta-data, but functionality is limited. There is support for error propagation but the only error class so far is for standard deviations, and it does not deal with correlated errors, so this will need improving. We will also need to add functionality such as slicing (which deals correctly with the WCS). We also need to make sure that it is suited for non-numpy arrays that behave like -
astropy.units: this was adapted from the pnbody code, and implements
arbitrary unit conversion, has support for various 'vocabularies' (i.e.
syntax for interpreting unit strings such as 'm / s^2 / g'), and also has
support for converting units that aren't technically related (e.g. microns
and Hz) through equivalencies. We discussed minor changes to the name of the
equivalency functions (e.g.
sp->spectral). - astropy.cosmology: allows one to use existing cosmologies, as well as define new ones. Someone asked whether this was restricted to lambdaCDM cosmologies, and the answer is no - these cosmologies can be very general, and one could even implement a MOND cosmology if needed.
- astropy.io.fits: currently just a port of the latest pyfits. One of the main goals in future will be to integrate astropy.io.fits with other astropy components. For example, reading a FITS file could return astropy Table or NDData objects, and could make use of the Unit class. During the transition phase from pyfits to astropy.io.fits, pyfits could even include some of these changes by back-porting the astropy functionality. A couple of participants raised the idea that the various sub-packages in the astropy core package be developed in separate repositories and then included in the core package via git sub-modules, which would allow pyfits to also check out those sub-modules, but this was not thought to be a good solution, and the idea was dropped. A better idea might be to have a branch of Astropy that contains only the required sub-modules that is then used by PyFITS.
-
astropy.io.vo: currently only contains VO table I/O support, but this is
stable, with the caveat that most tables out there are non-standard
compliant. In that sense,
astropy.io.vois a great validation tool! We discussed that it would be useful to have easy-to-set-up VO server functionality to make it easier for people to serve their own data (Demitri in particular requested this). This could go in an affiliated package initially. - astropy.wcs: port of PyWCS, is basically a thin wrapper around the stable and well-tested Calabretta wcslib library. It does tend to me more strict than e.g. libwcs (because Calabretta writes the standard), so astropy.wcs has to implement some support for non-standard features, such as SIP distortions.
- astropy.time: a wrapper around the SOFA time library that supports very high precision time representation (good for pulsar people for example). The wrapper supports arrays. In future, we should improve the parsing of strings, and also find a good way to deal with deta_ut1_utc, leap seconds, etc. Adrian asked if it is easy to add a converter for datetime <-> astropy.time, and Tom A. pointed out it is easy to write a new handler.
-
astropy.table: generalized Table class, which supports slicing, sorting,
and can be initialized from a number of objects (lists, arrays, etc.). A lot
of room for future features, such as masking and merging/joining. There was
some interest in adding a handler for HDF5, and Tom can port this over from
ATpy. We will need to decide how to consistently make use of
meta, which can contain meta-data, for FITS, VO tables, etc.
We talked about knowing at what point to deprecate stand-alone packages. Mike D. said that the vo module could probably be deprecated soon, but pywcs is used in the JWST pipeline, so while we can encourage new tools and users to use astropy.wcs, we cannot stop distributing the stand-alone versions for a while. Erik B. suggested that astropy.io.fits could be recommended as the main package once astropy reaches critical mass (which was defined as ~equal number of downloads compared to pyfits). However, several people pointed out that there is a circular issue here, as users won't use astropy unless we recommend it, so we cannot reach critical mass without encouraging users to switch over.
We decided that once the first public release (0.2) is out, any package that can be replaced by astropy should include a note encouraging users to try out astropy, or pointing out that the same functionality is available in astropy. Since some people had concerns about being too strong in forcing users to switch, we made it clear that we would have to be careful with the wording. For example, for PyFITS, Erik B. agreed to adding a note saying something along the lines of 'If you already use Astropy, you do not need to install PyFITS', which is different from saying 'you should definitely switch to astropy.io.fits'. STScI will start noting in future releases of pyfits that it will migrate to astropy, and that all users of pyfits should make plans to do so eventually as at some point, pyfits as a standalone package will become unsupported.
A decision was made that after the first public release (0.2), all online Python tutorials for astronomers that we have access to (e.g. python4astronomers) should be updated to use astropy rather than the standalone package, which will boost uptake for beginners.
We discussed whether to keep the astropy.io intermediate subpackage, since all other sub-packages in astropy are at the top level (so we would move astropy.io.fits to astropy.fits for example). Erik T. pointed out it's not always clear what to do with packages that have some I/O but not completely. However, we ultimately decided that since the sub-packages inside astropy.io do provide a similar kind of functionality, it made sense to group them into another sub-package. Other packages that are currently sub-packages could end up becoming groups of sub-packages (e.g. astropy.wcs might have astropy.wcs.fitswcs, astropy.nddata might have sub-packages, etc.). However, we did decide that it made sense to move astropy.io.vo to astropy.io.votable and to add VO-querying functionality to astropy.vo (the former is only a file format, while the latter are actual online-querying tools).
We discussed the current status of astropy.coordinates, which we want to get in before the first public release (0.2). The code is still under development, and a pull request should appear shortly after the meeting for initial feedback. Some participants expressed concerns over whether we would have enough time for testing this new component before 0.2, since we wanted to release 0.2 soon. However, we eventually agreed that it should be good enough to release in 0.2, and we can make it clear to users that it is still new functionality, and therefore not as stable as other components. It will be good to have feedback from the community to improve this and other sub-packages.
As a side-discussion about astropy.coordinates, we agreed that it would be good to make it clear to users what functionality in astropy is mature, stable, in development, etc. Of course, we never want to release unstable code, but the reality is that some components might be less mature than others.
We agreed that it would make sense to have a release candidate for 0.2 since it will be an important release. The deadline for the release candidate is a few weeks from now, arbitrarily chosen to be US election day! (November 6th 2012). Note that this is for the release candidate, not the actual release (which should obviously be done at a better date where people will notice).
We discussed the possibility of binary releases. Tom R. agreed to look into how to adapt the Numpy/Scipy MacOS X DMG generation for Astropy. We also discussed coordinating the inclusion of 0.2 in package managers, which should be straightforward since there are several people on the Astropy team that have access to add packages in package managers.
We discussed what channels we should advertise the 0.2 release on. It was generally perceived that Twitter would be fine because the Twitter demographic is generally more computer savvy than the general astronomy community. It was felt that advertising on the Facebook Astronomers group would not be a good idea, as it would likely trigger a flame war on IDL vs Python...
The question was raised as to whether the Astropy project should get involved in the whole IDL vs Python debate, and whether we should advertise Astropy as an alternative to the IDL astrolib. However, most participants agreed that we should stay out of IDL vs Python issues until we are comfortable that most of the astrolib functionality is in astropy (which would hopefully be the case by 1.0).
Perry mentioned that Greg Wilson was asking again about publishing an article in Computers in Science and Engineering. The consensus was to wait until 0.2 was out before starting such an effort.
This lead to discussion of the issue of how to get credit for the work we've been doing on Astropy. Tom R. brought up the idea of publishing an Astropy paper in a refereed journal and proposed that we should aim for as high profile an astronomy journal as possible. It was pointed out that A&A and MNRAS both have code/software sections, and it was felt that this would be better than a specialized computational research journal. Tom R. relayed an idea that David Hogg (NYU) had previously mentioned, which would be to have one paper per major release, as it will properly give credit over time (i.e. if people join the project after the first paper, they can still get credit by being on subsequent papers, and people who stay on the project longer get on more publications). No objections were made, so Tom R. will coordinate the effort of getting a first publication out. We agreed the first author could be 'The Astropy project' or equivalent to avoid any issues with first authors/author order/etc.
We should also make sure that Astropy is indexed in the Astrophysics Source Code Library: http://asterisk.apod.com/viewforum.php?f=35 - which is indexed by ADS - though this should not replace the idea of a paper as it does not count as a 'refereed' publication.
- A few participants familiarized themselves with git/github and the Astropy workflow, and took a set of notes on anything that was unclear to them, so as to be able to clarify our instructions.
- Tom R. started triaging issues for 0.2, 0.3, etc.
- Mike D., Wolfgang K., and Demitri M. worked on finalizing the discussion about Units and integration with various sub-packages.
- Adrian and Matt discussed plotting in AstroPy. Details can be seen here: https://github.com/astropy/astropy-api/pull/3.
The discussion on this day was focused on future directions for Astropy and affiliated packages after the 0.2 release.
We discussed what the next priorities should be for the core package and affiliated packages. We felt that more efforts should be focused on the affiliated packages than have been until now.
Demitri asked for clarifications on whether the aim is to keep specutils separate from the core package, or whether it will eventually be merged into the core package - the answer is that once part or all of it is mature, it can be merged into the core package (and similarly for photometry).
Going back to the discussion of what the priorities should be for development, we agreed that ultimately, it would be nice to have 'killer' features that can attract users. The following priorities were identified:
- survey querying inside the core astropy package. This was originally suggested by Demitri, who has developed an API to access the latest SDSS data release. This component of the core package would be separate from, but might use, the VO cone search capabilities that are also planned. The difference is that the 'surveys' sub-package would just provide an interface to high-quality survey data for specific surveys, while the VO cone search gives access to any VO resources, which is not quality controlled. Tom R. will coordinate this, with help from Demitri.
- photometry utilities, in particular a DAOPhot replacement, since this is something many people use. We could make use of all the existing DAOPhot documentation which describes in detail how the algorithms work. Our implementation could be tested against DAOPhot. More generally, we should work on both aperture and PSF photometry. The aperture photometry is already under development by Rene Breton, Kyle Barbary, and Tom R. but no-one present at the meeting volunteered to coordinate the PSF-fitting capabilities. We also discussed the possibility of having a ccdproc equivalent in Astropy, and defining classes of fluxes/magnitudes/bandpasses/etc. and conversions between these.
- plotting capabilities for objects in the Astropy core package (e.g. NDData and Table). It would be worthwhile designing an API to be able to easily visualize these objects (e.g. via a plot() method). This would ultimately include APLpy-style functionality of plotting the WCS correctly for any of the data types. Adrian will coordinate this.
In addition, we agreed that it was important to think about how to generalize the WCS code architecture.
We discussed how to recruit more existing third-party packages as affiliated packages, and what we could offer them. The definition of affiliated packages is currently vague, and could range from any package that 'joins' the Astropy project in spirit to packages that conform to specific documentation, testing, and layout standards, though we agreed that we should be broad in what we accept. At this time, the number of affiliated packages remains small, and this is still something that we will have to assess as time goes on.
An example of a project we might want to talk with is the yt project, which provides sophisticated visualization for simulation data. We should find out whether they rely on any dependencies that could be replaced by Astropy. Adrian will talk with the yt developers to see if there is any interest.
We also discussed the idea of having a 'showcase' page of affiliated packages, including instructions or link to instructions for installing each package. This would provide visibility to these packages, which could be the incentive to bring some on board. The list should only include packages that are suitable for user consumption.
-
Pey Lian Lim, Tom R, and Demitri M. discussed how the survey querying will work, and a preliminary API. Demitri will create an API proposal document, and Tom R. will start looking into some of the coding aspects.
-
"Generalized" WCS: the WCS classes will be re-organized as soon as possible to avoid breaking backward compatibility in future. The basic API for (to-be-renamed) WCS objects should be:
wcs = WCS() out = wcs(in) # data to world transformation in = wcs.invert(out) # optional world to data
There was a lot of discussion on the best way to organize such a class and connect it to the rest of astropy. Wolfgang K. (together with Nadia D.) agreed to write a base class formalizing the first parts of this class design. Nadia D. has a system that has implemented most of the WCS transforms already. Hence, she agreed to integrate this into astropy once the base class is done, starting with a basic scale-and-offset class for use as a first experimental realization. Part of the existing software developed by Nadia is a set of model/fitting classes that decouple model and fitting engine from each other making it easier to add new models or fitting engines without having to make modifications to the other. It was agreed that this sort of facility is of general use beyond WCS and should be separated out for astropy (discussed further below)
Much of the discussion centered around contrasting this approach with that used by AST (Tim J. being the expert on AST). In the end, it it was decided to take an approach very similar to AST - although simplified in the first instance (e.g. input coordinate system always refers to pixels, unitless transforms). Some of these features might appear after a first draft of the generalized WCS is done. Fitting of WCS-systems is (as in the AST package) not part of the actual WCS-core but will be done via the Model-framework developed by Nadia D.
-
Erik B. worked on triaging of issues and updating the changelog for the 0.2 release.
-
<please add details of your breakout session here>
In responding to question about matplotlib having interactive manipulation of plots in the browser, Mike D. stated that it appeared very unlikely given the technical challenges. Then the next day he produced a working example!
Tom A. and Tom R. implemented support for masking in Table objects. The code is mostly functional, although small additions need be done, and tests and documentation still need to be written, but this will be ready for a pull request reasonably soon now that the hard work is done.
Erik T. and Nadia D. worked on separating out the model-fitting framework used in Nadia's WCS work, with the intent that it be eventually included as an astropy.models subpackage.
The specutils-package is currently an affiliated package in which utilities for spectra are being developed. It was agreed upon that some of the mature code will eventually make it into the astropy core. In the specutils breakout session we (Astropy developers Erik T., Tim, Wolfgang K. joined by the COS+STIS Team Azalee, Justin, Phil Hodge and Charles Proffitt) mainly discussed the implementation of 1D spectrum objects. In the first instance, it was realized that we also need a framework to store analytical spectrum like object (in the simplest case a Blackbody). This can be done with a subclass of the Model-framework being developed by Nadia D.
The next item that was discussed, is the API for a Spectrum1D class. It seems to be a useful idea to first develop much more specialized classes (e.g. COSSpectrum1D) and then see what these have in common to come up with a Spectrum1D class that these can inherit from. We also discussed how metadata in array form is often needed (apart from any uncertainty classes). One example are exposure time arrays (needed when adding spectra that do not overlap on the entire wavelength) that need to do arithmetic in a sensible way. There seem to be two approaches: Using the wcs class to store more information in the NDData.data array or storing some of this information in a flags-class. Both of these approaches might be important for different problems. We ended the discussion by agreeing that the COS+STIS team would make a quick example API for what they need and go from there.
Add details of your sprint here!
- Do a 3-day meeting with a sprint day in the middle so that there can be more actual sprinting!