Transparently support uint16 and uint32 in astropy.io.fits (for tables)

[astrofrog]

It would be great if PyFITS could transparently support unsigned 16- and 32-bit integers in Tables. Even though the FITS standard does not directly support this, a 16-bit unsigned integer can be written as a 16-bit signed integer with BSCALE=1 and BZERO= 32768. Similarly, a 32-bit unsigned integer can be written as a 32-bit signed integer with BSCALE=1 and BZERO= 2147483648. This could work for both input and output - i.e. if a 16-bit unsigned integer is written, then BZERO and BSCALE are set appropriately, and if BSCALE=1 and BZERO=32768 and the column is a 16-bit signed integer, it is converted to a 16-bit unsigned integer upon reading.

For info, cfitsio already does this: http://heasarc.gsfc.nasa.gov/fitsio/c/c_user/node23.html

[Note: This functionality already works in PyFITS using uint=True when opening the file. However it only works for images, not tables.]

[mdboom]

Doesn't pyfits/astropy.io.fits already do this? It definitely did years ago when I worked on it.

[astrofrog]

I forgot to mention that I meant in tables:

In [1]: import numpy as np

In [2]: from astropy.io import fits

In [3]: x = np.array(zip([1,2,3]), dtype=[('x', np.uint16)])

In [4]: fits.writeto('test_uint16.fits', x, clobber=True)
ERROR: ValueError: Illegal format uint16. [astropy.io.fits.column]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/Users/tom/<ipython-input-4-3963fca66b62> in <module>()
----> 1 fits.writeto('test_uint16.fits', x, clobber=True)

/Users/tom/Library/Python/2.7/lib/python/site-packages/astropy-0.2.dev1595-py2.7-macosx-10.6-x86_64.egg/astropy/io/fits/convenience.pyc in writeto(filename, data, header, output_verify, clobber, checksum)
    400     """
    401 
--> 402     hdu = _makehdu(data, header)
    403     if hdu.is_image and not isinstance(hdu, PrimaryHDU):
    404         hdu = PrimaryHDU(data, header=header)

/Users/tom/Library/Python/2.7/lib/python/site-packages/astropy-0.2.dev1595-py2.7-macosx-10.6-x86_64.egg/astropy/io/fits/convenience.pyc in _makehdu(data, header)
    760         if ((isinstance(data, np.ndarray) and data.dtype.fields is not None)
    761             or isinstance(data, np.recarray)):
--> 762             hdu = BinTableHDU(data)
    763         elif isinstance(data, np.ndarray):
    764             hdu = ImageHDU(data)

/Users/tom/Library/Python/2.7/lib/python/site-packages/astropy-0.2.dev1595-py2.7-macosx-10.6-x86_64.egg/astropy/io/fits/hdu/table.pyc in __init__(self, data, header, name)
    162                     self.data = data
    163                 else:
--> 164                     self.data = data.view(self._data_type)
    165 
    166                 self._header['NAXIS1'] = self.data.itemsize

/Users/tom/Library/Python/2.7/lib/python/site-packages/astropy-0.2.dev1595-py2.7-macosx-10.6-x86_64.egg/astropy/io/fits/fitsrec.pyc in __array_finalize__(self, obj)
    227 
    228             if self._coldefs is None:
--> 229                 self._coldefs = ColDefs(self)
    230             self.formats = self._coldefs.formats
    231 

/Users/tom/Library/Python/2.7/lib/python/site-packages/astropy-0.2.dev1595-py2.7-macosx-10.6-x86_64.egg/astropy/io/fits/column.pyc in __init__(self, input, tbtype)
    441                     format = 'A' + str(ftype.itemsize)
    442                 else:
--> 443                     format = _convert_format(ftype, reverse=True)
    444                 # Determine the appropriate dimensions for items in the column
    445                 # (typically just 1D)

/Users/tom/Library/Python/2.7/lib/python/site-packages/astropy-0.2.dev1595-py2.7-macosx-10.6-x86_64.egg/astropy/io/fits/column.pyc in _convert_format(format, reverse)
   1168 
   1169     if reverse:
-> 1170         return _convert_record2fits(format)
   1171     else:
   1172         return _convert_fits2record(format)

/Users/tom/Library/Python/2.7/lib/python/site-packages/astropy-0.2.dev1595-py2.7-macosx-10.6-x86_64.egg/astropy/io/fits/column.pyc in _convert_record2fits(format)
   1156         output_format = repeat + NUMPY2FITS[dtype + option]
   1157     else:
-> 1158         raise ValueError('Illegal format %s.' % format)
   1159 
   1160     return output_format

ValueError: Illegal format uint16.

Things work fine for Image HDUs, so it'd be great for table columns to operate in the same way.

[embray]

Yeah, this has been on my to-do list for a while. I don't think it's too hard a change either so I might try doing it sooner rather than later. Though where pyfits is concerned I really want to focus at this point on replacing the table interface in its entirety.

[embray]

See also the related ticket for pyfits: https://trac.assembla.com/pyfits/ticket/192

[weaverba137]

I am preparing a pull request that addresses this issue.