Can't update data in a fits table

[csnap001]

Hello, I asked this question in the wrong section previously. I'm trying to update a .fits file using astropy.io.fits. I do not have PyFits, I'm using astropy in its place.
for the following code the contents of Qname are ['003511.48-004918.0' '003657.17-100810.6' '005118.28+135448.1' '011845.55+133327.2']. However, this data was originally grabbed from a .fits file as follows

thdu = fits.open('dr7qso.fits')#raw information
tdata = thdu[1].data#gathering the data
name = tdata.field('SDSSJ')

name then rewritten into a numpy array and saved as Qname.
In addition I have the following import at the start of my code,

from astropy.io import fits

c1 = fits.Column('sdss_name', format = 'A18', array = Qname)
c2 = fits.Column('Plate', format = 'J', array = Pl)
c3 = fits.Column('Fiber', format = 'J', array = Fib)
c4 = fits.Column('MJD', format = 'J', array = date)
c5 = fits.Column('OIIrew', format = 'D', array = OIIrew)
c6 = fits.Column('OIIfwhm', format = 'D', array = OIIfwhm)
c7 = fits.Column('OIIIrew', format = 'D', array = OIIIrew)
c8 = fits.Column('OIIIfwhm', format = 'D', array = OIIIfwhm)
c9 = fits.Column('z_dr7', format = 'D', array = reds)
c10 = fits.Column('ContPow', format = 'D', array = contPower)
c11 = fits.Column('OIIPow', format = 'D', array = OIIpower)
c12 = fits.Column('r', format = 'D', array = rMag)
c13 = fits.Column('g-i', format = 'D', array = giMag)
c14 = fits.Column('r-z', format = 'D', array = rzMag)

OIIrew = np.array(OIIrew, dtype = np.float64)
OIIfwhm = np.array(OIIfwhm, dtype = np.float64)
OIIIrew = np.array(OIIIrew, dtype = np.float64)
OIIIfwhm = np.array(OIIIfwhm, dtype = np.float64)
Qname = np.array(Qname, dtype = np.object_)
reds = np.array(reds, dtype = np.float64)
contPower = np.array(contPower, dtype = np.float64)
OIIpower = np.array(OIIpower, dtype = np.float64)
rMag = np.array(rMag, dtype = np.float64)
giMag = np.array(giMag, dtype = np.float64)
rzMag = np.array(rzMag, dtype = np.float64)
Pl = np.array(Pl, dtype = np.string_)
Fib = np.array(Fib, dtype = np.string_)
date = np.array(date, dtype = np.string_)

if(not(path.exists('new.fits'))):
    newhdu = fits.new_table([c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,c14])
    newhdu.writeto('new.fits')
else:
    fits.open('new.fits')
    fits.update('new.fits', Qname, 'sdss_name')
    fits.update('new.fits', Pl, 'Plate')
    fits.update('new.fits', Fib, 'Fiber')
    fits.update('new.fits', date, 'MJD')
    fits.update('new.fits', OIIrew, 'OIIrew')
    fits.update('new.fits', OIIfwhm, 'OIIfwhm')
    fits.update('new.fits', OIIIrew, 'OIIIrew')
    fits.update('new.fits', OIIIfwhm, 'OIIIfwhm')
    fits.update('new.fits', reds, 'z_dr7')
    fits.update('new.fits', contPower, 'ContPow')
    fits.update('new.fits', OIIpower, 'OIIPow')
    fits.update('new.fits', rMag, 'r')
    fits.update('new.fits', giMag, 'g-i')
    fits.update('new.fits', rzMag, 'r-z')
    fits.flush()
    fits.close()

and here is the traceback

raceback (most recent call last):

File "", line 1, in 
runfile('C:/Users/owner/Documents/.spyder2/fittingTest.py', wdir='C:/Users/owner/Documents/.spyder2')

File "C:\Users\owner\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 699, in runfile
execfile(filename, namespace)

File "C:\Users\owner\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile
exec(compile(scripttext, filename, 'exec'), glob, loc)

File "C:/Users/owner/Documents/.spyder2/fittingTest.py", line 854, in 
fits.update('new.fits', Qname, 'sdss_name')

File "C:\Users\owner\Anaconda2\lib\site-packages\astropy\io\fits\convenience.py", line 525, in update
new_hdu = _makehdu(data, header)

File "C:\Users\owner\Anaconda2\lib\site-packages\astropy\io\fits\convenience.py", line 772, in _makehdu
hdu = ImageHDU(data)

File "C:\Users\owner\Anaconda2\lib\site-packages\astropy\io\fits\hdu\image.py", line 1011, in init
scale_back=scale_back)

File "C:\Users\owner\Anaconda2\lib\site-packages\astropy\io\fits\hdu\image.py", line 138, in init
self.data = data

File "C:\Users\owner\Anaconda2\lib\site-packages\astropy\utils\decorators.py", line 526, in set
ret = self.fset(obj, val)

File "C:\Users\owner\Anaconda2\lib\site-packages\astropy\io\fits\hdu\image.py", line 236, in data
self._bitpix = DTYPE2BITPIX[data.dtype.name]

KeyError: 'object'

Thanks, I really don't understand where the issue is coming from

[pllim]

c/c @SaraOgaz

[SaOgaz]

Hi again, @csnap001, are there missing indents in this code?

[csnap001]

Yes, I apologize. The if/else statements are all properly indented in the coding environment that I use. I simply copied and pasted the code and it didn't transfer the indents, I apologize again.

[MSeifert04]

@csnap001 for code blocks you can also use a gist to share the code and just link it here. Or use triple backticks (```) before and after the code block to format it as code.

Does it work if you omit the QName column?

[SaOgaz]

There's a couple things I'd like to point out here. First,io.fits.update is a convenience function, so you do not need to open the file first http://docs.astropy.org/en/stable/io/fits/api/files.html#update.

But you should probably not be using update, or any of the convenience functions when you're adding this many changes to the file. You might want to read over some of the examples in http://docs.astropy.org/en/stable/io/fits/.

[csnap001]

I tried running the code without the first five updates after opening the file and I still get an error.

Traceback (most recent call last):

  File "<ipython-input-5-9c667a8986f9>", line 1, in <module>
    runfile('C:/Users/owner/Documents/.spyder2/fittingTest.py', wdir='C:/Users/owner/Documents/.spyder2')

  File "C:\Users\owner\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 699, in runfile
    execfile(filename, namespace)

  File "C:\Users\owner\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)

  File "C:/Users/owner/Documents/.spyder2/fittingTest.py", line 895, in <module>
    fits.update('new.fits', OIIIrew, 'OIIIrew')

  File "C:\Users\owner\Anaconda2\lib\site-packages\astropy\io\fits\convenience.py", line 530, in update
    hdulist[_ext] = new_hdu

  File "C:\Users\owner\Anaconda2\lib\site-packages\astropy\io\fits\hdu\hdulist.py", line 213, in **setitem**
    _key = self.index_of(key)

  File "C:\Users\owner\Anaconda2\lib\site-packages\astropy\io\fits\hdu\hdulist.py", line 534, in index_of
    raise KeyError('Extension %s not found.' % repr(key))

KeyError: "Extension ('OIIIrew', 1) not found."

is there some other function that I can use to update fits files?

[MSeifert04]

The documentation has a good example of how to create a FITS Table: http://docs.astropy.org/en/stable/io/fits/#creating-a-new-table-file

Besides, I don't think fits.update should be used to update one column. I might be wrong but the documentation suggests that one can only update the complete extension.

[SaOgaz]

I posted my links wrong, apologies for that. @csnap001 please read over the documentation on astropy.io.fits here:
http://docs.astropy.org/en/stable/io/fits/

And if your curious about how to use update or any of the convenience functions in the future, you can find that page here: http://docs.astropy.org/en/stable/io/fits/api/files.html
The convenience functions are only meant to be used when you have a quick edit/data pull from a fits file. When doing multiple edits to a fits file (as it seems you're doing in your code), it's best to use the main astropy.io.fits functions.

[csnap001]

Okay, I've looked through the documentation and I have change the final else statement as follows:

else:
   test = fits.open('new.fits')
   files = test[1].data
   files['sdss_name'] = Qname

but now my Issue is different. I want to update my .fits file in a dynamic way. for instance, sometimes when I run my code Qname will be a list with 23 entries, then when I run the file again this may change to a higher or lower number. In this case the file needs to remove the extra entries if Qname is smaller than before, and it needs to add entries if Qname is larger than before. I believe that the append() function might be able to handle the case for when Qname is larger than before, but is there a function that will remove all items in the fits file that are larger than a certain index?

[SaOgaz]

Hi @csnap001, for this kind of question, you may have better luck asking on http://stackoverflow.com/ or another programming question website, as this is more of a general programming question and not an issue with the astropy library itself. The github issue pages are more geared towards bugs and or feature request.

[pllim]

@csnap001

Not sure if I understand... When you assign a column to store Qname, there is no limit on how many entries it can have. Your snippet should still work, as long as all your other columns also have the same number of entries and you remember to update all of them.

[saimn]

Closing as there is no issue in astropy.io.fits here (I think). Note that for FITS table you have also the more convenient Table interface: http://docs.astropy.org/en/latest/io/unified.html#fits