Fpack compression stopped working?

[dstndstn]

Hi,

In fitsio 0.9.12, this code:

import fitsio
import numpy as np
img = np.zeros((1000,1000), np.float32)
fitsio.write('/tmp/test.fits.fz[compress R 100,100; qz -1e-4]', img, clobber=True)

produces a FITS file that is fpack compressed.

In fitsio 1.0.5, the same code results in a non-compressed file.

Also, clobber=True doesn't work in the latter case.

This is in python3, with fitsio installed on a fresh ubuntu 18.04 container using
pip3 install fitsio==0.9.12
or
pip3 install fitsio==1.0.5

cheers,
--dustin

[dstndstn]

1.0.2 shows the same behavior as 1.0.5

[dstndstn]

Same with 1.0.1

[dstndstn]

Down at DESI LegacySurvey.org headquarters, this is forcing us to revert back to 0.9.12

[esheldon]

I never even knew that this worked, putting compress information into the filename, it isn't something I ever tested.

[esheldon]

the clobber fails because it looks for that filename, including the bracketed information

[dstndstn]

The clobber is minor, but the fpack compression is a blocker. I assume it's just cfitsio doing its Extended Filename thing

…

On Tue, Dec 24, 2019 at 9:56 AM Erin Sheldon ***@***.***> wrote: the clobber fails because it looks for that filename, including the bracketed information — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#283?email_source=notifications&email_token=AAIEH7IQMNCF3SUHWIBODITQ2IPI5A5CNFSM4JZI3K5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHTJ66A#issuecomment-568762232>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIEH7OF4QOWH23VIHR75DLQ2IPI5ANCNFSM4JZI3K5A> .

[dstndstn]

(that used to allow the compression to work, that is)

…

On Tue, Dec 24, 2019 at 10:01 AM Dustin Lang ***@***.***> wrote: The clobber is minor, but the fpack compression is a blocker. I assume it's just cfitsio doing its Extended Filename thing On Tue, Dec 24, 2019 at 9:56 AM Erin Sheldon ***@***.***> wrote: > the clobber fails because it looks for that filename, including the > bracketed information > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#283?email_source=notifications&email_token=AAIEH7IQMNCF3SUHWIBODITQ2IPI5A5CNFSM4JZI3K5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHTJ66A#issuecomment-568762232>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAIEH7OF4QOWH23VIHR75DLQ2IPI5ANCNFSM4JZI3K5A> > . >

[beckermr]

Ugh. The fits standard has too many features. Supporting everything and making sure it doesn’t break is a pain for sure. :/

[esheldon]

In the short term you can keep 1.x (and proper python 3 strings etc) and instead add this information to a header that you write, and then run fpack yourself on it

[esheldon]

As for a solution, the right thing to do is probably to just try opening the file with cfitsio and looking for error conditions rather than doing filename processing in python

[dstndstn]

It's not really that simple, since we do more fancy stuff: we write the file to memory, compute a checksum, then write it to disk...

…

On Tue, Dec 24, 2019 at 10:06 AM Erin Sheldon ***@***.***> wrote: In the short term you can keep 1.x (and proper python 3 strings etc) and instead add this information to a header that you write, and then run fpack yourself on it — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#283?email_source=notifications&email_token=AAIEH7KE435FMISDR3V7JSDQ2IQOVA5CNFSM4JZI3K5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHTKLKY#issuecomment-568763819>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIEH7IXGJTL2CGHSDRVPUDQ2IQOVANCNFSM4JZI3K5A> .

[esheldon]

(above was solution for clobber)

a solution for the compression issue is probably something to do with python 3 strings and conversion to a c string in the C code

[beckermr]

There is an open issue about using this lib with an in memory file btw. If you have an example of how to do that, can you post it?

[esheldon]

the string that's being fed to cfitsio looks fine... I'm a bit baffled. Maybe there is a problem with the updated version of cfitsio we are using, or our hacks to cfitsio

[moustakas]

@esheldon @beckermr any additional thoughts on this issue? We're a couple short weeks away from launching DR9 (http://legacysurvey.org) and although 0.9.12 doesn't suffer from the fpack compression bug reported here, it has a number of other problems that were fixed in fitsio/1.0.x.

[beckermr]

I've got nothing here. Can you check cfitsio 3470 at the C level directly?

Assuming that works, then it is us. Otherwise, we will need to find / patch the bug or rollback the update.

[beckermr]

BTW, the version of cfitsio we bundle is here: https://github.com/esheldon/fitsio/tree/master/cfitsio3470

We apply patches to the build from here: https://github.com/esheldon/fitsio/tree/master/patches

So you might also check the actual version we use with patches at the C level as well.

[esheldon]

I've been unusually swamped, but I can try to look at this soon