Cannot open compressed file

[mileslucas]

I cannot open a zipped FITS file (astropy has no problem, though).

julia> using FITSIO, PyCall

julia> download("https://www.spacetelescope.org/static/projects/fits_liberator/datasets/eagle/502nmos.zip", "502nmos.zip")
"502nmos.zip"

julia> py"""import astropy
       print(astropy.__version__)
       """
3.2.3

julia> pf = pyimport("astropy.io.fits")
PyObject <module 'astropy.io.fits' from '/Users/miles/.pyenv/versions/3.7.4/Python.framework/Versions/3.7/lib/python3.7/site-packages/astropy/io/fits/__init__.py'>

julia> pf.open("502nmos.zip")
2-element Array{PyObject,1}:
 PyObject <astropy.io.fits.hdu.image.PrimaryHDU object at 0x1469b0390>
 PyObject <astropy.io.fits.hdu.table.TableHDU object at 0x1469ad0d0>

julia> FITS("502nmos.zip")
ERROR: While processing file `502nmos.zip`: error uncompressing image
Stacktrace:
 [1] error(::String) at /Applications/Julia-1.4.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 [2] fits_assert_ok at /Users/miles/.julia/packages/FITSIO/BavHZ/src/libcfitsio.jl:206 [inlined]
 [3] fits_open_file(::String, ::Int64) at /Users/miles/.julia/packages/FITSIO/BavHZ/src/libcfitsio.jl:281
 [4] FITS(::String, ::String) at /Users/miles/.julia/packages/FITSIO/BavHZ/src/FITSIO.jl:109 (repeats 2 times)
 [5] top-level scope at REPL[33]:1
 [6] eval(::Module, ::Any) at /Applications/Julia-1.4.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 [7] eval_user_input(::Any, ::REPL.REPLBackend) at /Applications/Julia-1.4.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 [8] run_backend(::REPL.REPLBackend) at /Users/miles/.julia/packages/Revise/AMRie/src/Revise.jl:1023
 [9] top-level scope at none:0

relevant manifest info:

Details

  [525bcba6] FITSIO v0.14.0
  [438e738f] PyCall v1.91.4

[mileslucas]

I've checked using the fitsio python package (which also wraps libcfitsio)-

>>> import fitsio
>>> f = fitsio.FITS("502nmos.zip")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/fitsio/fitslib.py", line 477, in __init__
    self._FITS = _fitsio_wrap.FITS(filename, self.intmode, create)
OSError: FITSIO status = 414: error uncompressing image
failed to uncompress file into memory (compress_open)
failed to find or open the following file: (ffopen)
502nmos.zip

I'll keep digging a little- this seems like a trivial operation that's failing

[giordano]

We can access files compressed with the algorithms supported by cfitsio, which doesn't include zip: https://heasarc.gsfc.nasa.gov/docs/software/fitsio/compression.html. See #125, where a user is able to open compressed files (but they have a problem because of a cfitsio bug).

My guess is that Astropy high-level interface automatically uncompresses the files, even those not supported internally by cfitsio

[giordano]

My guess is that Astropy high-level interface automatically uncompresses the files, even those not supported internally by cfitsio

Oh well: https://github.com/astropy/astropy/tree/a45af44fdd406cd7b53f3e7a48ff4d22dca3b67e/astropy/io/fits/src

[mileslucas]

Would it be worth trying to parse the file for zips and decompress ourselves?

[giordano]

We could use CodecZlib.jl for decompressing in memory the files compressed with the algorithms not supported by cfitsio. The annoying part is that we should handle also writing back to disk.