Tile-image compression (read + write) and rename to fitskit#1
Merged
Conversation
Implement Phase 1 of the FITS tiled-image compression convention (ZIMAGE/ZCMPTYPE/ZTILEn/COMPRESSED_DATA), decoding compressed images out of the BINTABLE form into ImageData: - New src/tile_compress.rs: CompressionType/Quantize/TileGeometry, CompressedImage<'a> view (detect/from_bintable), RICE_1 decoder, tile reassembly (unravel/scatter_tile), and GZIP_1/GZIP_2 behind an optional, feature-gated `gzip` dependency (miniz_oxide) so the core stays zero-dependency. - Lazy, backward-compatible API: Hdu::as_compressed_image() -> Option<CompressedImage>, then .decompress()/.compression()/.geometry(). HduData is unchanged; compressed images stay stored as BinTable so the original tiles survive for lossless round-trip. - Fix RICE_1 seed handling: the first value is a seed (lastpix), not a stored pixel; decode a full nvals diffs so array[0] = seed + diff[0]. The previous off-by-one shifted every reconstructed pixel one position. - Tests: byte-exact integration tests against fpack-generated fixtures (RICE row-tiled, square edge-truncated, I32, gzip1), skip-if-absent; corrected synthetic RICE unit tests that had masked the bug. - scripts/gen_compressed_fixtures.sh generates/verifies fpack fixtures; CI downloads them best-effort (cache key v2). - COMPRESSION_PLAN.md documents the staged rollout (float quant+dither, PLIO_1, HCOMPRESS_1, write path remain TODO). Also includes accompanying in-progress crate refinements across ascii_table/bintable/header/keyword/types and their tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3 of FITS tiled-image compression: decode floating-point compressed images (RICE/GZIP-quantized) back to F32/F64. - unquantize/decompress_float in tile_compress.rs: per-tile ZSCALE/ZZERO columns, ZQUANTIZ NO_DITHER / SUBTRACTIVE_DITHER_1 / SUBTRACTIVE_DITHER_2, ZBLANK -> NaN, DITHER_2 exact-zero preservation, and lossless-float (ZQUANTIZ=NONE) passthrough. - Port cfitsio's dither RNG exactly: Park-Miller LCG (a=16807, m=2^31-1, seed 1) -> 10000 f32s via OnceLock; iseed = (tile + ZDITHER0 - 1) % 10000. - Reconstruct with a fused multiply-add (f64::mul_add) to bit-match cfitsio/funpack; without the FMA, large-ZZERO cancellation pixels diverge by ~1 ULP. - Tests: 6 float cases asserting bit-identity to the funpack CLI at test time, skip-if-absent on both fixture and binary so CI stays green without cfitsio. Added a SUBTRACTIVE_DITHER_2 fixture (fpack -qz5). - CI: float fixtures added to best-effort download, cache key v2 -> v3, clippy/test switched to --all-features. HCOMPRESS_1, PLIO_1, and the write path remain TODO. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Complete the tile-compression read path with the remaining two algorithms; all image compression types in the standard now decode. - PLIO_1 (plio_decompress): port of cfitsio pl_l2pi line-list decode. The 1PI VLA is read as big-endian i16 words; opcodes 0..7 reconstruct the integer mask. Byte-exact (lossless). Note: the ll_src[3] header test is a signed i16 comparison (magic -100) — comparing unsigned decodes garbage. - HCOMPRESS_1 (hcompress_decompress): full port of fits_hdecompress / fits_hdecompress64 — quadtree bit-plane decode, undigitize, inverse H-transform, and unshuffle. 32-bit transform for ZBITPIX 8/16, 64-bit otherwise (avoids overflow); all arithmetic wrapping to match C. Lossless for integer scale=0; float feeds the existing unquantize + dither path. SMOOTH != 0 is rejected with a documented Err. - Tests: plio_roundtrip_euv and hcompress_int_roundtrip_euv (byte-exact vs original), float_hcompress_dither1_matches_funpack (bit-exact vs funpack), plus PLIO unit tests. Skip-if-absent as before. - Added a lossless integer HCOMPRESS fixture (fpack -h -s 0); CI cache key v3 -> v4, new fixtures added to best-effort download. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- README: list PLIO_1 and HCOMPRESS_1 as supported; drop the "growing" hedge; note SMOOTH!=0 as the only HCOMPRESS gap; clarify which algorithms work without the gzip feature. - CLAUDE.md: add tile_compress.rs to the module table; add tile compression, the FMA/funpack bit-exactness note, and the compressed fixture workflow to Design Decisions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Describe the public type surface (FitsFile/Hdu/HduData/Header/ImageData/ BinTable/AsciiTable/CompressedImage/Bitpix) and the HDU data model so the README is a complete description of the crate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fits4 can now WRITE tile-compressed FITS images — the capability no other pure-Rust crate offers. Output is read back byte-for-byte by cfitsio's funpack (and imcopy/astropy). - ImageData::compress(&CompressOptions) -> Hdu produces a compressed-image BINTABLE HDU (HduData::BinTable + ZIMAGE header + COMPRESSED_DATA VLA); caller does fits.push_extension(img.compress(&opts)?). Mirrors the read side's as_compressed_image()/decompress(). - RICE_1 encoder: exact inverse of the decoder — seed verbatim, per-block zig-zag diffs, FS selection minimizing block_n*(fs+1)+(sum>>fs); constants match the decoder (fsbits/fsmax/bbits 5/25/32, 4/14/16, 3/6/8). - Lossless: RICE_1/GZIP_1/GZIP_2 for integer (8/16/32), GZIP_1 for raw float. Lossy: RICE_1/GZIP on quantized + dithered floats. PLIO_1, HCOMPRESS_1, NOCOMPRESS, 64-bit int, and GZIP_2-lossless-float rejected with a clear Error. - Z* keyword order is load-bearing: funpack rebuilds the image header by walking cards, so ZTENSION must precede ZBITPIX/ZNAXIS (else "1st key not SIMPLE or XTENSION"). build_z_header emits fpack's order. Documented. - Tests: synthetic RICE encode->decode (i8/i16/i32), internal round-trips, and 6 funpack-reads-fits4 interop tests (byte-exact), skip-if-absent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a write-compressed-image example (ImageData::compress), list encode in features/supported, note RICE works in the zero-dep build, and add a "Compressed write" column to the comparison table — fits4 is the only pure-Rust crate that writes compressed FITS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename the crate from fits4 to fitskit throughout (Cargo.toml, source, tests, doc examples, README, CLAUDE.md, COMPRESSION_PLAN.md). The fits4_samples GCS bucket name is unchanged (external resource). Add publish metadata: repository, documentation, readme, keywords, categories, authors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add binary-table and tile-compression (compress -> decompress round-trip) runnable examples, a Core types overview, and note that the gzip feature now enables encoding as well as decoding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds full FITS tiled-image compression support and renames the crate to fitskit.
Compression — read (bit/byte-exact vs cfitsio
funpack)NO_DITHER/SUBTRACTIVE_DITHER_1/_2); float reconstruction uses an FMA to bit-match cfitsiohdu.as_compressed_image()?.decompress()?;HduDatastaysBinTableso compressed tiles survive for lossless round-tripCompression — write (the unique niche)
ImageData::compress(&CompressOptions) -> Hdu; output verified byte-exact throughfunpack(andimcopy/astropy)Validation
fpack-generated fixtures; 6 funpack-reads-fitskit interop tests; float bit-exactness vs funpack. Tests skip cleanly when fixtures /funpackare absent.scripts/gen_compressed_fixtures.shgenerates the fixtures (served from thefits4_samplesGCS bucket); CI downloads them best-effort.Rename
fits4->fitskitwith crates.io publish metadata (repository, keywords, categories, readme, ...). Thefits4_samplesbucket name is unchanged.Docs
lib.rscrate docs expanded to mirror the READMEAll tests pass (
cargo test --all-features) and clippy is clean (--all-features --all-targets -D warnings).