refactor: drop sip_tpv, rely on astropy for TPV WCS (closes #713)#716
refactor: drop sip_tpv, rely on astropy for TPV WCS (closes #713)#716martinkilbinger merged 3 commits intodevelopfrom
Conversation
Closes #713. The only functional use of sip_tpv in shapepipe was `split_exp.create_hdus` calling `stp.pv_to_sip(h)` to rewrite each per-CCD header's TPV distortion keywords as SIP before saving. Modern astropy (>=5) parses TPV natively via WCSLIB, as do SExtractor, PSFEx, and every other downstream WCS consumer in the pipeline, so the keyword-level conversion was no longer buying us anything. Removing the conversion lets us drop sip_tpv entirely — an unmaintained 2017 package whose v1.1 imports `pkg_resources` at module load and is incompatible with setuptools>=81 (the underlying cause of the ModuleNotFoundError in #713; PR #714's setuptools<81 pin is a symptom workaround). Changes: - split_exp: remove sip_tpv import + pv_to_sip call; drop the now-dead `transf_coord` plumbing through `process` / `create_hdus`; update module docstring accordingly. - Remove sip_tpv from `depends=[...]` in split_exp_runner, python_example_runner, and find_exposures_runner (the last was a spurious declaration — find_exposures_package never imported it). - Drop sip_tpv from environment.yml, environment-dev.yml, Dockerfile, install_shapepipe, docs/source/dependencies.md, and the orphan shupe:12 BibTeX entry. - New: `src/shapepipe/tests/test_split_exp.py` — three unittests that build a synthetic multi-CCD FITS with TPV distortion, run SplitExposures, and verify (1) per-CCD header WCS matches the source for several pixel samples, (2) the saved headers .npy round-trips pixel→world→pixel, (3) flag output uses int16. `scripts/jupyter/wcs.ipynb` still imports sip_tpv; it's an exploratory notebook last touched ~2.5 years ago and left alone.
Adds integration tests that verify the premise of the sip_tpv removal:
that TPV distortion headers survive the split_exp → SExtractor → PSFEx
chain without keyword-level conversion.
- SExtractorTPVTestCase: builds a synthetic 512×512 image with a
CFIS-scale TPV header (non-trivial radial term), injects 9 Gaussian
point sources, runs source-extractor, and asserts each detection's
ALPHA_J2000/DELTA_J2000 agrees with astropy's TPV pix→world mapping
to within 10 mas (~0.05 px at 0.187"/pix). Observed: max separation
~0.5 mas on SExtractor 2.25.0.
- PSFExTPVTestCase: runs the same kind of image through source-extractor
with FITS_LDAC output, asserts LDAC_IMHEAD preserves the PV keywords
verbatim, runs psfex on the LDAC, and asserts a PSF model with
ACCEPTED > 0 and CHI2 < 5. Also re-reads the LDAC-embedded header via
astropy and asserts the WCS matches the source header to <1e-9 px.
Gated on `shutil.which("source-extractor")` / `which("psfex")`;
skipped on machines without the external binaries.
|
Follow-up: empirical verification of the TPV-downstream premise. The previous state of this PR relied on an inference — "SExtractor and PSFEx link against WCSLIB, which handles TPV, therefore they handle TPV" — without actually running the external tools. Adding integration tests to actually verify it:
Both tests are gated on Combined with the earlier code audit showing every shapepipe module consumes WCS via |
#715 removed treecorr from environment.yml, environment-dev.yml, Dockerfile, and install_shapepipe, but left the dependency row in docs/source/dependencies.md and the now-orphan `jarvis:04` BibTeX entry in docs/source/refs.bib (no other references). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Piggybacked a small follow-up: removed the TreeCorr row from |
|
I ran the two tests and they passed. Not sure how strict these really are, but I guess we can merge this PR. This package seems to be before support in astropy was implemented. |
Commit 78c2668 deleted read_ext_cat_package/ and read_ext_cat_runner.py while renaming consumers to read_ext_sexcat_runner, but never added the renamed module. Likewise the commit moved Vizier retry logic out of create_star_cat.py and mask.py into shapepipe.utilities.vizier but that module was never created. This left the imports broken on v2.0: - mask.py:22 and create_star_cat.py:30 import shapepipe.utilities.vizier (ImportError on module load — mask is a core pipeline stage) - config_tile_Uc.ini, test_tile_det.py, and run_job_sp_canfar_v2.0.bash all reference read_ext_sexcat_runner (module-not-found at pipeline run) Changes: * shapepipe.utilities.vizier — new module exposing query_vizier(ra, dec, radius_arcmin, cat_id), consolidating the retry logic from mask.py and create_star_cat.py (server fallback, timeout escalation, random stagger, double-precision coord fix inherited from mask.py). * read_ext_sexcat_package/ + read_ext_sexcat_runner.py — restored from the deleted read_ext_cat_package with: - Renamed symbols (read_ext_sexcat, make_ldac_from_ascii, etc). - Tile-id overflow guard: the encoding tile_id = RRR*1000 + DDD silently collided for dec_idx >= 1000; extracted into _tile_id_from_file_number_string with an explicit bounds check. * pyproject.toml: dropped dead deps — treecorr and Stile (removed in CosmoStat#715), sip_tpv (removed in CosmoStat#716, merged today). These were stale holdovers on v2.0 that never got swept after the corresponding develop cleanups. * Dockerfile.jupyter: FROM points at ghcr.io/cosmostat/shapepipe:v2.0 instead of the martinkilbinger fork tag, so upstream builds work. Addresses the two new blockers flagged in PR CosmoStat#706 re-review: CosmoStat#706 (comment) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#716 (merged 2026-04-24) removed sip_tpv from shapepipe, which was the transitive source of split_exp.py's pkg_resources import chain. On the post-merge develop container, both split_exp entries now import cleanly: - shapepipe.modules.split_exp_package.split_exp - shapepipe.modules.split_exp_runner Also dropped the matching runner_metadata entry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
Removes the
sip_tpvdependency from shapepipe entirely. Modern astropy (≥5) parses TPV distortion natively via WCSLIB, as do SExtractor, PSFEx, and every other downstream WCS consumer in the pipeline, so the keyword-levelpv_to_sipconversion thatsplit_expused to run was no longer buying us anything.Dropping the conversion lets us drop
sip_tpvitself — an unmaintained 2017 package whose v1.1 importspkg_resourcesat module load and is therefore incompatible withsetuptools>=81. That's the actual root cause of #713 (rewritten to reflect this). PR #714'ssetuptools<81pin is a symptom workaround that becomes unnecessary oncesip_tpvis gone.What changed
Code
split_exp_package/split_exp.py: dropimport sip_tpv as stpand thestp.pv_to_sip(h)call; remove the now-deadtransf_coordparameter fromprocess/create_hdus; fix the module docstring.split_exp_runner.py,python_example_runner.py: removesip_tpvfromdepends=[…].find_exposures_runner.py: removesip_tpvfromdepends=[…]— spurious declaration,find_exposures_packagenever imported it.Environment
environment.yml,environment-dev.yml,Dockerfile,install_shapepipe: dropsip_tpv.docs/source/dependencies.md: drop thesip_tpvrow.docs/source/refs.bib: drop the orphanshupe:12entry.Tests (new)
src/shapepipe/tests/test_split_exp.py: three unittests that build a synthetic multi-CCD FITS with TPV distortion, runSplitExposures, and verify:headers*.npyside-output round-tripspixel → world → pixel.flagoutput files are stored asint16.All four modules in
src/shapepipe/tests/still pass locally (7/7).Not touched
scripts/jupyter/wcs.ipynbstill importssip_tpv— it's an exploratory notebook last edited ~2.5 years ago. Left alone to keep the diff scoped.Interaction with open PRs
setuptools<81as a workaround. Once this PR merges, that pin can come out. If fix: small develop bugs (#709, #711, #712, #713) #714 merges first the pin becomes dead code; happy to follow up with a cleanup commit in either direction.tests/unit/and includes an xfailed import test forshapepipe.modules.split_exp_package.split_exp. That test should xpass once either this PR or thesetuptools<81pin lands.Reviewer Checklist
developbranch