Smokescreen-fork blinding: per-part-at-birth blind, hash-commitment custody, CAMB↔CCL cross-check (PR 6)

[cailmdaley]

Closes #252.

Blinds the tomographic data vector per part, at birth, through the UNIONS-WL/Smokescreen fork: each blindable intermediate SACC product — coarse ξ±, fine ξ±, pseudo-Cℓ — is shifted by t(hidden) − t(fiducial) (Muir et al. 2019) the moment the pipeline computes it, with the hidden cosmology drawn inside a fixed amplitude envelope by the fork's own CCL-native, per-key-independent draw. One seed per catalogue version drives every part's draw, so all parts share one hidden cosmology by construction. Only blinded parts persist on disk; derived statistics (COSEBIs, pure-E/B) are computed downstream from the already-blinded fine ξ± by the pipeline's own estimators, so they are born blinded — there is no re-derivation step inside the blinding code. No one can read S8 off the data until the collaboration unblinds; both the blind and the unblind are verifiable without a trusted keyholder.

Stacks on #245 (sacc_io); depends on the fork protocol (ConcealDataVector(fiducial_params, shifts_dict, sacc_data, *, seed, theory_fn), pinned fork SHA 588a6b9). No firecrown anywhere in this PR's diff — the fork is the concealment engine, and every theory vector is a direct CCL computation.

The three-verb surface

blind-init (once per catalogue version) — draws an OS-entropy seed, writes a repo-committable commitment.json (sha256(seed) + a canonical config digest binding the envelope and fiducial), and encrypts the seed into a Fernet bundle (smokescreen.encryption). The plaintext seed is never written. These outputs are the blind's fixed state; every other call reads them.

blind-part (per intermediate, at birth) — reads the encrypted seed (verifying both hashes against the commitment first), conceals one part SACC through its matching theory_fn backend under the shared seed, writes the blinded part with the part's true vector escrowed into a per-part encrypted bundle beside it, and deletes the plaintext part. Each escrow is self-contained: restoring a part needs only that part's bundle plus the seed bundle — corruption of one bundle loses one part, not all. Blinded metadata: concealed=True, blind label, blind_commitment, blind_config_digest; seed_smokescreen stripped.

unblind (per part, or the assembled file via grid-tag selection) — verifies both sha256(seed) and the config digest against commitment.json before subtracting anything, recomputes the part's shift from the seed through the same backend, and subtracts — the seed-subtracted vector is the authority. When the part's escrow bundle is beside it and its stored seed_sha256 matches the commitment, the escrow serves two subordinate roles: a tighter equality check (fail closed if it disagrees) and clearing the add-then-subtract ulp residue so the restore is bit-for-bit. An escrow bound to a different seed, or one that disagrees materially, never determines the output.

Terminal assembly — sacc_io.gather assembles the standalone part SACCs into the one-file {version}.sacc (tracer merge, ordered points with all tags including bandpower windows, block-diagonal covariance). Its single blind-aware touch is one call to blinding.assert_consistent_blind, which fails the gather closed unless every blindable part carries the identical blind_commitment (mixed blinded/plaintext assemblies also refuse), and stamps the shared commitment on the assembled file. ρ/τ diagnostic parts are exempt and pass through untouched.

What's in the box

sp_validation/blinding_theory.py — the blinding backend's theory surface. TheoryConfig (frozen dataclass; fail-fast overrides), the home of the blinding/inference fiducial (CosmoSIS v1.4.6.3 IA-fiducial values: S8=0.80, Ωm=0.30, h=0.70, n_s=0.96, Σm_ν=0.06 eV, HMCode2020+feedback with logT_AGN=7.5). Carries two halofit tokens (ccl_halofit_version, camb_halofit_version) naming the one recipe per stack; Neff=3.046/T_CMB=2.7255 are emitted explicitly to both stacks. Two independent ξ± paths live here: the CCL-native route (Boltzmann-CAMB P(k) → CCL Limber + FFTLog — the blinding recipe) and an independent pycamb P(k) → Pk2D → CCL projection with a closed-form σ8-matched A_s rescale. numpy-only at module scope.

No sp_validation/cosmology.py. develop deleted the local cosmology module (#223), moving cosmology to cs_util.cosmo, and this PR does not resurrect it. Per the UNIONS layering — cs_util = the cosmology library (generic machinery), sp_validation = configuration + survey-specific implementations — TheoryConfig (config) and the three part backends in blinding.py (coarse/fine ξ±, pseudo-Cℓ; survey-specific) are correctly homed in the blinding namespace. The generic pieces here (CCL ξ± path, CAMB P(k)→Pk2D path, σ8/A_s rescale) live in blinding_theory.py for now but carry an explicit module-docstring note that they are cosmology-library code destined for cs_util.cosmo (tracked in cs_util#80).

sp_validation/blinding.py —

• Envelope calibration: the intended (|ΔS8| ≤ 0.075, |ΔΩm| ≤ 0.1) box maps to {sigma8, Omega_c} half-widths at the fiducial (σ8 = S8/√(Ωm/0.3); Ω_c one-to-one). Half-widths are config, not code.
• Three theory_fn backends, one per part: coarse ξ±, fine ξ± (grid tags), and pseudo-Cℓ (W @ Cℓ_EE on the stored BandpowerWindows; ΔBB = ΔEB ≡ 0 — the shift is pure E-mode). Two-tracer cross-pairs; rows aligned to each part's own stored order, never an assumed pairing.
• Per-part conceal: extract the part's blindable block into a sub-SACC (exactly the rows the theory_fn spans — the fork's length guard enforces the agreement), drive ConcealDataVector, write the shifted values back at their recorded indices (row order preserved). Covariance (including the fine part's dense per-pair block) and BB/EB rows are byte-identical to the input.
• Custody: blind_init / blind_part / unblind_part as above, plus assert_consistent_blind for the gather. A part with no blindable block (ρ/τ diagnostics) refuses a blind call loudly.

sp_validation/sacc_io.py — gains gather (terminal part assembly, one blind-aware call site as above).

sp_validation/b_modes.py — gains the values-only estimator seams cosebis_from_xi and pure_eb_from_xi, exposing the same cosmo_numba kernels calculate_cosebis / calculate_pure_eb_correlation drive to callers holding ξ± arrays (e.g. a blinded fine-ξ± SACC part) rather than TreeCorr correlations. This is where born-blinded derived statistics come from — the estimators stay in the pipeline's namespace, not the blinding module's.

scripts/blind_data_vector.py — blind-init / blind-part / unblind / verify CLI; blind-init refuses to overwrite an existing blind's state before drawing anything.

Tests (all observed in the shared-venv run on candide)

test_blinding.py covers the per-part acceptance criteria AC1–AC9 (33 tests: 25 fast + 8 slow); test_camb_ccl_crosscheck.py covers AC10–14 unchanged. Heavy precision tests are @pytest.mark.slow; the derived-statistics tests importorskip("cosmo_numba"). Full suite on candide: 183 passed with only the four known environmental failures (config/catalog paths, workflow dry-run, pure-EB pinned reference — all outside blinding scope). CI runs the full suite inside the freshly built image, which carries cosmo_numba (proven by the blinding-stack import smoke test), so these ACs gate publication.

Observed numbers from the runs:

Quantity	Observed	Bound
AC2 on-file shift vs theory difference (all 3 parts, 2-bin; one hidden cosmology across parts asserted)	6.4e-14 rel	≤ 1e-10
AC3 realized shift vs independently written CCL reference	1.0e-20 abs (coarse ξ±)	< 1e-8
AC4 ΔBₙ (born-blinded COSEBIs) amplitude-independence	1.5e-10 of |ΔBₙ|	≤ 1e-9
AC4 leakage magnitude	ΔBₙ/Bₙ = 4.9e-6 (B=2e-5), 1.4e-5 (B=2e-6); Δξ⁺_B = 1.5e-9 = 0.08% of a 2e-6 B-mode	reported, not asserted
AC4 Δξ⁺_B amplitude-independence	8.7e-14 (5.8e-05 of the leakage)	≤ 5% of leakage
AC9 pure-E/B NaN pattern (born from blinded vs true parts)	identical (finite values shifted)	exact parity
AC10 nominal-A_s σ8 offset	3.4%	> 2% (the offset is real)
AC10 σ8-matched residual	1.1e-16	< 1e-4
AC11 ξ± floor at fiducial	ξ+ 0.24%, ξ− 0.10%	< 0.5% / 1.0%
AC12 ξ± floor off-fiducial (S8+0.075, Ωm−0.05)	ξ+ 0.28%, ξ− 0.12%	< 0.5% / 1.0%

The AC4/AC9 comparands run the pipeline's own downstream convention — measured coarse reporting ξ± + fine integration grid + edge-based bounds, through the b_modes seams — on blinded vs true parts, so they check the born-blinded derivation path against the pipeline convention rather than any blinding-internal seam. AC2 is fork-draw recovery + placement (both sides share the backend; backend correctness is AC3's independently written reference). AC1 (zero-shift identity, per part + downstream derivation), AC5 (covariance/row-order/BB-EB/ρτ untouched), AC6 (custody: init writes only commitment + encrypted bundle; plaintext parts deleted; gather fails closed on mismatched or mixed blinds; unblind fails closed on either tampered hash), AC7 (same seed ⇒ identical shifts), and AC8 (end-to-end init → three blind-parts → gather → per-part bit-for-bit unblind → true derived statistics reproduced) are asserted exactly.

Not in this PR

Snakemake wiring — which pipeline rule invokes blind-part, where blind-init fires per catalogue version — belongs to the Snakemake-parts row; the per-part surface here is designed for that wiring, but the DAG edges are authored there. Covariance blinding, unblinding-criteria policy, and the fiducial-cosmology group decision are likewise out of scope (config values, not code).

CI note

The blinding extra here pins the theory stack by git ref — the UNIONS-WL Smokescreen fork by commit (never ==1.5.6, which resolves to the incompatible upstream PyPI package that self-reports the same version), cosmo_numba by its numpy-2.x fork commit, and NumbaQuadpack at top level (it is a direct-URL transitive dep of cosmo_numba that pip/uv will not auto-install, so without it import cosmo_numba succeeds while the COSEBIs/pure-E/B kernels die at import NumbaQuadpack). The CI blinding-stack smoke test imports the fork-only draw_param_shifts and the actual kernel modules (cosmo_numba.B_modes.cosebis.COSEBIS, schneider2022.get_pure_EB_modes), so neither an upstream Smokescreen package nor a missing NumbaQuadpack can ship green. The numbers above were observed on candide (shared fork venv + cosmo_numba installed from Cail's fork), running the same suite CI runs in the image.

Fork-pin duplication is transitional. This branch's blinding extra pins the Smokescreen fork so the image is CI-self-sufficient before the deps PR (#243) lands. #243 carries the same fork as the core dependency (no [blinding] extra). The duplication resolves at whichever of the two merges second — the second merge's rebase onto develop drops the now-redundant extra pin. The PR base stays feat/sacc-2-sacc-io (a formal stack onto #243 was considered and declined: the branches have diverged architecturally — #243's base already moved cosmology to cs_util.cosmo, which this branch's blinding_theory.py layering now matches — so the rebase is a deliberate reconciliation for whichever merges second, not a mechanical restack now).

🤖 Generated with Claude Code

https://claude.ai/code/session_01KpaRHk3QwN13myduQ3hJyf