research: noise recovery notebooks with physical ground truth

[KedoKudo]

Summary

• Rebuilt noise recovery research notebooks (NB0 + NB1) from scratch using physical ground truth from sample metadata instead of the flawed NNLS-on-clean reference
• NB0 establishes ground truth, quantifies the NNLS normalization artifact (~63–218K ppm crosstalk), and characterizes the sparse data challenge (clipping, binning trade-offs)
• NB1 compares 8 recovery methods against physical truth: per-pixel NNLS, T-domain fitting, spatial denoising (NLM/Wavelet/Gaussian), NMF neighborhoods, and hierarchical subdivision
• All conclusions verified against executed output with deterministic seed

Key findings for March 4th presentation:

• Wavelet+NNLS is best at moderate noise (n=10): 181K ppm avg MAE
• Hierarchical bin=2 is the most robust method overall: 185K at n=10, 296K at n=2 (only method to beat raw NNLS at extreme noise)
• Blind NMF does not outperform informed NNLS — motivates semi-supervised approaches
• NNLS model bias floor (~100–335K ppm) limits all preprocessing methods

Test plan

• pixi run jupyter nbconvert --execute --to notebook examples/Notebooks/pleiades_noise_recovery_nb0_ground_truth.ipynb runs without error
• pixi run jupyter nbconvert --execute --to notebook examples/Notebooks/pleiades_noise_recovery_nb1_methods.ipynb runs without error
• Verify truth values: U-235=0.9974, Pu-241=0.4863
• Check _debug_images/NB0_*.png and NB1_*.png for reasonable plots

🤖 Generated with Claude Code

[KedoKudo]

Note: Poisson noise should be added to counts, not to transmission

The noise simulation in these notebooks adds Poisson noise directly to the transmission value T:

sigma = np.sqrt(T / n_incident)
T_noisy = T + noise  # or similar

This is physically incorrect. Transmission is not the Poisson random variable — raw counts are.

The correct simulation is:

counts = rng.poisson(n_incident * T_true)   # counts ~ Poisson(I₀ · T)
T_noisy = counts / n_incident               # normalize back to [0, 1]
sigma   = np.sqrt(np.maximum(counts, 1)) / n_incident  # propagated uncertainty

Why it matters:

• Adding noise to T can produce values outside [0, 1] at low I₀ (the "clipping catastrophe" you document in NB0). This is a symptom of the wrong approach, not an inherent limitation.
• Generating noise in count space then normalizing keeps T_noisy physical by construction, even at I₀ = 1.
• At deep resonance dips where T → 0, the correct formulation gives counts ≈ 0 (a few Poisson-distributed zeros), whereas the T-domain approach blows up the relative uncertainty.

The distinction also matters for fitting: the correct sigma from the count-domain simulation feeds directly into a Gaussian chi-squared fitter (valid when counts >> 1) or a Poisson NLL fitter (valid at all count levels).

For reference, NEREIDS uses the count-domain simulation for its noise sensitivity analysis in examples/notebooks/applications/01_spatial_mapping_demo.ipynb, and is adding a TRINIDI-style Poisson NLL fitter (fitter='poisson' in spatial_map()) for the sparse-counts regime.