Conversation
Comprehensive analysis of techniques, tests, and theories that could strengthen the SAGA statistical test suite for Falcon/FN-DSA. Covers Renyi divergence estimation, side-channel-motivated independence tests, discrete Anderson-Darling, Henze-Zirkler MVN testing, multiple testing correction, and lattice-specific structural tests. Prioritized by security impact with full mathematical formulations and 23 references.
Bug fixes: - 0.1: chisquare() rounding correction applied to wrong variable; now corrects the exp list actually passed to scipy and accounts for outliers in n_effective (crashed on scipy >= 1.7) - 0.2: sampler.c used PDT table with RCDT comparison logic, giving P(0) ~ 0.64 instead of 0.36; replaced with actual RCDT values. Also fixed doubled loop increment (i=i+1 inside for) - 0.3: doornik_hansen() had a raise before rank-deficient recovery path, making recovery dead code - 0.4: removed sampler_rep.py import and test_rejind() (descoped per unified plan; code preserved in pqcrypto2020-as-published tag) - 0.5: test_basesampler() now accepts optional data argument, fixing arity mismatch with csv_testing() - 0.6: mardia() p_B now two-sided (kurtosis can deviate either way) - 0.7: BerExp scaling factor corrected from sigma/sigma0 to sigmin/sigma (matches HPRR20 algorithm and Falcon reference) - 0.8: sigmin lowered from 1.3 to 1.2778 to accept spec-legal values (Falcon spec sigma_min = 1.277833697 for n=512) Test infrastructure: - generate_test_vectors.py: 83 seeded test vectors across 3 tiers (good/bad/mediocre) covering 10 flaw types - run_baseline.py: scores current SAGA against vectors Baseline results after fixes: - Errors: 52/78 -> 0/78 - Bad detection: 10/42 (23.8%) -> 30/42 (71.4%) - Good false alarms: 0/6 (0%) - Remaining blind spots: Markov correlation (needs Ljung-Box, Phase 4) and tail truncation (needs AD/tail exceedance, Phase 3)
- pyproject.toml replacing requirements.txt (Python >=3.10, scipy >=1.11, numpy >=1.24, pandas >=2.0, pytest + mpmath as dev) - Pytest test suite: 82 parametrized tests from generated vectors (good/bad/mediocre), plus table lint via mpmath recomputation - GitHub Actions CI: matrix on Python 3.10/3.12, generates vectors on the fly, no large testdata download needed - Table lint (test_tables.py): recomputes half-Gaussian PDT/CDT at 400-bit precision with mpmath, verifies Python and C tables match — deterministic detector for table errors like the Phase 0 PDT/RCDT confusion - Structured output: to_dict()/to_json() on UnivariateSamples and MultivariateSamples for machine-readable results - Configurable params: tau, chi2_bucket, pmin are now constructor args with defaults, no longer module-level globals - Fixed doornik_hansen rank-deficient path (column projection + index reset so dot products align) - Cleaned up unused imports from test_rejind removal - .gitignore for __pycache__, test_vectors, .eps, .pytest_cache
Certification layer (certification.py):
- Computes R_a(P_spec || P_ideal) exactly from the RCDT table and
BerExp quantization, without sampling
- Base sampler: R_128-1 ~ 9.1e-23 (informational; CDT quantization)
- Full sampler: R_128-1 ~ 1.3e-29 at typical (mu,sigma), 6 orders
of magnitude below the 2^{-75} target — BerExp rejection tightens
the distribution dramatically
- Tested at hostile fractional mu (0.0, 0.25, 0.5) and across sigma
range; all pass with large margin
- Finding: sigma=1.8 (near sigma0=1.8205 boundary) marginally fails
at R_128-1 = 3.94e-23 vs 2.65e-23 target — genuine BerExp
quantization effect at extreme sigma values
Effect-size diagnostics (saga.py):
- effect_sizes() method on UnivariateSamples: TV distance with
null-expected bias floor, R_2 divergence (= log(1 + chi2/n)),
resolution floor 1/sqrt(N)
- Framed as diagnostics, not security certification
Seeded determinism (test_sampler_determinism.py):
- Same seed produces identical output stream
- Different seeds produce different streams
README scope paragraph:
- Statistical layer detects bugs at delta ~ 1/sqrt(N)
- Certification layer certifies proof-level RD bounds
- Neither detects trace-level side-channel leakage
Test suite: 104 passed, 12 skipped (known blind spots), 0 failed
New tests (code/univariate_tests.py):
- Tail exceedance: exact binomial tests at 3-6 sigma thresholds;
detects truncation that chi-square's bucket aggregation misses
- Sign/half-Gaussian factorization: (a) chi-square on |z| vs folded
PDT, (b) per-|z| sign balance binomial test, (c) joint (sign,|z|)
chi-square. Tests the z=(2b-1)*z0+b recombination step.
- Moment CIs: 95% confidence intervals on mean, stdev, skewness,
kurtosis with asymptotic standard errors
- Discrete Anderson-Darling: Choulakian-Lockhart-Stephens tail-
weighted EDF statistic, MC-calibrated (null depends on mu,sigma)
- Higher criticism: max standardized CDF deviation, near-optimal
against sparse alternatives (single wrong table entry), MC-calibrated
- MC calibration helper: mc_calibrate() draws B replicates from the
exact PDT, enabling correct p-values for any test statistic
Integration:
- UnivariateSamples.run_extended_battery() runs all new tests
- is_valid_extended = is_valid AND all extended tests pass
- to_dict() includes extended results when available
- run_baseline.py uses extended battery for scoring
Test vector update:
- Tail truncation changed from 6-sigma (undetectable at n=10000,
mass ~e^{-18}) to 3-sigma (detectable, ~30 expected samples)
Baseline results:
- Bad detection: 71.4% -> 85.7% (tail truncation 0/6 -> 6/6)
- Good false alarms: 0/6 (0%) -- unchanged
- Remaining blind spot: Markov correlation (6/6 MISSED, Phase 4)
Test suite: 116 passed, 6 skipped (Markov), 0 failed
New tests added to univariate_tests.py: - Ljung-Box (4.1): Q(h) = n(n+2) * sum r_k^2/(n-k) ~ chi2(h), tests serial autocorrelation up to lag 20. Catches Markov- coupled samplers with perfect marginals. - Wald-Wolfowitz runs test (4.2): tests randomness of above/below- median sequence. ~20 lines, marginal power beyond Ljung-Box but catches non-linear dependence. - Block homogeneity (4.3): chi-square homogeneity across 10 stream blocks. Catches mid-run drift or state corruption invisible to pooled distributional tests. All three integrated into run_extended_battery(). Baseline results: - Bad detection: 85.7% -> 100.0% (Markov 0/6 -> 6/6) - Good false alarms: 0/6 (0%) — unchanged - ZERO remaining blind spots across all 7 flaw types Test suite: 122 passed, 0 skipped, 0 failed
New tests (code/multivariate_tests.py): - Squared-norm test (5.1): ||x||^2/sigma^2 ~ chi2(dim) via KS. Catches radial flaws from coordinate correlation. - Fisher+BH meta-layer (5.2): Fisher combination of per-coordinate p-values for global verdict + BH for localization. Fixes the informal "X out of dim pass" count. - Max off-diagonal correlation (5.3): max|rho_ij| with Jiang's Gumbel limit. Catches single-pair correlation that diagcov's diagonal sums dilute. - FFT-domain battery (5.4): per-frequency variance, Re/Im independence, higher criticism across frequencies. ffSampling flaws are localized in the FFT basis. - Cross-key homogeneity (5.5): two-sample energy distance with permutation test. Tests the GPV key-independence property. - Two-sample tests (5.6): per-coordinate KS + CvM with BH correction for cross-implementation comparison. - Henze-Zirkler (5.7): MC-calibrated (not analytic log-normal, which is uncalibrated for p >= 128). Shares pairwise distance computation with energy test. Integration: - MultivariateSamples.run_multivariate_battery() runs all new tests - to_dict() includes extended results New test vectors: - mv_bad_norm_inflated: all coords share common factor (norm test) - mv_bad_cross_key: mixed sigmas simulating key-dependent output - mv_med_fft_weak: frequency at 70% variance (subtle FFT flaw) Test suite: 125 passed, 0 skipped, 0 failed
Calibration layer (code/calibration.py): - Flaw-injection library: 16 parametrized broken samplers covering sigma shift (1-10%), mu shift, sign coupling (52-60%), tail truncation (3-4 sigma), table error (2-3x), contamination (1-5%), Markov coupling (rho 0.1-0.3) - Power matrix: detection rate for each (test, flaw) pair over configurable replicates. Run with: python calibration.py --reps 20 - p-value uniformity meta-test: KS test of chi-square p-values against U[0,1] under H0. Detects miscalibration of the suite itself. Passes (KS p=0.16, mean_p=0.45). - Default battery selection: tests that win on at least one flaw type. Result: chi2, tail_exceedance, sign_halfgauss, discrete_ad, higher_crit, ljung_box, runs_test (7/8 tests earn a spot; block_homog wins nowhere at n=5000 but may at higher n). Key power matrix findings (n=5000, reps=5 smoke): - tail_exceedance: sole detector of 3-sigma truncation - ljung_box + runs_test: sole detectors of Markov coupling - higher_crit: best at mu shift (catches at mu+0.1) - discrete_ad: tied-best on sign bias and table errors - chi2: strong on all diffuse alternatives Test suite: 127 passed, 0 failed
Bug 1 (blocking): sampler.c paired an ascending CDT with the RCDT
"v < row" count, sampling 18 - z0 (mass piled at |z| ~ 18). Flip the
count to "v >= row" so it matches the Python reference sampler0().
Add tests/test_sampler_c.py plus a CI step that compiles, runs, and
chi2-tests the output against the folded PDT -- the table lint checked
the table but never the comparison logic.
Bug 2 (serious): berexp accepted with prob ~255/256 when the quantized
acceptance probability floored to 0 (p = -1, whose byte is 0xff).
Guard with `if p < 0: return False`, matching certification.py's model.
Bug 3 (moderate): reported R_a - 1 as exp(ra*(a-1)) - 1, i.e.
sum p^a/q^(a-1) - 1 ~ (a-1)(R_a - 1) -- a factor a-1 too large vs the
[Pre17] convention the 2^-75 requirement is stated in. Use exp(ra) - 1.
Nits:
- max_offdiag_correlation: MC-calibrate (the log(dim) Gumbel centering
miscalibrated the max over dim(dim-1)/2 dependent pairs).
- fft_domain_battery: test the summed variance statistic against
chi2(2(n-1)) instead of the averaged form against chi2(n-1).
- mc_pvalue: (1 + #{null >= obs}) / (B + 1) so finite B stays valid.
- calibration: reproducible per-flaw seeds (drop PYTHONHASHSEED-salted
hash()); enumerate index instead.
- doornik_hansen: project onto the nonzero eigenvectors on the singular
path instead of dropping arbitrary data columns by eigenvalue index.
- _energy_distance: cdist-based, no subsampling.
Also: pyproject declared license = MIT, but the repo ships under the
PQShield non-commercial research licence (see LICENCE). Point at the
file instead of asserting a wrong SPDX id.
The battery verdict was an AND of 7 (uni) / 5 (multi) per-test passes,
each thresholded at alpha -- inflating the family false-alarm rate to
~1-(1-alpha)^k (about 0.7% at alpha=1e-3, k=7). Replace it with one
family verdict:
all_pass = (Fisher-combined global p > alpha) AND no test rejects at
the Bonferroni threshold alpha/k
Fisher controls the family-wise rate against diffuse alternatives; the
Bonferroni gate keeps a single strong localized flaw from being diluted.
Individual results and per-test p-values are still reported
(report-first); global_pvalue / fisher_stat / bonferroni_rejects are
exposed for downstream use.
The SAGA v2 suite runs on generated, seeded vectors, not the original HPRR20 reference sample files. Drop the ~530MB testdata/ directory from the working tree and gitignore it. The data is preserved verbatim in the pqcrypto2020-as-published tag; test_falcon() documents how to restore it (git checkout pqcrypto2020-as-published -- code/testdata). Note: removes it going forward only; the blobs remain in history. A full history purge is a separate, destructive step left for later.
- report_card.py: ANSI terminal "report card" that runs the full battery on a sample set and prints a per-section pass/fail summary with component p-values, the global verdict, and per-test "check:" hints plus a failure-fingerprint match (nearest calibration flaw family). Supports --flaw for demoing injected defects. - visualize_mockups.py: matplotlib figures (power-matrix heatmap, detection frontiers, p-value calibration, FFT spectrum, Manhattan, ACF, Renyi profile, QQ norms) for the paper. Illustrative layouts; swap in real run data before publication. - extensive_calibration.json: results of the 7-param x 65-flaw x 8-test x 100-rep power-matrix sweep (n=10000). Generated figures are gitignored (regenerate from the scripts).
This was referenced Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Routes the SAGA v2 overhaul (authored by @jameshoweee) from his fork to upstream. This is the reviewed, approved state of jameshoweee/SAGA#8 — see that PR for the full review history (one request-changes round with three bugs found and fixed, then approval with empirical verification of the compiled sampler).
What this brings
σ_min/σ, spec σ_min, Doornik-Hansen rank-deficient path, two-sided Mardia.testdata/(preserved at tagpqcrypto2020-as-published).Supersedes #1 and #2 (Dependabot): the dependency overhaul here replaces
requirements.txtentirely, and the chi² fix is a prerequisite for any scipy ≥ 1.7 bump (the scipy bump alone crashes the suite).🤖 Generated with Claude Code