jc: D-KIA-C1b — additive statistics battery (κ, ω, R/η², t-tests, φ) - #887
Conversation
…t-tests, phi) New module crates/jc/src/stats.rs. Closes the gap the C1 audit found: kappa was genuinely absent and blocks D3's fusion falsifier, while phi and KR-20 turned out to be computations jc already shipped under other names. Shipped: - cohen_kappa — chance-corrected agreement. A DIFFERENT estimator from ICC, not a rename: ICC decomposes variance for interval ratings, kappa corrects counts for marginal-expected agreement. This unblocks D3. - omega_total — McDonald's omega_t via Spearman's triad identity for the congeneric loadings. Where alpha assumes tau-equivalence, omega does not, so omega >= alpha whenever loadings differ. - phi — named wrapper DELEGATING to reliability::pearson on a 0/1 coding; takes &[bool] so the binary precondition is unforgeable. The marginal-capped ceiling caveat is documented at the function. - multiple_r / multiple_r_squared — OLS with intercept via normal equations and Gaussian elimination with partial pivoting; singular design returns None. - eta_squared + anova_one_way — explained variance and its F test. - t_test_one_sample / _paired / _welch / _student — each returning t, df and p. Every p-value comes from ONE shared regularised-incomplete-beta core (Lanczos ln-gamma + modified-Lentz continued fraction), pinned against textbook critical values rather than against this code's own output. The effect-size family is r (phi, R, R-squared, eta-squared). Cohen's d is out by construction; the t-tests are the significance companion to eta-squared/ R-squared, not a d-family back door. ADDITIVE CONSTRAINT HELD, TIGHTER THAN PERMITTED. The whole diff to existing files is `fn` -> `pub(crate) fn` on mean and all_finite, plus doc comments and one `pub mod` line. average_ranks and pop_var were NOT widened — the carve-out allowed it, but this module does not consume them; and sample_var/sample_cov here use the unbiased n-1 divisor, a different estimator from pop_var's n, not a duplicate. No existing statistic's arithmetic, signature or semantics moved. Validation is by CROSS-IDENTITY against independently-computed quantities, not self-assertion: phi vs pearson; R-squared vs pearson-squared at one predictor; eta-squared vs R-squared on a 0/1 dummy; eta-squared vs t^2/(t^2+df) and F=t^2 from the pooled t; omega vs alpha (equal under tau-equivalence, and 0.9473684 > 0.8684211 on a hand-computed congeneric fixture). Two defects found by the doc examples, both fixed (see EPIPHANIES E-EXACT-FIT-IS-WHERE-ABSOLUTE-ZERO-GUARDS-BREAK-1): 1. Real bug — the Heywood guard `psi < 0.0` rejected zero-residual models, because psi = 0 exactly in real arithmetic lands a few ulps either side of zero in f64. Now a variance-relative tolerance, PAIRED with a can-it-fire test proving the guard still bites on a real violation (relaxing a guard without that test is the strictly worse defect). 2. Wrong example — predictors `a` and `a+1` are collinear WITH THE INTERCEPT, so None was correct. A regression test now pins that case. 107 lib tests + 11 doctests green; stats.rs clippy-clean. The crate is pre-existing fmt-dirty and clippy-noisy, so only the new file was formatted — reformatting the rest would be the "while I was in there" cleanup the additive constraint forbids. Board hygiene in the same commit: STATUS_BOARD D-KIA-C1b -> In PR, plan C1b shipped-note, EPIPHANIES entry, PR_ARC + LATEST_STATE entries. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K3RyLEbuNSHxxB3NTTrGki
📝 WalkthroughWalkthroughThe PR adds a public Changesjc statistics extension
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_bc7afd8b-1220-4957-a085-6e0e8914b7da) |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f394d091a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if lam_sq < -tol || !lam_sq.is_finite() { | ||
| return None; // single-factor model violated (negative common variance) | ||
| } | ||
| lambda[i] = lam_sq.max(0.0).sqrt(); |
There was a problem hiding this comment.
When a scale contains a negatively loading item, taking sqrt here makes every loading positive and silently inflates ω because the numerator must use the signed sum of loadings. For example, three orthogonal-error items with loadings [1, -1, 1] and residual variances 0.25 produce approximately 0.923 here instead of the correct 0.571; infer each loading's sign from its covariance with a reference item before summing.
Useful? React with 👍 / 👎.
| if max.abs() < 1e-12 { | ||
| return None; // singular → collinear predictors |
There was a problem hiding this comment.
When an otherwise valid predictor is expressed in small units, this absolute pivot cutoff classifies the design as singular, making R² depend on the predictor's units. For example, y = [1,2,3,4,5] with x = [1e-7,2e-7,3e-7,4e-7,5e-7] is an exact one-predictor fit, but the centered pivot is about 1e-13, so the function returns None; use a matrix-relative tolerance or a scale-aware decomposition.
Useful? React with 👍 / 👎.
| let ss_w = ss_t - ss_b; | ||
| if ss_w <= 0.0 || !ss_w.is_finite() { |
There was a problem hiding this comment.
When between-group separation is much larger than nonzero within-group variation, subtracting these nearly equal large sums loses the within variance and incorrectly returns None. For groups [0,1,2] and [1e9,1e9+1,1e9+2], the actual within-group sum of squares is 4, but both ss_t and ss_b round to 1.5e18, making this subtraction zero; accumulate deviations from each group mean directly instead.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
crates/jc/src/stats.rs (2)
274-284: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueCount marginals in one pass to avoid quadratic cost.
The current loop scans
aandbonce per category. Labels are arbitraryusize, so the number of distinct categories can reach2n. Then the cost isO(n²). A single counting pass over each slice givesO(n log n).♻️ Proposed single-pass marginal counting
- let n = a.len() as f64; - let cats: BTreeSet<usize> = a.iter().chain(b.iter()).copied().collect(); - - let agree = a.iter().zip(b.iter()).filter(|(x, y)| x == y).count() as f64; - let p_o = agree / n; - - let mut p_e = 0.0; - for c in &cats { - let ma = a.iter().filter(|&v| v == c).count() as f64 / n; - let mb = b.iter().filter(|&v| v == c).count() as f64 / n; - p_e += ma * mb; - } + let n = a.len() as f64; + + let agree = a.iter().zip(b.iter()).filter(|(x, y)| x == y).count() as f64; + let p_o = agree / n; + + let mut count_a: BTreeMap<usize, usize> = BTreeMap::new(); + let mut count_b: BTreeMap<usize, usize> = BTreeMap::new(); + for (&x, &y) in a.iter().zip(b.iter()) { + *count_a.entry(x).or_default() += 1; + *count_b.entry(y).or_default() += 1; + } + let mut p_e = 0.0; + for (c, &ca) in &count_a { + if let Some(&cb) = count_b.get(c) { + p_e += (ca as f64 / n) * (cb as f64 / n); + } + }Then replace the
BTreeSetimport withBTreeMapat Line 77.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/jc/src/stats.rs` around lines 274 - 284, Update the agreement calculation around cats, p_e, and the category loop to build marginal counts for a and b in single passes using BTreeMap, then compute p_e from those counts without rescanning either slice per category. Replace the BTreeSet import with BTreeMap and preserve the existing category probabilities and result behavior.
462-464: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winScale the singularity threshold, and document it as hand-tuned.
1e-12is an absolute pivot threshold. The entries ofXᵀXscale with the square of the predictor magnitudes. A well-conditioned design with small values (for example predictors near1e-7) is then rejected as collinear. A near-singular design with large values passes the check. Compare the pivot against the largest initial magnitude in the matrix instead.The coding guidelines require hand-tuned thresholds to be documented as such. Add that note to the comment.
♻️ Proposed relative pivot threshold
fn solve(mut a: Vec<Vec<f64>>, mut rhs: Vec<f64>) -> Option<Vec<f64>> { let n = rhs.len(); + // Scale reference for the singularity test: the largest initial magnitude. + // The `1e-12` factor is HAND-TUNED, not derived — an absolute threshold + // would misclassify rank on very small or very large designs. + let scale = a + .iter() + .flat_map(|row| row.iter()) + .fold(0.0f64, |m, v| m.max(v.abs())) + .max(1.0); for col in 0..n { @@ - if max.abs() < 1e-12 { + if max.abs() < 1e-12 * scale { return None; // singular → collinear predictors }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/jc/src/stats.rs` around lines 462 - 464, Update the singularity check in the surrounding stats computation to use a relative threshold based on the largest initial magnitude in the matrix rather than the absolute 1e-12 cutoff, so scaling predictor values does not change the collinearity decision. Preserve the None return for pivots below the scaled threshold, and document the threshold as hand-tuned in the existing comment.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/board/PR_ARC_INVENTORY.md:
- Line 38: Update the file count and breakdown in the PR inventory entry to
accurately reflect the actual files in the stack: the current count of 5 files
is incorrect, and the breakdown should list the actual composition of three Rust
files and four board/plan files instead of the current enumeration that includes
visibility lines. Ensure visibility lines are not counted as part of the total
file count, only actual files.
In @.claude/board/STATUS_BOARD.md:
- Line 9: Restore the original D-KIA-C1b row in STATUS_BOARD.md, changing only
its permitted status or confidence field in place. Move the updated deliverable,
evidence, and shipment details into a new prepended entry while preserving all
other historical content unchanged.
In `@crates/jc/src/stats.rs`:
- Around line 384-396: Update the loading reconstruction in omega_total so it
preserves relative signs instead of always taking positive roots: keep the first
loading’s sign as the global convention and derive each subsequent sign from the
sign of its covariance with the first item, applying that sign to the recovered
magnitude before computing sum_lambda. Add a unit test for loadings (2, 3, -1)
and verify omega uses the signed hand-calculated total.
---
Nitpick comments:
In `@crates/jc/src/stats.rs`:
- Around line 274-284: Update the agreement calculation around cats, p_e, and
the category loop to build marginal counts for a and b in single passes using
BTreeMap, then compute p_e from those counts without rescanning either slice per
category. Replace the BTreeSet import with BTreeMap and preserve the existing
category probabilities and result behavior.
- Around line 462-464: Update the singularity check in the surrounding stats
computation to use a relative threshold based on the largest initial magnitude
in the matrix rather than the absolute 1e-12 cutoff, so scaling predictor values
does not change the collinearity decision. Preserve the None return for pivots
below the scaled threshold, and document the threshold as hand-tuned in the
existing comment.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 65a50232-442e-4f68-a17b-dc9ca527937e
📒 Files selected for processing (8)
.claude/board/EPIPHANIES.md.claude/board/LATEST_STATE.md.claude/board/PR_ARC_INVENTORY.md.claude/board/STATUS_BOARD.md.claude/plans/kanban-64k-inverted-awareness-v1.mdcrates/jc/src/lib.rscrates/jc/src/reliability.rscrates/jc/src/stats.rs
|
|
||
| ## 2026-08-04 — lance-graph #887 — D-KIA-C1b: the additive `jc` statistics battery (κ, ω, R/η², t-tests, φ) | ||
|
|
||
| **Head:** `<this branch>` (entry written in the same commit as the change, per the hygiene rule's "SAME commit" wording; merge SHA follows on merge). 5 files: 1 new module + 2 visibility lines + 3 board/plan. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Correct the arc file-count inventory.
Line 38 says 5 files, but the supplied stack lists three Rust files and four board/plan files. Update the count and breakdown to list the actual files. Do not count visibility lines as files.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/board/PR_ARC_INVENTORY.md at line 38, Update the file count and
breakdown in the PR inventory entry to accurately reflect the actual files in
the stack: the current count of 5 files is incorrect, and the breakdown should
list the actual composition of three Rust files and four board/plan files
instead of the current enumeration that includes visibility lines. Ensure
visibility lines are not counted as part of the total file count, only actual
files.
| | D-KIA-0 | jc capability map + dichotomous-statistics decision note (phi/KR-20/kappa naming; Spearman dropped at view 2) | lance-graph | Queued | plan W0 | | ||
| | D-KIA-A1 | ⊘ RESCOPED 2026-08-04 (E-ACTOR-IS-NOT-THE-PHASE-PATH-1): #879 is the complete phase-progression path; KanbanActor has no assigned architectural responsibility (legacy compatibility code). SHIPPED: held-owner reschedule/wake. OPEN: run_cycle drained-writer retry guard; missing-owner counter in cognitive_pass | lance-graph | Queued | plan W1 | | ||
| | D-KIA-C1b | jc additive-only extension: kappa + McDonald's omega + r-family effect size (R/R-squared, eta-squared = explained variance) + t-test (t/df/p) + a named phi wrapper. Cohen's d explicitly OUT — calculated separately if ever wanted. HARD CONSTRAINT: additive only — pearson/spearman/cronbach_alpha/icc keep their arithmetic, signature and semantics; any diff changing an existing jc statistic is an automatic reject. ONE sanctioned edit: widening reliability.rs private helpers (mean/all_finite/average_ranks/pop_var) to pub(crate) for reuse, visibility only, no body change. C1 audit found phi = pearson-on-binaries (already present in substance) and KR-20 = alpha-on-dichotomous (naming only); kappa absent = the real gap. Blocks D3's fusion falsifier | lance-graph | Queued | plan W0/C1b | | ||
| | D-KIA-C1b | jc additive-only extension: kappa + McDonald's omega + r-family effect size (R/R-squared, eta-squared = explained variance) + t-test (t/df/p) + a named phi wrapper. Cohen's d explicitly OUT — calculated separately if ever wanted. HARD CONSTRAINT: additive only — pearson/spearman/cronbach_alpha/icc keep their arithmetic, signature and semantics; any diff changing an existing jc statistic is an automatic reject. ONE sanctioned edit: widening reliability.rs private helpers (mean/all_finite/average_ranks/pop_var) to pub(crate) for reuse, visibility only, no body change. C1 audit found phi = pearson-on-binaries (already present in substance) and KR-20 = alpha-on-dichotomous (naming only); kappa absent = the real gap. SHIPPED as crates/jc/src/stats.rs: cohen_kappa, omega_total, phi, multiple_r/multiple_r_squared, eta_squared, t_test_one_sample/paired/welch/student, anova_one_way; 31 new tests (107 lib + 11 doctests green), clippy-clean. Existing-file diff is visibility-only (mean/all_finite -> pub(crate); average_ranks/pop_var NOT widened, unused). Unblocks D3's fusion falsifier | lance-graph | In PR | plan W0/C1b | |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Preserve the append-only board history.
This change rewrites the existing D-KIA-C1b row. It changes the deliverable text and evidence, not only the status or confidence field. Restore the historical row and limit the in-place edit to the allowed status/confidence field. Keep the new shipment details in a new prepended entry.
As per coding guidelines, “governance entries are append-only, with only status/confidence lines updated in place.” Based on learnings, merged historical entries must remain unchanged except for Status and Confidence updates.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.claude/board/STATUS_BOARD.md at line 9, Restore the original D-KIA-C1b row
in STATUS_BOARD.md, changing only its permitted status or confidence field in
place. Move the updated deliverable, evidence, and shipment details into a new
prepended entry while preserving all other historical content unchanged.
Sources: Coding guidelines, Learnings
| let lam_sq = acc / count as f64; | ||
| // Tolerance is RELATIVE to the item's own variance: a perfect-fit item | ||
| // has λ² == σ_ii exactly in real arithmetic but lands a few ulps either | ||
| // side of it in f64, and a bare `< 0.0` test would reject a valid model | ||
| // as misfit. Only a violation LARGER than rounding is a real one. | ||
| let tol = 1e-9 * cov[i][i].abs().max(1.0); | ||
| if lam_sq < -tol || !lam_sq.is_finite() { | ||
| return None; // single-factor model violated (negative common variance) | ||
| } | ||
| lambda[i] = lam_sq.max(0.0).sqrt(); | ||
| } | ||
|
|
||
| let sum_lambda: f64 = lambda.iter().sum(); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
omega_total loses loading signs, so ω is overstated for reverse-keyed items.
The triad identity recovers λ_i² only. Line 393 always takes the positive root. In a congeneric model the loadings share one global sign convention, but individual loadings may have opposite signs relative to each other. A reverse-keyed item has λ_i < 0.
Trace: take three items with true loadings λ = (2, 3, −1). Then σ₁₂ > 0, σ₁₃ < 0, σ₂₃ < 0. Every triad estimate stays positive, for example λ₃² = σ₁₃σ₂₃/σ₁₂ > 0. So all three loadings are taken as positive. Σλ becomes 2+3+1 = 6 instead of the true 2+3−1 = 4, and (Σλ)² is 36 instead of 16. ω is then substantially too high, and the function returns Some rather than None.
Recover the relative signs from the covariances. Fix the sign of the first loading and set every other sign from sign(σ_1j). If the module intends to reject reverse-keyed items instead, document that precondition and return None when the covariance signs are inconsistent with a single positive-loading factor.
🐛 Proposed sign recovery after the λ² loop
- let sum_lambda: f64 = lambda.iter().sum();
+ // λ² alone does not fix the sign. Under a single factor the relative signs
+ // follow from σ_ij = λ_iλ_j: fix λ_0 > 0, then sign(λ_j) = sign(σ_0j).
+ // A reverse-keyed item otherwise inflates (Σλ)² and overstates ω.
+ let signs: Vec<f64> = (0..k)
+ .map(|i| {
+ if i == 0 {
+ 1.0
+ } else if cov[0][i] < 0.0 {
+ -1.0
+ } else {
+ 1.0
+ }
+ })
+ .collect();
+ let sum_lambda: f64 = lambda
+ .iter()
+ .zip(signs.iter())
+ .map(|(&l, &s)| s * l)
+ .sum();Add a unit test with λ = (2, 3, −1) that asserts ω matches the hand-computed value for the signed loadings.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| let lam_sq = acc / count as f64; | |
| // Tolerance is RELATIVE to the item's own variance: a perfect-fit item | |
| // has λ² == σ_ii exactly in real arithmetic but lands a few ulps either | |
| // side of it in f64, and a bare `< 0.0` test would reject a valid model | |
| // as misfit. Only a violation LARGER than rounding is a real one. | |
| let tol = 1e-9 * cov[i][i].abs().max(1.0); | |
| if lam_sq < -tol || !lam_sq.is_finite() { | |
| return None; // single-factor model violated (negative common variance) | |
| } | |
| lambda[i] = lam_sq.max(0.0).sqrt(); | |
| } | |
| let sum_lambda: f64 = lambda.iter().sum(); | |
| let lam_sq = acc / count as f64; | |
| // Tolerance is RELATIVE to the item's own variance: a perfect-fit item | |
| // has λ² == σ_ii exactly in real arithmetic but lands a few ulps either | |
| // side of it in f64, and a bare `< 0.0` test would reject a valid model | |
| // as misfit. Only a violation LARGER than rounding is a real one. | |
| let tol = 1e-9 * cov[i][i].abs().max(1.0); | |
| if lam_sq < -tol || !lam_sq.is_finite() { | |
| return None; // single-factor model violated (negative common variance) | |
| } | |
| lambda[i] = lam_sq.max(0.0).sqrt(); | |
| } | |
| // λ² alone does not fix the sign. Under a single factor the relative signs | |
| // follow from σ_ij = λ_iλ_j: fix λ_0 > 0, then sign(λ_j) = sign(σ_0j). | |
| // A reverse-keyed item otherwise inflates (Σλ)² and overstates ω. | |
| let signs: Vec<f64> = (0..k) | |
| .map(|i| { | |
| if i == 0 { | |
| 1.0 | |
| } else if cov[0][i] < 0.0 { | |
| -1.0 | |
| } else { | |
| 1.0 | |
| } | |
| }) | |
| .collect(); | |
| let sum_lambda: f64 = lambda | |
| .iter() | |
| .zip(signs.iter()) | |
| .map(|(&l, &s)| s * l) | |
| .sum(); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/jc/src/stats.rs` around lines 384 - 396, Update the loading
reconstruction in omega_total so it preserves relative signs instead of always
taking positive roots: keep the first loading’s sign as the global convention
and derive each subsequent sign from the sign of its covariance with the first
item, applying that sign to the recovered magnitude before computing sum_lambda.
Add a unit test for loadings (2, 3, -1) and verify omega uses the signed
hand-calculated total.
…dence External review of the merged #887 found two P1 numerical defects. Both were REPRODUCED before fixing; both now carry regression tests. 1. omega_total erased loading SIGNS. The triad identity yields lambda_i^2, hence only |lambda_i|, and the code took the positive root for every item. omega depends on (sum lambda)^2, where a negatively-keyed item must SUBTRACT. On an exact signed-congeneric fixture (loadings [+1,-1,+1], equal orthogonal residuals) the true value is 0.25 and the shipped code reported 0.75. Fixed by recovering signs from the covariance row: under one factor sign(sigma_ij) = s_i*s_j, so anchor s_0 = +1 and read the rest off row 0. The anchor is free because the model is identified only up to a global flip and (sum lambda)^2 is invariant to one — asserted as its own test. Sign CONSISTENCY is now also checked, which is a genuine partial one-factor structure test. At k=3 it is provably subsumed by the existing negative- lambda^2 guard (the sign product of the single triad is -1 exactly when the pattern is inconsistent); it only becomes reachable at k>=4 where lambda^2 is averaged. The can-it-fire test therefore uses a k=4 fixture and pre-registers that every lambda^2 and psi is non-negative, so only the sign guard can be what rejects. 2. multiple_r_squared was SCALE-DEPENDENT. Normal equations were built on raw columns and a pivot was called singular below an ABSOLUTE 1e-12 — a statement about units, not rank. Measured: an exact linear fit at 1e-8 magnitude returned None, while the identical relationship at unit scale returned 1.0. Fixed by centering the response and every predictor and scaling predictors to unit norm. R-squared is affine-invariant so no correct answer changes, the rank test becomes relative (diagonal is exactly 1), and centering absorbs the intercept — so "collinear with the intercept" correctly reappears as two identical centered columns. The [0,1] clamp is now bounded to a rounding-scale band: a materially out-of-range value means the solve failed and surfaces as None rather than as a plausible 0 or 1. Also corrected / added: - omega's doc claimed rejection means "the congeneric model does not fit". It checks three NECESSARY conditions and cannot certify a one-factor matrix; the vanishing-tetrad constraints are not tested and k>=4 misfit can still return a number. Doc now states what is and is not verified. - BinaryAssociation + binary_association: counts, BOTH marginals, p_o/p_e alongside kappa and phi. The shipped phi doc said marginals are required to interpret it while the function returned a lone scalar; that was an internal contradiction. - kr20(&[Vec<bool>]): the dichotomous naming surface C2 asked for and C1b did not ship. Enforces binary input by type, delegates arithmetic to cronbach_alpha, does not duplicate it. - betacf now returns Option and reports non-convergence instead of presenting the 300th iterate as a p-value; reg_inc_beta uses ln_1p(-x) near x=1 and clamps only a rounding-scale excursion. Board corrections in the same commit: - The "unblocks D3's fusion falsifier" claim was WRONG and contradicted the plan's own C3 (validity requires an external criterion). kappa is chance- corrected agreement under the observed marginals; it measures overlap, not incremental value. D3 is split: D3a (descriptive overlap, unblocked) and D3b (held-out fusion falsifier, BLOCKED on an external criterion and a criterion-appropriate scoring rule). - C2's Spearman wording: on non-constant binary variables the average-rank transform is affine, so rho is REDUNDANT with phi, not degenerate. - C4's Jirak scope: jc::stats p-values are classical independent-sample p-values; jirak.rs is a fingerprint-specific empirical probe, not a general uncertainty engine. A dependent-cohort claim needs its own justified dependence model. - The exact-zero epiphany was too broad. Narrowed to theoretically non-negative FITTED quantities needing a scale-aware tolerance; it is not a licence to stop testing zero, since a zero determinant or zero within-group variance is a real degeneracy. - New epiphany E-THE-CROSS-IDENTITY-SUITE-INHERITED-MY-BLIND-SPOT-1: five independent cross-identities all passed while both defects were live, because every omega fixture had positive loadings and every R-squared fixture was O(1). A cross-identity suite inherits the blind spots of its fixtures silently, since every check reports green. The practice that would have caught both: name the invariances the estimator should have (affine rescaling, global sign flip) and write one test per dimension. 116 lib tests + 13 doctests green; stats.rs clippy-clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K3RyLEbuNSHxxB3NTTrGki
2 participants
New module
crates/jc/src/stats.rs. 107 lib tests + 11 doctests green;stats.rsclippy-clean.Closes what the C1 audit found: κ was genuinely absent and blocks D3's fusion falsifier, while φ and KR-20 turned out to be computations
jcalready shipped under other names.What ships
cohen_kappaomega_totalphireliability::pearsonon a 0/1 coding; takes&[bool]so the binary precondition is unforgeable. Marginal-capped ceiling documented at the function.multiple_r/multiple_r_squaredNone.eta_squared/anova_one_wayt_test_one_sample/_paired/_welch/_studentt,df,p.Every p-value comes from one shared regularised-incomplete-beta core (Lanczos
ln Γ+ modified-Lentz continued fraction), pinned against textbook critical values — never against this code's own output.The effect-size family is r (φ, R, R², η²). Cohen's d is out by construction; the t-tests are η²/R²'s significance companion, not a d-family back door.
The additive constraint held — tighter than permitted
The entire diff to existing files:
plus doc comments and one
pub modline.average_ranksandpop_varwere NOT widened — the carve-out allowed it, but this module doesn't consume them. The newsample_var/sample_covuse the unbiasedn−1divisor: a different estimator frompop_var'sn, not a duplicate. No existing statistic's arithmetic, signature or semantics moved.Validation is by cross-identity, not self-assertion
Each new estimator is checked against an independently-computed quantity, most of them against already-proven code:
pearsonon the 0/1 codingpearson²at one predictort²/(t²+df), andF = t², from the pooled tTwo defects found by the doc examples, both fixed
Filed as
E-EXACT-FIT-IS-WHERE-ABSOLUTE-ZERO-GUARDS-BREAK-1:psi < 0.0rejected the perfect model — ψ = 0 exactly in real arithmetic lands a few ulps either side of zero in f64, so zero-residual items were reported as misfit. Now a variance-relative tolerance, paired with a can-it-fire test proving the guard still bites on a genuine violation (relaxing a guard without that test is the strictly worse defect).aanda+1are collinear with the intercept, soNonewas correct — the code was right, my example wasn't. Pinned by a regression test, since collinearity-with-the-intercept is the variety that gets written by accident.Worth noting: 31 hand-written unit tests, including five cross-identity checks, passed while both defects were live. Unit tests here were built around hand-computable fixtures (deliberately non-degenerate); doc examples are written to be readable, which selects for the clean exact case. A suite optimised for verifiable references systematically under-samples the perfect case.
Gates
cargo test -p jc --lib→ 107 passed.cargo test -p jc --doc→ 11 passed.cargo clippy -p jc --lib --tests→ 0 warnings instats.rs.The crate is pre-existing fmt-dirty and clippy-noisy (HEAD's
reliability.rsalone carries 45 fmt-diff lines; the lib has 30 clippy warnings on main), so only the new file was formatted — reformatting the rest would be exactly the "while I was in there" cleanup the additive constraint forbids. That pre-existing debt is untouched and unhidden.Board hygiene (same commit)
STATUS_BOARDD-KIA-C1b → In PR; plan C1b shipped-note;EPIPHANIESentry;PR_ARC_INVENTORY+LATEST_STATEentries.Deferred, none blocking D3: ω²/ε² (bias-corrected η²), weighted κ, multi-rater Fleiss κ, the d family.
🤖 Generated with Claude Code
https://claude.ai/code/session_01K3RyLEbuNSHxxB3NTTrGki
Generated by Claude Code
Summary by CodeRabbit
New Features
Bug Fixes