panel: complete Tier-1 diagnostics and shared inference framework

[TheHiddenObserver]

Context

statgpu already provides PanelOLS, RandomEffects, PooledOLS, BetweenOLS, FirstDifferenceOLS, and FamaMacBeth, together with several covariance paths. The remaining core gap is not another estimator count; it is the absence of a shared panel inference framework and standard econometric diagnostics expected in routine FE/RE workflows.

The current implementation also duplicates validation, summary construction, OLS/inference logic, and covariance dispatch across panel estimators.

Roadmap: PR #89. Design reference: dev/plans/panel_framework_proposal.md.

Goal

Complete the Panel Tier-1 workflow through three bounded implementation stages while preserving all current fitted values, coefficients, prediction semantics, and documented covariance behavior.

Stage A — shared framework refactor

Introduce or finalize:

• a shared panel base abstraction for input/formula handling, fitted-state lifecycle, prediction validation, and summary construction;
• a covariance-estimator registry or equivalent centralized dispatch;
• shared structured result types for tests and fit statistics;
• common rank-deficiency, degrees-of-freedom, entity/time index, balanced/unbalanced panel, and missing-data validation;
• lazy effect handling where consistent with current public contracts.

This stage must be behavior-preserving. Golden regression tests should freeze current coefficients, fitted values, residuals, covariance matrices, and predictions before structural changes.

Stage B — specification tests and fit statistics

Add:

• Hausman FE-vs-RE test;
• pooling F-test / fixed-effect significance test;
• Breusch-Pagan LM test for random effects versus pooled OLS;
• within, between, and overall R-squared;
• adjusted R-squared with an explicitly documented degrees-of-freedom definition;
• model F-statistic;
• structured test output containing statistic, distribution, degrees of freedom, p-value, null, alternative, and applicability diagnostics.

Singular covariance differences, non-positive test statistics caused by numerical error, incompatible model pairs, and non-nested designs must have explicit behavior rather than silent clipping or generic exceptions.

Stage C — covariance completion

Add or complete:

• robust covariance for RandomEffects;
• HC0, HC2, and HC3 where statistically defined;
• Driscoll-Kraay covariance;
• explicit one-way and two-way cluster contracts;
• bandwidth and kernel selection semantics;
• small-sample and degrees-of-freedom corrections;
• backend-native accumulation without copying full GPU arrays to CPU.

Backend contract

• NumPy CPU;
• CuPy CUDA;
• Torch CUDA;
• explicit device requests must not silently fall back;
• final scalar/result conversion may synchronize, but core transformed design, residual, score, and covariance accumulation must remain on the selected backend.

Formula and data contract

Cover:

• entity and time effects;
• two-way effects;
• explicit and formula-generated intercept behavior;
• categorical terms and interactions where currently supported;
• balanced and unbalanced panels;
• index/order preservation;
• prediction with required entity/time identifiers;
• missing-data row alignment.

External alignment

Use aligned comparisons against:

• Python linearmodels;
• R plm;
• R/Python sandwich implementations where applicable;
• Stata definitions as documentation references when no open implementation is directly callable.

Every comparison must state effect specification, covariance type, small-sample correction, degrees-of-freedom convention, and any parameterization differences.

Non-goals

• no Panel IV/2SLS/GMM;
• no high-dimensional fixed-effect absorption;
• no DID/event study;
• no Arellano-Bond/dynamic-panel GMM;
• no Panel VAR;
• no repository-wide inference rewrite outside panel code.

Required validation

• focused unit tests for each statistic and covariance formula;
• golden behavior-preservation tests before/after Stage A;
• formula and array API parity;
• balanced/unbalanced and one-way/two-way effect cases;
• rank-deficient and invalid model-pair failure tests;
• NumPy/CuPy/Torch parity tests;
• external comparisons for coefficients, covariance, standard errors, test statistics, p-values, and R-squared variants;
• maintained physical-GPU validation on CuPy and Torch;
• synchronized EN/CN panel documentation and examples;
• benchmark evidence for any performance claim.

Acceptance criteria

• Stage A removes the targeted duplication without changing existing public numerical behavior.
• Hausman, pooling F, and Breusch-Pagan LM tests have structured, documented APIs.
• Within/between/overall/adjusted R-squared and model F-statistics are available where defined.
• RandomEffects robust covariance, HC0/HC2/HC3, and Driscoll-Kraay contracts are implemented or explicitly split with a documented blocker.
• All supported statistics work on NumPy, CuPy, and Torch without silent fallback.
• External alignment records definitions and degrees-of-freedom corrections explicitly.
• Complete CPU, compatibility, and physical-GPU gates pass.
• User documentation states applicability restrictions and failure behavior.

PR decomposition

This issue should normally close through at least three PRs:

1. shared base and covariance registry;
2. diagnostics and fit statistics;
3. covariance extensions.

A single PR combining all stages requires an explicit review justification and must not weaken the staged acceptance gates.

[TheHiddenObserver]

Stage A / PR-A is now implemented in draft PR #119 on exact head 246ce33a5b3c97b59563ce2be85fe233ee2e28a8.

Hosted/review status:

• pre-refactor golden behavior was frozen and validated before panel source migration;
• all six panel estimators now use the shared framework where statistically valid;
• existing covariance semantics/df/formula/prediction contracts remain frozen by golden tests;
• final fresh review #4884802467 has no unresolved CRITICAL/HIGH/in-scope MEDIUM findings;
• Tests #1472, Maintenance #485, Release notes #445, Release package #451, and Benchmark Frontend #517 all pass on the exact current head;
• full CPU suite: 2335 passed / 729 skipped / 20 warnings; dedicated Python 3.9 + Torch 2.0.1 CPU regression gate: 12 passed.

Stage A remains PARTIAL_REMOTE_PENDING only for physical CuPy/Torch CUDA validation. The exact clean-head command is documented in PR #119 and uses dev/benchmarks/validate_panel_stage_a_gpu.py.

Issue #93 remains open: after Stage A is physically accepted and merged, Stage B will add Hausman/pooling-F/Breusch-Pagan LM and fit statistics, followed by Stage C covariance expansion.

[TheHiddenObserver]

Panel Tier-1 Stage A update after resolving the physical-GPU blocker:

• PR fix: correct CuPy inverse-function LUT ordering #121 / Issue bug: CuPy inverse-special-function LUT cache swaps x/y grids #120 fixed the pre-existing CuPy inverse-beta/inverse-gamma LUT tuple-order defect that caused zero-width Panel confidence intervals; fix: correct CuPy inverse-function LUT ordering #121 is merged into master and bug: CuPy inverse-special-function LUT cache swaps x/y grids #120 is closed.
• PR refactor: establish shared panel framework Stage A #119 is synchronized with master through a5eb17fa76bd1d9c74d9cfdf60cf64d260a73562 using a two-parent merge. Root/EN/CN changelogs preserve both Stage-A and fix: correct CuPy inverse-function LUT ordering #121 entries.
• The first synchronized CI run exposed two fix: correct CuPy inverse-function LUT ordering #121 tests tied to pre-Stage-A private inference methods. Those tests were migrated to the inherited Stage-A _panel_store_ols_inference contract; no private API was restored and no statistical implementation changed.
• Current exact PR refactor: establish shared panel framework Stage A #119 head: 885c4dcdc04d3a7344f2c3bb917c74d0f81d82db.
• Hosted gates are all green: Tests #1492 (2359 passed, 729 skipped, 20 warnings), Maintenance #505, Release notes #454, Release package #460, Benchmark Frontend #537; Python 3.9–3.12, static/docs, and maintained Torch CPU gates pass.
• Fresh post-sync review 4887864185 found no unresolved CRITICAL, HIGH, or in-scope MEDIUM findings; review threads are empty.

Stage A remains PARTIAL_REMOTE_PENDING only for the final exact-head physical CuPy + Torch Panel matrix (10 representative cases × 2 GPU backends). Stage B diagnostics and Stage C covariance expansion remain pending under this issue.

[TheHiddenObserver]

Stage A is complete and merged via PR #119 at merge commit e9e0ec43b8b2cfcd7600013a60cb02100c72f4f5.

Final acceptance on PR #119 included all hosted gates green and physical Tesla P100 validation on exact clean head 885c4dcdc04d3a7344f2c3bb917c74d0f81d82db: 10/10 CuPy and 10/10 Torch panel cases passed with no silent fallback. The earlier CuPy CI blocker was independently fixed and merged through Issue #120 / PR #121.

Stage B is now active. Scope remains: Hausman FE-vs-RE, pooling F, Breusch-Pagan LM, within/between/overall/adjusted R², model F statistic, and structured applicability/failure outputs. Before implementation, the Stage-B plan will freeze formula/df conventions against plm, linearmodels, and Stata documentation, with explicit handling of singular Hausman covariance differences and unbalanced-panel BP-LM.

[TheHiddenObserver]

Stage B / PR #122 has completed the requested hosted review/fix cycles on exact head 636988751bcbfad3442d24d3073cdfcd2b3ac637.

Closed blockers include pooling-F df/centering, Hausman sample identity and diagnostic FE covariance df, HAC entity-metadata permutation, external FirstDifference transformed-sample alignment, strengthened physical GPU inference coverage, maintained Torch CI ownership, and EN/CN/changelog synchronization.

Exact-head hosted gates are all green: Tests #1531 (2384 passed, 731 skipped, 20 warnings), Python 3.9–3.12 matrix, Torch 2.0 CPU, linearmodels 7.0 external alignment, static/docs, Maintenance #544, Release notes #475, Release package #481, and Benchmark Frontend #548. Fresh review 4888203240 reports no new CRITICAL/HIGH/in-scope MEDIUM findings; review threads = 0.

Status: PARTIAL_REMOTE_PENDING. The only remaining Stage-B merge blocker is clean exact-head physical CuPy + Torch CUDA acceptance using dev/benchmarks/validate_panel_stage_b_gpu.py. PR #122 remains draft until that artifact is audited.

[TheHiddenObserver]

Panel Tier-1 Stage B / PR #122 has now passed its physical GPU gate on exact clean implementation head 636988751bcbfad3442d24d3073cdfcd2b3ac637.

Tesla P100 acceptance summary:

• Python 3.9.16, CuPy 13.6.0, PyTorch 2.0.0;
• schema v2, exact SHA, clean tree, status=success;
• 15/15 CuPy model cases and 15/15 Torch model cases succeeded with the requested backend and no silent CPU fallback;
• balanced/unbalanced Stage-B diagnostics completed consistently;
• Stage-B R²/model-F/pooling-F/BP-LM/Hausman/diagnostic-covariance checks passed;
• Stage-A coefficient inference (coef, BSE, t, p, CI, nobs, df_resid) remained within machine-precision-scale differences of NumPy.

A persistent evidence summary is recorded in dev/reviews/pr122_physical_gpu_validation.md. The current PR head is evidence/docs-only relative to the physically tested implementation SHA; no production or numerical code changed after the P100 run.

Issue #93 remains open after Stage B because Stage C still covers RandomEffects robust covariance, HC0/HC2/HC3, Driscoll-Kraay, and expanded cluster/small-sample corrections.

[TheHiddenObserver]

Stage B / PR #122 has completed its acceptance gates on exact head 9d5ee4c66d69418d301abd2c53bda5310c13be2c.

Final evidence:

• hosted Tests #1557: 2396 passed, 731 skipped, 20 warnings;
• Python 3.9-3.12 matrix, maintained Torch 2.0 CPU, linearmodels==7.0, static/docs, Maintenance #570, Release notes #501, Release package #507, Benchmark Frontend #574: all SUCCESS;
• exact-head Tesla P100 physical validation: schema v2, clean tree, CuPy 15/15 + 2 Hausman, Torch 15/15 + 2 Hausman, requested backend identities confirmed, no CPU fallback, machine-epsilon-scale agreement with NumPy;
• post-physical review 4888767976: no new CRITICAL/HIGH/in-scope MEDIUM findings;
• unresolved review threads: 0.

Stage B is therefore complete and PR #122 is being promoted from Draft to Ready for review. Issue #93 should remain open for Stage C covariance work (HC0/HC2/HC3, RandomEffects robust covariance, Driscoll-Kraay, and expanded cluster corrections).