Skip to content

feat(conformal): one-sided alternatives + covariates in the CWZ proxy#669

Merged
igerber merged 2 commits into
mainfrom
feature/sc-conformal-cwz-extensions
Jul 10, 2026
Merged

feat(conformal): one-sided alternatives + covariates in the CWZ proxy#669
igerber merged 2 commits into
mainfrom
feature/sc-conformal-cwz-extensions

Conversation

@igerber

@igerber igerber commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Summary

  • One-sided alternatives (alternative={"two-sided","greater","less"}) on conformal_test / conformal_confidence_intervals / conformal_average_effect. One-sided tests use the SIGNED average-effect statistic S(û)=T_*^{-1/2}·Σ û_t, grounded in CWZ Remark 1's statistic freedom + eq 2 — the paper has no §7; this PR also fixes the prior "(§7 signed-t)" citation in TODO/REGISTRY (a Firpo-Possebom cross-contamination). q=1 enforced for one-sided. CI inversion reports half-line HULLS: the infinite side attaches because p → 1 in the accepted direction; contiguous is recomputed against the final reported ray (Algorithm 1 has no monotonicity guarantee after per-candidate proxy refits); an all-rejected scanned grid reports the infinite side + an uncertified NaN finite endpoint as grid_limited, never a false empty set.
  • Covariates in the proxy (covariates=[...]): pivoted-variable matching rows stack RAW into the constrained-LS objective per the paper's note after eq 6; residuals/statistic stay outcome-only; exchangeability preserved. Exogeneity assumption warned on every covariate run: Z(θ0) null-imputes only the outcome, so covariates enter as observed (no-intervention) values — a policy-affected covariate leaks treated post-period information into the proxy. Block-collapsed average-effect covariates collapse with the same T*-block structure; pointwise CIs precompute the covariate panel once.
  • DiagnosticReport's conformal block carries alternative + covariates for all three kinds. Scope excludes (c) the AR/innovation-permutation path (TODO row rewritten to track only that remainder).

Methodology references (required if estimator / math changes)

  • Method name(s): CWZ conformal inference extensions (one-sided permutation tests; covariate-augmented constrained-LS proxy).
  • Paper / source link(s): Chernozhukov, Wüthrich & Zhu (2021), JASA 116(536); arXiv:1712.09089 (Remark 1; §2.3.2 note after eq 6). REGISTRY §SyntheticControl conformal scope note rewritten (citation fix + hull/exogeneity contracts).
  • Any intentional deviations from the source (and why): none in the statistics; conventions documented in REGISTRY — covariate rows stack raw (the paper's "(transformations of)" delegates scaling), one-sided sets report accepted-set hulls with ray-recomputed contiguity, exogeneity assumption surfaced as a per-run UserWarning.

Validation

  • Tests added/updated: TestConformalOneSided (hand-rolled signed-statistic permutation oracle vs eq 2 on full moving-block enumeration; directional rejection; half-line endpoints; missed-ray grid_limited on a 9-block fixture; deterministic non-contiguous scripted-p patterns incl. rejected-toward-infinity; report propagation) + TestConformalCovariates (proxy-weight mechanism with outcome-only residuals; exogenous fixture + assumption-warning assertion; validation errors; one-sided×covariates composition; permutation floor). All pre-existing conformal/SC/diagnostic-report suites pass unchanged.
  • Backtest / simulation / notebook evidence (if applicable): N/A. Reference scoping: the authors' scinference R package implements only the base machinery (neither extension), so the extensions anchor to the paper formulas via the oracle tests.
  • 5 local review rounds (2 methodology P1s found and fixed: ray-contiguity disclosure; covariate exogeneity leakage — including a test fixture that had blessed the unsafe case).

Security / privacy

  • Confirm no secrets/PII in this PR: Yes

Generated with Claude Code

Extends the SyntheticControl conformal layer (CWZ 2021) with both
user-locked extensions:

(a) alternative={"two-sided","greater","less"} on conformal_test /
conformal_confidence_intervals / conformal_average_effect. One-sided
tests use the SIGNED average-effect statistic S(u)=T_*^{-1/2} sum u_t,
grounded in Remark 1's statistic freedom + eq 2 (the paper has NO
section 7 - the prior TODO/REGISTRY "(§7 signed-t)" wording was a
Firpo-Possebom cross-contamination, fixed here). q=1 enforced for
one-sided. CI inversion reports half-line HULLS: the infinite side
attaches because p -> 1 in the accepted direction; contiguous is
RECOMPUTED against the final reported ray (Algorithm 1 has no
monotonicity guarantee after per-candidate proxy refits); an
all-rejected scanned grid reports the infinite side + an uncertified
NaN finite endpoint as grid_limited (never a false empty set).

(b) covariates=[...] stacks pivoted-variable matching rows RAW into the
constrained-LS proxy objective per the paper's note after eq 6;
residuals/statistic stay outcome-only; exchangeability preserved.
EXOGENEITY ASSUMPTION warned on every covariate run: Z(theta0)
null-imputes only the outcome, so covariates enter as observed
(no-intervention) values - a policy-affected covariate leaks treated
post-period information into the proxy. Block-collapsed average-effect
covariates collapse with the same T*-block structure; pointwise CIs
precompute the covariate panel once and slice per (pre, t) sub-series.

DiagnosticReport's conformal block carries alternative + covariates for
all three kinds. Scope excludes (c) the AR/innovation-permutation path
(TODO row rewritten to track only that remainder).

Tests (5 local review rounds): hand-rolled signed-statistic permutation
oracle; directional rejection; half-line endpoints bracketing the
two-sided interval; missed-ray grid_limited (9-block fixture - the
single-post-slot tie structure floors p at 1/n_blocks); deterministic
non-contiguous patterns incl. rejected-toward-infinity; proxy-weight
mechanism; exogenous covariate fixture + assumption-warning assertion;
report propagation; permutation floor with covariates. All pre-existing
conformal + SC + diagnostic-report suites pass unchanged. Reference
scoping: the authors' scinference R package anchors only the base
machinery; extensions anchor to paper formulas via the oracle tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lbd6nqWmg4snvvBmegwqiw
@github-actions

Copy link
Copy Markdown

Overall Assessment

✅ Looks good. No unmitigated P0/P1 findings.

Executive Summary

  • Affected method: SyntheticControl CWZ conformal inference: conformal_test, conformal_confidence_intervals, and conformal_average_effect.
  • Methodology checks line up with CWZ: null-imputed proxy fitting, eq. 2 p-values, Algorithm 1 inversion, and covariates in proxy estimation are consistent with the cited source and the updated registry. (arxiv.org)
  • The one-sided signed-statistic behavior is an implemented extension documented in docs/methodology/REGISTRY.md:L2443; no blocker.
  • New parameters are propagated through the three conformal surfaces and DiagnosticReport.
  • I could not run the targeted tests locally because pytest is not installed in this environment.

Methodology

  • Severity: P3 informational
    Impact: One-sided signed statistics are not a named CWZ procedure, but CWZ Remark 1 permits alternative statistics, and the registry explicitly documents the signed-statistic and half-line hull convention at docs/methodology/REGISTRY.md:L2443.
    Concrete fix: None required.

Code Quality

  • No findings.

Performance

  • No findings. The added covariate-row construction is negligible relative to the existing grid-refit cost.

Maintainability

  • No findings. The new logic is centralized in _coerce_alternative, _conformal_covariate_rows, _cwz_pvalue, and _apply_one_sided_endpoints.

Tech Debt

  • No blocking findings. The remaining CWZ AR / innovation-permutation path is tracked in TODO.md:L31, so it is not a blocker under the deferred-work rules.

Security

  • No findings.

Documentation / Tests

  • Severity: P3
    Impact: docs/methodology/REGISTRY.md:L2438 says covariate rows can be stacked into the proxy objective, but the same bullet still says “outcomes-only,” which can confuse whether covariates enter the objective or only the residual/statistic surface.
    Concrete fix: Reword that phrase to “outcome residuals/statistic only” or remove “outcomes-only” from the proxy-objective sentence.

  • Severity: P3 verification note
    Impact: Targeted tests were not run locally; pytest is unavailable (pytest: command not found).
    Concrete fix: Confirm CI runs the added TestConformalOneSided and TestConformalCovariates tests.

…proxy bullet

Review P3: the bullet both allowed covariate rows in the objective and
said outcomes-only; the invariant is that residuals/statistic are
outcome-only regardless of covariate rows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lbd6nqWmg4snvvBmegwqiw
@github-actions

Copy link
Copy Markdown

🔁 AI review rerun (requested by @igerber)

Head SHA: 0d7ebe5a4714efd333514657d7ca1bc37aa7162a


Overall Assessment

✅ Looks good. No unmitigated P0/P1 findings.

Executive Summary

  • Affected methods: SyntheticControlResults.conformal_test, conformal_confidence_intervals, and conformal_average_effect.
  • Methodology cross-check passes: null-imputed proxy fitting, eq. 2 p-values, Algorithm 1 inversion, Remark 1 statistic freedom, and covariate rows in proxy estimation are consistent with CWZ or documented in the registry. (arxiv.org)
  • Previous documentation concern is addressed: the registry now says “outcomes by default” and clarifies residuals/statistic are outcome-only at docs/methodology/REGISTRY.md:L2437-L2443.
  • New alternative / covariates metadata is propagated to conformal_inference and DiagnosticReport at diff_diff/diagnostic_report.py:L2687-L2698.
  • I could not run the targeted tests locally because pytest is not installed.

Methodology

  • Severity: P3 informational
    Impact: One-sided signed tests and half-line CI hulls are extensions rather than a named CWZ procedure, but CWZ Remark 1 permits alternative statistics and the exact convention is documented in docs/methodology/REGISTRY.md:L2443-L2443. The implementation is centralized in _cwz_pvalue and _apply_one_sided_endpoints at diff_diff/conformal.py:L188-L233 and diff_diff/conformal.py:L448-L512.
    Concrete fix: None required.

  • Severity: P3 informational
    Impact: Covariate proxy rows are supported by the CWZ note after eqs. 4/6, and the PR documents raw stacking plus the exogeneity warning. Implementation validates finite pivoted covariates and warns on every covariate run at diff_diff/synthetic_control_results.py:L2631-L2696.
    Concrete fix: None required.

Code Quality

No findings.

Performance

No findings. The added covariate-row work is small relative to the existing per-grid proxy refits.

Maintainability

No findings. The new behavior is reasonably localized in _coerce_alternative, _conformal_covariate_rows, _cwz_pvalue, _apply_one_sided_endpoints, and the three conformal public methods.

Tech Debt

  • Severity: P3 informational
    Impact: The remaining CWZ AR / innovation-permutation path is still deferred and tracked in TODO.md:L31-L31, so it is not a blocker under the deferred-work rules.
    Concrete fix: None required for this PR.

Security

No findings.

Documentation / Tests

  • Severity: P3 verification note
    Impact: I could not independently execute TestConformalOneSided or TestConformalCovariates; local command failed with pytest: command not found. The added test coverage is present at tests/test_methodology_synthetic_control.py:L4581-L4866.
    Concrete fix: Confirm CI runs the targeted conformal one-sided and covariate tests.

@igerber igerber added the ready-for-ci Triggers CI test workflows label Jul 10, 2026
@igerber igerber merged commit fd3cc0d into main Jul 10, 2026
35 of 36 checks passed
@igerber igerber deleted the feature/sc-conformal-cwz-extensions branch July 10, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-ci Triggers CI test workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant