LWDiD validation suite: skip-gated methodology acceptance tests + replication goldens (precursor #2 for #588)

[igerber]

Summary

• Second maintainer precursor for the LWDiD contribution (feat: add LWDiD estimator (Lee & Wooldridge 2025, 2026) #588), per the agreed plan there: the independent methodology-validation suite, merged to main ahead of the estimator. Motivation: fork PRs never receive CI AI review (the workflow's fork-skip security gate), so maintainer validation artifacts live on same-repo surfaces where they are reviewable and immutable by the contribution they gate.
• tests/test_methodology_lwdid.py: import-skip-gated on diff_diff.lwdid - collects and skips cleanly on main (zero estimator code exists here); activates automatically when feat: add LWDiD estimator (Lee & Wooldridge 2025, 2026) #588's branch rebases onto main. xfail markers (first use in this codebase; semantics documented in the module docstring) encode feat: add LWDiD estimator (Lee & Wooldridge 2025, 2026) #588's agreed outstanding work as executable acceptance criteria: strict markers must be removed by the commits that fix each item.
• Suite contents (10 classes, calibrated against feat: add LWDiD estimator (Lee & Wooldridge 2025, 2026) #588's current head - 30 pass / 18 xfail / 1 slow-pass there): LW (2026) Table 3/4/A1 Prop 99 replication goldens incl. exact-inference p; castle-doctrine tau_omega adjudicator with a from-scratch (7.18)/(7.19) composite-regression reference that reproduces the paper exactly (0.0917/0.0571 and 0.0666 vs printed 0.092/0.057 and 0.067); SE translation-invariance property tests (the IPW centering bug's repro); cross-estimator identities (plain DiD at 1e-10; per-period eq. (2.20); T=3 detrending closed form (5.7)); from-scratch Procedure 2.1/3.1 + staggered per-cohort references; exact T_{N-2}/T_{N-K-2} inference incl. N1=1; normative event-study API spec (CS-shape + band metadata) with Walmart Tables A4/A5 golden tests xfail until Appendix D lands; Monte Carlo bias-ordering under the paper's heterogeneous-trend DGP (slow-marked); minimum-pre-period contracts.
• benchmarks/data/lwdid_walmart_eventstudy_golden.json: Tables A4/A5 (14 relative periods x 5 estimator columns, ATT + bootstrap SE), extracted from the June 8 2026 SSRN revision via dual-mode pdftotext with anchor cross-checks; provenance (incl. PDF SHA-256) embedded.
• benchmarks/data/real/castle_lw_subset.csv: the real Cheng & Hoekstra (2013) panel (Cunningham 2021 packaging; 50 states, 2000-2010, cohorts {2005:1, 2006:13, 2007:4, 2008:2, 2009:1}, lhomicide outcome), following the benchmarks/data/real/ precedent. Committed because the causaldata/causal_datasets repo backing load_castle_doctrine is dead (404) - that silent-synthetic loader defect is now tracked as a Medium TODO row for a follow-up source PR.
• REGISTRY: one Note line pointing the LWDiD section at the suite; TODO: the upgraded loader-defect row + a real-data CI canary row.

Methodology references (required if estimator / math changes)

• Method name(s): LWDiD (validation artifacts only; no estimator/math code changes).
• Paper / source link(s):
  • Lee, S. J. & Wooldridge, J. M. (2025). https://ssrn.com/abstract=4516518 (June 8, 2026 revision) - Appendix F Tables A4/A5 goldens.
  • Lee, S. J. & Wooldridge, J. M. (2026). https://ssrn.com/abstract=5325686 (cover date February 3, 2026) - Tables 3/4/A1, castle-laws targets, Monte Carlo DGP.
• Any intentional deviations from the source (and why): None in assertions. Two calibration notes are documented in-test: the castle point targets require never-treated controls (matching the paper's (7.10) construction), and the RI p-value test is xfail-documented as a convention divergence pending reconciliation with the authors' Stata lwdid, ri.

Validation

• Tests added/updated: tests/test_methodology_lwdid.py (10 classes, 49 tests). On main: 1 module skip (verified). Against feat: add LWDiD estimator (Lee & Wooldridge 2025, 2026) #588's current head (calibration graft): 30 passed, 18 xfailed, slow Monte Carlo passes. Every xfail was calibrated against actual behavior, not assumed.
• Backtest / simulation / notebook evidence (if applicable): from-scratch composite regression reproduces the castle targets to printed precision; golden extraction cross-checked against the review-doc anchor WATT(1) = 0.032 (0.005).

Security / privacy

• Confirm no secrets/PII in this PR: Yes. New data files are public replication data (MIT-licensed / published-study panels) with embedded provenance.

---

Generated with Claude Code

[github-actions]

Overall Assessment

✅ Looks good — no unmitigated P0/P1 findings.

Executive Summary

• The PR adds LWDiD validation artifacts only; no estimator, math, default behavior, or variance implementation is changed.
• The new LWDiD targets match the Methodology Registry and paper-review notes for Prop 99, castle-law composite aggregation, exact t inference, event-study anchor exclusions, and multiplier bootstrap requirements.
• Known future gaps are documented with xfail markers and a REGISTRY note; because the estimator is absent on main, the module-level import skip is appropriate.
• The real-data skip/canary limitation is explicitly tracked in TODO.md, so it is informational, not blocking.
• I could not execute pytest in this sandbox because pandas is unavailable and the filesystem is read-only; I did validate AST parsing and the committed data files structurally.

Methodology

No P0/P1 findings.

P3 — Forward-looking xfail gates must be removed before LWDiD acceptance

Location: tests/test_methodology_lwdid.py:L122-L141, tests/test_methodology_lwdid.py:L439-L449, tests/test_methodology_lwdid.py:L626-L645, tests/test_methodology_lwdid.py:L692-L749

Impact: Several methodology-critical checks are currently xfail-marked: IPW IF centering, composite-regression overall SE, N_infinity >= 2, and event-study Algorithm 1 SEs. This is acceptable for the precursor suite because diff_diff.lwdid does not exist on main and the REGISTRY now documents that xfail markers encode PR #588 acceptance criteria. It must not be treated as acceptance once the estimator lands.

Concrete fix: In the estimator PR, remove the relevant xfail marker in the same commit that implements each requirement. For the non-strict event-study SE golden marker, either make it strict or replace it with a deterministic tolerance/seed strategy before relying on it as a regression guard.

Code Quality

No findings.

The helper implementations are local, readable, and scoped to validation. The from-scratch references use direct formulas rather than estimator internals, which is appropriate for methodology tests.

Performance

No blocking findings.

P3 — Slow Monte Carlo is correctly marked

Location: tests/test_methodology_lwdid.py:L790-L854

Impact: The 200-rep Monte Carlo is excluded from default pytest via the existing slow marker configuration.

Concrete fix: No action required.

Maintainability

No findings.

The REGISTRY note at docs/methodology/REGISTRY.md:L2146-L2147 gives future reviewers a clear pointer to the validation suite and golden data.

Tech Debt

No blocking findings.

P3 — Real-data canary is tracked

Location: TODO.md:L53-L54, tests/test_methodology_lwdid.py:L149-L168

Impact: Prop 99 and Walmart tests skip when loaders fall back to synthetic data, so a CI network/source issue could reduce coverage. This is explicitly tracked in TODO.md.

Concrete fix: No PR-blocking action required; implement the TODO canary in a follow-up CI change.

Security

No findings.

The new committed CSV/JSON data appear to be public replication artifacts and contain no obvious secrets or private identifiers.

Documentation/Tests

No blocking findings.

Validation performed:

• tests/test_methodology_lwdid.py AST parse succeeds.
• benchmarks/data/real/castle_lw_subset.csv has 550 rows, 50 states, years 2000-2010, and cohort counts {2005: 1, 2006: 13, 2007: 4, 2008: 2, 2009: 1}, matching the suite’s integrity assertions.
• benchmarks/data/lwdid_walmart_eventstudy_golden.json has both A4/A5 tables with event times 0..13.

I could not run pytest end-to-end in this sandbox because pandas is not installed, and bytecode compilation could not write __pycache__ under the read-only filesystem.