Add Tutorial 20: HAD for National Brand Campaign with Regional Spend Intensity

[igerber]

Summary

• Add docs/tutorials/20_had_brand_campaign.ipynb: practitioner walkthrough for HeterogeneousAdoptionDiD on a 60-DMA, 8-week panel where every market got the campaign at varying intensity and no untreated comparison group exists. Covers headline WAS on a 2-period collapse (design='auto' → continuous_at_zero), multi-week event study with per-week pointwise CIs and pre-launch placebos, and a stakeholder communication template. 23 cells (13 markdown / 10 code), mirrors T19's structure, no in-notebook drift cell.
• Add tests/test_t20_had_brand_campaign_drift.py: 12 drift tests with module-scoped fixtures pinning panel composition (60 DMAs, 1 at $0), design auto-detection (continuous_at_zero), overall WAS / SE / CI to one-decimal display, dose mean, n_units, and per-week event-study coverage of TRUE_SLOPE=100 plus zero-coverage at every placebo horizon. Mirrors T19's test-file-only pattern (T19's notebook itself has zero in-notebook asserts).
• Wire the tutorial into the existing diff_diff/had.py entry in docs/doc-deps.yaml; extend docs/practitioner_decision_tree.rst § "Varying Spending Levels" with a new subsection for the no-untreated-controls regime + tip cross-linking T20; add T20 entry to docs/tutorials/README.md and [Unreleased] section in CHANGELOG.md.

Methodology references (required if estimator / math changes)

• Method name(s): N/A - no methodology changes. Tutorial walks through the existing HeterogeneousAdoptionDiD estimator (shipped v3.3.0) on a synthetic panel.
• Paper / source link(s): N/A
• Any intentional deviations from the source (and why): None

Validation

• Tests added/updated: tests/test_t20_had_brand_campaign_drift.py (12 tests, 0.06s, all passing).
• Backtest / simulation / notebook evidence (if applicable): notebook executes cleanly via `pytest --nbmake docs/tutorials/20_had_brand_campaign.ipynb` (2.5s). Em-dash count on the notebook: 0 (per project convention). All locked numbers from `_scratch/had_tutorial/20_assemble_outputs.py` are stable; HAD's analytical SE path is bit-identical regardless of backend env (no Rust kernel involved on HAD).

Security / privacy

• Confirm no secrets/PII in this PR: Yes

---

Generated with Claude Code

[github-actions]

Overall Assessment

⛔ Blocker — I found one P0 and two P1 methodology/documentation issues in the new HAD tutorial package. The blocker is that the notebook/test DGP relabels one DMA as zero-dose/never-treated after outcomes are generated, so the quoted WAS/CI/coverage numbers are produced on internally inconsistent synthetic data.

Executive Summary

• P0 The tutorial/test create the “never-treated anchor” by mutating dose and first_treat after outcome generation, without recomputing that DMA’s post-treatment outcomes. The notebook’s reported HAD numbers are therefore locked to mislabeled data.
• P1 The new docs present Tutorial 20 as a pure “no untreated comparison group” / “every unit got some treatment” example, but the example actually relies on a zero-dose Design 1' / continuous_at_zero anchor. In the true no-QUG (d_lower > 0) regime, the estimand and assumptions change.
• P1 The notebook overstates the HAD pretest workflow as verifying parallel trends. Per the registry, the two-period workflow is QUG + linearity only; the pre-trends step is deferred there and only closed by the multi-period joint variants.
• P2 One new drift guard silently skips missing placebo horizons, so the tutorial can lose promised pre-launch rows without the test failing.
• I could not execute pytest/the notebook here because this environment lacks project Python deps (pytest, pandas), so this is a static review.

Methodology

• P0 docs/tutorials/20_had_brand_campaign.ipynb, diff_diff/prep_dgp.py, tests/test_t20_had_brand_campaign_drift.py. The notebook/test zero out one DMA’s dose and set first_treat=0 after generate_continuous_did_data() has already generated outcome and true_att from the original positive dose. Impact: the supposed zero-dose / never-treated reference still carries treated outcomes, so the quoted WAS, CI, and “covers TRUE_SLOPE=100” claims are not tied to a valid DGP. Concrete fix: generate a genuine never-treated unit in the DGP itself, or recompute that DMA’s post-period outcome/true_att after the mutation and then rebaseline all quoted notebook numbers and drift tests.

• P1 docs/tutorials/20_had_brand_campaign.ipynb, docs/practitioner_decision_tree.rst, docs/tutorials/README.md, CHANGELOG.md, docs/methodology/REGISTRY.md, docs/api/had.rst. The new prose repeatedly says this is the regime with “no untreated comparison group” / “every unit gets some treatment,” but the actual example forces a zero-dose anchor and the fit resolves to continuous_at_zero, i.e. Design 1' with target WAS. The registry/API explicitly distinguish that from true Design 1 (d_lower > 0), where the shipped estimator targets WAS_d_lower and warns about extra Assumptions 5/6. Impact: practitioners in a genuine no-zero-dose panel are being pointed at this tutorial as if the same WAS interpretation applies. Concrete fix: either rewrite the tutorial package as a Design 1' / QUG-at-zero example, or change the DGP to a genuine d_lower > 0 case and explicitly document target_parameter="WAS_d_lower" plus the Assumption 5/6 caveat.

• P1 docs/tutorials/20_had_brand_campaign.ipynb, docs/methodology/REGISTRY.md, docs/methodology/REGISTRY.md. The extensions section says the HAD “composite pretest combining QUG, Stute, and Yatchew-HR tests verifies the parallel-trends assumption holds at the boundary.” The shipped two-period workflow does not run the Assumption 7 pre-trends step; it only runs QUG + linearity and explicitly flags that gap. Impact: users can over-trust the workflow as a formal parallel-trends verification when the library documentation says not to. Concrete fix: reword this to distinguish the two-period QUG+linearity workflow from the multi-period aggregate="event_study" joint pre-trends/joint homogeneity workflow.

Code Quality

No additional material findings beyond the methodology issues above.

Performance

No material findings in the changed files.

Maintainability

No material findings beyond the documentation consistency problems above.

Tech Debt

No mitigating TODO.md tracking was added for the P0/P1 issues above, and those would not be mitigated by TODO tracking anyway.

Security

No material findings.

Documentation/Tests

• P2 tests/test_t20_had_brand_campaign_drift.py. test_event_study_pre_placebos_cover_zero() silently skips a claimed placebo horizon when it is missing (if e in event_times:). Impact: the tutorial can lose one or more of the promised e=-4,-3,-2 placebo rows and the drift test still passes. Concrete fix: first assert that all expected placebo horizons are present, then check their values/CIs.

Path to Approval

1. Make the tutorial/test DGP internally consistent: stop post-hoc relabeling a treated DMA as untreated, or recompute that DMA’s outcomes after the relabeling.
2. Choose one HAD regime for Tutorial 20 and make all new docs match it: either Design 1' (continuous_at_zero, WAS) or true Design 1 (WAS_d_lower with Assumption 5/6 caveats).
3. Correct the pretest wording so it no longer claims that the QUG + Stute + Yatchew workflow verifies parallel trends on the two-period path.

[igerber]

/ai-review

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: 6bd3c4e8218e62a859ac11c5933340b523b65f67

---

Overall assessment

✅ Looks good — the prior P0/P1 methodology issues appear resolved. I found only P2/P3 documentation/test consistency issues in the changed files; no unmitigated P0 or P1 findings remain.

Executive summary

• The previous blocker is fixed: the tutorial/test DGP no longer manufactures a zero-dose anchor after outcome generation, and the main notebook now matches Design 1 / continuous_near_d_lower.
• The notebook’s core methodology framing is now mostly aligned with the registry: it presents WAS_d_lower and surfaces the Assumption 5/6 caveat.
• P2 The notebook extensions section still misstates what the QUG test does.
• P2 Summary docs still contain stale WAS / continuous_at_zero descriptions for Tutorial 20, even though the tutorial was rewritten to WAS_d_lower.
• P2 The new decision-tree code block is not runnable as written.
• P3 The new drift file does not actually pin all of the newly quoted prose-level numbers.

Methodology

Previous re-review blockers look addressed: the notebook/test DGP now keep post-treatment doses strictly positive, the main tutorial narrative now documents Design 1 / WAS_d_lower, and the pretest discussion now distinguishes the two-period and event-study workflows.

• Severity: P2. docs/tutorials/20_had_brand_campaign.ipynb:L392, docs/methodology/REGISTRY.md:L2423-L2429, docs/methodology/REGISTRY.md:L2479-L2488. The extensions section says the workflow combines the QUG test with “constant-per-period effect,” but qug_test() is the support-boundary test H0: d̲ = 0. Impact: readers get the wrong picture of what the formal HAD pretest battery actually checks. Concrete fix: change that parenthetical to “quasi-untreated-group / support-infimum test” or remove the parenthetical entirely.

Code Quality

No material findings in changed executable library code.

Performance

No material findings.

Maintainability

• Severity: P2. CHANGELOG.md:L11-L27, docs/tutorials/README.md:L98-L102. The new Unreleased changelog entry correctly says Tutorial 20 is continuous_near_d_lower / WAS_d_lower, but the older Tutorial 20 changelog bullet and the README index still describe it as continuous_at_zero / WAS. Impact: repo docs disagree about the tutorial’s estimand and design path, which recreates part of the earlier review confusion for anyone scanning summary docs. Concrete fix: delete or rewrite the stale CHANGELOG.md bullet at L27, and update the README bullet to say WAS_d_lower.

Tech Debt

No separate tech-debt finding. TODO.md does not mitigate the items above.

Security

No material findings.

Documentation/Tests

• Severity: P2. docs/practitioner_decision_tree.rst:L267-L287, diff_diff/had.py:L2842-L2849. The new decision-tree snippet is not runnable as written: it gestures at zeroing pre-treatment doses, but then passes the unmodified 8-period panel straight into HAD.fit()’s default aggregate="overall" path, which requires exactly two time periods. Impact: copy/paste users will hit a front-door error instead of reproducing the documented workflow. Concrete fix: either show the actual preprocessing and 2-period collapse, or explicitly label the block as schematic and point readers to Tutorial 20 for executable code.
• Severity: P3. docs/tutorials/20_had_brand_campaign.ipynb:L349-L365, tests/test_t20_had_brand_campaign_drift.py:L182-L186, tests/test_t20_had_brand_campaign_drift.py:L251-L265. The drift harness says it locks the quoted markdown, but two new prose claims are only weakly checked: the notebook says placebo effects are within ±0.06 and spend is “median ~$25K”, while the test only enforces abs(placebo) < 0.5 and pins dose_mean, not the median. Impact: those prose claims can drift while the drift file stays green. Concrete fix: either add exact assertions for placebo magnitudes and the sample median, or soften the prose to qualitative wording.

Static-review note: I could not run pytest or nbmake here because this environment does not have pytest, numpy, or pandas installed.

[igerber]

/ai-review

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: d60f9c18af391ab43c028754a15527450af8f249

---

Overall Assessment

✅ Looks good — no unmitigated P0 or P1 findings in the changed files.

Executive Summary

• Prior re-review issues look addressed: the tutorial/docs now consistently frame Tutorial 20 as HAD Design 1 continuous_near_d_lower targeting WAS_d_lower, and the new decision-tree snippet is a real two-period HAD example.
• Methodology cross-check found no undocumented mismatch versus the registry/docstrings for Design 1 identification, event-study pointwise CIs, or the multi-period HAD pretest workflow.
• P3 The new drift test still does not fully pin the notebook’s quoted “within ±0.06” placebo claim; it allows values up to < 0.1.
• P3 The drift-test docstring points maintainers to _scratch/had_tutorial/*, but those paths are not present in the repo.
• Static review only: I could not run pytest here because pytest, numpy, and pandas are not installed in this environment.

Methodology

Affected method(s): HeterogeneousAdoptionDiD docs/tutorial only, specifically the no-untreated Design 1 path (continuous_near_d_lower / WAS_d_lower), event-study pointwise inference, and the HAD pretest workflow narrative.

No findings. The changed docs now line up with the registry on the Assumption 5/6 identification caveat and pointwise event-study CIs (docs/methodology/REGISTRY.md:L2528-L2535), and the tutorial’s extensions section now matches the shipped did_had_pretest_workflow(..., aggregate="event_study") contract (docs/tutorials/20_had_brand_campaign.ipynb:L392-L394, diff_diff/had_pretests.py:L4044-L4047). The new decision-tree example also now satisfies HAD’s two-period overall-path contract (docs/practitioner_decision_tree.rst:L257-L288).

Code Quality

No findings. The PR does not change estimator or inference implementation; executable changes are limited to the new drift test.

Performance

No findings. The test file uses module-scoped fixtures and is small enough that I do not see a meaningful regression risk.

Maintainability

• tests/test_t20_had_brand_campaign_drift.py:L15-L16 — Severity: P3. Impact: the docstring says the locked DGP / quoted outputs come from _scratch/had_tutorial/40_build_notebook.py and _scratch/had_tutorial/20_assemble_outputs.py, but those paths are not present in the repo, so future maintainers cannot inspect the stated source of truth. Concrete fix: either commit those scripts under a tracked path or replace the references with a note that the constants are duplicated directly in the test/notebook.

Tech Debt

No separate finding. TODO.md does not mitigate the minor items above.

Security

No findings. I did not see secrets, credentials, or unsafe new execution surfaces in the changed files.

Documentation/Tests

• docs/tutorials/20_had_brand_campaign.ipynb:L349-L349, tests/test_t20_had_brand_campaign_drift.py:L257-L272 — Severity: P3. Impact: the notebook says the placebo effects are “within ±0.06,” but the test only enforces abs(att) < 0.1, so the quoted prose can drift materially without the harness failing. Concrete fix: either tighten the assertion to the stated bound (or an exact rounded pin), or soften the prose to “within about ±0.1.”