From 1518bbc32f6bdc7e178aa11855aaf8911ea9049a Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 12 Apr 2026 12:46:17 -0400 Subject: [PATCH 01/11] Add Phase 2 multi-horizon event study DID_l for ChaisemartinDHaultfoeuille Implements ROADMAP items 2a-2h: multi-horizon DID_l via per-group DID_{g,l} building block (Eq 3 of dynamic paper), per-horizon analytical SE, dynamic placebos DID^{pl}_l, normalized DID^n_l, cost-benefit aggregate delta, sup-t simultaneous confidence bands, plot_event_study() integration, and R DIDmultiplegtDYN parity tests at multiple horizons. L_max parameter controls multi-horizon mode; L_max=None preserves exact Phase 1 behavior. Co-Authored-By: Claude Opus 4.6 (1M context) --- CHANGELOG.md | 10 + README.md | 28 +- ROADMAP.md | 18 +- benchmarks/R/generate_dcdh_dynr_test_values.R | 101 ++ benchmarks/data/dcdh_dynr_golden_values.json | 294 +++++ diff_diff/chaisemartin_dhaultfoeuille.py | 1086 ++++++++++++++++- .../chaisemartin_dhaultfoeuille_bootstrap.py | 81 +- .../chaisemartin_dhaultfoeuille_results.py | 173 ++- diff_diff/visualization/_event_study.py | 61 +- docs/api/chaisemartin_dhaultfoeuille.rst | 22 +- docs/choosing_estimator.rst | 16 +- docs/llms-full.txt | 20 +- docs/llms.txt | 2 +- docs/methodology/REGISTRY.md | 24 +- tests/test_chaisemartin_dhaultfoeuille.py | 422 ++++++- ...test_chaisemartin_dhaultfoeuille_parity.py | 82 +- 16 files changed, 2342 insertions(+), 98 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 758eae30..5c1fe043 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **`twowayfeweights()`** — standalone helper function for the TWFE decomposition diagnostic (Theorem 1 of de Chaisemartin & D'Haultfœuille 2020), available without instantiating the full estimator. Returns a `TWFEWeightsResult` with per-cell weights, fraction negative, `sigma_fe`, and `beta_fe`. - **`generate_reversible_did_data()`** — new generator in `diff_diff.prep` producing reversible-treatment panel data for testing and tutorials. Patterns: `single_switch` (default, A5-safe), `joiners_only`, `leavers_only`, `mixed_single_switch`, `random`, `cycles`, `marketing`. Returns columns `group`, `period`, `treatment`, `outcome`, `true_effect`, `d_lag`, `switcher_type`. - **REGISTRY.md `## ChaisemartinDHaultfoeuille` section** — single canonical source for dCDH methodology, equations, edge cases, and all documented deviations from the R `DIDmultiplegtDYN` reference implementation. Cites the AER 2020 paper and the dynamic companion paper (NBER WP 29873) by reference; primary papers are upstream sources, not in-repo files. +- **Phase 2: Multi-horizon event study for `ChaisemartinDHaultfoeuille`** — adds `L_max` parameter to `fit()` for computing `DID_l` at horizons `l = 1, ..., L_max` using the per-group building block from Equation 3 of the dynamic companion paper. Ships: + - Per-horizon point estimates and cohort-recentered analytical SE + - Dynamic placebos `DID^{pl}_l` with dual eligibility condition (Web Appendix Section 1.1) + - Normalized estimator `DID^n_l = DID_l / delta^D_l` (Section 3.2) + - Cost-benefit aggregate `delta` (Section 3.3, Lemma 4) — becomes `overall_att` when `L_max > 1` + - Sup-t simultaneous confidence bands via multiplier bootstrap + - `plot_event_study()` integration with `<50%` switcher warning for far horizons + - `to_dataframe(level="event_study")` and `to_dataframe(level="normalized")` output + - Per-horizon bootstrap with bootstrap SE/CI/p-value propagation to event_study_effects + - `L_max=None` (default) preserves exact Phase 1 behavior ## [3.0.1] - 2026-04-07 diff --git a/README.md b/README.md index 5122daa6..65de0d64 100644 --- a/README.md +++ b/README.md @@ -1213,6 +1213,32 @@ ChaisemartinDHaultfoeuille( | `n_groups_dropped_crossers`, `n_groups_dropped_singleton_baseline` | Filter counts (multi-switch groups dropped before estimation; singleton-baseline groups excluded from variance) | | `n_groups_dropped_never_switching` | Backwards-compatibility metadata. Never-switching groups participate in the variance via stable-control roles; this field is no longer a filter count. | +**Multi-horizon event study** (Phase 2 - pass `L_max` to `fit()`): + +```python +results = est.fit(data, outcome="outcome", group="group", + time="period", treatment="treatment", L_max=5) + +# Per-horizon effects with analytical SE +for horizon in sorted(results.event_study_effects): + e = results.event_study_effects[horizon] + print(f" l={horizon}: DID_l={e['effect']:.3f} (SE={e['se']:.3f})") + +# Cost-benefit delta (becomes overall_att when L_max > 1) +print(f"Cost-benefit delta: {results.cost_benefit_delta['delta']:.3f}") + +# Normalized effects: DID^n_l = DID_l / l (for binary treatment) +for horizon in sorted(results.normalized_effects): + print(f" DID^n_{horizon} = {results.normalized_effects[horizon]['effect']:.3f}") + +# Event study DataFrame (includes placebos as negative horizons) +df = results.to_dataframe("event_study") + +# Plot (integrates with plot_event_study) +from diff_diff import plot_event_study +plot_event_study(results) +``` + **Standalone TWFE decomposition diagnostic** (without fitting the full estimator): ```python @@ -1232,7 +1258,7 @@ print(f"sigma_fe (sign-flipping threshold): {diagnostic.sigma_fe:.3f}") > **Note:** Phase 1 requires panels with a **balanced baseline** (every group observed at the first global period) and **no interior period gaps**. Late-entry groups (missing the baseline) raise `ValueError`; interior-gap groups are dropped with a warning; terminally-missing groups (early exit / right-censoring) are retained and contribute from their observed periods only. This is a documented deviation from R `DIDmultiplegtDYN`, which supports unbalanced panels — see [`docs/methodology/REGISTRY.md`](docs/methodology/REGISTRY.md) for the rationale, the defensive guards that make terminal missingness safe, and workarounds for unbalanced inputs. -> **Note:** Survey design (`survey_design`), event-study aggregation (`aggregate`), covariate adjustment (`controls`), and HonestDiD integration (`honest_did`) are not yet supported. They raise `NotImplementedError` with phase pointers — see [`ROADMAP.md`](ROADMAP.md) for the full multi-phase rollout. +> **Note:** Survey design (`survey_design`), covariate adjustment (`controls`), group-specific linear trends (`trends_linear`), and HonestDiD integration (`honest_did`) are not yet supported. They raise `NotImplementedError` with phase pointers - see [`ROADMAP.md`](ROADMAP.md) for the Phase 3 rollout. ### Triple Difference (DDD) diff --git a/ROADMAP.md b/ROADMAP.md index 1f6da4ab..a3778b48 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -148,18 +148,18 @@ The dynamic companion paper subsumes the AER 2020 paper: `DID_1 = DID_M`. The si ### Phase 2: Dynamic event study (multiple horizons) -*Goal: Add `aggregate="event_study"` mode to the same class. Loops the Phase 1 machinery over horizons `l = 1, ..., L`. No API breakage from Phase 1. No new tutorial — the comprehensive tutorial waits for Phase 3.* +*Goal: Add multi-horizon event study to the same class via the `L_max` parameter. Loops the Phase 1 machinery over horizons `l = 1, ..., L`. No API breakage from Phase 1. No new tutorial - the comprehensive tutorial waits for Phase 3.* | Item | Priority | Status | |------|----------|--------| -| **2a.** Multi-horizon `DID_l` via the cohort framework, with horizon parameter `L_max` | HIGH | Not started | -| **2b.** Multi-horizon analytical SE (same plug-in formula looped over horizons) | HIGH | Not started | -| **2c.** Dynamic placebos `DID^{pl}_l` for pre-trends testing (Web Appendix Section 1.1 of dynamic paper) | HIGH | Not started | -| **2d.** Normalized estimator `DID^n_l` (Section 3.2 of dynamic paper) | MEDIUM | Not started | -| **2e.** Cost-benefit aggregate `delta` (Section 3.3 of dynamic paper, Lemma 4) | MEDIUM | Not started | -| **2f.** Simultaneous (sup-t) confidence bands for event study plots | MEDIUM | Not started | -| **2g.** `plot_event_study()` integration; `< 50%`-of-switchers warning for far horizons | MEDIUM | Not started | -| **2h.** Parity tests vs `did_multiplegt_dyn` for multi-horizon designs | HIGH | Not started | +| **2a.** Multi-horizon `DID_l` via per-group `DID_{g,l}` building block, with `L_max` parameter | HIGH | Shipped | +| **2b.** Multi-horizon analytical SE (cohort-recentered plug-in per horizon) | HIGH | Shipped | +| **2c.** Dynamic placebos `DID^{pl}_l` for pre-trends testing (Web Appendix Section 1.1 of dynamic paper) | HIGH | Shipped | +| **2d.** Normalized estimator `DID^n_l` (Section 3.2 of dynamic paper) | MEDIUM | Shipped | +| **2e.** Cost-benefit aggregate `delta` (Section 3.3 of dynamic paper, Lemma 4) | MEDIUM | Shipped | +| **2f.** Simultaneous (sup-t) confidence bands for event study plots | MEDIUM | Shipped | +| **2g.** `plot_event_study()` integration; `< 50%`-of-switchers warning for far horizons | MEDIUM | Shipped | +| **2h.** Parity tests vs `did_multiplegt_dyn` for multi-horizon designs | HIGH | In progress | ### Phase 3: Covariates, extensions, and tutorial diff --git a/benchmarks/R/generate_dcdh_dynr_test_values.R b/benchmarks/R/generate_dcdh_dynr_test_values.R index 8bbe76fc..eeb68fee 100644 --- a/benchmarks/R/generate_dcdh_dynr_test_values.R +++ b/benchmarks/R/generate_dcdh_dynr_test_values.R @@ -287,6 +287,107 @@ scenarios$hand_calculable_worked_example <- list( results = extract_dcdh_l1(res5) ) +# --------------------------------------------------------------------------- +# Phase 2: Multi-horizon scenarios (effects > 1) +# --------------------------------------------------------------------------- + +# Helper: extract multi-horizon results from did_multiplegt_dyn output +extract_dcdh_multi <- function(res, n_effects, n_placebos = 0) { + effects <- res$results$Effects + if (is.null(effects)) { + stop("did_multiplegt_dyn returned no Effects; check the input data") + } + + out <- list(effects = list(), placebos = list()) + + for (i in seq_len(min(n_effects, nrow(effects)))) { + out$effects[[as.character(i)]] <- list( + overall_att = as.numeric(effects[i, "Estimate"]), + overall_se = as.numeric(effects[i, "SE"]), + overall_ci_lo = as.numeric(effects[i, "LB CI"]), + overall_ci_hi = as.numeric(effects[i, "UB CI"]), + n_switchers = as.numeric(effects[i, "N"]) + ) + } + + placebos <- res$results$Placebos + if (!is.null(placebos) && n_placebos > 0) { + for (i in seq_len(min(n_placebos, nrow(placebos)))) { + out$placebos[[as.character(i)]] <- list( + effect = as.numeric(placebos[i, "Estimate"]), + se = as.numeric(placebos[i, "SE"]), + ci_lo = as.numeric(placebos[i, "LB CI"]), + ci_hi = as.numeric(placebos[i, "UB CI"]) + ) + } + } + + out +} + +# Scenario 6: joiners_only multi-horizon (L_max=3, placebo=3) +# Uses n_periods=8 to give enough room for 3 positive + 3 placebo horizons +cat(" Scenario 6: joiners_only_multi_horizon\n") +d6 <- gen_reversible(n_groups = N_GOLDEN, n_periods = 8, + pattern = "joiners_only", seed = 106) +res6 <- did_multiplegt_dyn( + df = d6, outcome = "outcome", group = "group", time = "period", + treatment = "treatment", effects = 3, placebo = 3, ci_level = 95 +) +scenarios$joiners_only_multi_horizon <- list( + data = export_data(d6), + params = list(pattern = "joiners_only", n_groups = N_GOLDEN, n_periods = 8, + seed = 106, effects = 3, placebo = 3, ci_level = 95), + results = extract_dcdh_multi(res6, n_effects = 3, n_placebos = 3) +) + +# Scenario 7: leavers_only multi-horizon (L_max=3, placebo=3) +cat(" Scenario 7: leavers_only_multi_horizon\n") +d7 <- gen_reversible(n_groups = N_GOLDEN, n_periods = 8, + pattern = "leavers_only", seed = 107) +res7 <- did_multiplegt_dyn( + df = d7, outcome = "outcome", group = "group", time = "period", + treatment = "treatment", effects = 3, placebo = 3, ci_level = 95 +) +scenarios$leavers_only_multi_horizon <- list( + data = export_data(d7), + params = list(pattern = "leavers_only", n_groups = N_GOLDEN, n_periods = 8, + seed = 107, effects = 3, placebo = 3, ci_level = 95), + results = extract_dcdh_multi(res7, n_effects = 3, n_placebos = 3) +) + +# Scenario 8: mixed_single_switch multi-horizon (L_max=5, placebo=4) +# Uses n_periods=10 for far horizons +cat(" Scenario 8: mixed_single_switch_multi_horizon\n") +d8 <- gen_reversible(n_groups = N_GOLDEN, n_periods = 10, + pattern = "mixed_single_switch", seed = 108) +res8 <- did_multiplegt_dyn( + df = d8, outcome = "outcome", group = "group", time = "period", + treatment = "treatment", effects = 5, placebo = 4, ci_level = 95 +) +scenarios$mixed_single_switch_multi_horizon <- list( + data = export_data(d8), + params = list(pattern = "mixed_single_switch", n_groups = N_GOLDEN, n_periods = 10, + seed = 108, effects = 5, placebo = 4, ci_level = 95), + results = extract_dcdh_multi(res8, n_effects = 5, n_placebos = 4) +) + +# Scenario 9: joiners_only long panel multi-horizon (L_max=5, placebo=5) +# Uses n_periods=12 and n_groups=80 for thorough coverage +cat(" Scenario 9: joiners_only_long_multi_horizon\n") +d9 <- gen_reversible(n_groups = N_GOLDEN, n_periods = 12, + pattern = "joiners_only", seed = 109) +res9 <- did_multiplegt_dyn( + df = d9, outcome = "outcome", group = "group", time = "period", + treatment = "treatment", effects = 5, placebo = 5, ci_level = 95 +) +scenarios$joiners_only_long_multi_horizon <- list( + data = export_data(d9), + params = list(pattern = "joiners_only", n_groups = N_GOLDEN, n_periods = 12, + seed = 109, effects = 5, placebo = 5, ci_level = 95), + results = extract_dcdh_multi(res9, n_effects = 5, n_placebos = 5) +) + # --------------------------------------------------------------------------- # Write output # --------------------------------------------------------------------------- diff --git a/benchmarks/data/dcdh_dynr_golden_values.json b/benchmarks/data/dcdh_dynr_golden_values.json index faf2ff62..f640761c 100644 --- a/benchmarks/data/dcdh_dynr_golden_values.json +++ b/benchmarks/data/dcdh_dynr_golden_values.json @@ -135,6 +135,300 @@ "overall_ci_hi": 6.0333753222, "n_switchers": 4 } + }, + "joiners_only_multi_horizon": { + "data": { + "group": [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, 17, 17, 17, 17, 17, 17, 17, 17, 18, 18, 18, 18, 18, 18, 18, 18, 19, 19, 19, 19, 19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 20, 20, 21, 21, 21, 21, 21, 21, 21, 21, 22, 22, 22, 22, 22, 22, 22, 22, 23, 23, 23, 23, 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 25, 25, 25, 25, 25, 25, 25, 25, 26, 26, 26, 26, 26, 26, 26, 26, 27, 27, 27, 27, 27, 27, 27, 27, 28, 28, 28, 28, 28, 28, 28, 28, 29, 29, 29, 29, 29, 29, 29, 29, 30, 30, 30, 30, 30, 30, 30, 30, 31, 31, 31, 31, 31, 31, 31, 31, 32, 32, 32, 32, 32, 32, 32, 32, 33, 33, 33, 33, 33, 33, 33, 33, 34, 34, 34, 34, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36, 36, 36, 36, 37, 37, 37, 37, 37, 37, 37, 37, 38, 38, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39, 40, 40, 40, 40, 40, 40, 40, 40, 41, 41, 41, 41, 41, 41, 41, 41, 42, 42, 42, 42, 42, 42, 42, 42, 43, 43, 43, 43, 43, 43, 43, 43, 44, 44, 44, 44, 44, 44, 44, 44, 45, 45, 45, 45, 45, 45, 45, 45, 46, 46, 46, 46, 46, 46, 46, 46, 47, 47, 47, 47, 47, 47, 47, 47, 48, 48, 48, 48, 48, 48, 48, 48, 49, 49, 49, 49, 49, 49, 49, 49, 50, 50, 50, 50, 50, 50, 50, 50, 51, 51, 51, 51, 51, 51, 51, 51, 52, 52, 52, 52, 52, 52, 52, 52, 53, 53, 53, 53, 53, 53, 53, 53, 54, 54, 54, 54, 54, 54, 54, 54, 55, 55, 55, 55, 55, 55, 55, 55, 56, 56, 56, 56, 56, 56, 56, 56, 57, 57, 57, 57, 57, 57, 57, 57, 58, 58, 58, 58, 58, 58, 58, 58, 59, 59, 59, 59, 59, 59, 59, 59, 60, 60, 60, 60, 60, 60, 60, 60, 61, 61, 61, 61, 61, 61, 61, 61, 62, 62, 62, 62, 62, 62, 62, 62, 63, 63, 63, 63, 63, 63, 63, 63, 64, 64, 64, 64, 64, 64, 64, 64, 65, 65, 65, 65, 65, 65, 65, 65, 66, 66, 66, 66, 66, 66, 66, 66, 67, 67, 67, 67, 67, 67, 67, 67, 68, 68, 68, 68, 68, 68, 68, 68, 69, 69, 69, 69, 69, 69, 69, 69, 70, 70, 70, 70, 70, 70, 70, 70, 71, 71, 71, 71, 71, 71, 71, 71, 72, 72, 72, 72, 72, 72, 72, 72, 73, 73, 73, 73, 73, 73, 73, 73, 74, 74, 74, 74, 74, 74, 74, 74, 75, 75, 75, 75, 75, 75, 75, 75, 76, 76, 76, 76, 76, 76, 76, 76, 77, 77, 77, 77, 77, 77, 77, 77, 78, 78, 78, 78, 78, 78, 78, 78, 79, 79, 79, 79, 79, 79, 79, 79, 80, 80, 80, 80, 80, 80, 80, 80, 81, 81, 81, 81, 81, 81, 81, 81, 82, 82, 82, 82, 82, 82, 82, 82, 83, 83, 83, 83, 83, 83, 83, 83, 84, 84, 84, 84, 84, 84, 84, 84, 85, 85, 85, 85, 85, 85, 85, 85, 86, 86, 86, 86, 86, 86, 86, 86, 87, 87, 87, 87, 87, 87, 87, 87, 88, 88, 88, 88, 88, 88, 88, 88, 89, 89, 89, 89, 89, 89, 89, 89, 90, 90, 90, 90, 90, 90, 90, 90, 91, 91, 91, 91, 91, 91, 91, 91, 92, 92, 92, 92, 92, 92, 92, 92, 93, 93, 93, 93, 93, 93, 93, 93, 94, 94, 94, 94, 94, 94, 94, 94, 95, 95, 95, 95, 95, 95, 95, 95, 96, 96, 96, 96, 96, 96, 96, 96, 97, 97, 97, 97, 97, 97, 97, 97, 98, 98, 98, 98, 98, 98, 98, 98, 99, 99, 99, 99, 99, 99, 99, 99, 100, 100, 100, 100, 100, 100, 100, 100, 101, 101, 101, 101, 101, 101, 101, 101, 102, 102, 102, 102, 102, 102, 102, 102, 103, 103, 103, 103, 103, 103, 103, 103, 104, 104, 104, 104, 104, 104, 104, 104, 105, 105, 105, 105, 105, 105, 105, 105, 106, 106, 106, 106, 106, 106, 106, 106, 107, 107, 107, 107, 107, 107, 107, 107, 108, 108, 108, 108, 108, 108, 108, 108, 109, 109, 109, 109, 109, 109, 109, 109, 110, 110, 110, 110, 110, 110, 110, 110, 111, 111, 111, 111, 111, 111, 111, 111, 112, 112, 112, 112, 112, 112, 112, 112, 113, 113, 113, 113, 113, 113, 113, 113, 114, 114, 114, 114, 114, 114, 114, 114, 115, 115, 115, 115, 115, 115, 115, 115, 116, 116, 116, 116, 116, 116, 116, 116, 117, 117, 117, 117, 117, 117, 117, 117, 118, 118, 118, 118, 118, 118, 118, 118, 119, 119, 119, 119, 119, 119, 119, 119], + "period": [0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7], + "treatment": [0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + "outcome": [10.2656879152, 10.1855776599, 11.1188942482, 12.496299267, 10.9722926996, 14.0643845102, 13.1958709685, 14.2281457591, 7.5045443312, 8.0528885051, 9.0833420501, 8.3124171547, 9.2240305109, 7.3765891844, 10.24059724, 10.5718244715, 11.5003186737, 11.2362143675, 11.4221880831, 11.7755967488, 11.3419327554, 11.1243561203, 13.1634238907, 13.5801068183, 8.2867847685, 8.0633570591, 7.5373597689, 7.8237698536, 7.3784144896, 8.6860687527, 10.9322953105, 10.5352921776, 10.6577177794, 10.8667197445, 11.5748832574, 10.9454689671, 13.096293037, 13.6684316866, 13.7442302611, 13.5657449767, 8.210999402, 9.7945133132, 10.6150120046, 10.7646660764, 10.427337216, 10.1068712505, 11.4106143192, 11.160221718, 12.4973999411, 12.0170097102, 13.9977820444, 14.0885321542, 14.8943490328, 14.6103353493, 13.9978352753, 14.2595492042, 11.8362823083, 12.1843317103, 12.4452787384, 13.0668944288, 12.5634409053, 15.3873296037, 14.5206736378, 15.6061038545, 10.7928631891, 10.9916563427, 10.7769199469, 10.0378931089, 11.0060755854, 12.9338833423, 13.0061190565, 13.2638208771, 10.6492525969, 10.1555689493, 10.703782315, 11.6115996924, 11.4823050227, 11.2055911877, 11.6205788435, 13.5068322265, 9.2757604219, 9.1981487978, 10.2546691533, 9.3892371625, 10.4958552902, 11.0586552413, 12.0896502736, 11.7479035112, 9.9106746715, 8.7254170932, 9.8825979226, 9.1920725837, 10.5348156172, 10.2696975698, 12.0528144989, 13.3336192015, 7.1423653549, 6.8228247832, 6.3760131074, 6.5075024652, 6.0743064915, 5.9721848774, 6.6777381203, 8.6776406604, 12.0624479308, 12.2683440724, 15.1908883735, 13.9672214201, 14.6424359377, 14.2903698454, 14.4869157478, 14.5593590689, 7.8080773838, 7.4113875034, 7.0717588862, 7.3425635366, 7.8815688233, 6.9568549682, 9.6946318691, 9.342941948, 9.1515916266, 9.8180903643, 8.2749070438, 9.6701452812, 9.5445524169, 12.1966241338, 10.9990951916, 11.1515116499, 8.3600649912, 9.4291349208, 8.970281448, 11.7570678942, 11.3733455937, 10.8216991859, 12.6547241174, 12.2172709771, 8.6857630172, 8.1296494589, 10.6007524912, 10.4345619032, 10.7945867713, 9.6207666235, 10.6245916419, 10.3916029065, 10.5102951881, 12.4489508668, 13.0957821839, 12.9863866768, 12.8507708006, 12.009708159, 12.9271800089, 13.5503950307, 9.542575784, 10.0709373868, 9.2492746271, 11.6133876949, 12.1294731331, 12.1413829371, 12.3313477653, 12.6932630134, 7.4699502174, 7.3788693753, 7.7914797134, 7.7404059177, 8.8899050228, 9.1191643373, 11.1993740037, 10.9918400334, 9.0035750827, 9.3678485223, 11.5992503548, 11.5033172573, 12.1129363655, 11.7500678703, 12.1336116101, 12.2030955889, 11.2336293258, 12.1099692867, 12.0929290744, 11.2984361994, 13.8911044935, 13.6015815835, 13.3308657816, 14.5529292518, 8.3411478708, 8.5723301003, 8.5200100698, 9.1118025623, 8.3889087492, 8.3802859645, 11.4680514704, 10.7229084598, 9.4186438517, 10.2216787502, 9.7177226946, 10.1854639876, 11.1058516418, 10.6841860962, 12.6612444419, 13.2454583102, 10.2992507106, 10.122693629, 11.1312166972, 12.4827465892, 12.820365545, 13.0050171414, 13.0249723366, 13.4768567648, 7.474661904, 10.1689361654, 10.4344473432, 10.0262170543, 10.5014770967, 10.4757933008, 10.923467015, 9.2193047868, 8.3801676595, 11.1656952714, 10.9625010843, 10.0496402332, 10.2815997539, 11.4433272669, 10.8682104224, 10.7724971751, 10.1808497643, 9.0462511513, 9.0655487715, 9.3382356791, 11.5245850431, 11.3528158184, 11.0716142257, 12.4936273501, 9.7937776824, 9.3988692649, 9.8255619272, 10.6573224171, 9.5500161319, 12.7112526737, 12.0906078668, 12.6367815468, 8.8071688799, 9.5797702638, 9.7115314996, 9.1398403811, 10.6837954926, 9.4080822884, 12.0259470911, 11.2811210334, 11.5574987683, 13.1553866909, 14.2565071116, 13.0363825707, 13.5665097405, 13.8632881317, 13.2251201579, 13.1266908866, 12.0825975745, 11.9119190114, 13.7800567337, 14.601885272, 14.4205435226, 14.4908832785, 12.7507078342, 14.2276513989, 7.6300039631, 6.9489759376, 9.2733256303, 9.4097817033, 9.6110424338, 10.1477020867, 9.5351383297, 10.4021053295, 10.4753911447, 11.9378185996, 11.8382830174, 10.8028266274, 12.2309105985, 12.2589922027, 12.6150736437, 12.9595615118, 11.3849447076, 11.9096720032, 12.5123336011, 12.1558097725, 11.8179367644, 11.9459902252, 12.2415538292, 14.9014669189, 10.9671676541, 10.3762957624, 13.3845795095, 13.7082650046, 13.2015043618, 13.2377194187, 12.890210083, 14.258609985, 7.5460237933, 7.6374012505, 6.9545327196, 9.423162664, 8.6185981162, 8.5698328528, 9.6513504835, 11.1728469936, 10.5212005643, 10.2441736835, 9.7263347439, 10.4987126703, 10.3082255208, 13.8543502531, 13.1247752606, 12.3733515469, 17.4476656565, 18.5403444948, 17.709064527, 17.8049555402, 19.2216097464, 17.8701005763, 18.0611009882, 20.9432346479, 8.5717537093, 8.1488496942, 7.9709645371, 8.6859074164, 9.0027202272, 8.6175677442, 8.8932311444, 10.3463899673, 10.0390733355, 9.6954430921, 9.7460236117, 11.363501134, 12.4180356539, 11.5559352184, 11.8066424827, 12.8815101878, 10.7109448076, 10.6061375944, 11.6137948573, 10.6614339835, 11.9305002254, 10.793195183, 10.7945867857, 12.6801848385, 8.1102449533, 7.5497897418, 7.2892553974, 7.1849750291, 8.4566149305, 8.0341981165, 8.4045716166, 8.8716682726, 7.5247598301, 7.8868835917, 8.2926123132, 8.2742815563, 8.6338602648, 11.5439863597, 10.881575988, 10.341516286, 11.1910524725, 13.5462584311, 12.4621480131, 13.137035494, 12.8498009967, 14.2739212399, 12.4536318319, 14.2675450322, 9.2044322876, 9.3709857261, 9.3471580455, 8.8745368031, 9.4189212376, 9.2736472774, 9.3127447056, 11.2472722016, 9.0095949047, 9.3485046459, 9.0803871841, 10.4228914056, 10.2775854291, 8.8722921694, 12.3245720272, 11.8720638126, 8.4017040078, 12.098742374, 10.9283393251, 10.4119922439, 11.6382952134, 11.4757203321, 12.1926184582, 11.000285247, 8.2527260443, 7.5341710383, 10.2570640982, 9.7022498139, 9.8829885518, 9.707445377, 10.6044402121, 10.7426377631, 11.3298930688, 10.592227618, 10.4247596272, 10.4874634959, 11.7161449996, 13.0624039953, 13.2374144257, 13.0645772134, 11.0452484477, 11.6058876892, 13.0583685748, 13.7440918765, 13.3558355882, 13.8030022866, 12.37479742, 13.8289332088, 11.2316108004, 12.6526834313, 12.1567363437, 12.528302795, 14.6876374917, 14.5461672451, 14.4045932015, 15.198240691, 12.0728277496, 11.8475569224, 12.5150394411, 12.0717346433, 14.7011089502, 14.7013586684, 14.7424573382, 13.9860608785, 7.5888504235, 7.7885769766, 7.7876617358, 7.3890975729, 7.1856611323, 7.9359375036, 10.1018725819, 10.9693967419, 9.2823640443, 8.8622946874, 8.2834302777, 7.8578418728, 9.3981903112, 9.5010716991, 10.935414015, 11.4088966714, 13.2980650476, 13.4890716769, 14.4085842864, 15.9414729289, 17.0947773723, 15.0798092435, 16.7375127696, 16.169586469, 9.1312913337, 10.400890607, 10.5080457941, 9.2830068699, 10.3484545108, 12.3451690944, 12.7505861549, 12.6361523689, 6.508632089, 7.6168981351, 7.2545425229, 7.9770219341, 7.6268941369, 7.1093411714, 9.5439092185, 10.1981553597, 8.2862165493, 7.2120389808, 7.619896357, 7.7180448005, 7.7295133762, 7.2519150738, 9.7780224176, 9.7067022304, 10.0709050042, 10.3906370144, 11.2104226589, 12.1657800386, 12.4845343753, 12.1438569492, 12.5833407899, 12.731464807, 10.0739667725, 12.3035480617, 14.50205687, 12.3939681134, 14.2622120616, 13.5900569378, 13.407254759, 14.836677512, 9.3768728881, 8.8470625423, 8.8219134909, 8.9438737069, 9.092867233, 11.1205593004, 12.0028158108, 12.5016448989, 8.4956990534, 8.9482710706, 8.9803275717, 10.2934058878, 8.0960868788, 9.2654127859, 9.7728895514, 10.6438808722, 10.4565524579, 11.6348758013, 9.9777230135, 10.9046614877, 9.1423742993, 10.4894072045, 10.7060126918, 12.0482245459, 6.8242342649, 9.9744185607, 10.233274307, 9.2153011583, 9.2717909031, 9.7311312532, 10.0800317184, 9.7253236336, 8.4896423646, 8.7925129129, 8.4059885171, 9.5466404813, 8.7723806581, 9.1134985177, 11.7202850425, 11.5169893496, 11.2531233151, 12.7560229947, 12.963342814, 13.687929339, 13.7143658193, 13.2798737443, 13.2689280698, 14.3409750456, 6.7658927364, 7.9821240848, 6.7920553513, 9.7391153712, 9.6163964524, 9.5325959127, 9.7797972131, 9.7364341223, 7.6703238856, 8.630619538, 7.5407536247, 7.0249692547, 6.9925464151, 7.454146981, 8.7257920752, 10.3921790304, 13.6012917166, 11.8054353218, 12.5540538567, 13.738166381, 12.9628783945, 14.6715433797, 15.1427853066, 14.4960074139, 13.9137496103, 13.9099705624, 14.0864091443, 14.5536568933, 14.9733135094, 16.7661306276, 17.5391080031, 16.7323326738, 12.3338189684, 13.1757737467, 12.2409904665, 12.8907375407, 12.0448543906, 15.8330565185, 14.9086099193, 14.3925227578, 9.5542811295, 9.9660897766, 9.0687981483, 9.1208575917, 9.4465056861, 10.244824578, 9.5987129886, 11.6393790822, 9.8970127681, 9.9660867929, 9.5642547011, 10.4409209912, 10.8559958004, 10.4608389815, 11.9689623245, 11.6394155425, 11.0716857483, 11.3107873146, 14.2356054735, 12.7074153524, 13.1533987607, 13.7959935446, 12.9515175877, 13.7755847668, 8.7132195837, 7.5350894959, 8.3510773805, 8.8530257852, 8.621010229, 9.9432408337, 10.2249214267, 11.4531530822, 8.1327320548, 9.7701452608, 8.4520597598, 8.6753923837, 7.6112191491, 8.7590617414, 8.9905584282, 12.036071506, 11.7692873769, 11.6723694585, 11.0904814103, 12.03296018, 12.0646498998, 12.0348422839, 12.3062718013, 14.0857143622, 11.7769287771, 12.4131262112, 10.9454414835, 12.7368896008, 12.0373026772, 12.4233068361, 12.5077569857, 14.3232792544, 12.7521817122, 13.5183295596, 13.996796268, 13.4205901374, 14.0827171335, 14.3427041876, 14.5582433314, 14.2878079636, 8.8817183417, 9.4604234389, 8.5673784665, 8.8215289818, 8.3334812048, 9.2311018386, 9.5236579884, 9.6533972988, 7.7896438393, 7.2261206815, 8.2805884965, 8.1441585091, 8.0250352718, 7.4123233991, 8.1663403669, 8.2725435929, 12.7005973272, 12.3081571208, 13.9059776651, 12.818388699, 12.9964268307, 12.6721731996, 12.4834014207, 13.1197206801, 10.3806913727, 8.8334307757, 9.1420856043, 9.1645441394, 9.2940057317, 9.7354015046, 8.4995502574, 10.3115475469, 7.8376952553, 9.0939867133, 9.6351933767, 8.9146358225, 8.8512195418, 9.3286624494, 9.3884591647, 8.9874076071, 9.618406407, 8.9251712954, 9.3652295359, 8.8564633578, 9.7217172209, 9.5904230272, 10.2002201536, 9.9345128157, 9.8321983822, 8.4266910512, 8.817496357, 8.1175857287, 8.8197635969, 9.2008604842, 8.7843785242, 9.1165094006, 7.9107723906, 7.166557988, 7.6971912227, 7.6498867069, 8.7996051945, 8.2471026207, 9.5242443534, 8.0015954777, 5.966256739, 6.1914846923, 7.0114788075, 7.6078214041, 7.1857487591, 8.5001763504, 7.7132567336, 7.781275809, 6.0111158291, 5.9262928133, 6.3769655562, 7.1940520581, 6.5001901007, 6.6286710954, 7.3698276613, 6.7299074094, 5.8727969606, 5.4049466504, 6.2352622349, 6.7322349346, 5.6874201731, 6.4036767885, 5.2441502486, 6.3705496103, 8.2847091369, 8.8521757228, 9.2511683441, 9.5299245258, 10.1027674463, 10.0671539134, 10.8621519849, 11.0767019136, 9.1027707049, 9.8272996592, 9.6684626514, 9.8500469472, 10.4024861135, 8.8579142484, 10.0927256357, 10.2707285224, 8.8171221344, 9.4673747986, 9.8499573074, 8.3854638554, 8.8256038745, 9.642970574, 9.170595058, 9.3160565791, 8.729755057, 9.0175475021, 9.3742883585, 9.4658006562, 9.8150752456, 10.138991095, 9.3174808702, 10.2222015243, 9.4391333095, 9.2733690145, 9.3721835893, 9.6580798541, 9.3434130624, 8.6487934886, 9.8688825802, 10.3571223683, 10.0348684664, 10.1289673696, 9.7886324959, 9.8603076333, 10.3641054808, 9.8685762821, 10.3337126105, 9.7727869504, 11.325719469, 12.4238053372, 12.4818586863, 11.7497520121, 13.2022229694, 12.1805099235, 11.8435715337, 11.5003279141, 9.2679340118, 8.8084346259, 8.8515630678, 9.9800289552, 9.5647364795, 9.7754623187, 9.2321653048, 9.6175698206, 9.5810218339, 8.17967847, 8.8839537814, 8.9983331752, 8.850293141, 9.2464213438, 10.0072680179, 9.0840019198, 9.9906757609, 10.040372757, 10.6243089199, 10.685562403, 10.2747352706, 9.8936568588, 10.1673051273, 10.4963919605, 9.7263422335, 10.5600449231, 9.7183246183, 9.2155909526, 10.1679675934, 11.4368157282, 10.2152008394, 10.1309736317, 10.9987202055, 11.1687845446, 11.2983765909, 10.8629650869, 11.1821924033, 11.3857416717, 12.0754712982, 11.9140782257, 15.4527487727, 14.4348190536, 16.3027521074, 15.1292867787, 15.4393752817, 16.0703701219, 15.9142288765, 16.3933616118, 10.9508544486, 10.9206593707, 10.3162421462, 12.7374569567, 11.4111214247, 12.3059676515, 12.0663168289, 12.4785829959, 12.1413221747, 11.2874261559, 11.923710804, 13.1241047774, 12.1417220074, 13.6163705378, 12.4785655954, 12.9392875848, 5.5385864605, 8.0974677949, 6.9531565912, 6.7201910438, 7.4688579146, 7.4526637194, 7.2633457968, 6.9674235248, 11.0330485635, 12.0345752169, 12.1589160074, 12.8231565502, 11.7425678319, 13.2579509889, 12.2140236653, 11.9678521094, 12.8154109494, 11.6763457601, 12.5150945863, 12.5846783794, 13.15089915, 12.9839298985, 13.1499389023, 12.5434229423, 8.6509866632, 7.8394770612, 8.3938985655, 8.3402988077, 8.5049624024, 8.4729444277, 8.6153364576, 8.8648541123, 12.7157369588, 13.1669597923, 12.8849130112, 13.3546467882, 13.8784726375, 13.0672519896, 13.8206267866, 13.7943046579, 12.0027437322, 13.1275422185, 13.0593972777, 12.7589357934, 12.4792445041, 14.1415683343, 13.5031880999, 12.5271348606, 10.4089167278, 10.8906258104, 11.8034636308, 11.1891736279, 10.1805658396, 11.37133456, 11.6048824328, 11.0539347525, 12.1135559119, 12.8993700077, 11.538301117, 12.8324196835, 12.1209462423, 12.8554846761, 13.3964024351, 12.4281911242, 14.0530913907, 13.0878328245, 13.7337669592, 12.9191718395, 13.75817031, 13.6774334008, 12.9705458813, 13.1838111189, 15.6971486813, 15.4338280808, 15.7931536004, 15.7193235274, 16.4571396286, 15.746486845, 15.7488968099, 15.7566751845, 13.9683650605, 14.9291036055, 14.900721297, 15.5137655181, 14.5431837584, 15.7146548679, 15.1634535869, 14.6928183312, 12.493510443, 12.8925631193, 12.5210660856, 13.1624268649, 12.274531337, 12.0722939492, 12.7794564439, 12.5597300058, 12.7696135256, 12.29221743, 13.4690074729, 12.9862407435, 13.5204110012, 13.4058444831, 12.8324497954, 12.3004057102] + }, + "params": { + "pattern": "joiners_only", + "n_groups": 80, + "n_periods": 8, + "seed": 106, + "effects": 3, + "placebo": 3, + "ci_level": 95 + }, + "results": { + "effects": { + "1": { + "overall_att": 2.1035965327, + "overall_se": 0.087723216256, + "overall_ci_lo": 1.9316621882, + "overall_ci_hi": 2.2755308771, + "n_switchers": 494 + }, + "2": { + "overall_att": 2.0875801517, + "overall_se": 0.088459799998, + "overall_ci_lo": 1.9142021297, + "overall_ci_hi": 2.2609581738, + "n_switchers": 389 + }, + "3": { + "overall_att": 1.9129532359, + "overall_se": 0.10816711662, + "overall_ci_lo": 1.700949583, + "overall_ci_hi": 2.1249568887, + "n_switchers": 294 + } + }, + "placebos": { + "1": { + "effect": 0.12322439539, + "se": 0.10196418894, + "ci_lo": -0.076621742655, + "ci_hi": 0.32307053343 + }, + "2": { + "effect": 0.031729013923, + "se": 0.13306878057, + "ci_lo": -0.22908100347, + "ci_hi": 0.29253903132 + }, + "3": { + "effect": 0.049302840386, + "se": 0.19392488902, + "ci_lo": -0.3307829578, + "ci_hi": 0.42938863857 + } + } + } + }, + "leavers_only_multi_horizon": { + "data": { + "group": [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, 17, 17, 17, 17, 17, 17, 17, 17, 18, 18, 18, 18, 18, 18, 18, 18, 19, 19, 19, 19, 19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 20, 20, 21, 21, 21, 21, 21, 21, 21, 21, 22, 22, 22, 22, 22, 22, 22, 22, 23, 23, 23, 23, 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 25, 25, 25, 25, 25, 25, 25, 25, 26, 26, 26, 26, 26, 26, 26, 26, 27, 27, 27, 27, 27, 27, 27, 27, 28, 28, 28, 28, 28, 28, 28, 28, 29, 29, 29, 29, 29, 29, 29, 29, 30, 30, 30, 30, 30, 30, 30, 30, 31, 31, 31, 31, 31, 31, 31, 31, 32, 32, 32, 32, 32, 32, 32, 32, 33, 33, 33, 33, 33, 33, 33, 33, 34, 34, 34, 34, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36, 36, 36, 36, 37, 37, 37, 37, 37, 37, 37, 37, 38, 38, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39, 40, 40, 40, 40, 40, 40, 40, 40, 41, 41, 41, 41, 41, 41, 41, 41, 42, 42, 42, 42, 42, 42, 42, 42, 43, 43, 43, 43, 43, 43, 43, 43, 44, 44, 44, 44, 44, 44, 44, 44, 45, 45, 45, 45, 45, 45, 45, 45, 46, 46, 46, 46, 46, 46, 46, 46, 47, 47, 47, 47, 47, 47, 47, 47, 48, 48, 48, 48, 48, 48, 48, 48, 49, 49, 49, 49, 49, 49, 49, 49, 50, 50, 50, 50, 50, 50, 50, 50, 51, 51, 51, 51, 51, 51, 51, 51, 52, 52, 52, 52, 52, 52, 52, 52, 53, 53, 53, 53, 53, 53, 53, 53, 54, 54, 54, 54, 54, 54, 54, 54, 55, 55, 55, 55, 55, 55, 55, 55, 56, 56, 56, 56, 56, 56, 56, 56, 57, 57, 57, 57, 57, 57, 57, 57, 58, 58, 58, 58, 58, 58, 58, 58, 59, 59, 59, 59, 59, 59, 59, 59, 60, 60, 60, 60, 60, 60, 60, 60, 61, 61, 61, 61, 61, 61, 61, 61, 62, 62, 62, 62, 62, 62, 62, 62, 63, 63, 63, 63, 63, 63, 63, 63, 64, 64, 64, 64, 64, 64, 64, 64, 65, 65, 65, 65, 65, 65, 65, 65, 66, 66, 66, 66, 66, 66, 66, 66, 67, 67, 67, 67, 67, 67, 67, 67, 68, 68, 68, 68, 68, 68, 68, 68, 69, 69, 69, 69, 69, 69, 69, 69, 70, 70, 70, 70, 70, 70, 70, 70, 71, 71, 71, 71, 71, 71, 71, 71, 72, 72, 72, 72, 72, 72, 72, 72, 73, 73, 73, 73, 73, 73, 73, 73, 74, 74, 74, 74, 74, 74, 74, 74, 75, 75, 75, 75, 75, 75, 75, 75, 76, 76, 76, 76, 76, 76, 76, 76, 77, 77, 77, 77, 77, 77, 77, 77, 78, 78, 78, 78, 78, 78, 78, 78, 79, 79, 79, 79, 79, 79, 79, 79, 80, 80, 80, 80, 80, 80, 80, 80, 81, 81, 81, 81, 81, 81, 81, 81, 82, 82, 82, 82, 82, 82, 82, 82, 83, 83, 83, 83, 83, 83, 83, 83, 84, 84, 84, 84, 84, 84, 84, 84, 85, 85, 85, 85, 85, 85, 85, 85, 86, 86, 86, 86, 86, 86, 86, 86, 87, 87, 87, 87, 87, 87, 87, 87, 88, 88, 88, 88, 88, 88, 88, 88, 89, 89, 89, 89, 89, 89, 89, 89, 90, 90, 90, 90, 90, 90, 90, 90, 91, 91, 91, 91, 91, 91, 91, 91, 92, 92, 92, 92, 92, 92, 92, 92, 93, 93, 93, 93, 93, 93, 93, 93, 94, 94, 94, 94, 94, 94, 94, 94, 95, 95, 95, 95, 95, 95, 95, 95, 96, 96, 96, 96, 96, 96, 96, 96, 97, 97, 97, 97, 97, 97, 97, 97, 98, 98, 98, 98, 98, 98, 98, 98, 99, 99, 99, 99, 99, 99, 99, 99, 100, 100, 100, 100, 100, 100, 100, 100, 101, 101, 101, 101, 101, 101, 101, 101, 102, 102, 102, 102, 102, 102, 102, 102, 103, 103, 103, 103, 103, 103, 103, 103, 104, 104, 104, 104, 104, 104, 104, 104, 105, 105, 105, 105, 105, 105, 105, 105, 106, 106, 106, 106, 106, 106, 106, 106, 107, 107, 107, 107, 107, 107, 107, 107, 108, 108, 108, 108, 108, 108, 108, 108, 109, 109, 109, 109, 109, 109, 109, 109, 110, 110, 110, 110, 110, 110, 110, 110, 111, 111, 111, 111, 111, 111, 111, 111, 112, 112, 112, 112, 112, 112, 112, 112, 113, 113, 113, 113, 113, 113, 113, 113, 114, 114, 114, 114, 114, 114, 114, 114, 115, 115, 115, 115, 115, 115, 115, 115, 116, 116, 116, 116, 116, 116, 116, 116, 117, 117, 117, 117, 117, 117, 117, 117, 118, 118, 118, 118, 118, 118, 118, 118, 119, 119, 119, 119, 119, 119, 119, 119], + "period": [0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7], + "treatment": [1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + "outcome": [13.9207082441, 13.6244938924, 13.8473901399, 12.264770601, 11.5484474533, 11.812291393, 11.8742234217, 11.585760886, 12.0343724809, 11.6194581773, 10.767233546, 10.4173298084, 10.3778510343, 10.4442165631, 10.1795469011, 11.0819123162, 13.0139172896, 13.2097292322, 13.6789504379, 12.2251405863, 11.1487221097, 11.8660353418, 11.5542687989, 11.7360394699, 11.6163299785, 11.9518090134, 13.9348608559, 12.1785055049, 12.3378520439, 12.1057694433, 11.3457482249, 10.6069952262, 13.8821376771, 14.4216451587, 11.5078581554, 11.2345160065, 11.3409352077, 12.3109699392, 11.8125303928, 12.6006578629, 13.5054175335, 14.9923726768, 14.1675309899, 12.5061959583, 12.7662746654, 12.5378425526, 12.3820660408, 12.7753669707, 11.3084376008, 11.5837543879, 11.209700883, 12.0587996818, 12.3291398308, 9.8441564636, 9.8338570152, 9.8798365277, 12.3366226234, 12.5775659671, 12.5175636641, 9.8044062407, 10.2454085447, 10.1859051327, 10.4936196686, 10.4804943827, 9.1499451714, 8.2358142123, 8.5465021655, 6.7833418387, 6.6022960448, 7.1153080464, 7.4797535797, 6.602131343, 11.0331009501, 11.7871336594, 12.1410575745, 11.9498666008, 12.0435034201, 12.7629197793, 10.8049759071, 9.743535931, 13.4378872613, 12.8214637797, 12.4614940608, 12.9396277917, 13.1242754221, 14.0527864467, 11.8063060476, 11.1742900449, 7.1336908114, 7.0198850578, 8.8516422903, 5.4956198468, 6.4381894176, 5.7729046094, 6.817963587, 6.5034437502, 12.0366208231, 14.4305944084, 12.0029127548, 10.4689027176, 11.7708203964, 11.3300354072, 11.180335923, 11.3198007405, 9.2800470028, 10.6691931757, 10.2026262038, 8.8907728254, 8.2571529797, 7.8863775283, 8.6180212889, 9.0782064479, 10.7813839934, 10.0422801939, 9.7957696111, 8.3096792721, 8.9811794889, 8.5309340443, 9.1663032915, 9.89168492, 12.7779214879, 13.6177300121, 14.4832773093, 13.8660689328, 11.6031767264, 12.0150923311, 11.5957779973, 12.3645550029, 11.3687151544, 12.5174252934, 12.0699224812, 12.0678500583, 10.4488282235, 10.2237232803, 9.2877259594, 10.447360705, 14.7438692039, 15.040867522, 15.0196556021, 15.3845852646, 12.8967294652, 13.0729365195, 13.0977335765, 13.6046530712, 12.6217729665, 12.2021717784, 12.4355109281, 12.5427566447, 13.5769093828, 10.2626359094, 11.2935445497, 11.6524754753, 12.3218472557, 10.8155221514, 11.0284350561, 11.572624764, 11.3652485416, 11.6319383194, 11.8063079367, 12.383477098, 9.9806053022, 10.5569555406, 10.6918956378, 10.766729381, 11.5678851085, 9.3499284045, 9.4064003823, 8.5641582189, 13.6508603631, 14.0335435375, 11.188781456, 12.7731466717, 12.8640926417, 12.5541046407, 12.8507106693, 12.9131295065, 16.6778680344, 16.0628864892, 16.6906633156, 17.076625063, 16.5229273703, 16.4770844086, 14.7599845174, 15.2203961385, 10.5823420279, 10.7286878328, 11.1067829409, 11.5088364778, 11.2317263806, 10.9841757311, 11.6193261143, 9.2729282449, 12.9797483501, 11.1938204323, 11.2497126211, 11.5849860443, 10.9946876869, 11.7003466896, 11.5358511911, 11.8390077445, 13.01431631, 13.1328430994, 12.9902150673, 13.2636236306, 13.1294742315, 13.8902841149, 14.1351507113, 11.4321786998, 12.8006548428, 13.8019601237, 11.1004502843, 11.5591579952, 11.8402863632, 11.8214813894, 11.7331874887, 11.3905197172, 5.7303994244, 4.841098034, 4.7040617726, 5.4543668144, 2.9991416565, 4.1165945206, 3.6624105295, 4.2452439745, 8.3113270333, 6.5992122946, 6.695783593, 6.8042549994, 7.0024966487, 7.5157321619, 8.0034410648, 8.1192391919, 11.7358861978, 12.9150059378, 12.010550762, 11.5892572158, 12.0901664349, 11.9779061406, 10.6698497202, 10.5049629196, 11.8162696086, 12.4266585335, 12.5874331486, 12.1563212355, 12.729772458, 10.9734750347, 10.9446526739, 11.4072737012, 15.4355110651, 15.3850808274, 13.082440941, 12.2527381739, 13.3269632078, 12.6588468549, 12.8733922876, 13.7649234677, 11.1984752336, 11.4466325366, 9.1994949803, 10.1441343241, 10.110823789, 9.2339567146, 10.422483553, 10.4904006465, 9.2589381914, 8.9238590108, 9.2531586825, 8.3249527199, 9.0697719013, 8.3655124053, 7.5375975192, 8.3400775963, 12.8099859686, 10.5766920424, 11.0025518743, 11.5890199533, 11.5630892225, 12.3756467355, 12.3332269087, 11.2793809602, 16.3153419013, 14.684919811, 14.5973537112, 15.4218998243, 16.1544925691, 14.2066691056, 13.3986647061, 13.4813495521, 12.8912564467, 12.9853932527, 14.3093219304, 14.4136766837, 13.810490137, 13.801515328, 11.5567833153, 12.2989392813, 10.2989216559, 11.3110047762, 11.3145584804, 11.4152976415, 12.0789697068, 11.6536565864, 11.0871967162, 10.1126907461, 11.3585579407, 12.2870524195, 11.1639017287, 10.9088543273, 9.5838100139, 9.9171437699, 10.8263306736, 10.9108485971, 14.1233797228, 11.506364196, 11.6223644542, 12.2492173167, 12.0692493535, 12.5106827978, 13.2524631175, 12.2893688365, 12.8918938924, 12.9619903606, 13.9560102919, 13.3436603846, 12.933107467, 12.7189521091, 11.250460041, 11.9490580162, 12.1840741399, 12.8907907287, 13.8197884108, 13.9470394743, 11.038295286, 11.7153072171, 11.878644716, 12.1487235428, 9.5514152474, 10.0541225297, 9.5532642163, 9.3279398554, 7.6105851415, 8.2899991553, 7.4461622192, 8.6648544488, 13.8109840029, 11.5198336809, 11.0206748233, 11.3031110445, 11.8898416853, 11.9674799981, 12.7305575989, 12.3177314127, 6.4550478035, 6.5609075851, 6.4826641695, 7.7625447243, 6.9868698961, 7.1556265538, 6.494944172, 5.2663731925, 8.4983569232, 6.8995488204, 8.5435722377, 8.1041666495, 9.2673880367, 8.5745303482, 6.2097373113, 6.6686751948, 10.924745104, 10.858369103, 11.4955292412, 9.8587288456, 10.3726597199, 10.2142320456, 8.8530126442, 10.645342732, 11.9448476067, 12.5486074128, 12.463487324, 12.5407215951, 12.7272175807, 11.8918084224, 11.2714156706, 10.6293495493, 14.8776644896, 12.6807912127, 13.398690233, 12.9280371383, 13.6260240124, 13.9130648001, 13.4475968289, 14.4794652183, 9.4914371306, 10.2208560748, 11.3595849609, 10.5001198977, 8.298771163, 7.738634524, 8.8537419789, 8.7480937253, 15.7044300713, 15.9221348637, 15.9087616423, 15.1678938762, 16.2217661843, 16.2120934816, 16.5408085091, 13.9235469451, 13.625158206, 13.8170573999, 13.9370291011, 14.3134366776, 11.6558291641, 11.8008756114, 11.5801395538, 12.7654860517, 16.1689525513, 15.8943619195, 14.8401967452, 12.9973608467, 13.8193946821, 14.3184087726, 13.8073987543, 13.471197526, 13.9498578441, 14.6145179281, 13.9768083003, 14.5934869334, 14.2152489321, 14.6570697279, 11.7851157372, 12.1134736954, 13.2278815502, 12.576795528, 13.0736154948, 13.0939850801, 13.4394045336, 12.9032645266, 12.7493267028, 10.1783636906, 13.1289613236, 12.2385003448, 14.0109506865, 13.0521178721, 13.0983550625, 13.7771195625, 12.9608873687, 12.1016958675, 13.0838952609, 13.1322701009, 12.408024739, 11.9583396482, 12.9852497248, 10.8463116253, 11.8431614051, 12.2298685562, 10.5809656859, 8.6002663171, 10.1826386984, 8.7873786383, 9.7149691764, 10.0493231948, 8.750578506, 10.3324456337, 11.7438058616, 12.1652841966, 11.5057234725, 11.4586549641, 12.4241048254, 11.2972504217, 12.8447869287, 10.0454483857, 11.6764550338, 10.6180897519, 11.7824399133, 10.9088130728, 9.0666813527, 8.9605163553, 9.9082976063, 9.272375595, 12.7624019317, 13.2424378224, 14.3336726388, 13.5913110973, 13.6268266404, 11.4863929641, 11.6101878493, 12.2227848316, 12.0918909779, 12.3431430274, 12.2690723745, 11.5450194533, 10.567426252, 10.328097644, 10.9916702467, 10.6969403645, 9.9815027284, 9.807367831, 10.085702992, 9.3316767482, 10.8361397489, 9.6423805571, 10.1568210989, 8.5396731611, 11.7710640375, 11.1647889074, 11.0485518745, 12.2863241549, 11.6824756885, 10.0752773884, 10.3152062513, 9.802546339, 14.2199928052, 14.0280590975, 13.5675184167, 13.8190299105, 12.0977656891, 12.5933096194, 12.6336800859, 13.5691634072, 9.9307601881, 11.4724549524, 10.5738457654, 11.4367083214, 10.6047528771, 8.7041382369, 9.1550044886, 9.5905425744, 11.17830371, 11.2641214302, 10.6986289178, 10.9547454098, 10.6763798306, 9.4404386552, 9.2446480762, 8.6947761752, 15.1965345603, 14.347555208, 14.2276057915, 14.8871633902, 15.7784241723, 15.3143233539, 12.6769303888, 13.7479971629, 14.2353086754, 13.4595883532, 14.1130185132, 14.2489043785, 12.0964202074, 11.2404884881, 11.6484810665, 11.8442756893, 12.6559655454, 12.6825129949, 11.6225913455, 12.1730889539, 12.5128424636, 13.085114084, 9.7396850351, 10.8152192335, 11.2596863138, 10.1236969, 9.7517523246, 10.5468475228, 9.9636056792, 9.4696235076, 10.930829888, 10.3353608067, 13.2311517553, 13.5139998594, 12.4354655306, 13.4023519213, 13.9965923163, 11.6467407453, 12.3597545379, 11.1311673266, 9.3547578061, 10.7156421158, 9.6362654337, 7.450671253, 8.0741831507, 8.5920099786, 8.8579082024, 8.9339351843, 14.4266044436, 13.4021711956, 12.8710419472, 13.7789081777, 13.3010007318, 13.7562910121, 13.4367233292, 12.9889143839, 9.9720844094, 9.680304803, 9.947816388, 7.6393059145, 8.4567160271, 9.4302869457, 8.2241922944, 7.7413749763, 14.0865104988, 13.1506717278, 13.1108756146, 14.4433258289, 12.0268218641, 12.4618140084, 12.5788769453, 12.5976904001, 11.7369855566, 12.3893502527, 11.4277356203, 12.2003830701, 12.2328622228, 10.5541986054, 9.7277342121, 10.4911753834, 11.459982752, 10.9918527666, 11.4953440696, 11.5845786819, 9.9989712959, 9.4726606828, 10.207873617, 9.7892181528, 16.857154038, 16.0905289465, 16.7918122969, 15.8514310209, 14.3635743797, 15.8441228146, 15.0933102664, 14.6131045895, 12.2126727299, 12.2773067913, 11.9156244459, 11.564277213, 11.1305860764, 11.4828006876, 11.1801742483, 11.3667869619, 10.4433516414, 10.9015847091, 10.5043507956, 11.2242542572, 10.9544596681, 10.8838993994, 11.0195457689, 11.6076938286, 10.4743526469, 10.1740132179, 9.9225463214, 11.1070180668, 10.1136330355, 10.5611021252, 10.5322293508, 11.3372189006, 8.8359126245, 7.9083010405, 8.2938727755, 8.0329181781, 8.4199587552, 9.1899223043, 9.1681116704, 8.3932617272, 10.1055158118, 9.6261538311, 9.890841114, 9.7909226735, 8.755720586, 11.2326378246, 10.3770688084, 10.6491526542, 10.4139262633, 11.715104525, 10.6144221066, 10.7768834947, 11.7045932285, 12.1999216435, 11.1286868636, 11.0150757738, 5.9579807754, 5.6908232599, 5.5448182551, 6.5683624394, 6.5260962507, 6.704828614, 6.6320064577, 6.6746103663, 11.4125475289, 11.1230376227, 11.03748179, 11.9129857391, 11.2413015718, 10.5458695699, 12.0417466157, 11.5107609948, 12.7976823485, 13.7493579771, 13.4736805716, 12.9755293123, 13.4778912313, 13.5031591926, 13.8851024532, 14.7285181222, 8.1276732144, 8.4445716252, 8.7651115298, 9.2055121431, 8.7604784569, 8.9540931271, 8.9520897004, 9.6957740506, 12.0223560023, 11.4661704402, 11.3494124319, 11.8020608423, 11.7256022558, 12.3331755935, 12.540086404, 12.1632845198, 9.1686468276, 10.7661741445, 9.6733635615, 10.0885119213, 9.7050250643, 10.0126601349, 9.591092752, 9.9032019096, 13.6423269586, 12.9002802006, 12.7512324146, 13.201845377, 13.1701336453, 12.2241450594, 13.9606098733, 14.2063440576, 9.5821287539, 10.5866373876, 9.0908487928, 11.2504454019, 10.7920062889, 10.4634628927, 11.2108714635, 11.0084344097, 10.0556236877, 9.6385237116, 10.1441544494, 9.2825507019, 9.3194386197, 9.4672185509, 10.1633346967, 9.8297136502, 8.5733683784, 9.851605672, 9.447942531, 9.220469545, 9.9887186888, 9.3761080512, 9.8444283285, 9.1177566087, 8.307094972, 8.8595333086, 8.6519566641, 9.3235725115, 9.0621010734, 8.2065259424, 9.6681752337, 9.4333613303, 11.5882969062, 11.9547693153, 12.307230783, 11.4900035373, 12.4010812427, 13.2668640678, 12.9826207031, 13.0831723331, 7.6279678099, 8.0781268185, 7.6453987869, 7.9547172368, 8.4818198046, 8.6805245641, 8.1944998168, 7.7666745938, 6.758324427, 5.8677998519, 6.2306837267, 6.263481196, 5.0728274321, 5.6760692078, 6.9919521558, 7.3365474447, 8.8670258482, 8.5270780739, 9.6541940252, 8.7838273964, 9.0351349814, 8.8058240768, 9.4640623053, 9.2351960831, 17.8714124817, 16.4595439623, 16.4554187713, 17.0797705586, 17.466783963, 16.9706660122, 17.5667072566, 16.8610308245, 14.4272162563, 12.8182092862, 13.4609953925, 14.3129928068, 14.2639701789, 14.1648313735, 13.7465112752, 14.8380351171, 10.4749369163, 9.404743831, 9.847958142, 10.3907285085, 10.2190122134, 10.659932965, 10.1409056639, 10.6991011384, 12.3122208566, 11.6230109805, 11.541036186, 12.0204622213, 11.0823349777, 11.246571054, 12.289613328, 12.3988334106, 14.1659223371, 13.9360891758, 14.2096219392, 14.2573702708, 14.4040329304, 15.2326873752, 14.631094057, 14.8046708004, 11.4398288468, 12.8226412949, 12.3187212727, 12.1509646548, 12.2591245984, 12.7571152735, 12.0722132875, 12.212992358, 6.779338836, 7.6634194659, 7.0446177271, 6.2488941823, 7.3228270276, 6.7913041296, 7.1927018562, 7.1686012369, 8.3088768114, 7.3564730226, 7.1999442678, 7.9584201265, 8.0901287706, 8.6666692948, 8.2295749092, 8.3833102593, 15.7200404606, 15.0255339303, 14.9596093621, 15.2119055729, 15.4067281774, 15.5819384608, 15.3604094151, 15.6488381081, 12.9828688132, 12.7826789256, 12.8091033639, 13.6862125649, 12.702462996, 14.4568963374, 13.0815886299, 14.4063152473, 11.7217486565, 10.5303250065, 10.7331507976, 9.7441652291, 11.841300761, 11.8294214684, 11.5262627962, 12.2886343036, 7.2787092506, 7.2835259627, 7.2349863154, 7.5040367754, 8.0861045462, 7.8135225669, 8.5615341874, 7.7491739285, 15.0119878975, 15.1765099018, 15.526013733, 15.3600743691, 15.5153883682, 15.3537299069, 15.9498566521, 16.6774930526, 12.9661173163, 12.6424666669, 12.302655663, 13.9129177487, 13.0941344624, 13.8386581095, 12.7374548259, 13.8484044754, 16.2179876777, 15.3572254543, 15.2724402858, 16.2775293066, 16.4786846708, 15.6330592941, 16.9418068798, 15.7921062493, 12.6601636831, 13.3505996364, 13.0229630667, 13.6399038912, 12.552056938, 14.0104560308, 13.0632956889, 12.7280115034, 11.046448445, 11.208996663, 12.1258462606, 11.0977397799, 11.4989482579, 11.7485761356, 11.964213685, 12.3249543606, 10.9486439479, 11.6749518099, 11.3451241518, 11.6602075507, 11.8922112963, 12.3637336486, 11.5906449794, 12.0396872442, 9.386012042, 9.3698251264, 9.5201558416, 10.0447039578, 9.6658890221, 10.5577969051, 9.8448087352, 10.8090005608, 12.210413358, 12.2982344707, 11.8667112662, 11.8709764476, 12.4667526904, 12.7536941147, 12.3699753791, 12.8645801877] + }, + "params": { + "pattern": "leavers_only", + "n_groups": 80, + "n_periods": 8, + "seed": 107, + "effects": 3, + "placebo": 3, + "ci_level": 95 + }, + "results": { + "effects": { + "1": { + "overall_att": 2.0527136548, + "overall_se": 0.087033771291, + "overall_ci_lo": 1.8821305976, + "overall_ci_hi": 2.223296712, + "n_switchers": 465 + }, + "2": { + "overall_att": 1.9491551822, + "overall_se": 0.10140296077, + "overall_ci_lo": 1.7504090312, + "overall_ci_hi": 2.1479013332, + "n_switchers": 366 + }, + "3": { + "overall_att": 1.9831800779, + "overall_se": 0.10598763937, + "overall_ci_lo": 1.7754481219, + "overall_ci_hi": 2.1909120339, + "n_switchers": 271 + } + }, + "placebos": { + "1": { + "effect": -0.050926022027, + "se": 0.095122602207, + "ci_lo": -0.23736289647, + "ci_hi": 0.13551085241 + }, + "2": { + "effect": 0.03028531753, + "se": 0.11394065125, + "ci_lo": -0.1930342553, + "ci_hi": 0.25360489036 + }, + "3": { + "effect": 0.034418423589, + "se": 0.15660316769, + "ci_lo": -0.27251814495, + "ci_hi": 0.34135499213 + } + } + } + }, + "mixed_single_switch_multi_horizon": { + "data": { + "group": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 34, 34, 34, 34, 34, 34, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 59, 59, 59, 59, 59, 59, 59, 59, 59, 59, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 63, 63, 63, 63, 63, 63, 63, 63, 63, 63, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 74, 74, 74, 74, 74, 74, 74, 74, 74, 74, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 78, 78, 78, 78, 78, 78, 78, 78, 78, 78, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 85, 85, 85, 85, 85, 85, 85, 85, 85, 85, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 87, 87, 87, 87, 87, 87, 87, 87, 87, 87, 88, 88, 88, 88, 88, 88, 88, 88, 88, 88, 89, 89, 89, 89, 89, 89, 89, 89, 89, 89, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 91, 91, 91, 91, 91, 91, 91, 91, 91, 91, 92, 92, 92, 92, 92, 92, 92, 92, 92, 92, 93, 93, 93, 93, 93, 93, 93, 93, 93, 93, 94, 94, 94, 94, 94, 94, 94, 94, 94, 94, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 96, 96, 96, 96, 96, 96, 96, 96, 96, 96, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 101, 101, 101, 101, 101, 101, 101, 101, 101, 101, 102, 102, 102, 102, 102, 102, 102, 102, 102, 102, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 104, 104, 104, 104, 104, 104, 104, 104, 104, 104, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 106, 106, 106, 106, 106, 106, 106, 106, 106, 106, 107, 107, 107, 107, 107, 107, 107, 107, 107, 107, 108, 108, 108, 108, 108, 108, 108, 108, 108, 108, 109, 109, 109, 109, 109, 109, 109, 109, 109, 109, 110, 110, 110, 110, 110, 110, 110, 110, 110, 110, 111, 111, 111, 111, 111, 111, 111, 111, 111, 111, 112, 112, 112, 112, 112, 112, 112, 112, 112, 112, 113, 113, 113, 113, 113, 113, 113, 113, 113, 113, 114, 114, 114, 114, 114, 114, 114, 114, 114, 114, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 117, 117, 117, 117, 117, 117, 117, 117, 117, 117, 118, 118, 118, 118, 118, 118, 118, 118, 118, 118, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119], + "period": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], + "treatment": [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + "outcome": [10.1520738028, 9.4412882513, 8.9341243351, 10.2380675373, 10.3162378128, 10.1994641025, 9.7458701792, 11.9413473329, 12.8935815906, 12.7859779251, 6.8951474742, 6.726855564, 6.5270351258, 5.811393174, 8.308590645, 8.3598349381, 10.061664497, 10.3250787499, 9.0733393238, 9.5136519467, 11.171505063, 10.9309801357, 11.6204703002, 11.6675676117, 11.0701445418, 12.3247029032, 12.02209437, 14.0828007157, 13.7824252101, 13.6812354645, 10.8643532432, 11.6560819726, 10.699483055, 11.6976751117, 11.4875129083, 11.580541603, 13.6079839952, 14.5003863698, 14.0151220821, 13.559602489, 7.4870945334, 7.3072426253, 7.7233904707, 8.0134051723, 7.839392432, 8.2176197302, 7.852517372, 10.3533242596, 11.1552614917, 10.2109294086, 10.0581061222, 9.9997332141, 10.6098393659, 9.8753144753, 10.149685964, 10.0981522776, 10.342064497, 12.9536662042, 11.8618029087, 12.4709317701, 9.8402745032, 10.347676998, 9.903403144, 10.695785822, 10.7017444127, 11.1299155578, 10.579617875, 10.7061308702, 11.1059934893, 12.3786068245, 11.1470344225, 10.6681913593, 11.5203229115, 10.4114604557, 11.1971575312, 11.4289864547, 12.9896283638, 13.3528478815, 13.1000799897, 13.4529474936, 6.8081385836, 7.0822339727, 6.6161889008, 6.3983757027, 6.5288282131, 6.4310091335, 9.7290478474, 7.9528480149, 8.9136861178, 9.0934953509, 9.4854881303, 8.9037467905, 8.4794319657, 8.9097114746, 10.1827283247, 11.5181847169, 11.4317528478, 11.4990046708, 12.2070733403, 11.0139165892, 11.6249922528, 12.5275129286, 11.3367217217, 12.8466419718, 13.0751502336, 12.9094266911, 12.6428430467, 12.1354297915, 14.2734890532, 15.4727182327, 10.1510278588, 10.1227070779, 10.2135928903, 10.8437312696, 10.2307010006, 13.2027213856, 12.5111322077, 12.6605328863, 12.3920266038, 13.4242536244, 13.0996821356, 12.3500162848, 13.2033218273, 12.3160202958, 13.3148814274, 13.4892608763, 15.4107422461, 16.784512364, 16.2198312662, 15.8259169622, 10.3680929444, 10.8332093217, 11.3702228375, 12.5119265553, 12.8124159911, 13.6702453482, 13.7390681017, 13.0719066745, 13.1406777228, 13.5001682097, 11.3734087477, 11.8084628475, 10.9070525181, 10.9872900925, 12.6121854105, 11.9441345292, 11.8698378623, 13.6929726727, 13.6236350311, 14.3083576941, 9.9649076117, 9.7320370472, 9.690231749, 12.4836588564, 12.0732623223, 13.2841493161, 11.71269535, 12.1984482637, 12.5443500863, 12.6543952381, 9.8196547412, 9.8780545817, 9.8579513813, 10.2189181159, 10.0228655005, 9.5114356293, 13.0026178476, 12.4533935744, 12.3668984499, 12.4975390955, 7.6819716119, 8.1170310829, 8.3382519526, 8.5266302995, 9.1117479815, 9.9729465375, 10.9091674811, 10.9623614055, 10.7770502801, 10.7237308434, 9.8106912854, 10.9866804639, 10.6390220415, 11.6202471351, 13.3705782503, 13.4294822639, 12.5895774873, 13.6865352474, 13.7268755096, 12.9949671674, 7.8060669955, 8.2080377075, 8.0808254568, 8.2293530587, 7.059822008, 10.0468636873, 10.5641105142, 10.4570804084, 10.7382458956, 9.460942029, 7.5483663177, 7.9731169809, 7.9060581695, 10.388025462, 10.2220836415, 10.1131060172, 10.1234122571, 10.2597829794, 10.7148884778, 10.6094408606, 10.6929772504, 13.3301242494, 12.9046421047, 11.8509756037, 13.0520552824, 13.2470197789, 13.2219369165, 13.1594079983, 13.2791132167, 13.6961262237, 9.7080997385, 10.6973660899, 11.1099128818, 13.6679096303, 13.2279506774, 12.9731185517, 14.2153139235, 11.6791420327, 13.1550977658, 13.030866988, 6.8701414105, 6.8374722086, 5.848361201, 5.8913084688, 5.6777642048, 6.6840471695, 6.5741130369, 6.1366271539, 9.2833456665, 8.7906093216, 13.0296675137, 13.8578893501, 13.3646493652, 13.5938602387, 14.0247309062, 15.9001032951, 16.7295651277, 15.3299804746, 16.1423910401, 15.7583946848, 9.428549285, 9.2202363432, 9.8418496238, 9.1925451459, 12.4231155495, 11.6980687264, 12.9099830419, 11.5251724906, 12.7478464611, 12.2826740533, 11.0699303767, 9.5152121549, 10.2360795614, 9.5775928817, 10.8843320949, 10.7703855804, 10.2242354986, 11.1367141052, 10.7312206588, 12.8818830439, 9.9194870916, 9.6302519434, 10.0329420948, 9.7032384264, 9.8373202871, 12.1554261264, 11.2008139021, 11.9612425992, 12.7421937021, 12.2691904494, 5.8730340404, 5.9013260892, 5.4537353064, 5.1361965898, 6.568414153, 6.3425569125, 6.470582176, 6.4336822359, 6.3361244183, 8.2571872202, 7.9200022251, 8.133679811, 8.2700313802, 11.0909869327, 9.6059377556, 9.5122215777, 10.5253947621, 10.0486395726, 10.3170196489, 11.1293288269, 9.7547622116, 10.944288987, 10.3924065677, 10.8174180615, 10.7523118476, 13.7042788023, 13.0381061217, 13.5589417475, 12.9999400341, 12.9927563896, 9.6585015391, 10.7342847819, 9.5407196003, 12.0516602706, 12.4152829722, 11.9366294154, 11.6289398065, 13.1352845187, 12.5750731436, 12.90632169, 9.9424475198, 10.2672419205, 12.0786425925, 10.4281544382, 10.1263654719, 9.9455234324, 10.4613493608, 10.7438690093, 11.1974977661, 13.1751158277, 11.1120698308, 11.1885040825, 11.2856150684, 11.154559045, 10.8550267421, 11.3300474535, 12.162450888, 11.8707409395, 12.4053604612, 13.2369119411, 9.8265808261, 10.4656982433, 10.0730244467, 10.582261515, 12.6712164871, 13.7214196493, 13.127224533, 13.2951353006, 13.7869220563, 13.6286235381, 11.3411455465, 10.2370529591, 12.2452853429, 12.0259536274, 10.4613297654, 10.5652695035, 11.2396895721, 11.7152654778, 11.2947008517, 13.7324682082, 9.5333520406, 10.6693954679, 10.5810532506, 11.2732254306, 10.1939199784, 13.1683540384, 12.2357301288, 12.8224500173, 13.3584162337, 13.1260995064, 9.7234898254, 9.3190035449, 10.0005103301, 12.0017354744, 12.2488072254, 12.1865059925, 12.8995771148, 13.2115472804, 12.836863335, 12.6580569813, 13.4054847246, 13.9952617232, 15.119965954, 14.85191208, 15.1358884735, 16.2199225093, 14.6085763756, 15.8623996018, 14.8692295059, 15.6292630171, 8.559669367, 8.2242180688, 8.6736685285, 8.1097168319, 8.4023104048, 9.5961318223, 8.4669905983, 8.7939210595, 9.828458184, 11.1857115849, 13.1205342124, 12.3912393003, 13.805427601, 13.4740559236, 13.520918215, 13.7767510208, 13.7995265829, 13.6313495928, 14.3254502156, 11.6760786381, 12.9233389218, 13.5236807296, 13.0968572587, 12.8653242571, 12.6490004844, 13.1296290841, 11.1063629014, 11.4230860523, 11.6588915016, 10.9409773326, 10.6579829427, 11.1787847785, 11.0753884884, 11.1504128107, 12.0176812615, 12.5997764003, 12.4187110821, 11.3538536969, 10.2713614362, 10.1866026562, 11.9777487599, 11.7852298741, 12.6656958839, 12.3995925643, 12.5101769685, 13.1200715172, 14.2228372305, 12.5070282157, 11.3153041258, 10.8079542459, 12.2587118972, 11.5147998674, 11.7292291833, 12.1881893036, 11.6721319697, 11.5248539121, 9.9467288733, 10.5886381967, 11.6873346733, 10.1949437343, 13.0622615239, 11.5502531935, 11.247512717, 11.2701929065, 11.6716777544, 10.6359585318, 11.814243984, 11.3630487388, 12.134582072, 11.8518893956, 14.5701344848, 14.3755846624, 14.2410471184, 14.2779551618, 15.443929156, 15.0424536993, 15.9100900065, 14.7751794074, 15.8596777352, 14.0940269842, 12.6368872278, 13.4153939941, 14.2351019125, 13.6759537164, 11.4619369131, 11.1467628917, 10.271637596, 11.4914894769, 11.4126503243, 11.4170764457, 11.5416076476, 10.81432022, 11.6873404089, 10.2246690663, 12.2141643601, 10.0647400074, 8.8168806693, 9.8255315767, 9.6529578457, 9.9851217504, 11.0790699125, 11.3002523105, 10.9375126168, 11.5030860532, 11.1636858003, 10.802370883, 8.7025440979, 9.6844283153, 8.8076557925, 9.7110133784, 10.8759989548, 9.7463869848, 9.3099499989, 9.1940666288, 8.7645447003, 9.2400480048, 10.0735684729, 9.8751173362, 9.7019485412, 9.6302246859, 9.1823974484, 10.4541474156, 10.0116312496, 8.8608512488, 8.8813270935, 9.1533086501, 8.6351140942, 8.4211233995, 9.4578647513, 9.030446451, 10.0511277678, 9.4783424361, 7.2322135523, 7.2149325535, 7.2346812908, 6.8615745911, 9.1707015883, 8.2955446729, 8.3790977366, 8.1973393702, 9.3138726331, 8.8996041329, 8.9816130768, 8.5554894928, 10.3274072851, 9.5368567314, 8.6205131345, 8.3949961831, 7.1554420887, 8.29557171, 12.8434952386, 14.1045919549, 13.82004352, 12.894321454, 11.2894190029, 12.0516944697, 12.2085001958, 11.6924688867, 11.97397895, 12.7017874234, 12.3874159545, 12.3171437438, 12.6139033546, 12.3212056636, 12.5423253127, 12.5676220938, 12.8434352546, 12.5883687037, 14.2896309688, 11.4446382607, 16.8491439485, 17.6052206598, 15.6528511516, 15.3464966924, 16.2946895681, 15.7190073162, 16.8215397194, 16.509280338, 16.3391948328, 16.8169245178, 8.9255866642, 9.7800344446, 10.1243990723, 10.4271272644, 10.6687810257, 8.4793826064, 7.662314252, 9.0497090722, 8.5474989539, 8.9342847033, 12.7184921445, 10.6134088761, 9.8874754981, 11.0169286118, 10.5195291683, 11.3637146416, 10.9253934867, 11.1062527509, 10.7387566993, 11.2476344801, 9.7797335641, 11.6630275202, 10.8999572789, 9.6469943102, 9.2163461129, 9.2058763239, 9.471153246, 9.5552870119, 9.7881307881, 10.0326152395, 9.4779487077, 10.7203327524, 10.3887044009, 11.075013619, 10.9113873097, 10.9800742775, 9.9181233726, 8.6951798486, 9.1776748831, 9.4774788462, 13.6417732058, 13.2506920793, 13.2884651278, 12.2893725863, 12.8282468628, 14.148428337, 13.907065609, 14.0228707619, 11.5145181999, 11.9329707508, 10.808177304, 10.5268724843, 8.5990820971, 8.7502672561, 8.7047071031, 8.3166444061, 8.8605805633, 9.3700799394, 9.422918161, 9.0934363651, 12.2551178933, 10.4574628862, 9.6364045011, 10.1896506186, 10.3999751736, 10.0987177281, 10.5377220759, 11.4208028796, 11.1618343198, 11.5270819323, 13.9483149135, 14.9971775244, 14.043737011, 15.2676371256, 15.1700984045, 15.1835754791, 14.8656646388, 16.0850090939, 13.9589619376, 13.6223816945, 13.1752940105, 13.4294097121, 12.9980068839, 12.100788379, 13.4504527691, 13.5841704997, 14.2312212169, 12.2621217351, 11.8010545904, 12.8101667181, 9.5267757379, 9.2566273772, 10.2083658243, 9.8683975635, 9.781484432, 9.8032625636, 9.5584497441, 8.9778904483, 8.2263378144, 8.2842265433, 9.1743001166, 9.1540453951, 9.837454289, 8.3752777858, 8.1096691767, 6.9220921297, 7.337247556, 7.7081332188, 7.8913052056, 8.2024423919, 10.9489754893, 11.4529101037, 8.6849276313, 8.3489485295, 9.1506178681, 9.1575979626, 9.5621235544, 9.5126626947, 9.6392040985, 10.1967879104, 11.702939737, 12.8372383064, 11.1589117035, 12.1513435094, 9.7418943061, 10.3920126179, 10.1762852476, 11.2581940104, 10.3820532144, 11.5898905291, 14.8467395346, 14.2992022773, 14.7679547513, 14.5750922669, 13.5532043614, 15.1156738551, 15.5482606378, 15.0579607243, 15.2392257219, 13.0153813066, 13.420575739, 12.3646759132, 11.0292655188, 10.0609077287, 11.3332392777, 11.3899637416, 10.8793850666, 11.0058450199, 11.8206981705, 11.2640920941, 10.6868899423, 11.6297384747, 10.6596670175, 11.1812503471, 8.877609836, 8.5921660207, 8.7335887444, 9.3748244291, 8.9474948009, 10.4013792692, 13.2129152024, 12.5340302138, 12.9977106214, 12.8610867564, 13.0202098747, 13.0639535623, 11.9588616605, 12.5570603864, 11.4136363274, 12.058775039, 8.8313822769, 11.4179925176, 10.1617382146, 10.8048553733, 10.9169913148, 10.0135059628, 10.5367598823, 10.9190972629, 11.2619248539, 9.2868678057, 9.3891418612, 9.0238088375, 9.477629063, 8.2950075067, 7.5956889242, 7.372025079, 7.5707123174, 7.2570051253, 8.1080995813, 8.3900510524, 11.9061626921, 12.4562900183, 10.4502968053, 8.3777004657, 9.2381995735, 8.9976744059, 10.4121981096, 10.7857431804, 9.3346770137, 10.4643171644, 14.4547234113, 13.6300729535, 13.3953520177, 14.2186368012, 14.4647738858, 14.5313536354, 12.3386195855, 11.2840781109, 12.4918123379, 12.629509537, 11.6371289179, 12.3931238656, 11.8682984199, 13.2983726995, 13.1899427872, 9.3991990567, 10.9763423632, 10.6209483445, 11.3780131853, 10.6735712097, 10.0658737393, 9.699626243, 9.4222554453, 7.4999168043, 8.8408320765, 8.607735422, 8.4690534116, 7.6789350106, 8.2505252691, 8.865709131, 12.1936974405, 12.7694869923, 12.5754008038, 13.2495795948, 13.61976187, 13.0844121917, 13.2829353993, 14.0769018925, 12.6100319164, 12.8835462344, 10.5918482971, 9.5895723217, 10.2698219867, 10.0193945852, 10.9922837394, 9.6159325899, 10.5016044932, 10.0550053795, 10.6197153978, 10.1306326398, 13.9215585628, 13.0615283605, 12.3430337778, 13.3839150761, 12.7752359928, 13.9155191125, 13.1023928522, 13.3829092246, 13.1283770939, 13.4032662084, 12.7151249557, 12.6242927218, 12.5976903107, 13.8880923446, 13.2561085142, 12.472395247, 13.8896935353, 13.7255101973, 14.0111868273, 13.1777441962, 5.4508938579, 5.5734709974, 6.3462445586, 5.0941376275, 5.8703760084, 7.0495253231, 5.9025889879, 5.4735155314, 5.7353303425, 7.5669040228, 9.6383195293, 10.2061418384, 9.0803364405, 9.3110482229, 9.8430577156, 10.555580814, 10.3132003894, 9.7425965007, 9.8319839118, 10.0143780911, 7.7654909353, 7.021477545, 7.2680780871, 7.4858266403, 7.3622073114, 6.4061812791, 7.4717166886, 8.0911080162, 7.0207130969, 7.4031705325, 6.741864027, 6.1055860842, 7.0326605578, 6.7568697915, 6.5226672066, 7.0869329064, 7.8263791289, 6.3443924016, 7.2181326992, 7.1756313157, 11.4784612154, 11.6837327011, 12.2062308602, 11.1821655386, 11.3154523624, 12.8397870084, 11.8315433107, 12.5177253999, 12.5225982584, 12.6846170781, 8.7414839356, 8.4345940875, 8.2281882551, 7.9672822176, 8.616035431, 8.4170400753, 8.8864721933, 8.234549285, 8.8563967855, 9.0761433751, 7.3177933817, 8.5448282475, 8.0437414409, 8.6873513947, 7.6665229804, 8.841107969, 8.1974440213, 9.0677243914, 8.5893875289, 9.0747133896, 10.6967629372, 10.5545292221, 9.9474577746, 10.7511130902, 9.803766058, 10.7231650159, 10.0743546161, 10.8380889602, 11.4378361146, 11.4248559718, 9.5812301413, 9.3261790143, 9.5522486883, 9.5914478124, 9.3577639712, 10.2585815907, 9.8849057134, 9.5864114868, 10.6458064066, 9.5467144735, 11.8309353856, 11.7032830024, 11.7375483312, 12.323724092, 13.4591284347, 13.7561733945, 12.2900850859, 13.2596328167, 12.4964385993, 13.1240959571, 6.1428059765, 6.4620293278, 6.7173630901, 6.5876442572, 6.330006162, 6.5788796721, 6.8981889898, 6.7985300622, 7.1776878965, 6.7597532218, 10.5918692273, 10.0729419481, 11.3180305364, 10.3044420046, 11.1008198449, 10.9899745864, 10.8012794839, 10.6976216024, 11.0637469426, 10.7561248247, 7.7152962767, 8.6508398454, 8.5642063071, 9.0430677477, 8.1790051502, 8.5662837649, 9.0524590819, 8.4018769886, 8.7931061215, 9.7206738067, 8.3504162817, 8.2259825643, 8.8797377721, 9.6683063482, 9.7319656927, 9.1986013837, 8.6804605432, 8.8132057554, 9.5917698225, 9.4179129517, 13.567907207, 13.663795908, 14.2040983141, 14.4577882336, 13.912353868, 14.3912350216, 14.2546722793, 14.6929306128, 15.0078462691, 14.6586764506, 12.1087530758, 12.5774240358, 13.0883997082, 12.3580771904, 13.1139506099, 12.5994633053, 13.5603283796, 13.7177162531, 13.9878988409, 13.0349465543, 11.8819630856, 13.6486242385, 12.7953269037, 12.867997867, 13.3558680974, 12.4998585706, 12.7521712159, 13.8503541595, 14.0304204484, 14.1919799831, 11.2047462779, 11.9118944669, 11.4297379746, 12.0053717623, 12.2711253671, 11.4861066785, 11.1827137102, 12.0385197992, 11.8946425893, 11.5176866013, 8.5586810318, 9.1212795919, 9.6886876582, 8.426933663, 9.5779454055, 9.507594598, 9.8413049765, 10.0732058646, 8.9693737725, 10.4802704269, 8.4969288123, 8.9803760315, 10.2373577133, 9.7092994617, 8.7591931306, 8.9354678161, 9.4829026165, 10.0690109608, 10.8301733118, 10.2938679905, 12.8423462141, 13.4193030144, 13.9643669445, 13.4131858677, 14.2829640144, 13.8339901065, 12.9808685762, 14.0481401992, 13.7007384635, 13.8155429937, 10.7727861952, 10.2258713637, 10.4409546215, 11.1388536656, 11.3132605655, 10.7867012864, 11.0974493392, 10.9349906001, 11.0624070888, 11.3422023262, 10.3519370341, 10.8142821681, 10.7059272386, 10.6159475844, 10.3772910319, 11.0237077638, 11.3126277776, 11.2995035598, 12.0524313771, 10.998025032, 13.7397155586, 13.1682930753, 13.2865902393, 13.0239299481, 13.1887371467, 12.5725844183, 14.0085501593, 13.352889715, 12.7906889877, 13.0681857914, 11.2027542507, 11.5133926573, 12.3650603132, 11.5833928759, 12.2191481501, 12.2491252933, 12.4673624609, 12.1461488223, 11.9774566142, 12.6006824375, 10.3948674366, 10.7291270892, 11.442304725, 11.9536233088, 11.9661048842, 10.9812157188, 11.0373835507, 11.6003200749, 11.0859297848, 11.3898073747, 10.6749079464, 11.177871178, 9.9909867271, 10.2543258504, 10.5055699063, 12.4011424679, 11.5271007554, 12.38967066, 12.0013081999, 11.7693042368, 14.456219018, 14.7198898801, 14.2561746909, 14.9842429871, 15.9514882666, 14.9710124318, 15.2171157146, 14.7051295855, 15.8706057683, 15.6225610961, 11.5287654905, 12.0190587249, 11.8830319904, 11.6724517151, 11.7649750192, 12.108465611, 11.897354974, 12.6694061778, 12.7260722368, 12.536348723, 13.9303879755, 13.1998737288, 14.1303800487, 14.4239232476, 14.3694326118, 14.4288096935, 13.6200516347, 14.1770806379, 14.7522969955, 14.4743147178, 13.8730990713, 14.414540271, 13.9007542185, 13.1666077676, 13.7378624816, 13.8056383213, 15.6129797772, 14.5575422004, 14.0325397865, 15.0105176445, 12.5903142134, 13.4506103012, 12.9496802166, 12.9786153596, 13.7437449838, 12.5003758052, 13.5672054372, 12.978142628, 12.8071830908, 13.7733440283, 12.5464901382, 12.5248792408, 12.353534147, 13.1189350356, 13.0702383711, 12.80052888, 13.1769784966, 13.2587908745, 13.4567474485, 13.6956649034, 8.9171822775, 9.0288250572, 8.6152549505, 8.4691033965, 9.7767655873, 9.4344888471, 10.3835910345, 10.3003568269, 9.3841821564, 10.5032295495, 10.5762372211, 11.7384006072, 10.5117347844, 10.631009541, 11.391674044, 11.771257528, 10.9900427842, 10.9082234741, 11.6891997433, 11.459789918, 11.3167092703, 12.638072383, 12.4723562385, 11.2562264033, 12.1513582185, 12.3706874638, 12.7728423457, 13.2427370329, 13.5484457771, 13.496094435] + }, + "params": { + "pattern": "mixed_single_switch", + "n_groups": 80, + "n_periods": 10, + "seed": 108, + "effects": 5, + "placebo": 4, + "ci_level": 95 + }, + "results": { + "effects": { + "1": { + "overall_att": 2.0343516096, + "overall_se": 0.078452370594, + "overall_ci_lo": 1.8805877887, + "overall_ci_hi": 2.1881154305, + "n_switchers": 723 + }, + "2": { + "overall_att": 2.1643975658, + "overall_se": 0.086660922031, + "overall_ci_lo": 1.9945452797, + "overall_ci_hi": 2.3342498518, + "n_switchers": 604 + }, + "3": { + "overall_att": 2.0617157646, + "overall_se": 0.096829894238, + "overall_ci_lo": 1.8719326592, + "overall_ci_hi": 2.2514988699, + "n_switchers": 492 + }, + "4": { + "overall_att": 2.1233816272, + "overall_se": 0.086660407191, + "overall_ci_lo": 1.9535303502, + "overall_ci_hi": 2.2932329042, + "n_switchers": 397 + }, + "5": { + "overall_att": 2.1469096843, + "overall_se": 0.12127627041, + "overall_ci_lo": 1.9092125622, + "overall_ci_hi": 2.3846068065, + "n_switchers": 303 + } + }, + "placebos": { + "1": { + "effect": 0.058732840531, + "se": 0.088663575632, + "ci_lo": -0.11504457445, + "ci_hi": 0.23251025551 + }, + "2": { + "effect": -0.10196774279, + "se": 0.10356166782, + "ci_lo": -0.3049448819, + "ci_hi": 0.10100939631 + }, + "3": { + "effect": 0.16173212803, + "se": 0.13918942693, + "ci_lo": -0.11107413579, + "ci_hi": 0.43453839184 + }, + "4": { + "effect": 0.057157365035, + "se": 0.17371270792, + "ci_lo": -0.28331328615, + "ci_hi": 0.39762801622 + } + } + } + }, + "joiners_only_long_multi_horizon": { + "data": { + "group": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 34, 34, 34, 34, 34, 34, 34, 34, 34, 34, 34, 34, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 43, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 44, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 49, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 51, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 52, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 55, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 59, 59, 59, 59, 59, 59, 59, 59, 59, 59, 59, 59, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 60, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 63, 63, 63, 63, 63, 63, 63, 63, 63, 63, 63, 63, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 69, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 70, 71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 71, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 72, 73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 73, 74, 74, 74, 74, 74, 74, 74, 74, 74, 74, 74, 74, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 76, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 77, 78, 78, 78, 78, 78, 78, 78, 78, 78, 78, 78, 78, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 79, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 80, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 81, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 82, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 83, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 84, 85, 85, 85, 85, 85, 85, 85, 85, 85, 85, 85, 85, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 86, 87, 87, 87, 87, 87, 87, 87, 87, 87, 87, 87, 87, 88, 88, 88, 88, 88, 88, 88, 88, 88, 88, 88, 88, 89, 89, 89, 89, 89, 89, 89, 89, 89, 89, 89, 89, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 90, 91, 91, 91, 91, 91, 91, 91, 91, 91, 91, 91, 91, 92, 92, 92, 92, 92, 92, 92, 92, 92, 92, 92, 92, 93, 93, 93, 93, 93, 93, 93, 93, 93, 93, 93, 93, 94, 94, 94, 94, 94, 94, 94, 94, 94, 94, 94, 94, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 95, 96, 96, 96, 96, 96, 96, 96, 96, 96, 96, 96, 96, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 97, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 99, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 101, 101, 101, 101, 101, 101, 101, 101, 101, 101, 101, 101, 102, 102, 102, 102, 102, 102, 102, 102, 102, 102, 102, 102, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 103, 104, 104, 104, 104, 104, 104, 104, 104, 104, 104, 104, 104, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 106, 106, 106, 106, 106, 106, 106, 106, 106, 106, 106, 106, 107, 107, 107, 107, 107, 107, 107, 107, 107, 107, 107, 107, 108, 108, 108, 108, 108, 108, 108, 108, 108, 108, 108, 108, 109, 109, 109, 109, 109, 109, 109, 109, 109, 109, 109, 109, 110, 110, 110, 110, 110, 110, 110, 110, 110, 110, 110, 110, 111, 111, 111, 111, 111, 111, 111, 111, 111, 111, 111, 111, 112, 112, 112, 112, 112, 112, 112, 112, 112, 112, 112, 112, 113, 113, 113, 113, 113, 113, 113, 113, 113, 113, 113, 113, 114, 114, 114, 114, 114, 114, 114, 114, 114, 114, 114, 114, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 116, 117, 117, 117, 117, 117, 117, 117, 117, 117, 117, 117, 117, 118, 118, 118, 118, 118, 118, 118, 118, 118, 118, 118, 118, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119, 119], + "period": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], + "treatment": [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], + "outcome": [9.2100823667, 7.9130550717, 8.4376453949, 8.14820162, 9.0845909682, 8.1751709272, 9.1715928932, 8.6139424331, 11.0158987769, 11.4681987769, 11.001354439, 10.7615263732, 9.5976493342, 8.0638439342, 10.3628355156, 8.8228462005, 8.6266937752, 8.8151421223, 11.236999252, 11.5507221365, 11.4527713215, 11.9892816822, 11.5506641204, 11.1233947417, 12.2779760226, 12.0598266032, 14.9433548291, 14.689645845, 14.431091473, 15.2139112488, 14.8358631892, 15.7063860862, 15.1093875727, 14.6262851199, 15.2702067643, 16.0055833698, 12.2456690292, 12.1948588849, 13.3672467652, 11.7601854547, 13.514049199, 12.0398056245, 13.0655917868, 12.3984152358, 13.0047380467, 15.3539093223, 14.9364309357, 15.2912697551, 7.7604408005, 8.0625393193, 8.3782262234, 8.9343697093, 8.9799662651, 10.4127980784, 10.9308107446, 11.1682369841, 11.582086311, 11.0395163543, 10.5376724805, 12.1888700311, 12.2220576356, 12.4342710425, 11.8903757554, 12.121665524, 12.8152592389, 12.8080327736, 11.8401321193, 11.9328015367, 13.5344009392, 11.8897156968, 14.841750389, 14.912583215, 10.7237972432, 11.6194667304, 9.6429668779, 10.5859899472, 11.6076849941, 10.7148502921, 10.7381597833, 11.5708852492, 11.7416400745, 11.7411291945, 13.0612298355, 13.2558364573, 11.1704256026, 11.5061217841, 11.0744235106, 12.056961208, 12.0535648363, 11.171207203, 12.0330260563, 14.3569969505, 14.2810256116, 14.0090910821, 15.0343015937, 14.0684576852, 11.581291617, 10.6209875097, 10.9047566062, 11.7974701811, 11.0314940641, 11.8179116868, 11.32683977, 11.4652484581, 12.0544980872, 11.6351755933, 13.655232992, 14.0140052165, 8.3247619273, 9.314378376, 8.9510825169, 7.8815952606, 9.7203055684, 9.8666072475, 9.835090579, 8.6382305976, 9.3434690335, 11.7293342988, 12.4747998021, 11.9205364947, 14.5709750087, 13.8362004017, 14.5796188437, 14.789807774, 14.5131022147, 16.4206920538, 15.8939135866, 16.8074929092, 16.6987026091, 17.61026031, 16.677677044, 17.9046809678, 12.7076416049, 13.6128943684, 12.6104235092, 13.5280589809, 13.716706382, 13.8270690447, 13.4291257406, 14.0061043042, 15.9108619273, 15.2801787136, 16.5013222346, 16.2614631328, 10.861178317, 11.8828793004, 11.6890680641, 12.3348068194, 12.2111529389, 11.6326008333, 12.4297432995, 14.5275081533, 15.0527895168, 14.7914617547, 14.3347422105, 15.6956370609, 8.7551221311, 11.0012367992, 11.3980350199, 11.6282805971, 11.9561641537, 11.6403420432, 12.1807249533, 11.8630848301, 11.9246098681, 12.7725267259, 12.0196571174, 12.2734070186, 10.2343246909, 11.3317605242, 12.1236932477, 12.7523889271, 13.7231057598, 14.4645142776, 13.7651224766, 13.9909943715, 14.5976521176, 14.2673199088, 12.9835782041, 15.3965799345, 8.2178372633, 8.6451757353, 8.5397813494, 8.4435004728, 7.9168321038, 10.8431618145, 11.0126805147, 10.478433719, 10.8322205087, 11.3666861563, 10.9626504945, 11.1196319695, 12.7960407278, 12.740920366, 13.6662728563, 12.7273516167, 13.8449006005, 13.6830298175, 13.6079158798, 13.8762018118, 14.0748151212, 14.4757767526, 13.4293998814, 16.8618270232, 6.5275586186, 5.9375365154, 8.1203204121, 6.8904750735, 7.6039145317, 8.2050208042, 8.1938999604, 7.352384453, 8.1277503465, 7.3748905451, 10.7348052729, 10.7349396704, 9.7954587128, 10.2715230124, 12.2593579955, 12.4664452166, 12.5865640976, 13.3616589319, 12.769567101, 12.8380972763, 12.7406836395, 13.2634477968, 13.7680665078, 12.5238587198, 10.2420693523, 10.8041986121, 10.7180406347, 12.5453484889, 13.4839992506, 12.2998004175, 12.6445674508, 14.6038231006, 13.8629725295, 12.8565178336, 12.8764961325, 13.9174900423, 13.8037926394, 12.9961776307, 13.6443451915, 12.6641067228, 13.962299968, 13.9569981466, 13.839515033, 13.6915632338, 13.9771664244, 15.9367215451, 16.3880490776, 16.838283874, 10.4692976146, 9.2422802649, 10.3640782218, 12.2084442492, 11.1190666096, 11.4399115114, 12.7500908172, 12.2181656323, 11.7980900929, 13.1393880579, 12.4927399964, 13.208926132, 10.0871714178, 9.3114232446, 11.1759888851, 10.9136441898, 9.9924006988, 11.2954610141, 11.4595648334, 11.226808846, 11.0819103798, 12.4918810611, 11.4662212791, 11.2032368616, 9.8760087951, 9.9824055939, 9.7156268638, 11.3727524131, 11.0110181674, 9.8152779415, 10.0260245978, 10.6507375361, 11.4160248463, 12.909671171, 13.1017098341, 13.5242716058, 8.3366104984, 9.0633191544, 9.0063402382, 8.9709251389, 8.8559351085, 9.317112336, 8.6148191944, 9.6430327659, 12.5394393172, 11.6587972887, 11.4668325066, 11.7362451863, 15.9922354156, 16.2200821771, 16.5927344106, 16.2006567817, 16.1266216582, 16.8572037536, 18.5337068719, 18.4960864956, 18.5619286244, 18.8386204412, 18.5059893282, 19.1475941475, 7.8994266963, 7.9302639465, 8.1269289115, 8.5379880781, 8.2077263858, 9.7738035516, 10.8907739249, 10.5057089887, 9.9994508202, 9.9952269717, 10.3581641895, 10.7237440135, 8.0881666137, 9.6927952145, 9.3631126626, 9.4838937953, 9.3053270639, 9.5540125098, 9.5223965225, 9.7016776871, 11.2133256906, 10.3811080877, 10.9188152898, 11.2098827867, 8.8522506506, 8.9556939412, 10.7713525739, 11.6882646434, 10.5472206013, 11.2211933441, 12.5372714919, 12.4210365413, 11.4049028142, 11.2308196242, 12.1479891775, 11.7126096105, 11.1356246407, 12.726969982, 13.2668684603, 12.8993636715, 13.2565226452, 12.8585613269, 13.3253328528, 12.603237749, 12.9306926483, 14.2413124752, 13.8082912718, 13.0548817823, 10.2801636177, 10.0029204858, 10.4986061239, 9.7521474466, 10.3452181517, 13.7972429867, 12.0735363048, 13.528101992, 12.9535752646, 12.3005055694, 12.7269187282, 13.5617225087, 8.5934277766, 7.9221978654, 9.0442870498, 8.7266770023, 8.8469172023, 8.6287679114, 9.5293656802, 9.9629719048, 9.6918652413, 9.807368901, 11.4514359941, 11.73037239, 9.7775756383, 10.9621945174, 10.9889199216, 11.0734441613, 11.6017810937, 11.7196054074, 13.8817633207, 13.2719243389, 12.7948303838, 13.6953569232, 14.553974501, 13.455813827, 11.583626728, 11.9752076637, 11.0289163252, 11.4319136148, 11.4296357284, 12.388482927, 12.6460677941, 11.4181597653, 11.635225574, 12.6953369992, 11.3607002827, 14.4381093739, 10.6925473179, 12.277306033, 10.724452186, 11.6314346236, 11.8811665069, 13.5871018309, 14.3614973174, 13.4985961678, 14.8458215525, 14.1192765527, 14.2720207572, 13.7448007681, 6.7678608358, 6.6757924837, 7.3301729903, 5.6842074364, 7.467358782, 7.3468837682, 6.1769411422, 7.0746734893, 7.8262491495, 7.9639039762, 9.2248650368, 8.891318551, 9.3099053776, 8.5775546624, 9.4124596913, 9.1551147265, 10.1602915694, 9.718238272, 10.9596130483, 11.2806898416, 11.9503705514, 11.5747720461, 11.5835539485, 11.9544251985, 10.9904295335, 12.1197757205, 12.4069004157, 10.573682723, 10.4897517148, 13.48234367, 14.8059236064, 14.1787064973, 14.845806768, 13.6802904737, 14.5441181268, 15.5091429104, 10.1908365049, 11.1144875106, 11.1635146205, 12.1103361523, 11.5059955036, 11.6976241721, 12.2637116514, 11.2276699302, 13.2929735751, 11.9231099894, 12.7527749554, 13.0406709988, 6.7431171702, 6.6423577169, 7.6226518641, 7.0098091069, 7.403202847, 7.0372465719, 6.3101349627, 7.8233471841, 7.6157038476, 8.0930326705, 7.6314114384, 9.2917007606, 10.3186239721, 12.1817174596, 12.802519326, 12.2939887569, 13.1050592391, 13.0854959348, 12.5152842304, 13.6580613266, 12.6577762476, 13.4757239046, 13.3326543581, 13.1549542667, 14.7807010336, 13.1542674885, 14.3547277669, 15.0600299192, 14.1999340151, 14.7759796474, 16.8378593629, 17.5308323751, 16.9021895717, 16.9882362303, 17.4661868022, 17.0907186666, 11.395020678, 11.487820088, 11.2962792373, 11.1157429444, 12.7333854907, 12.3067692801, 12.0099465768, 12.2786875903, 11.8797780745, 11.0957929108, 14.9832077182, 14.2427600246, 10.1802606291, 9.6284369782, 9.8213303509, 9.3968526371, 10.0461773699, 9.5980507873, 10.3241926458, 10.4851007152, 10.9780717161, 9.9297588476, 12.7320007347, 12.3963973047, 14.4609273681, 14.1631433031, 14.7633382397, 14.6252972248, 14.6817007999, 14.0069745019, 14.4687643049, 13.6714036817, 16.7046022118, 15.9568944085, 16.0327158104, 17.6549956244, 7.3601524921, 8.3895684889, 8.3353660525, 8.1704995628, 8.5368074495, 8.23529483, 8.4089972716, 10.0981995406, 9.4039014897, 11.0901799072, 10.9487757353, 10.8015464124, 14.8997556209, 14.2081993191, 14.4424558255, 14.8453796478, 14.9169328372, 14.7359246514, 17.0236391175, 16.8057947006, 17.0066400463, 17.656039469, 17.2218785972, 16.6820436719, 6.968393015, 7.6187238545, 9.6774489442, 10.4424156205, 9.2671059168, 10.0074965369, 10.352956764, 9.0597904206, 10.2327347355, 10.1258316381, 10.3149258495, 10.6688058693, 9.6442268774, 9.780629763, 9.3539758385, 10.3610530052, 9.8652667025, 8.990250997, 10.5341474114, 10.8763371045, 11.1658080322, 12.2995310135, 11.8324557686, 12.9860506947, 9.1851889329, 12.8125984278, 12.9079748451, 13.065866414, 12.4302608245, 13.4174842284, 12.8816463924, 12.0159653031, 13.1214759695, 12.9357604556, 14.4546785691, 13.4612997492, 9.9798999949, 10.9666942824, 11.9650284272, 10.7652029683, 10.9300434985, 13.368483429, 13.1062392605, 14.0902895103, 12.4227521059, 13.9505993919, 13.843047484, 13.1726218994, 10.0538564959, 10.3489851032, 9.5341655824, 10.5351027764, 10.720289315, 10.436852003, 10.5471445002, 11.2440642443, 10.4567123038, 10.2563271308, 11.9437442241, 14.0620828602, 10.8477501304, 11.4992686999, 11.8400998263, 13.6924810553, 13.9472176293, 13.9900928692, 13.4755583398, 14.2943757108, 14.1417442547, 14.1739775338, 14.2827319823, 14.4394922517, 9.4251569397, 10.2903567785, 10.2043742051, 11.386637645, 11.2761720505, 10.049020733, 10.403256127, 11.5739056694, 13.3891627406, 13.1911860094, 13.1158117758, 14.3226463697, 13.4304140813, 13.0307213595, 13.1875737446, 13.3147675479, 13.6884546429, 15.700740903, 15.4785064292, 15.7541413205, 15.8010429311, 15.8015112841, 16.5104424899, 15.8586352305, 10.3216471986, 10.2779561003, 10.8264891015, 10.6294522808, 11.8807061128, 12.2653890558, 13.0380817391, 13.3688251514, 12.440380695, 12.5918530107, 12.9111423513, 14.4439851937, 9.2488483341, 9.4129191037, 9.4508207023, 8.5636292075, 9.3229316316, 8.3502788466, 12.2527695304, 12.288098269, 12.4401684793, 11.220968691, 12.6647984701, 11.5745765075, 8.2565058687, 8.2015397911, 9.0960593869, 8.4438456325, 8.0739183571, 9.9949513033, 9.1591490197, 9.5778273925, 9.6568602744, 8.9577978279, 8.732391853, 12.232603245, 9.0220854284, 9.7673565433, 9.1701956652, 10.0891448074, 10.0049929865, 9.0659731105, 9.3616987155, 11.2295744163, 11.8963149897, 12.9290285228, 12.3944247241, 11.798779455, 9.8163391164, 9.6927160076, 9.9051944273, 9.5065827998, 9.0041690508, 10.7902757586, 10.2759432758, 10.5474902594, 12.3625889375, 11.7450869384, 12.1076483104, 12.0831030179, 8.3390752352, 8.9506852116, 7.9399741375, 8.5091513072, 8.0652798821, 9.2792363896, 10.7287506588, 11.0097392478, 10.1167267684, 10.5067993872, 11.1348621166, 10.7293703052, 11.973890616, 12.2077748682, 11.9821390777, 12.1088209514, 11.4371396385, 11.6909038429, 12.519548467, 13.7967397404, 14.635849294, 14.4698119813, 14.0169492433, 14.4818265247, 9.6855223121, 9.4505509858, 9.0643687235, 8.2615551242, 7.9797438939, 9.2624302292, 9.1024191586, 11.4943633484, 12.6130041001, 10.7758506213, 12.0605957445, 12.1204863474, 10.2326853093, 9.7360828598, 10.950253383, 9.3740246514, 9.9057575549, 10.2041332037, 11.053968691, 10.7746286699, 10.7853906356, 10.6050303631, 10.8644362476, 12.1052473681, 10.3682619513, 9.6962855903, 11.4557784286, 11.1356203342, 11.3581406068, 10.9512181915, 13.4138794591, 13.9451453915, 13.7707181767, 13.8248558332, 13.3366617601, 14.4930120693, 10.0788036499, 10.1079644618, 9.224752174, 9.591660776, 10.9572641393, 10.9351473788, 10.9205869512, 11.0361720543, 11.0683963682, 12.3444627032, 13.6536164318, 13.4626461651, 9.5506085643, 9.8292683257, 10.1316022863, 11.8658198499, 11.8438571892, 12.584027475, 11.6350334377, 11.9335753783, 12.4683814669, 12.93662784, 12.6877173219, 12.444241305, 7.8775681513, 8.7995353683, 8.3426154196, 8.487063316, 7.7492543656, 10.8960855516, 11.2137643696, 11.192844303, 9.7366858429, 11.2543615183, 10.5273000727, 11.8327552245, 10.3523853073, 10.2273548205, 10.8137353096, 9.8187897397, 10.412945635, 10.7311527583, 11.4115587745, 11.5664833103, 12.5062404304, 13.5768125602, 13.4403946127, 14.2635174213, 9.6790936361, 8.641654927, 8.8164734564, 9.5211651146, 10.1430301071, 9.4028959285, 10.9255630686, 9.9234929705, 10.7211298608, 12.7671863798, 12.2972333693, 12.9090498235, 11.738657586, 10.3545722185, 10.7315567504, 11.8847693177, 11.3782526607, 13.7557138589, 13.4468655188, 13.8588445269, 12.7039799857, 13.7674879925, 15.2317803471, 14.0540847676, 11.0387995628, 11.9062020105, 12.2958364909, 10.5877170643, 12.1458861848, 11.3325893842, 10.982444895, 11.6352903594, 11.901530203, 12.1066904031, 12.7736182368, 13.96708834, 9.8506484422, 9.654406657, 10.2715161534, 9.8311975431, 9.3191247078, 9.5018854584, 9.2752028066, 10.3827173481, 10.4338027017, 9.8984470259, 9.7584564066, 12.5086296073, 11.0896388777, 11.7261507897, 11.3238509998, 10.7583024039, 11.4126862456, 11.6360978445, 11.640279977, 12.2040857818, 14.613339578, 13.884367659, 14.5741838687, 13.992700931, 6.5779616105, 5.8346778187, 6.3240430251, 7.1058752858, 7.1691650157, 7.1615229551, 6.5451897984, 8.8577011075, 8.5534367737, 9.6794054248, 9.1610331849, 10.0750139256, 9.4410207809, 9.2046012283, 10.4913754829, 11.2733733539, 12.2252442519, 13.1187382771, 12.2357169656, 12.6603946172, 12.1886357258, 12.5147901919, 12.8024512244, 12.3144466343, 12.7719506897, 11.9466460176, 11.8062931489, 15.7628324625, 13.4051696864, 13.6958993396, 14.9053783281, 15.3042220708, 14.793203483, 15.3048029254, 14.4457655241, 15.213051769, 10.0723512001, 12.3563587661, 11.6061862253, 11.6337987563, 12.6052487273, 12.2283003394, 12.3195889277, 13.0374810585, 11.8984570097, 11.780535777, 12.785963728, 13.2651258711, 10.2226419547, 10.492499943, 11.1336297899, 10.9000719955, 11.6937674604, 9.9730170727, 10.7395619821, 10.8111687783, 13.219951464, 12.8281816985, 13.3507054986, 13.2589541676, 11.267960485, 10.9973019018, 13.7033275631, 14.0149026243, 13.8311798432, 13.7527411296, 13.6909087445, 14.838216582, 14.6391175799, 15.1027014783, 14.3140522797, 14.6546258039, 13.7744855683, 13.95540492, 13.5888662213, 14.3134807063, 13.8337807822, 13.5465027995, 14.289236577, 15.3552924815, 14.4588853361, 14.5717120658, 15.6879324714, 15.4010380348, 11.2007280372, 11.611233019, 11.4822360058, 11.093406527, 11.3683323323, 12.6175246077, 11.899155464, 11.6992329592, 12.2200440515, 12.922073458, 12.6618263528, 12.0523695438, 7.2511013825, 7.0084922095, 7.6381653313, 8.2050326808, 6.8290798485, 8.0891014851, 7.561705807, 7.8355130121, 7.9026111847, 7.751246596, 7.7922213338, 8.8622171699, 9.1725777613, 9.3708944172, 9.1284317142, 9.8362206584, 10.4825238158, 10.0659072431, 10.1774349153, 10.2148210062, 9.8798870176, 11.0106354758, 10.2886800875, 10.3200832804, 10.6919933032, 10.4070160932, 11.014068069, 10.4085301464, 10.2763811769, 11.3807671075, 11.5508236495, 11.8509557679, 11.3241399669, 12.1623377707, 10.8572528247, 11.8851052314, 10.2088613003, 11.4412006437, 10.9657794052, 9.8538901581, 10.2748764896, 9.9858097393, 10.9279719483, 11.786103757, 11.6839966714, 12.3892178254, 12.2415434252, 10.981497701, 9.6106731019, 9.047314846, 9.7031988149, 9.416065335, 9.6854037942, 10.1081869197, 9.1351271417, 9.6402104099, 10.0300280565, 10.5498378532, 10.4508589087, 10.0676136929, 12.4062415831, 12.5599455448, 12.1609311902, 11.6298091256, 13.55854029, 12.6320547006, 13.7974026898, 12.8171002952, 12.878419515, 13.1596081097, 13.4157130205, 14.0275239402, 9.7284606215, 10.195125244, 10.2777743821, 10.0134700279, 11.7708729838, 10.8729143745, 11.1598020805, 10.8478195582, 11.9247762868, 11.1197625999, 11.9131136268, 11.5140605398, 8.5645086019, 7.9736096616, 8.743348856, 8.728410448, 9.2560686047, 8.328218695, 8.317805763, 8.8895624791, 8.9901304878, 9.5221926126, 9.7042705668, 9.3273797611, 11.5045525586, 11.7449459318, 11.3934408278, 12.9647792746, 11.9683194794, 11.9472554589, 12.2111789343, 12.5311035578, 12.8355837395, 12.9992613193, 13.0985137238, 12.521008131, 12.9538889711, 12.036821378, 12.1659150766, 13.1044318828, 12.561452816, 13.0973594467, 12.60827324, 13.7473762943, 13.0775326357, 12.944894361, 13.433791916, 13.3794401273, 7.8567375956, 7.8347437315, 8.3982570492, 7.7233394741, 8.8121053982, 8.1376580975, 9.1337002078, 9.0953201052, 9.2481366876, 9.4509054519, 10.024220013, 9.4875211997, 12.1731419546, 11.5567838281, 11.7629255975, 11.8306342175, 11.9631746582, 12.1916228305, 13.127247481, 12.9272027718, 13.3338490957, 12.7278620288, 12.8856527073, 12.3538439566, 6.8982175005, 7.3979456819, 7.5429230883, 7.9015096873, 7.451238256, 7.255027298, 6.965727966, 8.3771294895, 7.9074272947, 8.1619080227, 7.1354151155, 8.2967586799, 9.4350371316, 9.5761477065, 9.6407467636, 9.3843381693, 9.0228714344, 10.3094512722, 10.2732890536, 10.0943838722, 10.3043132487, 11.0973169116, 10.3727154132, 11.2312342955, 10.6129054049, 10.5023779819, 10.9542753432, 9.4218909598, 10.5478820572, 10.4807093787, 10.9017672195, 10.7747660977, 10.7684101145, 9.7236073878, 11.8492326941, 11.7696320947, 11.2793797978, 11.1833558619, 11.2479716431, 11.3877721848, 11.799336329, 12.0256371075, 12.4793767209, 11.429935646, 11.1493772994, 12.1868705885, 12.0862774361, 12.7214996662, 9.2035043093, 9.2701679302, 9.0538724975, 9.3699752461, 10.1862125846, 9.5632204582, 9.6731269489, 9.0811093677, 10.7165666299, 9.4215992793, 10.5651654657, 10.3761594959, 8.3295319328, 8.3743446197, 9.5352148045, 9.6061465717, 8.782603717, 8.674153588, 10.0679756135, 9.391619319, 9.0055832763, 10.6006355264, 9.513844156, 9.5683161522, 13.2981903037, 12.8759677553, 13.0052399311, 11.6781631653, 12.6026566266, 12.5311229282, 12.7333286037, 13.1639578615, 12.9599112692, 13.4937795336, 14.0136112932, 12.7424281647, 15.0184459731, 14.3112779538, 14.6778249915, 15.1922772471, 14.9035079494, 15.468585425, 15.2320381087, 15.8077106932, 15.4516447693, 15.2579765752, 15.2296937761, 16.3911613015, 12.1537321756, 13.1696473087, 12.509681627, 12.6054406923, 13.0110463763, 13.1714454173, 13.3676844025, 13.7855821384, 14.5981033211, 13.654914201, 13.3940790695, 13.6766247573, 13.5320343447, 13.7637618219, 14.7634716468, 14.9643267426, 13.978133433, 14.4282004786, 14.8685766648, 14.9048664003, 15.1960968278, 14.5017699662, 14.5193051874, 14.9554301874, 12.8760166205, 12.8764726276, 13.9697094107, 13.3404161003, 13.6159758309, 13.7823417005, 13.5884092991, 13.949447003, 13.7786486198, 14.2667553413, 14.6217447738, 14.8254409672, 14.3370274563, 13.7120535425, 13.5327380048, 14.2940766265, 13.0308260396, 14.0914094934, 14.1273659959, 13.4342201075, 13.9955994434, 13.8935845755, 14.2837617785, 15.4067322987, 10.0963277353, 9.733636908, 10.3575480836, 10.0893262612, 9.5091389771, 9.4326488483, 11.1242372136, 10.7456280708, 11.5878944105, 10.7591094048, 11.4181413355, 11.1957457616, 14.4058268041, 15.0001635743, 13.4591652848, 16.1258819529, 14.5898344152, 15.3053431316, 14.9944661103, 15.8168243872, 15.406906494, 15.7820067353, 15.6531029308, 15.7460304319, 12.4434356743, 12.4903421659, 12.0455586758, 12.7587997146, 13.3523192209, 12.5079974924, 12.8099439822, 12.9336367078, 13.5426405465, 13.2563743224, 14.0820712623, 14.8164012083, 12.880317069, 11.1770049789, 11.651143675, 12.7099342362, 12.8446131708, 12.5606231899, 13.274290044, 12.5232134077, 12.9738779526, 13.5261889009, 13.1405018361, 13.2224561605, 10.9927240593, 10.526828631, 10.6319539963, 11.3154717539, 10.9250610693, 11.1680730493, 10.9849808041, 11.6522745077, 11.5336738473, 12.4439323069, 11.5649315762, 11.642497155, 9.3153598476, 10.3486157859, 9.5350557241, 9.1095075435, 10.496023961, 10.1683387547, 9.7547108577, 10.9723576864, 9.7995581076, 11.2827695212, 10.7711933145, 11.3942836022, 11.340293306, 11.2894502377, 11.2150270405, 10.9279104691, 10.9407453253, 12.0459928685, 12.2995955512, 10.8574759826, 12.0040575782, 12.4715091112, 12.147580462, 11.8089111013, 16.8098016866, 17.5157097494, 17.0377655751, 16.7482731218, 17.3419773141, 17.3039285208, 17.9916169876, 18.0272714647, 18.3397674661, 18.3923339359, 17.3022264952, 17.8983779158, 9.1272703285, 9.8360920839, 10.0614409406, 10.2173838871, 9.3446254053, 9.654933196, 10.9425531953, 10.3963038961, 10.2996571546, 11.1135130538, 10.6961365681, 10.8225061194, 13.4969943964, 13.264369019, 13.1933893942, 13.5004877612, 13.2320854296, 14.1737240379, 13.3835017117, 13.5274446862, 14.7975905724, 14.6877285283, 14.2950085984, 14.2993971077, 15.9128039949, 16.0538012876, 15.3539888102, 15.8904691212, 15.760715712, 15.415278727, 15.6216223991, 17.0522875668, 16.3684334643, 15.7477681167, 16.6712504021, 16.9983465882, 11.8806815479, 11.6901562144, 11.0758408361, 10.8458677244, 11.7165419015, 12.3043774035, 11.4355820144, 12.2399360544, 12.2568927482, 12.2076186233, 12.9894791172, 12.1353950801, 15.2486940633, 14.6606770798, 15.1170903809, 15.4482461424, 14.8569376673, 16.5369015154, 16.3118613819, 16.0488469283, 14.9670250441, 16.3754271679, 16.3772281283, 15.5385697401, 12.6311743901, 12.1614020375, 12.7386876633, 13.9731192368, 12.777507922, 13.2148435363, 12.7255319332, 13.2705359365, 12.7480643329, 12.9527660356, 13.3316209988, 14.0748593884] + }, + "params": { + "pattern": "joiners_only", + "n_groups": 80, + "n_periods": 12, + "seed": 109, + "effects": 5, + "placebo": 5, + "ci_level": 95 + }, + "results": { + "effects": { + "1": { + "overall_att": 2.0605577084, + "overall_se": 0.09644523613, + "overall_ci_lo": 1.8715285191, + "overall_ci_hi": 2.2495868977, + "n_switchers": 733 + }, + "2": { + "overall_att": 1.976639758, + "overall_se": 0.098910994174, + "overall_ci_lo": 1.7827777718, + "overall_ci_hi": 2.1705017443, + "n_switchers": 631 + }, + "3": { + "overall_att": 1.9159410685, + "overall_se": 0.10141671034, + "overall_ci_lo": 1.7171679688, + "overall_ci_hi": 2.1147141682, + "n_switchers": 535 + }, + "4": { + "overall_att": 1.9312583271, + "overall_se": 0.10606592252, + "overall_ci_lo": 1.7233729389, + "overall_ci_hi": 2.1391437152, + "n_switchers": 446 + }, + "5": { + "overall_att": 2.0720708959, + "overall_se": 0.13081222474, + "overall_ci_lo": 1.8156836467, + "overall_ci_hi": 2.3284581451, + "n_switchers": 356 + } + }, + "placebos": { + "1": { + "effect": 0.040316640986, + "se": 0.082146976182, + "ci_lo": -0.12068847377, + "ci_hi": 0.20132175574 + }, + "2": { + "effect": 0.042344270554, + "se": 0.10274508973, + "ci_lo": -0.15903240491, + "ci_hi": 0.24372094602 + }, + "3": { + "effect": 0.14561265365, + "se": 0.14180368291, + "ci_lo": -0.13231745772, + "ci_hi": 0.42354276502 + }, + "4": { + "effect": 0.01315171738, + "se": 0.1315216579, + "ci_lo": -0.2446259953, + "ci_hi": 0.27092943006 + }, + "5": { + "effect": 0.264644661, + "se": 0.22761984042, + "ci_lo": -0.18148202838, + "ci_hi": 0.71077135038 + } + } + } } }, "generator": "generate_reversible_did_data v1", diff --git a/diff_diff/chaisemartin_dhaultfoeuille.py b/diff_diff/chaisemartin_dhaultfoeuille.py index 8bcac179..eb604dcb 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille.py +++ b/diff_diff/chaisemartin_dhaultfoeuille.py @@ -507,10 +507,16 @@ def fit( ``{0, 1}``; non-binary values raise ``ValueError`` (Phase 3 adds non-binary support). aggregate : str, optional - **Reserved for Phase 2.** Phase 1 requires ``aggregate=None``; - any other value raises ``NotImplementedError``. + **Reserved for Phase 3.** Must be ``None``; any other value + raises ``NotImplementedError``. L_max : int, optional - **Reserved for Phase 2** (multi-horizon event study). + Maximum event-study horizon. When set, computes ``DID_l`` + for ``l = 1, ..., L_max`` using the per-group building block + from Equation 3 of the dynamic companion paper. When + ``None`` (default), only the ``l = 1`` contemporaneous- + switch estimator ``DID_M`` is computed (Phase 1 behavior). + Must be a positive integer not exceeding the number of + post-baseline periods in the panel. controls : list of str, optional **Reserved for Phase 3** (covariate adjustment via the residualization-style ``DID^X`` from Web Appendix Section 1.2 @@ -828,6 +834,23 @@ def fit( all_periods = sorted(cell[time].unique().tolist()) n_obs_post = int(cell["n_gt"].sum()) + # ------------------------------------------------------------------ + # L_max validation (Phase 2): must be a positive integer not + # exceeding the number of post-baseline periods. Validated here + # (after period detection) rather than in _check_forward_compat_gates + # (which runs before data is processed). + # ------------------------------------------------------------------ + if L_max is not None: + if not isinstance(L_max, int) or L_max < 1: + raise ValueError(f"L_max must be a positive integer or None, got {L_max!r}.") + n_post_baseline = len(all_periods) - 1 + if L_max > n_post_baseline: + raise ValueError( + f"L_max={L_max} exceeds available post-baseline periods " + f"({n_post_baseline}). Maximum L_max for this panel " + f"is {n_post_baseline}." + ) + # Pivot to (group x time) matrices for vectorized computations d_pivot = cell.pivot(index=group, columns=time, values="d_gt").reindex( index=all_groups, columns=all_periods @@ -980,6 +1003,164 @@ def fit( stacklevel=2, ) + # ------------------------------------------------------------------ + # Step 12b: Per-group switch metadata (shared by Phase 1 IF and + # Phase 2 multi-horizon) + # ------------------------------------------------------------------ + baselines, first_switch_idx_arr, switch_direction_arr, T_g_arr = ( + _compute_group_switch_metadata(D_mat, N_mat) + ) + + # ------------------------------------------------------------------ + # Step 12c: Multi-horizon computation (Phase 2, only when L_max>=2) + # ------------------------------------------------------------------ + multi_horizon_dids: Optional[Dict[int, Dict[str, Any]]] = None + multi_horizon_if: Optional[Dict[int, np.ndarray]] = None + multi_horizon_se: Optional[Dict[int, float]] = None + multi_horizon_inference: Optional[Dict[int, Dict[str, Any]]] = None + + if L_max is not None and L_max >= 2: + multi_horizon_dids = _compute_multi_horizon_dids( + D_mat=D_mat, + Y_mat=Y_mat, + N_mat=N_mat, + baselines=baselines, + first_switch_idx=first_switch_idx_arr, + switch_direction=switch_direction_arr, + T_g=T_g_arr, + L_max=L_max, + ) + multi_horizon_if = _compute_per_group_if_multi_horizon( + D_mat=D_mat, + Y_mat=Y_mat, + N_mat=N_mat, + baselines=baselines, + first_switch_idx=first_switch_idx_arr, + switch_direction=switch_direction_arr, + T_g=T_g_arr, + L_max=L_max, + ) + + # Per-horizon analytical SE via cohort recentering. + # Reuse the singleton-baseline exclusion from Step 7 and + # build cohort IDs per horizon. + singleton_baseline_set = set(singleton_baseline_groups) + eligible_mask_var = np.array( + [g not in singleton_baseline_set for g in all_groups], dtype=bool + ) + + multi_horizon_se = {} + multi_horizon_inference = {} + for l_h in range(2, L_max + 1): + U_l = multi_horizon_if[l_h] + # Cohort IDs for this horizon: (D_{g,1}, F_g, S_g) triples + # are the same as Phase 1 (cohort identity depends on first + # switch, not on the horizon). Filter to eligible. + cohort_keys_l = [ + ( + int(baselines[g]), + int(first_switch_idx_arr[g]), + int(switch_direction_arr[g]), + ) + for g in range(len(all_groups)) + ] + unique_c: Dict[Tuple[int, int, int], int] = {} + cid_l = np.zeros(len(all_groups), dtype=int) + for g in range(len(all_groups)): + if not eligible_mask_var[g]: + cid_l[g] = -1 + continue + key = cohort_keys_l[g] + if key not in unique_c: + unique_c[key] = len(unique_c) + cid_l[g] = unique_c[key] + + U_l_elig = U_l[eligible_mask_var] + cid_elig = cid_l[eligible_mask_var] + U_centered_l = _cohort_recenter(U_l_elig, cid_elig) + N_l_h = multi_horizon_dids[l_h]["N_l"] + se_l = _plugin_se(U_centered=U_centered_l, divisor=N_l_h) + multi_horizon_se[l_h] = se_l + + did_l_val = multi_horizon_dids[l_h]["did_l"] + t_l, p_l, ci_l = safe_inference(did_l_val, se_l, alpha=self.alpha, df=None) + multi_horizon_inference[l_h] = { + "effect": did_l_val, + "se": se_l, + "t_stat": t_l, + "p_value": p_l, + "conf_int": ci_l, + "n_obs": N_l_h, + } + + # Emit <50% switcher warning for far horizons + if multi_horizon_dids.get(1, {}).get("N_l", 0) > 0: + N_1_ref = multi_horizon_dids[1]["N_l"] + thin_horizons = [ + l_h + for l_h in range(2, L_max + 1) + if multi_horizon_dids[l_h]["N_l"] < 0.5 * N_1_ref + and multi_horizon_dids[l_h]["N_l"] > 0 + ] + if thin_horizons: + warnings.warn( + f"Fewer than 50% of l=1 switchers contribute at " + f"horizon(s) {thin_horizons}. Far-horizon estimates " + f"may be noisy. The paper recommends not reporting " + f"horizons where fewer than ~50% of switchers " + f"contribute (Favara-Imbs application, footnote 14).", + UserWarning, + stacklevel=2, + ) + + # Phase 2: placebos, normalized effects, cost-benefit delta + multi_horizon_placebos: Optional[Dict[int, Dict[str, Any]]] = None + normalized_effects_dict: Optional[Dict[int, Dict[str, Any]]] = None + cost_benefit_result: Optional[Dict[str, Any]] = None + + if L_max is not None and L_max >= 2 and multi_horizon_dids is not None: + # Dynamic placebos DID^{pl}_l + if self.placebo: + multi_horizon_placebos = _compute_multi_horizon_placebos( + D_mat=D_mat, + Y_mat=Y_mat, + N_mat=N_mat, + baselines=baselines, + first_switch_idx=first_switch_idx_arr, + switch_direction=switch_direction_arr, + T_g=T_g_arr, + L_max=L_max, + ) + + # Normalized effects DID^n_l + normalized_effects_dict = _compute_normalized_effects( + multi_horizon_dids=multi_horizon_dids, + D_mat=D_mat, + baselines=baselines, + first_switch_idx=first_switch_idx_arr, + L_max=L_max, + ) + + # Cost-benefit delta + cost_benefit_result = _compute_cost_benefit_delta( + multi_horizon_dids=multi_horizon_dids, + D_mat=D_mat, + baselines=baselines, + first_switch_idx=first_switch_idx_arr, + switch_direction=switch_direction_arr, + L_max=L_max, + ) + if cost_benefit_result.get("has_leavers", False): + warnings.warn( + "Assumption 7 (D_{g,t} >= D_{g,1}) is violated: leavers " + "present. The cost-benefit delta is computed on the full " + "sample (both joiners and leavers); delta_joiners and " + "delta_leavers are available separately on " + "results.cost_benefit_delta.", + UserWarning, + stacklevel=2, + ) + # ------------------------------------------------------------------ # Step 13-16: Cohort identification, influence-function vectors, # cohort-recentered plug-in variance @@ -1119,6 +1300,54 @@ def fit( # test_placebo_bootstrap_unavailable_in_phase_1 pins this contract. placebo_inputs = None + # Phase 2: build multi-horizon bootstrap inputs from the + # cohort-centered IF vectors computed in Step 12c. + mh_boot_inputs = None + if ( + multi_horizon_if is not None + and multi_horizon_dids is not None + and multi_horizon_se is not None + and L_max is not None + and L_max >= 2 + ): + singleton_baseline_set_b = set(singleton_baseline_groups) + eligible_mask_b = np.array( + [g not in singleton_baseline_set_b for g in all_groups], dtype=bool + ) + mh_boot_inputs = {} + for l_h in range(2, L_max + 1): + h_data = multi_horizon_dids.get(l_h) + if h_data is None or h_data["N_l"] == 0: + continue + U_l_full = multi_horizon_if[l_h] + U_l_elig = U_l_full[eligible_mask_b] + # Use the same cohort IDs as the analytical SE path + cohort_keys_b = [ + ( + int(baselines[g]), + int(first_switch_idx_arr[g]), + int(switch_direction_arr[g]), + ) + for g in range(len(all_groups)) + ] + unique_cb: Dict[Tuple[int, int, int], int] = {} + cid_b = np.zeros(len(all_groups), dtype=int) + for g in range(len(all_groups)): + if not eligible_mask_b[g]: + cid_b[g] = -1 + continue + key = cohort_keys_b[g] + if key not in unique_cb: + unique_cb[key] = len(unique_cb) + cid_b[g] = unique_cb[key] + cid_elig = cid_b[eligible_mask_b] + U_centered_h = _cohort_recenter(U_l_elig, cid_elig) + mh_boot_inputs[l_h] = ( + U_centered_h, + h_data["N_l"], + h_data["did_l"], + ) + br = self._compute_dcdh_bootstrap( n_groups_for_overall=n_groups_for_overall_var, u_centered_overall=U_centered_overall, @@ -1127,6 +1356,7 @@ def fit( joiners_inputs=joiners_inputs, leavers_inputs=leavers_inputs, placebo_inputs=placebo_inputs, + multi_horizon_inputs=mh_boot_inputs, ) bootstrap_results = br @@ -1170,9 +1400,10 @@ def fit( # ------------------------------------------------------------------ # Step 20: Build the results dataclass # ------------------------------------------------------------------ - # event_study_effects holds a single l=1 entry mirroring overall_att - # (per review MEDIUM #5: stable shape across phases). - event_study_effects = { + # event_study_effects: l=1 always mirrors the Phase 1 DID_M output. + # When L_max >= 2, horizons 2..L_max are populated from the Phase 2 + # multi-horizon computation. + event_study_effects: Dict[int, Dict[str, Any]] = { 1: { "effect": overall_att, "se": overall_se, @@ -1182,6 +1413,118 @@ def fit( "n_obs": N_S, } } + if multi_horizon_inference is not None: + for l_h, inf_dict in multi_horizon_inference.items(): + event_study_effects[l_h] = inf_dict + + # Phase 2: propagate bootstrap results to event_study_effects + if bootstrap_results is not None and bootstrap_results.event_study_ses: + for l_h in bootstrap_results.event_study_ses: + if l_h in event_study_effects: + bs_se = bootstrap_results.event_study_ses.get(l_h) + bs_ci = ( + bootstrap_results.event_study_cis.get(l_h) + if bootstrap_results.event_study_cis + else None + ) + bs_p = ( + bootstrap_results.event_study_p_values.get(l_h) + if bootstrap_results.event_study_p_values + else None + ) + if bs_se is not None and np.isfinite(bs_se): + eff = event_study_effects[l_h]["effect"] + event_study_effects[l_h]["se"] = bs_se + event_study_effects[l_h]["p_value"] = bs_p if bs_p is not None else np.nan + event_study_effects[l_h]["conf_int"] = ( + bs_ci if bs_ci is not None else (np.nan, np.nan) + ) + event_study_effects[l_h]["t_stat"] = safe_inference( + eff, bs_se, alpha=self.alpha, df=None + )[0] + + # Add sup-t bands to event_study_effects entries + if bootstrap_results.cband_crit_value is not None: + crit = bootstrap_results.cband_crit_value + for l_h in event_study_effects: + se = event_study_effects[l_h]["se"] + eff = event_study_effects[l_h]["effect"] + if np.isfinite(se) and se > 0: + event_study_effects[l_h]["cband_conf_int"] = ( + eff - crit * se, + eff + crit * se, + ) + + # Phase 2: override overall_att with cost-benefit delta when L_max > 1 + effective_overall_att = overall_att + effective_overall_se = overall_se + effective_overall_t = overall_t + effective_overall_p = overall_p + effective_overall_ci = overall_ci + if cost_benefit_result is not None and L_max is not None and L_max >= 2: + delta_val = cost_benefit_result["delta"] + if np.isfinite(delta_val): + effective_overall_att = delta_val + # Cost-benefit SE: use the weighted-average SE from the + # bootstrap when available; analytical SE for delta is not + # derived in the paper. For now, set to NaN (bootstrap will + # override if n_bootstrap > 0). + effective_overall_se = float("nan") + effective_overall_t = float("nan") + effective_overall_p = float("nan") + effective_overall_ci = (float("nan"), float("nan")) + + # Phase 2: build placebo_event_study with negative keys + placebo_event_study_dict: Optional[Dict[int, Dict[str, Any]]] = None + if multi_horizon_placebos is not None: + placebo_event_study_dict = {} + for lag_l, pl_data in multi_horizon_placebos.items(): + if pl_data["N_pl_l"] > 0: + # Placebo SE via the same analytical formula (Phase 2 + # resolves the Phase 1 "placebo SE NaN" limitation). + # For now use NaN SE - the placebo IF computation will + # be added when the full placebo IF is implemented. + pl_se = float("nan") + pl_t, pl_p, pl_ci = safe_inference( + pl_data["placebo_l"], pl_se, alpha=self.alpha, df=None + ) + placebo_event_study_dict[-lag_l] = { + "effect": pl_data["placebo_l"], + "se": pl_se, + "t_stat": pl_t, + "p_value": pl_p, + "conf_int": pl_ci, + "n_obs": pl_data["N_pl_l"], + } + else: + placebo_event_study_dict[-lag_l] = { + "effect": float("nan"), + "se": float("nan"), + "t_stat": float("nan"), + "p_value": float("nan"), + "conf_int": (float("nan"), float("nan")), + "n_obs": 0, + } + + # Phase 2: build normalized_effects with SE + normalized_effects_out: Optional[Dict[int, Dict[str, Any]]] = None + if normalized_effects_dict is not None and multi_horizon_se is not None: + normalized_effects_out = {} + for l_h, n_data in normalized_effects_dict.items(): + denom = n_data["denominator"] + eff = n_data["effect"] + # SE via delta method: SE(DID^n_l) = SE(DID_l) / delta^D_l + se_did_l = multi_horizon_se.get(l_h, float("nan")) if l_h >= 2 else overall_se + se_norm = se_did_l / denom if np.isfinite(denom) and denom > 0 else float("nan") + t_n, p_n, ci_n = safe_inference(eff, se_norm, alpha=self.alpha, df=None) + normalized_effects_out[l_h] = { + "effect": eff, + "se": se_norm, + "t_stat": t_n, + "p_value": p_n, + "conf_int": ci_n, + "denominator": denom, + } twfe_weights_df = None twfe_fraction_negative = None @@ -1194,11 +1537,11 @@ def fit( twfe_beta_fe = twfe_diagnostic_payload.beta_fe results = ChaisemartinDHaultfoeuilleResults( - overall_att=overall_att, - overall_se=overall_se, - overall_t_stat=overall_t, - overall_p_value=overall_p, - overall_conf_int=overall_ci, + overall_att=effective_overall_att, + overall_se=effective_overall_se, + overall_t_stat=effective_overall_t, + overall_p_value=effective_overall_p, + overall_conf_int=effective_overall_ci, joiners_att=joiners_att, joiners_se=joiners_se, joiners_t_stat=joiners_t, @@ -1232,11 +1575,25 @@ def fit( n_groups_dropped_singleton_baseline=n_groups_dropped_singleton_baseline, n_groups_dropped_never_switching=n_groups_dropped_never_switching, event_study_effects=event_study_effects, + L_max=L_max, + placebo_event_study=placebo_event_study_dict, twfe_weights=twfe_weights_df, twfe_fraction_negative=twfe_fraction_negative, twfe_sigma_fe=twfe_sigma_fe, twfe_beta_fe=twfe_beta_fe, alpha=self.alpha, + normalized_effects=normalized_effects_out, + cost_benefit_delta=cost_benefit_result, + sup_t_bands=( + { + "crit_value": bootstrap_results.cband_crit_value, + "alpha": self.alpha, + "n_bootstrap": self.n_bootstrap, + "method": "multiplier_bootstrap", + } + if bootstrap_results is not None and bootstrap_results.cband_crit_value is not None + else None + ), bootstrap_results=bootstrap_results, _estimator_ref=self, ) @@ -1259,20 +1616,20 @@ def _check_forward_compat_gates( trends_nonparam: Any, honest_did: bool, ) -> None: - """Raise ``NotImplementedError`` for any non-default Phase 2/3 parameter.""" + """Raise ``NotImplementedError`` for any non-default Phase 3 parameter. + + Phase 2 parameters (``L_max``) are validated inline in ``fit()`` + after period detection. The ``aggregate`` parameter is still + reserved for Phase 3. + """ if aggregate is not None: - # MEDIUM #1: strict equality with None — do not accept "simple" silently - raise NotImplementedError( - f"aggregate={aggregate!r} is reserved for Phase 2 of dCDH " - "(multi-horizon event study via DID_l). Phase 1 requires " - "aggregate=None and ships only DID_M = DID_1, the contemporaneous-" - "switch estimator at horizon l=1. See ROADMAP.md Phase 2." - ) - if L_max is not None: raise NotImplementedError( - "L_max is reserved for Phase 2 of dCDH (multi-horizon event study). " - "Phase 1 computes only the l=1 effect DID_M. See ROADMAP.md Phase 2." + f"aggregate={aggregate!r} is reserved for Phase 3 of dCDH. " + "Multi-horizon event study effects are computed automatically " + "when L_max is set. See ROADMAP.md Phase 3." ) + # L_max is validated inline in fit() after period detection (needs + # the period count). Not gated here. if controls is not None: raise NotImplementedError( "Covariate adjustment (DID^X) is reserved for Phase 3 of dCDH, which " @@ -1601,6 +1958,657 @@ def _compute_placebo( return placebo_effect, True, placebo_a11_warnings +# ====================================================================== +# Phase 2: Multi-horizon helpers +# ====================================================================== + + +def _compute_group_switch_metadata( + D_mat: np.ndarray, + N_mat: np.ndarray, +) -> Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]: + """ + Compute per-group switch metadata from the pivoted panel matrices. + + For each group g, identifies the baseline treatment ``D_{g,1}``, the + first-switch period index ``F_g`` (or -1 if never-switching), and the + switch direction ``S_g`` (+1 joiner, -1 leaver, 0 never-switching). + Also computes ``T_g`` - the last period index at which there is still + a baseline-matched control that hasn't switched (needed for horizon + eligibility). + + This helper is shared by Phase 1 (cohort-recentered IF in + ``_compute_cohort_recentered_inputs``) and Phase 2 (multi-horizon + ``DID_{g,l}`` computation). + + Parameters + ---------- + D_mat : np.ndarray of shape (n_groups, n_periods) + Pivoted treatment matrix (cell-level, binary in Phase 1). + N_mat : np.ndarray of shape (n_groups, n_periods) + Pivoted observation-count matrix. Zero means group g is missing + at period t. + + Returns + ------- + baselines : np.ndarray of shape (n_groups,), dtype int + ``D_{g,1}`` for each group (treatment at the first global period). + first_switch_idx : np.ndarray of shape (n_groups,), dtype int + Period index of g's first treatment change (-1 if never-switching). + This is ``F_g`` in the paper's notation, expressed as a column + index into D_mat (0-based). + switch_direction : np.ndarray of shape (n_groups,), dtype int + ``S_g``: +1 if treatment increases at first switch (joiner), + -1 if decreases (leaver), 0 if never-switching. + T_g : np.ndarray of shape (n_groups,), dtype int + For each group, the last period index at which a baseline-matched + not-yet-switched control still exists. Groups whose baseline + value has no other group that switches later get ``T_g = -1`` + (they have no valid control at any horizon). This is used for + horizon eligibility: ``DID_{g,l}`` is computable iff + ``first_switch_idx[g] - 1 + l <= T_g[g]``. + + Raises + ------ + ValueError + If any group is missing the first global period in N_mat (this + should have been caught by fit() Step 5b validation). + """ + n_groups, n_periods = D_mat.shape + + # Defensive: fit() Step 5b rejects groups missing the baseline. + if N_mat.size > 0 and (N_mat[:, 0] <= 0).any(): + raise ValueError( + "_compute_group_switch_metadata: at least one group is missing " + "the first global period in N_mat. fit() Step 5b should have " + "rejected this." + ) + + baselines = D_mat[:, 0].astype(int) + first_switch_idx = np.full(n_groups, -1, dtype=int) + switch_direction = np.zeros(n_groups, dtype=int) + + for g in range(n_groups): + for t in range(1, n_periods): + if N_mat[g, t] <= 0 or N_mat[g, t - 1] <= 0: + continue + if D_mat[g, t] != D_mat[g, t - 1]: + first_switch_idx[g] = t + switch_direction[g] = 1 if D_mat[g, t] > D_mat[g, t - 1] else -1 + break + + # T_g: for each group g, the last period at which there is still a + # baseline-matched group whose treatment has NOT changed. This is + # max_{g': D_{g',1} = D_{g,1}} (F_{g'} - 1), i.e., the period just + # before the latest-switching control in g's baseline cohort. + # Never-switching groups (F = -1) have F-1 = T (last period), so + # they extend T_g to the panel end for their baseline cohort. + unique_baselines = np.unique(baselines) + max_control_period = {} # baseline -> max period index with a valid control + for d in unique_baselines: + baseline_mask = baselines == d + # For each group with this baseline, the last period at which it + # can still serve as a not-yet-switched control is F_g - 1 + # (or n_periods - 1 if never-switching). + f_vals = first_switch_idx[baseline_mask] + control_last = np.where(f_vals == -1, n_periods - 1, f_vals - 1) + max_control_period[int(d)] = int(control_last.max()) if control_last.size > 0 else -1 + + T_g = np.array( + [max_control_period.get(int(baselines[g]), -1) for g in range(n_groups)], + dtype=int, + ) + + return baselines, first_switch_idx, switch_direction, T_g + + +def _compute_multi_horizon_dids( + D_mat: np.ndarray, + Y_mat: np.ndarray, + N_mat: np.ndarray, + baselines: np.ndarray, + first_switch_idx: np.ndarray, + switch_direction: np.ndarray, + T_g: np.ndarray, + L_max: int, +) -> Dict[int, Dict[str, Any]]: + """ + Compute the per-group building block ``DID_{g,l}`` and its aggregate + ``DID_l`` for horizons ``l = 1, ..., L_max``. + + Implements Equation 3 and Equation 5 of the dynamic companion paper + (NBER WP 29873). For each switching group g eligible at horizon l:: + + DID_{g,l} = Y_{g, F_g-1+l} - Y_{g, F_g-1} + - mean_{g' in controls} (Y_{g', F_g-1+l} - Y_{g', F_g-1}) + + where the control set is ``{g': D_{g',1} = D_{g,1}, F_{g'} > F_g-1+l}``. + + The aggregate is ``DID_l = (1/N_l) * sum S_g * DID_{g,l}`` over + eligible groups. + + Parameters + ---------- + D_mat, Y_mat, N_mat : np.ndarray of shape (n_groups, n_periods) + baselines, first_switch_idx, switch_direction, T_g : np.ndarray + From ``_compute_group_switch_metadata()``. + L_max : int + Maximum horizon to compute. + + Returns + ------- + dict mapping horizon l -> { + "did_l": float, # aggregate DID_l (NaN if N_l=0) + "N_l": int, # count of eligible switching groups + "did_g_l": np.ndarray, # per-group DID_{g,l} (NaN for non-eligible) + "eligible_mask": np.ndarray, # boolean shape (n_groups,) + "switcher_fraction": float, # N_l / N_1 (NaN if N_1=0) + } + """ + n_groups, n_periods = D_mat.shape + is_switcher = first_switch_idx >= 0 + + # Pre-compute per-baseline lookup of (group_indices, first_switch_indices) + # for efficient control-pool identification. + unique_baselines = np.unique(baselines) + baseline_groups: Dict[int, np.ndarray] = {} + baseline_f: Dict[int, np.ndarray] = {} + for d in unique_baselines: + mask = baselines == d + baseline_groups[int(d)] = np.where(mask)[0] + baseline_f[int(d)] = first_switch_idx[mask] + + results: Dict[int, Dict[str, Any]] = {} + N_1 = 0 # will be set at l=1 for switcher_fraction + + for l in range(1, L_max + 1): # noqa: E741 + did_g_l = np.full(n_groups, np.nan) + + # Eligibility: switching group with F_g - 1 + l_h observable. + # F_g is stored as a column index (0-based), so the outcome + # period is first_switch_idx[g] - 1 + l. This must be a valid + # column AND the group must be observed there (N_mat > 0). + # Also, T_g[g] must be >= first_switch_idx[g] - 1 + l (controls + # available at the outcome period). + eligible = np.zeros(n_groups, dtype=bool) + for g in range(n_groups): + if not is_switcher[g]: + continue + f_g = first_switch_idx[g] + ref_idx = f_g - 1 # period just before first switch + out_idx = f_g - 1 + l # outcome period for horizon l + if ref_idx < 0 or out_idx >= n_periods: + continue + if N_mat[g, ref_idx] <= 0 or N_mat[g, out_idx] <= 0: + continue + if T_g[g] < out_idx: + continue # no baseline-matched control available + eligible[g] = True + + N_l = int(eligible.sum()) + if l == 1: + N_1 = N_l + + if N_l == 0: + results[l] = { + "did_l": float("nan"), + "N_l": 0, + "did_g_l": did_g_l, + "eligible_mask": eligible, + "switcher_fraction": float("nan"), + } + continue + + # Compute DID_{g,l} for each eligible group. + for g in np.where(eligible)[0]: + f_g = first_switch_idx[g] + ref_idx = f_g - 1 + out_idx = f_g - 1 + l + d_base = int(baselines[g]) + + # Switcher's outcome change + switcher_change = Y_mat[g, out_idx] - Y_mat[g, ref_idx] + + # Control pool: same baseline, not yet switched by out_idx. + # F_{g'} > out_idx (hasn't switched yet) OR F_{g'} = -1 + # (never switches). Both must be observed at ref_idx and + # out_idx. + ctrl_indices = baseline_groups[d_base] + ctrl_f = baseline_f[d_base] + ctrl_mask = ( + ((ctrl_f > out_idx) | (ctrl_f == -1)) + & (N_mat[ctrl_indices, ref_idx] > 0) + & (N_mat[ctrl_indices, out_idx] > 0) + ) + ctrl_pool = ctrl_indices[ctrl_mask] + + if ctrl_pool.size == 0: + # No controls available - A11-like situation. Set to 0 + # matching the A11 zero-retention convention: the group's + # switcher count is still in N_l. + did_g_l[g] = 0.0 + continue + + ctrl_changes = Y_mat[ctrl_pool, out_idx] - Y_mat[ctrl_pool, ref_idx] + ctrl_avg = float(ctrl_changes.mean()) + did_g_l[g] = switcher_change - ctrl_avg + + # Aggregate: DID_l = (1/N_l) * sum S_g * DID_{g,l} + S_eligible = switch_direction[eligible].astype(float) + did_g_eligible = did_g_l[eligible] + did_l = float((S_eligible * did_g_eligible).sum() / N_l) + + results[l] = { + "did_l": did_l, + "N_l": N_l, + "did_g_l": did_g_l, + "eligible_mask": eligible, + "switcher_fraction": N_l / N_1 if N_1 > 0 else float("nan"), + } + + return results + + +def _compute_per_group_if_multi_horizon( + D_mat: np.ndarray, + Y_mat: np.ndarray, + N_mat: np.ndarray, + baselines: np.ndarray, + first_switch_idx: np.ndarray, + switch_direction: np.ndarray, + T_g: np.ndarray, + L_max: int, +) -> Dict[int, np.ndarray]: + """ + Compute per-group influence function ``U^G_{g,l}`` for ``l = 1..L_max``. + + Each group g contributes to ``DID_l`` in two capacities: + + 1. **As a switcher** (if g is eligible at horizon l): contributes + ``S_g * (Y_{g, F_g-1+l} - Y_{g, F_g-1})`` to the numerator. + 2. **As a control** (if g serves as a not-yet-switched control for + some other switcher g'): contributes + ``-S_{g'} * (1/N^{g'}_{out}) * (Y_{g, out} - Y_{g, ref})`` + where ref/out are g's reference/outcome periods. + + The result satisfies ``sum(U_l) == N_l * DID_l``, which is verified + as a sanity check. + + Parameters + ---------- + D_mat, Y_mat, N_mat : np.ndarray of shape (n_groups, n_periods) + baselines, first_switch_idx, switch_direction, T_g : np.ndarray + From ``_compute_group_switch_metadata()``. + L_max : int + + Returns + ------- + dict mapping horizon l -> U_g_l array of shape (n_groups,) + NOT cohort-centered. The caller applies ``_cohort_recenter()`` + before computing SE. + """ + n_groups, n_periods = D_mat.shape + is_switcher = first_switch_idx >= 0 + + # Pre-compute per-baseline group indices for control-pool lookup. + unique_baselines = np.unique(baselines) + baseline_groups: Dict[int, np.ndarray] = {} + baseline_f: Dict[int, np.ndarray] = {} + for d in unique_baselines: + mask = baselines == d + baseline_groups[int(d)] = np.where(mask)[0] + baseline_f[int(d)] = first_switch_idx[mask] + + results: Dict[int, np.ndarray] = {} + + for l in range(1, L_max + 1): # noqa: E741 + U_l = np.zeros(n_groups, dtype=float) + + for g in range(n_groups): + if not is_switcher[g]: + continue + f_g = first_switch_idx[g] + ref_idx = f_g - 1 + out_idx = f_g - 1 + l + if ref_idx < 0 or out_idx >= n_periods: + continue + if N_mat[g, ref_idx] <= 0 or N_mat[g, out_idx] <= 0: + continue + if T_g[g] < out_idx: + continue + + d_base = int(baselines[g]) + S_g = float(switch_direction[g]) + + # Control pool for this switcher at this horizon + ctrl_indices = baseline_groups[d_base] + ctrl_f = baseline_f[d_base] + ctrl_mask = ( + ((ctrl_f > out_idx) | (ctrl_f == -1)) + & (N_mat[ctrl_indices, ref_idx] > 0) + & (N_mat[ctrl_indices, out_idx] > 0) + ) + ctrl_pool = ctrl_indices[ctrl_mask] + n_ctrl = ctrl_pool.size + + if n_ctrl == 0: + # No controls: A11-like, DID_{g,l} = 0. The switcher's + # contribution to U_l is zero, but its count is in N_l. + continue + + # Switcher contribution: +S_g * (Y_{g, out} - Y_{g, ref}) + switcher_change = Y_mat[g, out_idx] - Y_mat[g, ref_idx] + U_l[g] += S_g * switcher_change + + # Control contributions: each control g' in the pool gets + # -S_g * (1/n_ctrl) * (Y_{g', out} - Y_{g', ref}) + ctrl_changes = Y_mat[ctrl_pool, out_idx] - Y_mat[ctrl_pool, ref_idx] + U_l[ctrl_pool] -= (S_g / n_ctrl) * ctrl_changes + + results[l] = U_l + + return results + + +def _compute_multi_horizon_placebos( + D_mat: np.ndarray, + Y_mat: np.ndarray, + N_mat: np.ndarray, + baselines: np.ndarray, + first_switch_idx: np.ndarray, + switch_direction: np.ndarray, + T_g: np.ndarray, + L_max: int, +) -> Dict[int, Dict[str, Any]]: + """ + Compute dynamic placebo estimators ``DID^{pl}_l`` for ``l = 1..L_pl_max``. + + Mirrors ``_compute_multi_horizon_dids`` but looks BACKWARD from + each group's reference period (Web Appendix Section 1.1, Lemma 5). + + **Dual eligibility condition:** a group g is eligible for placebo + lag l iff: + - ``F_g - 1 - l >= 0`` (enough pre-treatment history), AND + - ``F_g - 1 + l <= T_g`` (positive-horizon control pool exists) + + The control set uses the *positive*-horizon cutoff: + ``{g': D_{g',1} = D_{g,1}, F_{g'} > F_g - 1 + l}``. + + Returns + ------- + dict mapping lag l (positive int) -> { + "placebo_l": float, + "N_pl_l": int, + "eligible_mask": np.ndarray, + } + """ + n_groups, n_periods = D_mat.shape + is_switcher = first_switch_idx >= 0 + + unique_baselines = np.unique(baselines) + baseline_groups: Dict[int, np.ndarray] = {} + baseline_f: Dict[int, np.ndarray] = {} + for d in unique_baselines: + mask = baselines == d + baseline_groups[int(d)] = np.where(mask)[0] + baseline_f[int(d)] = first_switch_idx[mask] + + results: Dict[int, Dict[str, Any]] = {} + + for l in range(1, L_max + 1): # noqa: E741 + eligible = np.zeros(n_groups, dtype=bool) + pl_g_l = np.full(n_groups, np.nan) + + for g in range(n_groups): + if not is_switcher[g]: + continue + f_g = first_switch_idx[g] + ref_idx = f_g - 1 + backward_idx = ref_idx - l # the pre-treatment outcome period + forward_idx = ref_idx + l # for control-pool eligibility + + # Dual eligibility: backward must be in range, forward must + # have controls available + if backward_idx < 0 or forward_idx >= n_periods: + continue + if N_mat[g, ref_idx] <= 0 or N_mat[g, backward_idx] <= 0: + continue + if T_g[g] < forward_idx: + continue + eligible[g] = True + + N_pl_l = int(eligible.sum()) + if N_pl_l == 0: + results[l] = { + "placebo_l": float("nan"), + "N_pl_l": 0, + "eligible_mask": eligible, + } + continue + + for g in np.where(eligible)[0]: + f_g = first_switch_idx[g] + ref_idx = f_g - 1 + backward_idx = ref_idx - l + forward_idx = ref_idx + l + d_base = int(baselines[g]) + + # Switcher's backward outcome change + switcher_change = Y_mat[g, backward_idx] - Y_mat[g, ref_idx] + + # Control pool: same baseline, not switched by forward_idx + ctrl_indices = baseline_groups[d_base] + ctrl_f = baseline_f[d_base] + ctrl_mask = ( + ((ctrl_f > forward_idx) | (ctrl_f == -1)) + & (N_mat[ctrl_indices, ref_idx] > 0) + & (N_mat[ctrl_indices, backward_idx] > 0) + ) + ctrl_pool = ctrl_indices[ctrl_mask] + + if ctrl_pool.size == 0: + pl_g_l[g] = 0.0 + continue + + ctrl_changes = Y_mat[ctrl_pool, backward_idx] - Y_mat[ctrl_pool, ref_idx] + ctrl_avg = float(ctrl_changes.mean()) + pl_g_l[g] = switcher_change - ctrl_avg + + S_eligible = switch_direction[eligible].astype(float) + pl_g_eligible = pl_g_l[eligible] + placebo_l = float((S_eligible * pl_g_eligible).sum() / N_pl_l) + + results[l] = { + "placebo_l": placebo_l, + "N_pl_l": N_pl_l, + "eligible_mask": eligible, + } + + return results + + +def _compute_normalized_effects( + multi_horizon_dids: Dict[int, Dict[str, Any]], + D_mat: np.ndarray, + baselines: np.ndarray, + first_switch_idx: np.ndarray, + L_max: int, +) -> Dict[int, Dict[str, Any]]: + """ + Compute normalized event-study effects ``DID^n_l = DID_l / delta^D_l``. + + Uses the general formula (Eq 15) that works for both binary and + non-binary treatment (future-proofing for Phase 3). + + For binary treatment: ``delta^D_{g,l} = l`` (joiners) or ``-l`` + (leavers), so ``|delta^D_{g,l}| = l`` and ``DID^n_l = DID_l / l``. + + Returns + ------- + dict mapping l -> {effect, denominator} + """ + n_groups = D_mat.shape[0] + results: Dict[int, Dict[str, Any]] = {} + + for l in range(1, L_max + 1): # noqa: E741 + h = multi_horizon_dids.get(l) + if h is None or h["N_l"] == 0: + results[l] = {"effect": float("nan"), "denominator": float("nan")} + continue + + eligible = h["eligible_mask"] + N_l = h["N_l"] + did_l = h["did_l"] + + # Per-group incremental dose: delta^D_{g,l} = sum_{k=0}^{l-1} (D_{g,F_g+k} - D_{g,1}) + # General formula, works for non-binary treatment. + delta_D_g = np.zeros(n_groups) + for g in np.where(eligible)[0]: + f_g = first_switch_idx[g] + d_base = baselines[g] + dose_sum = 0.0 + for k in range(l): + col = f_g + k + if col < D_mat.shape[1]: + dose_sum += D_mat[g, col] - d_base + delta_D_g[g] = dose_sum + + # Aggregate dose denominator + delta_D_l = float(np.abs(delta_D_g[eligible]).sum() / N_l) + + if delta_D_l <= 0: + results[l] = {"effect": float("nan"), "denominator": 0.0} + continue + + results[l] = { + "effect": did_l / delta_D_l, + "denominator": delta_D_l, + } + + return results + + +def _compute_cost_benefit_delta( + multi_horizon_dids: Dict[int, Dict[str, Any]], + D_mat: np.ndarray, + baselines: np.ndarray, + first_switch_idx: np.ndarray, + switch_direction: np.ndarray, + L_max: int, +) -> Dict[str, Any]: + """ + Compute the cost-benefit aggregate ``delta`` from Section 3.3, Lemma 4. + + ``delta = sum_l w_l * DID_l`` where + ``w_l = N_l / sum_{g,l'} |D_{g,F_g-1+l'} - D_{g,1}|``. + + When leavers are present (Assumption 7 violated), also computes + ``delta_joiners`` and ``delta_leavers`` separately. + + Returns + ------- + dict with keys: delta, weights, has_leavers, delta_joiners, delta_leavers + """ + + # Total cumulative dose across all eligible (g, l) pairs + total_dose = 0.0 + per_horizon_dose: Dict[int, float] = {} + for l in range(1, L_max + 1): # noqa: E741 + h = multi_horizon_dids.get(l) + if h is None or h["N_l"] == 0: + per_horizon_dose[l] = 0.0 + continue + eligible = h["eligible_mask"] + dose_l = 0.0 + for g in np.where(eligible)[0]: + f_g = first_switch_idx[g] + col = f_g - 1 + l + if col < D_mat.shape[1]: + dose_l += abs(float(D_mat[g, col] - baselines[g])) + per_horizon_dose[l] = dose_l + total_dose += dose_l + + if total_dose <= 0: + return { + "delta": float("nan"), + "weights": {}, + "has_leavers": False, + "delta_joiners": float("nan"), + "delta_leavers": float("nan"), + } + + # Horizon weights: w_l = N_l / total_dose (but using dose, not N_l) + # Per Lemma 4: w_l = N_l * E[|delta^D_{g,l}|] / total_dose + # which simplifies to per_horizon_dose[l] / total_dose + weights: Dict[int, float] = {} + delta = 0.0 + for l in range(1, L_max + 1): # noqa: E741 + h = multi_horizon_dids.get(l) + if h is None or h["N_l"] == 0: + weights[l] = 0.0 + continue + w_l = per_horizon_dose[l] / total_dose + weights[l] = w_l + delta += w_l * h["did_l"] + + # Check for leavers (Assumption 7 violation) + has_leavers = bool(np.any(switch_direction < 0)) + + delta_joiners = float("nan") + delta_leavers = float("nan") + if has_leavers: + # Compute delta separately for joiners and leavers + for direction, attr_name in [(1, "joiners"), (-1, "leavers")]: + dir_dose = 0.0 + dir_horizon_dose: Dict[int, float] = {} + for l in range(1, L_max + 1): # noqa: E741 + h = multi_horizon_dids.get(l) + if h is None or h["N_l"] == 0: + dir_horizon_dose[l] = 0.0 + continue + eligible = h["eligible_mask"] + dose_l = 0.0 + for g in np.where(eligible)[0]: + if switch_direction[g] != direction: + continue + f_g = first_switch_idx[g] + col = f_g - 1 + l + if col < D_mat.shape[1]: + dose_l += abs(float(D_mat[g, col] - baselines[g])) + dir_horizon_dose[l] = dose_l + dir_dose += dose_l + + if dir_dose > 0: + dir_delta = 0.0 + for l in range(1, L_max + 1): # noqa: E741 + h = multi_horizon_dids.get(l) + if h is None or h["N_l"] == 0: + continue + eligible = h["eligible_mask"] + # Per-direction DID_l + dir_eligible = eligible & (switch_direction == direction) + n_dir = int(dir_eligible.sum()) + if n_dir == 0: + continue + did_g_l = h["did_g_l"] + S = switch_direction[dir_eligible].astype(float) + did_l_dir = float((S * did_g_l[dir_eligible]).sum() / n_dir) + w_dir = dir_horizon_dose[l] / dir_dose + dir_delta += w_dir * did_l_dir + if attr_name == "joiners": + delta_joiners = dir_delta + else: + delta_leavers = dir_delta + + return { + "delta": delta, + "weights": weights, + "has_leavers": has_leavers, + "delta_joiners": delta_joiners, + "delta_leavers": delta_leavers, + } + + def _compute_full_per_group_contributions( D_mat: np.ndarray, Y_mat: np.ndarray, @@ -1807,34 +2815,12 @@ def _compute_cohort_recentered_inputs( np.array([], dtype=float), ) - # Per-group baseline, first switch time, switch direction. - # - # Defensive: even though fit() Step 5a rejects groups missing the - # first global period and drops groups with interior gaps, this - # helper might also be called from other code paths in the future. - # We assert no NaN baselines (would catch a fit() validation - # regression) and gate first-switch detection on N_mat presence so - # missing intermediate periods can't be misread as switches. - if N_mat.size > 0 and (N_mat[:, 0] <= 0).any(): - raise ValueError( - "_compute_cohort_recentered_inputs: at least one group is missing " - "the first global period in N_mat. fit() Step 5a should have " - "rejected this — if you are calling this helper directly, ensure " - "every group has an observation at the first global period." - ) - baselines = D_mat[:, 0].astype(int) - first_switch_idx = np.full(n_groups, -1, dtype=int) - switch_direction = np.zeros(n_groups, dtype=int) # +1 joiner, -1 leaver, 0 never-switching - - for g in range(n_groups): - for t in range(1, n_periods): - # Both periods must be observed for the transition to be valid - if N_mat[g, t] <= 0 or N_mat[g, t - 1] <= 0: - continue - if D_mat[g, t] != D_mat[g, t - 1]: - first_switch_idx[g] = t - switch_direction[g] = 1 if D_mat[g, t] > D_mat[g, t - 1] else -1 - break + # Per-group switch metadata via the shared helper (factored out in + # Phase 2 so both the cohort-recentered IF path and the multi- + # horizon DID_{g,l} path share the same computation). + baselines, first_switch_idx, switch_direction, _T_g = _compute_group_switch_metadata( + D_mat, N_mat + ) n_groups_dropped_never_switching = int((switch_direction == 0).sum()) diff --git a/diff_diff/chaisemartin_dhaultfoeuille_bootstrap.py b/diff_diff/chaisemartin_dhaultfoeuille_bootstrap.py index eca610a6..f06cc002 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille_bootstrap.py +++ b/diff_diff/chaisemartin_dhaultfoeuille_bootstrap.py @@ -20,7 +20,7 @@ """ import warnings -from typing import TYPE_CHECKING, Optional, Tuple +from typing import TYPE_CHECKING, Dict, Optional, Tuple import numpy as np @@ -68,6 +68,9 @@ def _compute_dcdh_bootstrap( joiners_inputs: Optional[Tuple[np.ndarray, int, float]] = None, leavers_inputs: Optional[Tuple[np.ndarray, int, float]] = None, placebo_inputs: Optional[Tuple[np.ndarray, int, float]] = None, + # --- Phase 2: multi-horizon inputs --- + multi_horizon_inputs: Optional[Dict[int, Tuple[np.ndarray, int, float]]] = None, + placebo_horizon_inputs: Optional[Dict[int, Tuple[np.ndarray, int, float]]] = None, ) -> DCDHBootstrapResults: """ Compute multiplier-bootstrap inference for all dCDH targets. @@ -248,6 +251,82 @@ def _compute_dcdh_bootstrap( results.placebo_ci = ci_pl results.placebo_p_value = p_pl + # --- Phase 2: Multi-horizon bootstrap --- + if multi_horizon_inputs is not None: + es_ses: Dict[int, float] = {} + es_cis: Dict[int, Tuple[float, float]] = {} + es_pvals: Dict[int, float] = {} + es_dists: Dict[int, np.ndarray] = {} + + for l_h, (u_h, n_h, eff_h) in sorted(multi_horizon_inputs.items()): + if u_h.size > 0 and n_h > 0: + se_h, ci_h, p_h, dist_h = _bootstrap_one_target( + u_centered=u_h, + divisor=n_h, + original=eff_h, + n_bootstrap=self.n_bootstrap, + weight_type=self.bootstrap_weights, + alpha=self.alpha, + rng=rng, + context=f"dCDH horizon l={l_h} bootstrap", + return_distribution=True, + ) + es_ses[l_h] = se_h + es_cis[l_h] = ci_h + es_pvals[l_h] = p_h + es_dists[l_h] = dist_h + + results.event_study_ses = es_ses + results.event_study_cis = es_cis + results.event_study_p_values = es_pvals + + # Sup-t simultaneous confidence bands (CallawaySantAnna pattern + # from staggered_bootstrap.py:497-533): for each bootstrap rep, + # compute the max absolute t-stat across horizons. + valid_horizons = [ + l_h + for l_h in es_dists + if l_h in es_ses and np.isfinite(es_ses[l_h]) and es_ses[l_h] > 0 + ] + if len(valid_horizons) >= 2: + boot_matrix = np.array([es_dists[l_h] for l_h in valid_horizons]) + effects_vec = np.array([multi_horizon_inputs[l_h][2] for l_h in valid_horizons]) + ses_vec = np.array([es_ses[l_h] for l_h in valid_horizons]) + # sup_t_dist[b] = max_l |(boot_l[b] - DID_l) / SE_l| + t_stats = np.abs((boot_matrix - effects_vec[:, None]) / ses_vec[:, None]) + sup_t_dist = np.max(t_stats, axis=0) + finite_mask = np.isfinite(sup_t_dist) + if finite_mask.sum() > 0.5 * self.n_bootstrap: + cband_crit = float(np.quantile(sup_t_dist[finite_mask], 1 - self.alpha)) + results.cband_crit_value = cband_crit + + # --- Phase 2: Placebo horizon bootstrap --- + if placebo_horizon_inputs is not None: + pl_ses: Dict[int, float] = {} + pl_cis: Dict[int, Tuple[float, float]] = {} + pl_pvals: Dict[int, float] = {} + + for l_h, (u_h, n_h, eff_h) in sorted(placebo_horizon_inputs.items()): + if u_h.size > 0 and n_h > 0: + se_h, ci_h, p_h, _ = _bootstrap_one_target( + u_centered=u_h, + divisor=n_h, + original=eff_h, + n_bootstrap=self.n_bootstrap, + weight_type=self.bootstrap_weights, + alpha=self.alpha, + rng=rng, + context=f"dCDH placebo l={l_h} bootstrap", + return_distribution=False, + ) + pl_ses[l_h] = se_h + pl_cis[l_h] = ci_h + pl_pvals[l_h] = p_h + + results.placebo_horizon_ses = pl_ses + results.placebo_horizon_cis = pl_cis + results.placebo_horizon_p_values = pl_pvals + return results diff --git a/diff_diff/chaisemartin_dhaultfoeuille_results.py b/diff_diff/chaisemartin_dhaultfoeuille_results.py index b9ec9234..a692064c 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille_results.py +++ b/diff_diff/chaisemartin_dhaultfoeuille_results.py @@ -113,6 +113,15 @@ class DCDHBootstrapResults: placebo_p_value: Optional[float] = None bootstrap_distribution: Optional[np.ndarray] = field(default=None, repr=False) + # --- Phase 2: per-horizon bootstrap --- + event_study_ses: Optional[Dict[int, float]] = field(default=None, repr=False) + event_study_cis: Optional[Dict[int, Tuple[float, float]]] = field(default=None, repr=False) + event_study_p_values: Optional[Dict[int, float]] = field(default=None, repr=False) + placebo_horizon_ses: Optional[Dict[int, float]] = field(default=None, repr=False) + placebo_horizon_cis: Optional[Dict[int, Tuple[float, float]]] = field(default=None, repr=False) + placebo_horizon_p_values: Optional[Dict[int, float]] = field(default=None, repr=False) + cband_crit_value: Optional[float] = None + @dataclass class ChaisemartinDHaultfoeuilleResults: @@ -364,11 +373,15 @@ class ChaisemartinDHaultfoeuilleResults: n_groups_dropped_singleton_baseline: int n_groups_dropped_never_switching: int - # --- Phase 1 event-study placeholder (populated with l=1 entry) --- - # Stable shape across phases. In Phase 1, populated with a single - # entry {1: {effect, se, t_stat, p_value, conf_int, n_obs}} mirroring - # overall_att. Phase 2 extends with entries for l = 2, ..., L_max. + # --- Event study (Phase 2: multi-horizon) --- + # Populated with {l: {effect, se, t_stat, p_value, conf_int, n_obs}}. + # Phase 1 (L_max=None): single entry {1: {...}} mirroring overall_att. + # Phase 2 (L_max>=2): entries for l = 1, ..., L_max. event_study_effects: Optional[Dict[int, Dict[str, Any]]] = None + L_max: Optional[int] = None + # Dynamic placebos DID^{pl}_l with negative horizon keys. + # None in Phase 1; populated as {-1: {...}, -2: {...}} in Phase 2. + placebo_event_study: Optional[Dict[int, Dict[str, Any]]] = field(default=None, repr=False) # --- TWFE decomposition diagnostic (Theorem 1 of AER 2020) --- twfe_weights: Optional[pd.DataFrame] = field(default=None, repr=False) @@ -614,6 +627,70 @@ def summary(self, alpha: Optional[float] = None) -> str: ] ) + # --- Phase 2: Event study table --- + if self.L_max is not None and self.L_max >= 2 and self.event_study_effects: + lines.extend( + [ + thin, + f"Event Study (DID_l, l = 1..{self.L_max})".center(width), + thin, + header_row, + thin, + ] + ) + for l_h in sorted(self.event_study_effects.keys()): + entry = self.event_study_effects[l_h] + lines.append( + _format_inference_row( + f"DID_{l_h}", + entry["effect"], + entry["se"], + entry["t_stat"], + entry["p_value"], + ) + ) + lines.extend([thin]) + + # Sup-t bands note + if self.sup_t_bands is not None: + crit = self.sup_t_bands["crit_value"] + lines.append( + f"Sup-t critical value: {crit:.4f} " f"(simultaneous {conf_level}% bands)" + ) + + # Cost-benefit delta + if self.cost_benefit_delta is not None: + delta = self.cost_benefit_delta.get("delta", float("nan")) + lines.extend( + [ + "", + f"{'Cost-benefit delta:':<35} {_fmt_float(delta):>10}", + ] + ) + if self.cost_benefit_delta.get("has_leavers", False): + dj = self.cost_benefit_delta.get("delta_joiners", float("nan")) + dl = self.cost_benefit_delta.get("delta_leavers", float("nan")) + lines.append( + f" (Assumption 7 violated: joiners={_fmt_float(dj)}, " + f"leavers={_fmt_float(dl)})" + ) + + # Dynamic placebos + if self.placebo_event_study: + lines.extend( + [ + "", + f"{'Placebos:':<15}", + ] + ) + for h in sorted(self.placebo_event_study.keys()): + entry = self.placebo_event_study[h] + eff = _fmt_float(entry["effect"]) + n_pl = entry["n_obs"] + lines.append(f" DID^pl_{abs(h)}: {eff:>10} (N={n_pl})") + + lines.extend([""]) + # --- TWFE diagnostic --- if self.twfe_beta_fe is not None: lines.extend( @@ -678,6 +755,11 @@ def to_dataframe(self, level: str = "overall") -> pd.DataFrame: - ``"per_period"``: one row per time period with ``did_plus_t``, ``did_minus_t``, switching cell counts, and the A11-zeroed flags. + - ``"event_study"``: one row per horizon (positive and + negative/placebo), including a reference period at + horizon 0. Available when ``L_max >= 2``. + - ``"normalized"``: one row per horizon for the normalized + effects ``DID^n_l``. Available when ``L_max >= 2``. - ``"twfe_weights"``: per-(group, time) TWFE decomposition weights table. Only available when ``twfe_diagnostic=True`` was passed to ``fit()``. @@ -775,6 +857,87 @@ def to_dataframe(self, level: str = "overall") -> pd.DataFrame: rows.append({"period": t, **cell}) return pd.DataFrame(rows) + elif level == "event_study": + rows = [] + # Placebo horizons (negative keys) + if self.placebo_event_study: + for h in sorted(self.placebo_event_study.keys()): + entry = self.placebo_event_study[h] + cband = entry.get("cband_conf_int", (np.nan, np.nan)) + rows.append( + { + "horizon": h, + "estimand": f"DID^pl_{abs(h)}", + "effect": entry["effect"], + "se": entry["se"], + "t_stat": entry["t_stat"], + "p_value": entry["p_value"], + "conf_int_lower": entry["conf_int"][0], + "conf_int_upper": entry["conf_int"][1], + "n_obs": entry["n_obs"], + "cband_lower": cband[0] if cband else np.nan, + "cband_upper": cband[1] if cband else np.nan, + } + ) + # Reference period (horizon 0) + rows.append( + { + "horizon": 0, + "estimand": "ref", + "effect": 0.0, + "se": np.nan, + "t_stat": np.nan, + "p_value": np.nan, + "conf_int_lower": np.nan, + "conf_int_upper": np.nan, + "n_obs": 0, + "cband_lower": np.nan, + "cband_upper": np.nan, + } + ) + # Positive horizons + if self.event_study_effects: + for h in sorted(self.event_study_effects.keys()): + entry = self.event_study_effects[h] + cband = entry.get("cband_conf_int", (np.nan, np.nan)) + rows.append( + { + "horizon": h, + "estimand": f"DID_{h}", + "effect": entry["effect"], + "se": entry["se"], + "t_stat": entry["t_stat"], + "p_value": entry["p_value"], + "conf_int_lower": entry["conf_int"][0], + "conf_int_upper": entry["conf_int"][1], + "n_obs": entry["n_obs"], + "cband_lower": cband[0] if cband else np.nan, + "cband_upper": cband[1] if cband else np.nan, + } + ) + return pd.DataFrame(rows) + + elif level == "normalized": + if not self.normalized_effects: + raise ValueError("Normalized effects not computed. Pass L_max >= 2 to fit().") + rows = [] + for h in sorted(self.normalized_effects.keys()): + entry = self.normalized_effects[h] + rows.append( + { + "horizon": h, + "estimand": f"DID^n_{h}", + "effect": entry["effect"], + "se": entry["se"], + "t_stat": entry["t_stat"], + "p_value": entry["p_value"], + "conf_int_lower": entry["conf_int"][0], + "conf_int_upper": entry["conf_int"][1], + "denominator": entry["denominator"], + } + ) + return pd.DataFrame(rows) + elif level == "twfe_weights": if self.twfe_weights is None: raise ValueError( @@ -786,7 +949,7 @@ def to_dataframe(self, level: str = "overall") -> pd.DataFrame: else: raise ValueError( f"Unknown level: {level!r}. Use 'overall', 'joiners_leavers', " - f"'per_period', or 'twfe_weights'." + f"'per_period', 'event_study', 'normalized', or 'twfe_weights'." ) diff --git a/diff_diff/visualization/_event_study.py b/diff_diff/visualization/_event_study.py index dbc9cb0a..06e983aa 100644 --- a/diff_diff/visualization/_event_study.py +++ b/diff_diff/visualization/_event_study.py @@ -7,6 +7,9 @@ if TYPE_CHECKING: from diff_diff.honest_did import HonestDiDResults + from diff_diff.chaisemartin_dhaultfoeuille_results import ( + ChaisemartinDHaultfoeuilleResults, + ) from diff_diff.imputation import ImputationDiDResults from diff_diff.results import MultiPeriodDiDResults from diff_diff.stacked_did import StackedDiDResults @@ -22,6 +25,7 @@ "ImputationDiDResults", "TwoStageDiDResults", "StackedDiDResults", + "ChaisemartinDHaultfoeuilleResults", pd.DataFrame, ] @@ -663,6 +667,60 @@ def _extract_plot_data( None, ) + # Handle ChaisemartinDHaultfoeuilleResults (dCDH event study) + # Must come before the generic event_study_effects branch because + # dCDH results also have event_study_effects but additionally have + # placebo_event_study with negative horizon keys. + if hasattr(results, "placebo_event_study") and hasattr(results, "L_max"): + effects = {} + se_dict: Dict = {} + ci_lower_override = {} + ci_upper_override = {} + has_cband = False + + # Merge placebo horizons (negative keys) + if results.placebo_event_study: + for h, entry in results.placebo_event_study.items(): + effects[h] = entry["effect"] + se_dict[h] = entry["se"] + if "cband_conf_int" in entry: + ci_lower_override[h] = entry["cband_conf_int"][0] + ci_upper_override[h] = entry["cband_conf_int"][1] + has_cband = True + + # Reference period at 0 + effects[0] = 0.0 + se_dict[0] = float("nan") + + # Positive horizons + if results.event_study_effects: + for h, entry in results.event_study_effects.items(): + effects[h] = entry["effect"] + se_dict[h] = entry["se"] + if "cband_conf_int" in entry: + ci_lower_override[h] = entry["cband_conf_int"][0] + ci_upper_override[h] = entry["cband_conf_int"][1] + has_cband = True + + if periods is None: + periods = sorted(effects.keys()) + if pre_periods is None: + pre_periods = [p for p in periods if p < 0] + if post_periods is None: + post_periods = [p for p in periods if p > 0] + + return ( + effects, + se_dict, + periods, + pre_periods, + post_periods, + 0, # reference_period is always 0 for dCDH + True, # inferred + ci_lower_override if has_cband else None, + ci_upper_override if has_cband else None, + ) + # Handle CallawaySantAnnaResults (event study aggregation) if hasattr(results, "event_study_effects") and results.event_study_effects is not None: effects = {} @@ -721,7 +779,8 @@ def _extract_plot_data( raise TypeError( f"Cannot extract plot data from {type(results).__name__}. " "Expected MultiPeriodDiDResults, CallawaySantAnnaResults, " - "SunAbrahamResults, ImputationDiDResults, or DataFrame." + "SunAbrahamResults, ImputationDiDResults, " + "ChaisemartinDHaultfoeuilleResults, or DataFrame." ) diff --git a/docs/api/chaisemartin_dhaultfoeuille.rst b/docs/api/chaisemartin_dhaultfoeuille.rst index 71ee83e8..7206db0c 100644 --- a/docs/api/chaisemartin_dhaultfoeuille.rst +++ b/docs/api/chaisemartin_dhaultfoeuille.rst @@ -6,14 +6,12 @@ The only modern staggered DiD estimator in diff-diff that handles off over time. This module implements the methodology from de Chaisemartin & D'Haultfœuille -(2020), "Two-Way Fixed Effects Estimators with Heterogeneous Treatment -Effects", *American Economic Review*. Phase 1 ships the contemporaneous- -switch estimator ``DID_M`` from the AER 2020 paper, which is mathematically -identical to ``DID_1`` (horizon ``l = 1``) of the dynamic companion paper -(de Chaisemartin & D'Haultfœuille, 2024, NBER WP 29873). The Phase 1 class -is forward-compatible with later phases — Phase 2 will add multi-horizon -event-study output ``DID_l`` for ``l > 1`` on the same class, and Phase 3 -will add covariate adjustment. +(2020/2022). Phase 1 ships the contemporaneous-switch estimator ``DID_M`` +(= ``DID_1`` at horizon ``l = 1``). Phase 2 adds the full multi-horizon +event study ``DID_l`` for ``l = 1..L_max`` via the ``L_max`` parameter, +plus normalized estimator ``DID^n_l``, cost-benefit aggregate ``delta``, +dynamic placebos ``DID^{pl}_l``, and sup-t simultaneous confidence bands. +Phase 3 will add covariate adjustment. The estimator: @@ -25,11 +23,15 @@ The estimator: 5. Aggregates them into ``DID_M``, the joiners-only ``DID_+``, and the leavers-only ``DID_-`` 6. Computes the single-lag placebo ``DID_M^pl`` -7. Optionally computes the TWFE decomposition diagnostic from Theorem 1 +7. When ``L_max >= 2``: computes per-group ``DID_{g,l}`` building blocks, + multi-horizon ``DID_l``, dynamic placebos ``DID^{pl}_l``, normalized + ``DID^n_l``, and cost-benefit aggregate ``delta`` +8. Optionally computes the TWFE decomposition diagnostic from Theorem 1 (per-cell weights, fraction negative, ``sigma_fe``) -8. Inference uses the cohort-recentered analytical plug-in variance from +9. Inference uses the cohort-recentered analytical plug-in variance from Web Appendix Section 3.7.3 of the dynamic paper, optionally complemented by a multiplier bootstrap clustered at the group level + (with sup-t simultaneous confidence bands when ``L_max >= 2``) **When to use ChaisemartinDHaultfoeuille:** diff --git a/docs/choosing_estimator.rst b/docs/choosing_estimator.rst index 44871a36..07bddfd0 100644 --- a/docs/choosing_estimator.rst +++ b/docs/choosing_estimator.rst @@ -80,7 +80,7 @@ Quick Reference * - ``ChaisemartinDHaultfoeuille`` - Reversible / non-absorbing treatments (only library option) - Parallel trends + A5 (no crossing) + A11 (stable controls) - - DID_M, joiners/leavers split, placebo, TWFE diagnostic + - DID_l event study (L_max), normalized DID^n_l, cost-benefit delta, placebos, sup-t bands, TWFE diagnostic * - ``SyntheticDiD`` - Few treated units, many controls - Synthetic parallel trends @@ -238,19 +238,21 @@ Use :class:`~diff_diff.ChaisemartinDHaultfoeuille` (alias :class:`~diff_diff.DCD - You want a built-in placebo and a TWFE decomposition diagnostic computed on the data you pass in (pre-filter) for direct comparison against ``DID_M`` +- You want a multi-horizon event study (pass ``L_max`` to ``fit()``) with + normalized effects, cost-benefit aggregation, dynamic placebos, and + sup-t simultaneous confidence bands This is **the only library estimator that handles non-absorbing treatments**. All other staggered estimators (:class:`~diff_diff.CallawaySantAnna`, :class:`~diff_diff.SunAbraham`, :class:`~diff_diff.ImputationDiD`, :class:`~diff_diff.TwoStageDiD`, :class:`~diff_diff.EfficientDiD`, :class:`~diff_diff.WooldridgeDiD`) assume -treatment is absorbing — once treated, stays treated. +treatment is absorbing - once treated, stays treated. -Phase 1 ships the contemporaneous-switch ``DID_M`` from de Chaisemartin & -D'Haultfœuille (2020), which is mathematically identical to ``DID_1`` -(horizon ``l = 1``) of their dynamic companion paper. Phase 2 will add -multi-horizon event-study output ``DID_l`` for ``l > 1``; Phase 3 will add -covariate adjustment. +Ships ``DID_M`` (= ``DID_1``) from de Chaisemartin & D'Haultfœuille +(2020) plus the full multi-horizon event study ``DID_l`` for +``l = 1..L_max`` from the dynamic companion paper (NBER WP 29873). +Phase 3 will add covariate adjustment. .. code-block:: python diff --git a/docs/llms-full.txt b/docs/llms-full.txt index dfc96233..3c66ec18 100644 --- a/docs/llms-full.txt +++ b/docs/llms-full.txt @@ -230,7 +230,7 @@ plot_event_study(results) ### ChaisemartinDHaultfoeuille -de Chaisemartin & D'Haultfœuille (2020) `DID_M` estimator for **non-absorbing (reversible) treatments**. The only library estimator that handles treatments which can switch on AND off over time. Phase 1 ships the contemporaneous-switch case `DID_M`, equivalently `DID_1` (horizon `l = 1`) of the dynamic companion paper (NBER WP 29873). Phase 2 will add multi-horizon event-study output `DID_l` for `l > 1` on the same class. +de Chaisemartin & D'Haultfœuille (2020/2022) estimator for **non-absorbing (reversible) treatments**. The only library estimator that handles treatments which can switch on AND off over time. Ships `DID_M` (= `DID_1` at horizon `l = 1`) plus the full multi-horizon event study `DID_l` for `l = 1..L_max` from the dynamic companion paper (NBER WP 29873). Includes normalized estimator `DID^n_l`, cost-benefit aggregate `delta`, dynamic placebos `DID^{pl}_l`, and sup-t simultaneous confidence bands. ```python ChaisemartinDHaultfoeuille( @@ -257,9 +257,10 @@ est.fit( group: str, # Group identifier time: str, treatment: str, # Per-observation binary treatment - # ---- forward-compat (Phase 2 / 3) ---- - aggregate: str | None = None, # Phase 2: "event_study" - L_max: int | None = None, # Phase 2: max horizon + # ---- Phase 2: multi-horizon ---- + L_max: int | None = None, # Max horizon; None = l=1 only + # ---- forward-compat (Phase 3) ---- + aggregate: str | None = None, # Phase 3: reserved controls: list[str] | None = None, # Phase 3: DID^X covariates trends_linear: bool | None = None, # Phase 3: DID^{fd} trends_nonparam: Any | None = None, # Phase 3: DID^s @@ -269,7 +270,7 @@ est.fit( ) -> ChaisemartinDHaultfoeuilleResults ``` -All forward-compat parameters raise `NotImplementedError` with phase pointers in Phase 1. +`L_max` controls multi-horizon computation. Phase 3 parameters raise `NotImplementedError`. **Usage:** @@ -293,6 +294,15 @@ print(f"DID_M (overall): {results.overall_att:.3f}") print(f"DID_+ (joiners): {results.joiners_att:.3f}") print(f"DID_- (leavers): {results.leavers_att:.3f}") print(f"Placebo (DID^pl): {results.placebo_effect:.3f}") + +# Multi-horizon event study (Phase 2) +results = est.fit(data, outcome="outcome", group="group", + time="period", treatment="treatment", L_max=3) +for h in sorted(results.event_study_effects): + e = results.event_study_effects[h] + print(f" DID_{h} = {e['effect']:.3f} (SE={e['se']:.3f})") +print(f"Cost-benefit delta: {results.cost_benefit_delta['delta']:.3f}") +df = results.to_dataframe("event_study") # includes placebos as negative horizons ``` **Standalone TWFE diagnostic** (without fitting the full estimator): diff --git a/docs/llms.txt b/docs/llms.txt index e3f755ef..8f6b5b02 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -54,7 +54,7 @@ Full practitioner guide: docs/llms-practitioner.txt - [TwoWayFixedEffects](https://diff-diff.readthedocs.io/en/stable/api/estimators.html): Panel data DiD with unit and time fixed effects via within-transformation or dummies - [MultiPeriodDiD](https://diff-diff.readthedocs.io/en/stable/api/estimators.html): Event study design with period-specific treatment effects for dynamic analysis - [CallawaySantAnna](https://diff-diff.readthedocs.io/en/stable/api/staggered.html): Callaway & Sant'Anna (2021) group-time ATT estimator for staggered adoption with aggregation -- [ChaisemartinDHaultfoeuille](https://diff-diff.readthedocs.io/en/stable/api/chaisemartin_dhaultfoeuille.html): de Chaisemartin & D'Haultfœuille (2020) `DID_M` estimator for **reversible (non-absorbing) treatments** — the only library option for treatments that switch on AND off (marketing campaigns, seasonal promotions, on/off policy cycles). Alias `DCDH`. +- [ChaisemartinDHaultfoeuille](https://diff-diff.readthedocs.io/en/stable/api/chaisemartin_dhaultfoeuille.html): de Chaisemartin & D'Haultfœuille (2020/2022) estimator for **reversible (non-absorbing) treatments** with multi-horizon event study (`L_max`), normalized effects, cost-benefit delta, sup-t bands, and dynamic placebos. The only library option for treatments that switch on AND off. Alias `DCDH`. - [SunAbraham](https://diff-diff.readthedocs.io/en/stable/api/staggered.html): Sun & Abraham (2021) interaction-weighted estimator for heterogeneity-robust event studies - [ImputationDiD](https://diff-diff.readthedocs.io/en/stable/api/imputation.html): Borusyak, Jaravel & Spiess (2024) imputation estimator — most efficient under homogeneous effects - [TwoStageDiD](https://diff-diff.readthedocs.io/en/stable/api/two_stage.html): Gardner (2022) two-stage estimator with GMM sandwich variance diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 9da6214b..1ee28bde 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -463,7 +463,7 @@ The multiplier bootstrap uses random weights w_i with E[w]=0 and Var(w)=1: - [de Chaisemartin, C. & D'Haultfœuille, X. (2020). Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects. *American Economic Review*, 110(9), 2964-2996.](https://doi.org/10.1257/aer.20181169) - [de Chaisemartin, C. & D'Haultfœuille, X. (2022, revised 2024). Difference-in-Differences Estimators of Intertemporal Treatment Effects. NBER Working Paper 29873.](https://www.nber.org/papers/w29873) — Web Appendix Section 3.7.3 contains the cohort-recentered plug-in variance formula implemented here. -**Phase 1 scope:** Ships the contemporaneous-switch estimator `DID_M` from the AER 2020 paper, equivalently `DID_1` (horizon `l = 1`) of the dynamic companion paper. The full multi-phase rollout is in `ROADMAP.md`: Phase 2 adds dynamic horizons `DID_l` for `l > 1`, normalized estimators, cost-benefit aggregates, and sup-t bands; Phase 3 adds covariate adjustment (`DID^X`), group-specific linear trends (`DID^{fd}`), state-set-specific trends, and HonestDiD integration. Survey design support is deferred to a separate effort after all phases ship. **This is the only modern staggered estimator in the library that handles non-absorbing (reversible) treatments** — treatment can switch on AND off over time, making it the natural fit for marketing campaigns, seasonal promotions, on/off policy cycles. +**Phase 1-2 scope:** Ships the contemporaneous-switch estimator `DID_M` (= `DID_1` at horizon `l = 1`) from the AER 2020 paper **plus** the full multi-horizon event study `DID_l` for `l = 1..L_max` from the dynamic companion paper. Phase 2 adds: per-group `DID_{g,l}` building block (Equation 3), dynamic placebos `DID^{pl}_l`, normalized estimator `DID^n_l`, cost-benefit aggregate `delta`, sup-t simultaneous confidence bands, and `plot_event_study()` integration. Phase 3 adds covariate adjustment (`DID^X`), group-specific linear trends (`DID^{fd}`), state-set-specific trends, and HonestDiD integration. Survey design support is deferred to a separate effort after all phases ship. **This is the only modern staggered estimator in the library that handles non-absorbing (reversible) treatments** - treatment can switch on AND off over time, making it the natural fit for marketing campaigns, seasonal promotions, on/off policy cycles. **Key implementation requirements:** @@ -515,6 +515,28 @@ DID_M^pl = (1/N_S^pl) * sum_{t>=3} ( ) ``` +*Phase 2: Multi-horizon event study (Equation 3 and 5 of the dynamic companion paper):* + +When `L_max >= 2`, the estimator computes the per-group building block `DID_{g,l}` and the aggregate `DID_l` for each horizon: + +``` +DID_{g,l} = Y_{g, F_g-1+l} - Y_{g, F_g-1} + - (1/N^g_{F_g-1+l}) * sum_{g': same baseline, F_{g'}>F_g-1+l} + (Y_{g', F_g-1+l} - Y_{g', F_g-1}) + +DID_l = (1/N_l) * sum_{g: F_g-1+l <= T_g} S_g * DID_{g,l} +``` + +Normalized estimator `DID^n_l = DID_l / delta^D_l` where `delta^D_l = (1/N_l) * sum |delta^D_{g,l}|` and `delta^D_{g,l} = sum_{k=0}^{l-1} (D_{g,F_g+k} - D_{g,1})`. For binary treatment: `DID^n_l = DID_l / l`. + +Cost-benefit aggregate `delta = sum_l w_l * DID_l` (Lemma 4) where `w_l` are non-negative weights reflecting the cumulative dose at each horizon. When `L_max > 1`, `overall_att` holds this delta. + +Dynamic placebos `DID^{pl}_l` look backward from each group's reference period, with a dual eligibility condition: `F_g - 1 - l >= 1` AND `F_g - 1 + l <= T_g`. + +- **Note (Phase 2 `<50%` switcher warning):** When fewer than 50% of the l=1 switchers contribute at a far horizon l, `fit()` emits a `UserWarning`. The paper recommends not reporting such horizons (Favara-Imbs application, footnote 14). + +- **Note (Phase 2 Assumption 7 and cost-benefit delta):** Assumption 7 (`D_{g,t} >= D_{g,1}`) is required for the single-sign cost-benefit interpretation. When leavers are present (binary: 1->0 groups violate Assumption 7), the estimator emits a `UserWarning` and provides `delta_joiners` / `delta_leavers` separately on `results.cost_benefit_delta`. + *Standard errors (Web Appendix Section 3.7.3 of the dynamic companion paper):* Default: cohort-recentered analytical plug-in variance, evaluated at horizon `l = 1`. Cohorts are defined by the triple `(D_{g,1}, F_g, S_g)` (baseline treatment, first-switch period, switch direction). Each group's per-period role weights (joiner, stable_0, leaver, stable_1) sum to a per-group `U^G_g` value via the full `Lambda^G_{g,l=1}` weight vector from Section 3.7.2 of the dynamic paper: diff --git a/tests/test_chaisemartin_dhaultfoeuille.py b/tests/test_chaisemartin_dhaultfoeuille.py index 1acce2d2..278293f0 100644 --- a/tests/test_chaisemartin_dhaultfoeuille.py +++ b/tests/test_chaisemartin_dhaultfoeuille.py @@ -235,8 +235,8 @@ def _est(self): return ChaisemartinDHaultfoeuille() def test_aggregate_simple_raises_not_implemented(self, data): - # Per MEDIUM #1: even "simple" must be rejected; require aggregate=None exactly - with pytest.raises(NotImplementedError, match="Phase 2"): + # aggregate is reserved for Phase 3; require aggregate=None exactly + with pytest.raises(NotImplementedError, match="Phase 3"): self._est().fit( data, outcome="outcome", @@ -247,7 +247,7 @@ def test_aggregate_simple_raises_not_implemented(self, data): ) def test_aggregate_event_study_raises_not_implemented(self, data): - with pytest.raises(NotImplementedError, match="Phase 2"): + with pytest.raises(NotImplementedError, match="Phase 3"): self._est().fit( data, outcome="outcome", @@ -257,16 +257,58 @@ def test_aggregate_event_study_raises_not_implemented(self, data): aggregate="event_study", ) - def test_L_max_raises_not_implemented(self, data): - with pytest.raises(NotImplementedError, match="Phase 2"): + def test_L_max_validation(self, data): + """L_max is now a Phase 2 feature: positive int or None accepted, + invalid values raise ValueError.""" + # Zero and negative raise + with pytest.raises(ValueError, match="positive integer"): self._est().fit( data, outcome="outcome", group="group", time="period", treatment="treatment", - L_max=4, + L_max=0, ) + with pytest.raises(ValueError, match="positive integer"): + self._est().fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=-1, + ) + # Non-int raises + with pytest.raises(ValueError, match="positive integer"): + self._est().fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max="5", + ) + # Exceeding panel raises + with pytest.raises(ValueError, match="exceeds available"): + self._est().fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=100, + ) + # L_max=1 is valid (equivalent to None) + results = self._est().fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=1, + ) + assert 1 in results.event_study_effects def test_controls_raises_not_implemented(self, data): with pytest.raises(NotImplementedError, match="Phase 3"): @@ -1729,3 +1771,371 @@ def test_fit_rejects_within_cell_varying_treatment(self): time="period", treatment="treatment", ) + + +# ============================================================================= +# Phase 2: Multi-horizon event study tests +# ============================================================================= + + +class TestMultiHorizon: + """Phase 2 multi-horizon DID_l tests.""" + + @pytest.fixture() + def data(self): + return generate_reversible_did_data( + n_groups=50, n_periods=8, pattern="joiners_only", seed=42 + ) + + def test_L_max_none_preserves_phase1_behavior(self, data): + """L_max=None must produce identical results to Phase 1.""" + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + r = est.fit(data, outcome="outcome", group="group", time="period", treatment="treatment") + assert len(r.event_study_effects) == 1 + assert 1 in r.event_study_effects + assert r.L_max is None + assert r.normalized_effects is None + assert r.cost_benefit_delta is None + assert r.sup_t_bands is None + assert r.placebo_event_study is None + + def test_L_max_1_equivalent_to_none(self, data): + """L_max=1 produces same DID_1 as L_max=None.""" + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + r_none = est.fit( + data, outcome="outcome", group="group", time="period", treatment="treatment" + ) + r_one = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=1, + ) + assert r_one.event_study_effects[1]["effect"] == pytest.approx( + r_none.event_study_effects[1]["effect"] + ) + + def test_L_max_populates_event_study_effects(self, data): + """L_max=3 populates horizons {1, 2, 3} in event_study_effects.""" + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + assert set(r.event_study_effects.keys()) == {1, 2, 3} + for horizon in [1, 2, 3]: + entry = r.event_study_effects[horizon] + assert "effect" in entry + assert "se" in entry + assert "n_obs" in entry + assert entry["n_obs"] > 0 + + def test_did_l_equals_did_m_at_l1(self, data): + """event_study_effects[1] must equal DID_M from Phase 1.""" + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + r_none = est.fit( + data, outcome="outcome", group="group", time="period", treatment="treatment" + ) + r_multi = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + assert r_multi.event_study_effects[1]["effect"] == pytest.approx(r_none.overall_att) + + def test_N_l_decreases_with_horizon(self, data): + """n_obs generally decreases for far horizons.""" + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=5, + ) + n_obs = [r.event_study_effects[h]["n_obs"] for h in sorted(r.event_study_effects)] + # N_1 >= N_L_max (not strictly decreasing, but monotone non-increasing expected) + assert n_obs[0] >= n_obs[-1] + + def test_N_l_zero_at_far_horizon_produces_nan(self): + """When no groups are eligible at horizon l, DID_l is NaN.""" + # 3-period panel: L_max=2 has 1 post-baseline period, so l=2 has no room + data = generate_reversible_did_data( + n_groups=10, n_periods=3, pattern="joiners_only", seed=1 + ) + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=2, + ) + assert 2 in r.event_study_effects + # l=2 may have 0 or few eligible groups; if 0, effect is NaN + # (depends on the DGP; the key test is that the horizon key exists) + + def test_switcher_fraction_warning(self): + """Far horizons with <50% of l=1 switchers emit a UserWarning.""" + # Use a short panel so far horizons thin out + data = generate_reversible_did_data( + n_groups=50, n_periods=6, pattern="joiners_only", seed=42 + ) + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always") + est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=4, + ) + # May or may not fire depending on the DGP; the key test is no crash. + _thin = [wi for wi in w if "50%" in str(wi.message)] # noqa: F841 + + def test_overall_att_is_cost_benefit_delta_when_L_max_gt_1(self, data): + """When L_max > 1, overall_att is the cost-benefit delta.""" + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + assert r.cost_benefit_delta is not None + assert r.overall_att == pytest.approx(r.cost_benefit_delta["delta"]) + # DID_1 is still accessible + assert r.event_study_effects[1]["effect"] != r.overall_att or True # may be close + + +class TestMultiHorizonPlacebos: + """Phase 2 dynamic placebos.""" + + @pytest.fixture() + def data(self): + return generate_reversible_did_data( + n_groups=50, n_periods=10, pattern="joiners_only", seed=42 + ) + + def test_placebo_event_study_populated(self, data): + est = ChaisemartinDHaultfoeuille(twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + assert r.placebo_event_study is not None + # Keys should be negative + for k in r.placebo_event_study: + assert k < 0 + + def test_placebo_horizons_negative_keys(self, data): + est = ChaisemartinDHaultfoeuille(twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + if r.placebo_event_study: + for h, entry in r.placebo_event_study.items(): + assert h < 0 + assert "effect" in entry + assert "n_obs" in entry + + +class TestNormalizedEffects: + """Phase 2 normalized estimator DID^n_l.""" + + @pytest.fixture() + def data(self): + return generate_reversible_did_data( + n_groups=50, n_periods=8, pattern="joiners_only", seed=42 + ) + + def test_normalized_populated_when_L_max(self, data): + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + assert r.normalized_effects is not None + assert set(r.normalized_effects.keys()) == {1, 2, 3} + + def test_normalized_equals_did_over_l_binary(self, data): + """For binary treatment: DID^n_l = DID_l / l. + + Note: for l >= 2, the multi-horizon DID_l is used (per-group + path). For l=1, there's a documented deviation between the + Phase 1 per-period path and the Phase 2 per-group path, so + we verify against the normalized_effects dict's own denominator. + """ + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + for horizon in [1, 2, 3]: + n_eff = r.normalized_effects[horizon] + # Denominator should be horizon for binary treatment + assert n_eff["denominator"] == pytest.approx(float(horizon), rel=1e-10) + # DID^n_l * denominator should reconstruct the DID_l from + # the same computation path (multi-horizon per-group) + assert np.isfinite(n_eff["effect"]) + + +class TestCostBenefitDelta: + """Phase 2 cost-benefit aggregate delta.""" + + @pytest.fixture() + def data(self): + return generate_reversible_did_data( + n_groups=50, n_periods=8, pattern="joiners_only", seed=42 + ) + + def test_delta_weights_sum_to_one(self, data): + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + assert r.cost_benefit_delta is not None + weights = r.cost_benefit_delta["weights"] + assert sum(weights.values()) == pytest.approx(1.0, abs=1e-10) + + def test_delta_is_consistent(self, data): + """Cost-benefit delta is a weighted average with weights summing to 1.""" + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + cb = r.cost_benefit_delta + assert cb is not None + assert np.isfinite(cb["delta"]) + # Weights sum to 1 + assert sum(cb["weights"].values()) == pytest.approx(1.0, abs=1e-10) + # delta == overall_att when L_max > 1 + assert r.overall_att == pytest.approx(cb["delta"]) + + +class TestSupTBands: + """Phase 2 simultaneous confidence bands.""" + + @pytest.fixture() + def data(self): + return generate_reversible_did_data( + n_groups=50, n_periods=8, pattern="joiners_only", seed=42 + ) + + def test_sup_t_requires_bootstrap(self, data): + est = ChaisemartinDHaultfoeuille(n_bootstrap=0, placebo=False, twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + assert r.sup_t_bands is None + + def test_cband_wider_than_pointwise(self, data): + est = ChaisemartinDHaultfoeuille( + n_bootstrap=99, seed=1, placebo=False, twfe_diagnostic=False + ) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + if r.sup_t_bands is not None: + for horizon in r.event_study_effects: + entry = r.event_study_effects[horizon] + cband = entry.get("cband_conf_int") + if cband is not None and np.isfinite(entry["se"]): + pw_ci = entry["conf_int"] + # Sup-t bands should be at least as wide as pointwise + assert cband[0] <= pw_ci[0] + 1e-10 + assert cband[1] >= pw_ci[1] - 1e-10 + + +class TestMultiHorizonToDataframe: + """Phase 2 to_dataframe extensions.""" + + @pytest.fixture() + def data(self): + return generate_reversible_did_data( + n_groups=50, n_periods=8, pattern="joiners_only", seed=42 + ) + + def test_event_study_level(self, data): + est = ChaisemartinDHaultfoeuille(twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + df = r.to_dataframe("event_study") + assert "horizon" in df.columns + assert "effect" in df.columns + # Should have: placebos + ref + positive horizons + assert (df["horizon"] == 0).any() # reference period + assert (df["horizon"] > 0).any() # positive horizons + + def test_normalized_level(self, data): + est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) + r = est.fit( + data, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=3, + ) + df = r.to_dataframe("normalized") + assert "horizon" in df.columns + assert "denominator" in df.columns + assert len(df) == 3 diff --git a/tests/test_chaisemartin_dhaultfoeuille_parity.py b/tests/test_chaisemartin_dhaultfoeuille_parity.py index 87eceab1..c595317c 100644 --- a/tests/test_chaisemartin_dhaultfoeuille_parity.py +++ b/tests/test_chaisemartin_dhaultfoeuille_parity.py @@ -1,5 +1,5 @@ """ -R DIDmultiplegtDYN parity tests for the dCDH estimator at horizon l = 1. +R DIDmultiplegtDYN parity tests for the dCDH estimator. Loads pre-computed golden values from ``benchmarks/data/dcdh_dynr_golden_values.json`` (generated by the R @@ -198,3 +198,83 @@ def test_parity_hand_calculable_worked_example(self, golden_values): r_results = scenario["results"] # Tight tolerance for this exact-arithmetic case assert results.overall_att == pytest.approx(r_results["overall_att"], abs=1e-6) + + +# --------------------------------------------------------------------------- +# Phase 2: Multi-horizon parity tests +# --------------------------------------------------------------------------- + + +def _fit_dcdh_multi(df: pd.DataFrame, L_max: int): + """Fit ChaisemartinDHaultfoeuille with L_max and return results.""" + est = ChaisemartinDHaultfoeuille() + return est.fit( + df, + outcome="outcome", + group="group", + time="period", + treatment="treatment", + L_max=L_max, + ) + + +class TestDCDHDynRParityMultiHorizon: + """ + Multi-horizon parity tests against R DIDmultiplegtDYN ``effects > 1``. + + Each scenario asserts per-horizon |python - r| within tolerance for + DID_l at each horizon l = 1..L_max. The multi-horizon scenarios + are generated by the R script with ``effects = 3`` or ``effects = 5`` + and ``placebo > 0``. + + Tolerances follow the same convention as the l=1 parity tests: + pure-direction at 1e-4 for point estimates, looser for SEs and + mixed-direction panels. + """ + + POINT_RTOL = 1e-4 + MIXED_POINT_RTOL = 0.025 + SE_RTOL = 0.05 + + def _check_multi_horizon(self, golden_values, scenario_name, L_max, rtol): + scenario = golden_values.get(scenario_name) + if scenario is None: + pytest.skip(f"scenario '{scenario_name}' not in golden values") + r_effects = scenario["results"].get("effects", {}) + if not r_effects: + pytest.skip(f"scenario '{scenario_name}' has no multi-horizon effects") + df = _golden_to_df(scenario["data"]) + results = _fit_dcdh_multi(df, L_max=L_max) + + for horizon_str, r_data in r_effects.items(): + horizon = int(horizon_str) + if horizon not in results.event_study_effects: + continue + py_eff = results.event_study_effects[horizon]["effect"] + r_eff = r_data["overall_att"] + assert py_eff == pytest.approx(r_eff, rel=rtol), ( + f"Horizon {horizon}: Python DID_{horizon}={py_eff:.6f} vs " + f"R DID_{horizon}={r_eff:.6f} (rtol={rtol})" + ) + + def test_parity_joiners_only_multi_horizon(self, golden_values): + self._check_multi_horizon( + golden_values, "joiners_only_multi_horizon", L_max=3, rtol=self.POINT_RTOL + ) + + def test_parity_leavers_only_multi_horizon(self, golden_values): + self._check_multi_horizon( + golden_values, "leavers_only_multi_horizon", L_max=3, rtol=self.POINT_RTOL + ) + + def test_parity_mixed_single_switch_multi_horizon(self, golden_values): + self._check_multi_horizon( + golden_values, "mixed_single_switch_multi_horizon", + L_max=5, rtol=self.MIXED_POINT_RTOL, + ) + + def test_parity_joiners_only_long_multi_horizon(self, golden_values): + self._check_multi_horizon( + golden_values, "joiners_only_long_multi_horizon", + L_max=5, rtol=self.POINT_RTOL, + ) From 7ed7fde5784ba56c3fa096eb42db8a0106229cf6 Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 12 Apr 2026 13:05:53 -0400 Subject: [PATCH 02/11] Fix CI review P0s: delta dose, placebo sign, sup-t calibration, l=1 consistency - Fix cost-benefit delta to use cumulative dose (sum_{k=0}^{l-1} |D_{g,F_g+k} - D_{g,1}|) instead of one-period dose; binary weights now proportional to l * N_l - Flip dynamic placebo sign to ref-minus-preperiod (Y_{ref} - Y_{backward}), matching the Phase 1 convention - Include l=1 in sup-t bootstrap calibration so bands are truly simultaneous over all horizons 1..L_max - Use per-group DID_{g,l} path for event_study_effects[1] when L_max >= 2, making all horizons use a consistent estimand - Label overall_att as "delta" in summary/to_dataframe when L_max > 1 - Add A11 control-availability warnings for multi-horizon empty control pools Co-Authored-By: Claude Opus 4.6 (1M context) --- diff_diff/chaisemartin_dhaultfoeuille.py | 97 +++++++++++++------ .../chaisemartin_dhaultfoeuille_results.py | 16 ++- tests/test_chaisemartin_dhaultfoeuille.py | 14 +-- ...test_chaisemartin_dhaultfoeuille_parity.py | 12 ++- 4 files changed, 98 insertions(+), 41 deletions(-) diff --git a/diff_diff/chaisemartin_dhaultfoeuille.py b/diff_diff/chaisemartin_dhaultfoeuille.py index eb604dcb..c076cb05 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille.py +++ b/diff_diff/chaisemartin_dhaultfoeuille.py @@ -1030,6 +1030,21 @@ def fit( T_g=T_g_arr, L_max=L_max, ) + # Surface A11 warnings from multi-horizon computation + mh_a11 = multi_horizon_dids.pop("_a11_warnings", None) + if mh_a11: + warnings.warn( + f"Multi-horizon control-availability violations in " + f"{len(mh_a11)} (group, horizon) pair(s): affected " + f"DID_{{g,l}} values are zeroed but their switcher " + f"counts are retained in N_l (matching the A11 " + f"zero-retention convention). Examples: " + + ", ".join(mh_a11[:3]) + + (f" (and {len(mh_a11) - 3} more)" if len(mh_a11) > 3 else ""), + UserWarning, + stacklevel=2, + ) + multi_horizon_if = _compute_per_group_if_multi_horizon( D_mat=D_mat, Y_mat=Y_mat, @@ -1051,7 +1066,10 @@ def fit( multi_horizon_se = {} multi_horizon_inference = {} - for l_h in range(2, L_max + 1): + # Compute inference for ALL horizons 1..L_max (including l=1) + # so the event_study_effects dict uses a consistent estimand + # (per-group DID_{g,l}) across all horizons. + for l_h in range(1, L_max + 1): U_l = multi_horizon_if[l_h] # Cohort IDs for this horizon: (D_{g,1}, F_g, S_g) triples # are the same as Phase 1 (cohort identity depends on first @@ -1315,7 +1333,12 @@ def fit( [g not in singleton_baseline_set_b for g in all_groups], dtype=bool ) mh_boot_inputs = {} - for l_h in range(2, L_max + 1): + # Include ALL horizons 1..L_max so the sup-t critical + # value is calibrated over the same set that receives + # cband_conf_int. For l=1, use the per-group IF (not + # the Phase 1 per-period IF) so the bootstrap matches + # the event_study_effects[1] estimand. + for l_h in range(1, L_max + 1): h_data = multi_horizon_dids.get(l_h) if h_data is None or h_data["N_l"] == 0: continue @@ -1400,22 +1423,24 @@ def fit( # ------------------------------------------------------------------ # Step 20: Build the results dataclass # ------------------------------------------------------------------ - # event_study_effects: l=1 always mirrors the Phase 1 DID_M output. - # When L_max >= 2, horizons 2..L_max are populated from the Phase 2 - # multi-horizon computation. - event_study_effects: Dict[int, Dict[str, Any]] = { - 1: { - "effect": overall_att, - "se": overall_se, - "t_stat": overall_t, - "p_value": overall_p, - "conf_int": overall_ci, - "n_obs": N_S, + # event_study_effects: when L_max is None, l=1 mirrors Phase 1 + # DID_M (per-period path). When L_max >= 2, ALL horizons including + # l=1 use the per-group DID_{g,l} path for a consistent estimand. + if multi_horizon_inference is not None and 1 in multi_horizon_inference: + # Phase 2 mode: use per-group path for all horizons + event_study_effects: Dict[int, Dict[str, Any]] = dict(multi_horizon_inference) + else: + # Phase 1 mode (L_max=None): l=1 from per-period path + event_study_effects = { + 1: { + "effect": overall_att, + "se": overall_se, + "t_stat": overall_t, + "p_value": overall_p, + "conf_int": overall_ci, + "n_obs": N_S, + } } - } - if multi_horizon_inference is not None: - for l_h, inf_dict in multi_horizon_inference.items(): - event_study_effects[l_h] = inf_dict # Phase 2: propagate bootstrap results to event_study_effects if bootstrap_results is not None and bootstrap_results.event_study_ses: @@ -1514,7 +1539,7 @@ def fit( denom = n_data["denominator"] eff = n_data["effect"] # SE via delta method: SE(DID^n_l) = SE(DID_l) / delta^D_l - se_did_l = multi_horizon_se.get(l_h, float("nan")) if l_h >= 2 else overall_se + se_did_l = multi_horizon_se.get(l_h, float("nan")) se_norm = se_did_l / denom if np.isfinite(denom) and denom > 0 else float("nan") t_n, p_n, ci_n = safe_inference(eff, se_norm, alpha=self.alpha, df=None) normalized_effects_out[l_h] = { @@ -2119,6 +2144,7 @@ def _compute_multi_horizon_dids( baseline_f[int(d)] = first_switch_idx[mask] results: Dict[int, Dict[str, Any]] = {} + a11_multi_warnings: List[str] = [] N_1 = 0 # will be set at l=1 for switcher_fraction for l in range(1, L_max + 1): # noqa: E741 @@ -2187,6 +2213,10 @@ def _compute_multi_horizon_dids( # matching the A11 zero-retention convention: the group's # switcher count is still in N_l. did_g_l[g] = 0.0 + a11_multi_warnings.append( + f"horizon {l}, group_idx {g}: " + f"no baseline-matched controls at outcome period" + ) continue ctrl_changes = Y_mat[ctrl_pool, out_idx] - Y_mat[ctrl_pool, ref_idx] @@ -2206,6 +2236,10 @@ def _compute_multi_horizon_dids( "switcher_fraction": N_l / N_1 if N_1 > 0 else float("nan"), } + # Attach A11 warnings to the results for the caller to surface + if a11_multi_warnings: + results["_a11_warnings"] = a11_multi_warnings # type: ignore[assignment] + return results @@ -2393,8 +2427,9 @@ def _compute_multi_horizon_placebos( forward_idx = ref_idx + l d_base = int(baselines[g]) - # Switcher's backward outcome change - switcher_change = Y_mat[g, backward_idx] - Y_mat[g, ref_idx] + # Switcher's backward outcome change: reference minus pre-period + # (matching Phase 1 convention: Y_{ref} - Y_{earlier}) + switcher_change = Y_mat[g, ref_idx] - Y_mat[g, backward_idx] # Control pool: same baseline, not switched by forward_idx ctrl_indices = baseline_groups[d_base] @@ -2410,7 +2445,7 @@ def _compute_multi_horizon_placebos( pl_g_l[g] = 0.0 continue - ctrl_changes = Y_mat[ctrl_pool, backward_idx] - Y_mat[ctrl_pool, ref_idx] + ctrl_changes = Y_mat[ctrl_pool, ref_idx] - Y_mat[ctrl_pool, backward_idx] ctrl_avg = float(ctrl_changes.mean()) pl_g_l[g] = switcher_change - ctrl_avg @@ -2522,9 +2557,14 @@ def _compute_cost_benefit_delta( dose_l = 0.0 for g in np.where(eligible)[0]: f_g = first_switch_idx[g] - col = f_g - 1 + l - if col < D_mat.shape[1]: - dose_l += abs(float(D_mat[g, col] - baselines[g])) + # Cumulative dose: delta^D_{g,l} = sum_{k=0}^{l-1} |D_{g,F_g+k} - D_{g,1}| + # For binary treatment this equals l (each period contributes 1). + cum_dose = 0.0 + for k in range(l): + col_k = f_g + k + if col_k < D_mat.shape[1]: + cum_dose += abs(float(D_mat[g, col_k] - baselines[g])) + dose_l += cum_dose per_horizon_dose[l] = dose_l total_dose += dose_l @@ -2572,9 +2612,12 @@ def _compute_cost_benefit_delta( if switch_direction[g] != direction: continue f_g = first_switch_idx[g] - col = f_g - 1 + l - if col < D_mat.shape[1]: - dose_l += abs(float(D_mat[g, col] - baselines[g])) + cum_dose = 0.0 + for k in range(l): + col_k = f_g + k + if col_k < D_mat.shape[1]: + cum_dose += abs(float(D_mat[g, col_k] - baselines[g])) + dose_l += cum_dose dir_horizon_dose[l] = dose_l dir_dose += dose_l diff --git a/diff_diff/chaisemartin_dhaultfoeuille_results.py b/diff_diff/chaisemartin_dhaultfoeuille_results.py index a692064c..ff9c4242 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille_results.py +++ b/diff_diff/chaisemartin_dhaultfoeuille_results.py @@ -505,16 +505,22 @@ def summary(self, alpha: Optional[float] = None) -> str: ] ) - # --- Overall DID_M --- + # --- Overall --- + overall_label = ( + "Cost-Benefit Delta" + if self.L_max is not None and self.L_max >= 2 + else "DID_M (Contemporaneous-Switch ATT)" + ) + overall_row_label = "delta" if self.L_max is not None and self.L_max >= 2 else "DID_M" lines.extend( [ thin, - "DID_M (Contemporaneous-Switch ATT)".center(width), + overall_label.center(width), thin, header_row, thin, _format_inference_row( - "DID_M", + overall_row_label, self.overall_att, self.overall_se, self.overall_t_stat, @@ -772,7 +778,9 @@ def to_dataframe(self, level: str = "overall") -> pd.DataFrame: return pd.DataFrame( [ { - "estimand": "DID_M", + "estimand": ( + "delta" if self.L_max is not None and self.L_max >= 2 else "DID_M" + ), "effect": self.overall_att, "se": self.overall_se, "t_stat": self.overall_t_stat, diff --git a/tests/test_chaisemartin_dhaultfoeuille.py b/tests/test_chaisemartin_dhaultfoeuille.py index 278293f0..bb80a37f 100644 --- a/tests/test_chaisemartin_dhaultfoeuille.py +++ b/tests/test_chaisemartin_dhaultfoeuille.py @@ -1836,12 +1836,12 @@ def test_L_max_populates_event_study_effects(self, data): assert "n_obs" in entry assert entry["n_obs"] > 0 - def test_did_l_equals_did_m_at_l1(self, data): - """event_study_effects[1] must equal DID_M from Phase 1.""" + def test_did_l1_uses_per_group_path_when_L_max(self, data): + """When L_max >= 2, event_study_effects[1] uses the per-group + DID_{g,1} path (consistent with horizons 2..L_max), which may + differ from the Phase 1 per-period DID_M. The per-period DID_M + is still available via the L_max=None path.""" est = ChaisemartinDHaultfoeuille(placebo=False, twfe_diagnostic=False) - r_none = est.fit( - data, outcome="outcome", group="group", time="period", treatment="treatment" - ) r_multi = est.fit( data, outcome="outcome", @@ -1850,7 +1850,9 @@ def test_did_l_equals_did_m_at_l1(self, data): treatment="treatment", L_max=3, ) - assert r_multi.event_study_effects[1]["effect"] == pytest.approx(r_none.overall_att) + # event_study_effects[1] is populated and finite + assert np.isfinite(r_multi.event_study_effects[1]["effect"]) + assert np.isfinite(r_multi.event_study_effects[1]["se"]) def test_N_l_decreases_with_horizon(self, data): """n_obs generally decreases for far horizons.""" diff --git a/tests/test_chaisemartin_dhaultfoeuille_parity.py b/tests/test_chaisemartin_dhaultfoeuille_parity.py index c595317c..2dacd957 100644 --- a/tests/test_chaisemartin_dhaultfoeuille_parity.py +++ b/tests/test_chaisemartin_dhaultfoeuille_parity.py @@ -269,12 +269,16 @@ def test_parity_leavers_only_multi_horizon(self, golden_values): def test_parity_mixed_single_switch_multi_horizon(self, golden_values): self._check_multi_horizon( - golden_values, "mixed_single_switch_multi_horizon", - L_max=5, rtol=self.MIXED_POINT_RTOL, + golden_values, + "mixed_single_switch_multi_horizon", + L_max=5, + rtol=self.MIXED_POINT_RTOL, ) def test_parity_joiners_only_long_multi_horizon(self, golden_values): self._check_multi_horizon( - golden_values, "joiners_only_long_multi_horizon", - L_max=5, rtol=self.POINT_RTOL, + golden_values, + "joiners_only_long_multi_horizon", + L_max=5, + rtol=self.POINT_RTOL, ) From 32f7a5341941d886c1ee77b1363c86510224e0bd Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 12 Apr 2026 13:21:30 -0400 Subject: [PATCH 03/11] Fix Round 2: placebo sign, delta Lemma 4 weights, placebo A11 warnings, labels - Flip placebo to paper convention (backward - ref) for R parity - Revert cost-benefit delta to per-period dose (Lemma 4: w_l proportional to N_l for binary, not l * N_l) - Add placebo control-availability warnings mirroring DID path - Update __repr__ and joiners_leavers labels for delta when L_max > 1 Co-Authored-By: Claude Opus 4.6 (1M context) --- diff_diff/chaisemartin_dhaultfoeuille.py | 50 ++++++++++++------- .../chaisemartin_dhaultfoeuille_results.py | 6 ++- 2 files changed, 35 insertions(+), 21 deletions(-) diff --git a/diff_diff/chaisemartin_dhaultfoeuille.py b/diff_diff/chaisemartin_dhaultfoeuille.py index c076cb05..b8abe52b 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille.py +++ b/diff_diff/chaisemartin_dhaultfoeuille.py @@ -1149,6 +1149,19 @@ def fit( T_g=T_g_arr, L_max=L_max, ) + # Surface placebo A11 warnings + pl_a11 = multi_horizon_placebos.pop("_a11_warnings", None) + if pl_a11: + warnings.warn( + f"Multi-horizon placebo control-availability " + f"violations in {len(pl_a11)} (group, lag) pair(s): " + f"affected DID^{{pl}}_l values are zeroed but " + f"retained in N^{{pl}}_l. Examples: " + + ", ".join(pl_a11[:3]) + + (f" (and {len(pl_a11) - 3} more)" if len(pl_a11) > 3 else ""), + UserWarning, + stacklevel=2, + ) # Normalized effects DID^n_l normalized_effects_dict = _compute_normalized_effects( @@ -2388,6 +2401,7 @@ def _compute_multi_horizon_placebos( baseline_f[int(d)] = first_switch_idx[mask] results: Dict[int, Dict[str, Any]] = {} + a11_placebo_warnings: List[str] = [] for l in range(1, L_max + 1): # noqa: E741 eligible = np.zeros(n_groups, dtype=bool) @@ -2427,9 +2441,9 @@ def _compute_multi_horizon_placebos( forward_idx = ref_idx + l d_base = int(baselines[g]) - # Switcher's backward outcome change: reference minus pre-period - # (matching Phase 1 convention: Y_{ref} - Y_{earlier}) - switcher_change = Y_mat[g, ref_idx] - Y_mat[g, backward_idx] + # Switcher's backward outcome change: pre-period minus reference + # (paper convention: Y_{F_g-1-l} - Y_{F_g-1}) + switcher_change = Y_mat[g, backward_idx] - Y_mat[g, ref_idx] # Control pool: same baseline, not switched by forward_idx ctrl_indices = baseline_groups[d_base] @@ -2443,9 +2457,10 @@ def _compute_multi_horizon_placebos( if ctrl_pool.size == 0: pl_g_l[g] = 0.0 + a11_placebo_warnings.append(f"placebo lag {l}, group_idx {g}: no controls") continue - ctrl_changes = Y_mat[ctrl_pool, ref_idx] - Y_mat[ctrl_pool, backward_idx] + ctrl_changes = Y_mat[ctrl_pool, backward_idx] - Y_mat[ctrl_pool, ref_idx] ctrl_avg = float(ctrl_changes.mean()) pl_g_l[g] = switcher_change - ctrl_avg @@ -2459,6 +2474,9 @@ def _compute_multi_horizon_placebos( "eligible_mask": eligible, } + if a11_placebo_warnings: + results["_a11_warnings"] = a11_placebo_warnings # type: ignore[assignment] + return results @@ -2545,7 +2563,9 @@ def _compute_cost_benefit_delta( dict with keys: delta, weights, has_leavers, delta_joiners, delta_leavers """ - # Total cumulative dose across all eligible (g, l) pairs + # Per-horizon dose via Lemma 4: w_l uses the PER-PERIOD dose + # D_{g,F_g-1+l} - D_{g,1} (NOT the cumulative delta^D_{g,l}). + # For binary joiners this is 1 per (g,l) pair, so w_l = N_l / sum N_l'. total_dose = 0.0 per_horizon_dose: Dict[int, float] = {} for l in range(1, L_max + 1): # noqa: E741 @@ -2557,14 +2577,9 @@ def _compute_cost_benefit_delta( dose_l = 0.0 for g in np.where(eligible)[0]: f_g = first_switch_idx[g] - # Cumulative dose: delta^D_{g,l} = sum_{k=0}^{l-1} |D_{g,F_g+k} - D_{g,1}| - # For binary treatment this equals l (each period contributes 1). - cum_dose = 0.0 - for k in range(l): - col_k = f_g + k - if col_k < D_mat.shape[1]: - cum_dose += abs(float(D_mat[g, col_k] - baselines[g])) - dose_l += cum_dose + col = f_g - 1 + l + if col < D_mat.shape[1]: + dose_l += abs(float(D_mat[g, col] - baselines[g])) per_horizon_dose[l] = dose_l total_dose += dose_l @@ -2612,12 +2627,9 @@ def _compute_cost_benefit_delta( if switch_direction[g] != direction: continue f_g = first_switch_idx[g] - cum_dose = 0.0 - for k in range(l): - col_k = f_g + k - if col_k < D_mat.shape[1]: - cum_dose += abs(float(D_mat[g, col_k] - baselines[g])) - dose_l += cum_dose + col = f_g - 1 + l + if col < D_mat.shape[1]: + dose_l += abs(float(D_mat[g, col] - baselines[g])) dir_horizon_dose[l] = dose_l dir_dose += dose_l diff --git a/diff_diff/chaisemartin_dhaultfoeuille_results.py b/diff_diff/chaisemartin_dhaultfoeuille_results.py index ff9c4242..cfd75965 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille_results.py +++ b/diff_diff/chaisemartin_dhaultfoeuille_results.py @@ -411,9 +411,10 @@ class ChaisemartinDHaultfoeuilleResults: def __repr__(self) -> str: """Concise string representation.""" sig = _get_significance_stars(self.overall_p_value) + label = "delta" if self.L_max is not None and self.L_max >= 2 else "DID_M" return ( f"ChaisemartinDHaultfoeuilleResults(" - f"DID_M={self.overall_att:.4f}{sig}, " + f"{label}={self.overall_att:.4f}{sig}, " f"SE={self.overall_se:.4f}, " f"n_groups={len(self.groups)}, " f"n_switcher_cells={self.n_switcher_cells})" @@ -802,9 +803,10 @@ def to_dataframe(self, level: str = "overall") -> pd.DataFrame: # For the DID_M row, both quantities use the overall switching # cell set: n_cells = sum of joiner + leaver cells, and n_obs # is the same sum of raw observation counts. + overall_est_label = "delta" if self.L_max is not None and self.L_max >= 2 else "DID_M" rows = [ { - "estimand": "DID_M", + "estimand": overall_est_label, "effect": self.overall_att, "se": self.overall_se, "t_stat": self.overall_t_stat, From 00efe07e9bd145776ebaa1d965d9fa6d05a9445c Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 12 Apr 2026 13:36:57 -0400 Subject: [PATCH 04/11] Fix Round 3: delta SE via delta-method, placebo SE deferral Note, summary gate - Compute cost-benefit delta SE from per-horizon SEs via delta method: SE(delta) = sqrt(sum w_l^2 * SE(DID_l)^2), giving overall_att non-NaN inference when L_max > 1 - Document placebo SE NaN as intentional Phase 2 deferral in REGISTRY.md (placebo IF computation deferred; point estimates meaningful for visual pre-trends; bootstrap plumbing exists but not wired) - Gate summary() bootstrap note on non-NaN overall inference - Remove unused import Co-Authored-By: Claude Opus 4.6 (1M context) --- diff_diff/chaisemartin_dhaultfoeuille.py | 36 ++++++++++++++----- .../chaisemartin_dhaultfoeuille_results.py | 7 +++- docs/methodology/REGISTRY.md | 4 +++ 3 files changed, 38 insertions(+), 9 deletions(-) diff --git a/diff_diff/chaisemartin_dhaultfoeuille.py b/diff_diff/chaisemartin_dhaultfoeuille.py index b8abe52b..bebd01df 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille.py +++ b/diff_diff/chaisemartin_dhaultfoeuille.py @@ -1503,14 +1503,34 @@ def fit( delta_val = cost_benefit_result["delta"] if np.isfinite(delta_val): effective_overall_att = delta_val - # Cost-benefit SE: use the weighted-average SE from the - # bootstrap when available; analytical SE for delta is not - # derived in the paper. For now, set to NaN (bootstrap will - # override if n_bootstrap > 0). - effective_overall_se = float("nan") - effective_overall_t = float("nan") - effective_overall_p = float("nan") - effective_overall_ci = (float("nan"), float("nan")) + # Cost-benefit delta SE: compute from per-horizon bootstrap + # distributions if available (delta = sum w_l * DID_l, so + # delta_b = sum w_l * DID_l_b for each bootstrap rep). + delta_se = float("nan") + if bootstrap_results is not None and bootstrap_results.event_study_ses is not None: + # The mixin stores overall_dist for l=1; we need + # per-horizon distributions which were computed but + # not all stored. Use the delta-method SE as fallback: + # Var(delta) = sum_l w_l^2 * Var(DID_l) for indep. + weights = cost_benefit_result.get("weights", {}) + var_delta = 0.0 + for l_w, w_l in weights.items(): + se_l = event_study_effects.get(l_w, {}).get("se", float("nan")) + if np.isfinite(se_l): + var_delta += (w_l * se_l) ** 2 + if var_delta > 0: + delta_se = float(np.sqrt(var_delta)) + + if np.isfinite(delta_se): + effective_overall_se = delta_se + effective_overall_t, effective_overall_p, effective_overall_ci = safe_inference( + delta_val, delta_se, alpha=self.alpha, df=None + ) + else: + effective_overall_se = float("nan") + effective_overall_t = float("nan") + effective_overall_p = float("nan") + effective_overall_ci = (float("nan"), float("nan")) # Phase 2: build placebo_event_study with negative keys placebo_event_study_dict: Optional[Dict[int, Dict[str, Any]]] = None diff --git a/diff_diff/chaisemartin_dhaultfoeuille_results.py b/diff_diff/chaisemartin_dhaultfoeuille_results.py index cfd75965..897accf8 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille_results.py +++ b/diff_diff/chaisemartin_dhaultfoeuille_results.py @@ -540,12 +540,17 @@ def summary(self, alpha: Optional[float] = None) -> str: lines.append(f"{'CV (SE/|DID_M|):':<25} {cv:>10.4f}") lines.append("") - if self.bootstrap_results is not None: + if self.bootstrap_results is not None and np.isfinite(self.overall_se): lines.append("Note: p-value and CI are multiplier-bootstrap percentile inference") lines.append( f" ({self.bootstrap_results.n_bootstrap} iterations, " f"{self.bootstrap_results.weight_type} weights)." ) + elif self.bootstrap_results is not None: + lines.append( + f"Note: bootstrap ({self.bootstrap_results.n_bootstrap} iterations) " + f"used for event-study horizon inference." + ) else: lines.append( "Note: dCDH analytical CI is conservative under Assumption 8" diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 1ee28bde..41861916 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -537,6 +537,10 @@ Dynamic placebos `DID^{pl}_l` look backward from each group's reference period, - **Note (Phase 2 Assumption 7 and cost-benefit delta):** Assumption 7 (`D_{g,t} >= D_{g,1}`) is required for the single-sign cost-benefit interpretation. When leavers are present (binary: 1->0 groups violate Assumption 7), the estimator emits a `UserWarning` and provides `delta_joiners` / `delta_leavers` separately on `results.cost_benefit_delta`. +- **Note (Phase 2 cost-benefit delta SE):** When `L_max >= 2`, `overall_att` holds the cost-benefit `delta`. Its SE is computed via the delta method from per-horizon SEs: `SE(delta) = sqrt(sum w_l^2 * SE(DID_l)^2)`, treating horizons as independent (conservative under Assumption 8). When bootstrap is enabled, per-horizon bootstrap SEs flow through the delta-method formula. + +- **Note (Phase 2 dynamic placebo SE):** Dynamic placebos `DID^{pl}_l` (negative horizons in `placebo_event_study`) ship as point estimates with `NaN` inference in Phase 2. The placebo influence-function derivation follows the same cohort-recentered structure as the positive horizons but requires a separate IF computation for the backward outcome differences, which is deferred. The placebo point estimates are meaningful for visual pre-trends inspection; formal placebo inference will be added in a follow-up. Bootstrap placebo inference plumbing exists in the mixin but is not wired. + *Standard errors (Web Appendix Section 3.7.3 of the dynamic companion paper):* Default: cohort-recentered analytical plug-in variance, evaluated at horizon `l = 1`. Cohorts are defined by the triple `(D_{g,1}, F_g, S_g)` (baseline treatment, first-switch period, switch direction). Each group's per-period role weights (joiner, stable_0, leaver, stable_1) sum to a per-group `U^G_g` value via the full `Lambda^G_{g,l=1}` weight vector from Section 3.7.2 of the dynamic paper: From 41bc1e8c3ca5a03b8bc1965782a782d1d17b72bf Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 12 Apr 2026 13:52:38 -0400 Subject: [PATCH 05/11] Fix Round 4: delta SE on analytical path, shared bootstrap, equal-cell Note - Compute delta-method SE regardless of bootstrap (was gated on bootstrap_results != None, leaving analytical path with NaN) - Generate one shared bootstrap weight matrix for all horizons so sup-t bands are a valid joint multiplier-bootstrap band - Add REGISTRY Note for Phase 2 equal-cell weighting deviation Co-Authored-By: Claude Opus 4.6 (1M context) --- diff_diff/chaisemartin_dhaultfoeuille.py | 26 +++++++------- .../chaisemartin_dhaultfoeuille_bootstrap.py | 36 +++++++++++-------- docs/methodology/REGISTRY.md | 2 ++ 3 files changed, 36 insertions(+), 28 deletions(-) diff --git a/diff_diff/chaisemartin_dhaultfoeuille.py b/diff_diff/chaisemartin_dhaultfoeuille.py index bebd01df..0440d82d 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille.py +++ b/diff_diff/chaisemartin_dhaultfoeuille.py @@ -1506,20 +1506,18 @@ def fit( # Cost-benefit delta SE: compute from per-horizon bootstrap # distributions if available (delta = sum w_l * DID_l, so # delta_b = sum w_l * DID_l_b for each bootstrap rep). - delta_se = float("nan") - if bootstrap_results is not None and bootstrap_results.event_study_ses is not None: - # The mixin stores overall_dist for l=1; we need - # per-horizon distributions which were computed but - # not all stored. Use the delta-method SE as fallback: - # Var(delta) = sum_l w_l^2 * Var(DID_l) for indep. - weights = cost_benefit_result.get("weights", {}) - var_delta = 0.0 - for l_w, w_l in weights.items(): - se_l = event_study_effects.get(l_w, {}).get("se", float("nan")) - if np.isfinite(se_l): - var_delta += (w_l * se_l) ** 2 - if var_delta > 0: - delta_se = float(np.sqrt(var_delta)) + # Delta-method SE: Var(delta) = sum w_l^2 * Var(DID_l) + # (treating horizons as independent, conservative under + # Assumption 8). Works on both analytical and bootstrap + # SEs since event_study_effects[l]["se"] holds whichever + # was propagated. + weights = cost_benefit_result.get("weights", {}) + var_delta = 0.0 + for l_w, w_l in weights.items(): + se_l = event_study_effects.get(l_w, {}).get("se", float("nan")) + if np.isfinite(se_l): + var_delta += (w_l * se_l) ** 2 + delta_se = float(np.sqrt(var_delta)) if var_delta > 0 else float("nan") if np.isfinite(delta_se): effective_overall_se = delta_se diff --git a/diff_diff/chaisemartin_dhaultfoeuille_bootstrap.py b/diff_diff/chaisemartin_dhaultfoeuille_bootstrap.py index f06cc002..89894dc8 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille_bootstrap.py +++ b/diff_diff/chaisemartin_dhaultfoeuille_bootstrap.py @@ -251,25 +251,36 @@ def _compute_dcdh_bootstrap( results.placebo_ci = ci_pl results.placebo_p_value = p_pl - # --- Phase 2: Multi-horizon bootstrap --- + # --- Phase 2: Multi-horizon bootstrap with shared weight matrix --- + # Generate ONE shared (n_bootstrap, n_groups) weight matrix so all + # horizons use the same bootstrap draw, making the sup-t statistic + # a valid joint multiplier-bootstrap band. if multi_horizon_inputs is not None: es_ses: Dict[int, float] = {} es_cis: Dict[int, Tuple[float, float]] = {} es_pvals: Dict[int, float] = {} es_dists: Dict[int, np.ndarray] = {} + # Shared weight matrix sized for the group set + n_groups_mh = n_groups_for_overall + shared_weights = _generate_bootstrap_weights_batch( + n_bootstrap=self.n_bootstrap, + n_units=n_groups_mh, + weight_type=self.bootstrap_weights, + rng=rng, + ) + for l_h, (u_h, n_h, eff_h) in sorted(multi_horizon_inputs.items()): if u_h.size > 0 and n_h > 0: - se_h, ci_h, p_h, dist_h = _bootstrap_one_target( - u_centered=u_h, - divisor=n_h, - original=eff_h, - n_bootstrap=self.n_bootstrap, - weight_type=self.bootstrap_weights, + # Use the shared weight matrix truncated to u_h length + w_h = shared_weights[:, : u_h.size] + deviations = (w_h @ u_h) / n_h + dist_h = deviations + eff_h + + se_h, ci_h, p_h = _compute_effect_bootstrap_stats( + original_effect=eff_h, + boot_dist=dist_h, alpha=self.alpha, - rng=rng, - context=f"dCDH horizon l={l_h} bootstrap", - return_distribution=True, ) es_ses[l_h] = se_h es_cis[l_h] = ci_h @@ -280,9 +291,7 @@ def _compute_dcdh_bootstrap( results.event_study_cis = es_cis results.event_study_p_values = es_pvals - # Sup-t simultaneous confidence bands (CallawaySantAnna pattern - # from staggered_bootstrap.py:497-533): for each bootstrap rep, - # compute the max absolute t-stat across horizons. + # Sup-t simultaneous confidence bands using the shared draws. valid_horizons = [ l_h for l_h in es_dists @@ -292,7 +301,6 @@ def _compute_dcdh_bootstrap( boot_matrix = np.array([es_dists[l_h] for l_h in valid_horizons]) effects_vec = np.array([multi_horizon_inputs[l_h][2] for l_h in valid_horizons]) ses_vec = np.array([es_ses[l_h] for l_h in valid_horizons]) - # sup_t_dist[b] = max_l |(boot_l[b] - DID_l) / SE_l| t_stats = np.abs((boot_matrix - effects_vec[:, None]) / ses_vec[:, None]) sup_t_dist = np.max(t_stats, axis=0) finite_mask = np.isfinite(sup_t_dist) diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 41861916..030ccfbb 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -533,6 +533,8 @@ Cost-benefit aggregate `delta = sum_l w_l * DID_l` (Lemma 4) where `w_l` are non Dynamic placebos `DID^{pl}_l` look backward from each group's reference period, with a dual eligibility condition: `F_g - 1 - l >= 1` AND `F_g - 1 + l <= T_g`. +- **Note (Phase 2 equal-cell weighting, deviation from R `DIDmultiplegtDYN`):** The Phase 1 equal-cell weighting contract carries forward to all Phase 2 estimands (`DID_l`, `DID^{pl}_l`, `DID^n_l`, `delta`). Each `(g, t)` cell contributes equally regardless of within-cell observation count. On individual-level inputs with uneven cell sizes, this produces a different estimand than R `DIDmultiplegtDYN` which weights by cell size. The parity tests use one-observation-per-cell generators so parity holds. See the Phase 1 weighting Note above for the full rationale. + - **Note (Phase 2 `<50%` switcher warning):** When fewer than 50% of the l=1 switchers contribute at a far horizon l, `fit()` emits a `UserWarning`. The paper recommends not reporting such horizons (Favara-Imbs application, footnote 14). - **Note (Phase 2 Assumption 7 and cost-benefit delta):** Assumption 7 (`D_{g,t} >= D_{g,1}`) is required for the single-sign cost-benefit interpretation. When leavers are present (binary: 1->0 groups violate Assumption 7), the estimator emits a `UserWarning` and provides `delta_joiners` / `delta_leavers` separately on `results.cost_benefit_delta`. From a7f32990c6b2b9f8d5e3e1e630e0e08866775323 Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 12 Apr 2026 14:20:09 -0400 Subject: [PATCH 06/11] Fix Round 5: exclude empty-control groups from N_l instead of zero-retaining When a switcher's control pool is empty at a given horizon (e.g., due to terminal missingness), exclude the group from N_l / N_pl_l rather than zero-retaining it. Zero-retention biases DID_l toward zero on ragged panels. Reserve zero-contribution for the IF path only (where it naturally has no effect). Same fix applied to _compute_multi_horizon_placebos. Co-Authored-By: Claude Opus 4.6 (1M context) --- diff_diff/chaisemartin_dhaultfoeuille.py | 35 ++++++++++++++++++++---- 1 file changed, 30 insertions(+), 5 deletions(-) diff --git a/diff_diff/chaisemartin_dhaultfoeuille.py b/diff_diff/chaisemartin_dhaultfoeuille.py index 0440d82d..71ddbfa4 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille.py +++ b/diff_diff/chaisemartin_dhaultfoeuille.py @@ -2240,10 +2240,11 @@ def _compute_multi_horizon_dids( ctrl_pool = ctrl_indices[ctrl_mask] if ctrl_pool.size == 0: - # No controls available - A11-like situation. Set to 0 - # matching the A11 zero-retention convention: the group's - # switcher count is still in N_l. - did_g_l[g] = 0.0 + # No observed controls at this horizon (may be terminal + # missingness, not a true A11 violation). Exclude the + # group from N_l rather than zero-retaining, so the + # missing-data case doesn't bias DID_l toward zero. + eligible[g] = False a11_multi_warnings.append( f"horizon {l}, group_idx {g}: " f"no baseline-matched controls at outcome period" @@ -2254,6 +2255,20 @@ def _compute_multi_horizon_dids( ctrl_avg = float(ctrl_changes.mean()) did_g_l[g] = switcher_change - ctrl_avg + # Recompute N_l after control-pool exclusions + N_l = int(eligible.sum()) + if l == 1: + N_1 = N_l + if N_l == 0: + results[l] = { + "did_l": float("nan"), + "N_l": 0, + "did_g_l": did_g_l, + "eligible_mask": eligible, + "switcher_fraction": float("nan"), + } + continue + # Aggregate: DID_l = (1/N_l) * sum S_g * DID_{g,l} S_eligible = switch_direction[eligible].astype(float) did_g_eligible = did_g_l[eligible] @@ -2474,7 +2489,7 @@ def _compute_multi_horizon_placebos( ctrl_pool = ctrl_indices[ctrl_mask] if ctrl_pool.size == 0: - pl_g_l[g] = 0.0 + eligible[g] = False a11_placebo_warnings.append(f"placebo lag {l}, group_idx {g}: no controls") continue @@ -2482,6 +2497,16 @@ def _compute_multi_horizon_placebos( ctrl_avg = float(ctrl_changes.mean()) pl_g_l[g] = switcher_change - ctrl_avg + # Recompute N_pl_l after control-pool exclusions + N_pl_l = int(eligible.sum()) + if N_pl_l == 0: + results[l] = { + "placebo_l": float("nan"), + "N_pl_l": 0, + "eligible_mask": eligible, + } + continue + S_eligible = switch_direction[eligible].astype(float) pl_g_eligible = pl_g_l[eligible] placebo_l = float((S_eligible * pl_g_eligible).sum() / N_pl_l) From 4d71c59d7320d3a21343cb72aa6455a0fbe1fa98 Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 12 Apr 2026 14:34:04 -0400 Subject: [PATCH 07/11] Fix Round 6: align IF/SE eligibility with DID path, document l=1 and delta contracts - IF/SE/bootstrap paths now use the same combined eligibility mask as _compute_multi_horizon_dids (singleton-baseline + empty-control-pool exclusions), so point estimate and inference agree on terminal-missing panels - Add REGISTRY Note documenting that event_study_effects[1] uses per-group DID_{g,1} (cohort-based controls) when L_max >= 2, which may differ from Phase 1 DID_M (period-based controls) on mixed-direction panels - Add REGISTRY Note documenting that delta SE uses delta-method (normal-theory) even when bootstrap is enabled, as an intentional exception to the bootstrap-inference-surface contract Co-Authored-By: Claude Opus 4.6 (1M context) --- diff_diff/chaisemartin_dhaultfoeuille.py | 18 +++++++++++++----- docs/methodology/REGISTRY.md | 4 +++- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/diff_diff/chaisemartin_dhaultfoeuille.py b/diff_diff/chaisemartin_dhaultfoeuille.py index 71ddbfa4..7e693d5a 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille.py +++ b/diff_diff/chaisemartin_dhaultfoeuille.py @@ -1093,8 +1093,13 @@ def fit( unique_c[key] = len(unique_c) cid_l[g] = unique_c[key] - U_l_elig = U_l[eligible_mask_var] - cid_elig = cid_l[eligible_mask_var] + # Combine singleton-baseline exclusion with the finalized + # eligible_mask from _compute_multi_horizon_dids (which + # excludes groups with empty control pools). + did_eligible = multi_horizon_dids[l_h]["eligible_mask"] + combined_mask = eligible_mask_var & did_eligible + U_l_elig = U_l[combined_mask] + cid_elig = cid_l[combined_mask] U_centered_l = _cohort_recenter(U_l_elig, cid_elig) N_l_h = multi_horizon_dids[l_h]["N_l"] se_l = _plugin_se(U_centered=U_centered_l, divisor=N_l_h) @@ -1356,7 +1361,10 @@ def fit( if h_data is None or h_data["N_l"] == 0: continue U_l_full = multi_horizon_if[l_h] - U_l_elig = U_l_full[eligible_mask_b] + # Use same combined mask as analytical SE path + did_eligible_b = h_data["eligible_mask"] + combined_b = eligible_mask_b & did_eligible_b + U_l_elig = U_l_full[combined_b] # Use the same cohort IDs as the analytical SE path cohort_keys_b = [ ( @@ -1369,14 +1377,14 @@ def fit( unique_cb: Dict[Tuple[int, int, int], int] = {} cid_b = np.zeros(len(all_groups), dtype=int) for g in range(len(all_groups)): - if not eligible_mask_b[g]: + if not combined_b[g]: cid_b[g] = -1 continue key = cohort_keys_b[g] if key not in unique_cb: unique_cb[key] = len(unique_cb) cid_b[g] = unique_cb[key] - cid_elig = cid_b[eligible_mask_b] + cid_elig = cid_b[combined_b] U_centered_h = _cohort_recenter(U_l_elig, cid_elig) mh_boot_inputs[l_h] = ( U_centered_h, diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 030ccfbb..ce85863b 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -533,13 +533,15 @@ Cost-benefit aggregate `delta = sum_l w_l * DID_l` (Lemma 4) where `w_l` are non Dynamic placebos `DID^{pl}_l` look backward from each group's reference period, with a dual eligibility condition: `F_g - 1 - l >= 1` AND `F_g - 1 + l <= T_g`. +- **Note (Phase 2 `DID_1` vs Phase 1 `DID_M`):** When `L_max >= 2`, `event_study_effects[1]` uses the per-group `DID_{g,1}` building block (Equation 3 of the dynamic paper) with cohort-based controls, which may differ slightly from the Phase 1 `DID_M` value (Theorem 3 of AER 2020 with period-based stable-control sets). The Phase 1 `DID_M` value remains accessible via `fit(..., L_max=None).overall_att`. The difference arises because the per-group path conditions on baseline treatment `D_{g,1}` when selecting controls, while the per-period path does not. On pure-direction panels (all joiners or all leavers) the two agree; on mixed-direction panels they can differ by O(1%). This is the same period-vs-cohort control-set deviation documented in the Phase 1 Note above, extended to the `l=1` event-study entry. + - **Note (Phase 2 equal-cell weighting, deviation from R `DIDmultiplegtDYN`):** The Phase 1 equal-cell weighting contract carries forward to all Phase 2 estimands (`DID_l`, `DID^{pl}_l`, `DID^n_l`, `delta`). Each `(g, t)` cell contributes equally regardless of within-cell observation count. On individual-level inputs with uneven cell sizes, this produces a different estimand than R `DIDmultiplegtDYN` which weights by cell size. The parity tests use one-observation-per-cell generators so parity holds. See the Phase 1 weighting Note above for the full rationale. - **Note (Phase 2 `<50%` switcher warning):** When fewer than 50% of the l=1 switchers contribute at a far horizon l, `fit()` emits a `UserWarning`. The paper recommends not reporting such horizons (Favara-Imbs application, footnote 14). - **Note (Phase 2 Assumption 7 and cost-benefit delta):** Assumption 7 (`D_{g,t} >= D_{g,1}`) is required for the single-sign cost-benefit interpretation. When leavers are present (binary: 1->0 groups violate Assumption 7), the estimator emits a `UserWarning` and provides `delta_joiners` / `delta_leavers` separately on `results.cost_benefit_delta`. -- **Note (Phase 2 cost-benefit delta SE):** When `L_max >= 2`, `overall_att` holds the cost-benefit `delta`. Its SE is computed via the delta method from per-horizon SEs: `SE(delta) = sqrt(sum w_l^2 * SE(DID_l)^2)`, treating horizons as independent (conservative under Assumption 8). When bootstrap is enabled, per-horizon bootstrap SEs flow through the delta-method formula. +- **Note (Phase 2 cost-benefit delta SE):** When `L_max >= 2`, `overall_att` holds the cost-benefit `delta`. Its SE is computed via the delta method from per-horizon SEs: `SE(delta) = sqrt(sum w_l^2 * SE(DID_l)^2)`, treating horizons as independent (conservative under Assumption 8). When bootstrap is enabled, per-horizon bootstrap SEs flow through the delta-method formula, so `overall_se` reflects bootstrap-derived per-horizon uncertainty but the delta aggregation itself uses normal-theory (not bootstrap percentile). This is an intentional exception to the general bootstrap-inference-surface contract: `overall_p_value` and `overall_conf_int` for `delta` use `safe_inference(delta, delta_se)`, not percentile bootstrap, because the delta is a derived aggregate rather than a directly bootstrapped estimand. - **Note (Phase 2 dynamic placebo SE):** Dynamic placebos `DID^{pl}_l` (negative horizons in `placebo_event_study`) ship as point estimates with `NaN` inference in Phase 2. The placebo influence-function derivation follows the same cohort-recentered structure as the positive horizons but requires a separate IF computation for the backward outcome differences, which is deferred. The placebo point estimates are meaningful for visual pre-trends inspection; formal placebo inference will be added in a follow-up. Bootstrap placebo inference plumbing exists in the mixin but is not wired. From 8cac82d46f72e85861d031788783fb2b538ad382 Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 12 Apr 2026 14:51:52 -0400 Subject: [PATCH 08/11] Fix Round 7: NaN-consistent delta SE, full-group IF for variance - Delta SE now requires ALL positively-weighted horizons to have finite SE; if any has NaN, overall_se/p/CI are all NaN (NaN-consistent inference contract) - IF/SE/bootstrap paths use full variance-eligible group set (singleton- baseline filter only), not the switcher-only did_eligible mask. Never- switchers and later-switching controls with non-zero IF mass from their control roles are now included, matching the Phase 1 IF contract. Co-Authored-By: Claude Opus 4.6 (1M context) --- diff_diff/chaisemartin_dhaultfoeuille.py | 41 +++++++++++++++--------- 1 file changed, 25 insertions(+), 16 deletions(-) diff --git a/diff_diff/chaisemartin_dhaultfoeuille.py b/diff_diff/chaisemartin_dhaultfoeuille.py index 7e693d5a..e3f5126b 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille.py +++ b/diff_diff/chaisemartin_dhaultfoeuille.py @@ -1093,13 +1093,13 @@ def fit( unique_c[key] = len(unique_c) cid_l[g] = unique_c[key] - # Combine singleton-baseline exclusion with the finalized - # eligible_mask from _compute_multi_horizon_dids (which - # excludes groups with empty control pools). - did_eligible = multi_horizon_dids[l_h]["eligible_mask"] - combined_mask = eligible_mask_var & did_eligible - U_l_elig = U_l[combined_mask] - cid_elig = cid_l[combined_mask] + # Use the full variance-eligible group set (singleton- + # baseline exclusion only). Do NOT intersect with + # did_eligible — never-switchers and later-switching + # controls can have non-zero IF mass via their control + # roles, and dropping them understates the SE. + U_l_elig = U_l[eligible_mask_var] + cid_elig = cid_l[eligible_mask_var] U_centered_l = _cohort_recenter(U_l_elig, cid_elig) N_l_h = multi_horizon_dids[l_h]["N_l"] se_l = _plugin_se(U_centered=U_centered_l, divisor=N_l_h) @@ -1361,10 +1361,9 @@ def fit( if h_data is None or h_data["N_l"] == 0: continue U_l_full = multi_horizon_if[l_h] - # Use same combined mask as analytical SE path - did_eligible_b = h_data["eligible_mask"] - combined_b = eligible_mask_b & did_eligible_b - U_l_elig = U_l_full[combined_b] + # Full variance-eligible group set (matching + # analytical SE path: singleton-baseline only) + U_l_elig = U_l_full[eligible_mask_b] # Use the same cohort IDs as the analytical SE path cohort_keys_b = [ ( @@ -1377,14 +1376,14 @@ def fit( unique_cb: Dict[Tuple[int, int, int], int] = {} cid_b = np.zeros(len(all_groups), dtype=int) for g in range(len(all_groups)): - if not combined_b[g]: + if not eligible_mask_b[g]: cid_b[g] = -1 continue key = cohort_keys_b[g] if key not in unique_cb: unique_cb[key] = len(unique_cb) cid_b[g] = unique_cb[key] - cid_elig = cid_b[combined_b] + cid_elig = cid_b[eligible_mask_b] U_centered_h = _cohort_recenter(U_l_elig, cid_elig) mh_boot_inputs[l_h] = ( U_centered_h, @@ -1519,13 +1518,23 @@ def fit( # Assumption 8). Works on both analytical and bootstrap # SEs since event_study_effects[l]["se"] holds whichever # was propagated. + # Require ALL positively-weighted horizons to have finite + # SE. If any has NaN, delta SE is NaN (NaN-consistent + # inference contract: no partial aggregation). weights = cost_benefit_result.get("weights", {}) var_delta = 0.0 + all_finite = True for l_w, w_l in weights.items(): + if w_l <= 0: + continue se_l = event_study_effects.get(l_w, {}).get("se", float("nan")) - if np.isfinite(se_l): - var_delta += (w_l * se_l) ** 2 - delta_se = float(np.sqrt(var_delta)) if var_delta > 0 else float("nan") + if not np.isfinite(se_l): + all_finite = False + break + var_delta += (w_l * se_l) ** 2 + delta_se = ( + float(np.sqrt(var_delta)) if all_finite and var_delta > 0 else float("nan") + ) if np.isfinite(delta_se): effective_overall_se = delta_se From 39f3978296dbe18608febbd28c52a2a4b4c66afa Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 12 Apr 2026 15:10:04 -0400 Subject: [PATCH 09/11] Clean up P3 nits: warning text, README wording, ROADMAP accuracy - Fix multi-horizon control-availability warnings to say "excluded from N_l" (not "zeroed and retained") - Update README overall_att description to reflect delta when L_max > 1 - Update README placebo Note to reflect current state (not "Phase 2 will add") - Update ROADMAP 2c and 2h status to reflect placebo SE and parity SE/placebo deferrals Co-Authored-By: Claude Opus 4.6 (1M context) --- README.md | 4 ++-- ROADMAP.md | 4 ++-- diff_diff/chaisemartin_dhaultfoeuille.py | 9 ++++----- 3 files changed, 8 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 65de0d64..fab664da 100644 --- a/README.md +++ b/README.md @@ -1205,7 +1205,7 @@ ChaisemartinDHaultfoeuille( | Field | Description | |-------|-------------| -| `overall_att`, `overall_se`, `overall_conf_int` | `DID_M` and inference (cohort-recentered analytical SE by default; multiplier-bootstrap percentile inference when `n_bootstrap > 0`) | +| `overall_att`, `overall_se`, `overall_conf_int` | `DID_M` when `L_max=None`; cost-benefit `delta` when `L_max > 1` (delta-method SE from per-horizon SEs) | | `joiners_att`, `leavers_att` | Decomposition into the joiners (`DID_+`) and leavers (`DID_-`) views | | `placebo_effect` | Single-lag placebo (`DID_M^pl`) point estimate | | `per_period_effects` | Per-period decomposition with explicit A11-violation flags | @@ -1252,7 +1252,7 @@ print(f"Fraction of negative weights: {diagnostic.fraction_negative:.3f}") print(f"sigma_fe (sign-flipping threshold): {diagnostic.sigma_fe:.3f}") ``` -> **Note:** The Phase 1 placebo SE is intentionally `NaN` with a warning. The dynamic companion paper Section 3.7.3 derives the cohort-recentered analytical variance for `DID_l` only — not for the placebo `DID_M^pl`. Phase 2 will add multiplier-bootstrap support for the placebo via the dynamic paper's machinery. Until then, the placebo point estimate is meaningful but its inference fields are NaN-consistent (and `results.placebo_se`, `results.placebo_p_value`, etc. remain `NaN` even when `n_bootstrap > 0`). +> **Note:** Placebo SE is `NaN` for both the single-lag `DID_M^pl` and the dynamic placebos `DID^{pl}_l`. The point estimates are meaningful for visual pre-trends inspection; formal placebo inference (influence-function derivation) is deferred to a follow-up. See `REGISTRY.md` for the full contract. > **Note:** By default (`drop_larger_lower=True`), the estimator drops groups whose treatment switches more than once before estimation. This matches R `DIDmultiplegtDYN`'s default and is required for the analytical variance formula to be consistent with the point estimate. Each drop emits an explicit warning. diff --git a/ROADMAP.md b/ROADMAP.md index a3778b48..326e2e46 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -154,12 +154,12 @@ The dynamic companion paper subsumes the AER 2020 paper: `DID_1 = DID_M`. The si |------|----------|--------| | **2a.** Multi-horizon `DID_l` via per-group `DID_{g,l}` building block, with `L_max` parameter | HIGH | Shipped | | **2b.** Multi-horizon analytical SE (cohort-recentered plug-in per horizon) | HIGH | Shipped | -| **2c.** Dynamic placebos `DID^{pl}_l` for pre-trends testing (Web Appendix Section 1.1 of dynamic paper) | HIGH | Shipped | +| **2c.** Dynamic placebos `DID^{pl}_l` for pre-trends testing (Web Appendix Section 1.1 of dynamic paper) | HIGH | Shipped (point estimates; SE deferred) | | **2d.** Normalized estimator `DID^n_l` (Section 3.2 of dynamic paper) | MEDIUM | Shipped | | **2e.** Cost-benefit aggregate `delta` (Section 3.3 of dynamic paper, Lemma 4) | MEDIUM | Shipped | | **2f.** Simultaneous (sup-t) confidence bands for event study plots | MEDIUM | Shipped | | **2g.** `plot_event_study()` integration; `< 50%`-of-switchers warning for far horizons | MEDIUM | Shipped | -| **2h.** Parity tests vs `did_multiplegt_dyn` for multi-horizon designs | HIGH | In progress | +| **2h.** Parity tests vs `did_multiplegt_dyn` for multi-horizon designs | HIGH | Shipped (point estimates; SE/placebo parity deferred) | ### Phase 3: Covariates, extensions, and tutorial diff --git a/diff_diff/chaisemartin_dhaultfoeuille.py b/diff_diff/chaisemartin_dhaultfoeuille.py index e3f5126b..3d5f9dd4 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille.py +++ b/diff_diff/chaisemartin_dhaultfoeuille.py @@ -1036,9 +1036,8 @@ def fit( warnings.warn( f"Multi-horizon control-availability violations in " f"{len(mh_a11)} (group, horizon) pair(s): affected " - f"DID_{{g,l}} values are zeroed but their switcher " - f"counts are retained in N_l (matching the A11 " - f"zero-retention convention). Examples: " + f"groups are excluded from N_l (no observed baseline-" + f"matched controls at the outcome period). Examples: " + ", ".join(mh_a11[:3]) + (f" (and {len(mh_a11) - 3} more)" if len(mh_a11) > 3 else ""), UserWarning, @@ -1160,8 +1159,8 @@ def fit( warnings.warn( f"Multi-horizon placebo control-availability " f"violations in {len(pl_a11)} (group, lag) pair(s): " - f"affected DID^{{pl}}_l values are zeroed but " - f"retained in N^{{pl}}_l. Examples: " + f"affected groups are excluded from N^{{pl}}_l " + f"(no observed controls). Examples: " + ", ".join(pl_a11[:3]) + (f" (and {len(pl_a11) - 3} more)" if len(pl_a11) > 3 else ""), UserWarning, From a3b01bc706eaead4d89640cd8b06523341a9929d Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 12 Apr 2026 15:32:58 -0400 Subject: [PATCH 10/11] Fix Round 9: placebo control forward-observation guard, NaN delta fallback - Add N_mat[ctrl, forward_idx] > 0 to placebo control mask so terminally missing controls don't leak into DID^{pl}_l - When delta is NaN (non-estimable), set all overall_* to NaN instead of silently falling back to the Phase 1 DID_M values Co-Authored-By: Claude Opus 4.6 (1M context) --- diff_diff/chaisemartin_dhaultfoeuille.py | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/diff_diff/chaisemartin_dhaultfoeuille.py b/diff_diff/chaisemartin_dhaultfoeuille.py index 3d5f9dd4..72ab9bd1 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille.py +++ b/diff_diff/chaisemartin_dhaultfoeuille.py @@ -1507,7 +1507,17 @@ def fit( effective_overall_ci = overall_ci if cost_benefit_result is not None and L_max is not None and L_max >= 2: delta_val = cost_benefit_result["delta"] - if np.isfinite(delta_val): + if not np.isfinite(delta_val): + # Delta is non-estimable (e.g., no eligible switchers at + # any horizon). Set all overall_* to NaN rather than + # silently falling back to the Phase 1 DID_M values, + # since the results surface labels them as delta. + effective_overall_att = float("nan") + effective_overall_se = float("nan") + effective_overall_t = float("nan") + effective_overall_p = float("nan") + effective_overall_ci = (float("nan"), float("nan")) + else: effective_overall_att = delta_val # Cost-benefit delta SE: compute from per-horizon bootstrap # distributions if available (delta = sum w_l * DID_l, so @@ -2494,13 +2504,17 @@ def _compute_multi_horizon_placebos( # (paper convention: Y_{F_g-1-l} - Y_{F_g-1}) switcher_change = Y_mat[g, backward_idx] - Y_mat[g, ref_idx] - # Control pool: same baseline, not switched by forward_idx + # Control pool: same baseline, not switched by forward_idx, + # AND observed at all three relevant periods (ref, backward, + # AND forward - the last ensures terminally missing controls + # don't leak into the placebo computation). ctrl_indices = baseline_groups[d_base] ctrl_f = baseline_f[d_base] ctrl_mask = ( ((ctrl_f > forward_idx) | (ctrl_f == -1)) & (N_mat[ctrl_indices, ref_idx] > 0) & (N_mat[ctrl_indices, backward_idx] > 0) + & (N_mat[ctrl_indices, forward_idx] > 0) ) ctrl_pool = ctrl_indices[ctrl_mask] From 4350d91f29ae4df6f5a646bea9ed691f55d5819f Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 12 Apr 2026 15:48:55 -0400 Subject: [PATCH 11/11] Fix summary delta-method note, update stale Phase 2 doc references - summary() now describes delta SE as delta-method (normal-theory) when L_max >= 2 with bootstrap, instead of claiming percentile - Update README, choosing_estimator.rst, REGISTRY.md to reflect shipped Phase 2 state (was: "Phase 2 will add") Co-Authored-By: Claude Opus 4.6 (1M context) --- README.md | 2 +- diff_diff/chaisemartin_dhaultfoeuille_results.py | 10 +++++++++- docs/choosing_estimator.rst | 11 ++++------- docs/methodology/REGISTRY.md | 2 +- 4 files changed, 15 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index fab664da..724a07e4 100644 --- a/README.md +++ b/README.md @@ -1157,7 +1157,7 @@ EfficientDiD( `ChaisemartinDHaultfoeuille` (alias `DCDH`) is the only library estimator that handles **non-absorbing (reversible) treatments** — treatment can switch on AND off over time. This is the natural fit for marketing campaigns, seasonal promotions, on/off policy cycles. -Phase 1 ships the contemporaneous-switch estimator `DID_M` from the AER 2020 paper, which is mathematically identical to `DID_1` (horizon `l = 1`) of the dynamic companion paper (NBER WP 29873). Phase 2 will add multi-horizon event-study output `DID_l` for `l > 1` on the same class; Phase 3 will add covariate adjustment. +Ships `DID_M` (= `DID_1` at horizon `l = 1`) plus the full multi-horizon event study `DID_l` for `l = 1..L_max` via the `L_max` parameter. Phase 3 will add covariate adjustment. ```python from diff_diff import ChaisemartinDHaultfoeuille diff --git a/diff_diff/chaisemartin_dhaultfoeuille_results.py b/diff_diff/chaisemartin_dhaultfoeuille_results.py index 897accf8..85c6f678 100644 --- a/diff_diff/chaisemartin_dhaultfoeuille_results.py +++ b/diff_diff/chaisemartin_dhaultfoeuille_results.py @@ -540,12 +540,20 @@ def summary(self, alpha: Optional[float] = None) -> str: lines.append(f"{'CV (SE/|DID_M|):':<25} {cv:>10.4f}") lines.append("") - if self.bootstrap_results is not None and np.isfinite(self.overall_se): + is_delta = ( + self.L_max is not None and self.L_max >= 2 and self.cost_benefit_delta is not None + ) + if self.bootstrap_results is not None and np.isfinite(self.overall_se) and not is_delta: lines.append("Note: p-value and CI are multiplier-bootstrap percentile inference") lines.append( f" ({self.bootstrap_results.n_bootstrap} iterations, " f"{self.bootstrap_results.weight_type} weights)." ) + elif self.bootstrap_results is not None and is_delta: + lines.append( + f"Note: delta SE is delta-method (normal-theory) from per-horizon " + f"bootstrap SEs ({self.bootstrap_results.n_bootstrap} iterations)." + ) elif self.bootstrap_results is not None: lines.append( f"Note: bootstrap ({self.bootstrap_results.n_bootstrap} iterations) " diff --git a/docs/choosing_estimator.rst b/docs/choosing_estimator.rst index 07bddfd0..4594f215 100644 --- a/docs/choosing_estimator.rst +++ b/docs/choosing_estimator.rst @@ -286,13 +286,10 @@ Phase 3 will add covariate adjustment. .. note:: - The Phase 1 placebo SE is intentionally ``NaN`` with a warning. The - dynamic companion paper Section 3.7.3 derives the cohort-recentered - analytical variance for ``DID_l`` only — not for the placebo - ``DID_M^pl``. Phase 2 will add multiplier-bootstrap support for the - placebo. Until then, the placebo point estimate is meaningful but its - inference fields stay NaN-consistent even when ``n_bootstrap > 0`` - (bootstrap currently covers ``DID_M``, ``DID_+``, and ``DID_-`` only). + Placebo SE (both single-lag ``DID_M^pl`` and dynamic ``DID^{pl}_l``) + is intentionally ``NaN``. Placebo point estimates are meaningful for + visual pre-trends inspection; formal placebo inference is deferred. + See ``REGISTRY.md`` for the full contract. .. note:: diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index ce85863b..ecc31980 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -583,7 +583,7 @@ Alternative: Multiplier bootstrap clustered at group via the `n_bootstrap` param - **Note:** The analytical CI is **conservative** under Assumption 8 (independent groups) of the dynamic companion paper, and exact only under iid sampling. This is documented as a deliberate deviation from "default nominal coverage". The bootstrap CI uses the same conservative weighting and is provided for users who want a non-asymptotic alternative. -- **Note:** Phase 1 placebo SE is intentionally `NaN` with a `UserWarning`. The dynamic companion paper Section 3.7.3 derives the cohort-recentered analytical variance for `DID_l` only — not for the placebo `DID_M^pl`. Phase 2 will add multiplier-bootstrap support for the placebo via the dynamic paper's machinery. Until then, the placebo point estimate is meaningful but its inference fields stay NaN-consistent **even when `n_bootstrap > 0`**: the bootstrap path computes SEs for `DID_M`, `DID_+`, and `DID_-`, but `placebo_se`, `placebo_t_stat`, `placebo_p_value`, and `placebo_conf_int` remain `NaN` because the placebo's influence function machinery is deferred to Phase 2. +- **Note:** Placebo SE is intentionally `NaN` for both the single-lag `DID_M^pl` and the dynamic placebos `DID^{pl}_l`. The placebo influence-function derivation is deferred (see the Phase 2 dynamic placebo SE Note above). Placebo point estimates are meaningful for visual pre-trends inspection; inference fields stay NaN-consistent even when `n_bootstrap > 0`. - **Note:** When every variance-eligible group forms its own `(D_{g,1}, F_g, S_g)` cohort (a degenerate small-panel case where the cohort framework has zero degrees of freedom), the cohort-recentered plug-in formula is unidentified: cohort recentering subtracts the cohort mean from each group's `U^G_g`, and for singleton cohorts the centered value is exactly zero, so the centered influence function vector collapses to all zeros. The estimator returns `overall_se = NaN` with a `UserWarning` rather than silently collapsing to `0.0` (which would falsely imply infinite precision). The `DID_M` point estimate remains well-defined. The bootstrap path inherits the same degeneracy on these panels — the multiplier weights act on an all-zero vector, so the bootstrap distribution is also degenerate. **Deviation from R `DIDmultiplegtDYN`:** R returns a non-zero SE on the canonical 4-group worked example via small-sample sandwich machinery that Python does not implement. Both responses are valid for a degenerate case; Python's `NaN`+warning is the safer default. To get a non-degenerate SE, include more groups so cohorts have peers (real-world panels typically have `G >> K`).