LWDiD: default not-yet-treated staggered path averages unsupported calendar periods

[shawcharles]

Summary

The default control_group=\"not_yet_treated\" staggered path can report a large, highly significant ATT when the true treatment effect is zero. It applies eligibility as a unit-level filter and then averages transformed outcomes over unequal calendar-time windows, rather than aggregating supported cohort-time cells.

Reproduction

import warnings

import pandas as pd

from diff_diff.lwdid import LWDiD

rows = []
for unit in range(10):
    cohort = 3 if unit < 5 else 5
    for time in range(1, 7):
        rows.append(
            {
                "unit": unit,
                "time": time,
                "cohort": cohort,
                "treat": int(time >= cohort),
                "y": float(time),
            }
        )

data = pd.DataFrame(rows)

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    result = LWDiD(
        rolling="demean",
        estimator="ra",
        vce="classical",
        control_group="not_yet_treated",
    ).fit(
        data,
        outcome="y",
        unit="unit",
        time="time",
        treatment="treat",
        cohort="cohort",
    )

print(result.att)
print(result.cohort_effects)
print([str(w.message) for w in caught])

On the current PR head this returns an ATT of approximately 1.0, with a standard error near 5e-16 and a p-value near 2e-149, while warning that cohort 5 has no valid controls.

The outcome is a common linear time trend, so the valid comparisons are ATT(3, 3) = 0 and ATT(3, 4) = 0. For cohort 3 at times 5 and 6, no untreated/not-yet-treated controls remain. Those unsupported cells should not be folded into a cohort-level comparison against a later cohort measured only through time 4.

Cause

The staggered branch retains all post-treatment observations for a treated cohort, but stops the later cohort when it becomes treated. It then averages _ydot by unit, yielding unequal time windows:

diff-diff/diff_diff/lwdid.py

Lines 892 to 906 in c2694ff

	# - not_yet_treated (cohort_i > g): keep only t < cohort_i
	if self.control_group == "not_yet_treated":
	cohort_g_set = set(cohort_g_units)
	post_mask_g = sub_df[time].isin(post_periods_g) & ( # type: ignore[union-attr, call-overload]
	sub_df[unit].isin(cohort_g_set) # type: ignore[union-attr, call-overload]
	| (sub_df[cohort] == 0) # type: ignore[call-overload]
	| sub_df[cohort].isna() # type: ignore[union-attr, call-overload]
	| (sub_df[time] < sub_df[cohort]) # type: ignore[operator, call-overload]
	)
	else:
	post_mask_g = sub_df[time].isin(post_periods_g) # type: ignore[union-attr, call-overload]
	post_sub = sub_df.loc[post_mask_g] # type: ignore[union-attr]
	unit_post_avg_g = post_sub.groupby(unit)["_ydot"].mean().reset_index()

This conflicts with the documented time-specific control eligibility and with the Lee-Wooldridge staggered design, which conditions controls on each cohort-calendar-time pair.

Expected behaviour

Construct and aggregate supported (g, t) cells with their time-specific eligible controls. At a minimum, unsupported post-treatment cells must be skipped or rejected explicitly; they must not create unequal-window unit averages.

Acceptance criteria

• The reproduction above returns no non-zero ATT from the common time trend.
• Adding any common time-only shift h(t) to every unit's outcome leaves the ATT unchanged.
• Unsupported (g, t) cells are reported, skipped, or rejected explicitly.
• A regression test covers a later cohort exhausting the not-yet-treated control pool.
• A degenerate-SE guard prevents near-zero standard errors from being reported as ordinary finite-sample inference.
• Aggregation preserves the relevant cell support and inference structure.

Reference: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4516518