Add built-in "probability-of-effect" and lift summaries for ITS (and other designs)

[drbenvincent]

Problem

CausalPy returns posterior draws but lacks a standardized, decision-ready summary like CausalImpact’s report. Users must hand-compute tail probabilities, credible intervals, and relative/cumulative effects.

Proposal

Ship a small reporting layer that converts posterior draws into:

• Average effect over a window (post-period by default)
• Cumulative effect over a window
• Relative effect (%) vs the counterfactual
• Posterior tail-area metrics:
• one-sided $P(\text{effect}>0)$ or $P(\text{effect}<0)$
• two-sided tail area $p$ and “probability of a causal effect” $1-p$

Provide a tabular summary and a short prose report. Integrate with ITS first, then reuse for Synthetic Control, DiD, and GeoLift.

Scope

• Works out of the box for ITS where result.post_impact (or equivalent) exposes posterior draws of observed, counterfactual, and effect.
• Defaults to the full post-period window; allow custom windows.

API

import causalpy as cp

res = cp.InterruptedTimeSeries(...).fit()

# Stats object with .table (DataFrame) and .text (string)
stats = cp.reporting.effect_summary(
    res,
    window="post",                # "post" | ("2024-09-01","2024-10-31") | slice/int index
    direction="increase",         # "increase" | "decrease" | "two-sided"
    alpha=0.05,                   # HDI level
    cumulative=True,              # also report cumulative
    relative=True,                # also report relative %
    min_effect=None               # optional ROPE / minimal effect size
)

print(stats.table)
print(stats.text)

Computed quantities (from posterior draws)

Let $t \in W$ be the chosen window and $s=1,\dots,S$ posterior draws.

• Pointwise effects: $e_t^{(s)} = y_t - \mu_t^{\text{cf},(s)}$

• Average effect draws: $E^{(s)}{\text{avg}} = \frac{1}{|W|}\sum{t\in W} e_t^{(s)}$

• Cumulative effect draws: $E^{(s)}{\text{cum}} = \sum{t\in W} e_t^{(s)}$

• Relative effect draws: $R^{(s)}{\text{avg}} = \frac{1}{|W|}\sum{t\in W} \frac{e_t^{(s)}}{\mu_t^{\text{cf},(s)}+\epsilon}$ and similarly for cumulative

• Credible intervals: HDIs via ArviZ az.hdi at 1-alpha

• Tail metrics:

  • direction="increase": P(E>0) for both avg and cum
  • direction="decrease": P(E<0)
  • direction="two-sided": $p = 2 \min{P(E&gt;0), P(E&lt;0)}$, report “probability of effect” $1-p$

Default table schema

Columns for each of {average, cumulative}:

• mean, median, hdi_lower, hdi_upper
• relative_mean, relative_hdi_lower, relative_hdi_upper (if relative=True)
• p_gt_0, p_lt_0, p_two_sided (as applicable)

Prose report (templated)

Post-period (2024-09-01 to 2024-10-31), the average effect was +X (95% HDI [L, U]),
with a posterior probability of an increase of P_inc. The cumulative effect was +Y
(95% HDI [L, U]); probability of an increase P_inc_cum. Relative to the counterfactual,
this equals +Z% on average (95% HDI [L%, U%]).

Plot integration (optional)

Option to annotate the existing three-panel ITS plot with the average and cumulative effect plus $P(\text{effect}&gt;0)$

Edge cases

• Guard against division by zero in relative effects with a small $\epsilon$.
• For "stock" metrics where cumulative is not meaningful, allow cumulative=False.
• If users pass a min_effect (ROPE), report $P(|E|&gt;\text{min effect})$.

Acceptance criteria

• One line produces a DataFrame and a short narrative for ITS.
• Tail metrics match simple Monte Carlo calculations from posterior draws.
• Works with both date-indexed and integer-indexed time axes.
• Unit tests covering sign conventions, two-sided p, and HDI coverage on simulated data.

Why this helps

Gives a CausalImpact-style, decision-ready summary without custom post-processing. Teams get clear averages, cumulatives, relative lifts, and a “probability of effect” to support go/no-go and ROI decisions.