Chris and I were discussing short-series behaviour and he pointed me toward the repo. Two findings from pushing on very small T, and a question I'd genuinely value Tom's view on.
processbehavior 0.2.0 from PyPI, pandas 2.3.3, Python 3.11.15. Docs read: quickstart, coffee shop, design-state lineage, API reference. Everything below is reproducible — seed 7 for the synthetic series, default_rng(20260903) with 20,000 replicates for the sampling study, 400,000 for the null distribution in §2, and the CMS PY2024 Shared Savings public use file for the real series. Happy to contribute the scripts.
A related observation about signal reporting at small T touches the "results an analyst would mistake for correct output" language in SECURITY.md, so I am sending that one through the private advisory channel rather than including it here.
1. The design state is constant in T; the estimator is not
The lineage classifies replication structure — cell sizes N_kt, complete versus incomplete grid. Series length is a separate property, and ADS 2 is ADS 2 whether T = 4 or T = 400.
A stable simulated process, mean 5.50, sd 1.20 — true natural process limits span a width of 7.20. Feeding the first T points:
| T |
limit width |
share of true |
ADS |
recommended |
| 3 |
2.78 |
39% |
2 |
X |
| 4 |
3.16 |
44% |
2 |
X |
| 6 |
3.14 |
44% |
2 |
X |
| 8 |
4.37 |
61% |
2 |
X |
| 12 |
4.64 |
64% |
2 |
X |
| 20 |
5.34 |
74% |
2 |
X |
| 30 |
5.73 |
80% |
2 |
X |
| 60 |
6.64 |
92% |
2 |
X |
| 150 |
6.84 |
95% |
2 |
X |
The design state, the recommendation and the analysis menu are constant down the whole column.
The sampling distribution behind that. MR̄/d₂ under a stable normal process with σ fixed at 1.0, 20,000 replicates per length:
| n |
median |
p10 |
p90 |
p90/p10 |
RSD |
| 3 |
0.889 |
0.337 |
1.797 |
5.34× |
59% |
| 4 |
0.925 |
0.429 |
1.689 |
3.94× |
50% |
| 8 |
0.969 |
0.592 |
1.454 |
2.45× |
34% |
| 12 |
0.976 |
0.671 |
1.351 |
2.01× |
27% |
| 25 |
0.989 |
0.769 |
1.245 |
1.62× |
19% |
At four points the estimate spans a near-fourfold range between its 10th and 90th percentile on a process whose variation never moved.
The design report at T = 4:
Min cell size: 1 | K: 0 | T: 4
Structure: Complete structure
Available analyses (ADS 2):
Primary: Histogram, Xbar, S, X *, mR
Methods: Capability, Loss Function, Maximum Information
T: 4 is computed and printed. Nothing downstream consumes it. "Complete structure" is correct on the replication axis, and Capability and Loss Function are offered from four points.
2. Why I was at this end of the range, and what the real series did
My series are Medicare shared-savings settlements. Each unit is an accountable care organisation, each observation is an annual figure, and I get three or four points per unit. They also trend by construction, because medical costs rise and the benchmark years are built to reflect that.
476 ACOs, per-capita expenditure across three benchmark years plus the performance year:
mean level, BY1 -> PY $10,162 -> $10,184 -> $10,886 -> $12,622
year-over-year steps that increase 85.4%
median drift share |mean step| / mean|step| 1.000
MR̄ here is largely estimating the slope of medical cost growth rather than variation around a level. That is a property of the estimator at this length rather than anything the library does wrong — but it means the number the library correctly computes is not the number the analyst thinks they asked for.
And a drift test cannot help, for a reason worth stating. With three differences, an independent series is monotone with probability 2/4! = 1/12 ≈ 8.3%. That is above 5%, so the 95th percentile of the null distribution for drift share is 1.000 — the maximum attainable value. No four-point series can exceed it. Simulation at 400,000 replicates gives 8.3%, matching 1/12. The 95th percentile falls to 0.77 at T = 5 and 0.59 at T = 6, so this is specific to the shortest series rather than general.
So the two failure modes point in opposite directions — narrow limits and false alarms on a short stable series, wide limits and silence on a short trending one — and in both cases the design state reads as healthy.
3. A caveat on my own test
Those four points are not four observations of one process. Benchmark years are constructed inputs to a contractual benchmark, risk-adjusted separately and partially restated. A purist would say I fed the package something it should never have been given — and that objection makes the series worse than short, not better.
It was accepted without comment, and a user without the domain background would not know they had done anything wrong.
4. The question
This is the part I'd most value Tom's view on, and it is not rhetorical.
When the diagnostics that would report non-stationarity are disabled by the same shortness that makes the estimate imprecise, is there a defensible short-series variation estimator — or is the correct answer that four points does not carry a variation measure at all?
I've assumed the latter and gated my own work accordingly, but that is a judgement, not a result. Wheeler's own guidance is that useful limits can come from 5–6 points and firm up by 15–20, so a reasonable person could argue the estimate should be returned with its 3.94× interval attached and the reader left to discount it. My objection is that a number with a 3.94× interval gets quoted without the interval — which is an argument about institutions, not statistics.
Happy to move this to a Discussion if that is the better venue for a methodology question.
5. A suggestion, offered tentatively
Less "add a concept" than "connect two things already present": T is computed and printed in the design report, and SignalConfig.min_observations already exists with a default of 20.
- an adequacy flag alongside the design state, derived from T rather than N_kt
- below some threshold, a warning or
recommended_chart = None with a stated reason, in the style of the existing ads_reason
- optionally a monotone-trend flag, since a trending series changes what MR̄ means independently of length
The reason for putting it inside the design state rather than beside it: the package's distinguishing claim is that structure determines admissibility before anything is computed. Series length is a structural property, and it currently sits outside the gate everything else passes through.
6. What I liked, said plainly
The formulate() / execute() separation is the best decision in the API. Making "what does this structure permit" a first-class inspectable object, before computation, is the right decomposition — and it is precisely why the T question stood out, because the architecture is already shaped to hold that kind of constraint.
The ODS → ADS distinction is subtle and correct. Detecting incompleteness on raw data and then reporting what survives cleansing is careful work that most tools would collapse into one number.
Validating against Bishop's published reference results on every commit is the standard I would most want other analytical libraries to copy. Most demonstrate internal consistency; checking against an external authority is harder and different.
Chris and I were discussing short-series behaviour and he pointed me toward the repo. Two findings from pushing on very small T, and a question I'd genuinely value Tom's view on.
processbehavior 0.2.0from PyPI, pandas 2.3.3, Python 3.11.15. Docs read: quickstart, coffee shop, design-state lineage, API reference. Everything below is reproducible — seed 7 for the synthetic series,default_rng(20260903)with 20,000 replicates for the sampling study, 400,000 for the null distribution in §2, and the CMS PY2024 Shared Savings public use file for the real series. Happy to contribute the scripts.A related observation about signal reporting at small T touches the "results an analyst would mistake for correct output" language in SECURITY.md, so I am sending that one through the private advisory channel rather than including it here.
1. The design state is constant in T; the estimator is not
The lineage classifies replication structure — cell sizes N_kt, complete versus incomplete grid. Series length is a separate property, and ADS 2 is ADS 2 whether T = 4 or T = 400.
A stable simulated process, mean 5.50, sd 1.20 — true natural process limits span a width of 7.20. Feeding the first T points:
The design state, the recommendation and the analysis menu are constant down the whole column.
The sampling distribution behind that. MR̄/d₂ under a stable normal process with σ fixed at 1.0, 20,000 replicates per length:
At four points the estimate spans a near-fourfold range between its 10th and 90th percentile on a process whose variation never moved.
The design report at T = 4:
T: 4is computed and printed. Nothing downstream consumes it. "Complete structure" is correct on the replication axis, and Capability and Loss Function are offered from four points.2. Why I was at this end of the range, and what the real series did
My series are Medicare shared-savings settlements. Each unit is an accountable care organisation, each observation is an annual figure, and I get three or four points per unit. They also trend by construction, because medical costs rise and the benchmark years are built to reflect that.
476 ACOs, per-capita expenditure across three benchmark years plus the performance year:
MR̄ here is largely estimating the slope of medical cost growth rather than variation around a level. That is a property of the estimator at this length rather than anything the library does wrong — but it means the number the library correctly computes is not the number the analyst thinks they asked for.
And a drift test cannot help, for a reason worth stating. With three differences, an independent series is monotone with probability 2/4! = 1/12 ≈ 8.3%. That is above 5%, so the 95th percentile of the null distribution for drift share is 1.000 — the maximum attainable value. No four-point series can exceed it. Simulation at 400,000 replicates gives 8.3%, matching 1/12. The 95th percentile falls to 0.77 at T = 5 and 0.59 at T = 6, so this is specific to the shortest series rather than general.
So the two failure modes point in opposite directions — narrow limits and false alarms on a short stable series, wide limits and silence on a short trending one — and in both cases the design state reads as healthy.
3. A caveat on my own test
Those four points are not four observations of one process. Benchmark years are constructed inputs to a contractual benchmark, risk-adjusted separately and partially restated. A purist would say I fed the package something it should never have been given — and that objection makes the series worse than short, not better.
It was accepted without comment, and a user without the domain background would not know they had done anything wrong.
4. The question
This is the part I'd most value Tom's view on, and it is not rhetorical.
I've assumed the latter and gated my own work accordingly, but that is a judgement, not a result. Wheeler's own guidance is that useful limits can come from 5–6 points and firm up by 15–20, so a reasonable person could argue the estimate should be returned with its 3.94× interval attached and the reader left to discount it. My objection is that a number with a 3.94× interval gets quoted without the interval — which is an argument about institutions, not statistics.
Happy to move this to a Discussion if that is the better venue for a methodology question.
5. A suggestion, offered tentatively
Less "add a concept" than "connect two things already present":
Tis computed and printed in the design report, andSignalConfig.min_observationsalready exists with a default of 20.recommended_chart = Nonewith a stated reason, in the style of the existingads_reasonThe reason for putting it inside the design state rather than beside it: the package's distinguishing claim is that structure determines admissibility before anything is computed. Series length is a structural property, and it currently sits outside the gate everything else passes through.
6. What I liked, said plainly
The
formulate()/execute()separation is the best decision in the API. Making "what does this structure permit" a first-class inspectable object, before computation, is the right decomposition — and it is precisely why the T question stood out, because the architecture is already shaped to hold that kind of constraint.The ODS → ADS distinction is subtle and correct. Detecting incompleteness on raw data and then reporting what survives cleansing is careful work that most tools would collapse into one number.
Validating against Bishop's published reference results on every commit is the standard I would most want other analytical libraries to copy. Most demonstrate internal consistency; checking against an external authority is harder and different.