The 441 pull request replay corpus used to validate dashboard classification changes was collected without commits, so it cannot be used to compare whole-PR routes before and after a change.
normalize_events reads raw["commits"], and the harnesses work around this by passing {"commits": [], "review_comments": []}. That is fine for deriving discussion items and review threads, but it means compute_facts never sees pushes or CI, so route_pr cannot be evaluated. Any classification change can therefore only be measured at the discussion layer.
Concretely, the review-thread rewrite could be measured as "134 of 144 pull requests unchanged in whether any thread demands the author", but not as "N of 441 routes unchanged", which is the number that actually matters.
To close this:
- re-collect the corpus with
commits and whatever else compute_facts reads (status checks, mergeable state)
- add a replay that computes
route_pr per pull request and diffs against the route recorded in the otelbot/pull-request-dashboard-state/<repo> branches
- keep the collector in the repo rather than as local scratch, so the corpus can be rebuilt
Until then, route-level claims about classification changes are not supported by evidence.
The 441 pull request replay corpus used to validate dashboard classification changes was collected without
commits, so it cannot be used to compare whole-PR routes before and after a change.normalize_eventsreadsraw["commits"], and the harnesses work around this by passing{"commits": [], "review_comments": []}. That is fine for deriving discussion items and review threads, but it meanscompute_factsnever sees pushes or CI, soroute_prcannot be evaluated. Any classification change can therefore only be measured at the discussion layer.Concretely, the review-thread rewrite could be measured as "134 of 144 pull requests unchanged in whether any thread demands the author", but not as "N of 441 routes unchanged", which is the number that actually matters.
To close this:
commitsand whatever elsecompute_factsreads (status checks, mergeable state)route_prper pull request and diffs against therouterecorded in theotelbot/pull-request-dashboard-state/<repo>branchesUntil then, route-level claims about classification changes are not supported by evidence.