Skip to content

pull-request-dashboard: replay corpus cannot validate routes end to end #192

Description

@trask

The 441 pull request replay corpus used to validate dashboard classification changes was collected without commits, so it cannot be used to compare whole-PR routes before and after a change.

normalize_events reads raw["commits"], and the harnesses work around this by passing {"commits": [], "review_comments": []}. That is fine for deriving discussion items and review threads, but it means compute_facts never sees pushes or CI, so route_pr cannot be evaluated. Any classification change can therefore only be measured at the discussion layer.

Concretely, the review-thread rewrite could be measured as "134 of 144 pull requests unchanged in whether any thread demands the author", but not as "N of 441 routes unchanged", which is the number that actually matters.

To close this:

  • re-collect the corpus with commits and whatever else compute_facts reads (status checks, mergeable state)
  • add a replay that computes route_pr per pull request and diffs against the route recorded in the otelbot/pull-request-dashboard-state/<repo> branches
  • keep the collector in the repo rather than as local scratch, so the corpus can be rebuilt

Until then, route-level claims about classification changes are not supported by evidence.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions