devops(test-results-db): CLI to compact CI test results into DuckDB#41673
Conversation
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
| continue; | ||
| const id = String(artifact.id); | ||
| if (ingested.has(id)) { | ||
| if (++seen >= stopAfterSeen) |
There was a problem hiding this comment.
I don't quite understand this logic, but I guess we can give it a shot.
There was a problem hiding this comment.
I expect update to be run locally from machines with a not-so-great uplink to the Github API; and I want to make the minimum amount of API requests we need to ensure correctness. Fetching the entire last 7 days took 40s on my machine, which is too long. I'm using three characteristics of Github Artifacts to cut down work:
- the artifacts list API we're using here orders descending by artifact ID
- artifact IDs are auto-increment, and they're assigned upon upload time (not workflow start time)
- uploading an artifact is fast, so the time between the artifact ID being assigned, and it showing up in the artifacts list API, is very small
The fetching logic here combines this into a watermark like approach: Once we've seen 100 consecutive artifacts we already ingested, we assume all following pages are also fully ingested. On my machine, this trimmed down this phase from 40s to 3s.
Test results for "MCP"2 failed 7707 passed, 1235 skipped Merge workflow run. |
Test results for "tests 1"49510 passed, 1164 skipped Merge workflow run. |
Builds on #41669 (per-job parquet report artifacts). Adds a small CLI in
utils/test-results-db/plus a scheduled workflow that compacts thoseper-run parquet reports into a single queryable DuckDB file.
download— fetch the latest maintained DuckDB artifact (starts from an empty db on the first run).update— ingest parquet report artifacts from the last N days that aren't already stored, deduped byartifact_id, downloading + ingesting in parallel (tunable--concurrency).truncate— drop the oldest runs so the db keeps at most N runs. At ~500 test-workflow runs/week, the default cap of 2000 covers roughly the last 4 weeks.test-results-dbartifact. Aplaywright-test-resultsskill documents how to query it.Trial-run end-to-end on CI: 215 artifacts → 345k rows in ~50s.
Example
updaterun: