feat(ci): auto-bump recall floors after stable releases - #2162
Conversation
Adds a workflow that downloads the pre-publish benchmark artifact after a successful stable Publish run and raises any per-language precision/recall floor that improved, opening a PR. Floors are a one-way ratchet — they are never lowered. Impact: 1 functions changed, 1 affected
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if: > | ||
| github.event.workflow_run.conclusion == 'success' && | ||
| github.event.workflow_run.event != 'push' |
There was a problem hiding this comment.
When a successful scheduled Publish run completes, this condition accepts it even though scheduled runs do not produce benchmark-results-json, causing the follow-up workflow to fail at gh run download.
| if: > | |
| github.event.workflow_run.conclusion == 'success' && | |
| github.event.workflow_run.event != 'push' | |
| if: > | |
| github.event.workflow_run.conclusion == 'success' && | |
| (github.event.workflow_run.event == 'release' || | |
| github.event.workflow_run.event == 'workflow_dispatch') |
| git fetch --tags | ||
| VERSION=$(git describe --tags --match "v*" --abbrev=0 2>/dev/null | sed 's/^v//' || echo "unknown") | ||
| BRANCH="chore/recall-floors-v${VERSION}" |
There was a problem hiding this comment.
When an older workflow_dispatch retry completes after a newer release tag is reachable from main, the workflow downloads the older run's artifact but derives the newer version with git describe, causing the older metrics to be labeled and force-pushed onto the newer release's branch.
| const pattern = new RegExp( | ||
| `( ${lang}: \\{ precision: )([\\d.]+)(, recall: )([\\d.]+)( \\})`, | ||
| ); |
There was a problem hiding this comment.
Interpolating lang directly into the regular expression makes fixture names containing regex metacharacters, such as a future c++ key, fail to match their threshold row or match unintended text; the script then silently leaves that language's improved floor unchanged.
Greptile SummaryAdds automated recall-floor ratcheting after Publish runs.
Confidence Score: 2/5This PR should not merge until scheduled Publish runs are excluded and floor-update branches are bound to the version that produced each benchmark artifact. The workflow fails after successful scheduled runs and can apply an older run's benchmark data to a newer release branch when releases complete out of order; the updater also does not safely match future language keys containing regex metacharacters. Files Needing Attention: .github/workflows/update-recall-floors.yml, scripts/update-recall-floors.mjs Important Files Changed
Sequence DiagramsequenceDiagram
participant Publish
participant Workflow as Update Recall Floors
participant Script as Floor updater
participant GitHub
Publish->>Workflow: workflow_run completed
Workflow->>GitHub: Download benchmark artifact
Workflow->>Script: Apply improved metrics
Script-->>Workflow: Modified threshold table
Workflow->>GitHub: Push branch and open PR
Reviews (1): Last reviewed commit: "feat(ci): auto-bump recall floors after ..." | Re-trigger Greptile |
Codegraph Impact Analysis1 functions changed → 1 callers affected across 1 files
|
1 participant
Summary
Test plan
scripts/update-recall-floors.mjsdetects an improved floor, and is skipped otherwise