feat(cli): report distinct exit codes for benchmark, auth and upload failures - #527
Open
akx wants to merge 2 commits into
Open
feat(cli): report distinct exit codes for benchmark, auth and upload failures#527akx wants to merge 2 commits into
akx wants to merge 2 commits into
Conversation
Greptile SummaryThis PR introduces typed CLI failure markers and maps benchmark, authentication, and upload errors to distinct process exit codes for downstream automation.
Confidence Score: 3/5The PR should not merge until authentication coverage and tracker/profiler status classification match the documented exit-code meanings. Reachable authentication failures still return the generic status, while independent memory-tracker and wall-time-profiler failures are incorrectly reported as failures of the benchmark command itself. Files Needing Attention: src/executor/orchestrator.rs, src/executor/memory/executor.rs, src/executor/wall_time/executor.rs
|
| Filename | Overview |
|---|---|
| src/exit_code.rs | Defines the new marked-error mechanism, public exit-code contract, help text, and chain-preserving classification. |
| src/executor/orchestrator.rs | Adds upload and token-refresh classification, but earlier provider authentication failures remain generic. |
| src/executor/memory/executor.rs | Classifies the memtrack process status as a benchmark failure even when the tracker itself fails. |
| src/executor/wall_time/executor.rs | Classifies profiler process failures as benchmark failures because the workload status is not captured separately. |
| src/executor/valgrind/measure.rs | Correctly distinguishes the Valgrind tool status from the benchmark command's separately recorded status. |
| src/upload/uploader.rs | Marks unauthorized upload-target responses as authentication failures while leaving other upload errors for the enclosing upload marker. |
| src/main.rs | Converts classified anyhow error chains into the corresponding process exit status after logger cleanup. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
CLI[run or exec] --> Init[Initialize environment provider]
Init --> Execute[Execute benchmark]
Execute --> Upload[Upload completed results]
Init -->|auth error| Generic[Generic exit 1]
Execute -->|outer process nonzero| Benchmark[Benchmark exit 3]
Upload -->|auth error| Auth[Auth exit 4]
Upload -->|other error| UploadCode[Upload exit 5]
Prompt To Fix All With AI
### Issue 1
src/executor/orchestrator.rs:274-277
**Authentication classification starts too late**
When provider initialization rejects missing CI credentials or local repository fallback encounters an expired token, the failure occurs before this marked token-refresh call and exits with generic code 1 instead of the documented authentication code 4. Downstream automation therefore cannot reliably distinguish and optionally ignore these authentication failures.
### Issue 2
src/executor/memory/executor.rs:191-195
**Tracker failures mimic benchmark failures**
When memtrack fails to load or finish tracking, its ring buffer overflows, or a wall-time profiler fails independently, these checks pass the outer tool process status to `benchmark_failed`. The CLI consequently returns code 3 even when the benchmark succeeded, causing downstream automation to apply benchmark-failure policy to a measurement-tool failure.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "feat(cli): report distinct exit codes fo..." | Re-trigger Greptile
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In a downstream project using codspeed-action, we were wondering if it would be possible to make the action pass when only uploading/auth failed (e.g. since a fork doesn't have CodSpeed enabled).
I looked at codspeed-action, and there wasn't an obvious way to do that (aside from capturing the runner output and grepping it for a human-readable error string, yeugh), so here's a PR that adds a handful of exit codes to the runner.
If this gets merged, the follow-up in the action is to add an
inputfor ignoring error codes 4 and 5.