Skip to content

feat(cli): report distinct exit codes for benchmark, auth and upload failures - #527

Open
akx wants to merge 2 commits into
CodSpeedHQ:mainfrom
akx:exeunt-codes
Open

feat(cli): report distinct exit codes for benchmark, auth and upload failures#527
akx wants to merge 2 commits into
CodSpeedHQ:mainfrom
akx:exeunt-codes

Conversation

@akx

@akx akx commented Sep 3, 2026

Copy link
Copy Markdown

In a downstream project using codspeed-action, we were wondering if it would be possible to make the action pass when only uploading/auth failed (e.g. since a fork doesn't have CodSpeed enabled).

I looked at codspeed-action, and there wasn't an obvious way to do that (aside from capturing the runner output and grepping it for a human-readable error string, yeugh), so here's a PR that adds a handful of exit codes to the runner.

If this gets merged, the follow-up in the action is to add an input for ignoring error codes 4 and 5.

@greptile-apps

greptile-apps Bot commented Sep 3, 2026

Copy link
Copy Markdown

Greptile Summary

This PR introduces typed CLI failure markers and maps benchmark, authentication, and upload errors to distinct process exit codes for downstream automation.

  • Adds the public exit-code definitions, help text, error wrappers, and top-level process mapping.
  • Classifies selected benchmark process failures across memory, wall-time, and Valgrind execution.
  • Classifies selected authentication and upload failures while preserving nested authentication markers.

Confidence Score: 3/5

The PR should not merge until authentication coverage and tracker/profiler status classification match the documented exit-code meanings.

Reachable authentication failures still return the generic status, while independent memory-tracker and wall-time-profiler failures are incorrectly reported as failures of the benchmark command itself.

Files Needing Attention: src/executor/orchestrator.rs, src/executor/memory/executor.rs, src/executor/wall_time/executor.rs

Important Files Changed

Filename Overview
src/exit_code.rs Defines the new marked-error mechanism, public exit-code contract, help text, and chain-preserving classification.
src/executor/orchestrator.rs Adds upload and token-refresh classification, but earlier provider authentication failures remain generic.
src/executor/memory/executor.rs Classifies the memtrack process status as a benchmark failure even when the tracker itself fails.
src/executor/wall_time/executor.rs Classifies profiler process failures as benchmark failures because the workload status is not captured separately.
src/executor/valgrind/measure.rs Correctly distinguishes the Valgrind tool status from the benchmark command's separately recorded status.
src/upload/uploader.rs Marks unauthorized upload-target responses as authentication failures while leaving other upload errors for the enclosing upload marker.
src/main.rs Converts classified anyhow error chains into the corresponding process exit status after logger cleanup.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    CLI[run or exec] --> Init[Initialize environment provider]
    Init --> Execute[Execute benchmark]
    Execute --> Upload[Upload completed results]
    Init -->|auth error| Generic[Generic exit 1]
    Execute -->|outer process nonzero| Benchmark[Benchmark exit 3]
    Upload -->|auth error| Auth[Auth exit 4]
    Upload -->|other error| UploadCode[Upload exit 5]
Loading

Fix all with Greploop Fix All in Claude Code Fix All in Codex

Prompt To Fix All With AI
### Issue 1
src/executor/orchestrator.rs:274-277
**Authentication classification starts too late**

When provider initialization rejects missing CI credentials or local repository fallback encounters an expired token, the failure occurs before this marked token-refresh call and exits with generic code 1 instead of the documented authentication code 4. Downstream automation therefore cannot reliably distinguish and optionally ignore these authentication failures.

### Issue 2
src/executor/memory/executor.rs:191-195
**Tracker failures mimic benchmark failures**

When memtrack fails to load or finish tracking, its ring buffer overflows, or a wall-time profiler fails independently, these checks pass the outer tool process status to `benchmark_failed`. The CLI consequently returns code 3 even when the benchmark succeeded, causing downstream automation to apply benchmark-failure policy to a measurement-tool failure.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (1): Last reviewed commit: "feat(cli): report distinct exit codes fo..." | Re-trigger Greptile

Comment thread src/executor/orchestrator.rs
Comment thread src/executor/memory/executor.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant