🤖 ci: replace colons in TB artifact names with hyphens #488
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Artifact names cannot contain colons due to filesystem restrictions. Convert model names like 'anthropic:claude-sonnet-4-5' to 'anthropic-claude-sonnet-4-5' in artifact names. This fixes the nightly benchmark workflow failures where artifacts were generated successfully but failed to upload.
Add console.error logging to confirm which model is being used when running terminal-bench. This helps verify that model_name is correctly passed through the workflow -> Makefile -> agent -> CLI chain. Logging to stderr to avoid interfering with stdout-based result parsing.
Add a step to output the full results.json and per-task summary before uploading artifacts. This ensures we can see task-level results even if artifact upload fails. Output includes: - Full results.json (with jq formatting if available) - Per-task summary: task_id: ✓ PASS / ✗ FAIL This will help verify that different models are producing different results, not just different timeout patterns.
ammario
changed the title
github-merge-queue bot
pushed a commit
that referenced
this pull request
Oct 29, 2025
## Problem PR #488 introduced a syntax error by using the `replace()` function in a GitHub Actions workflow expression: ```yaml name: terminal-bench-results-${{ inputs.model_name && replace(format('{0}-{1}', inputs.model_name, github.run_id), ':', '-') || format('{0}', github.run_id) }} ``` **GitHub Actions does not support `replace()` as a function in workflow expressions.** This caused the nightly workflow to fail with a syntax error. ## Solution Use a shell script step to compute the artifact name using the `tr` command: ```yaml - name: Set artifact name id: artifact-name run: | if [ -n "${{ inputs.model_name }}" ]; then ARTIFACT_NAME="terminal-bench-results-$(echo "${{ inputs.model_name }}-${{ github.run_id }}" | tr ':' '-')" else ARTIFACT_NAME="terminal-bench-results-${{ github.run_id }}" fi echo "name=$ARTIFACT_NAME" >> $GITHUB_OUTPUT - name: Upload benchmark results uses: actions/upload-artifact@v4 with: name: ${{ steps.artifact-name.outputs.name }} ``` This uses standard shell string manipulation which is fully supported. ## Testing The workflow syntax now validates correctly. The next nightly run should: - ✅ Parse successfully without syntax errors - ✅ Replace colons with hyphens: `anthropic-claude-sonnet-4-5` - ✅ Upload artifacts with valid names --- _Generated with `cmux`_
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
The nightly Terminal-Bench workflow runs successfully but fails at the artifact upload step because artifact names contain colons (from model names like
anthropic:claude-sonnet-4-5).GitHub Actions artifact names cannot contain colons due to filesystem restrictions (NTFS compatibility).
Solution
1. Fixed Artifact Names
Use the
replace()function in the artifact name template to convert colons to hyphens:anthropic:claude-sonnet-4-5→anthropic-claude-sonnet-4-5openai:gpt-5-codex→openai-gpt-5-codex2. Added Results Logging
Important: Since the previous run had 720 files that failed to upload, added a new step to print
results.jsonto workflow logs before artifact upload:This ensures task-level results are preserved in logs even if artifact upload fails.
3. Added Model Verification Logging
Added logging in
agentSessionCli.tsto confirm model names:Investigation: Identical 42.50% Accuracy
During the manual run (#18913267878), both models achieved exactly 42.50% accuracy (34/80 tasks). Investigation revealed:
Facts:
runs/2025-10-29__15-29-47vs15-29-29)Code Path Verification:
Traced model parameter through the entire chain:
TB_ARGS: --agent-kwarg model_name=<model>$TB_ARGSto terminal-benchmodel_name, setsCMUX_MODELenv var--model "${CMUX_MODEL}"to CLI--modelflag and uses itThe code is correct. The identical scores are statistically unlikely but possible with offsetting timeout patterns.
Next Steps:
With the new results logging, the next benchmark run will show:
This allows full verification that models produce different task-level results.
Testing
The next nightly run (tonight at 00:00 UTC) will:
terminal-bench-results-anthropic-claude-sonnet-4-5-<run_id>terminal-bench-results-openai-gpt-5-codex-<run_id>[cmux-cli] Using model: <model_name>Generated with
cmux