fix(eval): sanitize jobbench DTU names for Incus - #136
Merged
David Koleczek (DavidKoleczek) merged 1 commit intoAug 19, 2026
Conversation
Incus instance names accept only alphanumerics and hyphens. JobBench occupation slugs come from the dataset directory names and are underscore-separated (`civil_engineers`, `financial_analysts`), so `_dtu_name` was emitting names Incus rejects outright -- the container never launches and the trial fails before the agent runs. Most occupations in the dataset contain an underscore, so this affects the majority of the suite. Transliterate any non-alphanumeric character to a hyphen for both the occupation slug and the agent name before assembling `jb-<agent>-<occupation>-t<N>-<uuid6>`. The agent name is sanitized too because it is caller-supplied and can carry dots or slashes. Scope: eval harness only. No engine, CLI, HTTP, protocol, or wrapper surface is touched. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Incus instance names accept only alphanumerics and hyphens. JobBench occupation slugs come from the dataset directory names and are underscore-separated (
civil_engineers,financial_analysts), so_dtu_namewas emitting names Incus rejects outright. The container never launches and the trial fails before the agent runs.Most occupations in the dataset contain an underscore, so this affects the majority of the suite.
Fix
Transliterate any non-alphanumeric character to a hyphen for both the occupation slug and the agent name before assembling
jb-<agent>-<occupation>-t<N>-<uuid6>. The agent name is sanitized too because it is caller-supplied and can carry dots or slashes.The 60-character budget and the uniqueness properties of the existing scheme are unchanged: sanitization is length-preserving and happens after the slice.
Scope of impact
Eval harness only. No engine, CLI, HTTP, protocol, or wrapper surface is touched, so no spec update, e2e case, or CHANGELOG entry applies.
Verification
ruff checkandruff format --checkclean on the changed file.