feat(docker): bake litellm extra into the docker image - #142
Merged
Conversation
The litellm judge dispatch route needs litellm available in-container since the checker runs there too under DockerRunner isolation, not just on the host. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
akshaylive
requested review from
CarlesUIPath,
bai-uipath,
tmatup and
uipreliga
as code owners
August 27, 2026 20:15
akshaylive
added a commit
to UiPath/skills
that referenced
this pull request
Aug 27, 2026
0.11.4 bakes --extra litellm into the coder-eval-agent base image (UiPath/coder_eval#142), which tests/experiments/*.yaml's litellm-routed checker_context needs available in-container under driver: docker. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
akshaylive
added a commit
to UiPath/skills
that referenced
this pull request
Aug 28, 2026
0.11.4 bakes --extra litellm into the coder-eval-agent base image (UiPath/coder_eval#142), which tests/experiments/*.yaml's litellm-routed checker_context needs available in-container under driver: docker. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
akshaylive
added a commit
to UiPath/skills
that referenced
this pull request
Aug 28, 2026
…-eval 0.11.5 (#2864) * feat(tests): route checker LLM through litellm gpt-5.6-luna via coder-eval 0.11.3 Bump coder-eval to 0.11.3 and set checker_context.api_route defaults (default/smoke/smoke-windows/nightly/activation experiments) so the success-checker/judge calls litellm -> gpt-5.6-luna instead of the built-in default, resolving credentials from CODEX_BASE_URL/CODEX_API_KEY. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * chore(tests): bump coder-eval pin to 0.11.4 0.11.4 bakes --extra litellm into the coder-eval-agent base image (UiPath/coder_eval#142), which tests/experiments/*.yaml's litellm-routed checker_context needs available in-container under driver: docker. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(tests): scope checker_context.api_route to per-task, not experiment defaults Addresses PR #2864 review. coder-eval rejects route: litellm outright on any task with simulation.enabled or an enabled agent_judge criterion (the simulator/agent_judge run as real Claude Code subprocesses, incompatible with an arbitrary litellm-fronted gateway) -- see orchestrator._reject_litellm_eval_route_if_unsupported. 303 of 359 llm_judge tasks in this repo have simulation.enabled, so the previous experiment-level default broke all of them at setup, including one smoke-tagged task (uipath-maestro-flow/interactive/solution_select.yaml). - Remove checker_context from experiments/{default,smoke,smoke-windows, nightly,activation}.yaml. activation.yaml has zero llm_judge criteria, so it gets no replacement. - Add checker_context.api_route directly to the 56 task YAMLs that have an llm_judge criterion and no simulation.enabled -- the only tasks it's safe and meaningful for. This also fixes same-ground-headtohead.yaml and flow-v2-preview.yaml, which never got the experiment-level block: a task-level setting applies under any experiment that runs the task. - Fix model: azure/gpt-5.6-luna, not the bare id -- litellm resolves an unprefixed model to provider `openai`, which would silently ignore the Azure-shaped `params.api_version`. Matches coder_eval's own TASK_DEFINITION_GUIDE.md example verbatim. - tests/Makefile: install the litellm extra for local `make install`, and point at the new README section for CODEX_BASE_URL/CODEX_API_KEY. - tests/README.md: document checker_context.api_route, the litellm/ simulation incompatibility, and what each driver needs (host extra for tempdir, baked into the agent image for docker -- coder-eval-agent bags it as of 0.11.4). - smoke-skills.yml: note the one deliberate exception to "the grader stays on Claude" now that a few tasks route their own llm_judge through the gate's CODEX_* endpoint. Windows RPA smoke (smoke-rpa-skills.yml) and skill-activation-gate.yml need no change: no task either runs carries checker_context now that it's task-scoped. run-coder-eval.yml's docker/Windows jobs already installed the litellm extra and passed CODEX_BASE_URL/CODEX_API_KEY through. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * revert(tests): restore experiment-level checker_context.api_route default Reverts e36dce6. The per-task scoping worked around coder_eval rejecting route: litellm on any task with simulation.enabled or an enabled agent_judge criterion -- but the real fix belongs upstream. coder_eval now resolves the simulator's route independently of checker_context.api_route entirely (Orchestrator.simulator_route), so route: litellm is safe as an experiment default even with simulation.enabled: true. - Restore checker_context.api_route to defaults/{default,smoke, smoke-windows,nightly,activation}.yaml, keeping the azure/gpt-5.6-luna model fix (bare gpt-5.6-luna resolves to provider openai, which silently ignores the Azure-shaped params.api_version). - Remove the 56 per-task checker_context blocks -- redundant now that the experiment default covers every llm_judge task safely. - tests/Makefile: keep the [litellm] extra for local `make install` (still needed for driver: tempdir now that this is the default, not a per-task exception). - tests/README.md: rewrite Checker Context to describe the new simulator-decoupled default instead of "set this per-task, never as an experiment default." Still calls out that agent_judge + route: litellm remains unsupported (no agent_judge criteria in this repo today). - smoke-skills.yml: drop the per-task-exception comment on AGENT_MODEL, no longer accurate now that ~all llm_judge tasks route through litellm by default, not "a handful." NOT bumping tests/.coder-eval-version yet -- the coder_eval fix (simulator_route decoupling) is committed on akshaya/scope-litellm-route-to-llm-judge but not released. Merging this against the current 0.11.4 pin would reintroduce the original CI failure (route: litellm rejected outright on simulation-enabled tasks). Bump the pin once coder_eval ships a release containing that fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
litellmextra into the docker image alongsidecodexandantigravitylitellmbacks thechecker_context.api_route.route: litellmjudge dispatch — needed in-container since the checker runs there too underDockerRunnerisolation, not just on the hostuv.lock, so no private-index credentials are neededTest plan
make testpasses (4367 passed, 2 skipped) before and after rebase onto latestmain