Provision Toolathlon service runtime sidecars#889
Merged
Conversation
…ifier scoring Follow-up to #885. Makes the materialized Toolathlon tasks actually runnable. 1. Token spawn-time resolution. The ${token.X}->${TOOLATHLON_X} env rewrite could never resolve per-task or runtime-computed tokens (Drive folder ids written by preprocess, groundtruth allowlists, per-student Canvas tokens): MCP env/args pass through verbatim and args cannot be resolved from env at all. New _toolathlon_container.py ships into each container: `write-config` bakes the global token_key_session.py from injected TOOLATHLON_* secrets; `launch` wraps every MCP server and resolves ${token.X}/${config}/paths across argv AND env at spawn, mirroring upstream utils/mcp/tool_servers.py (global merged with per-task override). Dep-free via an addict shim; absolute interpreter + PATH fallback so FastMCP can spawn it. 2. Pre-create declared server dirs. arxiv_local (--storage-path) and the emails server create working dirs lazily; an agent that reaches the goal another way leaves them absent and evaluators that os.listdir them crash. The launcher mkdir -p's an explicit allow-list of directory-valued flags. 3. Exit-code verifier scoring. Upstream evaluators score by exit code and several signal task-failure by raising (e.g. raise ValueError("Some tests FAILED")) rather than returning False. Treat any non-zero evaluator exit as reward 0 (healthy fail); only escalate to a verifier error on signatures that cannot be an agent failure (ModuleNotFoundError/ImportError/PermissionError). 4. Tolerate task_config entries naming local harness tools (e.g. web_search) with no MCP yaml instead of failing the whole adaptation. Validated on Daytona+OpenHands+Azure gpt-5.5: 37/58 headless tasks healthy on first pass; find-alita-paper and google-cloud/maps/github/hf probes all reward 1.0. 9 adapter tests, ruff + ty clean.
… access Upstream preprocess/eval read all_token_key_session via attributes (all_token_key_session.github_token) because upstream builds it with addict.Dict. The generated global token_key_session.py used a plain dict, so every github/HF/woocommerce/notion/snowflake task whose preprocess reads a token that way crashed with AttributeError. Emit a self-contained attribute-accessible dict subclass.
…ass launch_time Two upstream-parity fixes to the preprocess/verifier invocation: - Run as tasks.finalpool.<task>.preprocess.main (and .evaluation.main) from /workspace, exactly as upstream (uv run -m <dotted path>), so 'from ..utils' relative imports resolve. Running it as a bare 'preprocess.main' broke them. - Pass --launch_time (a single value shared by preprocess and the verifier via /workspace/.toolathlon/launch_time.txt), matching upstream, but only to scripts that declare it (argparse errors on unknown args otherwise).
…al config The notion preprocess (utils.app_specific.notion.notion_remove_and_duplicate) reads source_notion_page_url / eval_notion_page_url off the global all_token_key_session, but the container helper only baked the two integration keys — the URLs resolved to None and every notion task's setup crashed before duplicating the target subpage. Forward TOOLATHLON_NOTION_SOURCE_PAGE_URL and TOOLATHLON_NOTION_EVAL_PAGE_URL through the token env allowlist and add both to _GLOBAL_TOKENS. Bumps adapter version to invalidate the materialization cache.
… credential
Snowflake tasks need configs/snowflake_rsa_key.p8 for keypair auth, but the
credential-injection path validated every payload as JSON and chmod'd 0644 —
wrong for a PEM private key. Add content_format ('json'|'pem') and file_mode to
the credential spec: PEM payloads skip JSON validation and land at 0600. Register
the .p8 spec (b64_env TOOLATHLON_SNOWFLAKE_RSA_KEY_B64, trigger server snowflake).
Removes the last adapter blocker for the 4 Snowflake tasks (account still needs
a human trial signup).
Unblocks the poste-email Toolathlon tasks (7 poste-only fully; 16 more partially, pending canvas/woo/notion/snowflake sidecars). The official variant is single-container, so tasks that talk to a poste.io mail server at localhost never ran. A new _toolathlon_services module emits an environment/docker-compose.yaml with a poste sidecar (which auto-routes the task to the Daytona DinD compose strategy), seeds the 503-user roster (vendored), re-applies poste's plaintext-auth on each boot, and rewrites the task's email configs to reach the sidecar before preprocess seeds. Proven end-to-end on Daytona DinD (apply-phd-email -> clean reward 0.0, no infra error). Bumps _ADAPTER_VERSION to bust the materialization cache.
The notion_official MCP server (npx mcp-remote https://mcp.notion.com/mcp) needs an OAuth token to duplicate the source page during preprocess. Inject the pre-authorized mcp-remote token dir (base64 via TOOLATHLON_NOTION_MCP_AUTH_B64) into /workspace/configs/.mcp-auth at setup, pin mcp-remote to the version whose on-disk token path matches, and make MCP_REMOTE_CONFIG_DIR absolute. Unblocks the 8 notion tasks.
mcp-remote dynamically re-registers a new OAuth client on each run, which invalidates the injected token and forces an interactive re-auth (blocking notion tasks in the sandbox). Pass the pre-registered client via --static-oauth-client-info so mcp-remote reuses it and refreshes the token headlessly. Verified: 'Proxy established successfully' with an expired token.
…in-container notion_official isn't in tasks' needed_mcp_servers — the preprocess spawns mcp-remote itself from configs/mcp_servers/notion_official.yaml. So detect notion tasks by the 'notion' (token) server they all list, and patch that yaml in-container (pin version, absolute auth dir, --static-oauth-client-info) so the preprocess page-duplication reuses the injected token headlessly.
…ntainer yaml dep) The container's /usr/bin/python3 lacks PyYAML, so parse-and-patch failed. Overwrite the yaml wholesale with a known-good headless-reuse config instead, and suppress the macOS-xattr tar warning on the auth-dir extract.
bingran-you
temporarily deployed
to
pypi-internal-preview
July 5, 2026 01:43 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5d98439dbd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+215
to
+219
| ports: | ||
| - "10005:80" | ||
| - "2525:25" | ||
| - "1143:143" | ||
| - "1587:587" |
There was a problem hiding this comment.
For ordinary email tasks, poste_config_rewrite_command rewrites the task configs to reach the sidecar as poste on its internal ports, so these host bindings are unused; Docker documents HOST:CONTAINER port mappings as binding the first port on the Docker host (https://docs.docker.com/get-started/docker-concepts/running-containers/publishing-ports/). When two local Docker rollouts with Poste sidecars run concurrently, both compose projects try to reserve 10005/2525/1143/1587 and one compose up fails before the agent starts. Only publish fixed host ports for the k8s/host-network case, or keep non-k8s email tasks on the compose-internal network.
Useful? React with 👍 / 👎.
Collaborator
Author
There was a problem hiding this comment.
Addressed in 515ae08: ordinary Poste sidecars no longer publish fixed host ports; those bindings are only emitted when the k8s host-network runtime is also required. Added a regression assertion that email-only tasks have no ports block, while the k8s+Poste test still requires the fixed mappings.
bingran-you
force-pushed
the
bry/toolathlon-poste-sidecar
branch
from
July 5, 2026 01:49
5d98439 to
a39687d
Compare
bingran-you
temporarily deployed
to
pypi-internal-preview
July 5, 2026 01:49 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
bingran-you
force-pushed
the
bry/toolathlon-poste-sidecar
branch
from
July 5, 2026 01:55
a39687d to
89e9d6c
Compare
bingran-you
temporarily deployed
to
pypi-internal-preview
July 5, 2026 01:55 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
bingran-you
temporarily deployed
to
pypi-internal-preview
July 5, 2026 02:06 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
uv run pytest tests/test_source_adapters.py tests/test_daytona_dind_compose_up.py tests/test_sandbox.py::TestDockerExecEnvSecrecy -q-> 31 passeduv run ruff check src/benchflow/adapters/_toolathlon.py src/benchflow/adapters/_toolathlon_services.py src/benchflow/adapters/source.py src/benchflow/sandbox/docker.py src/benchflow/sandbox/daytona_dind.py tests/test_source_adapters.py tests/test_daytona_dind_compose_up.py tests/test_sandbox.py-> passedwoocommerce-update-coverwith--agent oracle --sandbox dockercompleted environment setup and reached expected missing-oracle boundary (environment_setup=142.6s).canvas-list-testwith--agent oracle --sandbox dockercompleted environment setup and reached expected missing-oracle boundary (environment_setup=84.4s).k8s-mysqlwith--agent oracle --sandbox dockercompleted environment setup and reached expected missing-oracle boundary (environment_setup=66.4s).Notes
mainatb7bbac3f9a1f381b095c878debe1a47dd164ad85.oracle/solve.sh; the expected terminal error for these infrastructure probes is the missing-oracle error after setup completes.