Skip to content

rlm harness: remove sandbox-side git shim (replaced by rlm tool-level block, rlm#70)#1262

Merged
rasdani merged 1 commit intomainfrom
feat/remove-git-shim-from-rlm-harness
Apr 28, 2026
Merged

rlm harness: remove sandbox-side git shim (replaced by rlm tool-level block, rlm#70)#1262
rasdani merged 1 commit intomainfrom
feat/remove-git-shim-from-rlm-harness

Conversation

@rasdani
Copy link
Copy Markdown
Contributor

@rasdani rasdani commented Apr 28, 2026

Why

The rlm harness has been dropping a git refusal shim into $HOME/.local/bin/git so the agent's PATH resolves git to the shim before /usr/bin/git. The shim's been through three iterations — added on a system PATH dir (#1225), relocated to $HOME/.local/bin so scoring's real-git calls keep working (#1244), reworked into a rename-to-twit + pre_score_script restoration design (#1254). Each iteration carried more sandbox-plumbing weight while still missing legitimate sandbox-side git usage that happens to traverse the agent's PATH (e.g. an agent-driven pip install git+..., build tools / language toolchains that internally shell out to git).

The block now lives at the rlm tool layer instead: https://github.com/PrimeIntellect-ai/rlm/pull/70.

  • Catches every shell escape (!cmd, %%bash, bash tool invocation).
  • AST-walks subprocess.run / subprocess.Popen / os.system / os.popen for a literal first arg of git.
  • Opt-out via RLM_ALLOW_GIT=1 (pass through ComposableEnv(environment_vars=...) for envs that genuinely need git inside the agent's tools).
  • Documented bypasses: pure-Python git libraries (pygit2, dulwich, GitPython API) and dynamic-attr / dynamic-import obfuscation. Both are accepted — the block targets the ~95% of incidental git invocations, not adversarial circumvention.

Tool-level wins because:

  1. Doesn't break legitimate git inside the sandbox. The shim shadowed every git invocation on the agent's PATH, including ones the agent legitimately wants to make: pip install git+https://... and build tools that internally call git. With the rlm-side block, only direct git from the agent (shell escape / subprocess literal) is refused; anything called transitively through pip/uv/build systems sees the real git.
  2. No filesystem mutation of agent-PATH binaries. No $HOME/.local/bin/git shim, no rename-to-twit, no pre-score restoration step.
  3. Lives with the agent, not the sandbox plumbing. Every harness/env using rlm gets it for free; nothing for ComposableEnv to wire up.

What this PR changes

  • verifiers/envs/experimental/composable/harnesses/rlm.py: drop _GIT_SHIM_BODY, drop the allow_git kwarg, drop the post_install_uploads / post_install_script wiring. Docstring now points callers at RLM_ALLOW_GIT=1 for the opt-out path.
  • verifiers/envs/experimental/composable/harness.py, composable_env.py: keep post_install_uploads / post_install_script as generic extension points (they are dataclass fields and dispatchers, not git-specific). Docstrings reframed accordingly — they no longer name the rlm git shim as the example.
  • tests/test_rlm_composable_env.py: drop test_rlm_harness_blocks_git_by_default and test_rlm_harness_allow_git_uploads_nothing; replace with a single test asserting the harness leaves post_install_uploads / post_install_script as None. The generic dispatcher tests (test_post_install_uploads_and_script_run_after_install, test_post_install_script_failure_raises) stay — they exercise the Harness field plumbing, not the rlm shim.
  • docs/environments.md, environments/AGENTS.md, assets/lab/environments/AGENTS.md: drop "RLM's /usr/local/bin/git refusal shim" from the Harness field summary.

Supersedes / closes

Test plan

  • uv run pytest tests/ → 1059 passed, 2 skipped (pre-existing OpenEnv smoke skips), 0 failed.
  • uv run ruff check verifiers/ tests/ → clean.
  • uv run ruff format --check verifiers/ tests/ → clean.

Note

Medium Risk
Medium risk because it changes rlm_harness behavior/API (removes allow_git and stops installing a git-blocking shim), which may affect environments relying on the previous sandbox post-install hooks.

Overview
Removes the RLM harness’s sandbox-side git refusal shim and its associated allow_git configuration, leaving rlm_harness to no longer set post_install_uploads/post_install_script.

Updates ComposableEnv/Harness docs and environment docs to describe post-install hooks as generic extension points (not RLM-specific), and adjusts tests to only assert that the RLM harness does not wire post-install hooks while keeping the generic post-install dispatcher coverage.

Reviewed by Cursor Bugbot for commit 64c9591. Bugbot is set up for automated code reviews on this repo. Configure here.

… block, rlm#70)

Drop the rlm harness's `$HOME/.local/bin/git` refusal shim and the
`allow_git` kwarg. Git access from inside the agent is now blocked at
the rlm tool layer (PrimeIntellect-ai/rlm#70):
shell-escape detection plus an AST walk over `subprocess.run` /
`os.system` / `os.popen` for a literal first arg of `git`. Opt out
with `RLM_ALLOW_GIT=1` via `ComposableEnv(environment_vars=...)`.

Tool-level beats the shim: legitimate sandbox-side git
(`pip install git+...`, build tools that internally shell out to git,
language toolchains) keeps working, no filesystem mutation of agent-PATH
binaries, and every harness/env using rlm gets the block for free.

`Harness.post_install_uploads` and `Harness.post_install_script` stay as
generic post-install extension points; their docstrings now say so
instead of pointing at the rlm git-shim example.

Supersedes #1225 / #1244 / #1254.
@rasdani rasdani merged commit 5c18d70 into main Apr 28, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant