Skip to content

feat(sandbox): broker SCM credentials for git operations#679

Merged
ColeMurray merged 5 commits into
ColeMurray:mainfrom
watchdog-no:fix/sandbox-scm-credential-helper
May 28, 2026
Merged

feat(sandbox): broker SCM credentials for git operations#679
ColeMurray merged 5 commits into
ColeMurray:mainfrom
watchdog-no:fix/sandbox-scm-credential-helper

Conversation

@hreiten
Copy link
Copy Markdown
Contributor

@hreiten hreiten commented May 23, 2026

Why

We hit this in our own Open-Inspect deployment: long-running or revisited sessions could no longer run git commands after the GitHub App installation token expired. The common workflow was to kick off an agent, come back more than an hour later, and continue the same session. At that point follow-up fetch/push/ls-remote operations were still using the token captured when the sandbox was created, so git failed even though the session itself was otherwise healthy.

What changed

  • Add a sandbox-authenticated POST /sessions/:id/scm-credentials broker endpoint that mints fresh SCM credentials through the configured provider.
  • Add provider-owned credential-helper auth for GitHub and GitLab, including GitHub installation token expiry metadata.
  • Install a sandbox git credential helper in Modal and Daytona images so git fetch/push/clone/LFS operations request fresh credentials on demand.
  • Stop injecting static git tokens into fresh interactive sandboxes. Keep explicit fallback behavior only for image-build, legacy snapshot, and repo-image compatibility paths.
  • Rewrite stale credentialed origin remotes to plain HTTPS URLs before fetch so older snapshots route through the helper.
  • Add a gh wrapper for GitHub CLI calls, since gh reads env tokens rather than git's credential protocol.

Security and compatibility

The helper fails closed by scoping credential responses to HTTPS and the configured SCM host. It intentionally preserves the existing installation-wide repository access model for that host, so setup/start hooks can still clone auxiliary private repositories the installation can access. It caches credentials locally with a short refresh buffer and 0600 permissions, and it refuses stale-cache fallback on broker failures. Live sessions also refuse to fall back to static VCS_CLONE_TOKEN when control-plane context is present but incomplete.

Modal snapshot restores already mint a fresh fallback token during restore; the stale-token motivation here applies primarily to continuously running sessions and Daytona persistent resumes.

Validation

  • npm test -w @open-inspect/control-plane
  • npm run test:integration -w @open-inspect/control-plane -- test/integration/scm-credentials.test.ts
  • npm run typecheck -w @open-inspect/control-plane
  • npm run lint -w @open-inspect/control-plane
  • npm run build -w @open-inspect/control-plane
  • cd packages/sandbox-runtime && uv run --extra dev pytest tests/test_git_credential_helper.py tests/test_gh_wrapper.py tests/test_entrypoint_build_mode.py tests/test_entrypoint_urls.py -q
  • cd packages/sandbox-runtime && uv run --extra dev pytest tests -q
  • cd packages/sandbox-runtime && uv run --extra dev ruff check src tests && uv run --extra dev ruff format --check src tests
  • npx prettier --check README.md docs/HOW_IT_WORKS.md packages/control-plane/README.md packages/github-bot/README.md
  • cd packages/modal-infra && uv run --extra dev pytest tests/test_sandbox_env_vars.py tests/test_web_api_create_sandbox.py -q
  • cd packages/modal-infra && uv run --extra dev ruff check src tests && uv run --extra dev ruff format --check src tests
  • python3 -m compileall packages/daytona-infra/src

Summary by CodeRabbit

  • New Features

    • Brokered git credential flow with a sandbox-facing endpoint to request short-lived SCM credentials; providers can emit credential-helper credentials.
  • Documentation

    • Updated security model, token architecture, HOW_IT_WORKS, provider checklist, and READMEs to describe brokered credentials, token taxonomy, and legacy fallbacks.
  • Infrastructure / Runtime

    • Sandbox/runtime images and startup configure a git credential helper and GH wrapper; fresh sandboxes avoid embedding clone tokens.
  • Tests

    • Expanded unit/integration tests for credential brokering, providers, sandbox boot/restore, and helper behavior.

Review Change Stack

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 23, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 2a49732a-8ff5-4a8f-9e30-436cd9ec1359

📥 Commits

Reviewing files that changed from the base of the PR and between ed148b9 and 803b965.

📒 Files selected for processing (1)
  • packages/control-plane/src/session/durable-object.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/control-plane/src/session/durable-object.ts

📝 Walkthrough

Walkthrough

Refactors git auth to a credential-helper flow: control plane exposes POST /sessions/:id/scm-credentials; providers implement generateCredentialHelperAuth(); sandboxes obtain short-lived per-request credentials via a system git credential helper and a GH CLI wrapper; clone-token embedding is removed (legacy fallbacks preserved).

Changes

Git Credential Helper Brokering

Layer / File(s) Summary
Provider types and implementations
packages/control-plane/src/source-control/types.ts, packages/control-plane/src/source-control/providers/github-provider.ts, packages/control-plane/src/source-control/providers/gitlab-provider.ts, tests
Adds CredentialHelperAuth, extends SourceControlProvider, and implements generateCredentialHelperAuth() for GitHub and GitLab with tests and 5xx transient-classification update.
Control-plane routing and service
packages/control-plane/src/router.ts, packages/control-plane/src/session/scm-credentials-service.ts, packages/control-plane/src/session/http/handlers/sandbox.handler.ts, tests
Adds POST /sessions/:id/scm-credentials, provider gating, ScmCredentialsService, handler wiring, and router/handler tests.
Session Durable Object wiring
packages/control-plane/src/session/durable-object.ts, packages/control-plane/src/session/contracts.ts
Adds internal /internal/scm-credentials route, uses ScmCredentialsService, and switches to env-driven provider factory.
Provider factory and shared wiring
packages/control-plane/src/source-control/provider-from-env.ts, packages/control-plane/src/source-control/index.ts, packages/control-plane/src/routes/shared.ts, tests
Introduces createSourceControlProviderFromEnv(env) and simplifies shared provider construction; re-exports factory and adds tests.
Daytona provider changes
packages/control-plane/src/sandbox/providers/daytona-provider.ts, tests
Removes getCloneToken from constructor and stops embedding VCS/GitHub tokens into sandbox env vars; updates factory and tests.
Sandbox credential helper & runtime
packages/sandbox-runtime/src/sandbox_runtime/credentials/git_credential_helper.py, packages/sandbox-runtime/src/sandbox_runtime/credentials/__init__.py, packages/sandbox-runtime/src/sandbox_runtime/entrypoint.py, tests
Adds Python git credential helper with caching/lock/refresh, installs credential helper and GH wrapper, configures git to use helper and useHttpPath, normalizes plain HTTPS origins, and updates entrypoint tests.
Modal infra and SandboxManager
packages/modal-infra/src/sandbox/manager.py, packages/modal-infra/src/web_api.py, tests
Distinguishes fresh boots vs prebuilt/snapshot boots for clone_token injection; resolves clone_token only for legacy boots and marks fallback tokens.
Infra image/toolchain
packages/daytona-infra/src/toolchain.py, packages/modal-infra/src/images/base.py
Bumps sandbox image version/cache-busters and installs/configures oi-git-credentials shim in images.
Docs and contributor guidance
README.md, docs/HOW_IT_WORKS.md, docs/adr/0001-single-provider-scm-boundaries.md, docs/provider-contribution-checklist.md, packages/control-plane/README.md, packages/github-bot/README.md
Documents the brokered credential flow, token taxonomy updates, provider responsibility (generateCredentialHelperAuth), and sandbox prerequisites (gh + credential helper).
Extensive tests and integrations
many files under packages/control-plane, packages/sandbox-runtime, packages/modal-infra
Adds unit, integration, and concurrency tests covering provider factory, routing, session handling, ScmCredentialsService, credential helper behavior, GH wrapper, env-var injection behavior, and image/manager flows.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Suggested reviewers

  • ColeMurray

Poem

🐇 I broker creds in a hop and twirl,

Fresh tokens for each little swirl.
Helpers fetch when git asks "please",
Secrets no longer tucked in keys.
Hooray — secure cloning for every swirl!

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/sandbox-runtime/src/sandbox_runtime/entrypoint.py`:
- Around line 62-65: The current entrypoint wrapper condition gates all token
checks behind OI_GITHUB_TOKEN_IS_FALLBACK != "1", which prevents
GITHUB_APP_TOKEN from being treated as a real override when fallback is set;
update the shell conditional around the exec of REAL_GH so that GITHUB_APP_TOKEN
is accepted regardless of OI_GITHUB_TOKEN_IS_FALLBACK (i.e., execute REAL_GH if
GITHUB_APP_TOKEN is set OR if OI_GITHUB_TOKEN_IS_FALLBACK != "1" and either
GH_TOKEN or GITHUB_TOKEN is set), leaving the exec "$REAL_GH" "$@" call intact.
- Around line 230-245: In _ensure_credential_helper_configured, the code sets
git's "credential.helper" even when writing the shim (shim_path.write_text with
shim_body) fails; change the flow so the credential.helper is only configured
when the shim actually exists/wrote successfully: after attempting to write and
chmod shim_path (inside the try/except around shim_path.write_text/chmod) set a
local success flag (e.g., shim_written) or re-check shim_path.exists() and only
proceed to set ("credential.helper", str(shim_path)) when that flag/check is
true; ensure the except block does not fall through to configure
credential.helper pointing at a non-existent shim and keep logging the original
warning (self.log.warn("credential_helper.shim_write_failed", error=str(e))).

In `@packages/sandbox-runtime/tests/test_git_credential_helper.py`:
- Around line 85-590: Tests repeat raw timing literals (3600, 60, 0.1); add
module-level named constants (e.g. ONE_HOUR_SECONDS = 3600, ONE_MINUTE_SECONDS =
60, SHORT_SLEEP_SECONDS = 0.1) and replace the numeric literals used in tests
such as test_get_returns_credentials_on_success,
test_refreshes_when_cache_within_expiry_buffer, test_uses_cache_within_buffer,
test_concurrent_invocations_share_one_refresh (slow_handler sleep),
test_control_plane_response_invalid_expiry_is_fatal,
test_token_action_prints_bare_token, and any other occurrences so all
timeouts/TTLs use the new constants to encode units consistently.
- Around line 551-568: The problem is concurrent per-thread patching of
module-global stdio in run_one(), which can clobber patches when two threads
run; remove the patch.object(helper.sys, "stdin"/"stdout"/"stderr") calls from
inside run_one() and instead apply a single patch around the threaded section
(wrap the t1/t2 start/join block) so the global sys streams are patched once, or
alternatively provide thread-local I/O to helper.main without modifying
helper.sys; update the test_concurrent_invocations_share_one_refresh test to
call run_one() without patching helper.sys and perform any required
sys.stdin/stdout/stderr setup outside the threads (or via thread-local
redirect_context) using the unique symbols run_one(), helper.sys.stdin,
helper.sys.stdout, helper.sys.stderr, and helper.main.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 054d978c-199b-42a9-ae48-0d5e505331e1

📥 Commits

Reviewing files that changed from the base of the PR and between 90473af and 8f52cb7.

📒 Files selected for processing (44)
  • README.md
  • docs/HOW_IT_WORKS.md
  • docs/adr/0001-single-provider-scm-boundaries.md
  • docs/provider-contribution-checklist.md
  • packages/control-plane/README.md
  • packages/control-plane/src/auth/github-app.test.ts
  • packages/control-plane/src/auth/github-app.ts
  • packages/control-plane/src/router.scm-credentials.test.ts
  • packages/control-plane/src/router.ts
  • packages/control-plane/src/routes/shared.ts
  • packages/control-plane/src/sandbox/providers/daytona-provider.test.ts
  • packages/control-plane/src/sandbox/providers/daytona-provider.ts
  • packages/control-plane/src/session/contracts.ts
  • packages/control-plane/src/session/durable-object.ts
  • packages/control-plane/src/session/http/handlers/sandbox.handler.test.ts
  • packages/control-plane/src/session/http/handlers/sandbox.handler.ts
  • packages/control-plane/src/session/http/routes.test.ts
  • packages/control-plane/src/session/http/routes.ts
  • packages/control-plane/src/session/scm-credentials-service.test.ts
  • packages/control-plane/src/session/scm-credentials-service.ts
  • packages/control-plane/src/source-control/errors.ts
  • packages/control-plane/src/source-control/index.ts
  • packages/control-plane/src/source-control/provider-from-env.test.ts
  • packages/control-plane/src/source-control/provider-from-env.ts
  • packages/control-plane/src/source-control/providers/github-provider.test.ts
  • packages/control-plane/src/source-control/providers/github-provider.ts
  • packages/control-plane/src/source-control/providers/gitlab-provider.test.ts
  • packages/control-plane/src/source-control/providers/gitlab-provider.ts
  • packages/control-plane/src/source-control/types.ts
  • packages/control-plane/test/integration/scm-credentials.test.ts
  • packages/daytona-infra/src/toolchain.py
  • packages/github-bot/README.md
  • packages/modal-infra/src/images/base.py
  • packages/modal-infra/src/sandbox/manager.py
  • packages/modal-infra/src/web_api.py
  • packages/modal-infra/tests/test_sandbox_env_vars.py
  • packages/modal-infra/tests/test_web_api_create_sandbox.py
  • packages/sandbox-runtime/src/sandbox_runtime/credentials/__init__.py
  • packages/sandbox-runtime/src/sandbox_runtime/credentials/git_credential_helper.py
  • packages/sandbox-runtime/src/sandbox_runtime/entrypoint.py
  • packages/sandbox-runtime/tests/test_entrypoint_build_mode.py
  • packages/sandbox-runtime/tests/test_entrypoint_urls.py
  • packages/sandbox-runtime/tests/test_gh_wrapper.py
  • packages/sandbox-runtime/tests/test_git_credential_helper.py

Comment thread packages/sandbox-runtime/src/sandbox_runtime/entrypoint.py
Comment thread packages/sandbox-runtime/src/sandbox_runtime/entrypoint.py Outdated
Comment thread packages/sandbox-runtime/tests/test_git_credential_helper.py
Comment thread packages/sandbox-runtime/tests/test_git_credential_helper.py
Comment thread packages/sandbox-runtime/src/sandbox_runtime/credentials/git_credential_helper.py Outdated
env would silently fail once it expires (or immediately, for
providers with short-lived tokens like GitHub Apps).

For image-build sandboxes (one-shot, no control-plane access)
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not required in this PR, but it likely makes sense as a follow-up to have all token auth run through the credentials helper to avoid two different credential approaches

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, that direction makes sense as a follow-up so we do not keep two token paths around longer than necessary. I left it out of this PR per your note and kept this pass focused on preserving same-host auxiliary repo compatibility.

Copy link
Copy Markdown
Owner

@ColeMurray ColeMurray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comments

@ColeMurray
Copy link
Copy Markdown
Owner

ColeMurray commented May 26, 2026

Small correction on the restore motivation: for Modal snapshot restores, we already mint and inject a fresh auth token. api_restore_sandbox calls _resolve_clone_token() per restore request, passes that into restore_from_snapshot, and the restored entrypoint rewrites origin via _ensure_remote_auth() before fetching.

Daytona is different: it does not use Modal-style snapshot restore. It resumes the same persistent sandbox via startSandbox/recoverSandbox, so env vars and credentialed remotes are not rebuilt. The stale-token issue applies to continuously running sandboxes and Daytona persistent resumes, but not Modal snapshot restores.

One extra Daytona note after tracing this more closely: existing Daytona sandboxes created before this helper lands will not automatically get the new helper on resume, since they keep the old filesystem/env. They likely need a fresh spawn or an explicit resume-time migration/patch.

@hreiten hreiten requested a review from ColeMurray May 27, 2026 00:34
@ColeMurray ColeMurray merged commit ce3a14d into ColeMurray:main May 28, 2026
30 of 31 checks passed
@ColeMurray
Copy link
Copy Markdown
Owner

thanks!

hreiten added a commit to watchdog-no/background-agents that referenced this pull request May 28, 2026
chore: sync main with upstream (SCM credential broker ColeMurray#679, dashboard link glyph)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants