Repository navigation
colab ssh -s B runs on session A's runtime when ~/.ssh/config enables ControlMaster (root cause + tested fix)
#247
gianlucamazza
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Summary
With OpenSSH connection multiplexing enabled in
~/.ssh/config(a commonHost *setup),colab ssh -s B -- CMDruns on session A's runtime whenever session A has an active or persisted control master.colab upload/ls/status -s Bhit the correct runtime, so files and commands silently diverge across VMs. In our case arm -rf /content/<project>meant for a new session ran on the VM of a long training job.Reproduced on google-colab-cli 0.8.0 (Linux, OpenSSH client). The code is unchanged on current
main(10f442a).Root cause
_ssh_base_argsspawnsssh ... root@colab-runtimefor every session. Only theProxyCommanddiffers between sessions. A config like:resolves every session to the same socket,
~/.ssh/sockets/root@colab-runtime-22. OpenSSH reuses the existing master without ever running the newProxyCommand, so session B's command travels over session A's WebSocket.Side effect: the persisted master keeps holding A's only SSH slot for
ControlPersist. Any later non-multiplexed connection to A then fails withAlready-active SSH session (HTTP 429), or occasionallyConnection timed out during banner exchange.Repro
Observed output with two live runtimes:
Proposed fix (tested)
Disable multiplexing for the
sshthatcolab sshspawns. Command-line-ooptions take precedence over~/.ssh/config, so this works regardless of the user's config. User-writtenHost <alias>+ProxyCommand colab ssh --proxy-mode -s NAMEentries are unaffected, because each alias gets its own socket.@@ def _ssh_base_args(proxy_command: str, identity: Optional[str]) -> list[str]: "-o", f"UserKnownHostsFile={os.devnull}", "-o", + "ControlMaster=no", + "-o", + "ControlPath=none", + "-o", "LogLevel=ERROR", ]Complete change, following
AGENTS.md(tests first, design doc log entry, live verification): branchfix/ssh-disable-connection-multiplexing, one commit,5ec8186.tests/test_ssh_multiplexing.py:-- CMDspawn paths;ssh -G -F <multiplexing config>) and assertscontrolmaster falseand no control path.mainand pass with the fix.tests/test_ssh_cmd.py.docs/06_ssh_access.md.ruff check/ruff format --checkreport nothing new in the touched files.I know external PRs aren't accepted right now. The branch is there in case it saves you time; feel free to take the diff as-is.
Unrelated:
maincurrently failsruff check/ruff format --checkuv run ruff check .reports 5 × E402 inexamples/finetune_run.py. These are intentional, because the imports must follow the in-scriptbitsandbytesinstall.ruff format --checkflags 7 files. A formatting-only fix that marks the E402s with a justified# noqais on a separate branch,chore/fix-preexisting-lint(eeff687). Test suite unchanged.Workaround until a fix ships, in
~/.ssh/configplaced beforeHost *:All reactions