Skip to content

fix(security): implement real TOFU for accept-new host key mode - #241

Merged
inureyes merged 6 commits into
mainfrom
fix/issue-239-tofu-host-key-verification
Jul 30, 2026
Merged

fix(security): implement real TOFU for accept-new host key mode#241
inureyes merged 6 commits into
mainfrom
fix/issue-239-tofu-host-key-verification

Conversation

@inureyes

@inureyes inureyes commented Jul 30, 2026

Copy link
Copy Markdown
Member

Summary

The documented, recommended default --strict-host-key-checking accept-new performed no host key verification at all: it mapped to ServerCheckMethod::NoCheck, accepted any server key, never wrote to ~/.ssh/known_hosts, and never rejected changed keys. This PR implements real TOFU for accept-new, fixes strict (yes) mode's silent downgrade to NoCheck when the known_hosts file is missing, makes changed keys a first-class error in every known_hosts-based check method, and, after two review passes, closes the gaps those reviews found: an OpenSSH-compatible any-matching-entry acceptance rule, hostname sanitization before recording, @revoked/@cert-authority known_hosts marker line handling, case-insensitive hostname matching, and a closed permission-creation window on ~/.ssh/known_hosts.

Correction to the issue's verified facts

The issue's "Verified Implementation Facts" section claims russh 0.62.1 exposes no learn_known_hosts helper and that bssh must implement the known_hosts append itself. That claim is incorrect: russh::keys::known_hosts::learn_known_hosts_path(host, port, pubkey, path) exists in russh 0.62.1 and already handles parent directory creation, newline-safe appending, the [host]:port entry form for non-22 ports, and OpenSSH key serialization. This implementation uses that library helper instead of hand-rolling the append; what the helper does not do, and what this PR adds on top, is restrictive permissions on newly created files (created up front rather than tightened afterward), serialization of concurrent writers, an OpenSSH-compatible any-matching-entry acceptance rule (the helper's own check_known_hosts_path short-circuits on the first non-matching same-algorithm entry), @revoked/@cert-authority marker line handling (the helper's parser reads the marker itself as the literal host field and never matches these lines), and hostname sanitization before recording.

What changed

  • src/ssh/tokio_client/authentication.rs: new path-carrying ServerCheckMethod::AcceptNewKnownHostsFile(String) variant (plus with_accept_new_known_hosts_file), so tests can inject a temp known_hosts path instead of mutating HOME.
  • src/ssh/tokio_client/host_verification.rs (new): implements the trust decision for both accept-new and strict/DefaultKnownHostsFile mode through a shared lookup_known_host, which reads a host's known_hosts entries directly and applies OpenSSH's rule: any recorded key equal to the offered key verifies, and only a host whose recorded keys none of them match counts as changed. This also closes the alternate-algorithm pinning bypass (a key offered under an algorithm the host had not used yet, which check_known_hosts_path cannot distinguish from an unknown host) and avoids the false-positive rejection check_known_hosts_path's short-circuiting collection produces for legitimate layouts such as a stale key kept beside a rotated one or a cluster-style shared line beside a per-node line. Before recording or looking anything up, the module also: rejects a hostname that cannot round-trip through a known_hosts entry (empty, over 255 bytes, whitespace/control characters, or #/,/|/*/?/!/\ metacharacters, since learn_known_hosts_path writes the hostname with no escaping and a malformed entry can never be verified again either); scans for @revoked/@cert-authority marker lines naming the host (host matching supports the exact, comma-list, and */? glob forms, erring toward matching for @revoked when in doubt) and hard-rejects a key matching an @revoked entry with a new HostKeyRevoked error while warning and falling through to ordinary TOFU for a @cert-authority match, since bssh has no CA signature validation to fall back to; and normalizes the hostname to lowercase once, using the normalized value for both the lookup and the recorded entry, matching OpenSSH's case-insensitive known_hosts comparison. Unknown hosts are recorded via learn_known_hosts_path with the OpenSSH-style Permanently added '<host>' (<ALGO>) to the list of known hosts. notice; conflicting keys are rejected with the OpenSSH warning banner (SHA256 fingerprint of the offered key, offending file and line, ssh-keygen -f "<file>" -R "<entry>" remediation hint naming the actual [host]:port-qualified entry) without touching the recorded entry. Recording failure warns and accepts, matching OpenSSH accept-new semantics.
  • Concurrency: the whole check-then-record sequence runs under a process-wide tokio::sync::Mutex, so parallel first-time connects to the same new host record exactly one entry and readers never observe a half-written line. A tokio mutex (not std::sync::Mutex) was chosen because the guard lives inside the async check_server_key handler; the critical section is purely synchronous file I/O today, and the tokio mutex keeps the guard sound if an await point is ever introduced while parking contending tasks instead of blocking runtime workers.
  • Permissions: ~/.ssh and known_hosts are created with mode 0700/0600 directly (DirBuilder/OpenOptions with an explicit mode, #[cfg(unix)]) before learn_known_hosts_path runs, rather than letting it create them with the process umask and tightening the mode only afterward, closing the window in which an unusually permissive umask could leave either one group- or world-writable; pre-existing files and directories are left untouched either way, verified by test.
  • src/ssh/tokio_client/error.rs: new Error::HostKeyChanged { host, port, line } and Error::HostKeyRevoked { host, port, line } variants; the KnownHostsFile and DefaultKnownHostsFile arms in check_server_key no longer collapse russh::keys::Error::KeyChanged into the generic ServerCheckFailed, so strict mode reports changed and revoked keys specifically too.
  • src/ssh/known_hosts.rs: get_check_method is now a pure mapping with no filesystem side effects (it no longer pre-creates an empty known_hosts file; learn_known_hosts_path creates it on demand). Yes maps to KnownHostsFile(default path) unconditionally, since russh treats a missing file as empty, so unknown hosts are rejected instead of unverified; an undeterminable home directory falls back to DefaultKnownHostsFile, which fails closed inside russh. AcceptNew maps to AcceptNewKnownHostsFile(default path). Stale "async-ssh2-tokio doesn't support TOFU" comments deleted.
  • src/jump/chain/tunnel.rs: removed the redundant strict-mode match before get_check_method (it already maps No to NoCheck).
  • src/commands/exec.rs: the port-forwarding path had its own hand-rolled mode mapping that sent accept-new through strict default-known_hosts checking (rejecting unknown hosts); it now routes through get_check_method, so TOFU covers every connection path: direct, first jump hop, intermediate hops, destination through the tunnel, SFTP, interactive (shell and PTY), and port forwarding. All get_check_method call sites were individually verified to flow into ClientHandler with the real hostname preserved (via ToSocketAddrsWithHostname or direct construction), so [host]:port entries round-trip on every path.
  • src/ssh/client/connection.rs and src/ssh/client/file_transfer.rs: guidance arms for HostKeyChanged and HostKeyRevoked (and, in the direct path, SshError(UnknownKey) for strict-mode unknown-host rejections) in connect_error_message and format_ssh_error, following both fix: show full anyhow error chain on interactive connect failure #240 invariants: the context layer does not restate its cause and does not end with a period, enforced by tests alongside the existing ones.
  • src/commands/interactive/connection.rs: regression tests that neither HostKeyChanged nor HostKeyRevoked ever triggers interactive password fallback (retrying with a password would hand credentials to an untrusted endpoint).
  • Docs: docs/architecture/ssh-client.md host key section updated to the actual behavior, including the any-matching-entry rule, @revoked/@cert-authority marker handling, and case-insensitive matching. CHANGELOG.md gains four ### Security entries under [Unreleased] covering each of the above. The CLI help text (accept-new - Accept new hosts, reject changed keys (recommended)) and man page were verified to match reality and are unchanged.

Deliberately out of scope

Two review passes surfaced additional lower-severity findings that are intentionally not addressed here:

  • Running the known_hosts lookup unlocked and taking the process-wide lock only when the host looks unknown. Measured cost of the current fully-serialized lookup is 25 to 400 ms across a 512-node fan-out for realistic known_hosts sizes, and moving the read out of the lock risks reintroducing the duplicate-append race this PR exists to prevent.
  • Cross-process locking (KNOWN_HOSTS_LOCK is per-process only).
  • get_check_method(AcceptNew)'s no-home-directory NoCheck fallback.
  • The dead, divergent duplicate ServerCheckMethod enum in src/shared/auth_types.rs.

Test plan

  • cargo test --lib (1314 passed, 0 failed, exit 0)
  • cargo test --lib ssh::tokio_client (36 passed, including the host_verification suite: first-use recording, no duplicate on reconnect, changed key rejected with entry preserved, any-matching-entry acceptance for stale-beside-rotated and cluster-shared-line layouts, alternate-algorithm rejection, 8-way concurrent first connects, [host]:2222 round-trip, created dir/file get 0700/0600 with pre-existing modes preserved, @revoked matching-key rejection and non-matching-key pass-through, @cert-authority warning-then-TOFU, comma-list/glob/non-standard-port marker matching, case-insensitive hostname matching for both accept-new and strict mode, strict mode parity with accept-new, hostname-sanitization rejection and regression coverage, handler end-to-end TOFU arm)
  • cargo test --lib ssh::known_hosts (5 passed)
  • cargo test --lib ssh::client (22 passed, including the HostKeyChanged/HostKeyRevoked guidance-message invariant tests) and cargo test --lib jump::chain (17 passed)
  • cargo test --lib commands::interactive (20 passed, including the password-fallback exclusion tests for both error variants) and cargo test --lib commands::exec (0 tests; exec.rs has no unit tests of its own)
  • cargo check --lib --tests, cargo clippy --lib --tests -- -D warnings, and cargo fmt --all clean
  • cargo check --lib --target x86_64-pc-windows-gnu could not complete in this environment: the target is installed but the aws-lc-sys build script requires x86_64-w64-mingw32-gcc, which is not present; the failure occurs in a C dependency before any bssh code compiles and is unrelated to this change. Windows compatibility is preserved by gating the permission code on #[cfg(unix)] with a non-unix fallback arm.

Note on an unrelated pre-existing flake: commands::interactive::utils::tests::test_expand_path_with_tilde fails nondeterministically under the full parallel cargo test --lib because several tests call EnvGuard::set("HOME", ...), mutating process-global env while that test reads dirs::home_dir() twice. It is untouched by this PR, and no test added here mutates HOME; the new ServerCheckMethod::AcceptNewKnownHostsFile variant carries a path so every new test injects a temp known_hosts location explicitly.

Closes #239

The documented and recommended default `--strict-host-key-checking accept-new` mapped to `ServerCheckMethod::NoCheck`, so `check_server_key` accepted any server key unconditionally, nothing was ever written to `~/.ssh/known_hosts`, and changed keys were never rejected, giving zero MITM protection contrary to the help text. Strict (`yes`) mode had the same defect class: a missing known_hosts file or an undeterminable path silently downgraded to `NoCheck`. Every known_hosts check error also collapsed into the generic `ServerCheckFailed`, making a changed key indistinguishable from an unreadable file.

accept-new now maps to a new path-carrying `ServerCheckMethod::AcceptNewKnownHostsFile` variant handled by a new `ssh::tokio_client::host_verification` module: a key matching its known_hosts entry is accepted, an unknown host is recorded via russh's `learn_known_hosts_path` (which handles entry formatting including the `[host]:port` form for non-22 ports) and accepted with the OpenSSH-style "Permanently added" notice, and a conflicting key is rejected with the OpenSSH warning banner carrying the offered key's SHA256 fingerprint, the offending file and line, and an `ssh-keygen -R` remediation hint, without overwriting the entry. The whole check-then-record sequence runs under a process-wide `tokio::sync::Mutex` so parallel first-time connects to the same new host record exactly one entry and never observe torn writes; the critical section is purely synchronous file I/O, and a tokio mutex keeps the guard sound inside the async handler. When the recording creates `~/.ssh` or `known_hosts`, they get modes 0700 and 0600 respectively (Unix only; pre-existing files keep their modes).

Strict mode now returns `KnownHostsFile(path)` unconditionally: russh treats a missing file as empty, so unknown hosts are rejected instead of unverified, and an undeterminable home directory falls back to `DefaultKnownHostsFile`, which fails closed. Changed keys surface as a dedicated `Error::HostKeyChanged { host, line }` in the `KnownHostsFile`, `DefaultKnownHostsFile`, and accept-new arms alike, with guidance arms added to `connect_error_message` and `format_ssh_error` that follow the #238 invariants (no cause restatement, no trailing period), and interactive password fallback explicitly never triggers on it. `get_check_method` is now a pure mapping with no filesystem side effects, the redundant strict-mode match in `jump/chain/tunnel.rs` is removed, the stale async-ssh2-tokio comments are deleted, and the hand-rolled mapping in the exec port-forwarding path (which sent accept-new through strict checking) is routed through `get_check_method` so TOFU covers every connection path: direct, first jump hop, intermediate hops, tunnel destination, SFTP, interactive, and port forwarding.

Validated with cargo test --lib for ssh::tokio_client, ssh::known_hosts, ssh::client, jump::chain, and commands::exec (all passing, including 13 new tests covering first-use recording, dedup under 8-way concurrency, non-standard port round-trip, changed-key rejection, permission handling, and strict-mode rejection with a missing file), plus cargo clippy --lib --tests -D warnings and cargo fmt. Tests inject temp known_hosts paths through the new variant instead of mutating HOME.

Refs #239
@inureyes inureyes added type:bug Something isn't working type:security Security vulnerability or fix priority:high High priority issue status:review Under review labels Jul 30, 2026
@inureyes inureyes self-assigned this Jul 30, 2026
Two defects found reviewing #239's TOFU implementation.

`russh::keys::check_known_hosts_path` reports `Err(KeyChanged)` only when an entry for the host exists with the same key algorithm, and `Ok(false)` both for a hostname with no entry at all and for a hostname whose entries are all of a different algorithm. accept-new treated the second case as a first use, appending the offered key next to the existing entry and accepting the connection, so a man-in-the-middle that advertises support for only an algorithm the host has not used yet bypassed pinning entirely: SSH negotiation picks the first client algorithm the server also supports, so the attacker chooses it. OpenSSH resists this by reordering its per-host key algorithm proposal to prefer already-known types, which russh cannot do per host, so `verify_accept_new` now consults `known_host_keys_path` before recording and treats any existing entry for the host as a changed key.

The printed removal command did not work for non-standard ports. known_hosts records those as `[host]:port`, but the warning banner and both client-facing guidance messages emitted `ssh-keygen -R <host>`, which matches nothing, and `Error::HostKeyChanged` did not carry the port for the downstream messages to use. The variant now carries the port, the banner follows OpenSSH's `ssh-keygen -f "<file>" -R "<entry>"` form with the resolved default known_hosts path, and every printed argument is double quoted so an unmatched `[...]` glob does not make zsh reject the command.

Both #240 invariants are preserved: no context layer restates its own cause, and no context message ends with a period.

Refs #239
@inureyes

Copy link
Copy Markdown
Member Author

Implementation Review Summary

Intent

Replace the accept-new host key mode's NoCheck no-op with real TOFU (check, record unknown hosts, reject changed keys), and stop strict yes mode from silently downgrading to NoCheck when known_hosts is missing.

Findings Addressed

  • accept-new recorded and accepted an alternate-algorithm key for an already-pinned host (HIGH). russh::keys::check_known_hosts_path returns Err(KeyChanged) only when an entry of the same algorithm holds a different key; a host pinned only under other algorithms returns Ok(false), indistinguishable from a hostname with no entry. The Ok(false) arm therefore appended the offered key next to the existing entry and accepted. Reproduced end to end: pin an ed25519 key for node1.example.com, offer an RSA key for the same host, and the result was Ok(true) plus a second entry, with only the ordinary Permanently added ... (RSA) notice. Since SSH negotiation selects the first host key algorithm on the client's list that the server also supports, a man-in-the-middle that advertises support for just one algorithm the real host has not used yet chose that algorithm and bypassed pinning completely, defeating the help text's "reject changed keys". OpenSSH resists this by reordering its host key algorithm proposal per host (order_hostkeyalgs), which russh cannot do per host, so the protection has to live in this check. verify_accept_new now consults known_host_keys_path before recording and treats any existing entry for the host as a changed key. Fixed in 328515a.
  • The printed ssh-keygen -R remediation command did not work for non-standard ports (HIGH). known_hosts records those as [host]:port, but the warning banner and both client-facing guidance messages emitted ssh-keygen -R <host>, which matches nothing, so a user following the instructions on a possible-MITM error stayed stuck. Error::HostKeyChanged did not even carry the port for the downstream messages to use. The variant now carries the port, the banner follows OpenSSH's own ssh-keygen -f "<file>" -R "<entry>" form with the resolved known_hosts path, and every printed argument is double quoted so an unmatched [...] glob does not make zsh refuse the command. Fixed in 328515a.

Remaining Items

  • accept-new with no determinable home directory still maps to NoCheck (MEDIUM), so verification is fully disabled with only a stderr warning. This is defensible (no persistent trust state means every host is a first use, and OpenSSH accept-new likewise accepts when known_hosts cannot be opened), and it is not a regression, but it is a silent accept-all in the CLI's default mode. The same class of gap applies when known_hosts exists but is unreadable: russh's known_host_keys_path treats an unopenable file as empty, so every connection is a first use. A process-lifetime in-memory pin of (host, port) -> key alongside the file check would close both cases and also protect bssh's fan-out pattern, where one run opens many connections. Worth a follow-up issue rather than a change in this PR.
  • Permission tightening happens after creation, not at creation (LOW). learn_known_hosts_path creates ~/.ssh and known_hosts with the process umask and restrict_created_permissions chmods them to 0700/0600 afterward, leaving a brief window at a wider mode. known_hosts holds public keys, not secrets, so read exposure is minor; the window only becomes interesting under an unusually permissive umask, where the file could briefly be group or world writable. Creating them directly with DirBuilder::mode(0o700) and OpenOptions::mode(0o600) instead of relying on the library helper would remove the window entirely.
  • The first append to a brand-new known_hosts is preceded by a blank line (LOW), because learn_known_hosts_path seeks to End(-1) to test for a trailing newline and that seek fails on an empty file. Harmless (check_known_hosts_path skips the line) and already accounted for by the tests' entry_lines filter, but OpenSSH does not do it.
  • Strict mode's message for a host pinned only under other algorithms is misleading (LOW). It correctly rejects, but connect_error_message reports "The host is not in known_hosts", when the host is in known_hosts under a different key type.
  • src/shared/auth_types.rs holds a dead duplicate ServerCheckMethod enum (LOW, pre-existing) that is now divergent, lacking the new TOFU variant. Nothing in production imports it; only its own tests do. Out of scope here, but it is a trap for the next person.

Verification

  • All stated requirements implemented. Trust decision confirmed empirically: Ok(true) accepts, an unknown host with no entry records then accepts, and any existing non-matching entry is rejected as HostKeyChanged with the file left byte-for-byte unchanged. Non-standard ports round-trip through the [host]:port form and the error carries the connection port. Strict yes mode rejects an unknown host with known_hosts missing and reports a changed key specifically. no mode writes nothing.
  • Strict mode fails closed in both branches. Some(path) maps to KnownHostsFile, and russh returns Ok(vec![]) for an unopenable file, so an unknown host becomes Ok(false) and then russh::Error::UnknownKey. The None branch's claim checks out against russh 0.62.1: known_hosts_path() returns Err(Error::NoHomeDir) without a home directory, which map_known_hosts_error turns into Error::ServerCheckFailed, an Err that rejects. get_check_method(Yes) can no longer return NoCheck on any path.
  • Concurrency guard is sound. verify_accept_new contains exactly one .await, the lock acquisition itself; the critical section is synchronous file I/O with no await, so no guard is held across one. No std::sync::Mutex is involved. The guard drops at function exit and nothing else takes the lock, so there is no reentrancy or deadlock path, and the lock is never held across network I/O. The 8-way concurrent first-connect test records exactly one entry.
  • No placeholder or mock code. The request_port_forward / cancel_port_forward TODOs in tokio_client/connection.rs are pre-existing and untouched.
  • Integrated into the project code flow. All ten get_check_method call sites cover direct connect, first jump hop, intermediate hops, the tunnel destination, SFTP, both interactive paths, port forwarding, and chain_connection.rs. All three production ClientHandler constructions preserve the real hostname, so [host]:port round-trips on every path. The PR's claim about src/commands/exec.rs is accurate: main mapped AcceptNew to DefaultKnownHostsFile by hand, and that is now routed through get_check_method. Every remaining ServerCheckMethod::NoCheck literal outside get_check_method sits inside a #[cfg(test)] module (src/forwarding/{local,remote,dynamic}) or in tests/streaming_test.rs.
  • Both fix: show full anyhow error chain on interactive connect failure #240 invariants hold for the new guidance arms. Neither HostKeyChanged message restates its cause's wording nor ends with a period, and the new port-2222 variants were added to the existing test_connect_error_messages_have_no_trailing_period and test_format_ssh_error_messages_have_no_trailing_period lists so a regression is caught.
  • Project conventions followed and existing modules reused. The recording delegates to russh::keys::known_hosts::learn_known_hosts_path rather than hand-rolling the append, and known_hosts_entry_name is shared with the client message builders instead of duplicating the [host]:port rule.
  • No unintended structural changes. The only visibility change is mod host_verification becoming pub(crate) to enable that reuse.
  • Help text, comments, docs, and behavior agree. accept-new - Accept new hosts, reject changed keys (recommended) is now true, and the stale "async-ssh2-tokio doesn't support TOFU" comments are gone.
  • Tests pass: ssh::tokio_client 20, ssh::known_hosts 5, ssh::client 20, jump::chain 17, commands::interactive 19, plus cargo check --lib --tests, cargo clippy --lib --tests -- -D warnings, and cargo fmt --all --check all clean. Note the PR body's test plan lists cargo test --lib commands::exec as 15 passed; that filter matches 0 tests, since exec.rs has no unit tests of its own.
  • Windows compilation not verified here, same limitation the PR reports: aws-lc-sys needs x86_64-w64-mingw32-gcc, which is unavailable, so the build fails in a C dependency before any bssh code compiles. By inspection the gating is correct: restrict_created_permissions is #[cfg(unix)], and the #[cfg(not(unix))] arm consumes both flags via let _ = (dir_preexisted, file_preexisted);, so there are no unused-variable warnings on either target.

inureyes added 2 commits July 30, 2026 14:11
`russh::keys::check_known_hosts_path` maps every known_hosts line that matches the host and then gathers the per-line results with `collect::<Result<Vec<bool>, _>>()`, which short-circuits on the first `Err(KeyChanged)`. A host with two or more entries of the same key algorithm therefore fails the check as soon as one of them is not the offered key, even when another entry holds exactly that key. Position does not matter: a correct entry on line 1 with a stale entry on line 2 errors just the same. OpenSSH's `check_key_in_hostkeys` does the opposite, returning HOST_OK if any recorded key for the host matches, and reporting HOST_CHANGED only when the host has keys of the same type and none of them match.

Now that `accept-new` is the CLI default, that mismatch rejected legitimate, OpenSSH-accepted known_hosts layouts with the full "REMOTE HOST IDENTIFICATION HAS CHANGED" banner. Two realistic layouts trigger it: a stale host key kept beside a rotated one for the same host and algorithm, and a cluster-style file with a comma-separated `node1,node2,node3 ssh-ed25519 <shared key>` line plus a per-node `node2 ssh-ed25519 <node2's own key>` line. In the second case, connecting to node2 while it offered its own genuine key was rejected as a changed key, and the banner pointed at the shared line rather than at any entry that actually conflicted. A false man-in-the-middle alarm is a security problem in its own right: the banner's own remediation advice tells the user to run `ssh-keygen -R <host>`, which deletes the legitimate pin and turns the next connection into a fresh trust-on-first-use that accepts whatever key is offered.

`lookup_known_host` now reads the host's entries directly and applies OpenSSH's rule, so any recorded key equal to the offered key is a match and only a host with recorded keys none of which match counts as changed. Strict (`yes`) mode and `DefaultKnownHostsFile` route through the new `verify_known_hosts_file`, which shares that lookup, so every mode agrees on which keys verify and which count as changed. `DefaultKnownHostsFile` also resolves the path itself rather than letting russh resolve it with `std::env::home_dir()` while the banner text was built from `dirs::home_dir()`, closing a latent inconsistency where the check and the `ssh-keygen -f "<file>"` hint could name different files. A missing home directory still fails closed, now as `ServerCheckFailed` directly instead of via russh's `NoHomeDir`.

The offending line reported in the banner now prefers an entry with the same key algorithm as the offered key, since that is the entry OpenSSH would name, and falls back to the host's first entry so an algorithm-only conflict (the alternate-algorithm pinning bypass) still points at a real line. The unknown-host path also parses known_hosts once instead of twice while holding the process-wide lock, because the entries it needs in order to tell "unknown" from "changed" are the same ones the check already read.

Four regression tests cover the layouts OpenSSH accepts (a key matching either of two same-algorithm entries, and a per-host key beside a shared cluster line), the same-algorithm offending line, and strict mode agreeing with accept-new.

Validation:
- cargo check --lib --tests (clean)
- cargo test --lib ssh::tokio_client (24 passed, including 4 new)
- cargo test --lib ssh::known_hosts (5 passed)
- cargo test --lib ssh::client (20 passed)
- cargo clippy --lib --tests -- -D warnings (clean)
- cargo fmt --all --check (clean)

Refs #239
`russh::keys::known_hosts::learn_known_hosts_path` appends the entry as `{host} <key>` (or `[{host}]:{port} <key>`) applying no escaping and no validation to the hostname, while russh's own parser splits each line on `' '`, treats `,` as the host-list separator, skips any line whose first byte is `#`, and reads a leading `|1|` as a hashed host field. Before this branch nothing was ever written to known_hosts, so the hostname string never reached a file; now that accept-new records it, a hostname carrying any of those characters no longer round-trips.

Nothing on the way to `ClientHandler` validates the hostname. `security::validate_hostname` is the only validator that rejects whitespace, newlines and `#`, it is reached only from the `-H/--hosts` branch, and its result is then discarded as the connect hostname anyway because `app::nodes` overwrites it with the unvalidated `~/.ssh/config` `HostName`. `Config::from_backendai_env` builds the node list from the platform-supplied `BACKENDAI_CLUSTER_HOSTS` variable with only `str::trim()` and a non-empty check, which leaves interior spaces, tabs, newlines and `#` intact, and that source outranks every config file with no `-c` flag. YAML cluster hosts, `^hostfile` lines and `expand_env_vars` splicing get no character validation either. `AuthContext::new` is the only check on the exec, SFTP and interactive paths and rejects only NUL, LF and CR, not space, tab, other control characters, `#` or `,`. The port-forwarding path in `commands::exec` builds its `AuthMethod` inline without `AuthContext` at all, so it has no hostname check whatsoever, and that is exactly the path this branch newly gave recording powers.

Four concrete consequences, all reproduced end to end by driving `ClientHandler::check_server_key` with `AcceptNewKnownHostsFile`: a hostname containing a newline writes two lines, so the second one pins the key the server just offered for a host the user never named, and every later connection to that host either accepts the attacker's key or is refused as a changed key; a hostname containing a space writes a key field that `parse_public_key_base64` cannot parse, so the lookup returns `Err` forever and the host becomes permanently unconnectable behind a generic "Server check failed"; a hostname starting with `#` writes a line the parser skips, so every connection is a fresh first use that accepts whatever key is offered and appends another entry, growing the file without bound; and a hostname containing `,` writes a host-list entry that silently pins one server's key for every name in the list.

The guard `ensure_recordable_hostname` therefore lives in `ssh::tokio_client::host_verification` at the point of use rather than in any caller, so it covers every connection path regardless of which caller skipped validation. It rejects an empty or over-long hostname, any whitespace or control character, and `#`, `,`, `|`, `*`, `?`, `!` and `\` (the last four because the file is shared with OpenSSH, whose pattern matching would read such an entry as covering hosts it was never meant to). Both `verify_accept_new` and `verify_known_hosts_file` call it first, so strict mode reports the real reason instead of a misleading "unknown host" and the two modes stay consistent. It fails closed with `ServerCheckFailed`: a hostname that cannot be recorded cannot be verified on any later connection either, so connecting unverified would be the worse outcome.

Tests cover the newline, space, tab, `#`, `,`, `|`, `*`, empty and over-long cases (asserting nothing is written), and a regression guard proves ordinary names, IPv4, IPv6 literals, zone identifiers and the bracketed `[::1]` form still record exactly one entry.

Verified with `cargo check --lib --tests`, `cargo clippy --lib --tests -- -D warnings`, `cargo fmt --all --check`, and the `ssh::tokio_client` (27), `ssh::known_hosts` (5), `ssh::client` (20), `jump::chain` (17) and `commands::interactive` (19) library test suites.

Refs #239
@inureyes

Copy link
Copy Markdown
Member Author

Security and Performance Review (independent second pass)

This pass took the earlier analysis in #241 (comment) as read and went looking for what it missed. Both of its HIGH fixes hold up (see the confirmations at the end). Two new HIGH findings are fixed here, and five lower-severity ones are reported for triage.

Every claim below was reproduced empirically by driving ClientHandler::check_server_key through the public API against a temp known_hosts file, not inferred from reading alone.

Fixed: HIGH, accept-new rejected a host key that IS recorded (ca6425a)

russh::keys::check_known_hosts_path maps each matching known_hosts line and then gathers the results with collect::<Result<Vec<bool>, _>>(), which short-circuits on the first Err(KeyChanged). So a host with two or more entries of the same key algorithm failed the whole check as soon as one of them was not the offered key, even when another entry held exactly that key. Line position did not save it: a correct entry on line 1 with a stale entry on line 2 errored just the same. OpenSSH's check_key_in_hostkeys does the opposite, returning HOST_OK if any recorded key for the host matches and reporting HOST_CHANGED only when the host has keys of the same type and none match.

Because this PR makes accept-new the CLI default, that mismatch rejected legitimate, OpenSSH-accepted known_hosts layouts with the full remote-host-identification-has-changed banner. Two layouts I reproduced:

  1. A stale host key kept beside a rotated one, same host, same algorithm.
  2. A cluster-style file with node1,node2,node3 ssh-ed25519 <shared key> plus a per-node node2 ssh-ed25519 <node2 own key> line. Connecting to node2 while it offered its own genuine key was refused as a changed key, and the banner pointed at line 1, the shared line, rather than at any entry that actually conflicted.

A false man-in-the-middle alarm is a security problem in its own right, not just a UX one: the banner's own remediation advice is ssh-keygen -R <host>, which deletes the legitimate pin and turns the next connection into a fresh trust-on-first-use that accepts whatever key is offered. Training users to clear pins in response to this banner is exactly the habit TOFU is supposed to build against.

The fix reads the host's entries directly through lookup_known_host and applies OpenSSH's rule. Strict yes mode and DefaultKnownHostsFile now route through a shared verify_known_hosts_file, so every mode agrees on which keys verify and which count as changed; strict mode had the identical defect. DefaultKnownHostsFile also resolves its own path now instead of letting russh resolve it with std::env::home_dir() while the banner text was built from dirs::home_dir(), which could have named two different files in one message. It still fails closed with no home directory. The offending line in the banner now prefers an entry with the same algorithm as the offered key, so a user is no longer told to delete an unrelated pin, with a fallback to the host's first entry so the alternate-algorithm conflict still points somewhere real. As a side effect the unknown-host path now parses known_hosts once instead of twice under the process-wide lock.

Fixed: HIGH, the hostname is written into known_hosts with no sanitization (5590918)

learn_known_hosts_path appends the entry with write!(file, "{host} "), or write!(file, "[{host}]:{port} ") for a non-22 port, with no escaping at all. russh's parser splits each line on ' ', reads , as the host-list separator, skips any line whose first byte is #, and reads a leading |1| as a hashed host field. ClientHandler stores its hostname verbatim, and before this PR nothing was ever written to known_hosts, so the string never reached a file. Now it does.

I traced every hostname source to ClientHandler. Nothing on the way validates it in a way that matters here:

  • bssh::security::validate_hostname (src/shared/validation.rs:296-323) is the only validator that rejects whitespace, newlines and #, and it is reached only from the -H/--hosts branch. Its result is then discarded as the connect hostname anyway, because src/app/nodes.rs:68 overwrites it with ssh_config.get_effective_hostname(...), an unvalidated ~/.ssh/config HostName value.
  • Config::from_backendai_env (src/config/loader.rs:49-92) builds the node list from BACKENDAI_CLUSTER_HOSTS applying only str::trim() and a non-empty check. It is priority 2 in load_with_priority, above every config file, and is auto-selected with no -c flag. Inside a Backend.AI session that variable is set by the platform, not typed by the user, and trim() removes only leading and trailing whitespace, not interior spaces, tabs, newlines or #.
  • YAML cluster hosts and ^hostfile lines get no character validation, and expand_env_vars splices arbitrary environment values into the host string.
  • AuthContext::new (src/ssh/auth.rs:144-153) is the only check on the exec, SFTP and interactive paths, and it rejects only NUL, LF and CR, not space, tab, other control characters, #, or ,.
  • src/commands/exec.rs:153-172, the port-forwarding path, builds its AuthMethod inline without AuthContext at all, so it has no hostname check whatsoever. This PR is what newly routed that path through get_check_method, so it became a recording path with zero validation.

Four consequences, all reproduced:

  1. A newline writes two lines. With host = victim]:2200 ssh-ed25519 AAAA + newline + [node1, the second written line is [node1]:2200 <the key the server just offered>, so a server gets its own key pinned for a host the user never named. Every later connection to that host then either accepts the attacker's key or is refused as changed. Exploitation additionally needs the odd name to resolve, which DNS labels do permit, so this is gated rather than blocked.
  2. A space writes evil.example.com plaintext ssh-ed25519 AAAA..., parsed as host evil.example.com with key field plaintext. parse_public_key_base64 then fails, so the host returns Err forever and becomes permanently unconnectable behind a generic "Server check failed".
  3. A hostname starting with # writes a line the parser skips, so the host is never matched again: every connection is a fresh first use that accepts whatever key is offered and appends another entry, growing the file without bound.
  4. A , writes a host-list entry, silently pinning one server's key for every name in the list.

The guard lives in host_verification.rs at the point of use rather than in any one caller, so every connection path is covered regardless of which one skipped validation. It rejects empty, over 255 bytes, any whitespace or control character, and #, ,, |, *, ?, !, \. It fails closed, because a hostname that cannot be recorded cannot be verified on a later connection either, so accepting it would mean permanently unverified access. A regression test pins that IPv6 literals, zone identifiers (fe80::1%eth0), bracketed forms, and ordinary names still record normally.

Behavior change worth calling out in the PR description: a hostname from BACKENDAI_CLUSTER_HOSTS or a YAML cluster with an interior space now produces a connection refusal where it previously produced a differently broken connection attempt. That is the intended fail-closed tradeoff, but it is a visible change.

New, not fixed

MEDIUM: @revoked and @cert-authority markers are silently ignored. russh's match_hostname compares the line's first space-separated field against the host, so on a line like @revoked node1 ssh-ed25519 <key> the first field is the marker and the line never matches node1. Reproduced: offering the revoked key to node1 in accept-new mode returned Ok(true), recorded the revoked key as a new entry, and printed the ordinary "Permanently added" notice. @revoked is OpenSSH's documented way to blocklist a compromised host key, so a key the operator explicitly revoked is trusted and re-pinned. @cert-authority lines are ignored the same way, meaning CA-signed host keys are not honored and every such host is treated as a first use. Fixing this needs marker handling in the lookup rather than a one-line change, so it is left for a follow-up. Detecting a leading @ field and failing closed on it would be a smaller intermediate step.

MEDIUM (performance): the process-wide lock serializes a full known_hosts parse on every connection, including hosts already known. Measured known_host_keys_path on a release build (LTO off, codegen-units 16), one call, host absent:

entries plain hashed (HashKnownHosts yes)
1,000 48 us 161 us
5,000 300 us 807 us
20,000 1.04 ms 3.50 ms

Verdict: acceptable as written. For a typical known_hosts of a few hundred to a few thousand entries, a 512-node fan-out adds roughly 25 ms to 400 ms of fully serialized work, small next to 512 TCP and SSH handshakes, and the holder blocks a worker thread for well under the fraction of a millisecond where spawn_blocking starts to matter. It only becomes visible with a pathological file: 20,000 hashed entries across 1,000 nodes is about 3.5 s of serialized parsing, and at 3.5 ms per call the sync read is long enough to be worth moving off the worker. The cheap improvement, if you want it, is to narrow the lock to the record path: run lookup_known_host unlocked, return immediately on Match or Conflict, and take the lock only when the host looks unknown, re-running the lookup under it. That drops the already-known case (the overwhelmingly common one) to zero contention and keeps the check-then-record sequence atomic, since learn_known_hosts_path appends the whole line in one buffered write under O_APPEND. ca6425a already halved the first-use cost by parsing once instead of twice.

LOW: host matching is case sensitive where OpenSSH lowercases. OpenSSH canonicalizes the hostname to lowercase before the known_hosts lookup; russh's match_hostname does a byte comparison. So connecting to NODE1.example.com when node1.example.com is pinned looks like a first use: the existing pin is bypassed, a second entry is recorded, and any offered key is accepted. Not attacker-reachable on its own, since the case comes from the user's own config, but it does mean the pin depends on spelling. Lowercasing the hostname before the lookup and the record would match OpenSSH.

LOW: impl ToSocketAddrsWithHostname for &[SocketAddr] returns a comma-joined hostname (src/ssh/tokio_client/to_socket_addrs_with_hostname.rs:104-109). If that impl were ever used in production with more than one address, hostname() would return 1.2.3.4,5.6.7.8 and record a single known_hosts entry pinning one server's key for both addresses. Only test code uses it today, and 5590918's guard now rejects it outright, so this is a note rather than a live bug.

LOW: no cross-process locking. KNOWN_HOSTS_LOCK is per process. Two concurrent bssh runs can both record. Benign in the normal case, since both write the same key, but if one run reaches a man-in-the-middle while another reaches the real host, the file ends up with two conflicting same-algorithm entries and the host becomes unusable until an operator intervenes. An flock on the known_hosts file during the record would close it.

Confirming the earlier pass

  • The alternate-algorithm pinning bypass fix holds. I re-derived it independently from russh's source: check_known_hosts_path maps (true, false) to Err(KeyChanged) and everything else with a differing algorithm to Ok(false), so a host pinned only under another algorithm is indistinguishable from an unknown host at that call. ca6425a preserves the protection through lookup_known_host's fallback branch, and test_accept_new_rejects_alternate_algorithm_key_for_known_host still passes.
  • The ssh-keygen -f "<file>" -R "<entry>" hint fix holds, including the double quoting for the unmatched [...] glob.
  • Its open MEDIUM (no determinable home directory maps to NoCheck, and an unreadable known_hosts is treated as empty) is confirmed. russh's known_host_keys_path returns Ok(vec![]) whenever File::open fails, so it cannot distinguish missing from unreadable. Its suggested process-lifetime in-memory pin of (host, port) -> key would close both, and would also cover the fan-out case where one run opens many connections to the same host.
  • Its LOW permission-creation window is confirmed: learn_known_hosts_path creates with the process umask and restrict_created_permissions chmods afterward.
  • One of its LOWs is now resolved as a side effect of ca6425a: strict mode no longer reports "the host is not in known_hosts" for a host pinned only under another algorithm, because that case is a Conflict and produces the changed-key error.

Suggested follow-ups (not blocking)

The two HIGH fixes are point-of-use backstops. The upstream validation gaps are still there and deserve their own issue: src/app/nodes.rs:68 discarding the validate_hostname result by overwriting with the ssh_config HostName, src/commands/exec.rs:153-172 building its AuthMethod without AuthContext, and Config::from_backendai_env applying only trim() to a platform-supplied host list. Also worth folding in: the @revoked and @cert-authority handling, and the dead divergent ServerCheckMethod duplicate in src/shared/auth_types.rs the earlier pass flagged.

Verification

cargo check --lib --tests, cargo clippy --lib --tests -- -D warnings and cargo fmt --all --check all clean. cargo test --lib ssh::tokio_client 27 passed (20 at 328515a, plus 4 from ca6425a and 3 from 5590918), ssh::known_hosts 5, ssh::client 20, jump::chain 17, commands::interactive 19, shared::auth_types 3. No test mutates HOME; the new tests inject the known_hosts path explicitly through AcceptNewKnownHostsFile. Both #240 invariants still hold for the guidance arms.

inureyes added 2 commits July 30, 2026 14:40
Two review passes on the accept-new TOFU implementation reported three remaining gaps that were not yet fixed: known_hosts `@revoked`/`@cert-authority` marker lines were silently ignored, host matching was case-sensitive where OpenSSH is not, and known_hosts permission tightening happened after creation instead of at creation.

russh's `known_host_keys_path` splits each line on `' '` and takes the first field as the host list, so on a marker line such as `@revoked node1 ssh-ed25519 <key>` that first field is the literal string `@revoked`, which never matches `node1`. The line was therefore invisible to the lookup, and a revoked key looked like an ordinary first use: recorded and accepted while printing the normal "Permanently added" notice. A new scan in `host_verification.rs` runs before the ordinary lookup in both accept-new and strict (`yes`)/`DefaultKnownHostsFile` mode: a `@revoked` line whose host matches (exact, comma-list, or `*`/`?` glob form, erring toward matching when in doubt since failing to honor a revocation is worse than an unnecessary rejection) and whose key equals the offered key is a hard rejection with a new `HostKeyRevoked` error, while a matching `@revoked` line naming a different key does not block the offered one. A `@cert-authority` match only warns and falls through to ordinary TOFU, since bssh has no CA signature validation path and failing closed would break every working CA setup with no workaround.

Hostnames are now lowercased once at entry to `verify_accept_new` and `verify_known_hosts_file`, and the normalized value is used for both the lookup and the recorded entry. russh's `match_hostname` is a plain byte compare, so without this a differently-cased hostname bypassed an existing pin and re-TOFU'd instead of hitting the same entry OpenSSH would have written.

`~/.ssh` and `known_hosts` are now created with mode 0700/0600 directly via `DirBuilder`/`OpenOptions` (`#[cfg(unix)]`, with a `#[cfg(not(unix))]` fallback) before `learn_known_hosts_path` runs, instead of letting it create them with the process umask and tightening the mode only afterward. A `mkdir`/`open` syscall that requests the final mode directly cannot be widened by any standard umask, since umask only clears requested bits and neither 0700 nor 0600 carries a group or other bit to clear, so this closes the window instead of narrowing it after the fact. Pre-existing files and directories are left untouched either way, and the prior post-creation chmod is kept as a fallback for the rare case where pre-creation could not run (for example a missing grandparent directory that only `learn_known_hosts_path`'s `create_dir_all` fills in).

`HostKeyRevoked` gets its own guidance arms in `connect_error_message` and `format_ssh_error`, verified against both #240 invariants (no restating the cause, no trailing period), and a regression test confirms it does not trigger interactive password fallback, matching `HostKeyChanged`.

Validation:
- cargo check --lib --tests (clean)
- cargo clippy --lib --tests -- -D warnings (clean)
- cargo fmt --all -- --check (clean)
- cargo test --lib ssh::tokio_client (36 passed, 9 new: revoked matching/non-matching, cert-authority, comma-list/glob/non-standard-port marker matching, case-insensitive hostname matching for accept-new and strict mode)
- cargo test --lib ssh::client (22 passed, 2 new guidance-invariant tests)
- cargo test --lib commands::interactive (20 passed, 1 new password-fallback exclusion test)
- cargo test --lib ssh::known_hosts (5 passed) and cargo test --lib jump::chain (17 passed)

Refs #239
CHANGELOG.md's `[Unreleased]` security entry only described the first commit on this branch (implementing accept-new TOFU) and had drifted from what the branch actually ships. It now has four bullets: the original accept-new/strict-mode/HostKeyChanged work, the OpenSSH-compatible any-matching-entry acceptance rule and alternate-algorithm bypass fix, hostname sanitization before recording, and this branch's last commit (known_hosts marker handling, case-insensitive hostname matching, and the closed permission-creation window).

`docs/architecture/ssh-client.md`'s host key verification section is updated to match: the accept-new bullet now describes any-matching-entry acceptance instead of "keys matching their known_hosts entry", and new bullets cover `@revoked`/`HostKeyRevoked`, the `@cert-authority` warn-and-fall-through behavior, and hostname case normalization and rejection.

The PR body was also updated (via the GitHub API, since `gh pr edit` fails on this repository's Projects-Classic GraphQL deprecation) to describe the final state across all four commits instead of only the first.

Refs #239
@inureyes

Copy link
Copy Markdown
Member Author

PR Finalization Complete

Summary

  • Security gaps closed (Task 1): @revoked/@cert-authority known_hosts marker lines are now honored in both accept-new and strict/DefaultKnownHostsFile mode (new HostKeyRevoked error, @cert-authority warns and falls through to TOFU rather than failing closed); hostname matching is normalized to lowercase once at entry, matching OpenSSH; ~/.ssh and known_hosts are now created with mode 0700/0600 directly instead of tightened after creation, closing the permission-creation window.
  • Tests: 9 new tests in ssh::tokio_client::host_verification (revoked matching-key rejection, revoked non-matching-key pass-through, cert-authority warn-then-TOFU, comma-list/glob/non-standard-port marker matching, case-insensitive hostname matching for accept-new and strict mode), 2 new guidance-invariant tests in ssh::client for HostKeyRevoked, and 1 new password-fallback exclusion test in commands::interactive.
  • Documentation: CHANGELOG.md's [Unreleased] security section now has four bullets covering the full final state across all commits on this branch (TOFU implementation, any-matching-entry rule, hostname sanitization, and this pass's marker/case/permission fixes). docs/architecture/ssh-client.md's host key section updated to match. PR body updated via the GitHub API (gh pr edit hits the Projects-Classic GraphQL deprecation on this repo) to describe the final state instead of only the first commit.
  • Explicitly out of scope, unchanged: the unlocked-lookup performance optimization, cross-process locking, the no-home-directory NoCheck fallback, and the dead duplicate ServerCheckMethod in src/shared/auth_types.rs.

Verification

  • cargo check --lib --tests, cargo clippy --lib --tests -- -D warnings, cargo fmt --all -- --check: all clean
  • cargo test --lib ssh::tokio_client: 36 passed
  • cargo test --lib ssh::known_hosts: 5 passed
  • cargo test --lib ssh::client: 22 passed
  • cargo test --lib jump::chain: 17 passed
  • cargo test --lib commands::interactive: 20 passed
  • cargo test --lib commands::exec: 0 tests (no unit tests in that module)

Commits: 8aa4627 (security fix + tests) and 85a4324 (CHANGELOG/docs reconciliation), on top of the four existing commits on this branch.

@inureyes inureyes added status:done Completed and removed status:review Under review labels Jul 30, 2026
@inureyes
inureyes merged commit c3b8ac2 into main Jul 30, 2026
3 checks passed
@inureyes
inureyes deleted the fix/issue-239-tofu-host-key-verification branch July 30, 2026 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority:high High priority issue status:done Completed type:bug Something isn't working type:security Security vulnerability or fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: accept-new host key mode is implemented as NoCheck, implement real TOFU verification and known_hosts recording

1 participant