Skip to content

test: isolate the keychain from the test suite - #214

Merged
pmaxhogan merged 4 commits into
mainfrom
test/keychain-isolation
Jul 30, 2026
Merged

test: isolate the keychain from the test suite#214
pmaxhogan merged 4 commits into
mainfrom
test/keychain-isolation

Conversation

@pmaxhogan

@pmaxhogan pmaxhogan commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Rebased onto main @ 9d67765 (after #201, #216, #213, #212, #203, #219).

#207 already fixed the root cause for the two crates it touched, so this PR is
now the audit, the consolidation, and the docs - the three halves it
did not cover.

Root cause (recap, for context)

keyring 4.x's Entry::new installs the PLATFORM-NATIVE store as the process
default on its first call, overwriting whatever default is already set
(keyring-4.1.5/src/v1.rs, the SET_CREDENTIAL_STORE latch). So installing
keyring-core's mock before the first real Entry is silently undone, and
every "mock" write lands in the real OS keychain. On macOS each rebuild is a new
binary identity, so the OS re-prompts on every single rebuild and the run blocks
on the modal dialog.

Confirmed against the maintainer's login keychain before touching anything - it
held the tests' own hard-coded account ids:

driven.google.refresh_token / acct-with-token       (created 20260729180400Z)
driven.google.client_creds  / acct-byo
driven.s3.credentials       / acct-s3-round-trip    (created 20260729181820Z)

1. The audit: are there any other paths?

Verified answer: no. Every test that reaches a real keyring::Entry is now
behind the shared helper - there are no unisolated paths left.

Done empirically rather than by inspection. A temporary probe panicked at all
four Entry construction sites in the workspace -
driven-crypto/src/keystore.rs:63, driven-drive/src/google/token_store.rs:102
and :157, driven-s3/src/config.rs:221 - and
cargo test --workspace --no-fail-fast ran every test binary on this exact
tree. Exactly 13 tests reach an entry, across 5 binaries:

Crate Tests reaching an entry Services touched
driven-backend 3 (pre-existing) driven.google.refresh_token, driven.google.client_creds
driven-s3 4 (pre-existing) driven.s3.credentials
driven-crypto 3 (new here) dev.maxhogan.driven
driven-drive 2 (new here) driven.google.refresh_token, driven.google.client_creds
src-tauri 1 (new here) dev.maxhogan.driven

All 13 go through driven_test_fixtures::keychain::isolated(). The probe hit
list and the isolated-test list match exactly, with no remainder.

Also checked and clear: driven-cli (its integration tests only exercise
--help / missing-argument paths, so the spawned binary never opens an entry),
every integration test under crates/*/tests and src-tauri/tests, and the two
new crates from this week - driven-localfs and driven-rclone reference
neither keyring nor any of the credential-store types.

2. Consolidation: one helper, not three

driven_test_fixtures::keychain (crates/driven-test-fixtures/src/keychain.rs)
is now the single implementation. Both of #207's local copies are repointed at
it and keyring-core is dropped from both crates' dev-deps:

  • crates/driven-backend/src/lib.rs - 60-line local helper -> one-line delegate
  • crates/driven-s3/src/config.rs - same

It keeps #207's load-bearing ordering (burn the latch with a throwaway Entry,
then install the mock) and adds one thing #207's version lacks:

An I/O-free precondition ahead of the proof write. The installed default
store must report CredentialPersistence::ProcessOnly, which by definition
means its credentials cannot outlive the process; a real OS keychain reports
UntilDelete/UntilReboot. So a defeated mock is caught before anything is
written
. Under #207's version the sentinel write is itself the first thing
that would leak into a real keychain when the mock fails. The sentinel round
trip still runs, after the precondition passes, as the functional check.

Constructing the throwaway Entry is safe: build in
apple-native-keyring-store-1.0.1/src/keychain.rs is pure struct construction
with no keychain I/O, so it raises no prompt and creates nothing.

Drift protection: the_test_suite_is_isolated_from_the_os_keychain in every
crate that owns a keychain call site - driven-crypto, driven-drive,
driven-s3, driven-backend, src-tauri. These assert rather than skip, so a
future keyring upgrade that breaks the mechanism fails loudly instead of the
suite quietly resuming real writes. All five crates now have the dev-dep wired,
so isolating a new test is a one-line change with no Cargo.toml work.

Production is untouched: no shipped code path changed, and keyring-core is a
direct dependency only of driven-test-fixtures, which is publish = false and
only ever a [dev-dependencies] entry.

3. The docs half

README - new "macOS re-prompts for keychain access after every update"

Placed in the existing macOS caveats, between the APFS locked-file section and
the auto-updater caveat. Verified, not restated:

  • Driven's macOS build carries no Developer ID signature. It is only ad-hoc
    (linker) signed, so it has no stable
    designated requirement
    and its cdhash changes with every build. (tauri.conf.json sets no
    signingIdentity; release.yml runs no codesign.)
  • macOS pins keychain ACLs to the identity of the binary that was granted
    access. Apple states the rule directly:
    "This dialog appears if you recently updated your system software or the app, or if the app has been modified".
    So "Always Allow" does not carry across an update, for any of the four
    services Driven uses (dev.maxhogan.driven,
    driven.google.refresh_token, driven.google.client_creds,
    driven.s3.credentials).
  • Denying it is safe by construction - an encrypted source whose master key
    cannot be read fails closed (crypto.key_missing) rather than uploading
    plaintext - but it stalls encrypted sources until the user re-authorizes.
  • The only fix is a Developer ID signature. Stated without overstating: no
    entitlement or partial workaround makes an ad-hoc-signed build's grants
    survive an update.

Aligned with #216, not contradicting it. #216's fdaBanner.unsignedNote
already covers the TCC/Full Disk Access half ("macOS ties this permission to the
app's signature, and Driven is not signed yet ... remove Driven from the Full
Disk Access list and add it back"). The README's FDA bullet gives the same
mechanism and the same remove-and-re-add remedy, and now explicitly says the
in-app banner says the same thing. The genuinely new material is the keychain
half, which nothing documented.

DESIGN

  • 3.6 gains "macOS, second cost: permission grants do not survive an update"
    next to the existing "no Apple Developer ID" material. It defers the TCC half
    to 5.3.3 rather than restating it, and documents the keychain half.
  • 5.3.3 (feat(ui): guide macOS users to grant Full Disk Access when files are denied #216's FDA onboarding section, which already explains the cdhash
    binding for TCC) gains a short pointer noting the same binding governs
    keychain ACLs, with a cross-reference to 3.6.

CONTRIBUTING

A local-gates subsection: the suite is keychain-isolated, how to isolate a new
test, why macOS makes it load-bearing, and cleanup commands for anyone who ran
the suite while the flawed helper was on main.

Bonus: coverage that was previously impossible

The old module docs in driven-crypto/src/keystore.rs and
driven-drive/src/google/token_store.rs claimed the mock store was unusable and
steered contributors away from testing these paths. Both are corrected, and the
paths they excluded are now covered for real against the in-memory store:

  • Keystore store/load/delete master key, NotFound on a wiped keychain,
    idempotent delete, per-account scoping
  • KeyringTokenStore and ClientCredsStore round trips, incl. the empty-secret
    PKCE case and per-account isolation
  • KeystoreCryptoProvider with an encrypted source that HAS a wrapped key but
    no master key - the only path there that actually opens the keystore, which no
    existing test reached (every existing one short-circuits on a missing wrapped
    key). This is the GA-critical fail-closed rule.

Verification: zero keychain prompts

Prompts are interactive and the machine is locked, so this is proved, not
asserted. Headless throughout - no GUI automation, no app launch.

  1. Complete audit. The 4-site panic probe above enumerated every test in the
    workspace that constructs a keychain entry; all 13 are isolated.
  2. The five guard tests pass, so the mock really is the effective default
    store in each of those binaries - the exact thing that was silently false.
  3. mdat unchanged. After a full cargo test --workspace, the S3 item's
    mdat is still 20260729181820Z - unchanged across multiple full runs and a
    targeted cargo test -p driven-s3, whose
    credentials_round_trip_through_the_keychain stores to that exact account.
    Nothing wrote.
  4. Nothing created. The two Google items were cleaned up mid-session, so
    runs since then started from an empty state for those services. After the
    final full run, dev.maxhogan.driven, driven.google.refresh_token,
    driven.google.client_creds, the helper's own driven.test.keyring-latch /
    driven.test.keyring-sentinel and the test round-trip service are all
    absent
    - despite the suite performing real store/load/delete round trips
    against every production service.
  5. No dialog. Every run executed non-interactively in the background and
    exited on its own, so nothing blocked on a modal prompt.

Gates (all re-run after the rebase onto 9d67765)

  • SQLX_OFFLINE=true cargo test --workspace - pass, 0 failures
  • SQLX_OFFLINE=true cargo clippy --workspace --all-targets -- -D warnings - clean
  • cargo fmt --all -- --check - clean
  • git diff --check - clean
  • cargo deny check - advisories ok, bans ok, licenses ok, sources ok
  • No em/en dashes in the diff (checked)
  • UI suite not run: no UI file is touched by this PR.

Note: test: is a hidden changelog type here, so the user-facing README/DESIGN
macOS caveat will not appear in the release notes. Flagging deliberately - a
follow-up docs: commit is your call.

@github-actions

github-actions Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Coverage

Area main this PR delta
Rust (lib crates) 82.84% 82.91% +0.08 (OK)
UI (vue/ts) 92.29% 92.29% +0.00 (OK)

Gate: passed - no coverage regression (epsilon 0.1 pp).

`cargo test --workspace` was writing into the developer's real macOS login
keychain, which produced a fresh "allow <binary> to access <service>" prompt
on every rebuild (each rebuild is a new binary identity) and could block the
run on a modal dialog.

Root cause: `keyring` 4.x's `Entry::new` installs the PLATFORM-NATIVE store as
the process default on its FIRST call, overwriting whatever default is already
set (`keyring-4.1.5/src/v1.rs`, the `SET_CREDENTIAL_STORE` latch). So the
"install keyring-core's mock as the default store" helper added in #200 was
silently undone by the first real `Entry`, and every "mock" write landed in the
OS keychain. Confirmed empirically: the login keychain contained
`driven.google.refresh_token`/`acct-with-token` and
`driven.google.client_creds`/`acct-byo` - the literal hard-coded ids of the
driven-backend tests.

A workspace-wide probe (panic at every `Entry::new` call site, full
`cargo test --workspace`) found exactly three offending tests, all in
`crates/driven-backend/src/lib.rs`. No other crate's tests reach a real entry.

The fix, as one shared mechanism rather than a per-crate copy:
`driven_test_fixtures::keychain::isolated()` burns the latch first, installs
the mock, then PROVES it is the effective store before any secret is stored -
first with an I/O-free precondition (the default store must report
`CredentialPersistence::ProcessOnly`, so a defeated mock is caught before
anything could leak), then with a sentinel round trip through the same
`keyring` facade production uses. It returns `None` (caller skips) rather than
ever writing for real. This adopts and generalises the sequence from #207.

Also:
- Guard tests (`the_test_suite_is_isolated_from_the_os_keychain`) in every
  crate that owns a keychain call site, so a future `keyring` upgrade that
  breaks the mechanism fails loudly instead of silently writing for real.
- Real round-trip coverage of `Keystore`, `KeyringTokenStore`,
  `ClientCredsStore` and the encrypted-source-with-no-master-key fail-closed
  path, all previously untestable. Corrects the module docs in driven-crypto
  and driven-drive that claimed the mock store was unusable.
- Documents the shipped-side cost of the same macOS mechanism: because Driven
  carries no Developer ID signature (only an ad-hoc/linker signature, so no
  stable designated requirement and a per-build cdhash), keychain ACLs and TCC
  grants are pinned to that exact build, so users are re-prompted for keychain
  access on every update and a granted Full Disk Access can lapse. README
  macOS caveats + DESIGN 3.6.

Verified: full `cargo test --workspace` from an empty login keychain completes
non-interactively with zero failures, and none of the six Driven keychain
services exists afterwards - despite the suite round-tripping real
store/load/delete against all three production services.
@pmaxhogan
pmaxhogan force-pushed the test/keychain-isolation branch from 89298a5 to 9379811 Compare July 29, 2026 23:47
@pmaxhogan
pmaxhogan merged commit 562c200 into main Jul 30, 2026
19 checks passed
@pmaxhogan
pmaxhogan deleted the test/keychain-isolation branch July 30, 2026 00:05
@github-project-automation github-project-automation Bot moved this from Todo to Done in Driven Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant