test(depot-client): stabilize fault harness

[NathanFlurry]

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[NathanFlurry]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• chore(rivetkit): note duplicate admission logic #4948
• test(rivetkit): cover bypass fetch header forwarding #4947
• feat(kitchen-sink): add actor probe controls #4946
• fix(gateway): split actor response wait errors #4945
• fix(guard): handle bypass preflight before query parse #4944
• feat(kitchen-sink): add mock agent event log copy #4943
• fix(kitchen-sink): default mock agent to local endpoint #4942
• test(rivetkit): enable driver lifecycle regressions #4941
• test(rivetkit): verify cached websocket db refs exceed grace #4940
• fix(rivetkit-core): keep sleeping actors reachable during grace #4939
• test(rivetkit): cover async websocket close db access #4938
• fix(rivetkit): keep actors awake until keepAwake work finishes #4937
• fix(rivetkit): persist native onSleep state on errors #4936
• fix(rivetkit-core): allow dispatch during sleep grace #4935
• fix(rivetkit): make duplicate sleep requests idempotent #4934
• fix(rivetkit): align logs with engine logfmt #4933
• fix(ci): standardize publish rust toolchain #4931
• SPLIT UP: agentic loop test #4932
• chore(rivetkit): doc skip ready wait #4927
• fix(guard): parse gateway bypass query booleans #4926
• feat(depot-client): add vfs staging cache ttl #4930
• feat(depot-client): wire sqlite optimization flags #4923
• feat(rivetkit): support gateway bypass client overrides #4928
• test(depot-client): stabilize fault harness #4924 👈 (View in Graphite)
• chore(depot-client): split sqlite vfs transports #4922
• feat(rivetkit): support gateway bypass probes #4915
• feat(kitchen-sink): add mock agentic lifecycle loop #4914
• feat(kitchen-sink): add mock agentic loop #4891
• feat(rivetkit): support gateway bypass client requests #4917
• fix(pegboard): restore hibernating request ids #4898 : 2 other dependent PRs (#4899 , #4916 )
• chore(guard): configure local timeout knobs #4913
• docs(rivetkit): raise serverless start payload limit #4911
• test(kitchen-sink): add sqlite memory soak harness #4910
• feat(rivetkit): expose runtime diagnostics #4909
• chore(depot-client): migrate to worker thread #4907 : 2 other dependent PRs (#4912 , #4918 )
• test(driver): select matrix cells with vitest filters #4906
• fix(pegboard): restore hibernating requests on serverful start #4904 : 1 other dependent PR (#4905 )
• fix(rivetkit): drain shutdown work before sleep #4902 : 1 other dependent PR (#4903 )
• feat(sqlite): wire remote sqlite execute handlers #4897
• test(serverless): avoid pinning header validation order #4896
• chore(driver-test-runner): add native/wasm runtime matrix to skill #4884 : 3 other dependent PRs (#4892 , #4894 , #4901 )
• chroe: rename rivetkit-sqlite to depot-client #4883
• fix(rivetkit-wasm): fix mem leaks #4880
• chore(rivetkit): wasm support #4860
• feat(sqlite): add cold read benchmarks and simplify optimizations #4857
• feat(sqlite): benchmark cold reads #4848 : 1 other dependent PR (#4864 )
• docs(actors): document manual lifecycle controls #4847
• test(sqlite): expand db stress coverage #4846 : 1 other dependent PR (#4849 )
• test(raw-websocket): add serverless smoke coverage #4845
• feat(serverless): configure drain grace period #4844
• fix(publish): enable frontend build for engine target and use turbo build instead of turbo build:engine #4843
• feat(rivetkit): expose conn handle in onWebSocket context #4842
• fix(rivetkit): normalize waitUntil/keepAwake promises to null to avoid serde undefined error #4841
• fix(rivetkit): remove requirement for token #4839 : 1 other dependent PR (#4840 )
• fix(pegboard-envoy): allow sqlite requests across lifecycle states #4838
• fix(pegboard): route scoped actor key reads by runner dc #4837
• fix(sqlite-storage): let sqlite open take over stale owners #4834
• fix(engine-tests): parallelize multi-DC startup + enable 6 multi-DC tests #4833
• fix(api): honor cursor in actor_ids list path #4832
• fix(api): silently filter invalid IDs from actor_ids query instead of 400 #4831
• fix(api-peer): make actors_delete idempotent for already-destroyed actors #4830
• fix(pegboard): use Iterator streaming mode in list_names so limit param is honored #4829
• chore(engine-tests): convert envoy tests to Sleep crash policy + triage ignored #4828
• feat(engine-tests): wire envoy actor names through prepopulate_actor_names #4827
• chore(depot): fault injection tests #4868
• feat(sqlite): pitr & forking #4859 : 1 other dependent PR (#4863 )
• chore(sqlite): stateless storage refactor #4856
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[claude]

Code Review: PR #4924 — test(depot-client): stabilize fault harness

Overview

This PR stabilizes the depot-client fault test harness by: (1) serializing fault scenario runs to prevent process-global state interference, and (2) strengthening the database pointer invariant check in verify.rs by cross-validating the scan result against resolve_database_branch.

scenario.rs

Mutex choice is correct. parking_lot::Mutex (already imported at line 33) is used in the sync run() function, which aligns with the project convention of using parking_lot for sync contexts.

Guard lifetime is correct. The named binding gives _run_guard a lifetime until the end of run(), not just the let statement. A bare underscore would drop immediately and release the lock.

Error message is excellent. The bail! message names the exact cargo test invocation needed to fix the problem, which is very helpful for developers hitting this unexpectedly.

Profile-based timeouts are appropriate. 30s vs. 120s for Simple vs. Chaos is a reasonable distinction given Chaos tests introduce fault injection delays.

No concerns with scenario.rs.

verify.rs

Bug fix is substantive (worth calling out in PR description). The original code had a silent control-flow issue: after calling self.violate("database pointer ... is missing") when the scan found nothing, it would fall through to Ok(current) -> Ok(None). Callers depending on a Some result would then panic rather than seeing the violation accumulate. The new early return after the violation is the correct fix.

BucketId::from_gas_id(Id::nil()) pattern is established. This mirrors the same call in vfs_support.rs line 306 and is the correct pattern for test fixtures that do not use bucket isolation.

Potential invariant gap to consider. The new cross-check only fires when both the scan and the resolver return Some. If resolved is Some but scanned_current is None, no violation is raised — whereas the old code would have violated with "database pointer ... is missing". Consider adding a check for the resolved-Some/scanned-None case. This may be intentional if resolve_database_branch can find pointers via its bucket-fallback path that the prefix scan misses, but it is worth confirming.

Summary

Area	Assessment
Serialization approach	Correct — parking_lot::try_lock in sync context
Timeout values	Reasonable for Simple/Chaos profiles
Bug fix in verify.rs	Real fix, not just cosmetic
Cross-check invariant	Strengthened, with one question about the resolved-Some/scanned-None gap
Conventions (Mutex, logging, comments)	All correct

One question to resolve before merging, otherwise this is clean stabilization work.

[NathanFlurry]

Landed in main via stack-merge fast-forward push. Commits are in main; closing to match.