Repository navigation
A stress harness for storage faults, process kills and concurrent commits: in-tree or a separate repo? #10801
wombatu-kun
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Since late October I have been running a seeded stress harness against Lance. It has found nine defects so far, each reported with a standalone reproducer; all of them are fixed on
main:It also reproduces the older #3718 in a focused run.
How it differs from lance-fuzz and lance-vibecheck (#9039): those vary the operations and the types. This one varies what happens around the operations: object store errors and torn writes injected through a
WrappingObjectStore, cancellation at named points, SIGKILL in the middle of a commit, cgroup memory limits, concurrent writers with forced commit races and a cleanup running under them, slow and hung storage calls, and a multi-hour soak that watches heap, threads, descriptors and manifest size. Each seed draws a plan, runs it in a child process and checks every version, row and index answer against a model. A failure prints its class and the command that replays it, and keeps the plan, the storage calls and the logs.It already covers what @wjones127 asked for in #9039: it runs locally with one cargo command, can focus on one kind of trouble or one operation, and is built to run on a schedule. Its self-tests check that every failure class is actually reported, and a mutation round (deliberate bugs patched into Lance, each of which must be caught) checks the checks.
Cost: about 17k lines of Rust under
rust/lance/tests/stress, Linux only; about 1.1k of them test the harness itself (24 self-tests and 18 unit tests). Outside that directory it adds a[[test]]entry torust/lance/Cargo.tomland a pointer inrust/lance/tests/README.md. It is atest = falsetarget likearrow_ordering, so the default test run skips it, and the clippy job could lint it with one more--test stress. Its memory scenario reads the trace events added in #10754, so it would land after that PR.Two questions:
lance-format/lance-stressthat runs againstmainand files reduced issues here?bug:issues, so that only reduced, confirmed defects reach the tracker?I will keep it green, triage what it finds, and update its model when Lance changes behavior on purpose.
The branch, harness commits only: test/stress-harness
cc @Xuanwo, who merged all of the fixes above.
All reactions