Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Jepsen] Read skew & data corruption in normal operation #894

Closed
aphyr opened this Issue Feb 22, 2019 · 1 comment

Comments

3 participants
@aphyr
Copy link

commented Feb 22, 2019

In YugaByte DB 1.1.9.0 CE, in healthy clusters without faults, reads routinely fail to observe snapshot isolation: transactions could observe part, but not all, of another transaction's effects. It appears that this is not limited to read-only transactions: update transactions could take information based on inconsistent reads and write it back to tables, causing logical corruption of state.

bank

For instance, in this Jepsen test, a simulated system of bank accounts containing $100 fluctuated between $95 and $154 in what appears to be a random walk. These variations can occur quickly: we've observed values doubling in under 2 minutes.

You can reproduce this behavior with Jepsen d4cf55ada5764a7ddca2f296c708b6acec134586 by running something like:

lein run test -o debian --version 1.1.9.0 --concurrency 2n --time-limit 60 --test-count 1 -w bank

This problem was fixed by 90bc062, which was released in 1.1.10; we have not observed read skew since.

@aphyr aphyr closed this Feb 22, 2019

@kmuthukk kmuthukk added this to To do in Jepsen Testing via automation Feb 24, 2019

@kmuthukk kmuthukk added the duplicate label Feb 24, 2019

@kmuthukk kmuthukk added this to To Do in Distributed transactions via automation Feb 24, 2019

@kmuthukk kmuthukk moved this from To do to Done in Jepsen Testing Mar 2, 2019

@kmuthukk kmuthukk moved this from To Do to In progress in Distributed transactions Mar 2, 2019

@kmuthukk kmuthukk moved this from In progress to Done in Distributed transactions Mar 2, 2019

@kmuthukk

This comment has been minimized.

Copy link
Collaborator

commented Mar 2, 2019

Fixed in 1.1.10 (90bc062). Duplicate of #673.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.