Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
[Jepsen] Read skew & data corruption in normal operation #894
In YugaByte DB 188.8.131.52 CE, in healthy clusters without faults, reads routinely fail to observe snapshot isolation: transactions could observe part, but not all, of another transaction's effects. It appears that this is not limited to read-only transactions: update transactions could take information based on inconsistent reads and write it back to tables, causing logical corruption of state.
For instance, in this Jepsen test, a simulated system of bank accounts containing $100 fluctuated between $95 and $154 in what appears to be a random walk. These variations can occur quickly: we've observed values doubling in under 2 minutes.
You can reproduce this behavior with Jepsen d4cf55ada5764a7ddca2f296c708b6acec134586 by running something like:
This problem was fixed by 90bc062, which was released in 1.1.10; we have not observed read skew since.