Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Jepsen] [YCQL] Consistency violation in bank test under partition #1946

Open
frozenspider opened this issue Jul 31, 2019 · 2 comments

Comments

@frozenspider
Copy link
Contributor

commented Jul 31, 2019

(Tested against 1.3.1.0 (b16), latest official release ATM)
Running ycql/bank workflow with partition nemesis encountered a consistency violation - at some point total amount of money for all bank accounts became 96 (diverged from 100) and remained so till the end

Full logs: logs-jepsen_2019-07-31_20-52-14__ycql-bank_n-part_c10_t600.tar.gz

I encountered this once, re-running the test 2nd and 3rd time caught nothing.

Command used to run test:

lein run test -o debian --version 1.3.1.0 --concurrency 2n --workload ycql/bank --nemesis partition --time-limit 600

@frozenspider frozenspider added this to To Do in Query Language via automation Jul 31, 2019

@frozenspider frozenspider added this to To do in Jepsen Testing via automation Jul 31, 2019

@frozenspider

This comment has been minimized.

Copy link
Contributor Author

commented Jul 31, 2019

More details.
Relevant history snippet: https://gist.github.com/frozenspider/a876f86559054c0cc695a2bf672fe819
Between 15:39:33,196 and 15:39:50,530, all 13 read ops failed, while 5 out of 14 transfer ops succeeded

@frozenspider

This comment has been minimized.

Copy link
Contributor Author

commented Aug 8, 2019

I was able to reproduce it, 1 out of 100 attempts.
Note that this time complete yb-tserver logs for n5 node are 1.3 GB
Here's a truncated logs, where I've removed tserver log entries after 3 mins (i.e. after consistency violation):
logs-jepsen_2019-08-08_15-59-42__ycql-bank_n-part_c10_t600_violation!_cut.zip
bank

@frozenspider frozenspider moved this from To do to In progress in Jepsen Testing Aug 12, 2019

@frozenspider frozenspider self-assigned this Aug 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
1 participant
You can’t perform that action at this time.