Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Jepsen] [YCQL] Consistency violation in bank test under partition #1946

Open
frozenspider opened this issue Jul 31, 2019 · 2 comments
Open

[Jepsen] [YCQL] Consistency violation in bank test under partition #1946

frozenspider opened this issue Jul 31, 2019 · 2 comments
Assignees
Labels

Comments

@frozenspider
Copy link
Contributor

@frozenspider frozenspider commented Jul 31, 2019

(Tested against 1.3.1.0 (b16), latest official release ATM)
Running ycql/bank workflow with partition nemesis encountered a consistency violation - at some point total amount of money for all bank accounts became 96 (diverged from 100) and remained so till the end

Full logs: logs-jepsen_2019-07-31_20-52-14__ycql-bank_n-part_c10_t600.tar.gz

I encountered this once, re-running the test 2nd and 3rd time caught nothing.

Command used to run test:

lein run test -o debian --version 1.3.1.0 --concurrency 2n --workload ycql/bank --nemesis partition --time-limit 600
@frozenspider frozenspider added this to To Do in YCQL via automation Jul 31, 2019
@frozenspider frozenspider added this to To do in Jepsen Testing via automation Jul 31, 2019
@frozenspider
Copy link
Contributor Author

@frozenspider frozenspider commented Jul 31, 2019

More details.
Relevant history snippet: https://gist.github.com/frozenspider/a876f86559054c0cc695a2bf672fe819
Between 15:39:33,196 and 15:39:50,530, all 13 read ops failed, while 5 out of 14 transfer ops succeeded

Loading

@frozenspider
Copy link
Contributor Author

@frozenspider frozenspider commented Aug 8, 2019

I was able to reproduce it, 1 out of 100 attempts.
Note that this time complete yb-tserver logs for n5 node are 1.3 GB
Here's a truncated logs, where I've removed tserver log entries after 3 mins (i.e. after consistency violation):
logs-jepsen_2019-08-08_15-59-42__ycql-bank_n-part_c10_t600_violation!_cut.zip
bank

Loading

@frozenspider frozenspider moved this from To do to In progress in Jepsen Testing Aug 12, 2019
@frozenspider frozenspider self-assigned this Aug 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Jepsen Testing
  
In progress
YCQL
  
To Do
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant