Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Jepsen][YCQL][v2.6] Inconsistency in bank test under partition #1946

Closed
frozenspider opened this issue Jul 31, 2019 · 3 comments
Closed

[Jepsen][YCQL][v2.6] Inconsistency in bank test under partition #1946

frozenspider opened this issue Jul 31, 2019 · 3 comments
Assignees
Labels
kind/bug This issue is a bug

Comments

@frozenspider
Copy link
Contributor

frozenspider commented Jul 31, 2019

(Tested against 1.3.1.0 (b16), latest official release ATM)
Running ycql/bank workflow with partition nemesis encountered a consistency violation - at some point total amount of money for all bank accounts became 96 (diverged from 100) and remained so till the end

Full logs: logs-jepsen_2019-07-31_20-52-14__ycql-bank_n-part_c10_t600.tar.gz

I encountered this once, re-running the test 2nd and 3rd time caught nothing.

Command used to run test:

lein run test -o debian --version 1.3.1.0 --concurrency 2n --workload ycql/bank --nemesis partition --time-limit 600
@frozenspider frozenspider added the kind/bug This issue is a bug label Jul 31, 2019
@frozenspider frozenspider added this to To Do in YCQL via automation Jul 31, 2019
@frozenspider frozenspider added this to To do in Jepsen Testing via automation Jul 31, 2019
@frozenspider
Copy link
Contributor Author

More details.
Relevant history snippet: https://gist.github.com/frozenspider/a876f86559054c0cc695a2bf672fe819
Between 15:39:33,196 and 15:39:50,530, all 13 read ops failed, while 5 out of 14 transfer ops succeeded

@frozenspider
Copy link
Contributor Author

frozenspider commented Aug 8, 2019

I was able to reproduce it, 1 out of 100 attempts.
Note that this time complete yb-tserver logs for n5 node are 1.3 GB
Here's a truncated logs, where I've removed tserver log entries after 3 mins (i.e. after consistency violation):
logs-jepsen_2019-08-08_15-59-42__ycql-bank_n-part_c10_t600_violation!_cut.zip
bank

@frozenspider frozenspider moved this from To do to In progress in Jepsen Testing Aug 12, 2019
@frozenspider frozenspider self-assigned this Aug 12, 2019
@qvad qvad changed the title [Jepsen] [YCQL] Consistency violation in bank test under partition [Jepsen][YCQL] Inconsistency in bank test under partition Jun 27, 2022
@qvad qvad changed the title [Jepsen][YCQL] Inconsistency in bank test under partition [Jepsen][YCQL][v2.6] Inconsistency in bank test under partition Jun 27, 2022
@qvad
Copy link
Contributor

qvad commented Jul 29, 2022

Not reproducible in current stable 2.8 version

@qvad qvad closed this as completed Jul 29, 2022
YCQL automation moved this from To Do to Done Jul 29, 2022
Jepsen Testing automation moved this from In progress to Done Jul 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug This issue is a bug
Projects
YCQL
  
Done
Status: Done
Development

No branches or pull requests

2 participants