Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(longevity): add longevity with rf=1 #7297

Merged
merged 1 commit into from
May 29, 2024

Conversation

juliayakovlev
Copy link
Contributor

@juliayakovlev juliayakovlev commented Mar 25, 2024

This PR is trying to cover customer issue scylladb/scylladb#16759.
This is from the issue description:

After Upgrading Scylla from 5.2.11 via 5.4.0 to 5.4.1, we started observing missing data in Scylla. 
Every once in a while, an INSERT is committed successfully but the inserted values are not visible 
until Scylla is restarted. 
As far as we can tell, version 5.2.11 was not affected while 5.4.1 is. 5.4.0 was not long enough 
in operation to make a reliable statement.

The suspected bug occurs extremely rarely, making it hard for us to reproduce. 
Out of ~800M inserts per day, only a dozen are affected.

The load is run on keyspace with RF 1 SCT test.
We use scylla-bench because its validate data better then in cassandra-stress

Testing

PR pre-checks (self review)

  • I added the relevant backport labels
  • I didn't leave commented-out/debugging code

Reminders

  • Add New configuration option and document them (in sdcm/sct_config.py)
  • Add unit tests to cover my changes (under unit-test/ folder)
  • Update the Readme/doc folder relevent to this change (if needed)

@juliayakovlev juliayakovlev force-pushed the longevity_rf_1 branch 5 times, most recently from 4fd637d to 6ac127b Compare May 26, 2024 06:35
sdcm/cluster.py Outdated Show resolved Hide resolved
@fruch fruch added the test-integration Enable running the integration tests suite label May 26, 2024
@juliayakovlev juliayakovlev marked this pull request as ready for review May 28, 2024 05:51
This PR is trying to cover customer issue scylladb/scylladb#16759.
This is from the issue description:

After Upgrading Scylla from 5.2.11 via 5.4.0 to 5.4.1, we started observing missing data in Scylla.
Every once in a while, an INSERT is committed successfully but the inserted values are not visible
until Scylla is restarted.
As far as we can tell, version 5.2.11 was not affected while 5.4.1 is. 5.4.0 was not long enough
in operation to make a reliable statement.

The suspected bug occurs extremely rarely, making it hard for us to reproduce.
Out of ~800M inserts per day, only a dozen are affected.

The load is run on keyspace with RF 1 SCT test.
We use scylla-bench because its validate data better then in cassandra-stress
@juliayakovlev
Copy link
Contributor Author

juliayakovlev commented May 29, 2024

@fruch As we discussed, I removed the filter out keyspace by replication factor from this PR.
So in the this new longevity expected disrupt_toggle_table_gc_mode failure with error:

ConfigurationException: tombstone_gc option with mode = repair not supported for table with RF one or local replication strategy

https://argus.scylladb.com/test/e6ef2a1c-7267-4566-9954-f49caa755c50/runs?additionalRuns[]=9c2c48ba-045b-4ef3-90c4-268beca319f3

It will be solved in the separate PR

@juliayakovlev juliayakovlev removed the test-integration Enable running the integration tests suite label May 29, 2024
@juliayakovlev
Copy link
Contributor Author

@fruch this PR is ready for merge. Please, review

Copy link
Contributor

@fruch fruch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fruch fruch merged commit 12974a7 into scylladb:master May 29, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants