Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rptest: Fix race condition in CloudRetentionTest #16361

Merged

Conversation

Lazin
Copy link
Contributor

@Lazin Lazin commented Jan 30, 2024

Fixes #15893

The test has race condition. It starts consumer before all data is produced and before retention kicks in. The original intention was to let the consumer validate the data after retention. Basically, to check that the data is readable. But because of that the consumer were reading data that was removed by retention. It was racing with the producer. Largely simplifed version of what was happening there.

  • The producer produces offsets in range [0...1000].
  • The retention removes offsets in range [0...500).
  • The consumer is started concurrently with the producer with the expectation that it will consume offsets in range [500...1000]. We're validating the number of messages consumed.
  • The consumer actually manages to consume offsets in range [0...600]. So the check based on number of messages passes.
  • In CDT the consumer doesn't manage to consume enough data because of different performance characteristics. It consumes in range [0...100] and check fails.

This PR fixes this by starting the consumer after retention is confirmed.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.3.x
  • v23.2.x
  • v23.1.x

Release Notes

  • none

@Lazin Lazin requested a review from andijcr January 30, 2024 16:42
@piyushredpanda piyushredpanda merged commit e6d3bc5 into redpanda-data:dev Jan 30, 2024
19 of 20 checks passed
@vbotbuildovich
Copy link
Collaborator

/backport v23.3.x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI Failure (Valid reads too low) in CloudRetentionTest.test_cloud_retention
4 participants