Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix intermittent failure (final I swear) #738

Merged
merged 2 commits into from Aug 15, 2022

Conversation

rukai
Copy link
Member

@rukai rukai commented Aug 10, 2022

I had originally thought that failures were a rare edge case that we should handle individually.
However its become clear that approach wont work.

To fix this issue once and for all I have replaced the illadvised SchemaAwaiter abstraction with logic in our CassandraConnection::execute method that will check for any schema altering commands and call await_schema_agreement() to block until the schema altering command is truly complete.

The rest of the PR should be just asyncification noise

@rukai rukai force-pushed the fix_intermittent_failure_final branch 2 times, most recently from d6a42fd to 878b476 Compare August 10, 2022 05:03
@shotover shotover deleted a comment from github-actions bot Aug 10, 2022
@rukai
Copy link
Member Author

rukai commented Aug 10, 2022

Still getting failures on this?
I wonder if a better fix would be to add a retry mechanism to CassandraConnection::execute?

Or maybe the problem is just that I am reusing the shotover address rather than connecting directly to the cluster.
Maybe a setter API would let us optionally set the schema waiter without making it easy to get the shotover/direct addresses mixed up.

Edit: Ended up solving this with the setter API.

@rukai rukai force-pushed the fix_intermittent_failure_final branch 4 times, most recently from 71b376c to 84fbad6 Compare August 11, 2022 01:41
@rukai rukai force-pushed the fix_intermittent_failure_final branch from 84fbad6 to eb15051 Compare August 11, 2022 03:15
@conorbros conorbros mentioned this pull request Aug 11, 2022
@rukai rukai enabled auto-merge (squash) August 15, 2022 00:21
@rukai rukai merged commit f4d4878 into shotover:main Aug 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants