Retry if MySQL Server is unreachable #190
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
to use json.load instead of .loads
paulomach
reviewed
Jan 18, 2024
paulomach
approved these changes
Jan 18, 2024
| @@ -23,6 +25,15 @@ | |||
| logger = logging.getLogger(__name__) | |||
|
|
|||
|
|
|||
| class _NoQuorum(server_exceptions.Error): | |||
There was a problem hiding this comment.
besides using only here, why not concentrate this in src/server_exceptions.py?
There was a problem hiding this comment.
that's why I put it in workload.py—since it's only used there
my preference is to avoid splitting code across multiple files since imo it's harder to read—I think it's worth splitting if multiple other files use it
carlcsaposs-canonical
added a commit
to canonical/mysql-router-operator
that referenced
this pull request
Jan 19, 2024
taurus-forever
approved these changes
Jan 19, 2024
carlcsaposs-canonical
added a commit
to canonical/mysql-router-operator
that referenced
this pull request
Mar 5, 2024
carlcsaposs-canonical
added a commit
to canonical/mysql-router-operator
that referenced
this pull request
Mar 5, 2024
carlcsaposs-canonical
added a commit
to canonical/mysql-router-operator
that referenced
this pull request
Mar 5, 2024
carlcsaposs-canonical
added a commit
to canonical/mysql-router-operator
that referenced
this pull request
Mar 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #86
Context
These are the situations the charm should handle without raising an uncaught exception:
mysqlrouter --bootstrapfails because server does not have quorummysqlrouter --bootstrapfails because server is unreachable (error code 2003)shell.connect()fails because server is unreachable (error code 2003)(mysqlsh commands seem to not fail even if server does not have quorum)
Solution
Catch exceptions in the aforementioned situations & set waiting status. Retry connections on the next Juju event
Alternatives considered
We decided not to retry within the Juju event so that
A retry within the Juju event with a short timeout was considered. However, during initial deployment ~15 events are currently fired (so a 1 min timeout could block the charm for 15 minutes). It would be possible to mitigate that by storing information about the last retry—but the complexity was not worth it.
Discussion about Juju limitation that we may not get another event until update status: https://matrix.to/#/!xdClnUGkurzjxqiQcN:ubuntu.com/$_SXnGJWUQCeFGu9vpBxD5GJ5hgMMAHchOmWBPiqjDFY?via=ubuntu.com&via=matrix.org&via=cutefunny.art
Implementation/refactoring
DBErrorexceptions, serialize them to JSON, and raise corresponding exception in charm code. This was done to move complexity (i.e. exception handling) from mysqlsh python code to charm code