DPE-3547 mitigations for container restart #377
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* set floor for max_connections in 100 * function retries * flush logs in single call * + test coverage
dpe-3547-mitigations-for-container-kills # Conflicts: # tests/unit/test_mysql_k8s_helpers.py
There was a problem hiding this comment.
see vm review comments canonical/mysql-operator#398 (review)
Comment on lines
+758
to
+762
| content = self.container.list_files(MYSQL_DATA_DIR) | ||
| content_set = {item.name for item in content} | ||
| logger.debug("Resetting MySQL data directory.") | ||
| for item in content_set: | ||
| self.container.remove_path(f"{MYSQL_DATA_DIR}/{item}", recursive=True) |
There was a problem hiding this comment.
can you just remove the parent directory?
https://github.com/canonical/mysql-router-k8s-operator/blob/f2cbb11ba9c333563acbb9f9b1e159adbded15b6/src/rock.py#L64-L65
(or remove parent dir & mkdir)
There was a problem hiding this comment.
I've observed an eventual failure to remove the parent directory due some process accessing it, but could not determine the process. Removing the content instead did not present the issue
| @@ -93,7 +93,7 @@ def test_on_leader_elected_secrets(self): | |||
| secret_data = self.harness.model.get_secret(label="mysql-k8s.app").get_content() | |||
|
|
|||
| # Test passwords in content and length | |||
| required_passwords = ["root-password", "server-config-password", "cluster-admin-password"] | |||
There was a problem hiding this comment.
Nothing about his in PR description. Q: Why?
There was a problem hiding this comment.
forgot to pop from local stash, fixed ea1e96d
carlcsaposs-canonical
approved these changes
Feb 15, 2024
taurus-forever
approved these changes
Feb 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue
Workload containers get restarted due timeout on livenessProbe pebble endpoint.
Discussion at juju lp bug: https://bugs.launchpad.net/bugs/2052517
Solution
Here some mitigations and optimizations, not a final solution though.