mgr/cephadm: make scheduler able to accomodate offline/maintenance hosts #42690
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes: https://tracker.ceph.com/issues/51027 Signed-off-by: Adam King <adking@redhat.com>
cfsnyder
approved these changes
Aug 12, 2021
|
@ceph/orchestrators if possible, it would be nice to get this into the next Pacific release. Having monitors go down in a manner that requires manual intervention to bring back up during host reboots seems like relatively high priority. I'll handle the backport PR if we can get this reviewed and merged? |
|
super red, but as good as it gets right now. |
sebastian-philipp
approved these changes
Aug 18, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes: https://tracker.ceph.com/issues/51027
Signed-off-by: Adam King adking@redhat.com
For reference, the reason this is marked as fixing that tracker is that daemons being removed from offline hosts was causing the issue. As part of the pre-remove for mons we remove them from the monmap. Any host that was rebooted could be potentially marked offline at which point cephadm would try to remove the mon. It would complete the pre-remove step of removing the mon from the monmap but fail to remove the daemon itself because it would be unable to ssh into the host. When the host came back up the mon would be stuck in a stopped state because it had been removed from the monmap. By not allowing cephadm to remove any daemons from offline hosts, this issue should be avoided.
Also, right now the name I 'm using for hosts in maintenance or offline mode that we do not want to add or remove daemons on is "unreachable hosts" but if anyone has a better idea for that name I'll change to that.
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume tox