mgr: avoid false alarm of MGR_MODULE_ERROR #35760
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mgr sends healthy report periodically, the report includes the information whether the always-on modules are loaded or not. but the modules are loaded with two steps: 1. load the options and command exposed by modules. the options and commands are registered using static methods of the subclasss of MgrModule. 2. create an instance of the subclass of MgrModule. this is performed in background by a Finisher thread. upon finishing of the construction of the instance, ActivePyModules::start_one() adds the module which successfully creates the class to `modules`. but there is chance that when mgr sends healthy report, the always-on module is still creating its instance of MgrModule subclass, or that task is still pending in the finisher thread. in that case, mgr would add a false error message like ``` 4 mgr modules have failed (MGR_MODULE_ERROR) ``` in the healthy report in this change, the number of modules in pending state is tracked, and mgr will not take the missing always-on modules into account unless the number of pending modules is 0. Signed-off-by: Kefu Chai <kchai@redhat.com>
|
this change should address the test failure like http://pulpito.ceph.com/rzarzynski-2020-06-24_11:08:39-rados-wip-bl-reenable-hc-distro-basic-smithi/5175357/ |
jdurgin
approved these changes
Jun 25, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
mgr sends healthy report periodically, the report includes the
information whether the always-on modules are loaded or not. but the
modules are loaded with two steps:
commands are registered using static methods of the subclasss of
MgrModule.
in background by a Finisher thread. upon finishing of the construction
of the instance, ActivePyModules::start_one() adds the module which
successfully creates the class to
modules.but there is chance that when mgr sends healthy report, the always-on
module is still creating its instance of MgrModule subclass, or that
task is still pending in the finisher thread. in that case, mgr would
add a false error message like
in the healthy report
in this change, the number of modules in pending state is tracked,
and mgr will not take the missing always-on modules into account unless
the number of pending modules is 0.
Signed-off-by: Kefu Chai kchai@redhat.com
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test dashboard backendjenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume tox