pacific: mgr/progress: optimize global recovery && introduce 5 seconds interval #43353
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kamoltat
force-pushed
the
wip-ksirivad-backport-pacific-37544
branch
from
f5c026f
to
9027323
Compare
kamoltat
changed the title
kamoltat
force-pushed
the
wip-ksirivad-backport-pacific-37544
branch
from
9027323
to
c6c70c8
Compare
kamoltat
requested review from
avanthakkar and
pereman2
and removed request for
a team
kamoltat
force-pushed
the
wip-ksirivad-backport-pacific-37544
branch
from
c6c70c8
to
ddc3ad9
Compare
kamoltat
requested review from
jdurgin
and removed request for
avanthakkar and
pereman2
because reported_epoch is an int, not a string Signed-off-by: Neha Ojha <nojha@redhat.com> (cherry picked from commit a8f3a0e)
Current progress module only checks pg stats and osdmap when it is notified by the cluster. However, this is expensive in large cluster with many pools and osds. we change it to only check both pg stats and osdmap every 5 seconds. in the function _osd_in_out() we now calculate `is_relocated` by: old_osds != new_osds such that it does not matter if the difference between osds are positive or negative. Signed-off-by: Kamoltat <ksirivad@redhat.com> (cherry picked from commit 4504749)
Changes some the tests in teuthology to make the test more deterministic. Using: `ceph osd set norecover` and `ceph osd set nobackfill` when marking osds in or out. As this will delay the recovery and make sure it the test cases get the chance to check that there is actually events poping up in the progress module. took out test_osd_cannot_recover from tasks/mgr/test_progress.py since it is no longer a relevant test case since recovery will get triggered regardless if pg is unmoved. Ignoring `OSDMAP_FLAGS` in teuthology because we are using norecover and nobackfill to delay the recovery process, therefore, it will create a health warning and fails the teuthology test. Signed-off-by: Kamoltat <ksirivad@redhat.com> (cherry picked from commit 5f33f2f)
Instead of fetching `pg_stats` from the python part of manager module, we filter out the pgs that are in active + clean state in ActivePyModules.cc then parse these pgs along with `reported_epoch` and the `total_num_pgs` of the clusters to global recovery module. Signed-off-by: Kamoltat <ksirivad@redhat.com> (cherry picked from commit fa92db1)
kamoltat
force-pushed
the
wip-ksirivad-backport-pacific-37544
branch
from
ddc3ad9
to
f8f4c40
Compare
|
@kamoltat I have created https://tracker.ceph.com/issues/52794 to track the backport for pacific. |
|
thank you! |
neha-ojha
approved these changes
Oct 5, 2021
jdurgin
approved these changes
Oct 11, 2021
15 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport tracker: https://tracker.ceph.com/issues/52794
This backport includes:
reported_epoch.Backporting the relevant commits from master PRs:
#37544
#41907
#42034
Signed-off-by: Kamoltat ksirivad@redhat.com
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test dashboard cephadmjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume tox