Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mgr/progress: optimize global recovery module #37544

Merged
merged 1 commit into from Sep 23, 2021
Merged

mgr/progress: optimize global recovery module #37544

merged 1 commit into from Sep 23, 2021

Conversation

kamoltat
Copy link
Member

@kamoltat kamoltat commented Oct 5, 2020 •
edited

Instead of traversing through each PGs for active+clean state.
We traverse through the PGs in
ActivePyModules.cc which is in C++ and use get
method to directly get active+clean PG nums, as
well as number of PGs.

Signed-off-by: Kamoltat ksirivad@redhat.com

Checklist

  • References tracker ticket
  • Updates documentation if necessary
  • Includes tests for new functionality or reproducer for bug
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox

@kamoltat kamoltat requested a review from jdurgin October 5, 2020 09:44
@tchaikov
Copy link
Contributor

tchaikov commented Oct 5, 2020

jenkins test docs

jdurgin
jdurgin reviewed Oct 6, 2020
View reviewed changes
src/mgr/ActivePyModules.cc Outdated Show resolved Hide resolved
@kamoltat
Copy link
Member Author

kamoltat commented Oct 8, 2020

jenkins test docs

@kamoltat kamoltat force-pushed the wip-mgr-progress-global-efficiency branch 2 times, most recently from 5d7019b to ee4e4a4 Compare November 9, 2020 10:56
@kamoltat
Copy link
Member Author

kamoltat commented Nov 9, 2020

jenkins retest this please

1 similar comment
@kamoltat
Copy link
Member Author

kamoltat commented Nov 9, 2020

jenkins retest this please

@kamoltat kamoltat force-pushed the wip-mgr-progress-global-efficiency branch from ee4e4a4 to e11b788 Compare November 9, 2020 14:52
@kamoltat
Copy link
Member Author

jenkins retest this please

@kamoltat

This comment has been minimized.

Sign in to view
@kamoltat
Copy link
Member Author

jenkins retest this please

@kamoltat
Copy link
Member Author

https://pulpito.ceph.com/ksirivad-2020-11-16_04:02:02-rados-wip-mgr-progress-global-efficiency-distro-basic-smithi/

@kamoltat kamoltat force-pushed the wip-mgr-progress-global-efficiency branch from e11b788 to 040a4e9 Compare November 19, 2020 16:15
@kamoltat kamoltat removed the needs-qa label Nov 21, 2020
@kamoltat
Copy link
Member Author

reran: dead/fail jobs https://pulpito.ceph.com/ksirivad-2020-11-20_07:21:48-rados-wip-mgr-progress-global-efficiency-distro-basic-smithi/

Seems like the tests went well

@github-actions
Copy link

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@kamoltat kamoltat force-pushed the wip-mgr-progress-global-efficiency branch from 040a4e9 to 2adcf85 Compare February 1, 2021 15:41
tchaikov
tchaikov reviewed Feb 1, 2021
View reviewed changes
src/mgr/ActivePyModules.cc Outdated Show resolved Hide resolved
@kamoltat kamoltat force-pushed the wip-mgr-progress-global-efficiency branch from 2adcf85 to 36ca17c Compare February 2, 2021 09:12
@github-actions
Copy link

github-actions bot commented Feb 4, 2021

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@tchaikov

This comment has been minimized.

Sign in to view
@kamoltat kamoltat force-pushed the wip-mgr-progress-global-efficiency branch from 36ca17c to 4ec2068 Compare February 5, 2021 08:22
@tchaikov
Copy link
Contributor

tchaikov commented Feb 5, 2021

@kamoltat i have following build failure

/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-572-gccc67fce/rpm/el8/BUILD/ceph-17.0.0-572-gccc67fce/src/mgr/ActivePyModules.cc: In member function 'PyObject* ActivePyModules::get_python(const string&)':
/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-572-gccc67fce/rpm/el8/BUILD/ceph-17.0.0-572-gccc67fce/src/mgr/ActivePyModules.cc:485:15: error: 'tstate' was not declared in this scope
         [&f, &tstate](const PGMap &pg_map) {
               ^~~~~~
/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-572-gccc67fce/rpm/el8/BUILD/ceph-17.0.0-572-gccc67fce/src/mgr/ActivePyModules.cc:485:15: note: suggested alternative: '_xstate'
         [&f, &tstate](const PGMap &pg_map) {
               ^~~~~~
               _xstate

while compiling my batch when testing this PR. could you fix the build failure and repush?

@github-actions
Copy link

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@kamoltat kamoltat force-pushed the wip-mgr-progress-global-efficiency branch from 4ec2068 to ca1ab3b Compare February 22, 2021 14:02
@kamoltat
Copy link
Member Author

jenkins test docs

@github-actions
Copy link

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@jdurgin
Copy link
Member

jdurgin commented Sep 15, 2021

@kamoltat this was looking good, seems like it just needs a rebase

@neha-ojha neha-ojha self-requested a review September 17, 2021 14:50
@kamoltat kamoltat force-pushed the wip-mgr-progress-global-efficiency branch from ca1ab3b to beee50e Compare September 20, 2021 23:49
@kamoltat kamoltat force-pushed the wip-mgr-progress-global-efficiency branch 2 times, most recently from b6c5d88 to 0eb3d21 Compare September 21, 2021 16:23
@kamoltat
Copy link
Member Author

rados:mgr progress test 18/18 passed
https://pulpito.ceph.com/ksirivad-2021-09-21_06:03:18-rados:mgr-wip-mgr-progress-global-efficiency-distro-basic-smithi/

rados 365/374 passed (failures and dead jobs seems all unrelated to me)
https://pulpito.ceph.com/ksirivad-2021-09-21_06:19:27-rados-wip-mgr-progress-global-efficiency-distro-basic-smithi/

@kamoltat
mgr/progress: optimize global recovery module
fa92db1 
Instead of fetching `pg_stats` from the python
part of manager module, we filter out the pgs
that are in active + clean state in ActivePyModules.cc
then parse these pgs along with `reported_epoch` and
the `total_num_pgs` of the clusters to global recovery
module.

Signed-off-by: Kamoltat <ksirivad@redhat.com>
@kamoltat kamoltat force-pushed the wip-mgr-progress-global-efficiency branch from 0eb3d21 to fa92db1 Compare September 22, 2021 19:18
@kamoltat kamoltat merged commit e3bbfcf into ceph:master Sep 23, 2021
@kamoltat kamoltat mentioned this pull request Sep 29, 2021
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tchaikov tchaikov tchaikov left review comments

@jdurgin jdurgin jdurgin approved these changes

@neha-ojha neha-ojha Awaiting requested review from neha-ojha

Assignees
No one assigned
Labels
core mgr pybind
Projects
None yet
Milestone
No milestone
3 participants
@kamoltat @tchaikov @jdurgin