mon/OSDMonitor, osd: Add warning on filestore deprecation and force use of wpq scheduler for filestore OSDs #39440

pdvian · 2021-02-12T13:49:33Z

This PR addresses 2 changes :

Notify end user on filestore deprecation through ceph health detail warning.
The 'mclock_scheduler' is not supported for filestore. The default scheduler type is 'wpq' for filestore OSDs and will enforce scheduler type to 'wpq' if user tries to override it.

Fixes: https://tracker.ceph.com/issues/49275

Signed-off-by: Prashant D pdhange@redhat.com

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

src/mon/OSDMonitor.cc

vumrao · 2021-06-02T05:17:48Z

ping @pdvian

neha-ojha · 2021-10-19T17:00:14Z

@pdvian I think we could incorporate FileStore deprecation message in Quincy in this PR based on https://www.spinics.net/lists/ceph-users/msg66668.html

pdvian · 2021-11-12T13:59:12Z

Moving discussion about commit message here.

Prashant, Thu 11:52 PM
@Neha About filestore deprecation message in PR, does below message in commit looks good to you ?

"Filestore will be deprecated in the next release (Quincy)
considering bluestore is the default objectstore now and
more widely used since quite some time."

Also should we include this message in PendingRelease as well ?

I am doing testing on latest changes according to David's inputs and will be pushing changes to PR#39440 by tomorrow.

Brad, Yesterday 8:11 AM
@prashant maybe change 'since quite some time' to 'for quite some time'?

Neha, Yesterday 8:12 AM
Hey @prashant @brad Hubbard!

Neha, Yesterday 8:15 AM
Let's move this discussion to the PR, just for everyone's record.

Something like "Filestore will be deprecated in Quincy,
considering BlueStore has been the default objectstore for quite some time" makes sense

and yes to release notes!

pdvian · 2021-11-13T02:40:35Z

@neha-ojha @badone I did not consider the scenario where filestore OSD(s) getting removed from the cluster and those filestore OSDs should not be seen in the health warn. Re-pushing the changes for handling this scenario.

badone

This looks OK to me mate.

neha-ojha · 2021-12-01T16:27:07Z

@pdvian we need to do some more work to silence the health warnings in tests, http://pulpito.front.sepia.ceph.com/yuriw-2021-12-01_02:24:37-rados-wip-yuri5-testing-2021-11-30-1628-distro-default-smithi/

pdvian · 2021-12-16T14:24:41Z

Does this looks good from ceph status and health detail specific ??

ceph status :
health: HEALTH_WARN
9 osd(s) are on filestore; Filestore has been deprecated, the 'osd_op_queue' is enforced to 'wpq' for filestore OSDs.

ceph health detail :
[WRN] OSD_FILESTORE: 9 osd(s) are on filestore; Filestore has been deprecated, the 'osd_op_queue' is enforced to 'wpq' for filestore OSDs.
Filestore OSDs [osd.0,osd.1,osd.2,osd.3,osd.4,osd.5,osd.6,osd.7,osd.8]

github-actions · 2021-12-16T15:02:41Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

src/mon/OSDMonitor.cc

PendingReleaseNotes

src/mon/OSDMonitor.cc

neha-ojha · 2021-12-17T01:07:34Z

Does this looks good from ceph status and health detail specific ??

@pdvian @sseshasa How about?

ceph status : health: HEALTH_WARN 9 osd(s) are on filestore; Filestore has been deprecated, the 'osd_op_queue' is enforced to 'wpq' for filestore OSDs.

ceph status : health: HEALTH_WARN 9 osd(s) are running Filestore [Deprecated]

ceph health detail : [WRN] OSD_FILESTORE: 9 osd(s) are on filestore; Filestore has been deprecated, the 'osd_op_queue' is enforced to 'wpq' for filestore OSDs. Filestore OSDs [osd.0,osd.1,osd.2,osd.3,osd.4,osd.5,osd.6,osd.7,osd.8]

ceph health detail : [WRN] OSD_FILESTORE: 9 osd(s) are running Filestore, which has been deprecated and not been optimized for QoS (Filestore OSDs will use osd_op_queue=wpq and cannot be used with mclock_scheduler). List Filestore OSDs using <command>.

I think we should avoid listing all the osds in ceph health detail, given that for a large Filestore cluster, this list will be huge. Instead we should provide a way for users to list all the Filestore OSDs using existing commands like ceph osd metadata or ceph report. @badone had a good idea of providing a jquery command that the parses the output of such commands.

src/mon/OSDMonitor.cc

pdvian · 2021-12-21T01:43:31Z

Does this looks good from ceph status and health detail specific ??

@pdvian @sseshasa How about?

ceph status : health: HEALTH_WARN 9 osd(s) are on filestore; Filestore has been deprecated, the 'osd_op_queue' is enforced to 'wpq' for filestore OSDs.
ceph status : health: HEALTH_WARN 9 osd(s) are running Filestore [Deprecated]
ceph health detail : [WRN] OSD_FILESTORE: 9 osd(s) are on filestore; Filestore has been deprecated, the 'osd_op_queue' is enforced to 'wpq' for filestore OSDs. Filestore OSDs [osd.0,osd.1,osd.2,osd.3,osd.4,osd.5,osd.6,osd.7,osd.8]
ceph health detail : [WRN] OSD_FILESTORE: 9 osd(s) are running Filestore, which has been deprecated and not been optimized for QoS (Filestore OSDs will use osd_op_queue=wpq and cannot be used with mclock_scheduler). List Filestore OSDs using <command>.
I think we should avoid listing all the osds in ceph health detail, given that for a large Filestore cluster, this list will be huge. Instead we should provide a way for users to list all the Filestore OSDs using existing commands like ceph osd metadata or ceph report. @badone had a good idea of providing a jquery command that the parses the output of such commands.

@neha-ojha We can use either of below commands to list filestore OSDs :

list with id and osd_objectstore
ceph report | jq -c '."osd_metadata" | .[] | {id, osd_objectstore} | select(.osd_objectstore | contains("filestore"))
list with id only
ceph report | jq -c '."osd_metadata" | .[] | {id, osd_objectstore} | select(.osd_objectstore | contains("filestore"))|{id}

The health detail will look like :
HEALTH_WARN 3 osd(s) are running Filestore [Deprecated] [WRN] OSD_FILESTORE: 3 osd(s) are running Filestore [Deprecated] 3 osd(s) are running Filestore, which has been deprecated and not been optimized for QoS (Filestore OSDs will use 'osd_op_queue = wpq' and cannot be used with mclock_scheduler). List Filestore OSDs using 'ceph report | jq -c '."osd_metadata" | .[] | {id, osd_objectstore} | select(.osd_objectstore | contains("filestore"))''

@badone Does this looks good to you ?

src/mon/OSDMonitor.cc

Filestore will be deprecated in Quincy, considering that BlueStore has been the default objectstore for quite some time. Fixes: https://tracker.ceph.com/issues/49275 Signed-off-by: Prashant D <pdhange@redhat.com>

…OSDs The 'mclock_scheduler' is not supported for filestore OSDs. Enforce the usage of 'wpq' scheduler for such OSDs to avoid issues. Also, in this scenario, the override of various config settings for the 'mclock_scheduler' are not performed. Signed-off-by: Sridhar Seshasayee <sseshasa@redhat.com>

Signed-off-by: Prashant D <pdhange@redhat.com>

neha-ojha · 2022-01-05T18:11:34Z

jenkins test api

neha-ojha · 2022-01-05T18:27:23Z

@pdvian filestore is used by a lot of test suites, we should probably have @yuriw run this PR through sanity testing of all suites, including upgrades

sseshasa · 2022-01-11T10:58:50Z

Teuthology Test report

RADOS Suite Runs

Unrelated Failures (Across both runs)

Upgrade Suite Run

http://pulpito.front.sepia.ceph.com/yuriw-2022-01-06_15:53:32-upgrade-wip-yuri6-testing-2022-01-05-1255-distro-default-smithi/

Unrelated Failures

pdvian added core mon labels Feb 12, 2021

pdvian requested review from neha-ojha and jdurgin February 18, 2021 11:23

neha-ojha reviewed Feb 18, 2021

View reviewed changes

src/mon/OSDMonitor.cc Outdated Show resolved Hide resolved

pdvian force-pushed the wip-warn-filestore-osds branch from eda3bb4 to 0164250 Compare March 2, 2021 10:13

neha-ojha reviewed Mar 4, 2021

View reviewed changes

src/mon/OSDMonitor.cc Outdated Show resolved Hide resolved

src/mon/OSDMonitor.cc Show resolved Hide resolved

neha-ojha mentioned this pull request Oct 21, 2021

build: Remove LevelDB support #43612

Merged

pdvian force-pushed the wip-warn-filestore-osds branch from 0164250 to a64c2ec Compare November 12, 2021 14:58

github-actions bot added common documentation labels Nov 12, 2021

pdvian requested a review from neha-ojha November 12, 2021 15:01

vumrao requested a review from badone November 12, 2021 17:39

pdvian force-pushed the wip-warn-filestore-osds branch from a64c2ec to f52cf06 Compare November 13, 2021 02:40

pdvian added the wip-pdhange-testing label Nov 17, 2021

badone approved these changes Nov 22, 2021

View reviewed changes

badone added the needs-qa label Nov 22, 2021

yuriw added wip-yuri6-testing wip-yuri5-testing and removed wip-yuri6-testing labels Dec 1, 2021

yuriw removed the wip-yuri5-testing label Dec 1, 2021

pdvian force-pushed the wip-warn-filestore-osds branch from f52cf06 to 661c9f5 Compare December 14, 2021 06:12

pdvian changed the title ~~mon/OSDMonitor: Raise health warning for filestore osds~~ mon/OSDMonitor, osd: Add warning on filestore deprecation and force use of wpq scheduler for filestore OSDs Dec 14, 2021

pdvian force-pushed the wip-warn-filestore-osds branch from 661c9f5 to 4f8dfb6 Compare December 14, 2021 06:29

github-actions bot added the tests label Dec 14, 2021

pdvian force-pushed the wip-warn-filestore-osds branch from 35ef208 to 579998b Compare December 16, 2021 14:17

pdvian requested review from neha-ojha and sseshasa December 16, 2021 14:21

github-actions bot added the needs-rebase label Dec 16, 2021

sseshasa reviewed Dec 16, 2021

View reviewed changes

src/mon/OSDMonitor.cc Outdated Show resolved Hide resolved

sseshasa reviewed Dec 16, 2021

View reviewed changes

PendingReleaseNotes Outdated Show resolved Hide resolved

sseshasa reviewed Dec 16, 2021

View reviewed changes

src/mon/OSDMonitor.cc Outdated Show resolved Hide resolved

wjwithagen reviewed Dec 19, 2021

View reviewed changes

src/mon/OSDMonitor.cc Outdated Show resolved Hide resolved

pdvian force-pushed the wip-warn-filestore-osds branch from 579998b to 188a12e Compare December 21, 2021 01:26

pdvian requested review from sseshasa and wjwithagen December 21, 2021 01:29

github-actions bot removed the needs-rebase label Dec 21, 2021

pdvian requested a review from badone December 21, 2021 01:44

neha-ojha reviewed Dec 21, 2021

View reviewed changes

src/mon/OSDMonitor.cc Show resolved Hide resolved

pdvian force-pushed the wip-warn-filestore-osds branch from 188a12e to 5ed5f94 Compare December 21, 2021 20:03

pdvian requested a review from neha-ojha December 21, 2021 20:04

Prashant D and others added 3 commits January 5, 2022 10:08

mon/OSDMonitor: Raise health warning for filestore osds

39f5a61

Filestore will be deprecated in Quincy, considering that BlueStore has been the default objectstore for quite some time. Fixes: https://tracker.ceph.com/issues/49275 Signed-off-by: Prashant D <pdhange@redhat.com>

doc/rados/operations: Document health check OSD_FILESTORE

658dbf4

Signed-off-by: Prashant D <pdhange@redhat.com>

pdvian force-pushed the wip-warn-filestore-osds branch from 5ed5f94 to 658dbf4 Compare January 5, 2022 12:54

neha-ojha approved these changes Jan 5, 2022

View reviewed changes

yuriw added the wip-yuri6-testing label Jan 5, 2022

yuriw merged commit a8bb49d into ceph:master Jan 12, 2022
8 checks passed

mon/OSDMonitor, osd: Add warning on filestore deprecation and force use of wpq scheduler for filestore OSDs #39440

mon/OSDMonitor, osd: Add warning on filestore deprecation and force use of wpq scheduler for filestore OSDs #39440

pdvian commented Feb 12, 2021 •

edited

vumrao commented Jun 2, 2021

neha-ojha commented Oct 19, 2021

pdvian commented Nov 12, 2021 •

edited

pdvian commented Nov 13, 2021 •

edited

badone left a comment

neha-ojha commented Dec 1, 2021

pdvian commented Dec 16, 2021 •

edited

github-actions bot commented Dec 16, 2021

neha-ojha commented Dec 17, 2021

pdvian commented Dec 21, 2021

neha-ojha commented Jan 5, 2022

neha-ojha commented Jan 5, 2022

sseshasa commented Jan 11, 2022

mon/OSDMonitor, osd: Add warning on filestore deprecation and force use of wpq scheduler for filestore OSDs #39440

mon/OSDMonitor, osd: Add warning on filestore deprecation and force use of wpq scheduler for filestore OSDs #39440

Conversation

pdvian commented Feb 12, 2021 • edited

Checklist

vumrao commented Jun 2, 2021

neha-ojha commented Oct 19, 2021

pdvian commented Nov 12, 2021 • edited

pdvian commented Nov 13, 2021 • edited

badone left a comment

Choose a reason for hiding this comment

neha-ojha commented Dec 1, 2021

pdvian commented Dec 16, 2021 • edited

github-actions bot commented Dec 16, 2021

neha-ojha commented Dec 17, 2021

pdvian commented Dec 21, 2021

neha-ojha commented Jan 5, 2022

neha-ojha commented Jan 5, 2022

sseshasa commented Jan 11, 2022

Teuthology Test report

RADOS Suite Runs

Upgrade Suite Run

pdvian commented Feb 12, 2021 •

edited

pdvian commented Nov 12, 2021 •

edited

pdvian commented Nov 13, 2021 •

edited

pdvian commented Dec 16, 2021 •

edited