cmake: enable ndctl when building PMDK for WITH_BLUESTORE_PMEM #46260

CongMinYin · 2022-05-13T13:24:04Z

In order to support the character device of pmem usage in bluestore via
libpmem built by Ceph itself, we need to enable daxctl and ndctl
dependency. And this PR is used to add ndctl and daxctl dependency
when WITH_BLUSE_STORE_PMEM is ON.

add the installation of ndctl and daxctl, then find them.
ndctl library requires >63. daxctl library requires >63.

Because after find_ndctl and find_daxctl are added, the dependency packages
required by WITH_BLUESTORE_PMEM and WITH_RBD_RWL become more different.
The DWITH_RBD_RWL option is turned on by default in some cases (see #39049),
while WITH_BLUESTORE_PMEM is usually turned off by default. These two should not be put together.
So separated these two. libpmem has no version required. libpmemobj
required version >=1.8.

the installation of ndctl-devel and daxctl-devel in ceph.spec.in has not been verified.

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

CongMinYin · 2022-05-13T13:29:21Z

Hi @tchaikov , this pr passed ./do_cmake.sh -DWITH_BLUESTORE=ON -DWITH_BLUESTORE_PMEM=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_CCACHE=ON, but failed after: cd build && ninja -j 96, report:
ninja: error: 'ndctl::ndctl-NOTFOUND', needed by 'bin/ceph-osd', missing and no known rule to make it.
Do you have any ideas?

part of output during do_cmake:
-- Found ndctl: /usr/lib/x86_64-linux-gnu/libndctl.so (Required is at least version "63")

install-deps.sh

src/CMakeLists.txt

cmake/modules/Findndctl.cmake

tchaikov · 2022-05-13T13:55:57Z

my guess is that WITH_PMEM=true never gets set in https://github.com/ceph/ceph-build/blob/master/ceph-pull-requests/build/build

CongMinYin · 2022-05-16T02:32:34Z

Hi @tchaikov , this pr passed ./do_cmake.sh -DWITH_BLUESTORE=ON -DWITH_BLUESTORE_PMEM=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DWITH_CCACHE=ON, but failed after: cd build && ninja -j 96, report: ninja: error: 'ndctl::ndctl-NOTFOUND', needed by 'bin/ceph-osd', missing and no known rule to make it. Do you have any ideas?

part of output during do_cmake: -- Found ndctl: /usr/lib/x86_64-linux-gnu/libndctl.so (Required is at least version "63")

the detail output: https://gist.github.com/CongMinYin/db3c96be013f63df139f9674452782f9

cmake/modules/Findndctl.cmake

CongMinYin · 2022-05-23T09:14:06Z

my guess is that WITH_PMEM=true never gets set in https://github.com/ceph/ceph-build/blob/master/ceph-pull-requests/build/build

take a look this pr: ceph/ceph-build#2011

CongMinYin · 2022-05-23T09:20:07Z

Hi @tchaikov , thanks your guide. Now passed the local test. Please review.

CongMinYin · 2022-05-30T02:48:21Z

ping @tchaikov

install-deps.sh

github-actions · 2022-06-02T17:51:36Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

CongMinYin · 2022-06-05T15:08:10Z

rebased, ping @tchaikov

CongMinYin · 2022-06-05T16:19:13Z

I rebased to newest upstream/main, but lable needs-rebase still exists.

dismissed for now.

tchaikov · 2022-06-06T08:22:56Z

-- mgr module disabled for 3.8.10: diskprediction_local
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find ndctl (missing: ndctl_LIBRARY ndctl_INCLUDE_DIR) (Required
  is at least version "63")

CongMinYin · 2022-06-06T14:27:19Z

-- mgr module disabled for 3.8.10: diskprediction_local
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find ndctl (missing: ndctl_LIBRARY ndctl_INCLUDE_DIR) (Required
  is at least version "63")

Because WITH_PMEM is not enabled in the ceph-build project. ndctl and daxctl is not installed in the test environment. so ndctl can't be found, then resulte in an error of make check. Please take a look at ceph/ceph-build#2011.

github-actions · 2022-06-28T13:58:12Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

CongMinYin · 2022-06-29T05:52:44Z

Hi @idryomov , could you please also have a look at this PR? I want to separate the dependencies of BLUESTORE_PMEM and RWL. Another reason is that the DWITH_RBD_RWL option is turned on by default in some cases (see #39049), while WITH_BLUESTORE_PMEM is usually turned off by default. These two should not be put together.

CongMinYin · 2022-07-03T08:56:52Z

ping @tchaikov

src/CMakeLists.txt

idryomov

Also, please fold "cmake: add findndctl function" and "cmake: add finddaxctl function" into a single commit as they are really the same.

src/CMakeLists.txt

cmake/modules/Buildpmdk.cmake

src/CMakeLists.txt

cmake/modules/Findndctl.cmake

cmake/modules/Finddaxctl.cmake

CongMinYin · 2022-07-10T06:23:13Z

jenkins test make check

tchaikov · 2022-07-10T10:46:18Z

being built at https://shaman.ceph.com/builds/ceph/pr-46260-kefu/15a806c002be3c496a32bfe91312343a81e6d8dc/

tchaikov · 2022-07-10T14:41:23Z

hi @idryomov , could you take another look? i think CongMin has addressed all of the comments.

idryomov

Ack from the RBD perspective

ceph.spec.in

In order to support the character device of pmem usage in bluestore via libpmem built by Ceph itself, we need to enable daxctl and ndctl dependency. add the installation of ndctl and find it. the version of ndctl and daxctl library requires >63. "apt-get install" meet the version under ubuntu focal. the installation of ndctl-devel in ceph.spec.in has not been verified. Signed-off-by: Yin Congmin <congmin.yin@intel.com>

In order to enable the pmem character device, add ndctl=y to the parameter of compiling the pmdk library when WITH_BLUESTORE_PMEM is ON. Because after find_ndctl and find_daxctl are added, the dependency packages required by WITH_BLUESTORE_PMEM and WITH_RBD_RWL become more different. So separated these two. libpmem has no version required. libpmemobj required version >=1.8. Signed-off-by: Yin Congmin <congmin.yin@intel.com>

CongMinYin · 2022-07-12T02:43:33Z

Hi @tchaikov @idryomov , added the version requirement in ceph.spec.in, thanks. please check.

idryomov · 2022-07-12T07:22:41Z

LGTM!

idryomov · 2022-07-12T07:22:56Z

jenkins test windows

idryomov · 2022-07-12T08:08:15Z

@tchaikov I'm going to run this through the part of the rbd suite that exercises WITH_RBD_RWL. If additional testing is desired, please re-add needs-qa label.

tchaikov · 2022-07-12T08:45:37Z

@tchaikov I'm going to run this through the part of the rbd suite that exercises WITH_RBD_RWL. If additional testing is desired, please re-add needs-qa label.

hi Ilya, that should suffice. thank you!

idryomov · 2022-07-14T13:16:30Z

https://pulpito.ceph.com/dis-2022-07-14_11:30:45-rbd:pwl-cache-wip-dis-testing-distro-default-smithi/

github-actions bot added the build/ops label May 13, 2022

CongMinYin changed the title ~~[cmake] add findndctl.cmake~~ [WIP] cmake: add findndctl.cmake May 13, 2022

CongMinYin commented May 13, 2022

View reviewed changes

install-deps.sh Outdated Show resolved Hide resolved

CongMinYin commented May 13, 2022

View reviewed changes

src/CMakeLists.txt Outdated Show resolved Hide resolved

cmake/modules/Findndctl.cmake Outdated Show resolved Hide resolved

tchaikov reviewed May 19, 2022

View reviewed changes

cmake/modules/Findndctl.cmake Show resolved Hide resolved

CongMinYin force-pushed the wip-enable-ndctl branch from 383da22 to 4fe7386 Compare May 23, 2022 09:01

CongMinYin changed the title ~~[WIP] cmake: add findndctl.cmake~~ cmake: add findndctl.cmake May 23, 2022

CongMinYin force-pushed the wip-enable-ndctl branch from 4fe7386 to 5180e83 Compare May 25, 2022 08:32

djgalloway changed the base branch from master to main May 25, 2022 19:59

tchaikov reviewed May 30, 2022

View reviewed changes

install-deps.sh Outdated Show resolved Hide resolved

CongMinYin force-pushed the wip-enable-ndctl branch from 5180e83 to 78502f3 Compare May 30, 2022 08:11

CongMinYin commented May 30, 2022

View reviewed changes

install-deps.sh Outdated Show resolved Hide resolved

CongMinYin force-pushed the wip-enable-ndctl branch 2 times, most recently from 9dfe5f6 to a1693db Compare May 31, 2022 04:21

github-actions bot added the needs-rebase label Jun 2, 2022

CongMinYin force-pushed the wip-enable-ndctl branch from a1693db to 080103b Compare June 5, 2022 15:05

github-actions bot removed the needs-rebase label Jun 5, 2022

tchaikov previously approved these changes Jun 6, 2022

View reviewed changes

CongMinYin force-pushed the wip-enable-ndctl branch from d29e407 to 646a1ef Compare June 29, 2022 04:35

github-actions bot removed the needs-rebase label Jun 29, 2022

tchaikov requested changes Jul 3, 2022

View reviewed changes

src/CMakeLists.txt Show resolved Hide resolved

src/CMakeLists.txt Outdated Show resolved Hide resolved

CongMinYin force-pushed the wip-enable-ndctl branch from 646a1ef to 268563e Compare July 3, 2022 12:12

idryomov reviewed Jul 4, 2022

View reviewed changes

CongMinYin force-pushed the wip-enable-ndctl branch from 268563e to ca044df Compare July 5, 2022 08:34

tchaikov requested changes Jul 6, 2022

View reviewed changes

cmake/modules/Finddaxctl.cmake Outdated Show resolved Hide resolved

CongMinYin force-pushed the wip-enable-ndctl branch from ca044df to 15a806c Compare July 6, 2022 15:03

tchaikov approved these changes Jul 10, 2022

View reviewed changes

tchaikov added the needs-qa label Jul 10, 2022

idryomov approved these changes Jul 11, 2022

View reviewed changes

ceph.spec.in Outdated Show resolved Hide resolved

CongMinYin added 2 commits July 12, 2022 10:32

CongMinYin force-pushed the wip-enable-ndctl branch from 15a806c to eb94cda Compare July 12, 2022 02:38

idryomov added wip-dis-testing and removed needs-qa labels Jul 12, 2022

idryomov changed the title ~~cmake: add findndctl.cmake~~ cmake: enable ndctl when building PMDK for WITH_BLUESTORE_PMEM Jul 14, 2022

idryomov merged commit 795daff into ceph:main Jul 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmake: enable ndctl when building PMDK for WITH_BLUESTORE_PMEM #46260

cmake: enable ndctl when building PMDK for WITH_BLUESTORE_PMEM #46260

CongMinYin commented May 13, 2022 •

edited

CongMinYin commented May 13, 2022 •

edited

tchaikov commented May 13, 2022

CongMinYin commented May 16, 2022

CongMinYin commented May 23, 2022

CongMinYin commented May 23, 2022

CongMinYin commented May 30, 2022

github-actions bot commented Jun 2, 2022

CongMinYin commented Jun 5, 2022

CongMinYin commented Jun 5, 2022 •

edited

tchaikov commented Jun 6, 2022

CongMinYin commented Jun 6, 2022

github-actions bot commented Jun 28, 2022

CongMinYin commented Jun 29, 2022

CongMinYin commented Jul 3, 2022

idryomov left a comment

CongMinYin commented Jul 10, 2022

tchaikov commented Jul 10, 2022

tchaikov commented Jul 10, 2022 •

edited

idryomov left a comment

CongMinYin commented Jul 12, 2022

idryomov commented Jul 12, 2022

idryomov commented Jul 12, 2022

idryomov commented Jul 12, 2022 •

edited

tchaikov commented Jul 12, 2022

idryomov commented Jul 14, 2022

cmake: enable ndctl when building PMDK for WITH_BLUESTORE_PMEM #46260

cmake: enable ndctl when building PMDK for WITH_BLUESTORE_PMEM #46260

Conversation

CongMinYin commented May 13, 2022 • edited

Contribution Guidelines

Checklist

CongMinYin commented May 13, 2022 • edited

tchaikov commented May 13, 2022

CongMinYin commented May 16, 2022

CongMinYin commented May 23, 2022

CongMinYin commented May 23, 2022

CongMinYin commented May 30, 2022

github-actions bot commented Jun 2, 2022

CongMinYin commented Jun 5, 2022

CongMinYin commented Jun 5, 2022 • edited

tchaikov commented Jun 6, 2022

CongMinYin commented Jun 6, 2022

github-actions bot commented Jun 28, 2022

CongMinYin commented Jun 29, 2022

CongMinYin commented Jul 3, 2022

idryomov left a comment

Choose a reason for hiding this comment

CongMinYin commented Jul 10, 2022

tchaikov commented Jul 10, 2022

tchaikov commented Jul 10, 2022 • edited

idryomov left a comment

Choose a reason for hiding this comment

CongMinYin commented Jul 12, 2022

idryomov commented Jul 12, 2022

idryomov commented Jul 12, 2022

idryomov commented Jul 12, 2022 • edited

tchaikov commented Jul 12, 2022

idryomov commented Jul 14, 2022

CongMinYin commented May 13, 2022 •

edited

CongMinYin commented May 13, 2022 •

edited

CongMinYin commented Jun 5, 2022 •

edited

tchaikov commented Jul 10, 2022 •

edited

idryomov commented Jul 12, 2022 •

edited