cmake: WITH_SPDK=ON by default #18944

tchaikov · 2017-11-15T13:50:38Z

No description provided.

tchaikov · 2017-11-15T13:54:58Z

debian/control

@@ -29,6 +29,7 @@ Build-Depends: bc,
               libbabeltrace-ctf-dev,
               libbabeltrace-dev,
               libblkid-dev (>= 2.17),
+               libcunit1-dev,


@liupan1111 this what i mean by debian/control.

tchaikov · 2017-11-15T13:59:36Z

ceph.spec.in

@@ -183,6 +183,7 @@ BuildRequires:	keyutils-libs-devel
 BuildRequires:	libibverbs-devel
 BuildRequires:  openldap-devel
 BuildRequires:  openssl-devel
+BuildRequires:  CUnit-devel


@smithfarm does this work for SLE? or do you want to WITH_SPDK=ON for the build also?

tchaikov · 2017-11-15T14:02:42Z

cmake/modules/BuildDPDK.cmake

+    set(machine "armv8a")
+  elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "(powerpc|ppc)64")
+    set(arch "ppc_64")
+    set(machine "power8")


@smithfarm do you think we should enable DPDK on "power8" also? as http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html claims that DPDK supports power8 as well.

tchaikov · 2017-11-15T16:25:53Z

cmake/modules/BuildDPDK.cmake

+  endif()
+
+  if(WITH_KERNEL_DIR)
+    if(NOT IS_DIRECTORY ${WITH_KERNEL_DIR})


@liupan1111 this is my biggest concern. the target machine's kernel is not always the same as the build host. even if we managed to fulfill the build-requirement by installing the package of linux-headers-`uname -r` , there is good chance that the built DPDK kernel module does not work for the deployed machine.

or we don't need to build the kernel module at all? if yes, how?

I'm not sure I understand the kernel version dependency. Does the build have to be built against a specific kernel? I.e., if we build on the centos 7.3 kernel will the built resuilt break on 7.4? Or a build on the 16.04 kernel break on a later HWE kernel?

As long as those scenerios behave okay, I think it's fine to continue. If there is a hard kernel dependency, then I'm not sure what we should do.. that would suck. :(

For SLE, I'm pretty sure kernel module code needs to be integrated into, built as part of, and shipped in the kernel package itself. That may be an oversimplification, but certainly I wouldn't expect to be building kernel module packages via ceph.spec.

@ddiss Can you vet my comment, above?

yeah, to my best knowledge, kernel modules are not portable across different versions. anyway, i am disabling the kernel modules in DPDK.

@liupan1111 could you help test the build w/o the kernel modules? thanks

smithfarm · 2017-11-15T16:55:17Z

@tchaikov As usual, I'm running behind. . . I haven't tried to build master on SLE lately. I will try building this branch and let you know the result.

tchaikov · 2017-11-15T18:50:03Z

cmake/modules/patch-dpdk-conf.sh

+setconf CONFIG_RTE_KNI_KMOD n
+setconf CONFIG_RTE_KNI_PREEMPT_DEFAULT n
+
+setconf CONFIG_RTE_APP_EVENTDEV n


i know, this option is not in the used DPDK yet.

tchaikov · 2017-11-16T05:21:28Z

cmake/modules/patch-dpdk-conf.sh

+machine=$1
+shift
+
+setconf CONFIG_RTE_MACHINE "${machine}"


@yuyuyu101 do you think we can have a functional SPDK backend without all these kernel modules?

tchaikov · 2017-11-17T04:39:52Z

@smithfarm shaman is happy, see https://shaman.ceph.com/builds/ceph/wip-18944-kefu/bcc3cb0ba31d40a500391819ae9933cc84c204f9/. could you give the branch a spin?

tchaikov · 2017-11-17T07:20:06Z

CMakeLists.txt

@@ -348,6 +348,10 @@ endif()

 option(WITH_DPDK "Enable DPDK messaging" OFF)
 if(WITH_DPDK)
+  if(NOT USE_CRYPTOPP)
+    message(FATAL_ERROR "CRYPTOPP must be supported when enable DPDK.")


the the incompatible option settings at the toplevel cmake script.

tchaikov · 2017-11-17T07:20:59Z

src/CMakeLists.txt

@@ -687,6 +657,32 @@ if(HAVE_ARMV8_CRC)
  list(APPEND ceph_common_deps common_crc_aarch64)
 endif(HAVE_ARMV8_CRC)

+if(WITH_DPDK)


so we can have the WITH_DPDK support in a single place.

tchaikov · 2017-11-17T07:22:36Z

src/CMakeLists.txt

+  target_link_libraries(common_async_dpdk
+    ${DPDK_LIBRARIES})
+  set_target_properties(common_async_dpdk PROPERTIES
+    COMPILE_FLAGS "-march=native -I${DPDK_INCLUDE_DIR}")


so we can avoid adding "-march=native" globally. this will be a show stopper for downstream packagers/maintainers, if we want to enable DPDK by default.

Signed-off-by: Kefu Chai <kchai@redhat.com> Signed-off-by: wanjun.lp <wanjun.lp@alibaba-inc.com> Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

Signed-off-by: wanjun.lp <wanjun.lp@alibaba-inc.com> Signed-off-by: Kefu Chai <kchai@redhat.com>

liu-chunmei · 2017-11-17T22:41:16Z

@tchaikov does your PR support the following 3 cases:

only enable WITH_DPDK = ON
only enable WITH_SPDK = ON
enable both WITH_DPDK =ON and WITH_SPDK =ON
Is there build error for these 3 scenarios? and for these 3 scenarios, can the ceph system start up?
if yes, I think we can merge the patch first. Then for the functional issue we can fix later.

liu-chunmei · 2017-11-17T22:43:26Z

@tchaikov Maybe you need squash the several commits into one before merge it.

* move the check of `USE_CRYPTOPP` to $top_srcdir/CMakeLists.txt * remove reference of DPDK_LIBRARY, it's defined nowhere * move the dpdk code to a single place Signed-off-by: Kefu Chai <kchai@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

see also http://pkgs.fedoraproject.org/cgit/rpms/dpdk.git/tree/dpdk.spec Signed-off-by: Kefu Chai <kchai@redhat.com>

* also refactor it a little bit Signed-off-by: Kefu Chai <kchai@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

Signed-off-by: Liu-Chunmei <chunmei.liu@intel.com>

tchaikov · 2017-11-18T02:56:10Z

@liu-chunmei i squashed the two DPDK cleanup changes into a single one. but i'd like to keep other changes separated commits. as in this way, smaller changes will help the posterity to understand changes.

and all combinations of {WITH_DPDK,WITH_SPDK}={ON/OFF} build, and are tested using rados bench. WITH_SPDK=ON WITH_DPDK=OFF has been passed rados suite at http://pulpito.ceph.com/kchai-2017-11-17_09:57:45-rados-wip-kefu-testing-2017-11-17-1613-distro-basic-smithi/, the failures are not related.

tchaikov added build/ops feature labels Nov 15, 2017

tchaikov requested a review from liupan1111 November 15, 2017 13:50

tchaikov mentioned this pull request Nov 15, 2017

cmake: WITH_SPDK=ON by default #16997

Closed

tchaikov requested a review from smithfarm November 15, 2017 13:53

tchaikov commented Nov 15, 2017

View reviewed changes

tchaikov force-pushed the wip-dpdk-spdk branch 2 times, most recently from b13d8a7 to ab4bf0c Compare November 15, 2017 15:35

tchaikov commented Nov 15, 2017

View reviewed changes

tchaikov force-pushed the wip-dpdk-spdk branch 7 times, most recently from 2aa0b61 to 2415725 Compare November 15, 2017 18:46

tchaikov commented Nov 15, 2017

View reviewed changes

tchaikov added the needs-qa label Nov 16, 2017

tchaikov commented Nov 16, 2017

View reviewed changes

tchaikov added the wip-kefu-testing label Nov 16, 2017

tchaikov force-pushed the wip-dpdk-spdk branch 7 times, most recently from 2fd5c65 to 88a1081 Compare November 16, 2017 13:35

tchaikov force-pushed the wip-dpdk-spdk branch 5 times, most recently from 97e824d to bcc3cb0 Compare November 17, 2017 03:39

tchaikov force-pushed the wip-dpdk-spdk branch 2 times, most recently from 4b11a59 to 42f6fc2 Compare November 17, 2017 07:13

tchaikov mentioned this pull request Nov 17, 2017

msg/async/dpdk: rebase to spdk/dpdk #18927

Closed

tchaikov requested a review from yuyuyu101 November 17, 2017 07:18

tchaikov force-pushed the wip-dpdk-spdk branch from 42f6fc2 to c1dfc49 Compare November 17, 2017 07:24

tchaikov commented Nov 17, 2017

View reviewed changes

cmake: build spdk/dpdk in cmake modules

01a9f17

Signed-off-by: Kefu Chai <kchai@redhat.com> Signed-off-by: wanjun.lp <wanjun.lp@alibaba-inc.com> Signed-off-by: Ziye Yang <optimistyzy@gmail.com>

tchaikov force-pushed the wip-dpdk-spdk branch from c1dfc49 to 34bb7d3 Compare November 17, 2017 07:38

tchaikov added 2 commits November 17, 2017 15:40

rpm,deb: add cunit1 as build dependency

22d8dde

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: WITH_SPDK=ON by default

df446f0

Signed-off-by: wanjun.lp <wanjun.lp@alibaba-inc.com> Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov force-pushed the wip-dpdk-spdk branch from 34bb7d3 to 0e5e3af Compare November 17, 2017 07:40

tchaikov and others added 6 commits November 18, 2017 10:43

cmake: refactor dpdk related code

dbe04b4

* move the check of `USE_CRYPTOPP` to $top_srcdir/CMakeLists.txt * remove reference of DPDK_LIBRARY, it's defined nowhere * move the dpdk code to a single place Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated spdk linkage

79db2c4

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: BuildDPDK: do not build kernel modules

9964ac9

see also http://pkgs.fedoraproject.org/cgit/rpms/dpdk.git/tree/dpdk.spec Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add rte_mempool_{ring,stack} to Finddpdk.cmake

e8add07

* also refactor it a little bit Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use BuildDPDK if libdpdk not found

a69bd61

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: rebase msg/async/dpdk to spdk/dpdk

0610332

Signed-off-by: Liu-Chunmei <chunmei.liu@intel.com>

tchaikov force-pushed the wip-dpdk-spdk branch from 0e5e3af to 0610332 Compare November 18, 2017 02:48

tchaikov merged commit b9c822a into ceph:master Nov 18, 2017

tchaikov deleted the wip-dpdk-spdk branch November 18, 2017 05:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmake: WITH_SPDK=ON by default #18944

cmake: WITH_SPDK=ON by default #18944

tchaikov commented Nov 15, 2017

tchaikov Nov 15, 2017

tchaikov Nov 15, 2017

tchaikov Nov 15, 2017

tchaikov Nov 15, 2017 •

edited

liewegas Nov 15, 2017

smithfarm Nov 15, 2017

smithfarm Nov 15, 2017

tchaikov Nov 15, 2017

smithfarm commented Nov 15, 2017

tchaikov Nov 15, 2017

tchaikov Nov 16, 2017

tchaikov commented Nov 17, 2017 •

edited

tchaikov Nov 17, 2017

tchaikov Nov 17, 2017

tchaikov Nov 17, 2017 •

edited

liu-chunmei commented Nov 17, 2017

liu-chunmei commented Nov 17, 2017

tchaikov commented Nov 18, 2017 •

edited

cmake: WITH_SPDK=ON by default #18944

cmake: WITH_SPDK=ON by default #18944

Conversation

tchaikov commented Nov 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tchaikov Nov 15, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smithfarm commented Nov 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tchaikov commented Nov 17, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tchaikov Nov 17, 2017 • edited

Choose a reason for hiding this comment

liu-chunmei commented Nov 17, 2017

liu-chunmei commented Nov 17, 2017

tchaikov commented Nov 18, 2017 • edited

tchaikov Nov 15, 2017 •

edited

tchaikov commented Nov 17, 2017 •

edited

tchaikov Nov 17, 2017 •

edited

tchaikov commented Nov 18, 2017 •

edited