dmclock: initial commit of dmclock QoS library #14330

ivancich · 2017-04-04T21:25:02Z

This commit brings in the dmclock library, located in a separate repo, as a git subtree. It also contains a README.git-subtree to describe this technique and some of the pitfalls surrounding it. And it links the cmake files so the library and support can be built.

Make targets include "dmclock", the static library; "dmclock-tests", unit tests for the dmclock code and supporting data structures; and "dmclock-sims", simulations that can be used to test and understand the library.

ivancich · 2017-04-04T21:31:12Z

Hmm, the two commits created by the "git subtree pull ..." command are not signed, and there do not appear to be command line options to add a signature.

liewegas · 2017-04-04T21:42:37Z

This looks good to me... @jdurgin ?

jdurgin · 2017-04-04T23:27:18Z

Don't worry about the signed-off-by for the subtree commits.

The 'make check' error looks real though - it got a failure from not running the dmclock tests.

ivancich · 2017-04-05T12:40:31Z

I'll handle the make check issue.

liewegas · 2017-04-07T14:52:38Z

retest this please

tchaikov · 2017-04-07T15:34:29Z

src/CMakeLists.txt

+
+if(WITH_TESTS)
+install(PROGRAMS
+  ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/dmclock-tests


might want to include this in "ceph-test". see debian/ceph-test.install and ceph.spec.in (search for %files -n ceph-test)

liewegas · 2017-04-20T21:48:05Z

-- Set runtime path of "/tmp/buildd/ceph-12.0.1-1259-g49dfa5a/debian/tmp/usr/bin/ceph-mds" to "/usr/lib/ceph"
CMake Error at src/cmake_install.cmake:193 (file):
file INSTALL cannot find
"/tmp/buildd/ceph-12.0.1-1259-g49dfa5a/obj-x86_64-linux-gnu/bin/dmclock-tests".
Call Stack (most recent call first):
cmake_install.cmake:37 (include)

https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=xenial,DIST=xenial,MACHINE_SIZE=huge/2765//consoleFull

the pitfalls surrounding them. Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

git-subtree-dir: src/dmclock git-subtree-split: d6586d73679f4a1bdf335235d309e2352f0c76c6

dmclock tests in ceph-test for builds. Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

liewegas · 2017-05-02T01:29:37Z

http://pulpito.ceph.com/sage-2017-05-01_20:13:55-rados-wip-sage-testing---basic-smithi/

smithfarm · 2017-05-19T11:43:15Z

This breaks SUSE builds:

[  158s] CMake Error at src/dmclock/CMakeLists.txt:20 (find_package):
[  158s]   By not providing "Findgtest.cmake" in CMAKE_MODULE_PATH this project has
[  158s]   asked CMake to find a package configuration file provided by "gtest", but
[  158s]   CMake did not find one.
[  158s] 
[  158s]   Could not find a package configuration file provided by "gtest" with any of
[  158s]   the following names:
[  158s] 
[  158s]     gtestConfig.cmake
[  158s]     gtest-config.cmake
[  158s] 
[  158s]   Add the installation prefix of "gtest" to CMAKE_PREFIX_PATH or set
[  158s]   "gtest_DIR" to a directory containing one of the above files.  If "gtest"
[  158s]   provides a separate development package or SDK, be sure it has been
[  158s]   installed.
[  158s] 
[  158s] 
[  158s] -- Configuring incomplete, errors occurred!

The FindGTest.cmake module is part of the upstream cmake [1] (so it should be the same in all distros), and it is named "FindGTest.cmake", not "Findgtest.cmake". Fix the find_package() call introduced by ceph#14330 so the case matches. This fixes SUSE builds broken by that PR. [1] https://cmake.org/cmake/help/v3.0/module/FindGTest.html Signed-off-by: Nathan Cutler <ncutler@suse.com>

ivancich · 2017-05-22T20:22:34Z

Hi Nathan, Thanks for looking into that. When ceph is built, gtest and gtest_main are targets, and I thought that would have prevented the find_package of gtest. Would you mind looking into that?

Nonetheless, I'll fix the cases to match cmake's upstream. I'll likely include that in #14997 , which is currently under eval.

smithfarm · 2017-05-22T20:28:31Z

@ivancich in #14997 you are dropping (i.e. not packaging) the binaries dmclock-tests and dmclock-data-struct-tests, right? In that case, instead of if(WITH_TESTS) you could guard the build of these binaries with some other conditional that defaults to OFF?

I ask because WITH_TESTS=ON means "the binaries in ceph-test are to be built". It does not mean "ctest is to be run". . .

smithfarm · 2017-05-22T20:35:02Z

@ivancich See #10872 for how/why of WITH_TESTS

smithfarm · 2017-05-22T20:46:22Z

src/CMakeLists.txt

+    ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/dmclock-tests
+    ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/dmclock-data-struct-tests
+    DESTINATION bin)
+endif(WITH_TESTS)


@ivancich This is where the build of dmclock-tests and dmclock-data-struct-tests is made conditional upon WITH_TESTS.

If we could change the guard to some other conditional (and not package these two binaries), then the problem I'm having will be solved I think!

e.g. in CMakeLists.txt add a line that says set(WITH_DMCLOCK_TESTS "enable the build of dmclock-tests and dmclock-data-struct-tests binaries" OFF)

and then s/WITH_TESTS/WITH_DMCLOCK_TESTS/

Then somebody who wanted to build these binaries could pass -DWITH_DMCLOCK_TESTS=ON to cmake, but for purposes of building RPMs and debs we would use -DWITH_DMCLOCK_TESTS=OFF.

If you look at where this is heading (https://github.com/ivancich/ceph-fork/tree/wip-bring-in-dmclock-p2-gtest) all that is gone. dmclock tests won't even be built unless you do something like "make dmclock-tests". They're detached from ceph's tests. I think that'll give you what you want, no?

dmclock tests won't even be built unless you do something like "make dmclock-tests". They're detached from ceph's tests. I think that'll give you what you want, no?

Yes.

although this [1] appears to be building the dmclock tests

[1] https://github.com/ivancich/ceph-fork/blob/wip-bring-in-dmclock-p2-gtest/src/dmclock/CMakeLists.txt#L19-L25

smithfarm · 2017-05-22T20:53:49Z

src/CMakeLists.txt

+# dmClock
+
+add_subdirectory(dmclock) # after gmock
+add_dependencies(tests dmclock-tests dmclock-data-struct-tests)


this line would get moved into the proposed WITH_DMCLOCK_TESTS conditional

ivancich requested review from liewegas and jdurgin April 4, 2017 21:25

liewegas added the build/ops label Apr 4, 2017

liewegas approved these changes Apr 4, 2017

View reviewed changes

ivancich force-pushed the wip-bring-in-dmclock branch 2 times, most recently from 92df9bc to 3c07720 Compare April 6, 2017 22:47

ivancich force-pushed the wip-bring-in-dmclock branch from 3c07720 to 3cc1f90 Compare April 7, 2017 15:11

tchaikov reviewed Apr 7, 2017

View reviewed changes

ivancich force-pushed the wip-bring-in-dmclock branch from 3cc1f90 to dee3d05 Compare April 7, 2017 20:37

liewegas added needs-qa wip-sage-testing labels Apr 20, 2017

liewegas removed needs-qa wip-sage-testing labels Apr 20, 2017

ivancich changed the title ~~dmclock: initial commit of dmclock QoS library~~ [DNM yet] dmclock: initial commit of dmclock QoS library Apr 25, 2017

ivancich added 4 commits April 27, 2017 18:18

Add a README to explain some of the nature of git subtrees and some of

d6b04e3

the pitfalls surrounding them. Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Squashed 'src/dmclock/' content from commit d6586d7

0bca9fe

git-subtree-dir: src/dmclock git-subtree-split: d6586d73679f4a1bdf335235d309e2352f0c76c6

Merge commit '0bca9fe991c7e1f623d2a387f54e63f18d3227eb' as 'src/dmclock'

06b037b

Modify ceph cmake files to bring in dmclock library and tests. Include

e80a884

dmclock tests in ceph-test for builds. Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

ivancich force-pushed the wip-bring-in-dmclock branch from dee3d05 to e80a884 Compare May 1, 2017 12:44

ivancich changed the title ~~[DNM yet] dmclock: initial commit of dmclock QoS library~~ dmclock: initial commit of dmclock QoS library May 1, 2017

liewegas added needs-qa wip-sage-testing labels May 1, 2017

liewegas merged commit 5f1c03e into ceph:master May 2, 2017

smithfarm mentioned this pull request May 19, 2017

build/ops: fix dmclock test-related build breakage by conditionalizing dmclock tests #15174

Closed

smithfarm reviewed May 22, 2017

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dmclock: initial commit of dmclock QoS library #14330

dmclock: initial commit of dmclock QoS library #14330

ivancich commented Apr 4, 2017

ivancich commented Apr 4, 2017

liewegas commented Apr 4, 2017

jdurgin commented Apr 4, 2017

ivancich commented Apr 5, 2017

liewegas commented Apr 7, 2017

tchaikov Apr 7, 2017

ivancich Apr 7, 2017

liewegas commented Apr 20, 2017

liewegas commented May 2, 2017

smithfarm commented May 19, 2017

ivancich commented May 22, 2017

smithfarm commented May 22, 2017

smithfarm commented May 22, 2017

smithfarm May 22, 2017

smithfarm May 22, 2017

smithfarm May 22, 2017 •

edited

smithfarm May 22, 2017

ivancich May 22, 2017

smithfarm May 22, 2017

smithfarm May 22, 2017

smithfarm May 22, 2017

dmclock: initial commit of dmclock QoS library #14330

dmclock: initial commit of dmclock QoS library #14330

Conversation

ivancich commented Apr 4, 2017

ivancich commented Apr 4, 2017

liewegas commented Apr 4, 2017

jdurgin commented Apr 4, 2017

ivancich commented Apr 5, 2017

liewegas commented Apr 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liewegas commented Apr 20, 2017

liewegas commented May 2, 2017

smithfarm commented May 19, 2017

ivancich commented May 22, 2017

smithfarm commented May 22, 2017

smithfarm commented May 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smithfarm May 22, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smithfarm May 22, 2017 •

edited