cmake changes #10283

tchaikov · 2016-07-13T13:55:46Z

this changeset addresses following issues

silence the "no such option: --disable-pip-version-check" warnings
"make tests all" for preparing the "make test" // yeah, tests + all builds more artifacts than what is actually needed for "make test", we will fix this later
run tests in two steps as a workaround of the annoying jenkins "make check" timeout
cmake cleanup

tchaikov · 2016-07-13T17:53:06Z

[  4%] ceph-disk is being created
New python executable in /tmp/ceph.hod/ceph-disk-virtualenv/bin/python2.7
Also creating executable in /tmp/ceph.hod/ceph-disk-virtualenv/bin/python
Installing setuptools, pip...[  4%] Built target kv_objs
[  4%] Scanning dependencies of target librados_objs
Building CXX object src/cls/CMakeFiles/cls_lock_client.dir/lock/cls_lock_types.cc.o
[  4%] Building CXX object src/CMakeFiles/librados_objs.dir/librados/IoCtxImpl.cc.o
done.
Running virtualenv with interpreter /usr/bin/python2.7

Usage:   
  pip <command> [options]

no such option: --disable-pip-version-check
[  4%] Built target common_util_obj
[  4%] Building CXX object src/CMakeFiles/librados_objs.dir/librados/RadosXattrIter.cc.o

Usage:   
  pip <command> [options]

no such option: --disable-pip-version-check

Usage:   
  pip <command> [options]

no such option: --disable-pip-version-check
make[3]: *** [src/ceph-disk/CMakeFiles/ceph-disk] Error 2
make[2]: *** [src/ceph-disk/CMakeFiles/ceph-disk.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....

tchaikov · 2016-07-13T17:53:33Z

retest this please.

tchaikov · 2016-07-14T03:14:16Z

Building CXX object src/test/CMakeFiles/unittest_subprocess.dir/test_subprocess.cc.o
CMakeFiles/unittest_mime.dir/mime.cc.o: file not recognized: File truncated
collect2: error: ld returned 1 exit status
make[3]: *** [bin/unittest_mime] Error 1
make[2]: *** [src/test/CMakeFiles/unittest_mime.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....

https://jenkins.ceph.com/job/ceph-pull-requests/9047/console

tchaikov · 2016-07-14T07:37:18Z

retest this please.

ghost · 2016-07-14T08:51:06Z

@tchaikov the cmake part is above my paygrade but the tox/python part looks good to me :-)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tchaikov · 2016-07-14T09:09:11Z

thanks for your review! @dachary

tchaikov · 2016-07-14T09:10:19Z

src/tools/setup-virtualenv.sh

+    DISABLE_PIP_VERSION_CHECK=
+fi
+
+pip $DISABLE_PIP_VERSION_CHECK --version 1>&2


please remove these lines, they are only for debugging purpose.

tchaikov · 2016-07-14T09:14:13Z

changelog

removed debugging messages.

tchaikov · 2016-07-14T12:32:40Z

retest this please.

tchaikov · 2016-07-14T14:18:19Z

changelog

label rados-striper.sh with "Racing"

tchaikov · 2016-07-14T14:47:53Z

Linking CXX executable ../../../bin/unittest_erasure_code_example
CMakeFiles/unittest_erasure_code_example.dir/TestErasureCodeExample.cc.o: file not recognized: File truncated
collect2: error: ld returned 1 exit status

i am getting this error in https://jenkins.ceph.com/job/ceph-pull-requests/9095/consoleFull

@ErwanAliasr1 could this be related with some env problem in jenkins slave?

tchaikov · 2016-07-14T14:49:12Z

retest this please.

alimaredia · 2016-07-14T19:08:12Z

@tchaikov what was the purpose of changing the name of the "check" target to "test"? Is it to override the current "test" target that just runs ctest?

this follows the pattern in ceph-disk. this enables us to run ceph-detect-init/run-tox.sh from the ${CMAKE_BINARY_DIRECTORY} as well. Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov · 2016-07-15T02:01:23Z

@alimaredia that commit is not necessary at all. i just removed it. i am using "tests" for preparing the test dependency, which does not conflict with "test" or "check".

alimaredia · 2016-07-15T17:20:12Z

run-make-check.sh

-        $DRY_RUN make $BUILD_MAKEOPTS check || return 1
+        $DRY_RUN make $BUILD_MAKEOPTS all tests || return 1
+        $DRY_RUN ctest -L Racing -j1 --output-on-failure || return 1
+        $DRY_RUN ctest -LE Racing $CHECK_MAKEOPTS --output-on-failure || return 1


@tchaikov why are there two runs of ctest here?

please see commit message of 19a8b5c.

please note "make test" is used by cmake to run tests, so we cannot just repurpose it to *build* them. * AddCephTest.cmake: depends on "tests" * CMakeLists.txt: let "check" depend on "tests" * src/CMakeLists.txt: update the run-tox tests * run-make-check.sh: use "make tests" and "ctest" instead of "make check" * ceph-detect-init/CMakeLists.txt: let "tests" depend on "ceph-detect-init" * ceph-disk/CMakeLists.txt: let "tests" depend on "ceph-disk" Signed-off-by: Kefu Chai <kchai@redhat.com>

run-tox-ceph-disk and run-tox-ceph-detect-init are already added as test, so "ctest" and "make {test,check}" will run them without extra settings. Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov · 2016-07-16T03:38:59Z

changelog

s/check/tests/ at ce5724e#diff-9e70c23860593dc53da7c47ae633bbe7R481
let "check" depend on "tests"

two tests timesout for unknown reasons, so label them with "Racing" and "LongRunning" labels. Signed-off-by: Kefu Chai <kchai@redhat.com>

this is a workaround of the timeout found in jenkins. currently three tests are found timeout, and they are labeld with "Racing" and "LongRunning". so, to workaround this issue, we run the tests in two phases: 1. run the racing tests with -j1 2. run the non-racing tests with -jN if we all all tests with -j1, the total test time is 2683.57 sec Signed-off-by: Kefu Chai <kchai@redhat.com>

as ceph_crypt.cc is using the symbols in it, and libcommon contains ceph_crypt.cc. Signed-off-by: Kefu Chai <kchai@redhat.com>

they are included by rgw_a as well. and ceph_rgw_jsonparser is linked against rgw_a. Signed-off-by: Kefu Chai <kchai@redhat.com>

on older versions of pip, this option is not supported, and --disable-pip-version-check is implied with --no-index. so no need to use them when --no-index is passed to pip. this partially reverts 395f2c5 Signed-off-by: Kefu Chai <kchai@redhat.com>

Signed-off-by: Kefu Chai <kchai@redhat.com>

quote from https://cmake.org/cmake/help/v3.0/prop_test/TIMEOUT.html?highlight=timeout > If it exceeds that the test process will be killed and ctest will move > to the next test. this helps us to identify test hang. Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov · 2016-07-16T04:06:54Z

changelog

removed "Racing" label from rados-striper.sh

alimaredia · 2016-07-18T01:50:46Z

@tchaikov make test and make check ran fine for me but cephtool-test-mds.sh and cephtool-test-mon.sh didn't pass for me. Got a segfault for the mds test and a "process not found" for the mon test. Do those ring any bells?

tchaikov · 2016-07-18T02:13:42Z

@alimaredia

make test and make check ran fine for me

make check by itself would suffice, iiuc, make test is identical to ctest.

Got a segfault for the mds test

do you have the log and the backtrace? so we can look at it?

and a "process not found" for the mon test.

i spotted this fault before in the jenkins job, but it does not show up anymore.

tchaikov · 2016-07-18T17:14:23Z

@cbodley is this good to merge? as "make check" is working as expected.

cbodley · 2016-07-18T17:20:53Z

@cbodley is this good to merge? as "make check" is working as expected.

I haven't tested it, but you addressed my concern about the 'check' dependencies. Thanks!

alimaredia · 2016-07-18T17:52:37Z

@tchaikov I think this should get merged asap. I understand that this passes jenkins but you're not seeing the cephtool-test-mds.sh errors on your local machine like I am?

tchaikov · 2016-07-18T17:54:28Z

not yet. if i had seen them, i'd fixed them already.
Ali Maredia notifications@github.com于2016年7月19日周二01:52写道：

@tchaikov https://github.com/tchaikov I think this should get merged
asap. I understand that this passes jenkins but you're not seeing the
cephtool-test-mds.sh errors on your local machine like I am?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10283 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AADmvzNtXzaLDIy8A7W4lN31WXPbHBuhks5qW71lgaJpZM4JLcWi
.

tchaikov force-pushed the wip-cmake branch 3 times, most recently from 3787aec to 014c443 Compare July 13, 2016 16:37

tchaikov mentioned this pull request Jul 13, 2016

CMake - stop pip checking for updates #10161

Merged

tchaikov force-pushed the wip-cmake branch 5 times, most recently from fba9180 to c4271e1 Compare July 14, 2016 07:21

tchaikov changed the title ~~[DNM] cmake changes~~ cmake changes Jul 14, 2016

tchaikov assigned dang, ghost and alimaredia Jul 14, 2016

tchaikov added the build/ops label Jul 14, 2016

tchaikov reviewed Jul 14, 2016
View reviewed changes

tchaikov force-pushed the wip-cmake branch from c4271e1 to c633504 Compare July 14, 2016 09:12

tchaikov force-pushed the wip-cmake branch from c633504 to b1550ae Compare July 14, 2016 14:17

tchaikov force-pushed the wip-cmake branch from b1550ae to 766e59d Compare July 14, 2016 16:01

test: ceph-detect-init/run-tox.sh: run it from any path

3be6e1c

this follows the pattern in ceph-disk. this enables us to run ceph-detect-init/run-tox.sh from the ${CMAKE_BINARY_DIRECTORY} as well. Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov force-pushed the wip-cmake branch from 766e59d to 25963da Compare July 15, 2016 01:58

tchaikov unassigned dang and ghost Jul 15, 2016

alimaredia reviewed Jul 15, 2016
View reviewed changes

tchaikov added 2 commits July 16, 2016 10:57

cmake: no need to depend run-tox-ceph-disk on test

5b14e28

run-tox-ceph-disk and run-tox-ceph-detect-init are already added as test, so "ctest" and "make {test,check}" will run them without extra settings. Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov force-pushed the wip-cmake branch from 25963da to b290183 Compare July 16, 2016 02:58

tchaikov added 7 commits July 16, 2016 12:00

cmake: label some tests with "Racing"

3a21d00

two tests timesout for unknown reasons, so label them with "Racing" and "LongRunning" labels. Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link libcommon against $CRYPTO_LIBS

64a8bfb

as ceph_crypt.cc is using the symbols in it, and libcommon contains ceph_crypt.cc. Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated src from ceph_rgw_jsonparser

941605d

they are included by rgw_a as well. and ceph_rgw_jsonparser is linked against rgw_a. Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: restructure src/CMakeLists.txt in a more hierarchical way

5a69597

Signed-off-by: Kefu Chai <kchai@redhat.com>

tchaikov force-pushed the wip-cmake branch from 76de696 to c30c522 Compare July 16, 2016 04:01

tchaikov force-pushed the wip-cmake branch from 858e7ef to c30c522 Compare July 16, 2016 12:17

alimaredia merged commit 132b6ff into master Jul 18, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmake changes #10283

cmake changes #10283

tchaikov commented Jul 13, 2016 •

edited

tchaikov commented Jul 13, 2016

tchaikov commented Jul 13, 2016

tchaikov commented Jul 14, 2016 •

edited

tchaikov commented Jul 14, 2016

ghost commented Jul 14, 2016

tchaikov commented Jul 14, 2016

tchaikov Jul 14, 2016

tchaikov commented Jul 14, 2016

tchaikov commented Jul 14, 2016

tchaikov commented Jul 14, 2016

tchaikov commented Jul 14, 2016

tchaikov commented Jul 14, 2016

alimaredia commented Jul 14, 2016

tchaikov commented Jul 15, 2016 •

edited

alimaredia Jul 15, 2016

tchaikov Jul 16, 2016

tchaikov commented Jul 16, 2016

tchaikov commented Jul 16, 2016

alimaredia commented Jul 18, 2016

tchaikov commented Jul 18, 2016

tchaikov commented Jul 18, 2016

cbodley commented Jul 18, 2016

alimaredia commented Jul 18, 2016

tchaikov commented Jul 18, 2016

cmake changes #10283

cmake changes #10283

Conversation

tchaikov commented Jul 13, 2016 • edited

tchaikov commented Jul 13, 2016

tchaikov commented Jul 13, 2016

tchaikov commented Jul 14, 2016 • edited

tchaikov commented Jul 14, 2016

ghost commented Jul 14, 2016

tchaikov commented Jul 14, 2016

tchaikov Jul 14, 2016

Choose a reason for hiding this comment

tchaikov commented Jul 14, 2016

tchaikov commented Jul 14, 2016

tchaikov commented Jul 14, 2016

tchaikov commented Jul 14, 2016

tchaikov commented Jul 14, 2016

alimaredia commented Jul 14, 2016

tchaikov commented Jul 15, 2016 • edited

alimaredia Jul 15, 2016

Choose a reason for hiding this comment

tchaikov Jul 16, 2016

Choose a reason for hiding this comment

tchaikov commented Jul 16, 2016

tchaikov commented Jul 16, 2016

alimaredia commented Jul 18, 2016

tchaikov commented Jul 18, 2016

tchaikov commented Jul 18, 2016

cbodley commented Jul 18, 2016

alimaredia commented Jul 18, 2016

tchaikov commented Jul 18, 2016

tchaikov commented Jul 13, 2016 •

edited

tchaikov commented Jul 14, 2016 •

edited

tchaikov commented Jul 15, 2016 •

edited