Tests too verbose

[davetcoleman]

MoveIt! is using the following alias for running its tests on Travis via Docker:

run_tests: build --verbose --catkin-make-args run_tests --

However the verbose argument results in really long log files that get cut off by Travis' limit. e.g. from this build a small snippet of the noise:

make[3]: Leaving directory '/root/ws_moveit/build/moveit_ikfast'
/usr/bin/make -f CMakeFiles/run_tests.dir/build.make CMakeFiles/run_tests.dir/build
make[3]: Entering directory '/root/ws_moveit/build/moveit_ikfast'
make[3]: Nothing to be done for 'CMakeFiles/run_tests.dir/build'.
make[3]: Leaving directory '/root/ws_moveit/build/moveit_ikfast'
Built target run_tests
make[2]: Leaving directory '/root/ws_moveit/build/moveit_ikfast'
/usr/bin/cmake -E cmake_progress_start /root/ws_moveit/build/moveit_ikfast/CMakeFiles 0
make[1]: Leaving directory '/root/ws_moveit/build/moveit_ikfast'
cd /root/ws_moveit/build/moveit_ikfast; catkin build --get-env moveit_ikfast | catkin env -si  /usr/bin/make run_tests --jobserver-fds=6,7 -j; cd -
Finished << moveit_ikfast:make
Starting >> moveit_ikfast:symlink
Output << moveit_ikfast:symlink /root/ws_moveit/logs/moveit_ikfast/build.symlink.002.log
Linked: (/root/ws_moveit/devel/.private/moveit_ikfast/share/moveit_ikfast/cmake/moveit_ikfastConfig-version.cmake, /root/ws_moveit/devel/share/moveit_ikfast/cmake/moveit_ikfastConfig-version.cmake)
Linked: (/root/ws_moveit/devel/.private/moveit_ikfast/share/moveit_ikfast/cmake/moveit_ikfastConfig.cmake, /root/ws_moveit/devel/share/moveit_ikfast/cmake/moveit_ikfastConfig.cmake)
Linked: (/root/ws_moveit/devel/.private/moveit_ikfast/lib/pkgconfig/moveit_ikfast.pc, /root/ws_moveit/devel/lib/pkgconfig/moveit_ikfast.pc)

I tried removing the --verbose flag but then I also lose the important test results data, e.g. from this build it is missing the output that looks like:

[==========] Running 6 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from TestPropagationDistanceField
[ RUN      ] TestPropagationDistanceField.TestAddRemovePoints
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from LoadPlanningModelsPr2
[ RUN      ] LoadPlanningModelsPr2.InitOK
[       OK ] TestPropagationDistanceField.TestAddRemovePoints (11 ms)
[----------] 1 test from TestPropagationDistanceField (11 ms total)

How can I hide the build noise but still show the test results? catkin_test_results will give you a one line summary of failures, but not tell you which tests failed.

I suspect a new verb proposed in #397 might address this issue.

[VictorLamoine]

👍 the verbose arguments results in test being hidden in dozens of lines. This is un-readable.
@davetcoleman have you found a "trick" to make things better? I didn't!

I'm sticking to catkin_make when I want to read test results because of this issue.

[davetcoleman]

I have not found a good solution, but in moveit we are using this:
https://github.com/ros-planning/moveit_ci/blob/master/travis.sh#L164

[HWiese1980]

Is there a solution to this issue? I'm encountering it too at the moment. The MoveIt! solution is not sufficient for my needs as it only shows the summary of how many tests have succeeded, failed and so on, but not which and why.

[VictorLamoine]

@HWiese1980 I'm still sticking to catkin_make when reading failed tests.
I did not find a solution to this issue.

[Jmeyer1292]

A 2018 ping: I finally decided to test something and this is pretty rough. @VictorLamoine or @davetcoleman, did you guys find a solution ultimately?

[VictorLamoine]

No I'm relying on catkin_make for the tests.

[taketwo]

A workaround is to pipe the output through sed:

$ catkin run_tests | sed -En '/^-- run_tests.py/,/^-- run_tests.py/p'

This filters out everything besides the output produced by the commands executed via run_tests.py script.

[davetcoleman]

Love it!

[arpitg1304]

thanks @taketwo

[UniverseHan]

Thanks @taketwo

[UniverseHan]

Don't we have a plan to fix this unwanted default behavior or not?

Is there any idea of whether we should fix this or not?

[zflat]

Another workaround that I find helpful in having more control over gtest is to first build the test target and then run the tests manually. So assuming I have a test target defined in my CMakeLists.txt file catkin_add_gtest(${PROJECT_NAME}_test ...` I can do something like this:

export PACKAGE=my_package
catkin build --no-deps $PACKAGE --catkin-make-args ${PACKAGE}_test
rosrun $PACKAGE ${PACKAGE}_test

I like this because I can interact with gtest for example to get a list of tests or to filter tests. Get a list of available tests:

catkin build --no-deps $PACKAGE --catkin-make-args ${PACKAGE}_test \
&& rosrun $PACKAGE ${PACKAGE}_test --gtest_list_tests

[bren-dusty]

Will Just ping this. Its still very annoying to try and debug tests with the current output. You only get the error output and none of the prior ros_info / debug / error

[timonegk]

Thank you for the ping. I started implementing the catkin test verb in #676. Please test it on your setup and report any problems or requests in the linked pull request. Since I myself don't have that many tests (shame on me), I rely on your participation!