run_tests writes to wrong workspace

[bricerebsamen]

I have a chain of workspaces: third, stable, exp, in this order, i.e. stable sources third and exp sources stable.

if I run catkin_make run_tests in the stable workspace, the xml result file is written to exp/build/test_results instead of stable/build/test_results.

-- run_tests.py: execute commands
  /opt/ros/indigo/share/rostest/cmake/../../../bin/rostest --pkgdir=/home/brice/ws/stable/src/velodyne_localizer --package=velodyne_localizer --results-filename tests_short_loop_4-11-05-2014_12-01-05.xml /home/brice/ws/stable/src/velodyne_localizer/tests/short_loop_4-11-05-2014_12-01-05.launch 
... logging to /home/brice/.ros/log/rostest-boox-19288.log
[ROSUNIT] Outputting test results to /home/brice/ws/exp/build/test_results/velodyne_localizer/rostest-tests_short_loop_4-11-05-2014_12-01-05.xml
testtest_localizer_short_loop_4_11_05_2014 ... ok

[ROSTEST]-----------------------------------------------------------------------

[velodyne_localizer.rosunit-test_localizer_short_loop_4_11_05_2014/VelodyneLocalizer][passed]

SUMMARY
 * RESULT: SUCCESS
 * TESTS: 1
 * ERRORS: 0
 * FAILURES: 0

rostest log file is in /home/brice/.ros/log/rostest-boox-19288.log
-- run_tests.py: verify result "/home/brice/ws/stable/build/test_results/velodyne_localizer/rostest-tests_short_loop_4-11-05-2014_12-01-05.xml"
Cannot find results, writing failure results to '/home/brice/ws/stable/build/test_results/velodyne_localizer/MISSING-rostest-tests_short_loop_4-11-05-2014_12-01-05.xml'

$ ls /home/brice/ws/exp/build/test_results/velodyne_localizer
rostest-tests_short_loop_4-11-05-2014_12-01-05.xml
rosunit-test_localizer_short_loop_4_11_05_2014.xml
$ ls /home/brice/ws/stable/build/test_results/velodyne_localizer/
gtest-test_laser_map.xml
MISSING-rostest-tests_short_loop_4-11-05-2014_12-01-05.xml

note that in this examples there are 2 tests in the velodyne_localizer package:

• test_laser_map which is a unit test added using catkin_add_gtest. The result is written to the appropriate location
• short_loop_4-11-05-2014_12-01-05 which is a rostest added using add_rostest. The results are written to the wrong location

[bricerebsamen]

I cannot find the code for add_rostest. A grep for it in the catkin package only returns results in the doc folder.

[wjwwood]

That's because it's in the rostest package: https://github.com/ros/ros_comm/tree/indigo-devel/tools/rostest

[wjwwood]

There is a related issue here, though I couldn't extract enough information to track down the actual issue: catkin/catkin_tools#137

We have another user of catkin which has identified a similar problem, so I think this is issue is real and is caused by a conceptual problem with the way we set the test variable. It has to do with the compatibility with the older rosbuild system.

[dirk-thomas]

The problem is that the location is package specific but determined by a runtime setting defined through an environment hook. This is the case since rosmake used an environment variable which defines the location of the test results.

One option to allow package specific folders for the test results would be to change the code to make the decision about the result location at configure time. This would definitely require break backward compatibility. Therefore I don't see a good path forward currently.

[bricerebsamen]

what's confusing is that the rostest results and the gtest results are written in different locations and that catkin_test_results reports missing tests. Personally I don't care where the results are written, but I'd like something integrated with catkin_test_results. So what about writing them to .ros? Or making catkin_test_results search for test results in the current workspace and in the tip workspace (exp in my case)?

[dirk-thomas]

The problem should be addressed with #728. Please give it a try and confirm that it works for your use case (please remove the build space to perform a clean build).

[dirk-thomas]

I will close this ticker since the PR resolves the problem for me.