Added tests for ros2action and ros2msg #287

ivanpauno · 2019-06-24T12:53:29Z

The idea of this is start increasing testing coverage in all the ros2cli related command line tools, to avoid manual testing in the next release.

Continuation of ros2/system_tests#372.

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

hidmic

I left a couple comments, but overall it looks like it's going in the right direction.

Though I recall that @cevans87 was recently working on something over these lines. Not sure exactly, but you may join efforts, or at least not duplicate.

hidmic · 2019-06-24T18:46:45Z

test_ros2cli/package.xml

+  <name>test_ros2cli</name>
+  <version>0.7.4</version>
+  <description>
+    Test of the ROS2 command line tools Framework.


@ivanpauno consider:

Suggested change

Test of the ROS2 command line tools Framework.

Tests for ROS2 command line tools.

hidmic · 2019-06-24T18:53:30Z

test_ros2cli/test/config_ros2action_test.py

@@ -0,0 +1,107 @@
+# Copyright 2019 Open Source Robotics Foundation, Inc.


@ivanpauno both here and in the next config_*.py file, I wonder if we can aggregate the information that's currently distributed in multiple collections into a single one so that's easier to read and extend. For instance, a list of dictionaries that keep verb, flags, output, etc. What do you think?

The Config class could also use the __slots__ attribute and/or a __init__ method for setting the members that the test code expects. These could then be documented to make it easier for future readers adding tests.

cevans87 · 2019-06-24T21:16:26Z

Though I recall that @cevans87 was recently working on something over these lines. Not sure exactly, but you may join efforts, or at least not duplicate.

I think you're right. This looks similar to what I prototyped last week.

The only major difference I see is that I tested cli functionality by using python subprocess.

If this automates the manual release testing and it's easy to test with any rmw implementation, I'm on board. It looks like we can translate the tests I've written to live in this repo and look like these tests.

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno · 2019-06-25T20:07:44Z

If this automates the manual release testing and it's easy to test with any rmw implementation, I'm on board.

Yes, probably using ros_testing is better, as it should be the standard way to write this kind of tests.

It looks like we can translate the tests I've written to live in this repo and look like these tests.

Sounds good.
I think they should live with the code it's testing, in this case in https://github.com/ros2/demos itself.
I originally opened this in ros2/system_tests#372, and then I moved them here for the same reason.

Also, the cli will probably look different soon: #288.
I think ros2msg, ros2srv will be deleted. Also list and show verbs of ros2action.

So I will wait a little bit, until that PR gets merged.

jacobperron

I agree @hidmic that it would be easier to understand and extend the tests if the config structure was changed.

It would be nice if we defined something like a 'TestCase' object that contained the command, verb with options, other launch actions, and the expected output. This could then be passed as a single list as the test configuration.

E.g.

class TestCase:
    command = 'msg'
    verb = 'show std_msgs/msg/String'
    launch_actions = []
    expected_output = ['string data']

Edit: maybe command and verb could be combined?

test_ros2cli/CMakeLists.txt

jacobperron · 2019-06-25T22:50:15Z

test_ros2cli/test/config_ros2action_test.py

@@ -0,0 +1,107 @@
+# Copyright 2019 Open Source Robotics Foundation, Inc.


The Config class could also use the __slots__ attribute and/or a __init__ method for setting the members that the test code expects. These could then be documented to make it easier for future readers adding tests.

test_ros2cli/test/config_ros2msg_test.py

jacobperron · 2019-06-25T23:14:14Z

I think ros2msg, ros2srv will be deleted. Also list and show verbs of ros2action

I don't expect these to go away any time soon. ros2interface will be the recommend tool over ros2msg and ros2srv, but we may leave them for a bit to help with the transition. ros2action list will not change; it is listing action servers/clients, not action types like ros2interface will.

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno · 2019-06-26T14:38:03Z

I don't expect these to go away any time soon. ros2interface will be the recommend tool over ros2msg and ros2srv, but we may leave them for a bit to help with the transition. ros2action list will not change; it is listing action servers/clients, not action types like ros2interface will.

Sounds logical, thanks for clarifying.

Addressed your comments in: 3bef9d3.

ivanpauno · 2019-06-26T17:16:35Z

CI (build up to test_ros2cli, test test_ros2cli, including opensplice):

Linux
Linux-aarch64
macOS
Windows

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno · 2019-06-26T20:02:09Z

CI again, after addressing the failures (I guess ...).
(build up to test_ros2cli, test test_ros2cli, including opensplice):

Linux
Linux-aarch64
macOS
Windows

test_ros2cli/test/config_ros2action_test.py

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno · 2019-06-28T16:32:00Z

CI (build up to test_ros2cli, test test_ros2cli, including opensplice):

Linux
Linux-aarch64
macOS
Windows

jacobperron

LGTM

test_ros2cli/test/config_ros2launch_test.py

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno · 2019-06-28T18:06:31Z

The last CI have some failures related to connext, and it seems that some verbs of the action command aren't working well with it. I will try to dig a little bit more into the problem.

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno · 2019-07-04T15:34:09Z

CI (build up to test_ros2cli, test test_ros2cli, including opensplice):

Linux
Linux-aarch64
macOS
Windows

ivanpauno · 2019-07-08T13:15:33Z

@hidmic @jacobperron I think this is ready for merging, but it needs a new review after last commit.

jacobperron · 2019-07-08T16:44:06Z

test_ros2cli/test/test_process_output_customizable.py.in

@@ -40,11 +41,16 @@ def generate_test_description(config, ready_fn):
        output='screen',
        sigterm_timeout=LaunchConfiguration('sigterm_timeout', default=60)
    )
+    if '@RMW_IMPLEMENTATION@' == 'rmw_connext_cpp':


Since the test code is otherwise RMW agnostic, I would rather see this logic moved to the tests macro in CMakeLists.txt. Also, with a comment explaining why this is required.

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno · 2019-07-09T13:30:00Z

Sanity check:

Linux
Linux-aarch64
macOS
Windows

jacobperron

LGTM.

I'm curious, why is the delay needed for Connext? It's probably worth adding a comment.

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno · 2019-07-09T17:54:21Z

LGTM.

I'm curious, why is the delay needed for Connext? It's probably worth adding a comment.

Sorry, I forgot about adding the comment.
Connext startup is just too slow. I tested manually, and if you start the fibonacci_action_server and then run ros2 action list, the first few times it doesn't appear (with connext).

test_ros2cli/package.xml

Signed-off-by: Audrow Nash <audrow@hey.com>

Added tests for ros2action and ros2 msg

96c3a7d

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno added enhancement New feature or request in review Waiting for review (Kanban column) labels Jun 24, 2019

ivanpauno self-assigned this Jun 24, 2019

This was referenced Jun 24, 2019

Add tests for ros2action and ros2msg ros2/system_tests#372

Closed

Add PathJoinSubstitution ros2/launch#266

Merged

hidmic reviewed Jun 24, 2019

View reviewed changes

Address per review comments

8f87ced

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

jacobperron reviewed Jun 25, 2019

View reviewed changes

Address review comments

3bef9d3

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

Corrections after running CI

15dc4b4

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

jacobperron reviewed Jun 26, 2019

View reviewed changes

test_ros2cli/test/config_ros2action_test.py Outdated Show resolved Hide resolved

ivanpauno added 4 commits June 27, 2019 12:34

Extend timeout, to avoid flaky test

a9b52bd

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

Solve problem on windows

e3e7769

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

Make windows happy. Custom timeout for each test

5383caa

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

Some extra timeout for windows

8cc35bc

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno mentioned this pull request Jun 28, 2019

Release Testing Automation - Iteration #1 ros2/ros2#739

Closed

Refactor

3d39e7d

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno requested review from jacobperron and hidmic June 28, 2019 16:32

jacobperron reviewed Jun 28, 2019

View reviewed changes

test_ros2cli/test/config_ros2launch_test.py Outdated Show resolved Hide resolved

Delete unused files

6b684d2

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

jacobperron approved these changes Jun 28, 2019

View reviewed changes

Add timer to make connext pass tests

17c1ed8

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno changed the title ~~Added tests for ros2action and ros2 msg~~ Added tests for ros2action and ros2msg Jul 8, 2019

ivanpauno requested a review from jacobperron July 8, 2019 13:14

jacobperron reviewed Jul 8, 2019

View reviewed changes

Address review comments

38c69ea

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

jacobperron approved these changes Jul 9, 2019

View reviewed changes

Add comment

0326882

Signed-off-by: ivanpauno <ivanpauno@ekumenlabs.com>

ivanpauno merged commit ef61333 into master Jul 9, 2019

delete-merged-branch bot deleted the ivanpauno/add-tests-for-ros2action-ros2msg branch July 9, 2019 20:25

dirk-thomas reviewed Aug 9, 2019

View reviewed changes

test_ros2cli/package.xml Show resolved Hide resolved

ivanpauno mentioned this pull request Aug 9, 2019

Update action_tutorials dependency name #312

Closed

esteve pushed a commit to esteve/ros2cli that referenced this pull request Dec 16, 2022

Update maintainers in setup.py (ros2#287)

20aef20

Signed-off-by: Audrow Nash <audrow@hey.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added tests for ros2action and ros2msg #287

Added tests for ros2action and ros2msg #287

ivanpauno commented Jun 24, 2019

hidmic left a comment

hidmic Jun 24, 2019

hidmic Jun 24, 2019

jacobperron Jun 25, 2019

cevans87 commented Jun 24, 2019

ivanpauno commented Jun 25, 2019

jacobperron left a comment •

edited

Loading

jacobperron Jun 25, 2019

jacobperron commented Jun 25, 2019

ivanpauno commented Jun 26, 2019

ivanpauno commented Jun 26, 2019

ivanpauno commented Jun 26, 2019

ivanpauno commented Jun 28, 2019 •

edited

Loading

jacobperron left a comment

ivanpauno commented Jun 28, 2019

ivanpauno commented Jul 4, 2019

ivanpauno commented Jul 8, 2019

jacobperron Jul 8, 2019

ivanpauno commented Jul 9, 2019

jacobperron left a comment

ivanpauno commented Jul 9, 2019

	Test of the ROS2 command line tools Framework.
	Tests for ROS2 command line tools.

		@@ -0,0 +1,107 @@
		# Copyright 2019 Open Source Robotics Foundation, Inc.

Added tests for ros2action and ros2msg #287

Added tests for ros2action and ros2msg #287

Conversation

ivanpauno commented Jun 24, 2019

hidmic left a comment

Choose a reason for hiding this comment

hidmic Jun 24, 2019

Choose a reason for hiding this comment

hidmic Jun 24, 2019

Choose a reason for hiding this comment

jacobperron Jun 25, 2019

Choose a reason for hiding this comment

cevans87 commented Jun 24, 2019

ivanpauno commented Jun 25, 2019

jacobperron left a comment • edited Loading

Choose a reason for hiding this comment

jacobperron Jun 25, 2019

Choose a reason for hiding this comment

jacobperron commented Jun 25, 2019

ivanpauno commented Jun 26, 2019

ivanpauno commented Jun 26, 2019

ivanpauno commented Jun 26, 2019

ivanpauno commented Jun 28, 2019 • edited Loading

jacobperron left a comment

Choose a reason for hiding this comment

ivanpauno commented Jun 28, 2019

ivanpauno commented Jul 4, 2019

ivanpauno commented Jul 8, 2019

jacobperron Jul 8, 2019

Choose a reason for hiding this comment

ivanpauno commented Jul 9, 2019

jacobperron left a comment

Choose a reason for hiding this comment

ivanpauno commented Jul 9, 2019

jacobperron left a comment •

edited

Loading

ivanpauno commented Jun 28, 2019 •

edited

Loading