New launch system #74

wjwwood · 2018-06-05T23:12:10Z

This is the first pull request in a series, which adds the first pass at the reference implementation described in ros2/design#163.

I'm still working on this pull request, but some highlights to review in the mean time:

the architecture.rst document: https://github.com/ros2/launch/pull/74/files#diff-e25d31144ab0a51cf4d41fa107f6c9c3
the use of type checking with Python's typing module and mypy
- specifically, there are some import loops which can be avoided normally, but not when doing type checking, so there are cases where "fake" imports must be done to satisfy mypy, and I'd like feedback on that
- e.g. https://github.com/ros2/launch/pull/74/files#diff-84e2bbbeb5e4ca165d268d4962177343R23
the example: https://github.com/ros2/launch/pull/74/files#diff-450e2fd128e18f66fc9c0ca9d834ca43
console output from ExecuteProcess is currently modeled after roslaunch, but could be refactored to be like launch was, or something else entirely

What's known to be missing from this pr (and forth coming as soon as I can redo the commits) are tests and some example usage.

In a follow up pull request, I will add the launch_ros package and some demos using it, as well as the ros2launch package.

Things to open issues about after merging:

Tests for non-core classes (things like specific actions, utilities, substitutions, etc...)
Improvements to the introspection system.
Fix introspection of event handlers
Update introspection support for some Actions (e.g. the ExecuteProcess action could report more of what it might do in certain situations)
Add option for hidden versus non-hidden (default) description elements (so you can see important stuff, but also visualize deep details if desired)
Add extensions to the launch service
Create extension to launch service for standard logging behaviors
Conditions
GroupAction
Launch Arguments
UnsetLaunchConfiguration
Check iterable contains only types in ensure_argument_type(), see: New launch system #74 (comment)

wjwwood · 2018-06-06T02:11:17Z

I added an example in f353805 for anyone who wants to start reviewing.

wjwwood · 2018-06-06T08:09:24Z

45e63bc added the tests I that I had time to clean up for this pull request. The more detailed actions still need proper automated tests, but are otherwise pretty well covered in demos I have to pull request later. I'll add expanding the test coverage to the list of new issues that need to be made as follow up work.

dirk-thomas

the architecture.rst document

Looks very good to me. 👍👍

Only one minor comment to clarify required arguments and a nitpick.

Does the current code come with any callables which can be passed to the OnProcessIO handler to e.g. just print to stdout? Since the example defines its own local function for that.

the use of type checking with Python's typing module and mypy

I haven't used mypy before but the approach looks fine to me on a first look.

the example

Why is the example file called launch_counter_good_bad_ugly.py? It looks pretty clean and intuitive to read or do you consider any part of it bad / ugly style?

dirk-thomas · 2018-06-06T18:51:27Z

launch/doc/source/architecture.rst

+- :class:`launch.actions.DeclareLaunchDescriptionArgument`
+
+  - This action will declare a launch description argument, which can have a name, default value, and documentation.
+  - The argument will be exposed via a command line option for a root launch description, or as action configurations to the include launch description action for the included launch description.


Can an argument be marked as "required"? So if it doesn't have a default value but isn't provided the invocation would fail.

My plan was to make argument without default values required, I think this is how it works in roslaunch: https://wiki.ros.org/roslaunch/XML/arg#Usecase_pattern

dirk-thomas · 2018-06-06T18:54:23Z

launch/doc/source/architecture.rst

+Substitutions
+-------------
+
+A substitution is a something that cannot, or should not, be evaluated until it's time to execute the launch description that they are used in.


A substitution is a something that cannot ...

The second "a" seems to be wrong?

wjwwood · 2018-06-06T21:29:51Z

Does the current code come with any callables which can be passed to the OnProcessIO handler to e.g. just print to stdout? Since the example defines its own local function for that.

Not right now, the reason is that I plan to integrate the "default" one from launch_ros so it can hook into the ROS logging system, though I'm not 100% sure that's what we want. I might add one to launch just for convenience. I plan to still port the output handler concept from the old launch in some form.

Why is the example file called launch_counter_good_bad_ugly.py? It looks pretty clean and intuitive to read or do you consider any part of it bad / ugly style?

Sorry, that's because there are three counters, one that behaves, one that ignores sigint (bad), and one that ignores sigint and sigterm (ugly). I can shorten the name. It was late when I made that file :)

Output will likely be moved to an “OutputHandler” type class in the future.

Access to argv. Access to shutdown state (from LaunchService via LaunchContext). Ability to asynchronously add completion events. Add concept of globals to go with locals.

wjwwood · 2018-06-12T00:10:53Z

Ok, I'm going to put this one into review and build the ROS one on top.

CI:

Linux
Linux-aarch64
macOS
Windows

wjwwood · 2018-06-12T00:11:38Z

Note, I will create issues for the things listed at the end of the description of this issue after merging.

clalancette

I've done a partial review, mostly of the documentation. I can do some more on this on Thursday.

clalancette · 2018-06-12T21:26:14Z

launch/doc/source/architecture.rst

+========================
+
+`launch` is designed to provide core features like describing actions (e.g. executing a process or including another launch description), generating events, introspecting launch descriptions, and executing launch descriptions.
+While at the same time, it provides extension points so that the set of things on which these core features can operate on, or integrate with, can be expanded with additional packages.


Can probably drop the While in the above sentence, as it is a bit redundant.

clalancette · 2018-06-12T21:26:34Z

launch/doc/source/architecture.rst

+
+When visited, entities may yield additional entities to be visited, and this pattern is used from the "root" of the launch, where a special entity called :class:`launch.LaunchDescription` is provided to start the launch process.
+
+The :class:`launch.LaunchDescription` class encapsulates the intent of the user as a list of discrete :class:`launch.Action`'s (which are also derived from :class:`launch.LaunchDescriptionEntity`.


Close parens at the end here.

clalancette · 2018-06-12T21:27:07Z

launch/doc/source/architecture.rst

+These substitutions are things which can be evaluated during launch and can be used to do various things like: get a launch configuration, get an environment variable, or evaluate arbitrary Python expressions.
+
+Launch descriptions, and the actions contained therein, can either be introspected directly or launched by a :class:`launch.LaunchService`.
+A launch service, is a long running activity which handles the event loop and dispatches actions.


No comma needed in this sentence.

clalancette · 2018-06-12T21:29:09Z

launch/doc/source/architecture.rst

+^^^^^^^^^^^^^
+
+`launch` provides the foundational actions on which other more sophisticated actions may be built.
+This is a non-exhaustive list of actions that `launch` may provide:


Thinking about this as a user, why is this a may provide? It seems like this is a (non-exhaustive) list of things that launch does provide, or am I misunderstanding?

I also asked myself this question.

Simply because I didn't want to have to come back here and update the list every time I added a trivial action to the list in launch. These are the important ones (as I see it). This is just supposed to be an overview, not the complete API docs.

clalancette · 2018-06-12T21:29:51Z

launch/doc/source/architecture.rst

+
+- :class:`launch.actions.GroupAction`
+
+  - This action will yield other actions, but can associated with conditionals (allowing you to use the conditional on the group action rather than on each sub-action individually) and can optionally scope the launch configurations.


but can associated -> but can be associated

clalancette · 2018-06-12T21:30:40Z

launch/doc/source/architecture.rst

+
+- :class:`launch.actions.RegisterEventHandler`
+
+  - This action will register a :class:`launch.EventHandler` class, which takes user defined lambda to handle some event.


which takes user defined lambda -> which takes a user defined lambda

clalancette · 2018-06-12T21:34:48Z

launch/examples/counter.py

@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+
+# Copyright 2015 Open Source Robotics Foundation, Inc.


2018, probably :).

This one is based on the counter.py in the test folder for the old launch, so I think 2015 is ok here.

clalancette · 2018-06-12T21:36:54Z

launch/examples/launch_counters.py

@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+
+# Copyright 2015 Open Source Robotics Foundation, Inc.


dhood · 2018-06-13T16:50:14Z

launch/launch/event_handlers/on_process_exit.py

+        # TODO(wjwwood) check that it is not only callable, but also a callable that matches
+        # the correct signature for a handler in this case
+        self.__on_exit = on_exit
+        self.__actions_on_exit: List[LaunchDescriptionEntity] = []


yes, that looks better. I added 89d39de to be more consistent with how the OnShutdown handler is described, in the case a handler is passed in.

dhood · 2018-06-13T17:07:18Z

launch/launch/events/process/process_io.py

+        """
+        super().__init__(**kwargs)
+        self.__text = text
+        self.__from_stdin = fd == 0


is sys.stdin.fileno() appropriate in place of just 0 here and e.g. https://github.com/ros2/launch/pull/74/files/165e928a4021689017683033d75bf16bbb43a3ad#diff-76180495de0daffea911889f8030444fR34? I know it really just has to match between the two, so it might not make sense.

I don't think that's appropriate in this case, since the files uses for the stdout/stderr/stdin communication with the process are most often not sys.stdin.fileno(). I thought about using an enum or a string, but decided to just use an int. I could change the mapping mechanism if you want, but using the fileno from sys doesn't seem like the right thing to do.

OK as is I mean.

dhood · 2018-06-13T21:32:50Z

launch/launch/launch_service.py

+                        _logger.warn(base_msg)
+                        self._shutdown(reason='ctrl-c (SIGINT)', due_to_sigint=True)
+                        sigint_received = True
+                    else:


This is the opposite to what we've done in the previous launch: https://github.com/ros2/launch/pull/58/files
I think you've mentioned before ROS 1 does a similar thing, escalating based on the number of interrupts received, is that right?

the reason it's relevant is because I press it twice by habit sometimes (which is what prompted that PR)

The signal handling of the service needs more work, I also don't handle SIGTERM, which I probably should do in order to avoid zombie processes.

I'll spend some time on this right now, but at some point I'll need to just start making issues for things, otherwise this pr will never get merged in time.

To me it's fine if this is one of the things that gets ticketed (it will be easier to review as an incremental improvement anyway). My intent is to point out what I see while reviewing (because otherwise we might forget), then we can make the judgement on them individually.

For sure, didn't mean to sound like I was complaining, just saying I might fix it up some now and ticket other stuff for later, even if it ends up in the release as a follow and smaller change.

I pushed some enhancements in 010edcb, one of which is that repeated ctrl-c does not exit early.

dhood · 2018-06-13T23:03:20Z

launch/launch/substitutions/launch_configuration.py

+        if default is None:
+            self.__default = default
+        else:
+            # convert any items in default that are not a Substitution or str to a str


I want to note that it might trip users up that the default gets converted to a string underneath. It might be better to just enforce it's passed in as a string (if it's not a substitution) rather than doing it ourselves, so that a user that passes in a dictionary or something doesn't expect to be able to retrieve a dictionary later.

It's a trade-off, as it's convenient to pass literals or the result of expressions to the SetLaunchConfiguration action directly. I can change it.

I dunno, it's pretty convenient. I think I'd prefer to leave it.

If you want, I'll change it anyways, but what if I just improved the documentation a bit to be clear that the default must be a string in the end, so non-text/non-substitution things are converted to text implicitly?

yeah I already clarified a bit in 760fe46#diff-55c61397a0c771113a1e78e3143e31fdR77. I can see the convenience, so let's leave it; maybe one day we will store dicts as dicts even.

dhood · 2018-06-13T23:21:44Z

launch/launch/actions/execute_process.py

+        cwd: Optional[SomeSubstitutionsType] = None,
+        env: Optional[Dict[SomeSubstitutionsType, SomeSubstitutionsType]] = None,
+        shell: bool = False,
+        sigterm_timeout: SomeSubstitutionsType = LaunchConfiguration('sigterm_timeout', default=5),


I imagine you are setting the default via a LaunchConfiguration in order to retrieve its value when it's actually used (as opposed to just passing in a float). But is it actually possible that someone could modify it before then, externally? FWIU it gets evaluated here and we don't modify it internally, so maybe we can just pass it as a float?

answering myself: it is modifiable externally in the same way the launch-prefix configuration is.

I imagine you are setting the default via a LaunchConfiguration in order to retrieve its value when it's actually used (as opposed to just passing in a float).

Yes.

But is it actually possible that someone could modify it before then, externally?

Yes, consider:

ld = LaunchDescription([ SetLaunchConfiguration('sigterm_timeout', 42.0), ExecuteProcess(cmd=['ls']), ExecuteProcess( cmd=['ls'], sigterm_timeout=EnvironmentVariable('CUSTOM_SIGTERM_TIMEOUT')), ExecuteProcess(cmd=['ls', '-la']), # ExecuteProcess(cmd=['ls', '-la'], sigterm_timeout=1.0), ])

The first and third execute processes would use the timeout set in the launch configuration, but the second would not because it was explicitly passed and the fourth (commented out) would be a nice to have interface where sigterm_timeout could also take a non-text/non-substitution numeric, but it's not necessary.

FWIU it gets evaluated here and we don't modify it internally, so maybe we can just pass it as a float?

Not exactly, the substitutions that make up sigterm_timeout are passed to the TimerAction there, but they are not substituted until the TimerAction is visited, and the substitutions should not be evaluated until then I think. We couldn't expand the substitution there even if we wanted to, because when called via describe, it would not have a LaunchContext which is required to perform substitutions.

yep, it doesn't get evaluated until the TimerAction is visited/executed: https://github.com/ros2/launch/pull/74/files#diff-a40b801740c81b57ba0a59a56cc0d36fR79.

I documented this behaviour in f3417d4

dhood

Looks alright to me!

wjwwood · 2018-06-14T02:49:35Z

Thanks for the review @dhood! Here's some CI (with --packages-up-to launch):

Linux
Linux-aarch64
macOS
Windows

wjwwood · 2018-06-14T05:32:35Z

@dhood sorry, I should have gotten you to review the new changes before merging, CI was good with them, but if you have any time to review it postmortem that'd be great:

dhood · 2018-06-14T16:43:35Z

OK, looks good, I just have some small comments. I'm happy to open a followup PR if you agree with them.

It's fine that there's no else of this check for the original signal handler, because it came from getsignal. But for custom signal handlers passed in, we're just silently ignoring signal handlers that (by mistake) are not callable. I'd add a check to the setters.

this message should instead say the function needs to be called from the main thread, right? Did it used to get called on module import?

the use of "exception" in this text makes it sound like their handler won't be called if it raises a keyboard interrupt, where it actually will still be called.

wjwwood · 2018-06-14T18:49:39Z

It's fine that there's no else of this check for the original signal handler, because it came from getsignal. But for custom signal handlers passed in, we're just silently ignoring signal handlers that (by mistake) are not callable. I'd add a check to the setters.

There's no else because I believe signal.getsignal will return None if no handler was set. So it will be hard to know if the existing handler was intentionally not callable or in error. I'd say that's the responsibility of signal.signal to check.

this message should instead say the function needs to be called from the main thread, right? Did it used to get called on module import?

That's right, it started out as something I was doing on import and decided to change it to a function.

the use of "exception" in this text makes it sound like their handler won't be called if it raises a keyboard interrupt, where it actually will still be called.

Well there's not way of knowing if their handler will raise KeyboardInterrupt without running it, so I don't know how we'd do that. I didn't feel it was that confusing, but if you suggest a specific edit, I'll include it in a follow up pr.

dhood · 2018-06-14T18:53:41Z

There's no else because I believe signal.getsignal will return None if no handler was set. So it will be hard to know if the existing handler was intentionally not callable or in error. I'd say that's the responsibility of signal.signal to check.

Please re-read: I am referring to custom signal handlers.

There's no else because I believe signal.getsignal will return None if no handler was set. So it will be hard to know if the existing handler was intentionally not callable or in error. I'd say that's the responsibility of signal.signal to check.

Just removing the first part of the sentence is enough to clarify.

again, I'm happy to do it, but there wasn't an open PR. now I'll build on 77

wjwwood · 2018-06-14T18:57:27Z

Sorry, I thought you meant original handler since that's the line you linked to (117 is if callable(__original_sigint_handler):).

I'd check it in the on_sigint function before setting it then. Rather than checking it in the signal handler.

wjwwood · 2018-06-14T18:58:09Z

But you're right, your text makes total sense. I didn't read it closely enough.

dhood · 2018-06-14T18:59:00Z

yeah that was what I mentioned. I will do that; wasn't trying to create more work for you, just to get approval of the direction/intent :)

wjwwood added 2 commits June 5, 2018 15:09

update .gitignore and add default .flake8 config for 100 line length

9e87af8

add architecture documentation

e3da213

wjwwood added enhancement New feature or request in progress Actively being worked on (Kanban column) labels Jun 5, 2018

wjwwood self-assigned this Jun 5, 2018

first pass at implementation of architecture

6471f11

wjwwood force-pushed the new_launch branch from cba9142 to 6471f11 Compare June 5, 2018 23:15

add example and fix some imports

f353805

wjwwood added 2 commits June 6, 2018 01:05

fix missing find/replace for launch->launch.legacy

1cf60f1

add tests for core classes, fix some issues found testing

45e63bc

dirk-thomas reviewed Jun 6, 2018

View reviewed changes

wjwwood added 9 commits June 6, 2018 14:33

rename example

60d0e5e

typo

4ca35c1

use locals rather then a lambda in on shutdown handler

c40054b

add output and log_cmd options to ExecuteProcess

dd3b3ed

Output will likely be moved to an “OutputHandler” type class in the future.

add some missing attributes

522237c

add several features to the LaunchContext and LaunchService

0f7417c

Access to argv. Access to shutdown state (from LaunchService via LaunchContext). Ability to asynchronously add completion events. Add concept of globals to go with locals.

fix style to match ament_flake8

f1d4662

fix linter error

9d15139

add missing dependency on osrf_pycommon

165e928

wjwwood added in review Waiting for review (Kanban column) and removed in progress Actively being worked on (Kanban column) labels Jun 12, 2018

wjwwood mentioned this pull request Jun 12, 2018

ROS specific launch #75

Merged

clalancette reviewed Jun 12, 2018

View reviewed changes

Documentation fixes

8ec159a

dhood added 3 commits June 13, 2018 09:58

Avoid printing hex id of None

c8cd906

Clarify the 'first element' aspect of comment

9916821

Add todo for storing passed in action list

982aba6

dhood reviewed Jun 13, 2018

View reviewed changes

dhood added 2 commits June 13, 2018 14:01

Specialise shutdown docstring

2721df7

future .done() -> future.done()

fc4afea

dhood reviewed Jun 13, 2018

View reviewed changes

dhood and others added 3 commits June 13, 2018 15:14

Argument fixup

8ed32bd

Update substitutions' perform docstrings

760fe46

improve signal handling in LaunchService

010edcb

dhood reviewed Jun 13, 2018

View reviewed changes

Add timeouts types to docblock

f3417d4

dhood approved these changes Jun 14, 2018

View reviewed changes

remove .flake8

368f871

dhood mentioned this pull request Jun 14, 2018

remove ament_tools ros2/ros2#486

Merged

wjwwood added 2 commits June 13, 2018 19:02

fix signal handling of LaunchService outside of main thread and Windows

a9d82f9

another fix for Windows

54bc19e

wjwwood merged commit 88ae20e into master Jun 14, 2018

wjwwood deleted the new_launch branch June 14, 2018 03:16

clalancette removed the in review Waiting for review (Kanban column) label Jun 14, 2018

wjwwood mentioned this pull request Jun 14, 2018

Small fixups from launch PR #77

Merged

wjwwood mentioned this pull request Jul 12, 2018

[launch] check that iterable contains only types in ensure_argument_type() #109

Open


		When visited, entities may yield additional entities to be visited, and this pattern is used from the "root" of the launch, where a special entity called :class:`launch.LaunchDescription` is provided to start the launch process.

		The :class:`launch.LaunchDescription` class encapsulates the intent of the user as a list of discrete :class:`launch.Action`'s (which are also derived from :class:`launch.LaunchDescriptionEntity`.


		- :class:`launch.actions.GroupAction`

		- This action will yield other actions, but can associated with conditionals (allowing you to use the conditional on the group action rather than on each sub-action individually) and can optionally scope the launch configurations.


		- :class:`launch.actions.RegisterEventHandler`

		- This action will register a :class:`launch.EventHandler` class, which takes user defined lambda to handle some event.

		@@ -0,0 +1,79 @@
		#!/usr/bin/env python3

		# Copyright 2015 Open Source Robotics Foundation, Inc.

		@@ -0,0 +1,132 @@
		#!/usr/bin/env python3

		# Copyright 2015 Open Source Robotics Foundation, Inc.

New launch system #74

New launch system #74

Conversation

wjwwood commented Jun 5, 2018 • edited

wjwwood commented Jun 6, 2018

wjwwood commented Jun 6, 2018

dirk-thomas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wjwwood commented Jun 6, 2018

wjwwood commented Jun 12, 2018

wjwwood commented Jun 12, 2018

clalancette left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhood Jun 13, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wjwwood Jun 13, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhood left a comment

Choose a reason for hiding this comment

wjwwood commented Jun 14, 2018

wjwwood commented Jun 14, 2018

dhood commented Jun 14, 2018

wjwwood commented Jun 14, 2018

dhood commented Jun 14, 2018

wjwwood commented Jun 14, 2018

wjwwood commented Jun 14, 2018

dhood commented Jun 14, 2018

wjwwood commented Jun 5, 2018 •

edited

dhood Jun 13, 2018 •

edited

wjwwood Jun 13, 2018 •

edited