Record all topics #30

Martin-Idel-SI · 2018-08-31T10:43:02Z

This PR is based on #27

The goal of this PR is to allow to subscribe to all available topics. In addition, there are a few technical changes:

We publish serialized topics directly as shared pointers
We introduced logging macros for simple usage and to make sure that the correct package is given. The macros internally use the rcutils logging macros, so nothing has really changed on the surface
We improve and clarify why and how we filter topics to remove non-ROS topics that cannot be written

Karsten1987

I feel like having multiple copies of the logging file doesn't make sense. If there is something in the rcutils logging macros missing, we can address them upstream.

Karsten1987 · 2018-09-10T23:58:36Z

rosbag2/include/rosbag2/logging.hpp

@@ -0,0 +1,61 @@
+// Copyright 2018, Bosch Software Innovations GmbH.


what's the need for this file? Is there something in rcutils which is missing?

Karsten1987 · 2018-09-11T00:05:20Z

rosbag2/include/rosbag2/rosbag2.hpp

+   * Records topics to a bagfile. Subscription happens at startup time, hence the topics must
+   * exist when "record" is called.
+   *
+   * @param file_name Name of the bagfile to write


I believe we are using the notion of \param for doxygen

Karsten1987 · 2018-09-11T00:08:18Z

rosbag2/src/rosbag2/generic_subscription.cpp

-          RCUTILS_LOG_ERROR_NAMED(
-            "rosbag2",
-            "failed to destroy serialized message: %s", rcl_get_error_string_safe());
+          ROSBAG2_LOG_ERROR_STREAM(


there is still a problem with the allocator IMO.
So assume we create a serialized message using this subscription, keep it in memory and destroy the subscription before destroying the message.
The message now has an invalid allocator.

We had a close look and I believe it should be ok as is. The content of the allocator gets copied inside rmw_serialized_message_init(...) (see https://github.com/ros2/rcutils/blob/f04cc666dd06347050ff5c18b80079922e729d92/src/char_array.c#L50) so it should still be valid even after the pointer is destroyed (as long as its state is empty, of course, which seems to be the case for the default allocator, in the other case, even a delete won't help).

We did a check by prematurely destroying the allocator (create new rcutils_allocator_t, use it to initialize the serialized message and then immediately destroy it) and it ran perfectly and valgrind was also happy.

That said, I'd love to have a better way to do it.

true. Please ignore my comment then.

Karsten1987 · 2018-09-11T00:16:42Z

rosbag2/src/rosbag2/rosbag2_node.cpp

+{
+  std::vector<std::string> sanitized_topic_names;
+  for (const auto & topic_name : topic_names) {
+    sanitized_topic_names.push_back(topic_name[0] != '/' ? "/" + topic_name : topic_name);


https://github.com/ros2/rcl/blob/master/rcl/include/rcl/expand_topic_name.h

This function does exactly what you're looking for :)

Makes sense to use then. Thank you

Karsten1987 · 2018-09-11T00:18:44Z

rosbag2/src/rosbag2/rosbag2_node.cpp

+    }
+  }
+
+  return reduce_multiple_types_to_one(filter_topics_with_wrong_types(filtered_topics_and_types));


do you really need to do this? This copies the complete map. Maybe we could use the first element of the type just by convention.

Since this is done at startup time, it's not really a big overhead. What happens if there are multiple types and it's not a ros type? The subscription would fail, I guess, but isn't it easier to just leave it out?

Karsten1987 · 2018-09-11T00:19:36Z

rosbag2/src/rosbag2/rosbag2_node.cpp

+  // This should be replaced by an auto-discovery system in the future
+  std::this_thread::sleep_for(std::chrono::milliseconds(100));
+  return reduce_multiple_types_to_one(
+    filter_topics_with_wrong_types(this->get_topic_names_and_types()));


the topic types returned from here are correct.

Ok, I deleted our own filtering code, then. Thanks.

Karsten1987 · 2018-09-11T00:24:28Z

rosbag2/test/rosbag2/rosbag2_write_integration_fixture.hpp

+          while (rclcpp::ok() && messages_stored_counter_[topic_name] < number_expected_messages) {
+            publisher->publish(message->serialized_data);
+            // We need to wait a bit, in order for record to process the messages
+            std::this_thread::sleep_for(50ms);


isn't this potentially dangerous to assume? What if the publisher rate is higher than this?

This is just for regular functionality testing (no performance or the like), so I think it's fine to assume. Or what would be your preferred rationale?

Karsten1987 · 2018-09-11T00:25:04Z

rosbag2_storage/include/rosbag2_storage/logging.hpp

@@ -0,0 +1,61 @@
+// Copyright 2018, Bosch Software Innovations GmbH.


Isn't this a copy of the first file? Why is this necessary?

Karsten1987 · 2018-09-11T00:26:13Z

rosbag2_storage/src/impl/storage_factory_impl.hpp

@@ -69,17 +66,17 @@ get_interface_instance(
    auto unmanaged_instance = class_loader->createUnmanagedInstance(storage_id);
    instance = std::shared_ptr<InterfaceT>(unmanaged_instance);
  } catch (const std::runtime_error & ex) {
-    RCUTILS_LOG_ERROR_NAMED(ROS_PACKAGE_NAME,
-      "unable to load instance of read write interface: %s", ex.what());
+    ROSBAG2_STORAGE_LOG_ERROR_STREAM(


I don't really see a reason for this.

Karsten1987 · 2018-09-11T00:26:38Z

rosbag2_storage_default_plugins/src/rosbag2_storage_default_plugins/logging.hpp

@@ -0,0 +1,61 @@
+// Copyright 2018, Bosch Software Innovations GmbH.


that's the third copy of this file ?!

Martin-Idel-SI · 2018-09-11T08:05:05Z

Regarding the logging: We basically copied that behaviour from rviz where we always had similar macros in rviz_rendering, rviz_common - they weren't invented by us. However, we felt it would be a nice addition as it solves two problems:

The named macros need the package name. As far as I see, there is no nice way to make sure that the package name is uniform (there is no variable ROS_PACKAGE_NAME or the like).
We don't have a stream variant of the macros. We like the stream variant because it simplifies working with the different variables (no c_str() or std::to_string() or similar patterns) and is easier to read than the usual variant.

At least the first problem can't be addressed in rcutils as far as I can see.

- correctly expand topic names using rcl - do not check type correctness (supposed to be done internally)

Karsten1987 · 2018-09-11T15:24:58Z

rosbag2/src/rosbag2/rosbag2_node.cpp

+           });
+}
+
+std::string Rosbag2Node::expand_topic_name(std::string topic_name)


should this function take a const std::string &

Karsten1987 · 2018-09-11T15:28:32Z

rosbag2/src/rosbag2/generic_subscription.cpp

-          RCUTILS_LOG_ERROR_NAMED(
-            "rosbag2",
-            "failed to destroy serialized message: %s", rcl_get_error_string_safe());
+          ROSBAG2_LOG_ERROR_STREAM(


true. Please ignore my comment then.

…ackage (#48) * GH-111 Add package for converter plugins * GH-111 Add CDR converter plugin * GH-111 Add test for more primitives types * GH-116 Fix cdr converter after rebase on new converters interface * GH-116 Use rmw_serialize/rmw_deserialize directly in converter and link against rmw_fastrtps_cpp * Fix converter package.xml * Fix clang warnings * GH-30 Change interface to the same convention as rmw_(de)serialize * comply to new rcutils error handling API * use poco to load fastrtps * Update rosbag2_converter_default_plugins/src/rosbag2_converter_default_plugins/cdr/cdr_converter.cpp Co-Authored-By: Karsten1987 <karsten@osrfoundation.org>

- The TODO comments have been removed because they're no longer relevant: they have been discussed in the PR review

* GH-112 Open storage for reading handing in rmw_identifier * GH-113 Cleanup: better naming * GH-113 Introduce interface for StorageFactory to allow mocks in tests * GH-113 Add test for SequentialReader for using converters - Added mocks for storage and converters (and factories) * GH-113 Implement skeleton convert function - Use convert only if necessary (different input and output formats), converters are only loaded if really necessary. - Allocate_ros2_message is public to enable extensive tests for this function. - Helper function to get any typesupport by name - Helper function for empty ros2_message * GH-113 Implement allocate_ros2_message - Treats most messages already. - Some combinations of nested messages with arrays are still missing - Cleanup of DynamicArrayNested messages is failing - Main difficulty is the cleanup of the allocated ros2_message which needs to be done manually - The test_ros2_message is intended to be run with valgrind and there should be no leaks or problems with free! * GH-113 Fix DynamicArrayNested deallocation Swapping with empty container seems more stable than deleting the data pointer of the container. * GH-113 Add test for BoundedArrayNested deallocation * GH-113 Refactoring of deallocation code * GH-113 Fix string initialization in all types * GH-113 Fix vector<bool> initialization * GH-113 Add test for deallocation of topic name + Refactoring * GH-113 Minor refactoring of converter * GH-113 Make sure to throw an error if converters do not exist * GH-113 Delete superfluous imports * GH-113 Small fix for deleting vectors * GH-113 Fix build after rebase * GH-30 Minor refactoring - The TODO comments have been removed because they're no longer relevant: they have been discussed in the PR review * GH-30 Give an allocator as parameter to allocate_ros2_message() * GH-111 Add missing test dependencies for CDR converter test * GH-128 Extend message allocation test to also cover big strings - Big strings are not treated with small string optimization and need to be checked, too. * GH-128 Add tests for nested arrays * GH-128 always initialize vectors with a placement new * pass by ref * use new getter functions * consistent function naming * uncrustify * GH-30 Fix windows build * use visibility macros on all functions

tfoote added the in review label Aug 31, 2018

anhosi mentioned this pull request Aug 31, 2018

Improve sqlite usage and test stability #31

Merged

Martin-Idel-SI force-pushed the feature/record_all_topics branch from c63b58f to 83d9a34 Compare September 4, 2018 08:30

anhosi force-pushed the feature/record_all_topics branch from 83d9a34 to ebfb989 Compare September 5, 2018 14:44

Martin-Idel-SI and others added 10 commits September 5, 2018 18:57

ros2GH-23 Get all topics from node and sanitize

01947ca

ros2GH-23 Allow to write all available topics

c4098b2

ros2GH-23 Move methods to node for better interface

2351161

ros2GH-23 Use rmw_serialized_message_t consistently

9b1eb50

ros2GH-23 Improve santization of topics

c48c545

ros2GH-65 Introduce and use better logging macros

363cc67

ros2GH-23 Use publisher to serialized message directly

9f7701e

ros2GH-23 Improve readability of sanitizing topics and types

9cb06da

ros2GH-23 Add test for record all

8ffae59

ros2GH-23 Cleanup: add missing const ref to record interface

0eb18eb

anhosi force-pushed the feature/record_all_topics branch from ebfb989 to 0eb18eb Compare September 5, 2018 17:03

Karsten1987 reviewed Sep 11, 2018

View reviewed changes

Martin-Idel-SI added 2 commits September 11, 2018 11:04

Cleanup for doxygen

2a6c423

Improve topic sanitization

f1a173b

- correctly expand topic names using rcl - do not check type correctness (supposed to be done internally)

Karsten1987 approved these changes Sep 11, 2018

View reviewed changes

Pass topic_name by reference

7d77905

Karsten1987 merged commit 3b23acc into ros2:master Sep 11, 2018

Karsten1987 removed the in review label Sep 11, 2018

anhosi deleted the feature/record_all_topics branch September 12, 2018 06:25

Martin-Idel-SI added a commit to bosch-io/rosbag2 that referenced this pull request Oct 30, 2018

ros2GH-30 Change interface to the same convention as rmw_(de)serialize

1794ba3

botteroa-si added a commit to bosch-io/rosbag2 that referenced this pull request Nov 13, 2018

ros2GH-30 Minor refactoring

13f2e70

- The TODO comments have been removed because they're no longer relevant: they have been discussed in the PR review

botteroa-si added a commit to bosch-io/rosbag2 that referenced this pull request Nov 13, 2018

ros2GH-30 Give an allocator as parameter to allocate_ros2_message()

74602cd

anhosi added a commit to bosch-io/rosbag2 that referenced this pull request Nov 21, 2018

ros2GH-30 Fix windows build

aec25aa

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Record all topics #30

Record all topics #30

Martin-Idel-SI commented Aug 31, 2018

Karsten1987 left a comment

Karsten1987 Sep 10, 2018

Karsten1987 Sep 11, 2018

Martin-Idel-SI Sep 11, 2018

Karsten1987 Sep 11, 2018

Martin-Idel-SI Sep 11, 2018 •

edited

Loading

Karsten1987 Sep 11, 2018

Karsten1987 Sep 11, 2018

Martin-Idel-SI Sep 11, 2018

Karsten1987 Sep 11, 2018

Martin-Idel-SI Sep 11, 2018 •

edited

Loading

Karsten1987 Sep 11, 2018

Martin-Idel-SI Sep 11, 2018 •

edited

Loading

Karsten1987 Sep 11, 2018

Martin-Idel-SI Sep 11, 2018

Karsten1987 Sep 11, 2018

Karsten1987 Sep 11, 2018

Karsten1987 Sep 11, 2018

Martin-Idel-SI commented Sep 11, 2018

Karsten1987 Sep 11, 2018

Martin-Idel-SI Sep 11, 2018

Karsten1987 Sep 11, 2018

		@@ -0,0 +1,61 @@
		// Copyright 2018, Bosch Software Innovations GmbH.

Record all topics #30

Record all topics #30

Conversation

Martin-Idel-SI commented Aug 31, 2018

Karsten1987 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Martin-Idel-SI Sep 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Martin-Idel-SI Sep 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Martin-Idel-SI Sep 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Martin-Idel-SI commented Sep 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Martin-Idel-SI Sep 11, 2018 •

edited

Loading

Martin-Idel-SI Sep 11, 2018 •

edited

Loading

Martin-Idel-SI Sep 11, 2018 •

edited

Loading