Play old bagfiles (ros2#69)

* ros2GH-138 Move calculation of bag size - previously in rosbag2::Info - now in storage plugin * ros2GH-130 Add rosbag2_bag_v2_plugins package -This package will contain storage and converter plugins * ros2GH-131 don't build plugins on Windows * ros2GH-129 Add function to be generated - massive if/else between all message types - will be generated similar to ros1_bridge plugin * ros2GH-138 Write storage plugin for rosbag v2 bags * ros2GH-138 Make sure that no attempt to create a converter is made when trying to read a rsbag v2 bag file * ros2GH-138 Add play end-to-end test for rosbag v2 plugin * ros2GH-138 Use cmake files to find ros1 packages - Use files from ros1_bridge via PkgConfig * ros2GH-138 Add generator code * ros2GH-141 Add initial version of vendor package * ros2GH-141 Improve vendor package to build on Mac * ros2GH-138 Cleanup CMakeLists * ros2GH-141 Use unmanaged Instance of class-loader - managed instance somehow isn't available for gcc 6.3 * ros2GH-141 Reduce patch and copy new toplevle CMakeLists by hand * ros2GH-141 Fix Shared Instance usage * ros2GH-141 Improve maintainability of vendor package - Document what patches do and why changes are necessary - Load ros1 packages through cmake macro - Do not export ros1 packages via ament - use commit hash of current master which is more stable than using melodic-devel * ros2GH-138 Link against rclcpp - necessary for ros1_bridge * ros2GH-138 Avoid crash when trying to play v2 bags which contain unknown message types * ros2GH-138 Add CLI -s <storage_id> option to ros2 bag info and use it in rosbag2::info - this allows ros2 bag info to work also when the yaml metadata file does not exsist - this is always the case for rosbag1 bagfiles - it could also happen for sqlite or other storage based bagfiles * ros2GH-138 Add end-to-end info test for rosbag v2 files * ros2GH-138 Add unit tests to rosbag_storage * ros2GH-138 Add method to extract filename from path to FilesystemHelpers * ros2GH-138 Add proper logging for topics which cannot be converted * ros2GH-138 Improve finding dependencies of ros1 * ros2GH-141 Explicitly import transitive dependencies of vendor package * ros2GH-138 Skip tests via ament if ros1 is not available * ros2GH-133 First split of plugins * ros2GH-133 Write serialized rosbag message * ros2GH-133 Improve converter plugin - move generation templates outside of plugin folders as both plugins need it - use ros::serialization routines to deserialize the ros message * ros2GH-133 Add plugin to be found by pluginlib * ros2GH-133 Remove empty check in converter - With the rosbag_v2_converter_plugin, we don't need to treat rosbag_v2 storage any different * ros2GH-133 Assert serialization format in unit tests for storage * ros2GH-133 Delete superfluous include folder - Only needed if we want to link against the library * ros2GH-133 get_all_topics_and_types returns only valid ros2 types - This is necessary as the information is used by rosbag2_transport - ros2 bag info still shows all topics and types - rosbag::View::getConnections() can return multiple connections corresponding to the same topic * ros2GH-133 Improve end to end test - use a bagfile with messages not known to ros2 * ros2GH-133 Reformulate info message in case of missing ros1-ros2 mapping for a topic * ros2GH-14 Find messages first * Explicitly print message when on Windows Co-Authored-By: Martin-Idel-SI <external.Martin.Idel@bosch-si.com> * ros2GH-14 Refactor rosbag_storage vendor package - Improve toplevel CMakeLists - Put all patches into a resource subfolder * ros2GH-14 Reflect renames of converter interfaces * ros2GH-156 Workaround for path problems * ros2GH-156 Add documentation for plugin * ros2GH-156 Fix the pluginlib version to greater 2 * ros2GH-156 Prohibit CMake from declaring paths as system paths This switches the order of ros2 and ros1 directories resulting in build failures * ros2GH-156 Prohibit system include paths for rosbag plugins This can lead to switching ros1 and ros2 include paths resulting in missing symbols as the wrong pluginlib gets included * ros2GH-14 Split patches * make README more verbose * add plugin specific readme * more readme for bag_v2 plugin
thomas-moulard · Dec 15, 2018 · 5e6c491 · 5e6c491
1 parent 557e9a3
commit 5e6c491
Show file tree

Hide file tree

Showing 52 changed files with 2,640 additions and 28 deletions.
diff --git a/README.md b/README.md
@@ -1,12 +1,24 @@
 # rosbag2
 
-THIS IS WORK IN PROGRESS AND NOT READY TO BE USED YET
+THIS IS WORK IN PROGRESS AND SHOULD BE USED WITH CARE
 
-Repository for implementing ROSBag2 as described in its corresponding [design article](https://github.com/ros2/design/blob/f69fbbd11848e3dd6866b71a158a1902e31e92f1/articles/rosbags.md)
+Repository for implementing rosbag2 as described in its corresponding [design article](https://github.com/ros2/design/blob/f69fbbd11848e3dd6866b71a158a1902e31e92f1/articles/rosbags.md)
 
-## Build instructions
+## Installation instructions
 
-Create a new workspace:
+## Debian packages
+
+rosbag2 packages are available via debian packages and thus can be installed via
+
+```
+$ sudo apt-get install ros2bag rosbag2*
+```
+
+For other platforms than Linux, rosbag2 has to be built from source as it's currently not part of the latest (ros2.repos file)[https://github.com/ros2/ros2/blob/master/ros2.repos].
+
+## Build from source
+
+It is recommended to create a new overlay workspace on top of your current ROS 2 installation.
 
 ```
 $ mkdir -p ~/rosbag_ws/src
@@ -22,18 +34,117 @@ $ git clone https://github.com/ros2/rosbag2.git
 Then build all the packages with this command:
 
 ```
-$ colcon build --merge-install
+$ colcon build [--merge-install]
 ```
 
-The `--merge-install` flag is optional but ensures a cleaner environment which is helpful for development.
+The `--merge-install` flag is optional and installs all packages into one folder rather than isolated folders for each package.
 
 #### Executing tests
 
 The tests can be run using the following commands:
 
 ```
-$ colcon test --merge-install
+$ colcon test [--merge-install]
 $ colcon test-result --verbose
 ```
 
 The first command executes the test and the second command displays the errors (if any).
+
+## Using rosbag2
+
+rosbag2 is part of the ROS 2 command line interfaces.
+This repo introduces a new verb called `bag` and thus serves as the entry point of using rosbag2.
+As of the time of writing, there are three commands available for `ros2 bag`:
+
+* record
+* play
+* info
+
+### Recording data
+
+In order to record all topics currently available in the system:
+
+```
+$ ros2 bag record -a
+```
+
+The command above will record all available topics and discovers new topics as they appear while recording.
+This auto-discovery of new topics can be disabled by given the command line argument `--no-discovery`.
+
+To record a set of predefined topics, one can specify them on the command line explicitly.
+
+```
+$ ros2 bag record <topic1> <topic2> … <topicN>
+```
+
+The specified topics don't necessarily have to be present at start time.
+The discovery function will automatically recognize if one of the specified topics appeared.
+In the same fashion, this auto discovery can be disabled with `--no-discovery`.
+
+If not further specified, `ros2 bag record` will create a new folder named to the current time stamp and stores all data within this folder.
+A user defined name can be given with `-o, --output`.
+
+### Replaying data
+
+After recording data, the next logical step is to replay this data:
+
+```
+$ ros2 bag play <bag_file>
+```
+
+The bag file is by default set to the folder name where the data was previously recorded in.
+
+### Analyzing data
+
+The recorded data can be analyzed by displaying some meta information about it:
+
+```
+$ ros2 bag info <bag_file>
+```
+
+You should see something along these lines:
+
+```
+Files:             demo_strings.db3
+Bag size:          44.5 KiB
+Storage id:        sqlite3
+Duration:          8.501s
+Start:             Nov 28 2018 18:02:18.600 (1543456938.600)
+End                Nov 28 2018 18:02:27.102 (1543456947.102)
+Messages:          27
+Topic information: Topic: /chatter | Type: std_msgs/String | Count: 9 | Serialization Format: cdr
+                   Topic: /my_chatter | Type: std_msgs/String | Count: 18 | Serialization Format: cdr
+```
+
+## Storage format plugin architecture
+
+Looking at the output of the `ros2 bag info` command, we can see a field called `storage id:`.
+rosbag2 specifically was designed to support multiple storage formats.
+This allows a flexible adaptation of various storage formats depending on individual use cases.
+As of now, this repository comes with two storage plugins.
+The first plugin, sqlite3 is chosen by default.
+If not specified otherwise, rosbag2 will store and replay all recorded data in an SQLite3 database.
+Secondly, rosbag2 provides support for legacy ROS 1 bag data. __TODO(karsten1987) Add link when rosindex ran__.
+
+In order to use a specified (non-default) storage format plugin, rosbag2 has a command line argument for it:
+
+```
+$ ros2 bag <record> | <play> | <info> -s <sqlite3> | <rosbag2_v2> | <custom_plugin>
+```
+
+Have a look at each of the individual plugins for further information.
+
+## Serialization format plugin architecture
+
+Looking further at the output of `ros2 bag info`, we can see another field attached to each topic called `Serialization Format`.
+By design, ROS 2 is middleware agnostic and thus can leverage multiple communication frameworks.
+The default middleware for ROS 2 is DDS which has `cdr` as its default binary serialization format.
+However, other middleware implementation might have different formats.
+If not specified, `ros2 bag record -a` will record all data in the middleware specific format.
+This however also means that such a bag file can't easily be replayed with another middleware format.
+
+rosbag2 implements a serialization format plugin architecture which allows the user the specify a certain serialization format.
+When specified, rosbag2 looks for a suitable converter to transform the native middleware protocol to the target format.
+This also allows to record data in a native format to optimize for speed, but to convert or transform the recorded data into a middleware agnostic serialization format.
+
+By default, rosbag2 can convert from and to CDR as it's the default serialization format for ROS 2.
diff --git a/ros1_rosbag_storage_vendor/CMakeLists.txt b/ros1_rosbag_storage_vendor/CMakeLists.txt
@@ -0,0 +1,69 @@
+cmake_minimum_required(VERSION 3.5)
+project(ros1_rosbag_storage_vendor)
+
+# Default to C99
+if(NOT CMAKE_C_STANDARD)
+  set(CMAKE_C_STANDARD 99)
+endif()
+
+# Default to C++14
+if(NOT CMAKE_CXX_STANDARD)
+  set(CMAKE_CXX_STANDARD 14)
+endif()
+
+if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
+  add_compile_options(-Wall -Wextra -Wpedantic)
+endif()
+
+# find dependencies
+find_package(ament_cmake REQUIRED)
+
+# We don't need to build on Windows for now
+if(WIN32)
+  message("Replay ROS1 rosbags is currently not supported on Windows systems")
+  return()
+endif()
+
+set(git_apply git apply)
+
+include(ExternalProject)
+# We have to include a number of patches to the rosbag
+# 1. rosbag1_encryption_patch includes https://github.com/ros/ros_comm/pull/1499, without it, pluginlib won't link
+# 2. The next two patches patch the pluginlib usage:
+#    Missing boost/bind include
+#    Using pluginlib with boost results in compile errors on some systems, so we need to switch to pluginlib without boost
+# 3. The next patches patch the catkin package to ament and renames it
+#    Patching to ament is necessary as otherwise colcon needs to build a catkin package together with an ament package, which is impossible
+#    Renaming the package is necessary on some systems to not confuse the linker
+#
+# Note that we need to build with CMAKE_NO_SYSTEM_FROM_IMPORTED. This prohibits cmake from
+# declaring any paths as "internal" (flagged with -isystem instead of -I) which can result in this
+# vendor package building against the wrong pluginlib, because the paths get switched.
+# Symptoms of wrong paths: The build of the rosbag for ROS 1 plugins fails with missing symbols
+# like ros::package::getPlugins(...) when linking the vendor package
+ExternalProject_Add(ros1_rosbag_storage
+  DOWNLOAD_DIR ${CMAKE_CURRENT_BINARY_DIR}/download
+  URL https://github.com/ros/ros_comm/archive/669fbd32d2f92cc295f4b024fcb2f982fddec0f0.zip
+  URL_MD5 4c8b4c33165b223870f5d77bb697bef6
+  TIMEOUT 60
+  SOURCE_DIR ${CMAKE_CURRENT_BINARY_DIR}/sources
+  CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${CMAKE_CURRENT_BINARY_DIR}/rosbag_install -DCMAKE_NO_SYSTEM_FROM_IMPORTED=1
+  PATCH_COMMAND
+    ${git_apply} ${CMAKE_CURRENT_SOURCE_DIR}/resources/rosbag1_encryption_patch.diff &&
+    ${git_apply} ${CMAKE_CURRENT_SOURCE_DIR}/resources/bagh.diff &&
+    ${git_apply} ${CMAKE_CURRENT_SOURCE_DIR}/resources/bagcpp.diff &&
+    ${git_apply} ${CMAKE_CURRENT_SOURCE_DIR}/resources/bagcpp_name.diff &&
+    ${git_apply} ${CMAKE_CURRENT_SOURCE_DIR}/resources/plugin_descriptionxml.diff &&
+    ${git_apply} ${CMAKE_CURRENT_SOURCE_DIR}/resources/packagexml.diff &&
+    ${git_apply} ${CMAKE_CURRENT_SOURCE_DIR}/resources/cmakeliststxt.diff &&
+    ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_SOURCE_DIR}/cmake/CMakeLists.txt.in ${CMAKE_CURRENT_BINARY_DIR}/sources/CMakeLists.txt
+)
+
+install(
+  DIRECTORY
+    ${CMAKE_CURRENT_BINARY_DIR}/rosbag_install/
+  DESTINATION
+    ${CMAKE_INSTALL_PREFIX}/
+)
+
+ament_package()
diff --git a/ros1_rosbag_storage_vendor/cmake/CMakeLists.txt.in b/ros1_rosbag_storage_vendor/cmake/CMakeLists.txt.in
@@ -0,0 +1,5 @@
+# We download ros_comm, which does not contain a toplevel CMakeLists so colcon can't find the entry
+# point and claims there is nothing to build.
+# This is also an opportunity, because we want to only build rosbag_storage anyways
+cmake_minimum_required(VERSION 3.5)
+add_subdirectory(tools/rosbag_storage)
diff --git a/ros1_rosbag_storage_vendor/package.xml b/ros1_rosbag_storage_vendor/package.xml
@@ -0,0 +1,18 @@
+<?xml version="1.0"?>
+<?xml-model href="http://download.ros.org/schema/package_format2.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
+<package format="2">
+  <name>ros1_rosbag_storage_vendor</name>
+  <version>0.0.0</version>
+  <description>Vendor package for rosbag_storage of ROS1</description>
+  <maintainer email="karsten.knese@googlemail.com">Karsten Knese</maintainer>
+  <license>Apache License 2.0</license>
+
+  <buildtool_depend>ament_cmake</buildtool_depend>
+
+  <test_depend>ament_lint_auto</test_depend>
+  <test_depend>ament_lint_common</test_depend>
+
+  <export>
+    <build_type>ament_cmake</build_type>
+  </export>
+</package>
diff --git a/ros1_rosbag_storage_vendor/resources/bagcpp.diff b/ros1_rosbag_storage_vendor/resources/bagcpp.diff
@@ -0,0 +1,22 @@
+diff --git a/tools/rosbag_storage/src/bag.cpp b/tools/rosbag_storage/src/bag.cpp
+index 7e2e67171..073072934 100644
+--- a/tools/rosbag_storage/src/bag.cpp
++++ b/tools/rosbag_storage/src/bag.cpp
+@@ -40,6 +40,7 @@
+ #include <iomanip>
+
+ #include <boost/foreach.hpp>
++#include <boost/bind.hpp>
+
+ #include "console_bridge/console.h"
+
+@@ -222,7 +223,8 @@ void Bag::setEncryptorPlugin(std::string const& plugin_name, std::string const&
+     if (!chunks_.empty()) {
+         throw BagException("Cannot set encryption plugin after chunks are written");
+     }
+-    encryptor_ = encryptor_loader_.createInstance(plugin_name);
++    auto unmanaged_instance = encryptor_loader_.createUnmanagedInstance(plugin_name);
++    encryptor_ = std::shared_ptr<rosbag::EncryptorBase>(unmanaged_instance);
+     encryptor_->initialize(*this, plugin_param);
+ }
+
diff --git a/ros1_rosbag_storage_vendor/resources/bagcpp_name.diff b/ros1_rosbag_storage_vendor/resources/bagcpp_name.diff
@@ -0,0 +1,29 @@
+diff --git a/tools/rosbag_storage/src/bag.cpp b/tools/rosbag_storage/src/bag.cpp
+index 073072934..13a7fc8fc 100644
+--- a/tools/rosbag_storage/src/bag.cpp
++++ b/tools/rosbag_storage/src/bag.cpp
+@@ -58,12 +58,13 @@ using ros::Time;
+
+ namespace rosbag {
+
+-Bag::Bag() : encryptor_loader_("rosbag_storage", "rosbag::EncryptorBase")
++Bag::Bag() : encryptor_loader_("ros1_rosbag_storage", "rosbag::EncryptorBase")
+ {
+     init();
+ }
+
+-Bag::Bag(string const& filename, uint32_t mode) : encryptor_loader_("rosbag_storage", "rosbag::EncryptorBase")
++Bag::Bag(string const& filename, uint32_t mode)
++: encryptor_loader_("ros1_rosbag_storage", "rosbag::EncryptorBase")
+ {
+     init();
+     open(filename, mode);
+@@ -71,7 +72,7 @@ Bag::Bag(string const& filename, uint32_t mode) : encryptor_loader_("rosbag_stor
+
+ #ifndef BOOST_NO_CXX11_RVALUE_REFERENCES
+
+-Bag::Bag(Bag&& other) : encryptor_loader_("rosbag_storage", "rosbag::EncryptorBase") {
++Bag::Bag(Bag&& other) : encryptor_loader_("ros1_rosbag_storage", "rosbag::EncryptorBase") {
+     init();
+     swap(other);
+ }
diff --git a/ros1_rosbag_storage_vendor/resources/bagh.diff b/ros1_rosbag_storage_vendor/resources/bagh.diff
@@ -0,0 +1,22 @@
+diff --git a/tools/rosbag_storage/include/rosbag/bag.h b/tools/rosbag_storage/include/rosbag/bag.h
+index c65533011..e75ef9c78 100644
+--- a/tools/rosbag_storage/include/rosbag/bag.h
++++ b/tools/rosbag_storage/include/rosbag/bag.h
+@@ -35,6 +35,8 @@
+ #ifndef ROSBAG_BAG_H
+ #define ROSBAG_BAG_H
+
++#include <memory>
++
+ #include "rosbag/macros.h"
+
+ #include "rosbag/buffer.h"
+@@ -354,7 +356,7 @@ private:
+     // Encryptor plugin loader
+     pluginlib::ClassLoader<rosbag::EncryptorBase> encryptor_loader_;
+     // Active encryptor
+-    boost::shared_ptr<rosbag::EncryptorBase> encryptor_;
++    std::shared_ptr<rosbag::EncryptorBase> encryptor_;
+ };
+
+ } // namespace rosbag