New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[foxy backport] Latest rosbag2 #625
Conversation
08ee91a
to
9bd1d6e
Compare
Releasing pybind11_vendor for Foxy sounds fine to me. Note, we should also add it to the ros2.repos file for Foxy if we end up not ignoring rosbag2_py. |
Wasn't sure if we should create a foxy release branch, lmk what you think |
f17351a
to
316d830
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Leaving a comment on every changed line with exact justification
@@ -29,7 +29,7 @@ jobs: | |||
sqlite3_vendor | |||
rosbag2_test_common | |||
rosbag2_tests | |||
target-ros2-distro: rolling | |||
target-ros2-distro: foxy |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Allows action CI to run on the release branch correctly
@@ -14,7 +14,7 @@ | |||
<exec_depend>rosbag2_compression</exec_depend> | |||
<exec_depend>rosbag2_converter_default_plugins</exec_depend> | |||
<exec_depend>rosbag2_cpp</exec_depend> | |||
<exec_depend>rosbag2_py</exec_depend> | |||
<!-- <exec_depend>rosbag2_py</exec_depend> --> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Depends on pybind11_vendor (has been bloomed into Foxy) ros2/ros2#1088
@@ -67,7 +67,7 @@ void SequentialCompressionReader::preprocess_current_file() | |||
* Because we have no way to check whether the bag was written correctly, | |||
* check for the existence of the prefixed file as a fallback. | |||
*/ | |||
const rcpputils::fs::path base{base_folder_}; | |||
rcpputils::fs::path base{base_folder_}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be allowed to be const, but that was fixed in ros2/rcpputils#119 which is API-changing and therefore has not been backported to Foxy
@@ -110,6 +110,7 @@ void SequentialWriter::open( | |||
} | |||
|
|||
bool dir_created = rcpputils::fs::create_directories(db_path); | |||
dir_created &= db_path.is_directory(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This stands in for https://github.com/ros2/rcpputils/pull/98/files (maybe it should be backported to Foxy?)
@@ -166,7 +166,7 @@ function(create_tests_for_rmw_implementation) | |||
rosbag2_transport_add_gmock(test_record_regex | |||
test/rosbag2_transport/test_record_regex.cpp | |||
LINK_LIBS rosbag2_transport | |||
AMENT_DEPS test_msgs rosbag2_test_common | |||
AMENT_DEPS test_msgs rosbag2_cpp rosbag2_test_common |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Allows this to be correctly built against the local workspace, when rosbag2_cpp is binary installed in the environment. This could correctly be an independent fix to master.
throw std::runtime_error( | ||
std::string("Exception on calculating the size of directory :") + uri); | ||
} | ||
metadata.bag_size = rcutils_calculate_directory_size(uri.c_str(), allocator); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adapts to API change from ros2/rcutils#306
316d830
to
e835d2e
Compare
@j-rivero I am wondering if you have any ideas how we can get the ABI report for rosbag2. The problem right now is that ABI report: https://build.ros2.org/job/Fpr__rosbag2__ubuntu_focal_amd64/116/API_5fABI_20report/ |
This pull request has been mentioned on ROS Discourse. There might be relevant details there: https://discourse.ros.org/t/fast-forward-merging-rosbag2-master-api-to-foxy/18927/1 |
@emersonknapp thanks. In the Discourse post we mentioned the following:
Is this commit really the only API/ABI that has changed between the |
This pull request has been mentioned on ROS Discourse. There might be relevant details there: https://discourse.ros.org/t/fast-forward-merging-rosbag2-master-api-to-foxy/18927/13 |
You've misunderstood - that commit contains the changes necessary to make EDIT: It will be possible to list the changed APIs - listing the changed ABIs will be more difficult until we can make the reporting tool work - I have had no luck locally or on the buildfarm yet. |
* First attempt at time splitting logic Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Give default value for max duration Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Added bagfile duration as a storage option Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Initialize duration off of storage optios Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Switch duration in StorageOptions to take int Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Add duration to command line interface Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Suppress lint warning Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Add missing K parameter to the parsing string Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Fix typo Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Switch duration measurement to seconds Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Get project to compile again Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Change logic to allow simultaneous split modes Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Clarifying help comments on splitting behavior Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Fix typo Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Properly split by time Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Initial implementation of duration split test Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Linting whitespace Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Move curly brace for lint Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Remove magic constant Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Finally found and fixed unit error Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Force starting_time to be a system_clock time_point Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Change another high_resolution clock instance Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Convert everything to steady_clock Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Convert everything to use high_resolution_clock Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Switch Windows testing code to newer version Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com>
* expect only 60 percent of messages to arrive Signed-off-by: Karsten Knese <karsten@openrobotics.org> * test for only one message Signed-off-by: Karsten Knese <karsten@openrobotics.org>
* minimal c++ API test Signed-off-by: Karsten Knese <karsten@openrobotics.org> * linters Signed-off-by: Karsten Knese <karsten@openrobotics.org>
* Add per-message ZSTD compression This implements the per-messages compression and decompression functions for the ZSTD compressor and also adds unit tests for them. Distro A, OPSEC #2893 Signed-off-by: P. J. Reed <preed@swri.org>
Signed-off-by: Karsten Knese <karsten@openrobotics.org>
Signed-off-by: Dirk Thomas <dirk-thomas@users.noreply.github.com>
* reenable cppcheck Signed-off-by: Karsten Knese <karsten@openrobotics.org> * suppress unknown macro inline Signed-off-by: Karsten Knese <karsten@openrobotics.org>
* Use foxy testing apt repos to install linters for Actions Signed-off-by: Emerson Knapp <eknapp@amazon.com>
Signed-off-by: Emerson Knapp <eknapp@amazon.com>
* Consolidate ZSTD utility functions The zstd_compressor and zstd_decompressor implementations had a number of duplicated utility functions between them; this consolidates them into one file. Signed-off-by: P. J. Reed <preed@swri.org>
* added db directory creation to storage factory Signed-off-by: Marwan Taher <marokhaled99@gmail.com> * moved db directory creation to rosbag2_cpp Signed-off-by: Marwan Taher <marokhaled99@gmail.com> * rasing exception if dir already exists Signed-off-by: Marwan Taher <marokhaled99@gmail.com> * removed dir creation from record.py, added dir creation to sequential_compression_writer and refactored dir creation in sequential_writer Signed-off-by: Marwan Taher <marokhaled99@gmail.com> * fixed failing tests Signed-off-by: Marwan Taher <marokhaled99@gmail.com> * fixing review comments Signed-off-by: Marwan Taher <marokhaled99@gmail.com> * Apply suggestions from code review Co-authored-by: Karsten Knese <Karsten1987@users.noreply.github.com> Signed-off-by: Emerson Knapp <eknapp@amazon.com>
Signed-off-by: Barry Xu <barry.xu@sony.com>
* Updating performance writer - utilizes --storage-config-file with two examples - message reading from metadata yaml file (ready for cache double-buffers PR 546) - removed performance-affecting outputs - supports bag splitting in benchmarking Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * typo fix Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * corrected compression so that it doesn't break the command line in case it is empty (the default) Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * address CI warns/fails on MacOS and Windows Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>
1. play a specific known message type even if some unknown types exist. 2. add a warning message while a message type library not exist. Signed-off-by: Chen Lihui <lihui.chen@sony.com>
…bject (#593) Signed-off-by: Josh Langsfeld <josh.langsfeld@gmail.com>
* Use optimized pragmas by default in sqlite storage. Added option to use former behavior Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>
Signed-off-by: Ivan Santiago Paunovic <ivanpauno@ekumenlabs.com>
#608) Signed-off-by: Emerson Knapp <eknapp@amazon.com>
* Change all 'fastrtps-unspecified' duration values in test resources to 0 for cross-implementation readability, pending a clarification of the rmw qos duration api Signed-off-by: Emerson Knapp <eknapp@amazon.com>
…ge (#603) * Mutex-protected writes and topic creation/removal in sqlite_storage Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>
…ing team (#614) Signed-off-by: Emerson Knapp <eknapp@amazon.com>
* Regex and exclude options for recording topics - you can use -e or --regex option now to specify how topics are recorded - can use -x or --exclude to exclude topics from recording - regex is exclusive with -a and specifying topics (for simplicity) Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>
…to find bagfiles in incorrectly-written metadata (#612) * Deduplicate sequentialcompressionreader business logic, add fallback to find bagfiles in incorrectly-written metadata Signed-off-by: Emerson Knapp <eknapp@amazon.com>
* Refactoring of rosbag2 performance package: - renamed since now it no longer benchmarks writer only - generalized byte_producer so that it uses a callback instead of queue, so it can be reused in publisher scheme Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Benchmark publishers based on yaml configuration - can specify multiple groups of publishers (see attached example yaml) - reuses byte producer Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Applying configured QoS settings for publishers. Also included in yaml example. Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * linters Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Towards common configuration - separating out common structures - utility class for common parameter parsing Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Barebone launchfile for benchmarks. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * writer benchmark adapted to yaml file and publisher groups Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * refactored result writing and bag parameters Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * linters Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Launchfile for benchmarks Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Change storage config file from non optimized to resilient Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Max bag size for benchmark launchfile. Launchfile refactor. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Copy yaml configs after benchmark is finished. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Benchmark results csv file extended Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * added disclaimer about random data and compression Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Report gen tool for benchmarks Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Benchmarks out dir name changed Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * results writer node Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * documentation Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Transport and transportless in launchfile Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Benchmark launchfile refactor Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Wait for rosbag listening in benchmark launchfile Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Uncrustify and some comments for benchmarking tools Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Added new producers config for benchmarks Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Missing parameters in transport benchmark Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> added comment in storage_optimized.yaml * Missing rosbag record parameters in transport benchmark Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Wait for subscriptions parameter in producers config Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * moved utils code from header to source Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * changed compiler shortcut uint to unsigned int (should fix Windows build) Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> Co-authored-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> Signed-off-by: Emerson Knapp <eknapp@amazon.com>
* Synchronize compression shutdown correctly, avoiding occasional deadlock Signed-off-by: Emerson Knapp <eknapp@amazon.com>
* Fix relative metadata path writing in compression by deduplicating business logic Signed-off-by: Emerson Knapp <eknapp@amazon.com>
* Regex and exclude fix for rosbag recorder Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>
Several packages were failing to build from source when rosbag2_storage was installed from a binary package because the path to the binary install was being added to CMake's include path before the workspace path. This tweaks the dependencies and include orders to ensure the source workspace path is preferred. Fixes #583 Distro A, OPSEC #4584 Signed-off-by: P. J. Reed <preed@swri.org>
* Implement streaming compression/decompression Distro A, OPSEC #4584 Signed-off-by: P. J. Reed <preed@swri.org>
Make compressor implementation into a plugin via pluginlib Signed-off-by: Emerson Knapp <eknapp@amazon.com>
* Update CHANGELOG * 0.6.0 Signed-off-by: Emerson Knapp <eknapp@amazon.com>
Signed-off-by: Emerson Knapp <eknapp@amazon.com>
Signed-off-by: Emerson Knapp <eknapp@amazon.com>
7b52eda
to
e5c7951
Compare
Is there a next step for this MR? |
It looks like |
Yes - we are considering releasing the rosbag2 as |
After deciding to go forward with the Please see the Discourse post for more context |
Thank you! |
Proposal to fix #657
Backport the entire
master
branch to foxy. This is a proposal - not a final review, just to see what changes would be needed to be compatible.Notes: