[foxy backport] Latest rosbag2 #625
Closed
Commits on Feb 17, 2021
-
-
Add split by time to recording (#409)
* First attempt at time splitting logic Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Give default value for max duration Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Added bagfile duration as a storage option Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Initialize duration off of storage optios Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Switch duration in StorageOptions to take int Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Add duration to command line interface Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Suppress lint warning Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Add missing K parameter to the parsing string Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Fix typo Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Switch duration measurement to seconds Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Get project to compile again Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Change logic to allow simultaneous split modes Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Clarifying help comments on splitting behavior Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Fix typo Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Properly split by time Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Initial implementation of duration split test Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Linting whitespace Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Move curly brace for lint Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Remove magic constant Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Finally found and fixed unit error Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Force starting_time to be a system_clock time_point Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Change another high_resolution clock instance Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Convert everything to steady_clock Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Convert everything to use high_resolution_clock Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com> * Switch Windows testing code to newer version Distro A, OPSEC #2893 Signed-off-by: Jacob Hassold <jhassold@dcscorp.com>
-
More reliable topic remapping test (#456)
* expect only 60 percent of messages to arrive Signed-off-by: Karsten Knese <karsten@openrobotics.org> * test for only one message Signed-off-by: Karsten Knese <karsten@openrobotics.org>
-
* minimal c++ API test Signed-off-by: Karsten Knese <karsten@openrobotics.org> * linters Signed-off-by: Karsten Knese <karsten@openrobotics.org>
-
Add per-message ZSTD compression (#418)
* Add per-message ZSTD compression This implements the per-messages compression and decompression functions for the ZSTD compressor and also adds unit tests for them. Distro A, OPSEC #2893 Signed-off-by: P. J. Reed <preed@swri.org>
-
comment out unused variable (#460)
Signed-off-by: Karsten Knese <karsten@openrobotics.org>
-
use a single temp dir for the test class (#462)
Signed-off-by: Dirk Thomas <dirk-thomas@users.noreply.github.com>
-
reenable cppcheck for rosbag2_transport (#461)
* reenable cppcheck Signed-off-by: Karsten Knese <karsten@openrobotics.org> * suppress unknown macro inline Signed-off-by: Karsten Knese <karsten@openrobotics.org>
-
Use foxy testing apt repos to install linters for Actions (#463)
* Use foxy testing apt repos to install linters for Actions Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
Consolidate ZSTD utility functions (#459)
* Consolidate ZSTD utility functions The zstd_compressor and zstd_decompressor implementations had a number of duplicated utility functions between them; this consolidates them into one file. Signed-off-by: P. J. Reed <preed@swri.org>
-
Adding db directory creation to rosbag2_cpp (#450)
* added db directory creation to storage factory Signed-off-by: Marwan Taher <marokhaled99@gmail.com> * moved db directory creation to rosbag2_cpp Signed-off-by: Marwan Taher <marokhaled99@gmail.com> * rasing exception if dir already exists Signed-off-by: Marwan Taher <marokhaled99@gmail.com> * removed dir creation from record.py, added dir creation to sequential_compression_writer and refactored dir creation in sequential_writer Signed-off-by: Marwan Taher <marokhaled99@gmail.com> * fixed failing tests Signed-off-by: Marwan Taher <marokhaled99@gmail.com> * fixing review comments Signed-off-by: Marwan Taher <marokhaled99@gmail.com> * Apply suggestions from code review Co-authored-by: Karsten Knese <Karsten1987@users.noreply.github.com> Signed-off-by: Emerson Knapp <eknapp@amazon.com>
2 people authored and committedFeb 17, 2021 -
move wait_for_shutdown() call out of the context manager (#466)
Signed-off-by: Dirk Thomas <dirk-thomas@users.noreply.github.com>
-
introduce ros2 bag list <plugins> (#468)
* introduce ros2 bag list <plugins> Signed-off-by: Karsten Knese <karsten@openrobotics.org> * Apply suggestions from code review Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
introduce defaults for the C++ API (#452)
* minimal c++ API test Signed-off-by: Karsten Knese <karsten@openrobotics.org> * introduce defaults Signed-off-by: Karsten Knese <karsten@openrobotics.org> * open overload for simple uri string Signed-off-by: Karsten Knese <karsten@openrobotics.org> * set sqlite3 as a constant Signed-off-by: Karsten Knese <karsten@openrobotics.org>
-
add wait for closed file handles on Windows (#470)
Signed-off-by: Dirk Thomas <dirk-thomas@users.noreply.github.com>
-
Signed-off-by: Dirk Thomas <dirk-thomas@users.noreply.github.com>
-
Fix typo in error message (#475)
Signed-off-by: Jacob Perron <jacob@openrobotics.org>
-
performance testing packages (#442)
* Init commit Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Readme and package dependencies cleaned. Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Benchmark launch file Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Dummy warmup publishers and raport gen barebone Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * added todo for system usage Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Added I/O benchmarking section Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * config file wip Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Pc2 worker, bench and raport dirs Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * added benchmark sample config Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Moved scripts to a dir Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Simple system monitor Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Updated readme Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * added notice Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Update readme.md Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Mkdir fix Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Update readme.md Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * plotting WIP script plus an example file with logs Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Raport gen, monitor with iotop Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Disk bw fix. Instances in raport fix. Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Typo fix Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Fix in memory time axis Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Another axis fix Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * sqlite.conf for patched version of rosbag2 Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Improved plotting (added message count, count start includes fractions of second, color etc) Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * data examples Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * large examples over 10000 messages Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Byte array worker, instantiation improvements Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * total captured in raport Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * voyager wip Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * workers refactored Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * launchfile refactor Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * More improvement and voyager case Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * removed some commented out lines Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Update readme.md Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * moved directories of performance packages into rosbag2_performance folder Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * commented out 1000x1000 case in voyager Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Update readme.md Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * added html template to exported files Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * First version of no-transport benchmarking - Scalable with number of producer threads - Uses queues to simulate message loss in callback queues. Unlike some other implementations allows to count loss messages. - ASCII visualisation of work (better than it sounds). Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Improved parameters and added scripting to run batches. These were used to acquire first results Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Update readme.md Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Changes towards resolving WIP status - Added copyright and license to files - Split and updated READMEs - Refactored writer_benchmarking, splitting classes into separate files - added ctrl-c handling to writer_benchmarking, in script and in producers Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * lint changes Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Removed large files from repo Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * PEP8 issues refactor, rosbag2_benchmarking package info updated. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * fixed an issue with SQLite crash with cache>1 due to released message memory Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * uncrustify Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Introduced BUILD_PERFORMANCE flag to control the package build. Use colcon build --cmake-args -DBUILD_PERFORMANCE=1. Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * linter changes and missingcopyrights Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Update README.md Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Removed unnecessary wording Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Fixed bad spelling raport->report Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Report stats from time 0 Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * minor cleaning of writer_benchmarking package Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * removed other performance packages from PR. These will be added later and reviewed separately Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * removed accidental change Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Applied review changes Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * enforced shared pointer in queue api, added whitespaces Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * mVar -> _var, camelCase -> snake_case Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * writing results to a csv file Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * style touchups and linters Signed-off-by: Karsten Knese <karsten@openrobotics.org> * A fix in fio command, thanks to Karsten Knese Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Added spinning (in thread) to handle parameters Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Typo fix Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Applied review remarks - node name changed to reflect package name - value now in main and constructor takes name argument - benchmark folder management adjusted to changes after rebase - benchmark now remaps node (adding "_batch"). - removed unnecessary find on rosbag2_transport Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * portable auto type to remove Windows warning. Thanks KK Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> Co-authored-by: Piotr Jaroszek <jaroszekpiotr@gmail.com> Co-authored-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> Co-authored-by: Adam Dąbrowski <1961316-adamdbrw@users.noreply.gitlab.com> Co-authored-by: Karsten Knese <karsten@openrobotics.org> Signed-off-by: Emerson Knapp <eknapp@amazon.com>
5 people authored and committedFeb 17, 2021 -
Add pytest.ini back to ros2bag. (#492)
Signed-off-by: Chris Lalancette <clalancette@openrobotics.org>
-
rosbag2_py reader and writer (#308)
* Initialize new package rosbag2_py Signed-off-by: Jacob Perron <jacob@openrobotics.org> (cherry picked from commit f2ec0b7) * Proof-of-concept implementation using pybind11 Expose the sequential reader to iterate over messages from Python. Signed-off-by: Jacob Perron <jacob@openrobotics.org> (cherry picked from commit dc68894) * pybind StorageOptions and ConverterOptions; linter fixes Signed-off-by: Mabel Zhang <mabel@openrobotics.org> (cherry picked from commit 91f6763) * return timestamp Signed-off-by: Mabel Zhang <mabel@openrobotics.org> (cherry picked from commit deafbf8) * simplify binding of structs Signed-off-by: Mabel Zhang <mabel@openrobotics.org> (cherry picked from commit da561cc) * pybind TopicMetadata Signed-off-by: Mabel Zhang <mabel@openrobotics.org> (cherry picked from commit 4a4e1b8) * namespace rosbag2 -> rosbag2_cpp Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * small fixes Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * initial pytests Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * wrap Reader instead of SequentialReader Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * small fixes for PR comments Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * change workflow to autotest feature branch Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * revert workflow Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * added writer and writer tests Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * fix test path setup Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * fix and make more rigorous writer test Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * wrapper and test for topic filter Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * linters. change pybind package Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * simplify verbose import Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * tidy up Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * explicit subclasses of Reader and Writer Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * Use template specialization instead of inheritence This makes the implementation slightly more compact. And adding a new specialization should be more straight forward as we only have to modify the cpp file in one place. Signed-off-by: Jacob Perron <jacob@openrobotics.org> * Remove temporary variable Signed-off-by: Jacob Perron <jacob@openrobotics.org> * Remove unused member variable Signed-off-by: Jacob Perron <jacob@openrobotics.org> * Move to initializer list Signed-off-by: Jacob Perron <jacob@openrobotics.org> * refactor into reader writer storage Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * cleanup includes Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * switch back to pybind11_vendor Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * Packages dependency for CI Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * printout to debug CI test failure Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * use tempfile for writer test, add exec_depend Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * remove exec_depend Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * use pytest fixture tmp_path; add rosbag2_py to rosbag2 Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * add makedirs back to see CI result Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * fix import error Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * cleanup Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * add modules explicitly for windows CI Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * cleanup Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * try more <test_depend>s for mac test failure Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * wrap structs with named arguments for Python keyword params Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * add pytest.ini Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * add rosbag db3-shm and db3-wal files for macOS test Signed-off-by: Mabel Zhang <mabel@openrobotics.org> Co-authored-by: Jacob Perron <jacob@openrobotics.org> Co-authored-by: Andreas Klintberg <ankl@kth.se> Signed-off-by: Emerson Knapp <eknapp@amazon.com>
3 people authored and committedFeb 17, 2021 -
AMENT_IGNORE rosbag2_py for now (#509)
* AMENT_IGNORE rosbag2_py Signed-off-by: Mabel Zhang <mabel@openrobotics.org> * remove rosbag2_py from meta package Signed-off-by: Mabel Zhang <mabel@openrobotics.org>
-
Add compression and batch size to the README (#510)
* Add compression and batch size to the README Signed-off-by: Devin Bonnie <dbbonnie@amazon.com> * Update with split by duration and message compression mode Signed-off-by: Devin Bonnie <dbbonnie@amazon.com>
-
disable sanitizer by default (#517)
Signed-off-by: Dirk Thomas <dirk-thomas@users.noreply.github.com>
-
Mark flaky tests as xfail for now (#520)
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
-
Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
-
[ros2bag test_record] Gets rid of time.sleep and move to using comman…
…d.wait_for_output (#525) * Uses bag_command.wait_for_output with expected string instead of time.sleep in tests Signed-off-by: Jaison Titus <jaisontj@amazon.com> * Fixes code style errors Signed-off-by: Jaison Titus <jaisontj@amazon.com> * Moves to asserting expected output match outside of the process context to account for cases where wait_for_output is maybe called after the expected output is already printed. Signed-off-by: Jaison Titus <jaisontj@amazon.com> * Defines timeout with variables and better error messages for failed tests. Signed-off-by: Jaison Titus <jaisontj@amazon.com> * Fixes flake8 errors Signed-off-by: Jaison Titus <jaisontj@amazon.com>
-
Remove some code duplication between SequentialWriter and SequentialC…
…ompressionWriter (#527) Remove code duplication between SequentialWriter and SequentialCompressionWriter Signed-off-by: Jaison Titus <jaisontj@amazon.com>
-
Fix rosbag2_py bug when using libc++ (#529)
Signed-off-by: Mabel Zhang <mabel@openrobotics.org> Signed-off-by: Ivan Santiago Paunovic <ivanpauno@ekumenlabs.com> Co-authored-by: Mabel Zhang <mabel@openrobotics.org>
2 people authored and committedFeb 17, 2021 -
Fix rosbag2_py on Windows debug and stop ignoring the package (#531)
Signed-off-by: Ivan Santiago Paunovic <ivanpauno@ekumenlabs.com>
-
Do not expect empty StorageOptions URI to work in *CompressionWriterT…
…est (#526) Signed-off-by: Christophe Bedard <bedard.christophe@gmail.com>
-
Update the package.xml files with the latest Open Robotics maintainers (
#535) Signed-off-by: Michael Jeronimo <michael.jeronimo@openrobotics.org>
-
SequentialWriter to cache by message size instead of message count (#530
) * Fixes #464 - caches by message size and not message count Signed-off-by: Jaison Titus <jaisontj@amazon.com>
-
Change default cache size for sequential_writer to a non zero value (#…
…533) * Change default value for max-cache-size to 1MB Signed-off-by: Jaison Titus <jaisontj@amazon.com>
-
Removed duplicated code in record (#534)
* Removed duplicated code for record. Signed-off-by: Jaison Titus <jaisontj@amazon.com>
-
Workaround pybind11 bug on Windows when CMAKE_BUILD_TYPE=RelWithDebI…
…nfo (#538) Signed-off-by: Ivan Santiago Paunovic <ivanpauno@ekumenlabs.com>
-
Add record test for ros2bag (#523)
* Add record test for ros2bag A nominal smoke test to confirm that the tool is working. Signed-off-by: Jacob Perron <jacob@openrobotics.org> * Remove unused import Signed-off-by: Jacob Perron <jacob@openrobotics.org> * Fix lint Signed-off-by: Jacob Perron <jacob@openrobotics.org> * Expect exit code 2 from rclcpp Signed-off-by: Jacob Perron <jacob@openrobotics.org> * Remove unused import Signed-off-by: Jacob Perron <jacob@openrobotics.org> * Fix test for Windows Signed-off-by: Jacob Perron <jacob@openrobotics.org>
-
if cache data exists, it needs to flush the data into the storage bef…
-
Update deprecated qos policy value names (#548)
Signed-off-by: Ivan Santiago Paunovic <ivanpauno@ekumenlabs.com>
-
* add storage_config_uri Signed-off-by: Karsten Knese <karsten@openrobotics.org> * linters and tests Signed-off-by: Karsten Knese <karsten@openrobotics.org> * move storage options to rosbag2_storage Signed-off-by: Karsten Knese <Karsten1987@users.noreply.github.com> * use storage options to open storage backends Signed-off-by: Karsten Knese <Karsten1987@users.noreply.github.com> * add rosbag2_py to metapackage Signed-off-by: Karsten Knese <Karsten1987@users.noreply.github.com>
-
Signed-off-by: Karsten Knese <Karsten1987@users.noreply.github.com>
-
List all storage plugins in plugin xml file (#554)
Signed-off-by: Barry Xu <barry.xu@sony.com>
-
* read yaml config file Signed-off-by: Karsten Knese <karsten@openrobotics.org> * address review comments Signed-off-by: Karsten Knese <Karsten1987@users.noreply.github.com> * Sqlite storage pragmas handling and validation - includes documentation of storage-config-file option - includes adaptation of benchmarking script to cache size semantic change - tests with valid and invalid storage file Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Applied review remarks Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * - Typo fix - change pragma tests order as a workaround to fix tests issue on Windows: exception throwing tests don't clean temp folders Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Applied review: moved some parsing and key extraction to parse function Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> Co-authored-by: Adam Dabrowski <adam.dabrowski@robotec.ai> Signed-off-by: Emerson Knapp <eknapp@amazon.com>
2 people authored and committedFeb 17, 2021 -
* Update changelogs and setup.py Signed-off-by: Michael Jeronimo <michael.jeronimo@openrobotics.org> * Update versions in new packages to match (so that we can run bloom) Signed-off-by: Michael Jeronimo <michael.jeronimo@openrobotics.org> * 0.4.0 Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
Bump action-ros-ci to 0.1.0 (#564)
Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
Sqlite storage double buffering (#546)
* Double buffers Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Circular queue and FLUSH option as define Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Minor naming and lexical fixes. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Removed FLUSH_BUFFERS define check. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Sqlite3 storage logging fixes. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Sqlite3 storage circular buffer with pre allocated memory. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Sqlite3 storage buffers moved to shared_ptrs. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Uncrustify Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Moved double buffers to writer Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Buffer layer reset in seq compression writer in rosbag2 cpp Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Buffer layer for rosbag2 writer refactor Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Changed buffers in BufferLayer to std vectors. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * BufferLayer uncrustify Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Removed non-applicable test for writer cache. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * BufferLayer review fixes Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Rosbag metadata msgs count fixed for BufferLayer Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Condition variable for buffer layer sync. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Fixed buffer locks Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Buffers in BufferLayer refactored, moved into new class Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Buffer layer split bags fixed. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Storage options include fix in buffer layer header. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Mutex around swapping buffers in buffer layer. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Fixed cache 0 bug in buffer layer. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Minor buffer layer refactor. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Counting messages in writer refactored. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Changed default cache size to 100Mb and updated parameter description Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Applied review remarks: - significant refactoring: separation of cache classes - applied suggested improvements - some renaming - reduce code duplication that would otherwise increase with cache refactor, between compression and plain writers Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Applied review comments - cache consumer now takes a callback and is independent of storage - namespace changes, renaming, cleaning - counting and logging messages by topic Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * linter Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Changes after review: fixing flushing, topic counts, and more Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Fix for splitting - flushing state now correctly turns off Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * cache classes documentation Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * simplified signature Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * a couple of tests for cache Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * address review: explicit constructor and doxygen styling Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Windows warnings fix Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * fixed type mismatch warning on Windows Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * added minor comment Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> Co-authored-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> Signed-off-by: Emerson Knapp <eknapp@amazon.com>
2 people authored and committedFeb 17, 2021 -
Add back rosbag2_cpp::StorageOptions as deprecated (#563)
The struct was removed in #493, but in order to avoid a hard-break for users coming from Foxy I've added it back with a deprecation warning. Signed-off-by: Jacob Perron <jacob@openrobotics.org>
-
Update codes since rcutils_calculate_directory_size() is changed (#567)
Signed-off-by: Barry Xu <barry.xu@sony.com>
-
Signed-off-by: Chris Lalancette <clalancette@openrobotics.org>
-
Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
Stabilize test_record by reducing copies of executors and messages (#576
-
Compress bag files in separate threads (#506)
* Compress bag files in separate threads This offloads all compression into separate threads. Doing so helps to prevent rosbag2 from dropping messages due to the main thread being busy, and it also helps to improve performance by spreading the work across multiple CPU cores. It uses a producer/consumer model with a fixed number of pre-allocated threads that consume messages or files off of an incoming queue. This adds two new command line options: - --compression-queue-size - The number of messages or files that can be sitting in the queue waiting for a thread to consume them. If the queue is full, older messages or files will be discarded; this would lead to either messages being discarded or files not being compressed. The default value is 1, which should be sufficient if the system is capable of keeping up with the work load. - --compression-threads - The number of threads that can be compressing data at once. The default is 0, and values less than 1 will be interpreted to mean the number of concurrent threads supported by the current hardware. Closes #274. Distribution Statement A; OPSEC #2893 Signed-off-by: P. J. Reed <preed@swri.org> Co-authored-by: Emerson Knapp <eknapp@amazon.com> -
Only dereference the data pointer if it is valid. (#581)
If the static_cast returned a nullptr, then just print the message and don't attempt to deference it. Signed-off-by: Chris Lalancette <clalancette@openrobotics.org>
-
Re-enable Action CI for PRs (#465)
* Add target-ros2-distro to action CI to allow it to run correctly Signed-off-by: Emerson Knapp <emerson.b.knapp@gmail.com>
-
Fix a bug on parsing wrong description in plugin xml file (#578)
Signed-off-by: Barry Xu <barry.xu@sony.com>
-
Updating performance writer (#570)
* Updating performance writer - utilizes --storage-config-file with two examples - message reading from metadata yaml file (ready for cache double-buffers PR 546) - removed performance-affecting outputs - supports bag splitting in benchmarking Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * typo fix Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * corrected compression so that it doesn't break the command line in case it is empty (the default) Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * address CI warns/fails on MacOS and Windows Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>
-
Use std::filesystem compliant non-member
existsfunction for path o……bject (#593) Signed-off-by: Josh Langsfeld <josh.langsfeld@gmail.com>
-
SQLite storage optimized by default (#568)
* Use optimized pragmas by default in sqlite storage. Added option to use former behavior Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>
-
Signed-off-by: Ivan Santiago Paunovic <ivanpauno@ekumenlabs.com>
-
Add mjeronimo to codeowners to be auto-assigned in the review rotation (
#608) Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
Fix the tests on cyclonedds by translating qos duration values (#606)
* Change all 'fastrtps-unspecified' duration values in test resources to 0 for cross-implementation readability, pending a clarification of the rmw qos duration api Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
Mutex protection for db writing and stl collections in writer & stora…
…ge (#603) * Mutex-protected writes and topic creation/removal in sqlite_storage Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>
-
Remove mjeronimo from CODEOWNERS in favor of having him on the review…
…ing team (#614) Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
Recorder --regex and --exclude options (#604)
* Regex and exclude options for recording topics - you can use -e or --regex option now to specify how topics are recorded - can use -x or --exclude to exclude topics from recording - regex is exclusive with -a and specifying topics (for simplicity) Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai>
-
Deduplicate SequentialCompressionReader business logic, add fallback …
…to find bagfiles in incorrectly-written metadata (#612) * Deduplicate sequentialcompressionreader business logic, add fallback to find bagfiles in incorrectly-written metadata Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
Performance benchmarking refactor (#594)
* Refactoring of rosbag2 performance package: - renamed since now it no longer benchmarks writer only - generalized byte_producer so that it uses a callback instead of queue, so it can be reused in publisher scheme Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Benchmark publishers based on yaml configuration - can specify multiple groups of publishers (see attached example yaml) - reuses byte producer Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Applying configured QoS settings for publishers. Also included in yaml example. Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * linters Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Towards common configuration - separating out common structures - utility class for common parameter parsing Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Barebone launchfile for benchmarks. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * writer benchmark adapted to yaml file and publisher groups Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * refactored result writing and bag parameters Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * linters Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Launchfile for benchmarks Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Change storage config file from non optimized to resilient Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Max bag size for benchmark launchfile. Launchfile refactor. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Copy yaml configs after benchmark is finished. Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Benchmark results csv file extended Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * added disclaimer about random data and compression Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Report gen tool for benchmarks Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Benchmarks out dir name changed Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * results writer node Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * documentation Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * Transport and transportless in launchfile Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Benchmark launchfile refactor Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Wait for rosbag listening in benchmark launchfile Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Uncrustify and some comments for benchmarking tools Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Added new producers config for benchmarks Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Missing parameters in transport benchmark Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> added comment in storage_optimized.yaml * Missing rosbag record parameters in transport benchmark Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * Wait for subscriptions parameter in producers config Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> * moved utils code from header to source Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> * changed compiler shortcut uint to unsigned int (should fix Windows build) Signed-off-by: Adam Dabrowski <adam.dabrowski@robotec.ai> Co-authored-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai> Signed-off-by: Emerson Knapp <eknapp@amazon.com>
2 people authored and committedFeb 17, 2021 -
Fix deadlock race condition on compression shutdown (#616)
* Synchronize compression shutdown correctly, avoiding occasional deadlock Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
Fix relative metadata paths in SequentialCompressionWriter (#613)
* Fix relative metadata path writing in compression by deduplicating business logic Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
Regex and exclude fix for rosbag recorder (#620)
* Regex and exclude fix for rosbag recorder Signed-off-by: Piotr Jaroszek <piotr.jaroszek@robotec.ai>
-
Fix build issues when rosbag2_storage is binary installed (#585)
Several packages were failing to build from source when rosbag2_storage was installed from a binary package because the path to the binary install was being added to CMake's include path before the workspace path. This tweaks the dependencies and include orders to ensure the source workspace path is preferred. Fixes #583 Distro A, OPSEC #4584 Signed-off-by: P. J. Reed <preed@swri.org>
-
Use ZSTD's streaming interface for [de]compressing files (#543)
* Implement streaming compression/decompression Distro A, OPSEC #4584 Signed-off-by: P. J. Reed <preed@swri.org>
-
Make compressor implementations into a plugin via pluginlib (#624)
Make compressor implementation into a plugin via pluginlib Signed-off-by: Emerson Knapp <eknapp@amazon.com>
-
Prepare bloom release 0.6.0 (#628)
* Update CHANGELOG * 0.6.0 Signed-off-by: Emerson Knapp <eknapp@amazon.com>
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.