Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple serialization formats (encodings) in a single bag #52

Open
anhosi opened this issue Oct 25, 2018 · 1 comment
Open
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@anhosi
Copy link
Contributor

anhosi commented Oct 25, 2018

It should be possible to have data with different serialization formats (encodings) in the same bag.
All messages for a given topic, however, must be encoded with the same serialization format.

@clalancette clalancette added the enhancement New feature or request label Oct 25, 2018
sriramster pushed a commit to sriramster/rosbag2 that referenced this issue Feb 28, 2019
* Adapt new interface

* Try to write and read rcutils_char_array_t BLOBs in sqlite

* Add simple test for arbitrary char ptr

* Refactor SqliteWrapper and add tests

* Write and read actual timestamp from serialized message and add relative tests

* Add SqliteStatementWrapper class and refactor SqliteStorage and SqliteWrapper

* Refactor test fixture

* ros2GH-50 Assert message content in write_integration_test, and remove TODOs

* ros2GH-50 Remove sqlite_storage_plugin unit tests

* ros2GH-50 Refactor SqliteStatements and SqliteStorage

* ros2GH-50 Fix build after rebase

* ros2GH-50 Make has_next() method no more const

* ros2GH-52 Extend statement wrapper with a generic bind

* ros2GH-50 Refactor after rebase

* ros2GH-59 cleanup db interface

- Remove virtual on methods as this was added only for unit tests. We
  decided to use only integration tests for the sqlite plugins.
- Changes semantics of SqliteStatement: represents always a prepared
  statement if not null.
- Ensures that a SqliteStatementWrapper cannot be copied and does not
  publicly expose its sqlite_stmt as this would cause memory corruption.

* ros2GH-59 Introduce general read interface for sqlite statements

- Uses a std::tuple for row data
- Exposes an iterator interface for the query result

* ros2GH-59 Cleanup: remove unused files

* ros2GH-59 make sqlite interface fluent

* ros2GH-59 move creation of serialized message to rosbag2_storage

This is not storage plugin specific but will be needed by most (if
not all) plugins.

* Change rcutil_char_array_t to rmw_serialized_message_t in subscriber

* Remove debugging output in test
sriramster pushed a commit to sriramster/rosbag2 that referenced this issue Feb 28, 2019
* ros2GH-52 Extend db schema to include topic meta data

- Two table db layout (messages and topics)
- Messages table references topics table but without foreign key for
  improved write performance
- Create_topic must be called for every topic prior to storing a
  message of this topic.
- Sqlite_storage caches all known topics
- At least for now the type information is stored as a simple string.

* ros2GH-54 Make first rcl subscription prototype work

* ros2GH-54 find type name from topic

* ros2GH-54 Publish messages from database knowing only topic name and pass topic name by terminal

* ros2GH-54 Refactoring of typesupport helpers

* ros2GH-54 Use c++ typesupport

* ros2GH-54 Use cpp typesupport and rclcpp::Node for publisher

* ros2GH-54 Add raw subscription and use in rosbag_record

* ros2GH-54 Add Rosbag2Node and Rosbag2Publisher classes and use them in Rosbag2::play

* ros2GH-54 Rename Rosbag2Publisher to RawPublisher

* ros2GH-54 Minor refactoring of Rosbag2Node

* ros2GH-54 Extract and test waiting for topic into its own method

* ros2GH-54 Fix read integration tests and linters

* ros2GH-55 Refactor Rosbag2Node::create_raw_publisher()

* ros2GH-54 Add subscription method to rosbag node

* ros2GH-54 Keep subscription alive

* ros2GH-54: Extract subscription to correct class

* ros2GH-55 Change interface of raw_publisher to match subscriber

* ros2GH-54 Add test for rosbag node

* ros2GH-54 Unfriend rclcpp class

* ros2GH-54 Make test more robust

* ros2GH-54 Fix build

* ros2GH-54 Minor cleanup and documentation

* ros2GH-55 Minor refactoring + TODO comment

* ros2GH-54 Change dynamic library folder on Windows

* ros2GH-54 Fix build

* ros2GH-54 Add shutdown to test

* ros2GH-55 Add test helpers methods for usage in multiple tests

* ros2GH-55 Add new method to read all topics and types in BaseReadInterface and use it in Rosbag2::play

* ros2GH-55 Fix gcc and msvc

* ros2GH-54 Rename raw to generic in publisher/subscriber

* ros2GH-55 Check that topic and associated type in bag file are well defined before playing back messages

* ros2GH-54 Prevent unnecessary error message loading storage

* ros2GH-54 Fix memory leak

* ros2GH-54 stabilize node test

* ros2GH-55 Check if database exists when opening storage with READ_ONLY flag

* ros2GH-54 Minor cleanup of subscriber

* ros2GH-54 Wait a small amount of time to let node discover other nodes

* Add logging to false case

* ros2GH-54 Catch exceptions and exit cleanly

* Use rmw_serialized_message_t and rcutils_char_array_t consistently

* ros2GH-4 Refactoring for correctness

- pass a few strings as const reference
- throw error when no topics could be found

* Improve error messages when loading plugins

* alphabetical order

* type_id -> type
@emersonknapp emersonknapp added the help wanted Extra attention is needed label Jan 29, 2020
@amacneil
Copy link
Contributor

This is now supported in MCAP and the rosbag2_storage_mcap plugin.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

4 participants