Rosbag splitting in Writer #185

zmichaels11 · 2019-10-17T23:28:58Z

This is part of an effort to rework PR #158 into multiple, smaller PRs.

Changes

open in BaseIOInterface now must point to the storage file
Writer is responsible for writing and generating BagMetadata
Writer rolls over to a new storage when the bagfile exceeds max_bagfile_size by opening a new storage.

Dependent PRs

Issues

ros-security/aws-roadmap#11

zmichaels11 · 2019-10-18T19:15:59Z

Epr failed due to a slight change in db name - this PR removed the extension since Writer needs to specify the name and Writer can't know the file extension until after the storage opens. It also cannot assume db3 in case the storage plugin does not back as SQLite3.

I can resolve this by either:

blocking until Reader changes are synced up and rewriting all e2e tests
Append the extension in open and introduce a get_relative_paths to io interfaces

I'll probably do the second and introduce a new PR that should be merged in before this one.

Edit:
The work required was merged in with #184.

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

zmichaels11 · 2019-10-25T18:09:22Z

@Karsten1987 this PR is ready for review.
Please don't merge until I verify that it doesn't break any of the work done for reading split bagfiles.

Karsten1987

I would very much prefer if we tackle the removal of get_metadata() from the storage interfaces in this PR as well. That would make at least the writer API complete in a sense to support bag splitting.

rosbag2/src/rosbag2/writer.cpp

rosbag2_storage_default_plugins/src/rosbag2_storage_default_plugins/sqlite/sqlite_storage.cpp

Karsten1987 · 2019-10-25T21:02:33Z

...storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/storage_test_fixture.hpp

@@ -91,7 +91,9 @@ class StorageTestFixture : public TemporaryDirectoryFixture
    std::unique_ptr<rosbag2_storage::storage_interfaces::ReadWriteInterface> writable_storage =
      std::make_unique<rosbag2_storage_plugins::SqliteStorage>();

-    writable_storage->open(temporary_dir_path_);
+    auto db_file = rosbag2_storage::FilesystemHelper::concat({temporary_dir_path_, "rosbag"});


why that change? Why appending rosbag to the filename?

SqliteStorage is being invoked directly here, so db_file needs to be the path to the actual database storage. Before this change, it was expecting the base folder since storage also interacted with the metadata file.

Shouldn't it then be appended with .db3? It seems like you do this for the some of the tests below as well.

also, looking at this again, I don't understand why rosbag is essential here.

..._storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp

Karsten1987 · 2019-10-25T21:05:23Z

rosbag2_tests/test/rosbag2_tests/record_fixture.hpp

  std::string bag_path_;
  std::string database_path_;
+  std::string storage_path_;


What's the difference between these three variables now? What's the difference between database_path_ and storage_path_?

Writer consumes bag_path_, SqliteStorage consumes storage_path_, and SqliteWrapper can consume database_path_

This is related to what I commented here: #185 (comment).

The e2e test in rosbag2_tests opens the database either through Writer, SqliteStorage, or SqliteWrapper.
SqliteWrapper hasn't changed and expects the path to the db, which is database_path_
SqliteStorage now also expects the db as well, but without the extension since that may be plugin-specific. This is storage_path_
Writer expects the base path, which is bag_path_.

SqliteStorage now also expects the db as well, but without the extension since that may be plugin-specific. This is storage_path_

I don't think I understand this. In the metadata.yaml, there should be a path to an existing file listed in relative_file_paths. This filepath should then be passed to a Reader which opens it through the appropriate plugin. The Sqlite storage plugin should thus be able to handle a file which ends in .db3.

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

rosbag2/src/rosbag2/writer.cpp

rosbag2_storage_default_plugins/src/rosbag2_storage_default_plugins/sqlite/sqlite_storage.cpp

Karsten1987 · 2019-11-11T18:58:00Z

...storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/storage_test_fixture.hpp

@@ -91,7 +91,9 @@ class StorageTestFixture : public TemporaryDirectoryFixture
    std::unique_ptr<rosbag2_storage::storage_interfaces::ReadWriteInterface> writable_storage =
      std::make_unique<rosbag2_storage_plugins::SqliteStorage>();

-    writable_storage->open(temporary_dir_path_);
+    auto db_file = rosbag2_storage::FilesystemHelper::concat({temporary_dir_path_, "rosbag"});


Shouldn't it then be appended with .db3? It seems like you do this for the some of the tests below as well.

Karsten1987 · 2019-11-11T19:05:56Z

rosbag2_tests/test/rosbag2_tests/record_fixture.hpp

  std::string bag_path_;
  std::string database_path_;
+  std::string storage_path_;


SqliteStorage now also expects the db as well, but without the extension since that may be plugin-specific. This is storage_path_

I don't think I understand this. In the metadata.yaml, there should be a path to an existing file listed in relative_file_paths. This filepath should then be passed to a Reader which opens it through the appropriate plugin. The Sqlite storage plugin should thus be able to handle a file which ends in .db3.

Karsten1987

Do you have plans on removing the get_metadata functions on the interfaces? I feel this is one of the most important changes for the set of PRs as it will contain changes to reader and writer api simultaneously.

rosbag2_storage_default_plugins/src/rosbag2_storage_default_plugins/sqlite/sqlite_storage.cpp

Karsten1987 · 2019-11-11T19:31:51Z

...storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/storage_test_fixture.hpp

@@ -91,7 +91,9 @@ class StorageTestFixture : public TemporaryDirectoryFixture
    std::unique_ptr<rosbag2_storage::storage_interfaces::ReadWriteInterface> writable_storage =
      std::make_unique<rosbag2_storage_plugins::SqliteStorage>();

-    writable_storage->open(temporary_dir_path_);
+    auto db_file = rosbag2_storage::FilesystemHelper::concat({temporary_dir_path_, "rosbag"});


also, looking at this again, I don't understand why rosbag is essential here.

...storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/storage_test_fixture.hpp

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

Karsten1987 · 2019-11-11T23:19:31Z

@zmichaels11 do you mind not force pushing after a round of reviews? That makes it easier for me to keep track of individual changes.

zmichaels11 · 2019-11-11T23:30:05Z

@zmichaels11 do you mind not force pushing after a round of reviews? That makes it easier for me to keep track of individual changes.

sorry about that

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

rosbag2/src/rosbag2/writer.cpp

rosbag2_storage/include/rosbag2_storage/storage_interfaces/base_info_interface.hpp

..._storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp

rosbag2_storage_default_plugins/src/rosbag2_storage_default_plugins/sqlite/sqlite_storage.cpp

…ts with READ_WRITE Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

Karsten1987

This looks good now to me.
The only thing I would like to see here now is a test which tests for multiple files to be written and correctly mentioned in the metadata file.

I assume further, that the removal of get_metadata() will be done in another PR?

rosbag2_storage_default_plugins/src/rosbag2_storage_default_plugins/sqlite/sqlite_storage.cpp

..._storage_default_plugins/test/rosbag2_storage_default_plugins/sqlite/test_sqlite_storage.cpp

Karsten1987 · 2019-11-12T23:49:47Z

~~so when testing this I get the following:~~

~~➭ ros2 bag record -a~~
[ERROR] [rosbag2_storage]: Could not open 'rosbag2_2019_11_12-15_48_47/rosbag2_2019_11_12-15_48_47_0' with 'sqlite3'. Error: Failed to setup storage. Error: Could not read-write open database. Error code: 14
~~[ERROR] [rosbag2_storage]: Could not load/open plugin with storage id 'sqlite3'.~~
~~[ERROR] [rosbag2_transport]: Failed to record: No storage could be initialized. Abort~~

Nevermind, did compile my workspace completely.

Karsten1987 · 2019-11-13T23:01:35Z

Its going to be a little more intensive than just duplicating and modifying a test in rosbag2 tests that invokes ros2 bag record since enabling bagsplitting from ros2 bag is in a different PR.

Not sure I completely understand this. The command line option doesn't really play a role in this new PR. I do believe we (just) need a system test in ros2_tests which basically just opens a Writer instance, manually adds a couple of messages and verifies that all messages are correctly stored across various bag files. That tests has to be successful for all provided storage implementations.
This can be achieved independent of the command line tools and even the transport layer.

I'm thinking its probably best to add a unit test for bagsplitting on a mocked storage since the operation of bagsplitting is owned by Writer.

This makes sense to me for validating the splitting logic within the Writer itself.

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

zmichaels11 · 2019-11-13T23:51:28Z

Added a unit test that validates that Writer is splitting and the bagfiles increment correctly.

rosbag2/src/rosbag2/writer.cpp

zmichaels11 · 2019-11-14T16:48:49Z

Not sure I completely understand this. The command line option doesn't really play a role in this new PR. I do believe we (just) need a system test in ros2_tests which basically just opens a Writer instance, manually adds a couple of messages and verifies that all messages are correctly stored across various bag files. That tests has to be successful for all provided storage implementations.
This can be achieved independent of the command line tools and even the transport layer.

I took a crack at that and I think its going to require enough effort to justify a new PR. Most of what you suggested works, except that there isn't a cross-platform way to create directory structures in c++.

I think that we should wrap up this PR, port filesystem::create_directory from C++17 as another PR, and include the integration test with the work defined in https://github.com/ros-security/aws-roadmap/issues/10

I have added a unit test to Writer that verifies splitting. Can you please take a look at it?

Karsten1987 · 2019-11-14T22:28:20Z

CI:

Linux
Linux-aarch64
macOS
Windows

The changes for the writer API look good to me now. However, I am unable to play a bag file with this PR. I presume it's because the changes for loading the metadata appropriately on the reader side are not part of this PR.

 ➭ ros2 bag play rosbag2_2019_11_14-14_16_02
[ERROR] [rosbag2_storage]: Could not open 'rosbag2_2019_11_14-14_16_02' with 'sqlite3'. Error: Failed to setup storage. Error: Could not read-only open database. Error code: 10
[ERROR] [rosbag2_storage]: Could not load/open plugin with storage id 'sqlite3'.
[ERROR] [rosbag2_transport]: Failed to play: No storage could be initialized. Abort

Pointing me to the direction of

  if (is_read_write(io_flag)) {
    relative_path_ = uri + FILE_EXTENSION;

    // READ_WRITE requires the DB to not exist.
    if (database_exists(relative_path_)) {
      throw std::runtime_error(
              "Failed to create bag: File '" + relative_path_ + "' already exists!");
    }
  } else {  // APPEND and READ_ONLY
    relative_path_ = uri;

    // APPEND and READ_ONLY require the DB to exist
    if (!database_exists(relative_path_)) {
      throw std::runtime_error(
              "Failed to read from bag: File '" + relative_path_ + "' does not exist!");
    }
  }

which lets me believe that the relative path is still set to the base folder path and the actual metadata is not being read.

As a side note: This also means that database_exists() is not as accurate as it should be. It will check whether a file exists, not if that file actually represents a valid database.

How do you guys want to proceed with this? I can't merge this PR as-is as this would break rosbag2 because of not being able to replay bag files.

zmichaels11 · 2019-11-14T23:13:22Z

I had @piraka9011 look at this since he has context on the Reader.
He's going to push a commit with the minimal number of changes required to get this functionality working.

I'm going to look into making database_exists more correct.

Signed-off-by: Anas Abou Allaban <allabana@amazon.com>

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

zmichaels11 · 2019-11-14T23:36:37Z

@Karsten1987 I think we resolved your issues.
Anas pushed a commit that resolved the Reader issues.
I pushed a commit that has SqliteWrapper throw if the database is bad.

rosbag2/test/rosbag2/test_sequential_reader.cpp

rosbag2_storage_default_plugins/src/rosbag2_storage_default_plugins/sqlite/sqlite_storage.cpp

rosbag2/src/rosbag2/sequential_reader.cpp

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

rosbag2_storage_default_plugins/src/rosbag2_storage_default_plugins/sqlite/sqlite_storage.cpp

Karsten1987 · 2019-11-15T00:35:34Z

rosbag2/src/rosbag2/sequential_reader.cpp

-  storage_ = storage_factory_->open_read_only(storage_options.uri, storage_options.storage_id);
+  metadata_ = metadata_io_->read_metadata(storage_options.uri);
+  storage_ = storage_factory_->open_read_only(
+    metadata_.relative_file_paths[0], storage_options.storage_id);


this should at least check if relative_file_paths is non empty otherwise this will lead to a segfault.

Karsten1987 · 2019-11-15T01:43:56Z

Except that one outstanding comment, that looks good to me.

piraka9011 · 2019-11-15T01:59:17Z

@Karsten1987 added a check!

zmichaels11 · 2019-11-15T02:01:55Z

@piraka9011 I was actually about to do the same, except throw instead.
You need to add in test_sequential_reader.cpp

metadata_.relative_file_paths = {"dummy path"};

piraka9011 · 2019-11-15T02:03:13Z

I followed the same convention as the next check for the topics in L#58. Updating tests.

Karsten1987 · 2019-11-15T02:09:57Z

An immediate return is fine, but I think in both cases we should log a warning saying that either no concrete files are found or no topics are listed.

Signed-off-by: Anas Abou Allaban <aabouallaban@pm.me>

piraka9011 · 2019-11-15T02:19:29Z

@zmichaels11 Updated tests
@Karsten1987 Added warning

Karsten1987 · 2019-11-15T02:42:22Z

Linux
Linux-aarch64
macOS
Windows

zmichaels11 mentioned this pull request Oct 17, 2019

Add bagfile splitting support to storage_options and Writer #182

Merged

zmichaels11 mentioned this pull request Oct 18, 2019

Add function to BaseIOInterface to retrieve backing file #186

Closed

2 tasks

Implement rosbag splitting in Writer

b2411a3

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

zmichaels11 force-pushed the splitting/writer-storage-integration branch from db90555 to b2411a3 Compare October 24, 2019 22:44

Update unit tests for sqlite3 storage

0ef95cd

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

zmichaels11 force-pushed the splitting/writer-storage-integration branch from fd128ec to 0ef95cd Compare October 25, 2019 15:06

zmichaels11 added 3 commits October 25, 2019 09:16

Update unit tests for rosbag2_tests

dff5797

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

Add documentation

fd4c8a1

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

Cleanup code

c6e10f5

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

zmichaels11 mentioned this pull request Oct 25, 2019

Implement changes in io interfaces ros2/rosbag2_bag_v2#7

Merged

2 tasks

Karsten1987 reviewed Oct 25, 2019

View reviewed changes

zmichaels11 force-pushed the splitting/writer-storage-integration branch 2 times, most recently from 39b20fd to f8872d7 Compare October 28, 2019 20:43

Apply suggestions

b16a8bd

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

zmichaels11 force-pushed the splitting/writer-storage-integration branch from f8872d7 to b16a8bd Compare October 28, 2019 20:53

Add deleted test back in

acce1a9

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

Karsten1987 reviewed Nov 11, 2019

View reviewed changes

Apply suggestions

3197d1b

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

zmichaels11 force-pushed the splitting/writer-storage-integration branch from 22c6e6e to 3197d1b Compare November 11, 2019 22:35

Append file extension only when SqliteStorage::open is passed READ_WRITE

2ba3451

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

Karsten1987 reviewed Nov 12, 2019

View reviewed changes

rosbag2_storage_default_plugins/src/rosbag2_storage_default_plugins/sqlite/sqlite_storage.cpp Show resolved Hide resolved

Apply formatting suggestions and throw in storage open when file exis…

700ced6

…ts with READ_WRITE Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

Karsten1987 reviewed Nov 12, 2019

View reviewed changes

prajakta-gokhale mentioned this pull request Nov 13, 2019

expose max-bagfile-size for recording command line verb aws-ros-dev/rosbag2#3

Closed

Add unit test for validating splitting in Writer

1fba68b

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

piraka9011 reviewed Nov 14, 2019

View reviewed changes

rosbag2/src/rosbag2/writer.cpp Show resolved Hide resolved

zmichaels11 mentioned this pull request Nov 14, 2019

Add FilesystemHelper::create_directory #204

Closed

Anas Abou Allaban and others added 2 commits November 14, 2019 15:18

Make reader use load_metadata and update tests

bf9e0d8

Signed-off-by: Anas Abou Allaban <allabana@amazon.com>

Remove database_exists and make SqliteWrapper throw when database is bad

060ddc7

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

Karsten1987 reviewed Nov 14, 2019

View reviewed changes

rosbag2/test/rosbag2/test_sequential_reader.cpp Show resolved Hide resolved

rosbag2/test/rosbag2/test_sequential_reader.cpp Outdated Show resolved Hide resolved

rosbag2_storage_default_plugins/src/rosbag2_storage_default_plugins/sqlite/sqlite_storage.cpp Show resolved Hide resolved

Karsten1987 reviewed Nov 14, 2019

View reviewed changes

rosbag2/src/rosbag2/sequential_reader.cpp Outdated Show resolved Hide resolved

Fix formatting and use relative_file_paths[0] from metadata

b17a59e

Signed-off-by: Zachary Michaels <zmichaels11@gmail.com>

Karsten1987 reviewed Nov 15, 2019

View reviewed changes

Karsten1987 mentioned this pull request Nov 15, 2019

Create new interface for SequentialReader #191

Closed

Check if relative file paths is empty

bc92f10

Signed-off-by: Anas Abou Allaban <aabouallaban@pm.me>

piraka9011 force-pushed the splitting/writer-storage-integration branch from 092999f to bc92f10 Compare November 15, 2019 02:14

Update tests to reflect changes in reader

d770b64

Signed-off-by: Anas Abou Allaban <aabouallaban@pm.me>

Karsten1987 approved these changes Nov 15, 2019

View reviewed changes

Karsten1987 merged commit 4e79be4 into ros2:master Nov 15, 2019

zmichaels11 deleted the splitting/writer-storage-integration branch November 15, 2019 03:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rosbag splitting in Writer #185

Rosbag splitting in Writer #185

zmichaels11 commented Oct 17, 2019 •

edited

Loading

zmichaels11 commented Oct 18, 2019 •

edited

Loading

zmichaels11 commented Oct 25, 2019

Karsten1987 left a comment

Karsten1987 Oct 25, 2019

zmichaels11 Oct 25, 2019

Karsten1987 Nov 11, 2019

Karsten1987 Nov 11, 2019

Karsten1987 Oct 25, 2019

zmichaels11 Oct 25, 2019

Karsten1987 Nov 11, 2019

Karsten1987 Nov 11, 2019

Karsten1987 Nov 11, 2019

Karsten1987 left a comment

Karsten1987 Nov 11, 2019

Karsten1987 commented Nov 11, 2019

zmichaels11 commented Nov 11, 2019

Karsten1987 left a comment

Karsten1987 commented Nov 12, 2019 •

edited

Loading

Karsten1987 commented Nov 13, 2019

zmichaels11 commented Nov 13, 2019

zmichaels11 commented Nov 14, 2019 •

edited

Loading

Karsten1987 commented Nov 14, 2019

zmichaels11 commented Nov 14, 2019 •

edited

Loading

zmichaels11 commented Nov 14, 2019

Karsten1987 Nov 15, 2019

Karsten1987 commented Nov 15, 2019

piraka9011 commented Nov 15, 2019

zmichaels11 commented Nov 15, 2019 •

edited

Loading

piraka9011 commented Nov 15, 2019

Karsten1987 commented Nov 15, 2019

piraka9011 commented Nov 15, 2019

Karsten1987 commented Nov 15, 2019

Rosbag splitting in Writer #185

Rosbag splitting in Writer #185

Conversation

zmichaels11 commented Oct 17, 2019 • edited Loading

Changes

Dependent PRs

Issues

zmichaels11 commented Oct 18, 2019 • edited Loading

zmichaels11 commented Oct 25, 2019

Karsten1987 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Karsten1987 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Karsten1987 commented Nov 11, 2019

zmichaels11 commented Nov 11, 2019

Karsten1987 left a comment

Choose a reason for hiding this comment

Karsten1987 commented Nov 12, 2019 • edited Loading

Karsten1987 commented Nov 13, 2019

zmichaels11 commented Nov 13, 2019

zmichaels11 commented Nov 14, 2019 • edited Loading

Karsten1987 commented Nov 14, 2019

zmichaels11 commented Nov 14, 2019 • edited Loading

zmichaels11 commented Nov 14, 2019

Choose a reason for hiding this comment

Karsten1987 commented Nov 15, 2019

piraka9011 commented Nov 15, 2019

zmichaels11 commented Nov 15, 2019 • edited Loading

piraka9011 commented Nov 15, 2019

Karsten1987 commented Nov 15, 2019

piraka9011 commented Nov 15, 2019

Karsten1987 commented Nov 15, 2019

zmichaels11 commented Oct 17, 2019 •

edited

Loading

zmichaels11 commented Oct 18, 2019 •

edited

Loading

Karsten1987 commented Nov 12, 2019 •

edited

Loading

zmichaels11 commented Nov 14, 2019 •

edited

Loading

zmichaels11 commented Nov 14, 2019 •

edited

Loading

zmichaels11 commented Nov 15, 2019 •

edited

Loading