Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store DB schema versioning in .db3 files #1108

Closed
MichaelOrlov opened this issue Oct 1, 2022 · 2 comments · Fixed by #1156
Closed

Store DB schema versioning in .db3 files #1108

MichaelOrlov opened this issue Oct 1, 2022 · 2 comments · Fixed by #1156
Labels
enhancement New feature or request

Comments

@MichaelOrlov
Copy link
Contributor

MichaelOrlov commented Oct 1, 2022

Description

Some times we need to change DB schema. For instance add new field or even a new table connected by some index with another table.
Currently we are not storing schema version in .db3 file and it's very difficult to support backward compatibility in case of changing DB schema.
We need to store DB schema version in .db3 file directly to be able properly read out bag files with different versions.

Related Issues

Related to the #1086

Completion Criteria

    • Database schema version stored in .db3 files directly.
    • Backward compatibility for old db schema supported.

Implementation Notes / Suggestions

On "Rosbag2 and Tooling WG meeting" from 2022-09-23 we discussed possible implementation and considered 2 options for storing db schema.

  1. Using builtin sqlite pragma for db schema version https://www.sqlite.org/pragma.html#pragma_schema_version
  2. Creating new table in database for storing DB schema version in it.

We decided to go with second option because builtin pragma schema version is intended for internal usage by sqlite engine and supposed to be incremented by sqlite engine when doing internal operations on database. There are some notes in description that changing those value could lead to the undefined behavior.
One more argument towards to the second option is that we can store metadata version in the same table as well.

Presumably will need to add new table in SqliteStorage::Initialize() method.

 std::string create_stmt = "CREATE TABLE schema(" \
    "schema_version INTEGER PRIMARY KEY,"
    "metadata_version INTEGER NOT NULL);";
 database_->prepare_statement(create_stmt)->execute_and_reset();

Testing Notes / Suggestions

Will need to add coverage in unit tests to make sure that bag files created with previous version of the schema layout supported. i.e. possible to create, open reader and read messages from it.

@ros-discourse
Copy link

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-2022-10-13/28213/1

@ros-discourse
Copy link

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-2022-11-17/28457/1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants