Design/Doc: sqlite directory storage format

[chaoflow]

We're looking into storing additional files alongside a rosbag2, e.g. pcaps, pdfs, ... and are thinking to place these into the rosbag2 directory. Given that metadata.yaml lists the files relevent for reading rosbag2 this should not pose a problem, correct?

The rosbag2 design article linked from README does not specify the file system layout. Is there a more recent version?

---

Conclusions from discussions:

• Adding additional files including subdirectories to a rosbag2 directory is allowed and encouraged if the files belong to the recording
• metadata.yaml is primary location for metadata, db3-files may duplicate that data
• Additional metadata options:
  1. top-level ``metadata.yaml
  2. under top-level custom key; alternative name: userdata
  3. under rosbag2_bagfile_information subkey custom/userdata -- this would enable the BagMetadata class to pass this information to a single-file storage format
• Message definitions will be stored in BagMetadata version >4, probably TopicMetadata see Store message definition in bag #782

[emersonknapp]

The directory contains metadata.yaml and all of the sqlite databases (if splitting is enabled) - putting extra files into the directory should not cause any problems as rosbag2 is currently implemented. It certainly does make sense to store this extra information with the bag if it is related to the recording session.

[chaoflow]

@emersonknapp Great, thank you for confirming that. Would it make sense to extend the design article to include a section about file system layout or is there some other place better suited?

[emersonknapp]

I have a hard time deciding on that - I feel like design/documentation should live here with the rosbag2 project, instead of in ros2/design, for better synchronization, including updating across releases without losing the context for any given release. @Karsten1987 do you have an opinion on that?

[emersonknapp]

Considering starting a readthedocs page for this - or a dedicated documentation page like nav2

[chaoflow]

@emersonknapp We're looking for a place to store low volume additional metadata for a recording, like description, operator, software version, ... for cases where this is not published and recorded on a topic. One option is to add an additional file as currently in the scope of this issue, another would be to use a top-level key in metadata.yaml, e.g. custom under which people can store what they feel like. However, this would contradict that metadata.yaml is only duplicating data otherwise available within the db3 files.

• What are your thoughts on this?
• Shall we expand the scope of this issue to document not only additional files, but also things currently discussed elsewhere? like:
  • db3 self-contained
  • message definitions
  • additional metadata

[emersonknapp]

metadata.yaml is not strictly defined as a duplicate of what is in the database files. Actually, database files may store the BagMetadata for redundancy. If you want to include extra top-level fields in metadata.yaml, this will not break any assumptions that are currently made. I would be happy to define this as an explicitly supported use case, by saying that metadata.yaml reserves the top-level key rosbag2_bagfile_information, and all other keys are available for user-specific data. Or, we could make it slightly more specific, by saying that the top-level key custom is reserved for the user. Either way would be fine with me.

^ That's a first thought.

However, if we want to handle the case "DB3 self-contained" - it may make more sense to add a field custom that is a sub-field of rosbag2_bagfile_information - and reserve that for user use. This would then be part of the BagMetadata class, and be passed to the storage implementation to write into a single file.

Case "message definitions" - this is a separate concern and will need to be part of the BagMetadata schema (version > 4 - current is 4 and this will be an addition)

[chaoflow]

@emersonknapp I updated the description above, please take a look.

• Could BagMetadata also access a top-level custom key?
• Is it safe with current BagMetadata class to add a custom subkey of rosbag2_bagfile_information?

[emersonknapp]

Could BagMetadata also access a top-level custom key?

No - the way the serialization works it that the BagMetadata is serialized into the YAML file as the key rosbag2_bagfile_information, so it can't affect anything above that

Is it safe with current BagMetadata class to add a custom subkey of rosbag2_bagfile_information?

Yes, this will be safe to do. This key is not currently reserved by the BagMetadata schema, so we can reserve this key for user use from now on.

[t0ny-peng]

@emersonknapp Thanks for the valuable comments. I happened to be assigned the similar task of adding custom metadata support. If it hasn't been implemented anywhere else, I'm happy to create a PR once it's done.