Support Message metadata field (like Channel)

[alokpr]

Message does not support any metadata field.

Other options I have considered:

1. Channel metadata: While fields common to all messages on a channel can be added to the Channel metadata, there are fields that change every message on a channel. Adding them to the channel metadata will result in an explosion of number of channels
2. Message data: If we wanted to record some fields common to all messages, we would need to change the schema of all messages. It is also inefficient to wrap an already encoded payload into yet another byte array

[james-rms]

@alokpr A few thoughts:

• MCAP already includes a lot of per-message overhead (22 bytes for every message), I would not support adding more unless there was a clear good reason.
• Application-defined message data should be described in the message schema and encoded into the message data field.
• Do you have a concrete use-case that is driving this ask? Why do you believe it's inefficient to wrap an encoded payload in a byte-array?
• It might be that this ask is driven by requirements specific to your message serialization format.

[alokpr]

I understand the concern for per-message overhead. Perhaps there is another way - like creating a different kind of record like MessageWithMetadata.

Anyways, the concrete use-case is implementing a recorder for zenoh middleware, whose Sample type contains a bunch of fields not directly supported by mcap:

• express
• priority
• congestion_control
• reliability
• attachment

I expect most of these fields to be the same for all samples on a topic, which means those values can be stored as channel metadata. The "attachment" field however would change every sample on a topic. How do you propose recording the attachment?

The reason why an application would choose to publish a sample with attachment instead of encoding the attachment into the message payload is to avoid redundant copies. Imagine we need to publish an image which is already available in a byte buffer. If we were to encode it into a protobuf message, say:

message Image {
  Timestamp timestamp = 1;
  string format = 3;
  bytes data = 2;
}

This would require:

1. Constructing an Image message
2. Copy image data into the message (this can be a move in certain languages)
3. Encode message into a buffer which will require another copy of image data -> this copy is unavoidable

Compare the above with publishing image data as attachment that avoids at least one copy.