v1: Store an index by publish time

[defunctzombie]

The current index records are great when you want to read the messages by log_time. If you want to read the messages by publish_time, you have to create your own index by reading all the records and building it in memory. It would be nice to consider writer options for recording both index types in the file.

[wkalt]

I think the best solution would be to store (log time, publish time, offset) triples in the message index record and extend the chunk index record with publish start/end. Breaking change for the former + 8 bytes per record.

If we need something in the interim, we could append publish_start and publish_end to the chunk index record as optional fields, and append a publish_time_index array to the end of message index that mirrors the existing record time index structure. This would cost 8 bytes per record more than the breaking solution due to timestamp duplication.

[shuhaowu]

Can the concept of time be dynamically changed based on a flag somewhere? Maybe in the footer there could be a flag that dictates if "time" means log_time or publish_time? A binary compatible way would be to throw that information in the header.library, but that would require a read of the header which is an extra required read.

There are a few places where time means log_time for now:

• In ChunkIndex, the message_start_time and message_end_time is currently log_time.
• In MessageIndex, the records has its values as log_time.
• In Chunk, the message_start_time and message_end_time is currently log_time.

If a flag can universally switch them to mean publish_time, then no breaking change would be needed. Unfortunately I don't see any flexible/reserved fields for this other than the Header.profile or Header.library. I suppose could also use Metadata but that would require possibly up to 2 additional IO (at least 1 additional) which can be cost prohibitive in high-performance scenarios.

[shuhaowu]

The alternative is to maybe introduce another record type with a new opcode: ChunkIndex2 and MessageIndex2 which has this information. The 2 version can be in addition to the normal version to be forward compatible

[amacneil]

Cross-linking my write up on allowing less or more than 2 timestamps: #1196

And this thread about per-message metadata: #1369

If we ever do a binary-breaking change to mcap, improving handling of timestamps and metadata handling is up there for me.

The alternative is to maybe introduce another record type with a new opcode: ChunkIndex2 and MessageIndex2 which has this information. The 2 version can be in addition to the normal version to be forward compatible

The nice thing about this approach is that it degrades gracefully, old mcap client libraries can still read the mcap data, it will just appear unindexed to them.

[shuhaowu]

Yeah, having more than 2 timestamps is also a very important use case (more below). But really having the publish time as index is more of a show stopper. Having a 2 version that gracefully degrade would solve that problem.

It is actually possible to support per message timestamps and N timestamps without breaking the binary compatibility. The way to do this is to create a new payload type that itself can encapsulate protons/flat buffer/JSON/etc. Basically the message encoding for a channel would be something like encapsulated_proto or encapsulated_json. Then the data could contain the metadata and the timestamps as a prefix, and the proto and JSON can follow it. It's not the prettiest solution, but it will work. So like:

Message:

• publish_time
• data
  • timestamps: uint64[]
  • metadata: ...
  • inner_data: byte

In a way this is the same as if you changed the binary protocol (imagine if the above is flattened), except the versioning is controlledb y the channel message_encoding field, if that makes sense.

[shuhaowu]

If there is interest in this, I may have some time to create a proposal/PR.

[jtbandes]

Also related: #918 by @james-rms