CLI: mcap filter subcommand #445

james-rms · 2022-06-27T21:32:30Z

Public-Facing Changes

New mcap filter subcommand added, which you can use to filter MCAPs by topic and time range.
Bug fixed in the go MCAP library when writing attachment indexes, which would cause the attachment
content type to be malformed.

Description
This PR adds a new subcommand to the MCAP CLI tool that allows people to quickly filter topics into and out-of an MCAP file.

There was a bug where WriteAttachmentIndex would fail to pre-allocate 8 bytes for one of its uint64 timestamp fields. This would cause the write buffer to be too short to fit the final field (the content type string). In our existing test cases, the content type was more than 8 chars long, and the effect was to truncate the content type string by 8 chars. However, in implementing `mcap filter`, i created a test case with a 0-length content type string, which caused a segfault.

This subcommand allows command-line users to filter MCAP files by time range and topic name.

defunctzombie · 2022-07-07T15:10:24Z

What is the syntax for filter? Some examples?

go/mcap/writer.go

go/cli/mcap/cmd/filter.go

james-rms · 2022-07-07T21:53:44Z

@defunctzombie from the long help:

This subcommand filters an MCAP by topic and time range to a new file.
When multiple regexes are used, topics that match any regex are included (or excluded).
usage example:
  mcap filter in.mcap -o out.mcap -y /diagnostics -y /tf -y /camera_(front|back)

Some more examples:

# exclude topic by regex, using pipes
cat in.mcap | mcap filter -n /pose > out.mcap

# time ranges
mcap filter in.mcap -o out.mcap --start-secs 1657000000 --end-secs 1657030667

go/cli/mcap/cmd/filter.go

wkalt · 2022-07-07T22:19:30Z

go/cli/mcap/cmd/filter.go

+				}
+			} else {
+				channels[channel.ID] = markableChannel{channel, false}
+			}


if we write the schema+channel after observing the channel, rather than after observing the first message, would that let us drop the "markable" aspect of the channel map, or maybe even drop the channel map altogether?

That would also result in us returning channels with matching topics even if they have no messages on them, which seems more correct on balance to me. What do you think?

IMO a channel with no messages in it in an MCAP does not make a lot of sense to me, which is why I did it this way.

well, a channel in an mcap with no messages is definitely something you can record - so why wouldn't the filter command preserve channels on the supplied topic?

go/cli/mcap/cmd/filter.go

wkalt

few high-level comments - feel free to address as you like. I pulled it down and tested, seems to work fine with the stdout change.

james-rms force-pushed the feature/cli-filter branch from 5ce6425 to 7c6adc8 Compare June 27, 2022 22:39

james-rms marked this pull request as ready for review June 30, 2022 20:58

james-rms force-pushed the feature/cli-filter branch from 7c6adc8 to e274d65 Compare July 7, 2022 04:50

james-rms force-pushed the feature/cli-filter branch 2 times, most recently from bad50ee to 78204b6 Compare July 7, 2022 04:57

james-rms requested review from wkalt and defunctzombie July 7, 2022 04:58

james-rms force-pushed the feature/cli-filter branch from 78204b6 to 891a079 Compare July 7, 2022 05:01

CLI: implement filter subcommand

d14ef51

This subcommand allows command-line users to filter MCAP files by time range and topic name.

james-rms force-pushed the feature/cli-filter branch from 891a079 to d14ef51 Compare July 7, 2022 05:09