Parsing the Mutation Log

A mutation log was introduced to capture high-level mutation operations (e.g., split, merge of a label) in an append-only log. This page describes how to parse the data within a mutation log.

The mutation log is written as a binary stream of data with the following structure:

entryType 1 (uint16 - enum for each type of protobuf-defined entry)
entrySize 1 (uint32 - giving # bytes for the serialization of this entry)
entryData 1 (byte array of length “entrySize” that is the protobuf3 serialization)
entryType 2
entrySize 2
entryData 2
…
entryType n
entrySize n
entryData n

The enum for entry types is here:

https://github.com/janelia-flyem/dvid/blob/master/datatype/common/proto/entries.go

This will become a enum in the .proto file but for now is defined as a Go enum.

The format of each entry is in protobuf3 IDL:

https://github.com/janelia-flyem/dvid/blob/master/datatype/common/proto/labelops.proto

You run a protobuf generator on that IDL file and it generates parsing code in whatever language you want (python, Java, C++, etc).

Here’s info on protobuf: https://developers.google.com/protocol-buffers/