Need a custom Log Format

[prazanna]

Today we use Avro.

Drawbacks:

1. A file has a schema, and all objects stored in the file must be written according to that schema. Hoodie Log file schema can change.
2. Need for atomically publishing a Avro block, requires rolling back with a tombstone.
3. We need custom metadata like the delta commit time for the block
4. AvroReader will fail to scan a avro file which has a partial written block (because of a failed commit). We need to skip to the next block and we need to manualy scan till the sync marker.

We need a custom format to help with the above pains. The data can still be avro serialized, but the file format needs to be more flexible to store metadata and sync markers.