Add support for introspecting avro-wrapped protobufs

[nevillelyh]

Avro DataFileWriter has a .setMeta(String, String) method and we wrap ProtoBuf bytes in Avro files. Maybe set schema there to make ad-hoc file operations easier, e.g. cat, getschema?

[nevillelyh]

scala> TrackPB.getDescriptor.toProto
res1: com.google.protobuf.DescriptorProtos.DescriptorProto =
name: "TrackPB"
field {
  name: "trackId"
  number: 1
  label: LABEL_REQUIRED
  type: TYPE_STRING
}

[ravwojdyla]

Cool idea! That said - then the user would have to know that even those she is saving data as protobufs, to find schema she needs to use avro-tools and then there is this custom meta there etc. Isn't this too low level and "scio" specific?

[nevillelyh]

I mean we can make a proto-tools for quickly eyeballing a file. Not much work and maybe worth it if Protobuf becomes popular.

[nevillelyh]

https://github.com/nevillelyh/protobuf-generic

This lib can do the following:

• generate schema/JSON for any compiled protobuf type T, which we can attach to Avro files as metadata.
• encode/decode binary protobuf as JSON given the above schema.

I'll look into Scio side of things.

[nevillelyh]

BEAM
Issue: https://issues.apache.org/jira/browse/BEAM-813
PR: apache/beam#1177

[nevillelyh]

This will be supported in 0.2.6, due for release tomorrow. Closing this now.
https://github.com/spotify/scio/wiki/Protobuf#file-format