Explore protocol buffers vs Avro #1031

Closed
jpdna opened this Issue May 12, 2016 · 3 comments

Comments

Projects
None yet
3 participants
@jpdna
Member

jpdna commented May 12, 2016

GA4GH says they are moving to protocol buffers as opposed to Avro due to the streaming rpc support in PB - which is lacking in Avro. Is this use case, and compatibility with GA4GH, sufficient reason to consider a branch/option of ADAM and bdg that uses PB instead of Avro?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft May 12, 2016

Member

Since we just rely on Avro for serialization and model definition, we wouldn't benefit from the streaming RPC support. For compatibility, I like the "view" approach that we take, where we translate data from underlying storage format x (right now, legacy file formats) to our Avro schemas. From a serialization performance perspective, Avro/Thrift/Protobuf are pretty similar, with Avro being slightly faster IIRC. See here for a longer benchmarking experiment.

Since our long term roadmap is to move to Datasets (where the serialization framework is somewhat moot other than for defining schemas), I'm -0.5 on exploring protobuf.

Member

fnothaft commented May 12, 2016

Since we just rely on Avro for serialization and model definition, we wouldn't benefit from the streaming RPC support. For compatibility, I like the "view" approach that we take, where we translate data from underlying storage format x (right now, legacy file formats) to our Avro schemas. From a serialization performance perspective, Avro/Thrift/Protobuf are pretty similar, with Avro being slightly faster IIRC. See here for a longer benchmarking experiment.

Since our long term roadmap is to move to Datasets (where the serialization framework is somewhat moot other than for defining schemas), I'm -0.5 on exploring protobuf.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Jul 6, 2016

Member

Close as WontFix?

Member

heuermh commented Jul 6, 2016

Close as WontFix?

@fnothaft fnothaft added the wontfix label Jul 6, 2016

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Jul 6, 2016

Member

Closed as won't fix.

Member

fnothaft commented Jul 6, 2016

Closed as won't fix.

@fnothaft fnothaft closed this Jul 6, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment