[proposal] Pigeon should use Protobuf to share data between "HostApi" and "FlutterApi"

[iapicca]

Hi all,
the following proposal is largely inspired by this great article from @truongsinh

Use case

Common issues with platform channel data transfer:

• FlutterStandardTypedData issues
• often require to cast data
• frequent loss of type-safety
• loss of performance with List, Map<String,dynamic>

Proposal

Use protobuf as default for pigeon's platform channel communication.

Extras

• grpc uses protobuf, the proposed implementation
  would make very easy to cache data on native side from a grpc client

• AFAIK flutterfire uses a grpc client on the native side
  the proposed implementation should lead to better performance and stability

cc @gaaclarke as requested in the feedback tab of pigeon readme

Thank you

[gaaclarke]

Thanks @iapicca are you saying you want to be able to write in protobuf language, or you want pigeons to serialize in a binary format that is compatible with protobufs? You've listed problems with platform channels, but what are the problems you are trying to address in pigeon?

Protobufs were designed to help with network throughput which means they were optimized for size compression not speed of serialization / deserialization. That means protobufs aren't well suited for situations where there is no network communication. Also, since pigeons are ephemeral they can be optimized for speed by removing that feature. On the other hand, Protobufs can be saved and reloaded with versioning, at a cost.

The parser for pigeon is split into a frontend and a backend so that either of them could be replaced. We had things like protobufs in mind when we designed it and it isn't out of the cards.

[iapicca]

Hi @gaaclarke
first of all thank you for reading through my proposal,
in the back of my head I took for granted that protobuf was optimized for serialization,
this not being the case makes sense to prefer pigeons for speed over size;
nonetheless given your comment

[...] or you want pigeons to serialize in a binary format that is compatible with protobufs?

a compatibility between pigeon and protobuf sounds great
and having the option to use protobuf with pigeon,
maybe through a TypeAdapter could be desirable

Anyway those are just my 2 cents,
this solution being considered is all I ask for.

Thanks again for your answer

[Sunbreak]

Interesting proposal.

But I would recommand dart:ffi or https://github.com/dart-native/dart_native if performance is a concern