Schema support for Flatbuffers??

[cavanaug]

Is your feature request related to a problem? Please describe.
Organizationally we have been looking at a variety of serialization solutions, there has been significant interest around flatbuffers as a messaging model. This has a lot of benefits for IOT style devices and communication between services on device. This also has potential benefits for rpc and/or rest style binary payloads.

Describe the solution you'd like
I am interested in understanding how the pulsar project views flatbuffers vis-a-vis the currently supported schema models. Is this a yeah that seems interesting maybe someday? Or a yeah, we are definitely going to be working on that? Or a yeah, no, thats not part of our future strategy.

Describe alternatives you've considered
Obviously there are strategies that could be employed to using protobuf and simply convert etc, or revert to JSON. So these are all on the table for us, but the perceived performance gains resulting from a non-copy access seem to be something that could quite beneficial to pulsar.

[aahmed-se]

It's possible to add flatbuffers schema support just like protobuf. But it's not clear what the advantages would be. Flatbuffers is generally used in the context of low latency ipc not really suitable for persisting messages in log based pub sub systems.

[vicaya]

What about people want to use non-persistent topics for lower latency pubsub for certain topics? Flatbuffer also uses a lot less CPU and memory allocations (esp. vs protobuf). We'd welcome things that would reduce the CPU/memory usage of pulsar broker.

[merlimat]

We'd welcome things that would reduce the CPU/memory usage of pulsar broker.

Keep in mind that serialization/deserialization are done only by producers/consumer. Brokers will not do these at any point. The only thing that broker does is to validate the schema definition.

[sijie]

@cavanaug

As what Matteo pointed out, the serialization and deserialization are only done at the client side. Brokers only deal with the schema definition and validation. So adding a new type is trivial if we are able to get the schema definition. I am not familiar with flatbuffer. but if there is a way to get schema definition, we can easily add it.

[tisonkun]

Closed as stale and no one works on it. Please create a new issue if it's still relevant to the maintained versions.