(kafka sink): Add AVRO support to Kafka sink

[erikbos]

It would be nice to have AVRO support in Kafka sink so data read as JSON gets output'ed as AVRO. Advantages would be input/output validation against schema and less overhead compares to raw JSON.

[Igosuki]

Additionally, supporting confluent's schema registry (or other registries for that matter) would add great value. It's simply reading a magic byte then an integer and caching schema responses from the registry.

[raghu999]

+1 for avro support , I am new to rust I can try contributing back with some help is this feature an approved usecase for vector?

[Hoverbear]

@raghu999 Hi! We'd like to have this feature yes! The first step would be speccing out a proposal. (Eg, how would we know the schema?)

You can do this informally with us here while you play with some ideas and get started looking at the code. :)

[Igosuki]

You can use the avro-rs crate, but it needs to be patched to support fingerprinting schemas.

There are two types of schema resolution, either you have an id of a schema (after the magic byte), which you can use to query a schema registry, or a fingerprint, which you can use to internally resolve a schema from the fingerprint (i.e. the schema has been manually provided by the user).

You can then, map the schema to the internal representation of Vector and that's pretty much it.

I already have a branch started on my computer, just need to find the time

[Igosuki]

I'm using Avro instead of protobuf for logging consistent events in avro files or parquet files because the data engineering tooling around it is easier to work with, so that would make vector usable in more than just system logging for me.

[Hoverbear]

Sounds like it might be more and more worthwhile to support https://docs.confluent.io/current/schema-registry/index.html

[Igosuki]

Both fingerprints and cache registry client with ids are valid implementations. Make sure not to force ppl to deploy schema registry when they can just inline a schema in conf