extension to read Kafka Avro Serializer messages using Kafka Avro Decoder

[Kargreat]

I am reading messages built to an Avro schema. These messages use the Avro Kafka Serializers (https://github.com/confluentinc/schema-registry/tree/master/avro-serializer/src/main/java/io/confluent/kafka/serializers).

I am using the Logstash Input Plugin to read these messages. I found that currently there is no support in the plugin for this serializer and decoder. I think the group.rb and consumer.rb need enhancement (apart from spec file). The link for the consumer is: http://confluent.io/docs/current/schema-registry/docs/serializer-formatter.html.

Can you please enhance this one or create a new one?

[talevy]

After looking further into this... It would be possible to utilize the existing java implementation of the Serializer/Deserializer for KafkaAvro described here after a slight modification to jruby-kafka's create_consumer_config function in group.rb. Where it would be possible to add custom properties like schema.registry.url for configuring the serializer classes appropriately. Once this is set, Logstash can update to expose this as well.

https://github.com/joekiller/jruby-kafka/blob/master/lib/jruby-kafka/group.rb#L221

looks like Logstash can already start exposing this for the producer:
https://github.com/joekiller/jruby-kafka/blob/master/lib/jruby-kafka/producer.rb#L98

[Kargreat]

@tal: Exactly! we need to make few configuration changes to the logstash-Kafka-input plugin

[Kargreat]

Hey, wanted to quickly check if you are working at enhancing it?

[talevy]

@Kargreat unfortunately, I do not have immediate bandwidth to do so. Just wanted to follow up so that we have possible action plans once time is made available. maybe @joekiller has some suggestions.

[joekiller]

No bandwidth here either but a passing PR would really help me push a new
version out :)
On Jul 8, 2015 3:36 PM, "Tal Levy" notifications@github.com wrote:

@Kargreat https://github.com/Kargreat unfortunately, I do not have
immediate bandwidth to do so. Just wanted to follow up so that we have
possible action plans once time is made available. maybe @joekiller
https://github.com/joekiller has some suggestions.

—
Reply to this email directly or view it on GitHub
#29 (comment)
.

[Kargreat]

Sure!

Sent from my iPhone

On Jul 9, 2015, at 6:24 PM, Joseph Lawson notifications@github.com wrote:

No bandwidth here either but a passing PR would really help me push a new
version out :)
On Jul 8, 2015 3:36 PM, "Tal Levy" notifications@github.com wrote:

@Kargreat https://github.com/Kargreat unfortunately, I do not have
immediate bandwidth to do so. Just wanted to follow up so that we have
possible action plans once time is made available. maybe @joekiller
https://github.com/joekiller has some suggestions.

—
Reply to this email directly or view it on GitHub
#29 (comment)
.

—
Reply to this email directly or view it on GitHub.

[joekiller]

fyi @Kargreat a new version of jruby-kafka is out. It doesn't implement this request but if you have a fork, please check out the changes.

[joekiller]

Finally looked at this some. I think it'll be easiest to just include Avro by default. Related to logstash-plugins/logstash-input-kafka#37 (comment)