Confluent's Avro / Schema Registry support?

[simplesteph]

I'd be amazing to have an -avro http://schema-registry-url flag (i.e. confluent's avro) so we can deserialise avro messages while reading a topic

[edenhill]

What would it deserialize to? JSON?

[simplesteph]

Yes! Similar behaviour as the avro console consumer from confluent

[derekcline]

This would be a really helpful feature!

[Dabz]

+1

[llofberg]

#151

[gschmutz]

+1

[lcuquejo]

+1

[msiddiqi]

+1

[ehrlichja]

+1

[anderseriksson]

+1

[TsuyoshiUshio]

+1

[dkalcheu]

+1

[kkoch-od]

+1

[jboulineau]

+1

[spoissant]

+1

[raphaelsolarski]

+1

[Tempys]

so what the status of that ?

[lsimmons2]

+1

[lsimmons2]

Unless I'm missing something, seems like this is described as a feature in the README:

Decode Avro key (-s key=avro), value (-s value=avro) or both (-s avro) to JSON using schema from the Schema-Registry:

When I try to run though with kafkacat version 1.4.0 I get the error kafkacat: illegal option -- s.

[llofberg]

On a Mac with Homebrew you can try editing the kafkacat recipe with brew edit kafkacat as seen below and then reinstall kafkacat with brew install --HEAD kafkacat. You may need to add more dependencies too.

class Kafkacat < Formula
  desc "Generic command-line non-JVM Apache Kafka producer and consumer"
  homepage "https://github.com/edenhill/kafkacat"
  url "https://github.com/edenhill/kafkacat.git",
      :tag      => "1.4.0",
      :revision => "e685f7e05a184e06a9a1a34abffbb3676a2d6f95"
  head "https://github.com/edenhill/kafkacat.git",
      :revision => "6bb21640611621c13bd0567680389b7171286592"

  bottle do
    cellar :any
    sha256 "93f09459a4a835dc4506c78ffd2a22f28a357f1dc359f1083c1a68b6c59e914b" => :mojave
    sha256 "1a22bb79786c31a5c58114b79571171f339b901529e302c3d99311b385075d2a" => :high_sierra
    sha256 "f087b771ff8f6a7de9fa412980b4449139b1130bf1dd8fc0d58380bfcc334551" => :sierra
  end

  depends_on "librdkafka"
  depends_on "yajl"

  def install
    system "./configure", "--prefix=#{prefix}",
                          "--enable-json",
                          "--enable-avro"
    system "make"
    system "make", "install"
  end

  test do
    system bin/"kafkacat", "-X", "list"
  end
end

[TDanielsHL]

Hi,

I followed the steps above and received the following error:
% ERROR: This build of kafkacat lacks Avro/Schema-Registry support

[colaho]

Hi,

Got the same error as % ERROR: This build of kafkacat lacks Avro/Schema-Registry support after following the step. Any update on this?

[edenhill]

Use docker image edenhill/kafkacat:20190711

[homdoq]

docker run -t edenhill/kafkacat:20190711 kafkacat -b <host:port>

on Windows 10, docker desktop v 2.1.0.1 returns error:

Error: -b <broker,..> missing

Usage: kafkacat <options> [file1 file2 .. | topic1 topic2 ..]]
kafkacat - Apache Kafka producer and consumer tool
https://github.com/edenhill/kafkacat
Copyright (c) 2014-2019, Magnus Edenhill
Version 1.4.0 (JSON, Avro, librdkafka 1.1.0 builtin.features=gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer)
...

The same command with 20190709 "works" in that it complains about a missing topic rather than a missing broker (however on 20190709 -L does not work)

[edenhill]

@homdoq You shouldn't specify "kafkacat" as an argument, kafkacat is already the entrypoint.

[homdoq]

Thanks! I just followed the examples using the confluent image...

[vedal]

How do you include schema.registry.basic.auth.credentials.source and schema.registry.basic.auth.user.info in the request? there seems only to be the -r option?

[edenhill]

Try http://user:pass@srurl:port/

[a6kme]

For Confluent Schema registry, you might have to encode the password, since the password might contain URL unsafe characters.

I use https://www.url-encode-decode.com/ to quickly encode the password.

Then you can easily use -r https://USERNAME:URL_ENCODED_PASSWORD@SRURL and it will work like a charm.