Skip to content

Burrow returning empty topic/consumer lists #514

@g-lok

Description

@g-lok

After giving up on using our cluster SASL-Plaintext authentication/encryption (see #380 (comment)), I've tested burrow with simple plaintext (no encryption/authentication) to confirm that it can connect at all. It does seem to connect without error,

{"level":"info","ts":1554492061.736778,"msg":"Started Burrow"}
{"level":"info","ts":1554492061.7368414,"msg":"configuring","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1554492061.7371376,"msg":"configuring","type":"coordinator","name":"storage"}
{"level":"info","ts":1554492061.7371557,"msg":"configuring","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1554492061.7372136,"msg":"configuring","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1554492061.7372262,"msg":"configuring","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1554492061.7372491,"msg":"configuring","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1554492061.746594,"msg":"configuring","type":"coordinator","name":"notifier"}
{"level":"info","ts":1554492061.7466223,"msg":"configuring","type":"coordinator","name":"cluster"}
{"level":"info","ts":1554492061.7466445,"msg":"configuring","type":"module","coordinator":"cluster","class":"kafka","name":"myclustername"}
{"level":"info","ts":1554492061.7618148,"msg":"configuring","type":"coordinator","name":"consumer"}
{"level":"info","ts":1554492061.7618647,"msg":"configuring","type":"module","coordinator":"consumer","class":"kafka","name":"myconsumers"}
{"level":"info","ts":1554492061.7975335,"msg":"starting","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1554492061.8112516,"msg":"Connected to 52.27.94.190:2181","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1554492061.815502,"msg":"Authenticated: id=101861002933334153, timeout=6000","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1554492061.8155353,"msg":"Re-submitting `0` credentials after reconnect","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1554492061.820379,"msg":"starting","type":"coordinator","name":"storage"}
{"level":"info","ts":1554492061.820402,"msg":"starting","type":"module","coordinator":"storage","class":"inmemory","name":"default"}
{"level":"info","ts":1554492061.8204606,"msg":"starting","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1554492061.820469,"msg":"starting","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1554492061.8204806,"msg":"starting","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1554492061.8206592,"msg":"started listener","type":"coordinator","name":"httpserver","listener":"[::]:8443"}
{"level":"info","ts":1554492061.820674,"msg":"starting","type":"coordinator","name":"notifier"}
{"level":"info","ts":1554492061.820687,"msg":"starting","type":"coordinator","name":"cluster"}
{"level":"info","ts":1554492061.8206909,"msg":"starting","type":"module","coordinator":"cluster","class":"kafka","name":"myclustername"}

but returns empty lists of topics and consumers when queried:

$ curl https://my.burrow:8443/v3/kafka/myclustername/topic
{"error":false,"message":"topic list returned","topics":[],"request":{"url":"/v3/kafka/myclustername/topic","host":"my.burrow"}}

$ curl https://my.burrow:8443/v3/kafka/myclustername/consumer
{"error":false,"message":"consumer list returned","consumers":[],"request":{"url":"/v3/kafka/myclustername/consumer","host":"my.burrow"}}

My config:

[general]
pidfile="/var/lock/burrow/burrow.pid"
stdout-logfile="burrow.out"

[zookeeper]
servers=[
  "my.zookeeper:2181"
]
timeout=6
root-path="/burrow"

[client-profile.myclient]
kafka-version="0.9.0"
client-id="burrow-myclient"
tls="mytlsprofile"

[tls.mytlsprofile]
certfile="whatever.crt"
keyfile="whatever.key"
cafile="/etc/pki/tls/certs/ca-bundle.crt"
noverify=true

[httpserver.tlslistener]
address=":8443"
timeout=300
tls="mytlsprofile"

[cluster.myclustername]
class-name="kafka"
servers=[
  "my.kafka:9092"
]
client-profile="myclient"
topic-refresh=120
offset-refresh=30

[consumer.myconsumers]
class-name="kafka"
cluster="myclustername"
servers=[
  "my.kafka:9092"
]
client-profile="myclient"
offsets-topic="__consumer_offsets"
start-latest=true
group-whitelist=".*"

Kafka ACLs:

Current ACLs for resource `Topic:LITERAL:__consumer_offsets`:
  User:* has Allow permission for operations: All from hosts: *
  User:ANONYMOUS has Allow permission for operations: All from hosts: *

Current ACLs for resource `Group:LITERAL:*`:
  User:* has Allow permission for operations: All from hosts: *
  User:ANONYMOUS has Allow permission for operations: All from hosts: *

Current ACLs for resource `Topic:LITERAL:*`:
  User:ANONYMOUS has Allow permission for operations: Read from hosts: *

Using confluent kafka 2.0.0 (which I believe is kafka v. 0.9.0 or 0.9.0.0). I have tried this with various combinations of the above settings (such as noverify set to true or false), kafka 0.9.0 vs 0.9.0.0, direction connections to zookeeper/kafka vs. AWS ELBs, etc. Nothing seems to work

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions