Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka_consumer not working in 1.9.2 #5325

Closed
akamac opened this issue Jan 22, 2019 · 5 comments · Fixed by #5508
Closed

Kafka_consumer not working in 1.9.2 #5325

akamac opened this issue Jan 22, 2019 · 5 comments · Fixed by #5508
Assignees
Labels
area/kafka bug unexpected problem or unintended behavior

Comments

@akamac
Copy link

akamac commented Jan 22, 2019

Relevant telegraf.conf:

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  debug = true
  quiet = false
  logfile = ""

[[inputs.kafka_consumer]]
  brokers = ["kafka:9092"]
  topics = ["linux", "hyperv"]
  # client_id = "Telegraf"
  version = "2.0.0"
  consumer_group = "influxdb"
  offset = "oldest"
  # max_undelivered_messages = 1000
  data_format = "influx"

[[aggregators.basicstats]]
  period = "30s"
  drop_original = false
  stats = ["mean"]
  [aggregators.basicstats.tags]
    aggr = "basicstats"

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "linux"
  skip_database_creation = false
  retention_policy = "1month"
  username = "admin"
  password = "password"
  http_proxy = ""
  content_encoding = "gzip"
  influx_uint_support = false
  [outputs.influxdb.tagpass]
    kafka_topic = ["linux"]
  [outputs.influxdb.tagdrop]
    aggr = ["basicstats"]

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "linux"
  skip_database_creation = false
  retention_policy = "1year"
  username = "admin"
  password = "password"
  http_proxy = ""
  content_encoding = "gzip"
  influx_uint_support = false
  [outputs.influxdb.tagpass]
    kafka_topic = ["linux"]
    aggr = ["basicstats"]

System info:

Ubuntu 18.04
Confluent Kafka 2.11
Telegraf 1.9.2
InfluxDB 1.7.3

Actual behavior:

Logs: D! [outputs.influxdb] buffer fullness: 0 / 10000 metrics.
Offset is not updated for Kafka consumer group.

Additional info:

Rolling back to 1.9.1 fixes the issue.

@danielnelson danielnelson changed the title v1.9.2 is broken Kafka_consumer not working in 1.9.2 Jan 22, 2019
@danielnelson
Copy link
Contributor

Could you try adding these plugins to write some additional data to /tmp/telegraf, run Telegraf for several minutes, and attach the file here?

[[inputs.internal]]
  collect_memstats = false

[[outputs.file]]
  namepass = ["internal*"]
  files = ["/tmp/telegraf"]

@danielnelson danielnelson added bug unexpected problem or unintended behavior area/kafka labels Jan 22, 2019
@akamac
Copy link
Author

akamac commented Jan 26, 2019

Here you go:
telegraf.txt
Thanks!

@danielnelson
Copy link
Contributor

@akamac Is this is still an issue in 1.9.3?

@akamac
Copy link
Author

akamac commented Feb 18, 2019

@danielnelson, The issue persists as of 1.9.4.

@danielnelson
Copy link
Contributor

@akamac I believe I found the issue, any chance you can test it out with this build?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kafka bug unexpected problem or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants