Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

database routing breaks at a restart of InfluxDB #6471

Closed
x70b1 opened this issue Oct 1, 2019 · 1 comment · Fixed by #6486
Closed

database routing breaks at a restart of InfluxDB #6471

x70b1 opened this issue Oct 1, 2019 · 1 comment · Fixed by #6486
Labels
bug unexpected problem or unintended behavior
Milestone

Comments

@x70b1
Copy link
Contributor

x70b1 commented Oct 1, 2019

I setup database routing. Every service (MySQL, RabbitMQ and Jenkins) should use it's own database in InfluxDB. The used config is attached.
This works good so far.

If I restart the influxdb service the routing is broken for 3 to 4 data points (Interval is 10s).
This means that every collected data point is written to every configured database.

  • I got 3 to 4 MySQL, RabbitMQ and Jenkins data points in my MySQL database.
  • I got 3 to 4 MySQL, RabbitMQ and Jenkins data points in my RabbitMQ database.
  • I got 3 to 4 MySQL, RabbitMQ and Jenkins data points in my Jenkins database.

This is not a very critical error but it's causing confusion.
Also I am not sure if this is a problem with Telegraf, because InfluxDB is also involved.

I could not find any logs related to this. There are lines like read: connection reset by peer. But that's what I'm expecting.

Relevant telegraf.conf:

[[outputs.influxdb]]
  urls = ["http://127.0.0.1:8086"]
  username = "telegraf"
  password = "xxx"
  database = "RabbitMq"

  skip_database_creation = true
  database_tag = "influxdb_database"
  exclude_database_tag = true

[[outputs.influxdb]]
  urls = ["http://127.0.0.1:8086"]
  username = "telegraf"
  password = "xxx"
  database = "Jenkins"

  skip_database_creation = true
  database_tag = "influxdb_database"
  exclude_database_tag = true

[[outputs.influxdb]]
  urls = ["http://127.0.0.1:8086"]
  username = "telegraf"
  password = "xxx"
  database = "Mysql"

  skip_database_creation = true
  database_tag = "influxdb_database"
  exclude_database_tag = true

[[inputs.rabbitmq]]
  url = "https://rabbit.example.org"
  username = "xxx"
  password = "xxx"

  [inputs.rabbitmq.tags]
    influxdb_database = "RabbitMq"

[[inputs.jenkins]]
  url = "https://example.com"
  username = "xxx"
  password = "xxx"

  [inputs.jenkins.tags]
    influxdb_database = "Jenkins"

[[inputs.mysql]]
  servers = ["xxx"]

  [inputs.mysql.tags]
    influxdb_database = "Mysql"

System info:

1.12.0 as Docker Container

Steps to reproduce:

  1. check the list of measurements in all 3 databases
  2. restart InfluxDB
  3. check the list of measurements in all 3 databases

Expected behavior:

Every database should contain its own measurements.

Actual behavior:

A small amount of values get the wrong database.

@danielnelson danielnelson added the bug unexpected problem or unintended behavior label Oct 2, 2019
@danielnelson danielnelson added this to the 1.12.3 milestone Oct 2, 2019
@danielnelson
Copy link
Contributor

@x70b1 Thanks for the report, as a workaround for the bug you can use this alternate style:

 [[outputs.influxdb]]
  urls = ["http://127.0.0.1:8086"]
  username = "telegraf"
  password = "xxx"
  database = "RabbitMq"
  skip_database_creation = true

  tagexclude = ["influxdb_database"]
  [outputs.influxdb.tagpass]
    influxdb_database = ["RabbitMq"]


[[inputs.rabbitmq]]
  url = "https://rabbit.example.org"
  username = "xxx"
  password = "xxx"

  [inputs.rabbitmq.tags]
    influxdb_database = "RabbitMq"

This will actually be more efficient even when the bug is fixed because only the metrics that match tagpass are added to the output's metric buffer. With the original configuration all metrics are added to every output's metric buffer and then written to the database depending on the influxdb_database tag.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants