Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rsyslog-Kafka: After disconnect, never retries to reconnect #1234

Closed
4 of 9 tasks
djenriquez opened this issue May 26, 2017 · 5 comments
Closed
4 of 9 tasks

Rsyslog-Kafka: After disconnect, never retries to reconnect #1234

djenriquez opened this issue May 26, 2017 · 5 comments
Labels
wait-info

Comments

@djenriquez
Copy link

djenriquez commented May 26, 2017 •
edited

Description

Not sure if this is an rsyslog problem or a librdkafka one, but using Rsyslog-Kafka w/ librdkafka 0.9.5, if rsyslog-kafka is unable to connect to a broker, it will retry 10 times then stop....and it stops forever. Rsyslog continues to log files to the syslog, but the omkafka module no longer tries to send anything out to the kafka brokers. We can resume the omkafka producer by restarting rsyslog.

This is the complete inverse of issue #1089, where librdkafka had a backoff bug and continued to retry forever until the box DoS'd itself. When using this same rsyslog version with version < 0.9.5 of librdkafka, the producer continues to retry, making me believe this to be librdkafka related.

rsyslog-kafka now has the ability to specify librdkafka configuration with the confParam parameter. I'm curious if there a configuration option somewhere in librdkafka that will tell it to come back to life.

Logs:

May 25 23:44:15 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
May 25 23:44:15 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
May 25 23:44:15 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
May 25 23:44:15 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
May 25 23:44:15 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
May 25 23:44:15 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
May 25 23:44:15 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
May 25 23:44:15 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
May 25 23:44:15 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
May 25 23:44:15 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
May 25 23:44:49 ip-10-5-10-102 ec2-user: hello world 3
May 25 23:44:54 ip-10-5-10-102 ec2-user: hello world 4
May 25 23:45:00 ip-10-5-10-102 ec2-user: hello world 5
May 25 23:46:16 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 131810ms.
May 25 23:46:55 ip-10-5-10-102 ec2-user: hello world 6
May 25 23:48:28 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 125810ms.
May 25 23:50:21 ip-10-5-10-102 ec2-user: hello world 7
May 25 23:50:34 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 124650ms.
May 25 23:52:39 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 124800ms.
May 25 23:54:43 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 108990ms.
May 25 23:56:33 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 131520ms.
May 25 23:57:36 ip-10-5-10-102 ec2-user: kafka has not retried yet

How to reproduce

Using rsyslog-kafka v8.27.0 /w librdkafka 0.9.5, set the broker destination to a broker that is no longer listening on port 9092 such that the connection is dropped and the broker is disconnected. Then restore the connection--no new logs are forwarded over to broker.

Checklist

Please provide the following information:

  • librdkafka version (release number or git tag):
    Name : librdkafka
    Arch : x86_64
    Version : 0.9.5
    Release : 1.el6.remi
    Size : 681 k
    Repo : installed
    Summary : Apache Kafka C/C++ client library
    URL : https://github.com/edenhill/librdkafka
    License : BSD and MIT
    Description : librdkafka is a C library implementation of the Apache Kafka protocol,
    : containing both Producer and Consumer support.
    :
    : It was designed with message delivery reliability and high performance
    : in mind, current figures exceed 800000 msgs/second for the producer
    : and 3 million msgs/second for the consumer.
  • Apache Kafka version:
    v0.10.2.1
  • librdkafka client configuration:
    rsyslog-omkafka config:
action(
  broker=["{{KAFKA_DEST}}"]
  type="omkafka"
  topic="logs"
  template="tokafka"
  partitions.auto="on"
  confParam=["socket.max.fails=0", 
             "log.connection.close=false"
             "reconnect.backoff.jitter.ms=1000"]
  
)
  • Operating system:
    Amazon Linux 2017.03.0
  • Using the legacy Consumer
  • Using the high-level KafkaConsumer
  • Provide logs (with debug=.. as necessary) from librdkafka
  • Provide broker log excerpts
  • Critical issue
@djenriquez djenriquez mentioned this issue May 26, 2017
Closed
@djenriquez
Copy link
Author

Any thoughts or comments on this...? Maybe I'm just doing something wrong? I'm cool with that kind of feedback also 😄

@edenhill
Copy link
Contributor

librdkafka will reconnect automatically, but there is no corresponding log message for when it manages to connect (unless debugging is enabled (debug=broker).
But it sounds like you are also not seeing any messages being produced when the broker is back up.

Are you registering a delivery report callback? (dr_msg_cb or dr_cb)
If so: is it being called after message.timeout.ms (default 5 minutes) with a TIMED_OUT error?
You can enable debugging in librdkafka to track what it is doing, set the debug property to broker,msg,topic,metadata,queue to get a good (and verbose) picture of what is happening.

@djenriquez
Copy link
Author

djenriquez commented Jun 2, 2017 •
edited

hi @edenhill, I've added this debug config like so:

action(
  broker=["<REDACTED>"]
  type="omkafka"
  topic="logs"
  template="tokafka"
  partitions.auto="on"
  confParam=["socket.max.fails=0", 
	     "log.connection.close=false",
             "reconnect.backoff.jitter.ms=1000",
	     "debug=broker,msg,topic,metadata,queue" 
 ]
)

but I'm not seeing any additional logs. Does librdkafka throw these debug messages some where on the host?

It looks like librdkafka, whether through rsyslog or internally, will attempt to reconnect for ~10 minutes continuously (with proper backoff) with every log rsyslog picks up, then stop; notice the timestamp of the last few logs without any broker disconnect errors in-between.

Jun  2 19:16:08 ip-10-5-10-102 rsyslogd:  [origin software="rsyslogd" swVersion="8.27.0" x-pid="11473" x-info="http://www.rsyslog.com"] start
Jun  2 19:16:13 ip-10-5-10-102 ec2-user: hello world
Jun  2 19:16:13 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:16:13 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:16:19 ip-10-5-10-102 ec2-user: hello world 2
Jun  2 19:16:19 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:16:19 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:16:22 ip-10-5-10-102 ec2-user: hello world 3
Jun  2 19:16:22 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:16:22 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:16:25 ip-10-5-10-102 ec2-user: hello world 4
Jun  2 19:16:25 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:16:31 ip-10-5-10-102 ec2-user: hello world 5
Jun  2 19:16:31 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:16:31 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:16:31 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 128910ms.
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:02 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:17 ip-10-5-10-102 ec2-user: hello world 6
Jun  2 19:17:17 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:17 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:17 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:17 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:17 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:19 ip-10-5-10-102 ec2-user: hello world 7
Jun  2 19:17:19 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:17:21 ip-10-5-10-102 ec2-user: hello world 8
Jun  2 19:17:21 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 121560ms.
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:11 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:16 ip-10-5-10-102 ec2-user: hello world 9
Jun  2 19:19:16 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:16 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 ec2-user: hello world 10
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:19:49 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:20:14 ip-10-5-10-102 ec2-user: hello world 11
Jun  2 19:20:14 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:20:14 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:20:14 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:20:14 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:20:14 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:20:14 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:20:14 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:20:14 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:20:14 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:20:14 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 ec2-user: hello world 12
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:00 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:03 ip-10-5-10-102 ec2-user: hello world 13
Jun  2 19:21:03 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:05 ip-10-5-10-102 ec2-user: hello world 14
Jun  2 19:21:05 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:06 ip-10-5-10-102 ec2-user: hello world 15
Jun  2 19:21:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:13 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 108470ms.
Jun  2 19:21:13 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:21:13 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 124780ms.
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:01 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:08 ip-10-5-10-102 ec2-user: hello world 16
Jun  2 19:23:08 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:08 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:23:08 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 125590ms.
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:25:06 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 120510ms.
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:27:12 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  2 19:29:13 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 126860ms.
Jun  2 19:31:20 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 125120ms.
Jun  2 19:33:25 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 117590ms.
Jun  2 19:35:22 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 126350ms.
Jun  2 19:37:29 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 110640ms.

Another observation: It also looks like the larger the time gap between logs, the more "brokers are down" error messages pop up.

@djenriquez
Copy link
Author

Update: Looking through the logs over the weekend, omkafka remained down until a HUP was sent Sunday night at 3:16:03 UTC, probably via logrotated since these were the first lines in the log:

Jun  4 03:16:03 ip-10-5-10-102 rsyslogd:  [origin software="rsyslogd" swVersion="8.27.0" x-pid="11473" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jun  4 03:16:34 ip-10-5-10-102 dhclient[2151]: DHCPREQUEST on eth0 to 10.5.10.1 port 67 (xid=0x2d5b9c61)
Jun  4 03:16:34 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  4 03:16:34 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  4 03:16:34 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  4 03:16:34 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  4 03:16:34 ip-10-5-10-102 rsyslogd: omkafka: kafka message 1/1 brokers are down [v8.27.0 try http://www.rsyslog.com/e/2422 ]
Jun  4 03:16:34 ip-10-5-10-102 dhclient[2151]: DHCPACK from 10.5.10.1 (xid=0x2d5b9c61)
Jun  4 03:16:34 ip-10-5-10-102 dhclient[2151]: bound to 10.5.10.102 -- renewal in 1533 seconds.
Jun  4 03:16:34 ip-10-5-10-102 ec2net: [get_meta] Trying to get http://169.254.169.254/latest/meta-data/network/interfaces/macs/06:9f:c8:8e:63:cd/local-ipv4s
Jun  4 03:16:34 ip-10-5-10-102 ec2net: [rewrite_aliases] Rewriting aliases of eth0
.
.
.
Jun  5 18:41:46 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 109300ms.
Jun  5 18:43:36 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 131910ms.
Jun  5 18:43:57 ip-10-5-10-102 su: (to root) ec2-user on pts/0
Jun  5 18:45:48 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 131480ms.
Jun  5 18:47:59 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 131760ms.
Jun  5 18:50:11 ip-10-5-10-102 dhclient[2252]: XMT: Solicit on eth0, interval 131910ms.

It seems to have not come back ever since, current logs still aren't being sent to the brokers.

@edenhill
Copy link
Contributor

debug logs will go to the same place as standard librdkafka logs, but with the log level/severity set to LOG_DEBUG, so it might be filtered by the syslog daemon?

The consistent flood of 1/1 brokers are down indicates that it manages to reconnect before it is being closed down again - in this case you should check the broker logs for hints, unless there are numerous librdkafka producer instances running simultaneously?

@edenhill edenhill mentioned this issue Oct 6, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
wait-info
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@edenhill @djenriquez