Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installation of on premise erroring out with Connection to Kafka failed #472

Closed
khangtoh opened this issue May 5, 2020 · 21 comments · Fixed by #511
Closed

Installation of on premise erroring out with Connection to Kafka failed #472

khangtoh opened this issue May 5, 2020 · 21 comments · Fixed by #511
Assignees

Comments

@khangtoh
Copy link

khangtoh commented May 5, 2020

Cloning master version to ubuntu 18.04 and running the install script

2020-05-05 09:29:21,527 Connection to Kafka failed (attempt 59)
Traceback (most recent call last):
File "/usr/src/snuba/snuba/cli/bootstrap.py", line 56, in bootstrap
client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
Traceback (

ubuntu@ubuntu-s-6vcpu-16gb-sgp1-01:~/onpremise$ docker -v Docker version 19.03.8, build afacb8b7f0

ubuntu@ubuntu-s-6vcpu-16gb-sgp1-01:~/onpremise$ docker-compose -v docker-compose version 1.25.5, build unknown

@cristicotet
Copy link

Same error when updating Sentry from older 10.1.0-dev0 to latest but in case of update it seems to start correctly if i run docker-compose up -d
I'm not sure if the update is complete :)

@ChibangLW
Copy link

Looks like reverting #465 will "fix" this. With kafka and zookeeper on 5.5 both containers kept failing for me.

@maximal
Copy link

maximal commented May 5, 2020

Had a similar issue in January: see #331

Removing some containers and Kafka volume helped me back then. But it does not resolve the issue this time.

@BYK
Copy link
Member

BYK commented May 6, 2020

@maximal maybe you need to remove the zookeeper volumes and containers this time too?

Apologies for the issues, I was expecting this upgrade to be painless but seems like that's not really the case.

@BYK BYK self-assigned this May 6, 2020
@BYK BYK added the Type: Bug label May 6, 2020
@maximal
Copy link

maximal commented May 6, 2020

@maximal maybe you need to remove the zookeeper volumes and containers this time too?

Removing zookeeper volumes helped even with the newer kafka and zookeeper versions in docker-compose.yml, thanks!

docker-compose down
docker volume rm sentry-kafka sentry-zookeeper
docker volume rm sentry_onpremise_sentry-kafka-log sentry_onpremise_sentry-zookeeper-log
./install

Apologies for the issues, I was expecting this upgrade to be painless but seems like that's not really the case.

No worries! We’re using unstable branch here, testing it so I suppose we all realize the consequences :-)

@BYK
Copy link
Member

BYK commented May 6, 2020

No worries! We’re using unstable branch here, testing it so I suppose we all realize the consequences :-)

Eh, if only we were providing a "stable" version, you'd be right. This is around the corner though so stay tuned and thanks again!

@cristicotet
Copy link

Solution proposed by @maximal works in my case (update from older 10.1.0-dev0)

@BYK
Copy link
Member

BYK commented May 7, 2020

If anyone has still access to ZooKeeper logs, can you share them here? You can run docker-compose logs zookeeper to get them.

@BYK BYK added the needs-info label May 7, 2020
@BuSHari
Copy link

BuSHari commented May 7, 2020

I'm getting this error too.

Here is the zookeeper logs:

Attaching to sentry_onpremise_zookeeper_1
zookeeper_1                | ===> ENV Variables ...
zookeeper_1                | ALLOW_UNSIGNED=false
zookeeper_1                | COMPONENT=zookeeper
zookeeper_1                | CONFLUENT_DEB_VERSION=1
zookeeper_1                | CONFLUENT_PLATFORM_LABEL=
zookeeper_1                | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1                | CONFLUENT_VERSION=5.5.0
zookeeper_1                | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1                | HOME=/root
zookeeper_1                | HOSTNAME=a82a7743d431
zookeeper_1                | KAFKA_VERSION=
zookeeper_1                | LANG=C.UTF-8
zookeeper_1                | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1                | PWD=/
zookeeper_1                | PYTHON_PIP_VERSION=8.1.2
zookeeper_1                | PYTHON_VERSION=2.7.9-1
zookeeper_1                | SCALA_VERSION=2.12
zookeeper_1                | SHLVL=1
zookeeper_1                | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1                | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1                | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1                | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1                | _=/usr/bin/env
zookeeper_1                | ===> User
zookeeper_1                | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1                | ===> Configuring ...
zookeeper_1                | ===> Running preflight checks ...
zookeeper_1                | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1                | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1                | ===> Launching ...
zookeeper_1                | ===> Launching zookeeper ...
zookeeper_1                | [2020-05-07 19:40:27,042] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1                | [2020-05-07 19:40:28,135] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                | [2020-05-07 19:40:28,140] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                | [2020-05-07 19:40:28,833] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
zookeeper_1                | java.io.IOException: No snapshot found, but there are log entries. Something is broken!
zookeeper_1                | 	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
zookeeper_1                | 	at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
zookeeper_1                | 	at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
zookeeper_1                | 	at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
zookeeper_1                | 	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
zookeeper_1                | 	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
zookeeper_1                | ===> ENV Variables ...
zookeeper_1                | ALLOW_UNSIGNED=false
zookeeper_1                | COMPONENT=zookeeper
zookeeper_1                | CONFLUENT_DEB_VERSION=1
zookeeper_1                | CONFLUENT_PLATFORM_LABEL=
zookeeper_1                | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1                | CONFLUENT_VERSION=5.5.0
zookeeper_1                | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1                | HOME=/root
zookeeper_1                | HOSTNAME=a82a7743d431
zookeeper_1                | KAFKA_VERSION=
zookeeper_1                | LANG=C.UTF-8
zookeeper_1                | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1                | PWD=/
zookeeper_1                | PYTHON_PIP_VERSION=8.1.2
zookeeper_1                | PYTHON_VERSION=2.7.9-1
zookeeper_1                | SCALA_VERSION=2.12
zookeeper_1                | SHLVL=1
zookeeper_1                | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1                | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1                | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1                | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1                | _=/usr/bin/env
zookeeper_1                | ===> User
zookeeper_1                | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1                | ===> Configuring ...

And kafka:

Attaching to sentry_onpremise_kafka_1
kafka_1                    | ===> ENV Variables ...
kafka_1                    | ALLOW_UNSIGNED=false
kafka_1                    | COMPONENT=kafka
kafka_1                    | CONFLUENT_DEB_VERSION=1
kafka_1                    | CONFLUENT_PLATFORM_LABEL=
kafka_1                    | CONFLUENT_SUPPORT_METRICS_ENABLE=false
kafka_1                    | CONFLUENT_VERSION=5.5.0
kafka_1                    | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
kafka_1                    | HOME=/root
kafka_1                    | HOSTNAME=02cd0ee0887c
kafka_1                    | KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
kafka_1                    | KAFKA_LOG4J_LOGGERS=kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN
kafka_1                    | KAFKA_LOG4J_ROOT_LOGLEVEL=WARN
kafka_1                    | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
kafka_1                    | KAFKA_TOOLS_LOG4J_LOGLEVEL=WARN
kafka_1                    | KAFKA_VERSION=
kafka_1                    | KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
kafka_1                    | LANG=C.UTF-8
kafka_1                    | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
kafka_1                    | PWD=/
kafka_1                    | PYTHON_PIP_VERSION=8.1.2
kafka_1                    | PYTHON_VERSION=2.7.9-1
kafka_1                    | SCALA_VERSION=2.12
kafka_1                    | SHLVL=1
kafka_1                    | ZULU_OPENJDK_VERSION=8=8.38.0.13
kafka_1                    | _=/usr/bin/env
kafka_1                    | ===> User
kafka_1                    | uid=0(root) gid=0(root) groups=0(root)
kafka_1                    | ===> Configuring ...
kafka_1                    | ===> Running preflight checks ...
kafka_1                    | ===> Check if /var/lib/kafka/data is writable ...
kafka_1                    | ===> Check if Zookeeper is healthy ...
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=02cd0ee0887c
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.4.0-166-generic
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=955MB
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=14315MB
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=966MB
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d
kafka_1                    | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
kafka_1                    | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
kafka_1                    | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /192.168.192.10:42838, server: zookeeper/192.168.192.5:2181
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper/192.168.192.5:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.io.IOException: Connection reset by peer
kafka_1                    | 	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
kafka_1                    | 	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
kafka_1                    | 	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
kafka_1                    | 	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
kafka_1                    | 	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main] ERROR io.confluent.admin.utils.ClusterStatus - Timed out waiting for connection to Zookeeper server [zookeeper:2181].
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x0 closed
kafka_1                    | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x0
kafka_1                    | ===> ENV Variables ...
kafka_1                    | ALLOW_UNSIGNED=false
kafka_1                    | COMPONENT=kafka
kafka_1                    | CONFLUENT_DEB_VERSION=1
kafka_1                    | CONFLUENT_PLATFORM_LABEL=
kafka_1                    | CONFLUENT_SUPPORT_METRICS_ENABLE=false
kafka_1                    | CONFLUENT_VERSION=5.5.0
kafka_1                    | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
kafka_1                    | HOME=/root
kafka_1                    | HOSTNAME=02cd0ee0887c
kafka_1                    | KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
kafka_1                    | KAFKA_LOG4J_LOGGERS=kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN
kafka_1                    | KAFKA_LOG4J_ROOT_LOGLEVEL=WARN
kafka_1                    | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
kafka_1                    | KAFKA_TOOLS_LOG4J_LOGLEVEL=WARN
kafka_1                    | KAFKA_VERSION=
kafka_1                    | KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
kafka_1                    | LANG=C.UTF-8
kafka_1                    | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
kafka_1                    | PWD=/
kafka_1                    | PYTHON_PIP_VERSION=8.1.2
kafka_1                    | PYTHON_VERSION=2.7.9-1
kafka_1                    | SCALA_VERSION=2.12
kafka_1                    | SHLVL=1
kafka_1                    | ZULU_OPENJDK_VERSION=8=8.38.0.13
kafka_1                    | _=/usr/bin/env
kafka_1                    | ===> User
kafka_1                    | uid=0(root) gid=0(root) groups=0(root)
kafka_1                    | ===> Configuring ...
kafka_1                    | ===> Running preflight checks ...
kafka_1                    | ===> Check if /var/lib/kafka/data is writable ...
kafka_1                    | ===> Check if Zookeeper is healthy ...
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=02cd0ee0887c
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_212
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.4.0-166-generic
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=955MB
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=14315MB
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=966MB
kafka_1                    | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@cc34f4d
kafka_1                    | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
kafka_1                    | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
kafka_1                    | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
kafka_1                    | [main-SendThread()] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper: Temporary failure in name resolution
kafka_1                    | 	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
kafka_1                    | 	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
kafka_1                    | 	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper: Temporary failure in name resolution
kafka_1                    | 	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
kafka_1                    | 	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
kafka_1                    | 	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] ERROR org.apache.zookeeper.client.StaticHostProvider - Unable to resolve address: zookeeper:2181
kafka_1                    | java.net.UnknownHostException: zookeeper
kafka_1                    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
kafka_1                    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
kafka_1                    | 	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
kafka_1                    | [main-SendThread(zookeeper:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect
kafka_1                    | java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper:2181 because it's not resolvable
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:71)
kafka_1                    | 	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:39)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1087)
kafka_1                    | 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1139)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket error occurred: zookeeper/192.168.192.5:2181: Connection refused
kafka_1                    | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/192.168.192.5:2181. Will not attempt to authenticate using SASL (unknown error)

@pnghai
Copy link

pnghai commented May 14, 2020

Here's yours:

zookeeper_1                | ===> ENV Variables ...
zookeeper_1                | ALLOW_UNSIGNED=false
zookeeper_1                | COMPONENT=zookeeper
zookeeper_1                | CONFLUENT_DEB_VERSION=1
zookeeper_1                | CONFLUENT_PLATFORM_LABEL=
zookeeper_1                | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1                | CONFLUENT_VERSION=5.5.0
zookeeper_1                | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1                | HOME=/root
zookeeper_1                | HOSTNAME=0e076b9fc016
zookeeper_1                | KAFKA_VERSION=
zookeeper_1                | LANG=C.UTF-8
zookeeper_1                | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1                | PWD=/
zookeeper_1                | PYTHON_PIP_VERSION=8.1.2
zookeeper_1                | PYTHON_VERSION=2.7.9-1
zookeeper_1                | SCALA_VERSION=2.12
zookeeper_1                | SHLVL=1
zookeeper_1                | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1                | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1                | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1                | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1                | _=/usr/bin/env
zookeeper_1                | ===> User
zookeeper_1                | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1                | ===> Configuring ...
zookeeper_1                | ===> Running preflight checks ...
zookeeper_1                | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1                | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1                | ===> Launching ...
zookeeper_1                | ===> Launching zookeeper ...
zookeeper_1                | [2020-05-14 10:45:54,588] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1                | [2020-05-14 10:45:54,790] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                | [2020-05-14 10:45:54,790] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                | [2020-05-14 10:45:55,004] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
zookeeper_1                | java.io.IOException: No snapshot found, but there are log entries. Something is broken!
zookeeper_1                | 	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
zookeeper_1                | 	at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
zookeeper_1                | 	at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
zookeeper_1                | 	at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
zookeeper_1                | 	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
zookeeper_1                | 	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
zookeeper_1                | ===> ENV Variables ...
zookeeper_1                | ALLOW_UNSIGNED=false
zookeeper_1                | COMPONENT=zookeeper
zookeeper_1                | CONFLUENT_DEB_VERSION=1
zookeeper_1                | CONFLUENT_PLATFORM_LABEL=
zookeeper_1                | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1                | CONFLUENT_VERSION=5.5.0
zookeeper_1                | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1                | HOME=/root
zookeeper_1                | HOSTNAME=0e076b9fc016
zookeeper_1                | KAFKA_VERSION=
zookeeper_1                | LANG=C.UTF-8
zookeeper_1                | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1                | PWD=/
zookeeper_1                | PYTHON_PIP_VERSION=8.1.2
zookeeper_1                | PYTHON_VERSION=2.7.9-1
zookeeper_1                | SCALA_VERSION=2.12
zookeeper_1                | SHLVL=1
zookeeper_1                | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1                | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1                | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1                | ZULU_OPENJDK_VERSION=8=8.38.0.13
zookeeper_1                | _=/usr/bin/env
zookeeper_1                | ===> User
zookeeper_1                | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1                | ===> Configuring ...
zookeeper_1                | ===> Running preflight checks ...
zookeeper_1                | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1                | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1                | ===> Launching ...
zookeeper_1                | ===> Launching zookeeper ...
zookeeper_1                | [2020-05-14 10:46:13,605] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
zookeeper_1                | [2020-05-14 10:46:13,823] WARN o.e.j.s.ServletContextHandler@4d95d2a2{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                | [2020-05-14 10:46:13,823] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
zookeeper_1                | [2020-05-14 10:46:14,001] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
zookeeper_1                | java.io.IOException: No snapshot found, but there are log entries. Something is broken!
zookeeper_1                | 	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:240)
zookeeper_1                | 	at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
zookeeper_1                | 	at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
zookeeper_1                | 	at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
zookeeper_1                | 	at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
zookeeper_1                | 	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
zookeeper_1                | 	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)

If anyone has still access to ZooKeeper logs, can you share them here? You can run docker-compose logs zookeeper to get them.

@jellevanhees
Copy link

https://issues.apache.org/jira/browse/ZOOKEEPER-3056

this might fix the issue in zookeeper. but i don't know the correct way to pass this setting to the container in this setup.

@BYK
Copy link
Member

BYK commented May 15, 2020

I think there's a fix upcoming. In the meantime, can someone try adding ZOOKEEPER_SNAPSHOT_TRUST_EMPTY: 'true' under zookeeper's environment section and see if it helps?

@BYK BYK removed the needs-info label May 15, 2020
@xf-
Copy link
Contributor

xf- commented May 16, 2020

@BYK stops the loop (error appears once) and install continues

@BYK
Copy link
Member

BYK commented May 18, 2020

@xf- so that means adding this value solves the issue?

@xf-
Copy link
Contributor

xf- commented May 18, 2020

@BYK like i wrote. I get the same error but only once and the update continues.
= Looks like it works, but still the same error in console (only once) and maybe somebody will reopen/create a new issue.

@caugner
Copy link

caugner commented May 20, 2020

@maximal maybe you need to remove the zookeeper volumes and containers this time too?

Removing zookeeper volumes helped even with the newer kafka and zookeeper versions in docker-compose.yml, thanks!

docker-compose down
docker volume rm sentry-kafka sentry-zookeeper
docker volume rm sentry_onpremise_sentry-kafka-log sentry_onpremise_sentry-zookeeper-log
./install

This solution did not work for me when upgrading from 10.1.0.dev0 (e54cbeb).

I think there's a fix upcoming. In the meantime, can someone try adding ZOOKEEPER_SNAPSHOT_TRUST_EMPTY: 'true' under zookeeper's environment section and see if it helps?

This workaround by @BYK did not work either.

PS: According to docker stats, there are no issues with CPU or Memory consumption. Both kafka and zookeeper seem to be running despite the timeout.

Update: At least I was able to revert to my previous version (with latest git checkout): SENTRY_IMAGE=getsentry/sentry:e54cbeb ./install.sh

@BYK
Copy link
Member

BYK commented May 22, 2020

@caugner that's quite peculiar that going back to an earlier Sentry image fixing this. I bet it's not Sentry getting reverted itself but instead Snuba and other services getting updated or restarted. I'd recommend running ./install.sh without the SENTRY_IMAGE override and see what happens.

BYK added a commit that referenced this issue May 23, 2020
Fixes the "logs found but no snapshot" error when upgrading from a
short-lived older version. Fixes #472.

See https://issues.apache.org/jira/browse/ZOOKEEPER-3056.
@BYK
Copy link
Member

BYK commented May 23, 2020

Hey all, can you check whether #511 fixes the issue for you? I tried it locally with a broken ZK and it did for me but trying to be sure that it works for everyone.

/cc @caugner @xf-

@BYK BYK closed this as completed in #511 May 24, 2020
BYK added a commit that referenced this issue May 24, 2020
Fixes the "logs found but no snapshot" error when upgrading from a
short-lived older version. Fixes #472.

See https://issues.apache.org/jira/browse/ZOOKEEPER-3056.
@jansorg
Copy link

jansorg commented May 25, 2020

@BYK I've pulled 83160e8 and I'm still seeing the errors of this issue when I run ./setup.sh.

^T2020-05-25 11:20:03,768 Connection to Kafka failed (attempt 59)
Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 56, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
Traceback (most recent call last):
  File "/usr/local/bin/snuba", line 11, in <module>
    load_entry_point('snuba', 'console_scripts', 'snuba')()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 56, in bootstrap
    client.list_topics(timeout=1)
cimpl.KafkaExcepti

@BYK
Copy link
Member

BYK commented May 25, 2020

@jansorg it is normal to see it a few times. If the installation finishes successfully, just ignore it. If your install doesn't succeed, please file a new issue and share your Kafka and Zookeeper logs.

@caugner
Copy link

caugner commented May 28, 2020

I was able to update from e54cbeb to df57809 now, but it could also be that my connectivity issues (#505 (comment)) played a role here.

MaicolBen pushed a commit to hinthealth/onpremise that referenced this issue Jul 20, 2020
Fixes the "logs found but no snapshot" error when upgrading from a
short-lived older version. Fixes getsentry#472.

See https://issues.apache.org/jira/browse/ZOOKEEPER-3056.
@github-actions github-actions bot locked and limited conversation to collaborators Dec 14, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.