Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Zookeeper connection error Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] #16

Open
honghaic opened this issue Feb 9, 2015 · 12 comments
Open

Zookeeper connection error Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] #16

honghaic opened this issue Feb 9, 2015 · 12 comments

Comments

@honghaic
Copy link

honghaic commented Feb 9, 2015

2015-02-08 16:23:49,016 - [INFO] - from org.apache.zookeeper.ClientCnxn in kafka-manager-system-akka.actor.default-dispatcher-7-SendThread(10.65.196.37:2181)
Opening socket connection to server 10.65.196.37/10.65.196.37:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-08 16:23:49,312 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-2
Could not read WebJar RequireJS config for: dustjs-linkedin 2.4.0
Please file a bug at: http://github.com/webjars/dustjs-linkedin/issues/new

2015-02-08 16:23:49,354 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-2
Could not read WebJar RequireJS config for: json 20121008
Please file a bug at: http://github.com/webjars/json/issues/new

2015-02-08 16:23:49,378 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-2
Could not read WebJar RequireJS config for: requirejs 2.1.10
Please file a bug at: http://github.com/webjars/requirejs/issues/new

2015-02-08 16:23:58,829 - [ERROR] - from kafka.manager.ApiError in pool-3-thread-2
error : Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) ~[akka-actor_2.11-2.3.7.jar:na]
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) ~[scala-library-2.11.4.jar:na]
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) ~[akka-actor_2.11-2.3.7.jar:na]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_40-MS]

2015-02-08 16:24:04,065 - [ERROR] - from org.apache.curator.ConnectionState in kafka-manager-system-akka.actor.default-dispatcher-7
Connection timed out for connection string (10.65.196.35:2181,10.65.196.36:2181,10.65.196.37:2181) and timeout (15000) / elapsed (15114)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.7.0.jar:na]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87) [curator-client-2.7.0.jar:na]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) ~[curator-client-2.7.0.jar:na]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:492) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:675) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:671) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:453) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:443) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:423) ~[curator-framework-2.7.0.jar:na]
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44) ~[curator-framework-2.7.0.jar:na]
at kafka.manager.KafkaManagerActor$$anonfun$6.apply(KafkaManagerActor.scala:184) ~[kafka-manager_2.11-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at kafka.manager.KafkaManagerActor$$anonfun$6.apply(KafkaManagerActor.scala:184) ~[kafka-manager_2.11-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at scala.util.Try$.apply(Try.scala:191) ~[scala-library-2.11.4.jar:na]
at kafka.manager.KafkaManagerActor.(KafkaManagerActor.scala:184) ~[kafka-manager_2.11-1.0-SNAPSHOT.jar:1.0-SNAPSHOT]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_40-MS]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_40-MS]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_40-MS]
at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_40-MS]
at akka.util.Reflect$.instantiate(Reflect.scala:66) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.ArgsReflectConstructor.produce(Props.scala:352) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.Props.newActor(Props.scala:252) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.ActorCell.newActor(ActorCell.scala:552) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.ActorCell.create(ActorCell.scala:578) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.dispatch.Mailbox.run(Mailbox.scala:220) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.dispatch.Mailbox.exec(Mailbox.scala:231) ~[akka-actor_2.11-2.3.7.jar:na]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[scala-library-2.11.4.jar:na]
@honghaic
Copy link
Author

honghaic commented Feb 9, 2015

2015-02-08 16:40:44,135 - [INFO] - from org.apache.zookeeper.ZooKeeper in kafka-manager-system-akka.actor.default-dispatcher-7
Client environment:user.dir=D:\app\KafkaManager.A\siphon

2015-02-08 16:40:44,137 - [INFO] - from org.apache.zookeeper.ZooKeeper in kafka-manager-system-akka.actor.default-dispatcher-7
Initiating client connection, connectString=10.65.196.35:2181,10.65.196.36:2181,10.65.196.37:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@4c0dfaf6

2015-02-08 16:40:44,142 - [DEBUG] - from org.apache.zookeeper.ClientCnxn in kafka-manager-system-akka.actor.default-dispatcher-7
zookeeper.disableAutoWatchReset is false

2015-02-08 16:40:44,159 - [INFO] - from org.apache.zookeeper.ClientCnxn in kafka-manager-system-akka.actor.default-dispatcher-7-SendThread(10.65.196.36:2181)
Opening socket connection to server 10.65.196.36/10.65.196.36:2181. Will not attempt to authenticate using SASL (unknown error)

2015-02-08 16:40:45,054 - [ERROR] - from kafka.manager.ApiError in pool-3-thread-2
error : Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) ~[akka-actor_2.11-2.3.7.jar:na]
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) ~[scala-library-2.11.4.jar:na]
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) ~[scala-library-2.11.4.jar:na]
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423) ~[akka-actor_2.11-2.3.7.jar:na]
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375) ~[akka-actor_2.11-2.3.7.jar:na]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_40-MS]

2015-02-08 16:40:45,395 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-6
Could not read WebJar RequireJS config for: dustjs-linkedin 2.4.0
Please file a bug at: http://github.com/webjars/dustjs-linkedin/issues/new

2015-02-08 16:40:45,431 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-6
Could not read WebJar RequireJS config for: json 20121008
Please file a bug at: http://github.com/webjars/json/issues/new

2015-02-08 16:40:45,454 - [WARN] - from org.webjars.RequireJS in play-akka.actor.default-dispatcher-6
Could not read WebJar RequireJS config for: requirejs 2.1.10
Please file a bug at: http://github.com/webjars/requirejs/issues/new

@rycoxiao
Copy link

rycoxiao commented Feb 9, 2015

you can use the comand as below:
kafka-manager -Dkafka-manager.zkhosts="server-01:2181".
the shell can't load the rigth config file.

@miguno
Copy link

miguno commented Feb 10, 2015

Unfortunately I confirm this problem I assume it's because of incorrect Play/sbt-native-packager settings in kafka-manager.

How to reproduce

A kafka-manager packaged and deployed via a zipfile (./sbt clean dist) is apparently not loading conf/application.conf after a deployment.

$ ./sbt clean dist    # >>> creates a zipfile
$ unzip kafka-manager-1.0-SNAPSHOT.zip
$ cd kafka-manager-1.0-SNAPSHOT 

# Now modify `conf/application.conf` to e.g. configure your ZK connection string via the
# config setting `kafka-manager.zkhosts`.  Say, set `kafka-manager.zkhosts="myzookeeper:2181".
$ vi conf/application.conf

Now, when you launch kafka-manager without any additional CLI options you will run into the following error:

$ bin/kafka-manager
Play server process ID is 57526
[info] play - Application started (Prod)
[info] play - Listening for HTTP on /0.0.0.0:9000
[INFO] [02/10/2015 10:13:53.250] [kafka-manager-system-akka.actor.default-dispatcher-2] [akka://kafka-manager-system/user/kafka-manager] Starting curator...
[error] o.a.c.f.i.CuratorFrameworkImpl - Background exception was not retry-able or retry gave up
java.net.UnknownHostException: kafka-manager-zookeeper: kafka-manager-zookeeper
        at java.net.InetAddress.getAllByName0(InetAddress.java:1348) ~[na:1.7.0]
        at java.net.InetAddress.getAllByName(InetAddress.java:1257) ~[na:1.7.0]
        at java.net.InetAddress.getAllByName(InetAddress.java:1182) ~[na:1.7.0]
        at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61) ~[org.apache.zookeeper.zookeeper-3.4.6.jar:3.4.6-1569965]
        at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445) ~[org.apache.zookeeper.zookeeper-3.4.6.jar:3.4.6-1569965]
[INFO] [02/10/2015 10:13:53.275] [kafka-manager-system-akka.actor.default-dispatcher-2] [akka://kafka-manager-system/user/kafka-manager] zk=kafka-manager-zookeeper:2181
[INFO] [02/10/2015 10:13:53.275] [kafka-manager-system-akka.actor.default-dispatcher-2] [akka://kafka-manager-system/user/kafka-manager] baseZkPath=/kafka-manager
[error] k.m.ApiError - error : Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager)]] after [1000 ms]
        at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333) ~[com.typesafe.akka.akka-actor_2.11-2.3.7.jar:na]
        at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117) ~[com.typesafe.akka.akka-actor_2.11-2.3.7.jar:na]
        at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599) ~[org.scala-lang.scala-library-2.11.4.jar:na]
        at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109) ~[org.scala-lang.scala-library-2.11.4.jar:na]
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597) ~[org.scala-lang.scala-library-2.11.4.jar:na]
[error] o.a.c.ConnectionState - Connection timed out for connection string (kafka-manager-zookeeper:2181) and timeout (15000) / elapsed (15053)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
        at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [org.apache.curator.curator-client-2.7.0.jar:na]
        at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87) [org.apache.curator.curator-client-2.7.0.jar:na]
        at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) [org.apache.curator.curator-client-2.7.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:492) [org.apache.curator.curator-framework-2.7.0.jar:na]
        at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:691) [org.apache.curator.curator-framework-2.7.0.jar:na]

Note the java.net.UnknownHostException: kafka-manager-zookeeper error: kafka-manager-zookeeper is the default host for kafka-manager.zkhosts. So kafka-manager is not reading the app configuration file, because if it had, it would have used myzookeeper as the ZK host.

Workaround

Explicitly provide the location of the application.conf file by setting the config.file System property on the command line when launching kafka-manager:

$ bin/kafka-manager -Dconfig.file=/path/to/kafka-manager/conf/application.conf

@mindscratch
Copy link

+1 for the work around.

@miguno miguno mentioned this issue Feb 19, 2015
Open
@spuder
Copy link

spuder commented Mar 2, 2015

I'm also running into this problem on both ubuntu 14.04 and mac os x. Running the latest commit as of right now.

a9fbba0

Specifying the config directory is the only way I can get this to work.

@spuder
Copy link

spuder commented Mar 9, 2015

Duplicate of #11

@lewison
Copy link

lewison commented Aug 17, 2015

i download the newest version of kafka-manager. and try to start it by the command above which said by miguno. but I still meet the error of load conf file error.

only I set the zookeeper in the command : -Dkafka-manager-zkhosts="127.0.0.1:2181".

I can start the kafka-manager in a right way now .

so I think this issue had no been fixed yet.

@zheolong
Copy link
Contributor

@lewison Same problem, do you know where the default zk urls was stored when sbt clean dist

@guangping0215
Copy link

i hit error "Bad application path: -Dconfig.file=../config/application.conf" when i use command line to start it. help~

@memelet
Copy link

memelet commented Jul 16, 2016

I have just started to see this today. Sometimes the page renders, sometimes I get this error.

@vijaygupta358
Copy link

For mine this command was working -
bin/kafka-manager -Dconfig.file=conf/application.conf -Dkafka-manager.zkhosts="localhost:2181"

make sure , u are using both parameter -Dconfig.file and -Dkafka-manager.zkhost .

If I was trying with individual parameter , it was giving same error .
Try with use of both parameter .

@dinar-dalvi
Copy link

dinar-dalvi commented Sep 9, 2017 •
edited
Loading

I am encountering the same issue Yikes! Ask timed out on [ActorSelection[Anchor(akka://kafka-manager-system/), Path(/user/kafka-manager/prod/kafka-state)]] after [2000 ms]

I am running the kafka-manager on a node where i can confirm that i can telnet into the ZK on 2181.

I am running with
/home/ubuntu/kafka-manager-1.3.3.13/bin/kafka-manager -Dconfig.file=/home/ubuntu/kafka-manager-1.3.3.13/conf/application.conf -Dhttp.port=9292 -Dkafka-manager.zkhosts=kafka1:2181,kafka2:2181,kafka3:2181,kafka4:2181 > /var/log/kafka-manager.log 2>/home/ubuntu/kafka-manager-1.3.3.13/error.log &

When i load the test cluster it's fine. Help needed please.

@sundhaug92 sundhaug92 mentioned this issue Nov 20, 2017
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests