This repository has been archived by the owner on Dec 8, 2019. It is now read-only.
Vamp Router does not start #22
Assignees
Labels
Comments
|
@zutherb Which version of Ubuntu are you using? Did you notice the issue reported in the documentation regarding vamp-router? |
|
I'm using: Yes notice the issue. |
|
I'm able to reproduce this issue, when I install an old version of haproxy. The haproxy version in the default Ubuntu repo is 1.4.x; vamp-router required haproxy 1.5+ Could you check which version of haproxy you got installed?
See the instructions for adding the haproxy repo in our docs: |
|
Hi @plamola, vagrant@master1:~$ haproxy -v
HA-Proxy version 1.5.14 2015/07/02
Copyright 2000-2015 Willy Tarreau <willy@haproxy.org> |
|
Can you check the privileges on the The I've done a clean install on an Ubuntu 14.04 machine and this is what is been setup: $ ls -al /usr/share/vamp-router
total 10168
drwxr-xr-x 6 root root 4096 Aug 10 09:54 .
drwxr-xr-x 105 root root 4096 Aug 10 09:54 ..
drwxrwxr-x 2 root root 4096 Aug 10 09:54 bin
drwxr-xr-x 4 vamp-router vamp-router 4096 Aug 10 09:54 configuration
drwxr-xr-x 2 vamp-router vamp-router 4096 Aug 10 11:02 data
drwxrwxr-x 2 root root 4096 Aug 10 09:54 examples
lrwxrwxrwx 1 root root 20 Aug 3 09:12 logs -> /var/log/vamp-router
-rwxr-xr-x 1 root root 10383392 Jul 31 14:14 vamp-router |
|
Looks good vagrant@master1:~$ ls -al /usr/share/vamp-router
total 10168
drwxr-xr-x 6 root root 4096 Aug 7 06:04 .
drwxr-xr-x 118 root root 4096 Aug 7 06:06 ..
drwxrwxr-x 2 root root 4096 Aug 7 06:04 bin
drwxr-xr-x 2 vamp-router vamp-router 4096 Aug 7 07:27 configuration
drwxr-xr-x 2 vamp-router vamp-router 4096 Aug 7 07:27 data
drwxrwxr-x 2 root root 4096 Aug 7 06:04 examples
lrwxrwxrwx 1 root root 20 Aug 3 02:12 logs -> /var/log/vamp-router
-rwxr-xr-x 1 root root 10383392 Jul 31 07:14 vamp-routerI will try a clean installation of ubuntu as well. |
|
Hi @plamola, the fresh installation doesn't work. Here is what i did: #!/usr/bin/env bash
sudo apt-get install -y openjdk-8
sudo add-apt-repository -y ppa:vbernat/haproxy-1.5
sudo apt-get update
sudo apt-get install -y haproxy
echo "deb https://dl.bintray.com/magnetic-io/upstart trusty main" | sudo tee -a /etc/apt/sources.list
curl https://bintray.com/user/downloadSubjectPublicKey?username=bintray | sudo apt-key add -
sudo apt-get update
sudo apt-get install -y vamp-core
cp -vf /vagrant/scripts/vamp/core-application.conf /usr/share/vamp-core/conf/application.conf
sudo service vamp-core restart
sudo apt-get install -y vamp-pulse
cp -vf /vagrant/scripts/vamp/pulse-application.conf /usr/share/vamp-pulse/conf/application.conf
sudo service vamp-pulse restart
sudo apt-get install -y vamp-router
/usr/share/vamp-router/vamp-router --binary=/usr/sbin/haproxy
sudo apt-get install -y vamp-cli |
|
@zutherb Hi Bernd, we're checking this right now in a fresh Ubuntu 14.04 instance. We noticed a couple of things:
a => root does a For now, all is fine...a simple curl confirms its running: Things start going wrong once we start/stop the processes directly using root or sudo. This will work once...but then all pid files, logs, haproxy socket files etc. are written under root. From that moment on, everything should be started under root, otherwise vamp-router cannot communicate with haproxy. |
|
Hi Tim,
vagrant@master1:~$ java -version
java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)
The curl doesn't work as well. vagrant@master1:~$ curl localhost:10001/v1/info
curl: (7) Failed to connect to localhost port 10001: Connection refusedThe vamp-core logfile (/var/log/vamp-core/application.log) show that it can connect to the proxy. 09:55:30.467 | ERROR | io.vamp.common.notification.Notification | | Router response error.
09:55:30.467 | ERROR | io.vamp.common.notification.Notification | | java.net.ConnectException: Connection refused: localhost/127.0.0.1:10001 to http://localhost:10001/v1/routes
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:10001 to http://localhost:10001/v1/routes
at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:342) ~[vamp-core.jar:0.7.9.1]
at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:107) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:431) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:422) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:384) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:109) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[vamp-core.jar:0.7.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]
Caused by: java.net.ConnectException: Connection refused: localhost/127.0.0.1:10001 to http://localhost:10001/v1/routes
at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:103) ~[vamp-core.jar:0.7.9.1]
... 12 common frames omitted
Caused by: java.net.ConnectException: Connection refused: localhost/127.0.0.1:10001
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_51]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_51]
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) ~[vamp-core.jar:0.7.9.1]
... 8 common frames omitted
09:55:30.470 | ERROR | akka.actor.OneForOneStrategy | e/user/deployment-synchronization-actor$ | io.vamp.common.notification.NotificationErrorException cannot be cast to io.vamp.core.router_driver.DeploymentRoutes
java.lang.ClassCastException: io.vamp.common.notification.NotificationErrorException cannot be cast to io.vamp.core.router_driver.DeploymentRoutes
at io.vamp.core.operation.deployment.DeploymentSynchronizationActor.io$vamp$core$operation$deployment$DeploymentSynchronizationActor$$synchronize(DeploymentSynchronizationActor.scala:84) ~[vamp-core.jar:0.7.9.1]
at io.vamp.core.operation.deployment.DeploymentSynchronizationActor$$anonfun$receive$1.applyOrElse(DeploymentSynchronizationActor.scala:72) ~[vamp-core.jar:0.7.9.1]
at akka.actor.Actor$class.aroundReceive(Actor.scala:467) ~[vamp-core.jar:0.7.9.1]
at io.vamp.core.operation.deployment.DeploymentSynchronizationActor.aroundReceive(DeploymentSynchronizationActor.scala:43) ~[vamp-core.jar:0.7.9.1]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [vamp-core.jar:0.7.9.1]
at akka.actor.ActorCell.invoke(ActorCell.scala:487) [vamp-core.jar:0.7.9.1]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [vamp-core.jar:0.7.9.1]
at akka.dispatch.Mailbox.run(Mailbox.scala:220) [vamp-core.jar:0.7.9.1]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [vamp-core.jar:0.7.9.1]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [vamp-core.jar:0.7.9.1]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [vamp-core.jar:0.7.9.1]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [vamp-core.jar:0.7.9.1]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [vamp-core.jar:0.7.9.1]
09:55:30.562 | ERROR | io.vamp.common.notification.Notification | | Router response error.
09:55:30.563 | ERROR | io.vamp.common.notification.Notification | | java.net.ConnectException: Connection refused: localhost/127.0.0.1:10001 to http://localhost:10001/v1/info
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:10001 to http://localhost:10001/v1/info
at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:342) ~[vamp-core.jar:0.7.9.1]
at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:107) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:431) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.DefaultChannelFuture.addListener(DefaultChannelFuture.java:145) ~[vamp-core.jar:0.7.9.1]
at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.doConnect(NettyAsyncHttpProvider.java:1138) ~[vamp-core.jar:0.7.9.1]
at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:935) ~[vamp-core.jar:0.7.9.1]
at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:499) ~[vamp-core.jar:0.7.9.1]
at dispatch.HttpExecutor$class.apply(execution.scala:47) ~[vamp-core.jar:0.7.9.1]
at dispatch.Http.apply(execution.scala:12) ~[vamp-core.jar:0.7.9.1]
at dispatch.HttpExecutor$class.apply(execution.scala:42) ~[vamp-core.jar:0.7.9.1]
at dispatch.Http.apply(execution.scala:12) ~[vamp-core.jar:0.7.9.1]
at io.vamp.common.http.RestClient$.http(RestClient.scala:69) ~[vamp-core.jar:0.7.9.1]
at io.vamp.common.http.RestClient$.get(RestClient.scala:27) ~[vamp-core.jar:0.7.9.1]
at io.vamp.core.router_driver.DefaultRouterDriver.info(DefaultRouterDriver.scala:20) ~[vamp-core.jar:0.7.9.1]
at io.vamp.core.router_driver.RouterDriverActor.reply(RouterDriverActor.scala:45) [vamp-core.jar:0.7.9.1]
at io.vamp.common.akka.ReplyActor$$anonfun$receive$1.applyOrElse(ReplyActor.scala:19) [vamp-core.jar:0.7.9.1]
at akka.actor.Actor$class.aroundReceive(Actor.scala:467) [vamp-core.jar:0.7.9.1]
at io.vamp.core.router_driver.RouterDriverActor.aroundReceive(RouterDriverActor.scala:33) [vamp-core.jar:0.7.9.1]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [vamp-core.jar:0.7.9.1]
at akka.actor.ActorCell.invoke(ActorCell.scala:487) [vamp-core.jar:0.7.9.1]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [vamp-core.jar:0.7.9.1]
at akka.dispatch.Mailbox.run(Mailbox.scala:220) [vamp-core.jar:0.7.9.1]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [vamp-core.jar:0.7.9.1]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [vamp-core.jar:0.7.9.1]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [vamp-core.jar:0.7.9.1]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [vamp-core.jar:0.7.9.1]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [vamp-core.jar:0.7.9.1]
Caused by: java.net.ConnectException: Connection refused: localhost/127.0.0.1:10001 to http://localhost:10001/v1/info
at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:103) ~[vamp-core.jar:0.7.9.1]
... 25 common frames omitted
Caused by: java.net.ConnectException: Connection refused: localhost/127.0.0.1:10001
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_51]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_51]
at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[vamp-core.jar:0.7.9.1]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[vamp-core.jar:0.7.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_51]The vamp-router logfile (/var/log/vamp-router/vamp-router.log) show that vamp-router starts again and again. |
|
The problem of the constant restarting is due to the service scripts that continuously tries to start Vamp Router after it fails. I think Vamp Router fails because either Router or Haproxy cannot write the necessary files to disk. What we can do to pinpoint the problem is the following:
rm /usr/share/vamp-router/configuration/haproxy_new.cfg
rm /usr/share/vamp-router/data/haproxy.stats.sock
rm /usr/share/vamp-router/data/haproxy-private.pidand try again with the manual start. |
|
thx @tnolet that works, but when i'm using |
|
@zutherb this is due to the fact that now some files are owned by root and the 'service' command tries to use the |
|
Hi @zutherb I heard this was still an issue.
|
|
Hi @tnolet, I try to upload the vm to google drive. I will send you an invention when the upload is done. I did the following steps to active the logging: The Log is then avaible under /var/log/haproxy.log I got the following output: While trying to connect the mongodb: When i looked to the haproxy log i notice that haproxy try to connected multiple mongodb instances. So i checked the routes with the rest api http://172.31.1.11:10001/v1/routes As you can see there are multiple routes for the port 30006:
It seems the the router does not delete the old configuration of an old deployment When looking to the deployments call http://172.31.1.11:8081/api/v1/deployments only b017c339-9406-48aa-ac90-4307e7f6343a should deployed. Is there a way to reset the router? |
|
I unlock you for the vm. My Assumption seems to be correct. After i deleted the following to files in /usr/share/vamp-router/configuration the haproxy works as excepted even that the external port 9050 is not started:
|
|
|
@tnolet thx, username + password is vagrant + vagrant |
|
@zutherb I can see the machine booting, but I guess its missing some Puppet files as it throws errors about missing those files. When trying to log in it seems the user is either incorrect or not setup: |
|
@tnolet hmm ... normally puppet is only need for provisioning the vm. |
|
@zutherb That works! Great! |
|
There are a couple of issues with the way this particular setup. Luckily, they are all pretty easy to fix:
exec sudo -u vamp-router bin/vamp-routerThis means all .sock files and .pid files that Haproxy writes are not accessible to Vamp Router and therefore it cannot start. I fixed this in your VMBox image by first killing haproxy and then chowning all files to vamp-router: I also removed the config files as you did mentioned earlier. We now have a clean start. with the result of haproxy AND vamp-router running as the same user. You can of course ignore this and just start Vamp Router as sudo /usr/share/vamp-router/vamp-router > /dev/null 2>&1 & |
|
@tnolet thx a lot it works. |
|
added to the FAQ. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I try out vamp 0.7.9 and installed vamp like you have defined it in the guide (http://vamp.io/installation/ubuntu/). But the Vamp Router does not start, i got an endless loop in the log file (/var/log/vamp-router/vamp-router.log ) of the following message.
I could not find a log file of the haproxy to see what is wrong.
██╗ ██╗ █████╗ ███╗ ███╗██████╗ ██║ ██║██╔══██╗████╗ ████║██╔══██╗ ██║ ██║███████║██╔████╔██║██████╔╝ ╚██╗ ██╔╝██╔══██║██║╚██╔╝██║██╔═══╝ ╚████╔╝ ██║ ██║██║ ╚═╝ ██║██║ ╚═══╝ ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ router version 0.7.9 by magnetic.io 08:13:53.615 main NOTI ==> Attempting to load config at /usr/share/vamp-router/configuration 08:13:53.615 main NOTI ==> Did not find a config...initializing empty config 08:13:53.616 main CRIT ==> Could not render initial config, exiting... 08:13:53.649 main INFO ==> ██╗ ██╗ █████╗ ███╗ ███╗██████╗ ██║ ██║██╔══██╗████╗ ████║██╔══██╗ ██║ ██║███████║██╔████╔██║██████╔╝ ╚██╗ ██╔╝██╔══██║██║╚██╔╝██║██╔═══╝ ╚████╔╝ ██║ ██║██║ ╚═╝ ██║██║ ╚═══╝ ╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ router version 0.7.9 by magnetic.io 08:13:53.650 main NOTI ==> Attempting to load config at /usr/share/vamp-router/configuration 08:13:53.650 main NOTI ==> Did not find a config...initializing empty config 08:13:53.651 main CRIT ==> Could not render initial config, exiting... 08:13:53.670 main INFO ==>The text was updated successfully, but these errors were encountered: