Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing app in Marathon #48

Closed
jwerak opened this issue Nov 2, 2014 · 8 comments
Closed

Missing app in Marathon #48

jwerak opened this issue Nov 2, 2014 · 8 comments

Comments

@jwerak
Copy link

jwerak commented Nov 2, 2014

I have trouble adding app to bamboo from gui.

I've added app called /luigi-test and it says Missing app in Marathon.

I am running bamboo as docker container with command:

/usr/bin/docker run --rm --name bamboo -p 8000:8000 -p 80:80 -e MARATHON_ENDPOINT=10.0.1.149:8081 -e BAMBOO_ENDPOINT=10.0.1.151:8000 -e BAMBOO_ZK_HOST="10.0.1.149:2181,10.0.1.151:2181,10.0.1.150:2181" -e BAMBOO_ZK_PATH=/bamboo quay.io/appuri/bamboo -bind=":8000" -config="config/production.example.json"

docker container logs:
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56 Starting update loop
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052755 Using environment override MARATHON_ENDPOINT=10.0.1.149:8081
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052823 Using environment override BAMBOO_ENDPOINT=10.0.1.151:8000
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052891 Using environment override BAMBOO_ZK_HOST=10.0.1.149:2181,10.0.1.151:2181,10.0.1.150:2181
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052949 Using environment override BAMBOO_ZK_PATH=/bamboo

When app added:
From bamboo logs:
Nov 02 06:05:37 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/02 06:05:37.653968 Domain mapping: Stated changed
Nov 02 06:05:37 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/02 06:05:37.653983 Queuing an haproxy update.
Nov 02 06:05:37 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/02 06:05:37.655489 HAProxy: Same content, no need to reload

From marathon, only relevant info I've found:

Nov 02 06:06:29 ip-10-0-1-149.us-west-2.compute.internal bash[6143]: [2014-11-02 06:06:29,066] INFO 10.0.1.151 - - [02/Nov/2014:06:06:29 +0000] "GET /v2/tasks HTTP/1.1" 200 252 "-" "curl/7.26.0" (mesosphere.chaos.http.ChaosRequestLog:15)

I can add marathon callback and call to /v2/tasks returns following:
curl -X GET -H "Content-Type: application/json" 10.0.1.149:8081/v2/tasks
{"tasks":[{"appId":"/luigi-test","id":"luigi-test.70583548-5f32-11e4-8166-56847afe9799","host":"ip-10-0-1-149.us-west-2.compute.internal","ports":[31001],"startedAt":"2014-11-02T06:07:11.580Z","stagedAt":"2014-10-29T06:11:36.580Z","version":"2014-10-29T06:11:36.379Z","servicePorts":[9001]},{"appId":"/marathon-tcp-haproxy-dev","id":"marathon-tcp-haproxy-dev.9e92e270-606f-11e4-8166-56847afe9799","host":"ip-10-0-1-151.us-west-2.compute.internal","ports":[31187],"startedAt":"2014-11-02T06:07:11.602Z","stagedAt":"2014-10-30T20:02:04.602Z","version":"2014-10-29T09:20:19.048Z","servicePorts":[30000]},{"appId":"/demo-dev","id":"demo-dev.f56bbeb8-5ee8-11e4-8166-56847afe9799","host":"ip-10-0-1-149.us-west-2.compute.internal","ports":[31000],"startedAt":"2014-11-02T06:07:11.563Z","stagedAt":"2014-10-28T21:25:37.088Z","version":"2014-10-28T21:25:34.948Z","servicePorts":[9000]},{"appId":"/demo-dev","id":"demo-dev.f69d39da-5ee8-11e4-8166-56847afe9799","host":"ip-10-0-1-151.us-west-2.compute.internal","ports":[31000],"startedAt":"2014-11-02T06:07:11.572Z","stagedAt":"2014-10-28T21:25:39.089Z","version":"2014-10-28T21:25:34.948Z","servicePorts":[9000]}]}%

I am using mesos 0.20.1
marathon 0.7.3
latest bamboo

Do you have any advice on how I could debug this further?

@j1n6
Copy link
Contributor

j1n6 commented Nov 2, 2014

Hello,

Did you enable marathon http callback? And make sure the Bamboo endpoint can be reached by Marathon.

Sent from my iPhone

On 2 Nov 2014, at 06:49, veverjak notifications@github.com wrote:

I have trouble adding app to bamboo from gui.

I've added app called /luigi-test and it says Missing app in Marathon.

I am running bamboo as docker container with command:

/usr/bin/docker run --rm --name bamboo -p 8000:8000 -p 80:80 -e MARATHON_ENDPOINT=10.0.1.149:8081 -e BAMBOO_ENDPOINT=10.0.1.151:8000 -e BAMBOO_ZK_HOST="10.0.1.149:2181,10.0.1.151:2181,10.0.1.150:2181" -e BAMBOO_ZK_PATH=/bamboo quay.io/appuri/bamboo -bind=":8000" -config="config/production.example.json"

docker container logs:
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56 Starting update loop
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052755 Using environment override MARATHON_ENDPOINT=10.0.1.149:8081
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052823 Using environment override BAMBOO_ENDPOINT=10.0.1.151:8000
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052891 Using environment override BAMBOO_ZK_HOST=10.0.1.149:2181,10.0.1.151:2181,10.0.1.150:2181
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052949 Using environment override BAMBOO_ZK_PATH=/bamboo

When app added:
From bamboo logs:
Nov 02 06:05:37 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/02 06:05:37.653968 Domain mapping: Stated changed
Nov 02 06:05:37 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/02 06:05:37.653983 Queuing an haproxy update.
Nov 02 06:05:37 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/02 06:05:37.655489 HAProxy: Same content, no need to reload

From marathon, only relevant info I've found:

Nov 02 06:06:29 ip-10-0-1-149.us-west-2.compute.internal bash[6143]: [2014-11-02 06:06:29,066] INFO 10.0.1.151 - - [02/Nov/2014:06:06:29 +0000] "GET /v2/tasks HTTP/1.1" 200 252 "-" "curl/7.26.0" (mesosphere.chaos.http.ChaosRequestLog:15)

I can add marathon callback and call to /v2/tasks returns following:
curl -X GET -H "Content-Type: application/json" 10.0.1.149:8081/v2/tasks
{"tasks":[{"appId":"/luigi-test","id":"luigi-test.70583548-5f32-11e4-8166-56847afe9799","host":"ip-10-0-1-149.us-west-2.compute.internal","ports":[31001],"startedAt":"2014-11-02T06:07:11.580Z","stagedAt":"2014-10-29T06:11:36.580Z","version":"2014-10-29T06:11:36.379Z","servicePorts":[9001]},{"appId":"/marathon-tcp-haproxy-dev","id":"marathon-tcp-haproxy-dev.9e92e270-606f-11e4-8166-56847afe9799","host":"ip-10-0-1-151.us-west-2.compute.internal","ports":[31187],"startedAt":"2014-11-02T06:07:11.602Z","stagedAt":"2014-10-30T20:02:04.602Z","version":"2014-10-29T09:20:19.048Z","servicePorts":[30000]},{"appId":"/demo-dev","id":"demo-dev.f56bbeb8-5ee8-11e4-8166-56847afe9799","host":"ip-10-0-1-149.us-west-2.compute.internal","ports":[31000],"startedAt":"2014-11-02T06:07:11.563Z","stagedAt":"2014-10-28T21:25:37.088Z","version":"2014-10-28T21:25:34.948Z","servicePorts":[9000]},{"appId":"/demo-dev","id":"demo-dev.f69d39da-5ee8-11e4-8166-56847afe9799","host":"ip-10-0-1-151.us-west-2.compute.intern al","ports":[31000],"startedAt":"2014-11-02T06:07:11.572Z","stagedAt":"2014-10-28T21:25:39.089Z","version":"2014-10-28T21:25:34.948Z","servicePorts":[9000]}]}%

I am using mesos 0.20.1
marathon 0.7.3
latest bamboo

Do you have any advice on how I could debug this further?


Reply to this email directly or view it on GitHub.

@jwerak
Copy link
Author

jwerak commented Nov 2, 2014

I run marathon with following:
/opt/marathon/bin/start --master "zk://${ZOOKEEPER_ENDPOINTS}/mesos" --zk "zk://${ZOOKEEPER_ENDPOINTS}/marathon" --http_port ${HTTP_PORT-8081} --event_subscriber http_callback

I can create callback via curl but when I add bamboo app nothing changes in

curl -X GET -H "Content-Type: application/json" 10.0.1.149:8081/v2/eventSubscriptions

The should appear as non-missing when notified by marathon subscription, does not care about its initial call to /v2/tasks ?

@j1n6
Copy link
Contributor

j1n6 commented Nov 3, 2014

When you curl Marathon /v2/eventSubscriptions

e.g.

curl -X GET -H "Content-Type: application/json"  http://10.0.1.149:8081/v2/eventSubscriptions

Do you see Bamboo's callback URL is registered in the list? Can this private ip (10.0.1.149:8081 http://10.0.1.149:8081/v2/eventSubscriptions) be accessed inside docker? Also try add http:// in front of those IPs, in case some http library doesn't handle it.

Alternatively, I suggest try to build deb package and it's much easier to trace issue. There's a section in README about how to build package.

@jwerak
Copy link
Author

jwerak commented Nov 3, 2014

Hmm,

so the solution that I've found was to use http with env var MARATHON_ENDPOINT=http://10.0.1.149:8081

It almost seems that bamboo discovered the apps on its own.

Great work, thanks :)

I am using your dockerfile, is it supposed to behave this way?

btw I didn't find the way to build bamboo with debug using deb.
Is there any different build attribute for that in deb build that could be used during phase defined in your dockerfile?

@jwerak jwerak closed this as completed Nov 3, 2014
@j1n6
Copy link
Contributor

j1n6 commented Nov 3, 2014

Good to know you get it working!

The deb version builds a binary and deploy it without docker.

The way I usually suggest is deploying Bamboo without docker - there are some issues with docker that having both bamboo and haproxy inside, if bamboo stops, docker daemon quits and haproxy will be down too; same issue the other way around; but ensure haproxy is live is important. There's no upstart implemented for Docker images.

Docker is good way to try things out.

In the repo, you should find a script or build directory, it has a script we use to build deb package.

I have a issue to build it via travis and upload to release page, never find free time to do it.

On 3 Nov 2014, at 06:31, veverjak notifications@github.com wrote:

Hmm,

so the solution that I've found was to use http with env var MARATHON_ENDPOINT=http://10.0.1.149:8081

It almost seems that bamboo discovered the apps on its own.

Great work, thanks :)

I am using your dockerfile, is it supposed to behave this way?

btw I didn't find the way to build bamboo with debug using deb.
Is there any different build attribute for that in deb build that could be used during phase defined in your dockerfile?


Reply to this email directly or view it on GitHub.

@jwerak
Copy link
Author

jwerak commented Nov 3, 2014

I am running docker container as fleet unit (coreos) so its relaunched once docker container is down.

It might be suitable though to run bamboo under supervisord so that its respawned internally.
I can create pull request for you if you would care for this...?

@j1n6
Copy link
Contributor

j1n6 commented Nov 3, 2014

yah, so the docker entrypoint is executed for haproxy and supervisor is
managing bamboo. PR would be great!

On 3 November 2014 10:14, veverjak notifications@github.com wrote:

I am running docker container as fleet unit (coreos) so its relaunched
once docker container is down.

It might be suitable though to run bamboo under supervisord so that its
respawned internally.
I can create pull request for you if you would care for this...?


Reply to this email directly or view it on GitHub
#48 (comment).

This email may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone to

the wrong person. Thanks.

@jwerak
Copy link
Author

jwerak commented Nov 4, 2014

Created new issue for supervisord to keep it clean - #49

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants