Marathon-LB Keeps Dying

[akamalov]

Environment:

OS: RHEL7.2
Docker: 1.10
Mesos: 0.26.0
Marathon: 0.14

Deployed mesosphere/marathon-lb:latest, runs it fine for about 2 hours and then dies. Looking through logs, can't spot anything abnormal:

marathon_lb: generating config
marathon_lb: reading running config from /marathon-lb/haproxy.cfg
marathon_lb: updating tasks finished, took 19.396591901779175 seconds
marathon_lb: received event of type event_stream_attached
marathon_lb: received event of type event_stream_detached
marathon_lb: setting default value for HAPROXY_HEAD
marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_HEAD
marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_APPID_HEAD
marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_HEAD
marathon_lb: setting default value for HAPROXY_FRONTEND_HEAD
marathon_lb: setting default value for HAPROXY_BACKEND_REDIRECT_HTTP_TO_HTTPS
marathon_lb: setting default value for HAPROXY_BACKEND_HEAD
marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ACL
marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_APPID_ACL
marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_ACL
marathon_lb: setting default value for HAPROXY_BACKEND_HTTP_OPTIONS
marathon_lb: setting default value for HAPROXY_BACKEND_HTTP_HEALTHCHECK_OPTIONS
marathon_lb: setting default value for HAPROXY_BACKEND_TCP_HEALTHCHECK_OPTIONS
marathon_lb: setting default value for HAPROXY_BACKEND_STICKY_OPTIONS
marathon_lb: setting default value for HAPROXY_BACKEND_SERVER_OPTIONS
marathon_lb: setting default value for HAPROXY_BACKEND_SERVER_HTTP_HEALTHCHECK_OPTIONS
marathon_lb: setting default value for HAPROXY_BACKEND_SERVER_TCP_HEALTHCHECK_OPTIONS
marathon_lb: setting default value for HAPROXY_FRONTEND_BACKEND_GLUE
marathon_lb: SSE Active, trying fetch events from from http://master.mesos:8080/v2/events
marathon_lb: fetching apps
marathon_lb: GET http://master.mesos:8080/v2/apps?embed=apps.tasks
marathon_lb: got apps ['/nginx3', '/mesos-dns', '/marathon-lb-internal', '/product/database/mongo004', '/chronos-marathon', '/marathon-lb-internal-autoscale']
marathon_lb: generating config
marathon_lb: reading running config from /marathon-lb/haproxy.cfg
marathon_lb: updating tasks finished, took 23.323400735855103 seconds
marathon_lb: received event of type event_stream_attached
marathon_lb: received event of type event_stream_attached
marathon_lb: received event of type event_stream_detached
marathon_lb: setting default value for HAPROXY_HEAD
marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_HEAD
marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_APPID_HEAD
marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_HEAD
marathon_lb: setting default value for HAPROXY_FRONTEND_HEAD
marathon_lb: setting default value for HAPROXY_BACKEND_REDIRECT_HTTP_TO_HTTPS
marathon_lb: setting default value for HAPROXY_BACKEND_HEAD
marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ACL
marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_APPID_ACL
marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_ACL
marathon_lb: setting default value for HAPROXY_BACKEND_HTTP_OPTIONS
marathon_lb: setting default value for HAPROXY_BACKEND_HTTP_HEALTHCHECK_OPTIONS
marathon_lb: setting default value for HAPROXY_BACKEND_TCP_HEALTHCHECK_OPTIONS
marathon_lb: setting default value for HAPROXY_BACKEND_STICKY_OPTIONS

[root@node261 mesos-slave]# docker ps -a
CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS                        PORTS               NAMES
d134889fc94e        mesosphere/marathon-lb                   "/marathon-lb/run sse"   2 hours ago         Exited (137) 12 minutes ago                       mesos-add95ef3-c87b-4c84-98c2-2497129cf7f8-S3.9f47ae5c-47d3-4e2c-94df-2ccbdad1b1f9
5e59cf9be0eb        mesosphere/marathon-lb                   "/marathon-lb/run sse"   3 hours ago         Exited (137) 2 hours ago                          mesos-add95ef3-c87b-4c84-98c2-2497129cf7f8-S3.99bc34cc-e92a-4209-996a-b80f13ba1ee9

e4466bce5be4        mesosphere/marathon-lb-autoscale         "/marathon-lb-autosca"   30 hours ago        Up 30 hours                                       mesos-add95ef3-c87b-4c84-98c2-2497129cf7f8-S3.a686d337-e286-4dac-b722-d71706bf0315
[root@node261 mesos-slave]# 

Anyone has encountered similar issues ?

Thanks!!

Alex

[brndnmtthws]

Looks like an issue with the container itself perhaps. Is there anything in the Mesos agent logs around the time it fails?

[akamalov]

re-deployed and problem seem to go away...