Traefik's integration with Marathon is broken after upgrading to Marathon 1.4.2

[erickrmz]

What version of Traefik are you using (traefik version)?

v1.2.1

What is your environment & configuration (arguments, toml...)?

################################################################
# Global configuration
################################################################

graceTimeOut = 10
traefikLogsFile = "/var/log/traefik/traefik.log"
accessLogsFile = "/var/log/traefik/traefik-access.log"
logLevel = "DEBUG"
ProvidersThrottleDuration = 1
MaxIdleConnsPerHost = 200
InsecureSkipVerify = false
defaultEntryPoints = ["http"]

[web]
address = ":8080"

[marathon]

endpoint = "http://10.20.0.201:8080,10.20.10.201:8080,10.20.20.201:8080"
watch = true
domain = "traefik.service.consul"
exposedByDefault = false
groupsAsSubDomains = false

[marathon.basic]
httpBasicAuthUser = "XXXXX"
httpBasicPassword = "XXXXX"

[retry]

What did you do?

We had Traefik 1.2.1 using Marathon 1.39 as a backend. It's been working fine for the last weeks. We recently decided to upgrade Marathon to version 1.4.2. Once we did that, I checked the Traefik dashboard and we lost all the backends.

What did you expect to see?

Configuration received from provider marathon should be a big hash containing all of our apps. We want to continue to see the backends in the dashboard of course.

What did you see instead?

Dashboard looks totally empty. The logs are saying that it's not receiving anything from Marathon:
Configuration received from provider marathon: {}

If applicable, please paste the log output in debug mode (--debug switch)

time="2017-04-12T15:39:10-07:00" level=debug msg="Configuration received from provider marathon: {}" 
time="2017-04-12T15:39:10-07:00" level=info msg="Skipping same configuration for provider marathon" 
time="2017-04-12T15:39:12-07:00" level=debug msg="Marathon event receveivedtype: status_update_event, event: &{status_update_event 2017-04-12T22:39:12.153Z 9e473fe9-01c5-409a-91f6-ff65a208014b-S7 sami_qamr1_apps_abacus-errors-qamr1.d8e7fbb7-1fd0-11e7-a77e-4a8e00763272 TASK_RUNNING  /sami/qamr1/apps/abacus-errors-qamr1 appmr01-nops-us-east-1a.ec2.ssic.internal [%!s(int=31252)] [%!s(*marathon.IPAddress=&{10.20.104.100 IPv4})] 2017-04-12T05:00:33.977Z}" 
time="2017-04-12T15:39:12-07:00" level=debug msg="Configuration received from provider marathon: {}" 
time="2017-04-12T15:39:12-07:00" level=info msg="Skipping same configuration for provider marathon" 
time="2017-04-12T15:39:17-07:00" level=debug msg="Marathon event receveivedtype: status_update_event, event: &{status_update_event 2017-04-12T22:39:17.436Z 9e473fe9-01c5-409a-91f6-ff65a208014b-S7 sami_qamr1_apps_abacus-errors-qamr1.d8e7fbb7-1fd0-11e7-a77e-4a8e00763272 TASK_FAILED Container exited with status 1 /sami/qamr1/apps/abacus-errors-qamr1 appmr01-nops-us-east-1a.ec2.ssic.internal [%!s(int=31252)] [%!s(*marathon.IPAddress=&{10.20.104.100 IPv4})] 2017-04-12T05:00:33.977Z}"

[erickrmz]

time="2017-04-12T16:08:09-07:00" level=info msg="Traefik version v1.2.1 built on 2017-03-27_03:33:41PM" 
time="2017-04-12T16:08:09-07:00" level=info msg="Using TOML configuration file /opt/traefik/conf/traefik.toml" 
time="2017-04-12T16:08:09-07:00" level=debug msg="Global configuration loaded {\"GraceTimeOut\":10,\"Debug\":false,\"CheckNewVersion\":true,\"AccessLogsFile\":\"/var/log/traefik/t
raefik-access.log\",\"TraefikLogsFile\":\"/var/log/traefik/traefik.log\",\"LogLevel\":\"DEBUG\",\"EntryPoints\":{\"http\":{\"Network\":\"\",\"Address\":\":80\",\"TLS\":null,\"Redi
rect\":null,\"Auth\":null,\"Compress\":false}},\"Cluster\":null,\"Constraints\":[],\"ACME\":null,\"DefaultEntryPoints\":[\"http\"],\"ProvidersThrottleDuration\":1,\"MaxIdleConnsPe
rHost\":200,\"InsecureSkipVerify\":false,\"Retry\":{\"Attempts\":0},\"Docker\":null,\"File\":null,\"Web\":{\"Address\":\":8080\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":fa
lse,\"Statistics\":null,\"Metrics\":null,\"Auth\":null},\"Marathon\":{\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Endpoint\":\"http://10.20.0.201:8080,10.20.10.201:8080,
10.20.20.201:8080\",\"Domain\":\"traefik.service.consul\",\"ExposedByDefault\":false,\"GroupsAsSubDomains\":false,\"DCOSToken\":\"\",\"MarathonLBCompatibility\":false,\"TLS\":null
,\"DialerTimeout\":60,\"KeepAlive\":10,\"Basic\":{\"HTTPBasicAuthUser\":\"XXXX\",\"HTTPBasicPassword\":\"XXXXX\"}},\"Consul\":null,\"ConsulCatalog\":null,\"Etcd\":null,\"Zookeep
er\":null,\"Boltdb\":null,\"Kubernetes\":null,\"Mesos\":null,\"Eureka\":null,\"ECS\":null,\"Rancher\":null}" 
time="2017-04-12T16:08:09-07:00" level=info msg="Preparing server http &{Network: Address::80 TLS:<nil> Redirect:<nil> Auth:<nil> Compress:false}" 
time="2017-04-12T16:08:09-07:00" level=info msg="Starting provider *provider.Marathon {\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Endpoint\":\"http://10.20.0.201:8080,1
0.20.10.201:8080,10.20.20.201:8080\",\"Domain\":\"traefik.service.consul\",\"ExposedByDefault\":false,\"GroupsAsSubDomains\":false,\"DCOSToken\":\"\",\"MarathonLBCompatibility\":f
alse,\"TLS\":null,\"DialerTimeout\":60,\"KeepAlive\":10,\"Basic\":{\"HTTPBasicAuthUser\":\"XXX\",\"HTTPBasicPassword\":\"XXXXX\"}}" 
time="2017-04-12T16:08:09-07:00" level=info msg="Starting provider *main.WebProvider {\"Address\":\":8080\",\"CertFile\":\"\",\"KeyFile\":\"\",\"ReadOnly\":false,\"Statistics\":nu
ll,\"Metrics\":null,\"Auth\":null}" 
time="2017-04-12T16:08:09-07:00" level=info msg="Starting server on :80" 
time="2017-04-12T16:08:09-07:00" level=warning msg="clientTLS is nil" 
time="2017-04-12T16:08:09-07:00" level=debug msg="Configuration received from provider marathon: {}" 
time="2017-04-12T16:08:09-07:00" level=debug msg="Last marathon config received more than 1ns, OK" 
time="2017-04-12T16:08:09-07:00" level=info msg="Server configuration reloaded on :80" 
time="2017-04-12T16:08:09-07:00" level=warning msg="A new release has been found: 1.2.2. Please consider updating." 
time="2017-04-12T16:13:11-07:00" level=debug msg="Marathon event receveivedtype: status_update_event, event: &{status_update_event 2017-04-12T23:13:10.762Z 9e473fe9-01c5-409a-91f6
-ff65a208014b-S12 sami_qa_apps_helios-qa.3efc7d83-1f3d-11e7-8108-9e4ddc98cb0e TASK_KILLING  /sami/qa/apps/helios-qa appqa03-nops-us-east-1d.ec2.ssic.internal [%!s(int=31003) %!s(i
nt=31004)] [%!s(*marathon.IPAddress=&{172.17.0.7 IPv4})] 2017-04-12T05:02:21.312Z}" 
time="2017-04-12T16:13:11-07:00" level=debug msg="Configuration received from provider marathon: {}" 
time="2017-04-12T16:13:11-07:00" level=info msg="Skipping same configuration for provider marathon" 

[erickrmz]

As it turns out, this is not a Traefik issue, but a Marathon plugin issue. See below:

We were working with Marathon 1.3.9 using the marathon-ldap plugin (latest), and using Traefik-Marathon integration as well. Everything was working fine until yesterday when we decided to upgrade to Marathon 1.4.2 to get the latest goodies. We thought the Marathon upgrade went fine, but our Traefik integration broke (we could no longer see any backends).

In Traefik logs we could see:

time="2017-04-12T15:39:10-07:00" level=debug msg="Configuration received from provider marathon: {}"

After inspecting different Marathon endpoints, we noticed that we would get the expected hash from calling /v2/apps, however when we tried to access /v2/tasks we noticed the result was an empty hash. We tried different things to work around this, when we tried disabling the plugins (we were using marathon-ldap), we started to get the expected results from /v2/tasks, and Traefik started working again

With plugins disabled

root 29061 27832 1 Apr12 ? 00:09:31 java -Djava.library.path=/usr/local/lib:/usr/lib:/usr/lib64 -Djava.util.logging.SimpleFormatter.format=%2$s%5$s%6$s%n -server -Djava.awt.headless=true -Xms2048M -Xmx2048M -Xss256M -Xmn512M -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Dlogback.configurationFile=/etc/marathon/logback.xml -cp /tmp/marathon-5f0afe9cf782f22f11c81c848d23ca70d59fcbda.jar mesosphere.marathon.Main --enable_features task_killing --ssl_keystore_path /etc/marathon/hb.org.jks --mesos_authentication --mesos_authentication_principal mt --mesos_authentication_secret_file /etc/maratho/marathon_mesos_principal_secret_file --framework_name marathon --event_subscriber http_callback --mesos_role marathon --zk_max_versions 50 --max_tasks_per_offer 5 --http_max_concurrent_requests 1000 --logging_level info --master zk://10.20.0.201:2181,10.20.10.201:2181,10.20.20.201:2181/mesos --ssl_keystore_password xxxxx

$ curl localhost:8080/v2/tasks {"tasks":[{"ipAddresses":[{"ipAddress":"172.17.0.12","protocol":"IPv4"}],"stagedAt":"2017-04-13T05:02:45.387Z","state":"TASK_RUNNING",.... .... ...
With plugins enabled

root 29061 27832 1 Apr12 ? 00:09:31 java -Djava.library.path=/usr/local/lib:/usr/lib:/usr/lib64 -Djava.util.logging.SimpleFormatter.format=%2$s%5$s%6$s%n -server -Djava.awt.headless=true -Xms2048M -Xmx2048M -Xss256M -Xmn512M -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Dlogback.configurationFile=/etc/marathon/logback.xml -cp /tmp/marathon-5f0afe9cf782f22f11c81c848d23ca70d59fcbda.jar mesosphere.marathon.Main --enable_features task_killing --ssl_keystore_path /etc/marathon/hb.org.jks --mesos_authentication --mesos_authentication_principal mt --mesos_authentication_secret_file /etc/maratho/marathon_mesos_principal_secret_file --plugin_dir /etc/marathon/plugins --plugin_conf /etc/marathon/plugins/plugin_conf.json --framework_name marathon --event_subscriber http_callback --mesos_role marathon --zk_max_versions 50 --max_tasks_per_offer 5 --http_max_concurrent_requests 1000 --logging_level info --master zk://10.20.0.201:2181,10.20.10.201:2181,10.20.20.201:2181/mesos --ssl_keystore_password xxxxx

$ curl http://XXXX:XXXX@localhost:8080/v2/tasks {"tasks":[]}

Plugin configuration

# cat /etc/marathon/plugins/plugin_conf.json { "plugins": { "authorization": { "plugin": "mesosphere.marathon.plugin.auth.Authorizer", "implementation": "io.containx.marathon.plugin.auth.LDAPAuthorizor" }, "authentication": { "plugin": "mesosphere.marathon.plugin.auth.Authenticator", "implementation": "io.containx.marathon.plugin.auth.LDAPAuthenticator", "configuration": { "users": [ { "username": "XXXX", "password": "XXXX", "groups": [ "Admin" ] }, { "username": "guest", "password": "XXXX", "groups": [ "Developers" ] } ], "authorization": { "access": [ { "group": "Admin", "permissions": [ { "allowed": "*", "type" : "app" }, { "allowed": "*", "type" : "group" } ] }, { "group": "Developers", "permissions": [ { "allowed": "view", "type": "app", "path": "/" }, { "allowed": "view", "type": "group", "path": "/" } ] } ] } } } } }

[erickrmz]

So, disabling the marathon-ldap plugin was the workaround. We will have to explore a different auth mechanism or wait for marathon-ldap devs to make appropriate changes.

Thanks a lot for your help ttr

[timoreimann]

@erickrmz glad to hear you figured it out!