CPU usage increase on Marathon leader when Traefik is configured with Marathon provider

[renaudhager]

Hi guys,

I searched on stackoverflow but I did not find anything related to my issue, that is why I'm opening that issue.

Do you want to request a feature or report a bug?

Bug

What did you do?

We have a Mesos & Marathon (1.5.0) with 3 masters and nodes ~20 agents.

On each agent we run Traefik in a container configured with marathon as a backend, here is the configuration file:

logLevel = "INFO"

[entryPoints]
    [entryPoints.http]
    address = ":80"

[traefikLog]
  format = "json"

[accessLog]
  format = "json"

# API definition
[api]

  entryPoint = "traefik"
  dashboard = true
  debug = false

  [api.statistics]
    recentErrors = 10

# Metrics definition
[metrics]

  [metrics.datadog]

    address = "10.X.X.X:8125"

    pushInterval = "10s"

# enable /ping on the API port
[ping]
entryPoint = "traefik"

################################################################
# Mesos/Marathon Provider
################################################################

# Enable Marathon Provider.
[marathon]

endpoint = "http://mesos.lan:8080/"

watch = true

domain = "service.lan"

exposedByDefault = false

It works well, all Marathon tasks are correctly configured in Traefik and traffic is correctly served.

What did you expect to see?

No or a little impact on Marathon leader cpu usage.

What did you see instead?

Since we run Traefik (15/08) we observed a significant increase of the CPU usage on Marathon leader (see below graphs):

We stopped all Traefik containers on 21/08, CPU usage and load decrease immediately.

I quickly investigated how Traefik fetch Marathon informations (I did a tcpdump).
I saw the following request made repeatedly:

GET //v2/apps?embed=apps.tasks&embed=apps.deployments&embed=apps.readiness HTTP/1.1
Host: mesos.lan:8080
User-Agent: Go-http-client/1.1
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

This requests returns informations about all tasks running and seems to be quite resource consuming for the Marathon leader.

First I'm wondering if my configuration is ok or if I missed something.

Then, Marathon exposes an event bus endpoint (https://mesosphere.github.io/marathon/docs/event-bus.html),
I quickly checked Traefik dependencies, it seems that it uses the following lib to interact with Marathon https://github.com/gambol99/go-marathon, which seems to be able to handle that stream.

I would like to know if there is a way/possible to configure Traefik to subscribe to that event bus instead of querying /v2/apps endpoint.

Output of traefik version: (What version of Traefik are you using?)

docker run traefik:v1.6 version
Version:      v1.6.6
Codename:     tetedemoine
Go version:   go1.10.3
Built:        2018-08-20_01:10:06PM
OS/Arch:      linux/amd64

[timoreimann]

@renaudhager thanks for your report.

Traefik does leverage the Marathon event bus, but only as a trigger to be notified about changes in the cluster. For each trigger, we fetch the entire set of applications (including tasks).

The advantage is that this approach is very robust and allows us to operate statelessly. For instance, it doesn't matter too much if Traefik misses an event (due to, say, a short network interruption) because the next event will reconcile everything as we fetch the entire state from Marathon. The downside is that the given approach has limits with regards to scalability; especially when talking about large clusters with hundreds or more applications, or clusters for which state changes happen rather rapidly.

In order to better understand the problem, could I ask you to answer the following questions:

1. How big is your Marathon/Mesos cluster (in terms of number of applications and tasks)?
2. Roughly speaking, how frequently does your cluster change its state (where a state change can be anything affecting the availability of your services, such as healthiness changes, readiness changes, scaling up/down, tasks failing, tasks coming back up, etc.)?
3. what is the spec of your machines running Marathon (number of cores/CPUs and memory)?
4. Do Marathon/Mesos/Zookeeper run on dedicated machines each, or do they share machines?
5. what kind of CPU usage is shown in your graph? Specifically, is it not iowait?

Depending on where time and resources are spent, there are multiple trajectories we could consider to remedy the situation (ordered by least effort descending):

• check if we can improve the set of events we use as triggers. My rough recollection is that we never made sure they make up the minimum set of events needed for us to catch all relevant state changes.
• check if we can find a better selection of embed filters for the /v2/apps HTTP GET call. If I had to guess, I'd say that apps.tasks is eating up most cycles; maybe switching to apps.taskStats could help to reduce the volume we retrieve? (I have never tried that filter myself.)
• move our configuration update throttling logic closer to the providers. Right now, we allow providers to generate configuration updates very frequently and only throttle when they run through our server machinery. It might make more sense if we already throttled the periodicity by which we fetch updates from the providers, in turn easing the load on systems like Marathon and Kubernetes.
• retrieve the information on what has changed from the bus events instead of a generic fetch following the event triggers. This one comes with significant implications to reliability and correctness as we cannot guarantee to see all events if there's some kind of outage. Looking at how Kubernetes does it, they offer a watch API interface that can replay earlier events given the last ID seen, and the client libraries do this under the hood to catch up when missing events. The Marathon client go-marathon (and maybe even the streaming library it uses) may need to go through significant architectural changes to facilitate such a mode, possibly even requiring upstream changes to Marathon itself. I haven't looked at what the Marathon event stream supports these days -- last time I checked, the events would not always provide all information necessary to determine the state delta, which is why we had to follow up with that big fetch.

Let's dig deeper once we know your precise setup.

[renaudhager]

Hi @timoreimann

Thank you for looking at this.
Your insight about how Traefik works with Marathon provider are very interesting and useful.

Here the answers to your questions.

1. We have ~ 260 deployments

2. This one is though :-), few weeks ago I searched for this informations and I find out that with have around ~ 3k changes (new deployment, due to failed health) / per day.

3. 17 mesos agents (m4.2xlarge) 8vCPUs / 32GB. 3 mesos masters (m5.large) 2vCPUs / 8GB

4. We have dedicated host to run zookeeper cluster. On the mesos master we run mesos master and marathon process.

5. CPU metric is the sum of cpu.system + cpu.user.
   Here a graph of only the marathon leader during the period, this graph cpu system + user and iowait.
   As you can see there is no I/O wait.

Right now our instance for mesos master are quite small, we could easily increase the size to something bigger and cpu focus.
My concern was around the scalability because at the minute we don't have a lot of agents but in the future we could have more than 50.

For your information when I execute /v2/apps?embed=apps.tasks&embed=apps.deployments&embed=apps.readiness against our cluster the returned json has 42K lines and size is ~ 11KB.

[timoreimann]

@renaudhager thanks for the prompt feedback, appreciated.

My organization used to run a Marathon / Mesos cluster in pretty much the same configuration as yours, with the one difference that we were at ~30 slaves (as we decided to make the slaves relatively small to limit the impact of an outage). I probably can't recover monitoring metrics from that period anymore, but at least I don't recall issues in terms of CPU consumption. We might have had more beefy machines, which is something I can try to figure out.

The 3k changes per day doesn't sound too many -- that's about 2 per minute on average. However, you likely have a concentration of changes on specific times of the day (presumably when your developers are most active deploying new releases), which may or may not be an issue.

I found an interesting item in the change log: apparently, Marathon 1.6 received a 10x improvement in API request / response serialization, and it's mentioning load reduction explicitly. Assuming that you plan to upgrade to that version at one point, I'm tempted to say that we wait to see how that version of Marathon behaves with the current Traefik implementation before we take any actions, at least as far as the more costly options are concerned. What do you think, is that something you'd be okay with?

[renaudhager]

Hi @timoreimann,

Thank you for your information. Improvement Marathon team made on the API performance looks very good.
I agree with what you said it makes sense. I will see if we can update marathon version (I don't have any time line), in the mean time I have the choice between using a bigger instance or even switch to consul provider.

Again thank you for your time, feel free to close that issue, since so far nothing is wrong on Traefik side :-)

[dduportal]

Thanks for your time and feedbacks @renaudhager , this is good news to hear!