Still a memory leak with k8s - 1.1 RC4

[rrichardson]

I know this was addressed in #387

I also note that somebody reported an existing leak in the new solution (back in May).

I am running 1.1 RC4

Here is what I'm seeing:

The 2nd item in the list git the memory limit and was killed.

[emilevauge]

Ouch, can you give us more details? Which Kubernetes version are you using? Were you running traefik 1.0 before? Any difference?
/cc @containous/traefik

[rrichardson]

k8s version : Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

I haven't run a prior version of Traefik (at least not while monitoring and using the cluster much)

Are there any debug/gc metrics I can pull out of Traefik to help narrow this down?

[emilevauge]

Anything in the logs?

[rrichardson]

Here's an anonymized debug log, I'm not really sure what I'm looking for..

traefik.log.gz

There are a couple errors involving broken connections and failure to parse :

time="2016-11-14T16:43:45Z" level=error msg="Kubernetes connection error failed to decode watch event: GET "https://100.64.0.1:443/apis/extensions/v1beta1/ingresses\" : unexpected EOF, retrying in 605.782735ms"

time="2016-11-14T16:45:58Z" level=error msg="Kubernetes connection error failed to decode watch event: GET "https://100.64.0.1:443/api/v1/endpoints\" : invalid character 'o' looking for beginning of value, retrying in 276.215364ms"

Other than that, I can't find anything that might lead to a leak.

[errm]

Are you running a HA master with the --leader-elect flag set on kube-scheduler and/or kube-controller-manager ?

[rrichardson]

I am running HA with leader-elect=true on the controller-manager.
3 masters.

[emilevauge]

There was an old issue with event spamming, but it did not lead to a memory leak...
#449

[emilevauge]

I tried to reproduce this with Kubernetes v1.4.4 and have not run into a memory leak in ours, despite the leadership election events spamming.
Another way to investigate would be to kill -SIGABRT $PID_OF_TRAEFIK (while traefik uses a lot of memory) and send us the stack.

[rrichardson]

I wonder if it a function of the number of services/pods. We currently have about 70 pods across 30 services.

Either way, attached is the stack trace. (with 6 million goroutines :) )

traefik-abrt.log.gz

[emilevauge]

@rrichardson thanks for investigating on this, it will help us alot :)

[emilevauge]

Hey @rrichardson, it would be awesome if you could test with this docker image containous/traefik:k8s, it has been built with the fix #845.

[rrichardson]

The new image has been running for 1 hour and so far the results are encouraging. Each pod is using about 20MB of RAM. I don't have any pretty charts yet, but things look good so far.

[emilevauge]

@rrichardson I love what you are saying 👍

[rrichardson]

[rrichardson]

It hasn't leveled off yet, which is a bit of a concern. However, the previous build would increase at a rate of 35MB/hour. The current build went from 20MB to 40MB in 2.5 hours, or 8MB/hr. I'll check again this evening to see if it has stopped increasing.

[emilevauge]

@rrichardson

It hasn't leveled off yet, which is a bit of a concern

Indeed...
Could you kill -SIGABRT $PID_OF_TRAEFIK again?

[rrichardson]

traefik.log.gz

[emilevauge]

I updated the Docker image containous/traefik:k8s with another fix.
Goroutines and memory allocation are pretty stable on my laptop:

[rrichardson]

I've been breaking and fixing prometheus all day, so I don't have much historical data, but I have been running the new image for almost an hour, and so far.

In the previous build, the instances were at 25MB by now, these seem to be holding steady at 14MB. I'll report back tomorrow.

[rrichardson]

The latest fix definitely worked. All 3 instances are sitting at 14MB after almost 24 hours.

[emilevauge]

@rrichardson Awesome news :) I really would like to thank you for your help on this 👍