Fix k8s memory leak

[emilevauge]

This PR fixes memory leaks in Kubernetes backend.
/cc @nlf
Fixes #377

[emilevauge]

Waiting feedback from @nlf before merging

[nlf]

LGTM very very significant improvement 👍

[emilevauge]

Hum, I need more time on this...

[errm]

It seems much better, but it still looks like memory usage is growing unbounded, albeit at a much slower rate...

[emilevauge]

I just pushed a new version for testing purpose on docker hub: containous/traefik:k8s

[errm]

Ok just deployed containous/traefik:k8s will let you know what it looks like ...

[nlf]

i've got the latest k8s tag deployed too and am monitoring memory growth, i'll report back later today

[errm]

The leak is still there, but much slower. I am seeing the memory usage go from 20MiB to 256MiB in 5 hours, after 256MiB I am killing the process.

It seems that memory is leaking from somewhere in k8s.client#WatchAll as I have a branch where I am watching endpoints rather than pods and replication controllers, and the memory leak is much slower again... as watched events fire far less often.

[emilevauge]

Hi @errm, I can't reproduce your leak on my computer :'(
@nlf do you have the same behavior ?

[nlf]

i can't say for sure, our hosting provider had some issues so we had a bunch of services restart from that. if everything stays up ok today i'll report back later

[errm]

@emilevauge I am still seeing the issue, but I these changes are a big improvement so I think it would be good to merge them in. For me the issue is solved when I am running these changes, plus the stuff I am working on to build the backend with endpoints.

I will be able to open a PR with as soon as this is merged...

[emilevauge]

@errm OK ;) Let's merge this one then /cc @vdemeester :)
You could also try to modify the parameter providersThrottleDuration in traefik, as you have a high rate of kubernetes events. By default, it is set to 2s, maybe you could try with 500ms, it could help.

[nlf]

I tend to agree, it's so significantly better than it was it's worth merging as is to keep things moving forward. For what it's worth, of the 4 instances I'm running it looks like only one of them is leaking at all, and it appears that garbage collection is fighting to keep it under control so the growth is pretty minimal

[vdemeester]

Sounds good to me (based on the comments).
LGTM 🐯