Omiagent seems to leak memory with OmsAgent and Docker

[fjollberg]

I'm not sure yet what is going on here, but I'll report it anyway with what I know.

I see a steady growth in resident memory used by the omiagent process on Ubuntu 16.04.1 LTS VM:s in Azure running Docker 1.12.3 together with the latest OmsAgent, v1.2.0-148.

Attached is an image showing the result in OMS after a systemctl restart omid, freeing up about 300Mb of memory per node, then starting growing linearly again.

The growth is identical on all docker hosts, regardless of their running containers which are different (apart from one global service). The graph shows five swarm members with the same level of growth from different base levels. Note that the graph, though hard to see, also includes one management node not running the docker daemon (the flat dark green line), and it does not show this growth. The management VM is smaller (A1 vs A2), but otherwise runs the same software apart from the Docker daemon.

I cannot find anything in any logs seeming related.

$ uname -a
Linux kth-integral-2 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ sudo docker version
Client:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 22:01:48 2016
OS/Arch: linux/amd64

Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 22:01:48 2016
OS/Arch: linux/amd64

[fjollberg]

More specifics about the setup. The nodes are setup using docker-machine as a five node swarm cluster, new 1.12 swarm mode. Five swarm nodes are thus created and they are all joined to the swarm as managers.

The swarm started as 1.12.1 but has later been upgraded to 1.12.3 without issues. Also, OmsAgent started as the previous version but was upgraded. Of notice is that it seems to have tried to update itself at some point and failed.

The latest version of the OmsAgent was then manually reinstalled in order to re-establish logging to OMS. Previously we had the situation with OmsAgent bits frequently crashing. This seems to have been resolved with this later version, however the new behaviour with memory growth started. I did not see such memory growth previously. I have no idea what version of omiagent the previous version of OmsAgent used.

[fjollberg]

More weird behaviour. After deploying a new service to the swarm, the omiagent on that particular host the container was scheduled to suddenly stopped leaking memory. The others still do.

[jeffaco]

This is actually a problem with the Docker Provider, I believe. It's running omiagent, but omiagent launches code from another project.

The Docker Provider is released with OMS. Please report bugs there, as they are responsible for the overall OMS project, and they know what new versions are in the works for bug fixes.

Please be certain you're running the latest/greatest version of OMS. If you are, please open an issue there to report that. Thanks.

I'm going to close this issue since it's not related to OMI directly. I think the memory growth issue is known too, but check the OMS issues for more information.

[helderpinto]

I am observing the same problem in my Ubuntu 14.04.4 LTS hosts. I couldn't find any related issue in OMS. Do you know if the most recent OMS version solved the problem or if there is any work-around?

[jeffaco]

@helderpinto Please see my prior response posted on February 6th.

OMI is not the issue here. OMI is just the transport, any provider can leak memory, and OMI has nothing to do with that.

Please open an issue with OMS, and they should be able to help you. Thanks.