OOM of Image automation controller

[ZeroDeltaAlpha]

Problem

We are seeing the Ephemeral Storage of the image automation controller grow quite rapidly and, in the past have experienced OOM evictions due to this limit being reached. We have now given the controller a 100G limit as it routinely consumes 30G~ at any given time

What solutions are there other than feeding the deployment more ephemeral storage ?

Specs and setup

flux: v0.24.1
helm-controller: v0.12.0
image-automation-controller: v0.15.0
image-reflector-controller: v0.12.0
kustomize-controller: v0.15.2
notification-controller: v0.17.0
source-controller: v0.16.0

7 Git repositories with 2 sources of them watching two branches for each repository.

[stefanprodan]

Is this the case with source-controller?

[squaremo]

How's it look a few days later -- does it look unbounded but slower, or has the problem gone away?

[ZeroDeltaAlpha]

HI @squaremo , Seeing a slower rate of disk usage, Still getting Evicted but now due to DiskPressure on the node, would there be any down side to setting the tmp directory to use a EBS volume ? As I'm understanding that the controller does its own GC

[squaremo]

A couple of downsides:

• putting git checkouts on persistent storage means they are more accessible through side channels (e.g., by mounting that EBS volume in some VM); if everyone with access to your cluster (and AWS infrastructure) is trusted, this may be less of a concern for you
• If it grows without (apparent) bound in memory, it'll grow without bound in block storage -- though you will have a lot more headroom.

The "steady state" for the image-automation-controller is to be using no disk, so ultimately there's a bug there to fix. In the meantime, you could perhaps have fewer Evictions -- and thereby fewer automation outages and alerts -- if you use an EBS volume (modulo the concerns above).

[ZeroDeltaAlpha]

@squaremo to give you an idea of how our testing is going this copy of automation controller has been running for 3d4h and this is how it's ephemeral storage looks in grafana

The GC looks more aggressive this time and doesn't breach 2G of usage, which is a massive improvement and we haven't had the pod get evicted off the node for any reason.

[kingdonb]

It sounds like things turned around. I just want to highlight for anyone that sees a recurrence of this issue,

• Log the error when removing the working dir fails image-automation-controller#287

added logging which should make it possible to pinpoint times when removing a working directory with RemoveAll failed.