Flake: `iptables: create-instance-chains: iptables: Memory allocation problem.`

[topherbullock]

Flake Report

Occasionally it seems like Garden fails here when Concourse is trying to create a task container.

Examples

• https://ci.concourse-ci.org/teams/main/pipelines/concourse/jobs/testflight/builds/382
• PCI team in the Toronto office saw this in their Concourse ( 4.2.1 ) environment as well

Observations

• The worker metrics don't show any abnormalities in terms of memory utilization https://snapshot.raintank.io/dashboard/snapshot/2LRam0LlWhwwEMP38tr6MLNZ4FV6gaAL?orgId=2

Additional Context

@kcmannem has a stack dump from PCI's issue

Version Info

• Concourse version: 4.2.1 & 5.0.0
• Deployment type (BOSH/Docker/binary): BOSH
• Infrastructure/IaaS: GCP
• Did this used to work? 🤷‍♂️

[kcmannem]

Garden team investigated this issue. They said just increase the workers memory. There is no other fix they can provide.

[cirocosta]

Which seems like a very weird remediation in this case - the workers were all at 20% utilization in our case (w/out limits being provided to the containers) 🤔

[vito]

Yeah...gotta say I'm pretty skeptical. Our workers never really seem to show high memory usage. They already have 16GB each, I don't think this is a situation where we can just throw hardware at it.

The only scenario I can imagine is a very quick spike in memory allocation that goes down just as quickly, but I find it hard to imagine a spike big enough to allocate ~12GB of RAM, let alone one that's quick enough to escape our metrics.

[kcmannem]

Heres what they had to say:

"A hypothesis we had was that since this is only observed on iptables operations that perhaps the table within kernel memory keeping track of this stuff filled up. I’ve heard from people that iptable is pretty inefficient with memory when it’s persisting rules."

[vito]

One thing of note is that build runs containers in Docker Compose, in a container. Could iptables rules be sticking around between builds? Or do they get cleaned up properly when the network namespace is destroyed?

[topherbullock]

Yeah, I don't think more worker memory will help here; the workers' memory utilization ~15% when the flake occurred.

@vito Garden should destroy the iptables for the container when the container is destroyed (https://github.com/cloudfoundry/guardian/blob/7536c003d476be01a24401148db6238416f53c02/gardener/gardener_test.go#L830-L835, https://github.com/cloudfoundry/guardian/blob/7536c003d476be01a24401148db6238416f53c02/kawasaki/networker.go#L246-L278). I think any iptables entries created by Docker will go away; we'd probably notice a lot more issues if they were leaking

[davidjahn]

Just chiming in that we are noticing this flake as well when we are running slightly more containers/worker than usual (~160 on one with highest load).

[marco-m]

We started seeing the same with 5.0 RC54

[cirocosta]

Hey @marco-m ,

Do you also see this happening only in workers with more containers than the usual like @davidjahn ?

thx!

[wagdav]

@cirocosta

@marco-m and I summarized what we see in #3251

[edude03]

We're seeing this with 5.0.0-rc4 when there are more builds than usual as well.

[marco-m]

To summarize: we are currently using RC78, which contains the fix for #3251. We don't see any more the iptables error. We have a distribution of containers per worker of average 100 (very low because we are currently running overprovisioned).

@edude03 I strongly suggest to upgrade and to try fewest as container placement srategy; RC4 was pre #3251.