Size limits for Empty Dirs

[chaitanya14]

What this PR does / why we need it:

Empty directory without size limit is consuming all the ephemeral local storage resources. Need to have a capability to assign limits on it.

Which issue this PR fixes

(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

• fixes #

Special notes for your reviewer:

Checklist

[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

• PR is based off the current tip of the main branch.
• Changes are documented under the "Unreleased" header in CHANGELOG.md
• New or modified sections of values.yaml are documented in the README.md
• Commits follow the Kong commit message guidelines

[CLAassistant]

All committers have signed the CLA.

[rainest]

What were you seeing these growing to in practice? What were the largest files in the mounts?

I'd prefer to make these static limits rather than configurable if possible, since they should be small and consistent enough in size that users shouldn't need to worry about configuring them. I don't know offhand what'd consume much space in those locations other than logs (which we want to divert to stdout and stderr)--I'm checking with the gateway engineers, but info from your instance would be helpful.

If we do end up using configurable values, they need to be in the default values.yaml and in documentation, else you'll hit the error that's currently failing in CI:

Error:  templates/: template: kong/templates/deployment.yaml:288:10: executing "kong/templates/deployment.yaml" at <include "kong.volumes" .>: error calling include: template: kong/templates/_helpers.tpl:414:25: executing "kong.volumes" at <.Values.deployment.prefixDir.sizeLimit>: nil pointer evaluating interface {}.sizeLimit

[chaitanya14]

What were you seeing these growing to in practice? What were the largest files in the mounts?

I'd prefer to make these static limits rather than configurable if possible, since they should be small and consistent enough in size that users shouldn't need to worry about configuring them. I don't know offhand what'd consume much space in those locations other than logs (which we want to divert to stdout and stderr)--I'm checking with the gateway engineers, but info from your instance would be helpful.

If we do end up using configurable values, they need to be in the default values.yaml and in documentation, else you'll hit the error that's currently failing in CI:

Error:  templates/: template: kong/templates/deployment.yaml:288:10: executing "kong/templates/deployment.yaml" at <include "kong.volumes" .>: error calling include: template: kong/templates/_helpers.tpl:414:25: executing "kong.volumes" at <.Values.deployment.prefixDir.sizeLimit>: nil pointer evaluating interface {}.sizeLimit

I don't have certain numbers on limits but we are facing disk pressure issues on AKS nodes. These directories may or may not be directly effecting our issue but as a best practice it is better to evict the pods if empty dirs are using more ephemeral storage than defined. This will improve things in kubernetes cluster .

[chaitanya14]

What were you seeing these growing to in practice? What were the largest files in the mounts?
I'd prefer to make these static limits rather than configurable if possible, since they should be small and consistent enough in size that users shouldn't need to worry about configuring them. I don't know offhand what'd consume much space in those locations other than logs (which we want to divert to stdout and stderr)--I'm checking with the gateway engineers, but info from your instance would be helpful.
If we do end up using configurable values, they need to be in the default values.yaml and in documentation, else you'll hit the error that's currently failing in CI:

Error:  templates/: template: kong/templates/deployment.yaml:288:10: executing "kong/templates/deployment.yaml" at <include "kong.volumes" .>: error calling include: template: kong/templates/_helpers.tpl:414:25: executing "kong.volumes" at <.Values.deployment.prefixDir.sizeLimit>: nil pointer evaluating interface {}.sizeLimit

I don't have certain numbers on limits but we are facing disk pressure issues on AKS nodes. These directories may or may not be directly effecting our issue but as a best practice it is better to evict the pods if empty dirs are using more ephemeral storage than defined. This will improve things in kubernetes cluster .

Also, we have a data plane where we have used 256Mi size of empty dirs that never caused an issue since 4 months. So IMO default to 256Mi is more meaningful.

[rainest]

We'll want to merge #634 first. Several of those were wrong and would eventually fill the prefix on a long-lived Pod.

There's one unbounded consumer of /tmp/ space whose actual needs will vary depending on use characteristics, so we do indeed need to make these configurable. NGINX will buffer large client bodies to disk, and we set no default upper bound on their size. An instance with a large number of concurrent large bodies (e.g. something that handles video uploads) is expected to consume a bunch of /tmp/ space even though most instances won't, so IMO we should give that one more breathing room.

[chaitanya14]

@rainest When will this be released ?

[rainest]

Probably end of this week or start of next; there are a few other things I'd like to work in if possible.

[pmalek]

We missed a section name in here I suppose