Use K8s PodPresets

[arm4b]

Moved from https://github.com/StackStorm/st2enterprise-dockerfiles/issues/83

It turns out that instead of sharing same volumes, files, secrets and ENV vars for each Deployment, we can just create PodPresets for each resource like volume, vars, secrets with selectors that will apply resource sharing with specific containers.
This would help to avoid resource/code duplication and slightly simplify K8s objects.

See

• https://kubernetes.io/docs/tasks/inject-data-application/podpreset/
• https://unofficial-kubernetes.readthedocs.io/en/latest/tasks/run-application/podpreset/

[cognifloyd]

PodPresets were removed in kubernetes 1.20:

The v1alpha1 PodPreset API and admission plugin has been removed with no built-in replacement. Admission webhooks can be used to modify pods on creation. (kubernetes/kubernetes#94090, ...)
See: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md

Admission Controllers is the replacement, however it requires running an additional process in the k8s control plane.
I looked at adding my some of my changes via such an admissions controller, but I don't want to run or maintain such a controller at this point; Instead I opted for direct changes to the helm templates instead.

As this is no longer "low hanging fruit", I'm going to close this.