feat: add sane defaults to the pod and containers in pepr deployments

[cmwylie19]

Description

Adds sane defaults to securityContext in Pepr's Admission and Watcher Deployment in the podSpec and containerSpec.

Related Issue

Fixes #444

Relates to #

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Other (security config, docs update, etc)

Checklist before merging

• Test, docs, adr added or updated as needed
• Contributor Guide Steps followed

[netlify]

✅ Deploy Preview for pepr-docs canceled.

Name	Link
🔨 Latest commit	3e54803
🔍 Latest deploy log	https://app.netlify.com/sites/pepr-docs/deploys/6580d3db79533200087da7e9

[rjferguson21]

Testing this against DUBBD 0.15.0 looks good. All policies that are currently set to enforce are no longer showing failures.

There is one remaining policy that is currently failing but it is currently only in audit mode:

(⎈|k3d-uds-test-cluster:default) ~ kubectl get policyreports -o json -n pepr-system | jq '.items[] | .results[] | select(.result == "fail") | {policy: .policy, rule: .rule, message: .message, resources: .resources }'
{
  "policy": "disallow-auto-mount-service-account-token",
  "rule": "automount-service-accounts",
  "message": "validation error: Automount Kubernetes API Credentials isn't turned off. The field automountServiceAccountToken  must be set to false. rule automount-service-accounts failed at path /automountServiceAccountToken/",
  "resources": [
    {
      "apiVersion": "v1",
      "kind": "ServiceAccount",
      "name": "pepr-static-test",
      "namespace": "pepr-system",
      "uid": "9b5bf683-8955-49a9-b34a-8972a4546675"
    }
  ]
}

The policy is roughly based on this - https://kyverno.io/policies/other/disable-automountserviceaccounttoken/disable-automountserviceaccounttoken/

Looking into what is a reasonable way to address that I found this example: https://cert-manager.io/docs/installation/best-practice/#restrict-auto-mount-of-service-account-tokens

I would be fine to move this along and create a follow on issue to address that particular policy because it seems a bit more involved.

[cmwylie19]

totally got tripped up testing locally b/c I was not able to create the configMap with SSA... turns out the code was right, but I had assigned the role create for the SSA, which of course didn't work

[rjferguson21]

Retested this against a baseline DUBBD 0.15.0 after @cmwylie19 added fixes specifically for disallow-auto-mount-service-account-token and there are no longer any Kyverno policy violations.

LGTM

[jeff-mccoy]

let's discuss this after the break please

[RothAndrew]

Status update? Looking forward to this fix so I can stop excluding the pepr-system namespace from my deployment's kyverno policies

[jeff-mccoy]

If we break this up, I don't have a problem with security context at all. I have an issue with the automount portion. Is that possible @cmwylie19?

[cmwylie19]

If we break this up, I don't have a problem with security context at all. I have an issue with the automount portion. Is that possible @cmwylie19?

Yes, definitely possible. I think that the automount is taken care of by intentionally and purposefully assigning serviceAccounts to your resources and not using the defaults.

@RothAndrew should be done today and I will look to get it reviewed tomorrow

[cmwylie19]

Tested upgrade process manually details here -- #464

[jeff-mccoy]

sgtm