Disable PSP and configure securityContext properly for observability pods

[a-thaler]

Description

Changes proposed in this pull request:

• disabled all PSPs
• Configured securityContext were possible
• disabled garafanas initChownData initContainer as it is not required and needs special permissions

Left warnings for restricted security level are:

Warning: monitoring-prometheus-node-exporter-kj8bv: host namespaces, hostPort, restricted volume types
Warning: telemetry-fluent-bit-vbk9q: unrestricted capabilities, restricted volume types, runAsNonRoot != true
Warning: tracing-jaeger-7d6cdffb9b-9rfz5: allowPrivilegeEscalation != false, unrestricted capabilities

Left warnings for baseline security level are:

Warning: monitoring-prometheus-node-exporter-kj8bv: host namespaces, hostPath volumes, hostPort

The reasoning behind that is:

• fluent-bit must read from the host filesystem and requires capabilities for that and must run as root
• prometheus-node-exporter requires to access to the host as well
• with the jaeger-operator in current version a securityContext on container level cannot be set, we would need to update first

I applied these default settings:

podSecurityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault

containerSecurityContext:
  allowPrivilegeEscalation: false
  privileged: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

Related issue(s)

[kyma-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[a-thaler]

/test all

[a-thaler]

/retest