Update existing documentation after removal of prometheus pod annotations

[mflendrich]

Is there an existing issue for this?

• I have searched the existing issues

Problem Statement

KIC 2.0 removes prometheus.io/* annotations from the KIC/proxy pod, in favor of a PodMonitor.
There are some bits of documentation (most notably the Grafana guide) impacted by this.

Also, ensure that the KIC metrics (not only the proxy metrics) are referenced.

Proposed Solution

Audit our monitoring documentation (docs.konghq.com, chart repo) for dependency on those annotations, and fix these to work with the changes introduced in 2.0.

Additional information

No response

Acceptance Criteria

• Each bit of documentation (in particular, the Grafana guide) that requires Prometheus works with the changes to KIC pod annotations introduced in KIC 2.0
• Monitoring documentation that predates KIC 2.0 and only showcased proxy metrics now exposes the ingress controller metrics as well.

[rainest]

Kong/docs.konghq.com#3288 (specifically Kong/docs.konghq.com@7e8628c) revises the docs guide to use the Prometheus Operator. That guide was already using a ServiceMonitor from the chart and just updates it slightly to match the new Prometheus' selection criteria.

On the chart end, we probably want to sub in the PodMonitor with a renamed values.yaml setting (deprecating the old key/using it as an alias for now).

[rainest]

Nevermind: the chart can just keep using a ServiceMonitor since the magic of the Prometheus Operator is such that it's just looking for the associated Deployment and doesn't care that the port isn't actually part of that Service (which makes sense--the actual metrics port doesn't have to be exposed via a Service). The changes we already made to the chart work; controller metrics do show up in Prometheus with the docs as-is.