Prometheus cert provisioning is broken on istio-1.5.0-beta.5

[lei-tang]

Bug description
Prometheus cert provisioning works on istio-1.5.0-beta.4 but is broken on istio-1.5.0-beta.5. The new commits (e.g., the ones that change Istio Agent) in istio-1.5.0-beta.5 may be related to this problem.

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ X] Policies and Telemetry
[X ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior
Prometheus cert provisioning works for istio-1.5.0-beta.5.

Steps to reproduce the bug
Use the steps in the guide https://github.com/istio/istio.io/pull/6554/files, which works for istio-1.5.0-beta.4 but breaks for istio-1.5.0-beta.5.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
istio-1.5.0-beta.5

How was Istio installed?
istioctl

Environment where bug was observed (cloud vendor, OS, etc)
MacOS.

Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.

[lei-tang]

Cause for #21843: outputKeyCertToDir used by Prometheus cert provisioning reads its value from the environmental variable OUTPUT_KEY_CERT_TO_DIRECTORY. On release-1.5 branch, a commit in istio-1.5.0-beta.5 changes outputKeyCertToDir to read its value from the environmental variable OUTPUT_CERTS, which was not set in the Prometheus deployment.

The master branch does not have the problem in #21843 because the commit in istio-1.5.0-beta.5 that changes outputKeyCertToDir to read its value from the environmental variable OUTPUT_CERTS is only merged in the release-1.5 branch but not in the master branch.

[lei-tang]

The fix for this issue: #21848.

[myidpt]

For Istio 1.5.0 users, here's how to manually fix this issue:

kubectl edit deploy prometheus -n istio-system

When the deploy config is opened, find the OUTPUT_KEY_CERT_TO_DIRECTORY in the env stanza for the proxy, and change it to OUTPUT_CERTS.

Wait for the new Prometheus instance to come up. Check the key and certificates are loaded by:

kubectl exec -it `kubectl get pod -l app=prometheus -n istio-system -o jsonpath='{.items[0].metadata.name}'` -c prometheus -n istio-system -- ls -la /etc/istio-certs/