No CA with OUTPUT_CERTS when istio-proxy has no mTLS services configured

[Stono]

Bug description
Following on from #28050, I've noticed that when you use OUTPUT_CERTS, the pc secret command no longer returns the secrets:

In this case, I have OUTPUT_CERTS set to /etc/istio-output-certs:

/prometheus $ ls -lah /etc/istio-output-certs/
total 12K
drwxrwsrwt    2 root     1337          80 Oct 20 07:53 .
drwxr-xr-x    1 root     root        4.0K Oct 20 07:53 ..
-rwxr-xr-x    1 1337     1337        2.2K Oct 20 07:53 cert-chain.pem
-rwxr-xr-x    1 1337     1337        1.6K Oct 20 07:53 key.pem

But as you can see, they do not appear in the pc secret command any more:

❯ ./src/1.6.12/bin/istioctl pc secret prometheus-0
RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER                               NOT AFTER                NOT BEFORE
default           Cert Chain     ACTIVE     true           234734325757014090387286477695910919128     2020-10-21T07:53:58Z     2020-10-20T07:53:58Z

Affected product area (please put an X in all that apply)

[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[x] User Experience
[x] Developer Infrastructure

Affected features (please put an X in all that apply)

[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane

Expected behavior
pc secrets to still show them

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)
1.16.12

How was Istio installed?
Helm

Environment where bug was observed (cloud vendor, OS, etc)

Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.

[howardjohn]

I think this one is expected if I understand right, although you should see ROOTCA as well - is it truncated? I also see there is no root-cert.pem in output-certs?

Basically, before this Envoy was reading the files directly. pc secret reads the envoy config. Now, the secrets are served over SDS (default - probably should have been named "workload cert"), so that is what shows up, and the files are just a mirror not used by Envoy

[Stono]

Nope, no ca:

❯ kubectl -n app-metrics exec -it prometheus-0 -c prometheus -- ls /etc/istio-output-certs/
cert-chain.pem  key.pem

I'm not sure I understand your explanation either really, shouldn't pc secrets be showing the sds certs (of which there should be three) and then i should also see 3 certs in that folder too?

[howardjohn]

Bringing slack conversation here: the problem is the envoy does not request any mTLS destinations, so we have nothing configured for ROOTCA, and OUTPUT_CERTS is driven by envoy

[Stono]

Closing this issue as once we resolved #28050 - it fixed, so it's a side effect of the mount path for certs.

[howardjohn]

I am going to re-open this to track actually fixing the underlying issue.

cc @williamaronli @stevenctl this will impact VMs - if there is no mtls service we will never get a cert