Unable to find documentation for how to migrate prometheus to sds

[Stono]

Bug description
Our (multiple) prometheus servers use the istio-pilot created secrets in order to scape istio endpoints. As part of the move to istiod and subsequently sds, those secrets are no longer created.

I couldn't find any documentation to help me work out what to do next, until I was pointed at https://istio.io/v1.6/blog/2020/proxy-cert which says to use .Values.prometheus.provisionPrometheusCert to true.

However istioctl generate-manifest on 1.6.12 gives you: Error: unknown field "provisionPrometheusCert" in v1alpha1.ExternalComponentSpec:

So I couldn't.

We actually don't use the istio deployment of prometheus anyway, so i was only wanting to inspect the templated output. Instead I started looking at https://github.com/istio/istio/blob/release-1.6/manifests/charts/istio-telemetry/prometheus/templates/deployment.yaml

I can't really decipher what I need to do in order to migrate here.

Have I missed some documentation that clearly explains how people migrate to SDS?

[x] Docs
[x] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[x] User Experience
[ ] Developer Infrastructure

Expected behavior
Clear guidance when making breaking changes.

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)
1.6.12

How was Istio installed?
Helm

Environment where bug was observed (cloud vendor, OS, etc)

[Stono]

God this has been a massive rabbit hole... my next idea was to inject prometheus but effectively disable it as a sidecar

        sidecar.istio.io/inject: "true"
        traffic.sidecar.istio.io/includeInboundPorts: ""
        traffic.sidecar.istio.io/includeOutboundPorts: ""

We've modified our injector to support sidecar.istio.io/mountCerts: "true" which makes pilot write the certs to a shared volume mount of /etc/certs. The idea being that prometheus can mount the same shared directory and use it tls scrapes.

After all that i realised it won't work as prometheus currently lives in istio-system namespace, so sidecar injection doesn't even run there.

Now i'm faced with either manually injecting (complex in our pipeline, we rely in run time injection), or attempting to move prometheus (a nightmare, due to all the things that reference it).

[Stono]

The above doesn't work, outbound connections fail so it looks like includeOutboundPorts isn't setting up iptables properly.

I decided to try sidecar.istio.io/interceptionMode: NONE, however then istio-proxy doesn't start:

{"level":"warn","time":"2020-10-15T17:13:12.815538Z","msg":"Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected"}
{"level":"warn","time":"2020-10-15T17:13:17.814267Z","msg":"Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected"}
{"level":"warn","time":"2020-10-15T17:13:22.814314Z","msg":"Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected"}
{"level":"warn","time":"2020-10-15T17:13:27.814405Z","msg":"Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected"}
{"level":"warn","time":"2020-10-15T17:13:32.815822Z","msg":"Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected"}
{"level":"warn","time":"2020-10-15T17:13:37.814109Z","msg":"Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected"}

[Stono]

traffic.sidecar.istio.io/includeOutboundIPRanges: "" works, traffic.sidecar.istio.io/includeOutboundPorts: "" doesn't. 🤷

[Stono]

For anyone coming to this issue; this is how I got this working: https://karlstoney.com/2020/10/15/istio-upgrades-prometheus-sds

However the solution in that blog is blocked by this bug:
#28050

[Stono]

Closing this in favour of #28050