Regarding Annotations/Label

[Parth-Sabhadiya-Imprivata]

Hi Everyone! We want to use  sidecar.istio.io/inject (label, Beta since 2021) and sidecar.istio.io/nativeSidecar (annotation, Alpha since 2024) on specific pods -- Istio 1.29, K8s 1.35, both working well. Given the label has been Beta for 3+ years and the annotation is relatively new Alpha, are there any known risks or planned breaking changes we should be aware of before relying on these in production?

[Icaro0310]

The Beta label isn't the risky half here. On K8s 1.35 the underlying sidecar containers feature is GA, and recent Istio versions turn native sidecars on by default when the cluster supports them, so on 1.29 the annotation is probably redundant. Check one of your pods:

kubectl get pod <pod> -o jsonpath='{range .spec.initContainers[*]}{.name}={.restartPolicy}{" "}{end}'

If istio-proxy shows under initContainers with restartPolicy=Always, native mode is already active cluster-wide and you can drop the annotation.

The actual risk is ordering: with native sidecars your app containers wait on the proxy's startupProbe, so a wedged proxy delays app start, and older tooling that only reads .spec.containers won't see the sidecar at all. That tradeoff is what fixes Jobs hanging on the proxy though, so keep it. For inject, the pod label overrides the namespace label, which is what you want for per-pod opt-outs.

—
Related work of mine: github.com/Icaro0310 · Website: icaro0310.github.io

[Parth-Sabhadiya-Imprivata]

Thanks for checking! We confirmed ENABLE_NATIVE_SIDECARS=false is explicitly set cluster-wide in istiod (Istio 1.29.2, K8s 1.35) due to Dynatrace Operator incompatibility with native sidecars. So the sidecar.istio.io/nativeSidecar: "true" annotation is intentional — it overrides the cluster-wide false just for our testkube Job pods so they don't hang on proxy shutdown.

Given that context (per-pod override of a cluster-wide opt-out), is this pattern safe/stable enough for production use? Any known risks with the per-pod annotation override specifically, or is it mainly the cluster-wide ENABLE_NATIVE_SIDECARS path that's more mature?

[jmrplens]

@Parth-Sabhadiya-Imprivata the override itself is sound: on 1.29 the annotation gives those pods exactly the sidecar the flag would, and nothing else in istiod treats them differently. How stable it is comes down to its Alpha status, below.

The annotation "Takes precedence over the ENABLE_NATIVE_SIDECARS environment variable" (its description in istio/api), and the injection template reads $nativeSidecar from it, falling back to the flag only when it's absent (injection-template.yaml#L29). I checked it with istioctl kube-inject 1.29.2 on a Job, using the default profile's injector config, values and mesh config: ENABLE_NATIVE_SIDECARS=false plus the annotation gives the same pod spec as ENABLE_NATIVE_SIDECARS=true without it, except for the two env values that carry the Job's name. Outside the injector, the only code in 1.29.2 that reads the flag is the gateway deployment controller (deploymentcontroller.go#L537); everything else works from the pod spec.

For production:

• The annotation is featureStatus: Alpha on release-1.29. Istio's feature phases describe Alpha as "used to vet a solution in production without committing to it in the long term", with no backward compatibility guarantee and no deprecation policy, so an upgrade could change or drop it without notice. After each upgrade, check that an annotated pod still has istio-proxy under initContainers with restartPolicy: Always. istioctl analyze already flags these pods with IST0136 (AlphaAnnotation).
• If these are Test Workflow pods, Testkube runs every step except the last as an init container (processor.go#L608-L609). With native sidecars the injector puts istio-init and istio-proxy ahead of them, so those steps run inside the mesh and wait for the proxy's startup probe (webhook.go#L805-L812). Without native sidecars istio-init goes last, so they run before the iptables redirect exists (L813-L820). That comes with native sidecars, not with the override, but it is the biggest behavioural difference between these pods and the rest of your cluster.
• holdApplicationUntilProxyStarts doesn't apply to native pods, since the kubelet already waits on the proxy's startup probe, and a global.proxy.lifecycle in your values replaces the native preStop drain on them (L71-L73, L215-L236).
• I can't tell whether the Dynatrace problem affects them, since it isn't documented publicly, but these pods are as exposed to it as they would be with the flag on.