Istio Breaking Multus Network Management After Overwriting Existing Pod Annotations

[brobb7]

Bug Description

If a pod has an annotation that points to an existing NetworkAttachmentDefinition for Multus, when injecting a sidecar, Istio overwrites the the network annotation (ex: k8s.v1.cni.cncf.io/networks: macvlan-conf) with k8s.v1.cni.cncf.io/networks: istio-cni. This breaks Multus' network management by preventing macvlan-conf from executing. Istio should be appending istio-cni to the annotation, to get k8s.v1.cni.cncf.io/networks: macvlan-conf, istio-cni.

@dougbtv wrote an in-depth gist article on the effects of this issue here: https://gist.github.com/dougbtv/babe9b59e4d54e7a0fc67f134b74b908

Here is a configuration that can replicate the issue: https://gist.github.com/jacob-delgado/bb3a08d21e13130e6b685df75ec54a10

cc @dougbtv @jwendell @dgn

Version

Known Istioctl versions affected 1.11-current

Additional Information

No response

Affected product area

• Ambient
• Docs
• Installation
• Networking
• Performance and Scalability
• Extensions and Telemetry
• Security
• Test and Release
• User Experience
• Developer Infrastructure
• Upgrade
• Multi Cluster
• Virtual Machine
• Control Plane Revisions

[brobb7]

Discussion on #45036 has determined that this issue is being caused by the instructions to add the following annotation to installation overrides files:

    sidecarInjectorWebhook:
      injectedAnnotations:
        k8s.v1.cni.cncf.io/networks: istio-cni

This is duplicating the work of the appendMultusNetworks function in the helm configuration and is causing the appended multus configurations to be overwritten during helm installation.