istio operator stuck after adding new label

[arnoldyahad]

Bug Description

Hey folks,
when adding a new label to the istio operator it got stuck and caused our gateway pods to not work, this is the change i've made:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-ingress-gateway
  namespace: istio-system
spec:
  namespace: istio-system
  components:
    ingressGateways:
    - name: istio-ingressgateway
      enabled: false
    - name: test-lb
      enabled: true
      namespace: istio-ingress
      label:
        app: test-lb
        istio: test-lb
        co.elastic.logs/enabled: "true" # New label
        {{- end }}

this caused an error:

status:
  componentStatus:
    IngressGateways:
      error: >-
        failed to update resource with server-side apply for obj
        Deployment/istio-ingress/test-lb: Deployment.apps "test-lb" is invalid:
        spec.selector: Invalid value:
        v1.LabelSelector{MatchLabels:map[string]string{"app":"test-lb",
        "co.elastic.logs/enabled":"true", "istio":"test-lb",
        "type":"ingressgateway"},
        MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
      status: ERROR
  status: ERROR

and also when reading the logs of the operator it got stuck entierly on that error and doesnt print anymore messages:

2023-02-13T15:39:45.637227Z	info	installer	using server side apply to update obj: PodDisruptionBudget/istio-ingress/test-lb
2023-02-13T15:39:45.688361Z	info	installer	using server side apply to update obj: Service/istio-ingress/test-lb
- Processing resources for Ingress gateways.
✘ Ingress gateways encountered an error: failed to update resource with server-side apply for obj Deployment/istio-ingress/test-lb: Deployment.apps "test-lb" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"test-lb", "co.elastic.logs/enabled":"true", "istio":"test-lb", "type":"ingressgateway"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

new test-lb pods are not passing their readiness gate:

kubectl get pods -n istio-ingress -o wide
NAME                            READY   STATUS    RESTARTS   AGE     IP               NODE                             NOMINATED NODE   READINESS GATES
test-lb-59f9fb69b6-lqlgn        1/1     Running   0          22m     10.191.193.234   ip-10-191-207-7.ec2.internal     <none>           <none>
test-lb-59f9fb69b6-m8vqf        1/1     Running   0          22m     10.191.202.109   ip-10-191-197-112.ec2.internal   <none>           <none>

is this an intended behavior? because this means that if someone adds a label the istio-ingress pods will stop working.
is there any fix or other way to add labels to existing istio-ingress pods?

Version

istioctl version --remote
client version: 1.15.0
control plane version: 1.16.1
data plane version: 1.16.1 (43 proxies), 1.13.5 (5 proxies)

kubectl version --short
Client Version: v1.22.15-eks-fb459a0
Server Version: v1.23.14-eks-ffeb93d

Additional Information

No response

[hanxiaop]

I cannot reproduce. What is your operator version?

[arnoldyahad]

@hanxiaop the istio-operator image is:

    Image:         docker.io/istio/operator:1.16.1
    Image ID:      docker.io/istio/operator@sha256:9763f0d5278699648fb810db0d56332efff02c45b32bb7586ac67bb9e75e7151

if thats what you meant with version

[hanxiaop]

@arnoldyahad Sorry I misunderstood the issue. I think this is as expected. You want to change the label which will result in the matchLabels change, and this is not allowed in k8s.

[arnoldyahad]

@hanxiaop thanks but i would expect istio-operator to just block the change and not make my ingress pods not usable anymore.

is there a proper way to add a label to an existing gateway deployment?

[arnoldyahad]

@hanxiaop thanks for your reply! - also, do you know(or anyone else reading this) is it possible to do if i dont use an operator but i use the helm charts? (to add labels)

https://istio.io/latest/docs/setup/install/helm/ as istio announced this is now promoted to beta, but if it allows me to easily manipulate labels, maybe its worth switching to.

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2023-02-14. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

[josh-ferrell]

+1 to having the operator validate component labels haven't changed. It will update the service with new selectors but never be able to update immutable fields on the deployment bringing components like the ingress gateway down since the pods are running with the original label values.

[hanxiaop]

@hanxiaop thanks for your reply! - also, do you know(or anyone else reading this) is it possible to do if i dont use an operator but i use the helm charts? (to add labels)

https://istio.io/latest/docs/setup/install/helm/ as istio announced this is now promoted to beta, but if it allows me to easily manipulate labels, maybe its worth switching to.

Sorry for missing this. I think using helm is better, adding labels to existing gateway deployment should not work since it's k8s limitation.

[hanxiaop]

+1 to having the operator validate component labels haven't changed. It will update the service with new selectors but never be able to update immutable fields on the deployment bringing components like the ingress gateway down since the pods are running with the original label values.

I think we have an ordering logic where services should be applied only after the pods are ready. However, there might be an issue causing services to be applied even when the deployment application has failed.