Inconsistence in istioctl upgrade on 1.12 version

[cnvergence]

Bug Description

Installing default 1.12.1 Istio on the cluster with istioctl install --set profile=default -y
and upgrading it to 1.12.2 and then to 1.12.3 with istioctl upgrade --set profile=default -y
will result in the following error:

Control Plane - istio-ingressgateway pod - istio-ingressgateway-8c48d875-zkztr - version: 1.12.2
Control Plane - istiod pod - istiod-58d79b7bff-7qfrs - version: 1.12.2

Upgrade version check passed: 1.12.2 -> 1.12.3.

Upgrade check: Warning!!! The following IOPS will be changed as part of upgrade. Please double check they are correct:
spec:
  installPackagePath: /var/folders/w9/y0g091cn7fb7shnmqw84vp_m0000gn/T/istio-install-packages/istio-1.12.1/manifests
    -> <empty> (REMOVED)
  values:
    pilot:
      env:
        ENABLE_LEGACY_FSGROUP_INJECTION: <empty> -> false (ADDED)

2022-03-14T14:32:58.144294Z	info	Error: failed to apply the Istio Control Plane specs. Error: errors occurred during operation: creating default tag would conflict:
Error [IST0139] (MutatingWebhookConfiguration istio-sidecar-injector) Webhook overlaps with others: [istio-revision-tag-default/namespace.sidecar-injector.istio.io]. This may cause injection to occur twice.
Error [IST0139] (MutatingWebhookConfiguration istio-sidecar-injector) Webhook overlaps with others: [istio-revision-tag-default/object.sidecar-injector.istio.io]. This may cause injection to occur twice.
Error [IST0139] (MutatingWebhookConfiguration istio-sidecar-injector) Webhook overlaps with others: [istio-revision-tag-default/rev.namespace.sidecar-injector.istio.io]. This may cause injection to occur twice.
Error [IST0139] (MutatingWebhookConfiguration istio-sidecar-injector) Webhook overlaps with others: [istio-revision-tag-default/rev.object.sidecar-injector.istio.io]. This may cause injection to occur twice.

Error: failed to apply the Istio Control Plane specs. Error: errors occurred during operation: creating default tag would conflict:
Error [IST0139] (MutatingWebhookConfiguration istio-sidecar-injector) Webhook overlaps with others: [istio-revision-tag-default/namespace.sidecar-injector.istio.io]. This may cause injection to occur twice.
Error [IST0139] (MutatingWebhookConfiguration istio-sidecar-injector) Webhook overlaps with others: [istio-revision-tag-default/object.sidecar-injector.istio.io]. This may cause injection to occur twice.
Error [IST0139] (MutatingWebhookConfiguration istio-sidecar-injector) Webhook overlaps with others: [istio-revision-tag-default/rev.namespace.sidecar-injector.istio.io]. This may cause injection to occur twice.
Error [IST0139] (MutatingWebhookConfiguration istio-sidecar-injector) Webhook overlaps with others: [istio-revision-tag-default/rev.object.sidecar-injector.istio.io]. This may cause injection to occur twice.

Version

client version: 1.12.3
control plane version: 1.12.2
data plane version: 1.12.2 (1 proxies)

Additional Information

This is probably related to the fact that the upgrade command does not check if the default revision exists on the cluster, as the install command does.
The newer version should not have this issue, as there istioctl upgrade is an alias to the istioctl install.

The question is if it should be fixed in the release-1.12 or not

[litong01]

@cnvergence were you using upgrade command going from 1.12.1 to 1.12.2? would like to have your confirmation so that I can try this to recreate the issue. This webhook thing has been a headache lately.

[cnvergence]

yes, this is only related to upgrade command on 1.12.
The error pops up only after the second upgrade.

I may have the solution for this already, but don't know if it should be fixed on release-1.12.

[litong01]

using install command wont cause this issue. add it here so that other may try to use install rather than upgrade command for upgrading.

[litong01]

this procedure will also create the same error

1. install istio 1.12.2
2. use upgrade command to upgrade to 1.12.3 (not issue)
3. use upgrade command to upgrade to 1.12.4

The above procedure create the same error.

[litong01]

also tried from 1.12.3 to 1.12.4 to 1.12.5 same error.
This problem does not occur on 1.13.x. which basically direct the upgrade to install command. So in 1.13.0 or newer, upgrade and install are the same thing, but it is different in prior releases.

[cnvergence]

that's correct, I believe the revision tag logic from install, should be also included in the upgrade on 1.12
Specifically, the logic from this PR #33773