Istio v2 upgrade fails from 1.8.6 to 1.9.x when policy is enabled previously and the charts are uninstalled and reinstalled upgrade version check failed: 1.8.6 -> 1.9.8. Error: found 1 unsupported v1alpha1 security policy

[anupama2501]

Rancher Server Setup

• Rancher version: v2.5-head commit 47e2c3
• Installation option (Docker install/Helm Chart): Docker install
  • If Helm Chart, Kubernetes Info: k3s
    Cluster Type (RKE1, RKE2, k3s, EKS, etc):
    Node Setup: 1 node
    Version: v1.21.3+k3s1
• Proxy/Cert Details: self-signed

Information about the Cluster

• Kubernetes version: v1.20.10
• Cluster Type (Local/Downstream): Downstream infrastructure provider 3 worker, 1 etcd, 1 cp
• Proxy/Cert Details: self-signed

Describe the bug
If we install istio charts version v1.7.3 with policy enabled, upgrade it to later versions, installation fails. If we then uninstall istio charts and re-install with the version v1.8.6 [policy is not enabled, option not present] and upgrade it to v1.9.8, the upgrade still fails.
Note: Upgrade from v1.8.6 to v1.9.8 works fine on a freshly installed istio when no prior installation was made.

To Reproduce

1. Create a downstream cluster with the nodes config as mentioned above
2. Install istio v2 charts version v1.7.301 with policy enabled
3. Upgrade it to v1.8.600 [installation goes fine] and then upgrade to v1.9.8 [installation fails]
4. Uninstall istio from the apps - verified kiali crd and istio are removed and all the corresponding deployments
5. Install istio version v1.8.6
6. Deploy book demo app
7. Upgrade to v1.9.8 version [also happens if we upgrade to any 1.9.x version]

Result
Istio installation fails with the error in istioctl-installer

Warning [IST0002] (CustomResourceDefinition clusterrbacconfigs.rbac.istio.io) Deprecated: Custom resource type rbac.istio.io ClusterRbacConfig is removed
Warning [IST0002] (CustomResourceDefinition rbacconfigs.rbac.istio.io) Deprecated: Custom resource type rbac.istio.io RbacConfig is removed
Warning [IST0002] (CustomResourceDefinition servicerolebindings.rbac.istio.io) Deprecated: Custom resource type rbac.istio.io ServiceRoleBinding is removed
Warning [IST0002] (CustomResourceDefinition serviceroles.rbac.istio.io) Deprecated: Custom resource type rbac.istio.io ServiceRole is removed
Info [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run 'kubectl label namespace default istio-injection=enabled' to enable it, or 'kubectl label namespace default istio-injection=disabled' to explicitly mark it as not needing injection.
running istioctl upgrade
2021-09-09T21:03:27.552217Z	info	proto: tag has too few fields: "-"
Control Plane - istio-ingressgateway pod - istio-ingressgateway-6f5786b7cb-4776r - version: 1.8.6
Control Plane - istiod pod - istiod-cc6d7ccfc-lhn9b - version: 1.8.6
2021-09-09T21:03:27.932737Z	warn	found 6 CRD of unsupported v1alpha1 security policy: [clusterrbacconfigs.rbac.istio.io meshpolicies.authentication.istio.io policies.authentication.istio.io rbacconfigs.rbac.istio.io servicerolebindings.rbac.istio.io serviceroles.rbac.istio.io]. The v1alpha1 security policy is no longer supported starting 1.6. It's strongly recommended to delete the CRD of the v1alpha1 security policy to avoid applying any of the v1alpha1 security policy in the unsupported version
2021-09-09T21:03:27.932776Z	info	Error: upgrade version check failed: 1.8.6 -> 1.9.8. Error: found 1 unsupported v1alpha1 security policy: [meshpolicies.authentication.istio.io//default]. The v1alpha1 security policy is no longer supported starting 1.6. To continue the upgrade, Please migrate to the v1beta1 security policy and delete all the v1alpha1 security policy, See https://istio.io/news/releases/1.5.x/announcing-1.5/upgrade-notes/#authentication-policy and https://istio.io/blog/2019/v1beta1-authorization-policy/#migration-from-the-v1alpha1-policy
Error: upgrade version check failed: 1.8.6 -> 1.9.8. Error: found 1 unsupported v1alpha1 security policy: [meshpolicies.authentication.istio.io//default]. The v1alpha1 security policy is no longer supported starting 1.6. To continue the upgrade, Please migrate to the v1beta1 security policy and delete all the v1alpha1 security policy, See https://istio.io/news/releases/1.5.x/announcing-1.5/upgrade-notes/#authentication-policy and https://istio.io/blog/2019/v1beta1-authorization-policy/#migration-from-the-v1alpha1-policy
error found during istioctl upgrade

Expected Result
Istio installation should have gone through as it was un-installed and re-installed with no prior crds present.

Additional context
Ref issue: #33843

[brendarearden]

After checking through several different istio/istio sources, this is the recommended path to resolve these upgrade errors - this is a documentation fix as there is no code fix that could be implemented:

Check to see if you have resources to migrate
Run the following commands to generate a list of resources that will need migrated:

kubectl get policies.authentication.istio.io --all-namespaces
kubectl get meshpolicies.authentication.istio.io --all-namespaces
kubectl get rbacconfigs.rbac.istio.io --all-namespaces
kubectl get clusterrbacconfigs.rbac.istio.io --all-namespaces
kubectl get serviceroles.rbac.istio.io --all-namespaces
kubectl get servicerolebindings.rbac.istio.io --all-namespaces
If this does not output any resources, skip forward to the cleanup step.

Migrating Resources:
Using the resources that were output from the prevous section, determine if you would like to migrate them manually or utilize an Istio provided tool. Follow the Istio documentation to determine the specific changes you will need to make for your set up.

Additional Resources about the security policy deprecation:
https://istio.io/latest/news/releases/1.6.x/announcing-1.6/upgrade-notes/#support-ended-for-v1alpha1-security-policy
https://istio.io/latest/news/releases/1.5.x/announcing-1.5/upgrade-notes/#authentication-policy

Cleanup
Run the following commands to finish cleanup:

kubectl delete clusterrbacconfigs.rbac.istio.io --all-namespaces
kubectl delete meshpolicies.authentication.istio.io --all-namespaces
kubectl delete policies.authentication.istio.io --all-namespaces
kubectl delete rbacconfigs.rbac.istio.io --all-namespaces
kubectl delete servicerolebindings.rbac.istio.io --all-namespaces
kubectl delete serviceroles.rbac.istio.io --all-namespaces
And finally remove the istio-policy deployment:

kubectl delete deployment istio-policy --namespace istio-system
Once you have completed the steps to migrate and cleanup resources, you are ready to upgrade to a newer version of istio. Reminder: Istio supports one minor version upgrade at a time to enusre the best upgrade path. It is not recommended to go directly to the latest version if it is more than one minor version up.

[cbron]

Assigning to @ryansann for dev-validate.

[ryansann]

I was able to validate that the original error can be avoided by following the instructions @brendarearden provided above.

The steps I followed were:

1. Deploy downstream RKE cluster
2. Install Istio v1.7.301
3. Install BookInfo app
4. Migrate / Cleanup
5. Upgrade to v1.8.600
6. Migrate / Cleanup
7. Upgrade to v1.9.600
8. Observe no errors and BookInfo application functions normally