[GKE] Error when upgrading GKEv1 cluster to 1.19

[thehejik]

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible):

• deploy rancher 2.5.7 somewhere
• deploy GKE cluster with default values - 3 UBUNTU worker nodes - just use lower k8s version 1.18.17-gke.700 which will be upgraded later
• upgrade rancher to rancher:2.5.8-rc9
• In Ember UI select the GKE cluster, select 3 vertical dots and click Edit...
• select higher k8s version 1.19.9-gke.1400 from dropdown and press Save+Done

Result:

• After a while the GKE cluster is in Updating state but with following message will appear in info panel:
  Googleapi: Error 400: Basic authentication was removed for GKE cluster versions >= 1.19. The cluster cannot be upgraded until basic authentication is disabled. To disable basic authentication use: `gcloud container clusters upgrade c-4m97d --no-enable-basic-auth`. Instructions for choosing a new method can be found at: https://cloud.google.com/kubernetes-engine/docs/how-to/api-server-authentication., badRequest

but the command gcloud container clusters upgrade c-4m97d --no-enable-basic-auth is incorrect.

thehejik@cloudshell:~ (rancher-qa)$ gcloud container clusters upgrade c-4m97d --no-enable-basic-auth                                                                                                                                                 
ERROR: (gcloud.container.clusters.upgrade) unrecognized arguments: --no-enable-basic-auth

To search the help text of gcloud commands, run:
  gcloud help -- SEARCH_TERMS

According to my findings and tests the command should be gcloud container clusters update c-4m97d --no-enable-basic-auth --zone us-central1-f so not upgrade but update and also zone/region option is mandatory.

Other details that may be helpful:
Not sure if it is needed but I run also following commands before in Google Shell console:

gcloud auth login
gcloud container clusters get-credentials c-4m97d --zone us-central1-f
kubectl get pods -A
gcloud container clusters update c-4m97d --no-enable-basic-auth --zone us-central1-f # Finally the comand

• then the rancher will upgrade the managed master node(s?) automatically (just wait a bit and the info message will disappear after a while)
• worker nodes has to be upgraded in GKE Google kubernetes webui manually (pick the same version as master has)

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.5.7 -> 2.5.8-rc9
• Installation option (single install/HA): HA

Cluster information

• Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Hosted GKE (GKEv1 from 2.5.7)
• Machine type (cloud/VM/metal) and specifications (CPU/memory): default
• Kubernetes version (use kubectl version): see in description

[thehejik]

The message is probably form Google API so maybe a note in doc would be nice.

[aaronyeeski]

This issue is reproduced on a v2.5.8-rc9 setup upgraded from v2.5.7.

Steps:
deploy rancher 2.5.7 somewhere
deploy GKE cluster with default values - 3 UBUNTU worker nodes - k8s version 1.18
upgrade rancher to rancher:2.5.8-rc9
upgrade cluster to k8s version 1.19

Result:
Upgrade fails with:
Googleapi: Error 400: Basic authentication was removed for GKE cluster versions >= 1.19. The cluster cannot be upgraded until basic authentication is disabled. To disable basic authentication use: "gcloud container clusters upgrade c-zmm2k --no-enable-basic-auth". Instructions for choosing a new method can be found at: https://cloud.google.com/kubernetes-engine/docs/how-to/api-server-authentication., badRequest

Upgrades to k8s version 1.17 and 1.18 are successful.

[thedadams]

This is not an issue with Rancher, but is something that should be documented in Rancher.

If a user disables basic authentication with GCP (https://cloud.google.com/kubernetes-engine/docs/how-to/api-server-authentication#disabling_authentication_with_a_static_password), then the upgrade will be successful. In addition, if a user tries to upgrade a GKE cluster and gets this error, they should be able to disable basic authentication in GCP and Rancher will continue with the upgrade successfully.

[thehejik]

Thanks. Closing