upgrade_strategy.timeout on upgraded Rancher clusters sets to 0 instead of 120

[bentastic27]

What kind of request is this (question/bug/enhancement/feature request):
bug

Steps to reproduce (least amount of steps as possible):

1. Start with with Rancher 2.3.6 and a custom downstream cluster on any version.
2. Update Rancher to 2.4.3
3. Update cluster to a current supported version (I did 1.17.5)
4. edit cluster as yaml and hit save.

Result:

The UI will show the following error:

 Validation failed in API: rancherKubernetesEngineConfig upgradeStrategy=InvalidFormat 422: nodeDrainInput=InvalidFormat 422: timeout=MinLimitExceeded 422:

To fix, edit the cluster yaml and set the upgrade_strategy.timeout to something other than 0, like 120. The UI allows you to save then.

gzrancher/rancher#11317

[sowmyav27]

Reproduced on a fresh install of 2.4.5-rc6

• Deploy a custom cluster through the API
• Upgrade strategy: Drain - False
• When the cluster comes up active
• Do an Edit on the cluster. Select Drain - True (with default values)

• Enable Scheduled CIS scan on the cluster
• Save the changes made
• Do an edit on the cluster
• Drain timeout is set to 0

• Click on Save
• Error seen Validation failed in API: rancherKubernetesEngineConfig upgradeStrategy=InvalidFormat 422: nodeDrainInput=InvalidFormat 422: timeout=MinLimitExceeded 422:
• Choose Drain as True, Select Drain Timeout Keep trying for - enter 120 seconds. Choose Drain as False
• User is able to save without any error.

[StoneCut]

Same issue occurs on 2.4.5 after the first succesful upgrade of a custom cluster. Doing it again (or simply choosing "edit" and then "save") results in the same error:
Validation failed in API: rancherKubernetesEngineConfig upgradeStrategy=InvalidFormat 422: nodeDrainInput=InvalidFormat 422: timeout=MinLimitExceeded 422:
Is there any workaround for this?

[jloisel]

Same issue here on Rancher v2.4.5, when trying to upgrade a cluster from v1.17.6 to v1.18.6:

[StoneCut]

This is a frustrating bug.

As a workaround edit the cluster configuration as YAML file. Then find the section "upgrade_strategy" and edit "node_drain_input" -> "timeout: 0" to "timeout: 120".

[Tejeev]

We saw this on v2.4.5 when trying to update to k8s v1.18.5

[jloisel]

The workaround works well, it's probably just because the old data contains the wrong value.

[codyrancher]

What appears to be happening is the backend seems to be setting timeout to 0 when we save changes with these two settings:

• upgradeStrategy.drain = false
• upgradeStrategy.nodeDrainInput.timeout = undefined

I put in a stopgap from the frontend to resolve this but we should ultimately resolve this from the backend so the API users don't run into this.

[sowmyav27]

Verified on 2.4-head - commit id: 3e543f7c4

• Deploy a custom cluster through the API - Upgrade strategy: Drain - False
• When the cluster comes up active
• Do an Edit on the cluster. Select Drain - True (with default values)
• Enable Scheduled CIS scan on the cluster
• Save the changes made
• Do an edit on the cluster
• Drain timeout is set to 1
• Change the max-pods in the cluster.yml and click on Save
• The cluster goes into updating state and an error is seen [controlPlane] Failed to upgrade Control Plane: [[error draining node ip-<>: error when waiting for pod "cattle-cluster-agent-66849d8fb9-kbqmr" terminating: global timeout reached: 1s]]
• Default value 1second for Drain timeout causes this error.

Expected:

• Drain timeout 1 second is too small for an upgrade to go through when Drain is set to true.
• Drain can be set to 120 seconds when the default value is null or 0

[sowmyav27]

Verified on 2.4-head - commit id: 3e543f7, ui tag: latest-2.4

• Deploy a custom cluster through the API - Upgrade strategy: Drain - False
• When the cluster comes up active
• Do an Edit on the cluster. Select Drain - True (with default values)
• Enable Scheduled CIS scan on the cluster
• Save the changes made
• Do an edit on the cluster
• Drain timeout is seen set to 120
• edit cluster as yaml, Change the max-pods in the cluster.yml and click on Save
• Cluster is updated successfully.

On master-head commit id: e20f472d4 ui tag: latest2

• Deploy a custom cluster through the API - Upgrade strategy: Drain - False
• When the cluster comes up active
• Do an Edit on the cluster. Select Drain - True (with default values)
• Enable Scheduled CIS scan on the cluster
• Save the changes made
• Do an edit on the cluster
• Drain timeout is seen set to 120
• click on save
• Error is seen: "Timeout" should be between 1 and 10800

[sowmyav27]

Another way to reproduce the issue on an upgraded setup:

• Deploy a cluster in 2.3.6 in k8s 1.17
• Upgrade Rancher to 2.4.5
• Upgrade k8s version to 1.17.9. Save changes made.
• When the cluster comes back active, Edit cluster --> Enable drain, notice that the drain timeout field is "blank"
• Save changes made.
• When the cluster comes to active state, Edit cluster, notice drain timeout is now 0. Save the cluster.
• Error seen on UI: Validation failed in API: rancherKubernetesEngineConfig upgradeStrategy=InvalidFormat 422: nodeDrainInput=InvalidFormat 422: timeout=MinLimitExceeded 422:
• Edit Drain timeout as 120 seconds. And now save changes made. Cluster will get updated successfully.

On master-head - commit id: 9b0dd20b7 - Issue is seen fixed

• Deploy a custom cluster through the API - Upgrade strategy: Drain - False
• When the cluster comes up active
• Do an Edit on the cluster. Select Drain - True (with default values)
• Enable Scheduled CIS scan on the cluster
• Save the changes made
• Do an edit on the cluster
• Drain timeout is seen set to 120
• click on save. Cluster goes into updating state and no error is seen

Upgrade from 2.3.6 to 2.4-head commit id: 2c7dc4ba8 - issue is seen fixed

• Deploy a cluster in 2.3.6 in k8s 1.17
• Upgrade Rancher to 2.4-head
• Upgrade k8s version to 1.17.9. Save changes made.
• When the cluster comes back active, Edit cluster --> Enable drain, notice that the drain timeout field has value 120
• Save changes made.
• When the cluster comes to active state, Edit cluster, notice drain timeout is now 120. Save the cluster.
• No error seen