Breaking change to upgrade in v0.12.0

[CecileRobertMichon]

Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE

What version of acs-engine?:
v0.12.4

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes

v0.12.0 introduced generic configuration interfaces for Kubernetes cluster deployments for the kubelet, controller-manager, and apiserver run-time components. Some of the previously existing kubernetesConfig properties are no longer recognized by the apimodel parsing, namely:

• hardEvictionThreshold

• nodeStatusUpdateFrequency

• nonMasqueradeCidr

• ctrlMgrNodeMonitorGracePeriod, ctrlMgrPodEvictionTimeout, and ctrlMgrRouteReconciliationPeriod

If you try to upgrade a previously existing version with the new version of acs-engine (v0.12.4 and up), you will run into the following error message:

Workaround:

if you used default values for the aforementioned properties, or your orginal template did not include those properties (meaning default values were used):

1. Open _output/<your_DNS_prefix>/apimodel.json
2. Delete hardEvictionThreshold, nodeStatusUpdateFrequency, nonMasqueradeCidr, ctrlMgrNodeMonitorGracePeriod, ctrlMgrPodEvictionTimeout, and ctrlMgrRouteReconciliationPeriod from kubernetesConfig
3. Re-run your upgrade command. You should see your command succeed.

if you configured custom values for one of more of the properties above:

1. Open _output/<your_DNS_prefix>/apimodel.json
2. Add the property kubeletConfig to kubernetesConfig containing your hardEvictionThreshold, nodeStatusUpdateFrequency, and nonMasqueradeCidr custom properties as follows:

"kubernetesConfig": {
    `<...>
    "kubeletConfig": {
        "--eviction-hard": "memory.available<250Mi,nodefs.available<20%,nodefs.inodesFree<10%",
        "--node-status-update-frequency": "1m",
        "--non-masquerade-cidr": "10.0.0.0/8"
    }
    <...>
}`

4. Add the property controllerManagerConfig to kubernetesConfig containing your ctrlMgrNodeMonitorGracePeriod, ctrlMgrPodEvictionTimeout, and ctrlMgrRouteReconciliationPeriod custom properties as follows:

"kubernetesConfig": {
    <...>
    "controllerManagerConfig": {
          "--node-monitor-grace-period": "40s",
          "--pod-eviction-timeout": "5m0s",
          "--route-reconciliation-period": "10s"
    }
    <...>
}

5. Delete hardEvictionThreshold, nodeStatusUpdateFrequency, nonMasqueradeCidr, ctrlMgrNodeMonitorGracePeriod, ctrlMgrPodEvictionTimeout, and ctrlMgrRouteReconciliationPeriod from kubernetesConfig
6. Re-run your upgrade command. You should see your command succeed.

Your upgraded cluster should have TLS encryption for etcd communication.

Please let us know by commenting on this issue if you run into any additional issues while trying to upgrade a pre-existing acs-engine cluster with v0.12.4

[bq1756]

I have clusters running production workloads. Building new clusters and migrating workloads is a lot of work. Are there future plans to support upgrading clusters built with previous versions of acs-engine?

[mcops]

oh boy. this is an upgrade-path, ops people like us really hate!

I understand those technical upgrades concerning TLS, but as bq1756 already mentioned, we do have productive workloads as well and far away that we have everything well defined via helm charts :-(

All I needed was 1.8.6, that finally the azurefile storage class mount options work properly and now I've to re-create a new cluster...

If those radical changes get "normal", ACS-engine is for me a no-go.

[CecileRobertMichon]

@bq1756 @mcops v12.0.4 fixes backward compatibility for upgrade. Please try upgrading your pre-existing cluster after following the above steps and let me know if you run into any additional issues.

[mcops]

unfortunately i re-deployed 2-days ago a new cluster! Too late...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution. Note that acs-engine is deprecated--see https://github.com/Azure/aks-engine instead.