feat (cluster): [day2-ops] node update configuration

[ferantivero]

• remove Kubernetes Reboot Daemon (Kured)
• enable node update channel for K8s version automatic upgrades (node-image)
• enable node os level update channel for OS security NodeImage automatic upgrades
• add maintenance windows for k8s version and os level upgrades
• add some initial guidance to the docs

Tested end to end:

# auto-upgrade is configured for node-only updates (procedure: node image updates)
# manual-upgrade is configured for aks cluster version updates
az aks show -n $AKS_CLUSTER_NAME -g rg-bu0001a0008 --query "autoUpgradeProfile"

# maintenance window for this RI is once a week
az aks maintenanceconfiguration list --cluster-name $AKS_CLUSTER_NAME -g rg-bu0001a0008

# kured is no longer installed.  
kubectl get all -n cluster-baseline-settings

[skabou]

Some explanation: I could not get it to deploy as-is and had to make those two changes in cluster-stamp.bicep

Apparently there's a known bug with the configuration of "SecurityPatch & node-image" so we will need to go with "NodeImage & node-image"

[skabou]

Here's some context on the bug:
https://learn.microsoft.com/en-us/azure/aks/auto-upgrade-node-os-image#node-channel-known-bugs

"Currently, when you set the cluster auto-upgrade channel to node-image, it also automatically sets the node OS auto-upgrade channel to NodeImage. You can't change node OS auto-upgrade channel value if your cluster auto-upgrade channel is node-image. In order to set the node OS auto-upgrade channel value, check the cluster auto-upgrade channel value isn't node-image."

I know a handful of other changes were made to support the preview feature SecurityPatch but those may not be necessary now.

[skabou]

@ferantivero Looking good! Can you add some guidance / an example of a maintenance window for the updates? Thanks

For reference:
https://learn.microsoft.com/en-us/azure/architecture/operator-guides/aks/aks-upgrade-practices#automatic-node-image-upgrades
https://learn.microsoft.com/en-us/azure/aks/planned-maintenance#creating-a-maintenance-window

[ferantivero]

@ferantivero Looking good! Can you add some guidance / an example of a maintenance window for the updates? Thanks

For reference: https://learn.microsoft.com/en-us/azure/architecture/operator-guides/aks/aks-upgrade-practices#automatic-node-image-upgrades https://learn.microsoft.com/en-us/azure/aks/planned-maintenance#creating-a-maintenance-window

sure thing @skabou, we added both k8s and os level maint config windows.

done | from 1c82e86

[skabou]

Approved with some text suggestions

[ferantivero]

Approved with some text suggestions

really appreciate all contribs @skabou, accepted them all.

[skabou]

@ferantivero Really appreciate your work on this!

[skabou]

👍

[skabou]

🎉