Support for setting taints per node pool that persist through scale up/down

[jnoller]

AKS recently added support for taints and tolerations. Taints are preserved/copied/applied through all operations (eg upgrade) except for scale up/down operations. This is a tracking issue for the in-progress work.

[jluk]

@jnoller isn't only taints in scope since tolerations are a pod property?

Taints to persist on scale out via node pool should be good though via the June API: https://github.com/Azure/azure-rest-api-specs/blob/master/specification/containerservice/resource-manager/Microsoft.ContainerService/stable/2019-06-01/managedClusters.json#L1418

Understandable if we want to wait until CLI support lands before closing though, just lmk.

[jnoller]

@jluk removed tolerations (c/p error from the parent) - will wait until CLI landing to close.

[tmbgreaves]

Great to see this is imminent -- thank you, a greatly appreciated feature!

Do you know roughly when this is expected to land in the CLI?

[Pizzajongen]

@jluk Creating new node pools with taints works great via the api but i can't seem to update or add taints on my existing node pools, is this a bug or this by design?

[jluk]

@tmbgreaves hopefully this month, but latest update will still be through this issue as we're working through the backlog.

@Pizzajongen great question, this is by design that node taints are a create-time property only. We'll update the documentation to clarify this!

[Benjiiim]

@jluk, would you mind to provide an update on this one? Thanks a lot.

[jluk]

@Benjiiim taints should work for your agentpools, the only missing piece is the preview CLI which we're trying to close down in next 2 weeks. In the meantime though you can use ARM templates to deploy with taints defined, the multiple agent pool document has instructions too.

[jluk]

To provide an update, we have the taints support for the preview CLI in PR below:

Azure/azure-cli-extensions#962

[tmbgreaves]

@jluk A word of appreciation for all your work and in particular for the regular updates. Thank you very much!

[ptemmer]

Likewise here, thank you for the effort!
I've seen the PR was merged into master, however the latest version listed of the aks-preview extension is still 0.4.16, which doesn't include the taints support. As I'm not acquainted at all with the release procedure, I would appreciate if someone could shed some light on this.

[jluk]

@ptemmer @tmbgreaves thanks for kind words, it really is appreciated and happy to hear it's helpful.

As for CLI extension procedure, once the PR is merged we have to cut a release ourselves by doing some small updates so we'll be trying to get that done this week as v0.4.17.

[jluk]

This is the PR that will get us there, upon merge it should cut a new cli extension release enabling you to upgrade to it and get the functionality.

Azure/azure-cli-extensions#974

[tmbgreaves]

@jluk In case the information is of interest/use to you our use case for this is for Jupyterhub instances where we're running a notebook service in education (university sector), needing to scale cluster size up and down in response to classes being in/out of session. We've been working with @leestott to develop our Azure use over the last year with a pilot class starting this week aiming to have all the teaching and student resources hosted on Azure. So hopefully this feature will be going into production use very soon!

[jluk]

Thanks for the context @tmbgreaves that's awesome. Are you using any other preview features by chance?

The CLI preview released so you can upgrade and set taints per node pool from the preview CLI now.

I'll leave this ticket open until it goes GA production ready. We're closing down on it within the next month.

[Benjiiim]

Hi @jluk ! It seems that Azure/azure-cli#10809 did the trick, didn't it?
Shouldn't https://docs.microsoft.com/en-us/azure/aks/use-multiple-node-pools#schedule-pods-using-taints-and-tolerations be updated to add the taint with az aks nodepool add cmd and not with kubectl taint node cmd?
Thanks

[jluk]

Thanks for bumping this @Benjiiim you are correct! I am closing this issue as it is GA and I will open a PR on our docs to use this AKS native feature for setting taints. Thank you!