AKS Guidance for CVEs 2019-11477, 2019-11478, 2019-11479 (SACK)

[jnoller]

Note: This issue will be updated as new information is available

On Monday June 17th security researchers announced 3 critical security issues impacting the Linux kernel. These are:

• CVE-2019-11477: SACK Panic
• CVE-2019-11478: SACK Slowness
• CVE-2019-11479: Excess Resource Consumption Due to Low MSS Values

These CVEs have been patched by all major linux vendors. This means your clusters must be updated to mitigate these security issues.

All Azure Kubernetes Service (AKS) customers running unpatched kernels are potentially vulnerable to these security issues. We recommend all customers verify the running kernel and take action if required to apply these updates.

Canonical issued updated, patched kernels and these updated kernels were made available to the AKS customer base as of 2019-06-19.

AKS clusters using the default configuration were patched as of the automatic update on the evening of 2019-06-20 12:00, however users must reboot their clusters for the path to take effect.

Customers with clusters created before 2019-06-28 (Friday, June 28 2019) should confirm their nodes are updated.

Confirming if you are impacted:

Execute the command kubectl get nodes -o wide - in the output you will see the running kernel under the KERNEL-VERSION header, for example:

KERNEL-VERSION
4.15.0-1049-azure
4.15.0-1049-azure

If you see a kernel version lower than 4.15.0-1049 you will need to take action.

New clusters created on or before 2019-06-28 (Friday, June 28 2019) are potentially running impacted kernels, all users are asked to verify the kernel version. If your cluster was created after Friday June 28th, no action is required.

New AKS clusters

All clusters created after 2019-06-28 should have the patched kernel, users are encouraged to confirm and if needed manually reboot for the update to take effect.

Applying the updates

As noted, the updated kernel was deployed to all customer clusters. This means if your cluster has been running for at least 24 hours the kernel should be installed on your worker nodes. In order for the patches to take effect, you will need to reboot your nodes.

When checking your clusters after a reboot or otherwise verify that the kernels are not installed, see manual installation below.

How to handle node reboots

AKS recommends using the kured utility to manage OS updates to your worker nodes.

Kured will detect when a reboot is required (due to OS/security updates) and manage the reboot of each worker in your cluster.

Manual mitigation

If customers need to manually install the kernel, they are advised to do the following:

• Log into each node within your AKS cluster

• Execute the commands below on all cluster nodes:

   apt-get update
   apt-get install linux-azure
  

• Finally, you will need to reboot each node in the cluster. AKS does not reboot customer nodes due to possible workload disruption. See How to handle node reboots above.

Patched Kernels

• The CVEs were patched in the 4.15.0-1047 kernel
  • See: https://usn.ubuntu.com/4017-1/
  • Ubuntu SACKPanic Wiki

Hosted AKS components (API server)

AKS Engineering has patched all systems hosting customer cluster components.

[przemolb]

Then how to apply all the updates to AKS clusters deployed with VMSS ?

[victornoel]

Then how to apply all the updates to AKS clusters deployed with VMSS ?

@jnoller I second this question. I think that upgrading the whole cluster just to reboot the nodes is not acceptable.

[victornoel]

For the record, here is the discussion where it is said that kured can't be used with VMSS node pools: MicrosoftDocs/azure-docs#33657

[immunochomik]

@przemolb here is how to do it for a vmss deployment.
To get the new image data, (that is if you are using images provides by microsoft-aks), the provider will be probably different in your case.

az vm image list --all -p microsoft-aks -l northeurope

az vmss update \
  --resource-group <resourceGroup> \
  --name <vmssName> \
  --set virtualMachineProfile.storageProfile.imageReference.version=2019.07.01 \
  --set virtualMachineProfile.storageProfile.imageReference.sku=aks-ubuntu-1604-201907

One problem here is that it will update all the nodes at once :) Great F*** idea !!!!

[przemolb]

@immunochomik
asking to be sure: does the az vmss update command applies to AKS VMSS automatically ?

[przemolb]

And if it will update all the nodes at once it is not acceptable ...

[victornoel]

yes, the problem is that in the context of AKS, there are some kubernetes specific things that must happen for the reboot of a node to be correct (such as cordoning the node and moving the pods out of the node before restarting it). I'm not sure it is a good idea to directly interact with the VMSS apis…

[victornoel]

See MicrosoftDocs/azure-docs#33657 (comment) more precisely.

[immunochomik]

@victornoel yes but if you do not update VMSS then on scaling events the new nodes will start up with the old image. It is not acceptable as @przemolb put it. It seems to be possible to set up rolling updates for vmss, but it is not trivial in kubernetes context, as we need to set up application extension for health checks and then somehow provide an http endpoint on every node that will respond with 200 (I assume) https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-health-extension#deploy-the-application-health-extension

[yossaa]

az vmss update
--resource-group
--name
--set virtualMachineProfile.storageProfile.imageReference.version=2019.07.01
--set virtualMachineProfile.storageProfile.imageReference.sku=aks-ubuntu-1604-201907

One problem here is that it will update all the nodes at once :) Great F*** idea !!!!

It shouldn't when you have manual upgrade mode:

Manual - In this mode, when you update the scale set model, nothing happens to existing VMs.

Then you have to re-image all your worker nodes and everything is upgraded. Kured won't help in this case, but it's doable with custom scripts and it should work from the VMSS perspective. I'm trying to get an answer for questions:

• will it work from AKS perspective
• how to find proper VM for my AKS cluster. az vm image list has no information about kubernetes version. And it should probably match my cluster.

[palma21]

I believe this and all the linked threads have generated some misunderstanding.

To clarify the recommended method to upgrade in cases like this or not is the one documented here:
https://docs.microsoft.com/en-us/azure/aks/node-updates-kured
If you have any issues please do report.

This should work for both VMSS or non-VMSS cases, regardeless.
I jumped to the linked threads to clarify as well.

[palma21]

Closing

[victornoel]

@palma21 as discussed in MicrosoftDocs/azure-docs#33657 (comment), it may be a bit premature to think this question is clarified.

[palma21]

Feel free to join the discussion there, I already replied there too. What I stated above remains true.
We rely on that method ourselves, so if there are specific issues you're facing with it, we're happy to look into them.