Policy for disabling SMT

[bgilbert]

There have been multiple rounds of CPU vulnerabilities (L1TF and MDS) which cannot be completely mitigated without disabling Simultaneous Multi-Threading on affected processors. Disabling SMT reduces system performance and changes the apparent number of processors on the system, which might confuse container orchestrators and the like, so it's difficult to do without warning in existing OSes -- especially on an upgrade of existing nodes. (Container Linux opted to leave SMT enabled but advise users to turn it off.) With Fedora CoreOS, we have the opportunity to choose a different default.

The kernel supports two relevant command-line arguments:

• mitigations=auto,nosmt disables SMT when needed to mitigate a vulnerability.
• nosmt unconditionally disables SMT.

There are also ways to disable SMT from userspace, but they're not connected to the kernel's vulnerability-detection logic. The only way to do conditional enablement is via the karg.

Proposal

Set mitigations=auto,nosmt by default on Fedora CoreOS.

Good parts:

• Better security by default
• Does not incur an unnecessary performance hit on CPUs that don't need it

Bad parts:

• Requires passing a karg, which isn't trivial to change via Ignition (though see Support default kernel arguments ostreedev/ostree#479)
• Causes a performance hit on vulnerable systems running workloads that aren't affected by the vulnerability, such as trusted workloads
• Does not protect systems with unknown SMT vulnerabilities, until those vulnerabilities become known and a kernel update is pushed out
• Updates to address future vulnerabilities can substantially reduce the performance of a node, as well as its apparent number of processors. I think this is a reasonable policy for a new OS, but we'd certainly need to document it.

Regardless of what we choose, we should document the default behavior and the fact that the user can change it.

Alternative defaults

• Unconditionally disable SMT. This provides predictable machine performance over the long term, but also penalizes processor families that have not had any public vulnerabilities.
• Leave SMT enabled. This leaves users exposed to known vulnerabilities, which is unsatisfactory.

[bgilbert]

Implementation merged in coreos/fedora-coreos-config#95. Design doc PR in #184.

[cgwalters]

This is...bold. I think it's probably right but things are going to get interesting if we start doing more things like this because it basically makes saying "I use Fedora" less meaningful (to the degree it is today), as the divergence between FCOS and other editions grows.

Also we're now diverging from RHCOS for that matter which...I have to wrap my head around this. Maybe we push to do this by default in the product too at some later release for new installs?

[bgilbert]

as the divergence between FCOS and other editions grows.

I'm comfortable with that divergence. 😄 The usage model for FCOS is already pretty substantially different from the other editions.

Maybe we push to do this by default in the product too at some later release for new installs?

Works for me! But I don't know that it's necessary. AIUI OpenShift is surfacing the SMT decision further up the stack, whereas FCOS doesn't have that ability.

[cgwalters]

The other aspect to this is I'd like to be able to say that Silverblue is based on FCOS...but for e.g. people compiling software I think this is a huge hit. (Obviously builds also happen on FCOS too, but it's more likely to be multi-tenant)

I wonder if there's been discussion about solving this less crudely in userspace - for example, there's no benefit to disabling SMT sharing between processes in the "same container" (definition of "container" can be up to userspace). Maybe ultimately some logic would need to be pushed to the scheduler though.

[bgilbert]

Silverblue should certainly use whatever default policy makes sense for them. Desktop use cases are an anti-goal for Fedora CoreOS so they shouldn't influence our default policy.

There has been some work on a kernel scheduler that co-schedules mutually trusted processes. From userspace, one could also imagine pinning workloads to specific cores. I don't think there's any tooling ready to go, though.

[dhawal55]

What is a good way to enable SMT from ignition config?

I'm trying to run FCOS on AWS which has deployed protections for MDS for all ec2 instances as mentioned here:
https://aws.amazon.com/security/security-bulletins/AWS-2019-004/

[cgwalters]

#318

[lump]

Forgive me if I do not entirely understand how this works. The following is only my opinion.

From my perspective, the idea of defaulting simultaneous multithreading (SMT) to off because of hardware vulnerabilities only makes sense for raw iron, where an SMT core is an SMT core. It seems to me that it is a bit overcontrolling to force this for a virtual machine, where the hypervisor can present virtual CPUs to the guest any way it wants. Each core is simply a kernel thread on the host. Who knows whether that thread may be scheduled to run on an SMT core under the covers.

It seems like if there is a decent way to detect whether the machine is raw iron or virtualized, then it could be defaulted to off for raw-iron. Defaulting it to off for virtualized machines is not fair, especially because an SMT core may or may not actually be an SMT core. Some virtualization platforms even have mitigations that deal with CPU flaws and bugs under the covers.

[bgilbert]

From my perspective, the idea of defaulting simultaneous multithreading (SMT) to off because of hardware vulnerabilities only makes sense for raw iron, where an SMT core is an SMT core.

@lump mitigations=auto,nosmt gives the kernel permission to disable SMT if it thinks that's necessary for security. We don't decide the kernel's policy, and I'd expect that policy to evolve over time. In general, I'm not sure we'd want to override the decisions of the upstream developers on this topic.

(If the virtualization system is just running separate kernel threads without regard to SMT scheduling, then perhaps it shouldn't be presenting their topology as hardware threads within the same CPU core.)

[dhawal55]

@cgwalters Thank You.
This is how I'm enabling SMT for FCOS on AWS ec2 instance:

- name: enable-smt-firstboot.service
    enabled: true
    contents: |
      [Unit]
      Description=Enable SMT on first boot on Intel CPUs to disable MDS mitigation
      DefaultDependencies=no
      Before=sysinit.target shutdown.target
      Conflicts=shutdown.target
      ConditionFirstBoot=true

      # On the first boot only, enable SMT at runtime if it is disabled.
      # AWS has designed and implemented its infrastructure with protections against these types of bugs, and has also 
      # deployed additional protections for MDS. All EC2 host infrastructure has been updated with these new protections,
      # and no customer action is required at the infrastructure level. https://aws.amazon.com/security/security-bulletins/AWS-2019-004/
      [Service]
      Type=oneshot
      ExecStart=/bin/bash -c 'active="$(cat /sys/devices/system/cpu/smt/active)" && if [[ "$active" != 1 ]] && grep -q "vendor_id.*GenuineIntel" /proc/cpuinfo; then echo "Enabling SMT." && echo on > /sys/devices/system/cpu/smt/control; fi'

      [Install]
      WantedBy=sysinit.target
  - name: enable-smt-subsequentboot.service
    enabled: true
    contents: |
      [Unit]
      Description=Enable SMT on subsequent boots by changing kernel arguments
      ConditionFirstBoot=true

      # On the first boot only, update kernel arguments to not disable smt. This will take effect on subsequent boot
      [Service]
      Type=oneshot
      ExecStart=/usr/bin/rpm-ostree kargs --append "mitigations=auto tsx=auto"

      [Install]
      WantedBy=multi-user.target

[lump]

Please forgive me for standing on my soap box. Please take the following with a grain of salt...

mitigations=auto,nosmt gives the kernel permission to disable SMT if it thinks that's necessary for security. We don't decide the kernel's policy, and I'd expect that policy to evolve over time. In general, I'm not sure we'd want to override the decisions of the upstream developers on this topic.

The need for a kernel command-line argument is to override what is considered the default by the upstream kernel developers. If upstream kernel developers wanted this to be the default, then they would implement it as default without requiring downstream distributions to implement any explicit kernel parameters. If they did implement this as a default, then there would be no purpose for this ticket. It's not fair to pin this decision on upstream kernel developers because they are not the ones who made this decision.

If the kernel's policy evolves, then they will decide to turn it on by default. I surmise that they have not made this sweeping decision by default because there is no decent way to employ the proper heuristics to decide hands-off whether all SMT truly should be disabled. The kernel developers themselves aren't sure, so it remains an explicit parameter to this day.

As far as I can tell, this ticket isn't about the kernel's defaults, nor the upstream kernel's agenda. Instead, this is about the "Policy for disabling SMT" for Fedora CoreOS. In other words, the impetus is about FCOS making the decision to turn on a kernel argument unless explicitly overridden back to the kernel default after the fact. The case for overriding is when a user realizes, "what the crap, my CPU cores aren't all there." and, "oh crap, I pay my IaaS provider through the nose, but FCOS reduced the CPU resources by half by default."

If the virtualization system is just running separate kernel threads without regard to SMT scheduling, then perhaps it shouldn't be presenting their topology as hardware threads within the same CPU core.

Are you sure that all virtualization platforms actually pin SMT-only threads to SMT-only cores? Are you saying that providers are being stupid to present buggy SMT cores to their virtual machines? Shouldn't their decision to present SMT cores to a guest be their prerogative?

I can tell QEMU to present any number of virtual cores and virtual hardware threads to my KVM virtual machine arbitrarily. Do you truly wish to "pin" all responsibility upon human errors such as these because any human can type a command-line argument?

[bgilbert]

Upstream doesn't disable SMT by default because users would be unhappy if a kernel upgrade disabled compute resources that were previously available. However, there's a growing sense that SMT is not great for security, and since Fedora CoreOS is a new operating system, we have the opportunity to enable more secure defaults. At the moment we aren't documenting this well enough, though. We should:

• document more clearly that Fedora CoreOS disables SMT by default on systems with relevant CPU vulnerabilities, and
• document how to change that with a Fedora CoreOS Config.

The latter requires better support for changing kernel arguments (#318).

My point with virtualization systems was: if hypervisors don't want their threads to be treated as SMT threads, they shouldn't expose a topology that claims otherwise. 🙂 It's a side point, though. We're choosing to give the kernel permission to apply modern, secure defaults, and I don't think we should second-guess what the kernel thinks is necessary.