New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
retpoline+IBPB is NOT? needed to mitigate the vulnerability on some CPUs #181
Comments
I'd be really interested to know whether this is the case. I've got a lot of machines that are frustraightingly just outside the window of what Intel has decided they want to provide microcode updates for. (a LOT of institutions haven't bought all new computers in the last five years) I'm no kernel dev, but I wonder if anyone's looked at an equivalent of IBPB for older machines that Intel has recently decided never touch. If an attacker can stuff the branch predition buffer by repeatedly jumping, why can't we? I understand it'd be slow as all hell, but might provide a fallback for those left in the lurch. |
Oracle has a useful blog post on this subject: The author links to this Intel document: In this document, I see the following commentary: "5.1 Processor Models - Retpoline is known to be an effective branch target injection (Spectre variant 2) mitigation on Intel processors belonging to family 6 (enumerated by the CPUID instruction) that do not have support for enhanced IBRS. On processors that support enhanced IBRS, it should be used for mitigation instead of, or in combination with, retpoline." "Retpoline is a hybrid approach since it requires updated microcode to make the speculation hardware behavior more predictable on some processor models." I also understand that some peripheral devices include (x86) firmware that may be executed by the kernel which might use branching that is vulnerable to speculative attack. I am assuming that overrides the following document text: "Kernels which were built with a compiler that does support retpoline will indicate that |
Yeah, I vaugely recall reading that some of the microcode-supplied features were being used to protect against speculation inside of calls to firmware. (when it comes to spectre/meltdown, good luck getting support/patches for that stuff) Speaking of CPUs that won't receive IBRS or IBPB microcode updates and may not need them, I have an AMD Phenom-based machine which is reported to have "Full AMD Retpoline" yet is still "vulnerable" because it lacks IBPB. EDIT: I'm a dumb*** for not googling it. AMD has released IBPB microcode for at least some processors. Installing 'amd64-microcode' (this is a fresh install to get the new kernel patches) doesn't seem to change which version of the microcode I've got, though It's possible it hasn't trickled down to me yet. I'd be interested if anyone's looked at a slow software IBRS or IBPB, but all signs I've seen point to it being a microcode-only thing. |
I wasn't able to mentally parse that sentence, @Matthew-Bradley I've tested the intel microcode update which is available for one of my systems which is NOT on the list of CPUs: "this one absolutely requires IBPB because retpoline-only is ineffective" and the performance hit (increased latency) was measurable. If I get the time I'll shuffle around and cook the data into a human-readable report which isn't nearly as opaque as the raw data. (edit) Also, as @orachas mentioned - yeah binary blob firmware is hard to trust, and I'm uncertain to what extent BTI (spectre v2) is able to peer outside its process to look inside the memory space of things which were built using a retpoline-type compiler. ::ponder:: |
To be specific, the 0.35 version of the script tested this file:
In 0.35, if the word "Full" was found here, then the script reported "Not Vulnerable" with the following test:
This reporting was changed on April 4th with a commit titled "feat: rework Spectre V2 mitigations detection w/ latest vanilla & Red Hat 7 kernels" where the script began demanding IBPB. I don't see any commentary on why this was done. I would like to know more detail on the reasons for this change, and who was involved. On another subject, the Oracle blog post in my last comment also mentions 3rd-party kernel modules. These must also be prepared with a retpoline-aware compiler - otherwise they can be successfully targeted in a speculative attack, and a dmesg fragment is provided that specifically warns of this condition. Non-retpoline modules join device x86-firmware that the kernel executes as vectors to reintroduce the vulnerability. I am not convinced that "Full AMD retpoline" is now once again completely vulnerable, and it appears to me that all the code in my distro's kernel package is safe. I would like to know how to identify unsafe peripherals, activities, and what countermeasures are possible when IBPB is not available. |
To rephrase: As far as I know, nobody has looked at an alternative (even a horifyingly slow one) to the requirement that you have IBPB in microcode to go along with retpoline (both for Intel and AMD). I would have thought it would be possible but nothing specifically points to it. (if an attacker can do repeated jumps to poison the buffer, then instead of clearing it on context-switch with IBPB I would have thought we could do the same jumping around to un-poison it) To the best of my understanding, being a non-expert in this stuff: All of this comes with the asterisk though that this is just what I've been able to understand from what I've read on the subject. If a kernel developer who wrote the protections wants to chime in, I'd be really grateful. EDIT: I imagine kernel modules probably need retpoline too, not just the kernel itself. |
@orachas - Ah! Good catch. de02dad is a major rewrite, and this may just be a regression / mistake (unintended consequence from the logic changes). Edit: @Matthew-Bradley yes. I mentioned the recompilation (and/or update all premade binary packages, as appropriate) in the original post. I feel It's worth noting: retpoline was an early fix, and predates the development of IBPB-type microcode, which was then discussed on LKML (among other places) and various people weighed in using strong language (some calling it garbage, or worse) As I understand, one of the reasons IBPB microcode was released is because of legacy software (which wouldn't be recompiled in a timely manner) running in enterprise situations where shutting down production servers would disrupt mission critical infrastructure. |
For reference, some of the best descriptions I've found: EDIT: Overall this doesn't look like a mistake to me, and I'd want confirmation from someone who's really familiar with how IBPB is being used before deciding that some oder CPUs are safe. |
@Matthew-Bradley yeah, that stuff looks right to me. Those details are correct.
I was careful when I file this issue to use qualifying language: "on some CPUs", as skylake is on the list which is known to require IBPB. Addendum: One of the things mentioned in the february LKML thread (KVM-related patch) is guests being attacked by a different guest on the same host: https://news.ycombinator.com/item?id=16127800 It's been discussed by various people that guests can protect themselves with retpoline-compiled userland code & guest kernel in the absence of IBPB. (assuming not a skylake or other affected CPU which needs IBPB). I guess that makes sense to me. |
Thanks for all your research. I added IBPB as a requisite to "NOT VULNERABLE" on variant 2 after seeing how Red Hat changed the default protection in their latest kernels. In the vanilla kernel, IBPB is now also enabled in any case when it's available, including when the kernel has been compiled with retpoline (vanilla kernel actually doesn't support IBRS appart from the specific IBRS_FW for the firmware code): So it would seem that the code implies that IBPB is not useless with retpoline, quite the contrary. However it does mean that a lot of CPUs that don't yet (or never will) have a microcode update will get VULNERABLE with no way to fix it. To avoid freaking out people, I could create a I'd like to get this sorted before releasing v0.37. |
@kuzetsa Are you sure you're not confusing IBRS and IBPB? My understanding is that skylake needs IBRS, because it will occasionally fall back to the poisoned buffer on returns, so retpoline is insufficient by itself. Aparently "retpoline underflow" now fixes this and IBRS is no longer necessarily needed on skylake. EDIT: upon reading the code linked above, it seems IBRS is also used when dealing with firmware and such. |
How about the following warning: Retpoline WITHOUT IBPB - modules, firmware, and BIOS/ACPI/UEFI may introduce vulnerable code I read a quick discussion of [when] the OS executes various types of BIOS code here: |
@Matthew-Bradley You linked https://lkml.org/lkml/2018/2/1/638 and it mentions IBPB (no mention of IBRS) - I'm a bit confused by your question, honestly. @speed47 - yeah, IBPB covers things which won't otherwise be able to protect themselves using cheaper (less overhead) retpoline-based compilation. Like - maybe as an alternative to the more generic: "vulnerable" message, when "full retpoline" is detected in the kernel but IBPB is NOT detected, a brief notice that sensitive userspace processes need retpoline mitgations: "unless IBPB is enabled, older binaries built without a retpoline-type compiler are vulnerable to BTI (CVE-2017-5715) attacks" @orachas - yeah, binary blob firmware which can't be audited for security is never trustworthy IMO :( |
That link was mostly in reference to IBPB Reading over the code @speed47 linked, it seems IBRS is also used to protect firmware (X86_FEATURE_USE_IBRS_FW). I don't know what's going on with AMD in that case though, since I've only heard about IBPB from them. |
AMD plans to implement IBRS for at least some CPUs, at they have published information about how to detect an IBRS-capable AMD processor (a different CPUID bit than Intel). This CPUID bit is already checked by the script, but I haven't seen any AMD CPU with it yet. |
@Matthew-Bradley IBRS in particular has performance hits which torvalds described as a "vile hack", and my own hot take on the intel microcode this agrees with that kind of criticism: Compared to compiler-level changes, one-size-fits-all microcode "fixes" are a blunt instrument which breaks optimizations for the sake of security. (and I have some firsthand test data on IBPB. it's quite bad for my usual workloads. It's possible that a future version of microcode-type spectre mitigation(s) will be computationally LESS expensive (and able to be managed with a fine-grained approach by the kernel) compared to compiler-level (still optimized) retpoline. Seems more likely that a speedup-type microcode will be possible if it's selectively managed by the "IL to assembly" stage in optimizing compilers (or whichever other level that would need to happen at). I think "the good stuff" isn't coming to my sandybridge potato though. My only choices are retpoline or major peformance hits. Edit: Oh!!! intel CPUs older than skylake do have IBRS support, but at or around the skylake models (42nd generation i7 or whatever) is when the relevant hardware "under the hood" to make it less expensive existed. Older CPUs have a worse performance hit with the microcode updates. (oops) ^ getting a bit offtopic. thanks for looking at the nuance & considering a rework the IBPB-related notice(s), @speed47 |
Intel originally said that they would only produce new microcode for CPUs manufactured within the last 5 years. They have since widened that net, and some older CPUs will get firmware when convenient, so IBPB might appear on older systems. I have Core 2 Duos and Athlons that will never get it. |
@orachas The article you've linked would seem to support the opposite: That intel origionally intended to provide microcode updates furthur back, but have now issued a "stop" for anything older than five years. https://rcpmag.com/blogs/scott-bekker/2018/03/intel-meltdown-spectre-updates-done.aspx |
I just googled "microcode stopped" and found your hit: http://www.hexus.net/tech/news/cpu/116885-intel-halts-microcode-patch-development-230-cpus/ "We've now completed release of microcode updates for Intel microprocessor products launched in the last 9+ years that required protection against the side-channel vulnerabilities discovered by Google Project Zero." |
Ah, yes. I missed that bit added to the bottom from a separate source. Not sure how much I trust it though. Googling around, a bunch of places have the more complete quote: It seems to me there won't be many more patches for older stuff though, either way. |
Just pushed the IBPB is not required to get "not vulnerable" for Variant 2, however a warning will be shown below the status. If |
the spirit of #181 - ["... IBPB is NOT? needed..."]: I feel that the microcode "fix" (rather, its current blunt instrument incarnation) from intel is contentious, and it nearly comes across as [speaking for myself / my views]
... 0919f5c contains several ( Semi-unrelatedly, the wording for --disclaimer is great:
|
@kuzetsa All of the microcode patches are quite a bit more mature by this point, with Intel no longer choosing to patch any more chips. While early patches were plauged with issues, they're now probably as finalized as they're going to be considering how difficult microcode patches can be. They're also important enough to coverage that AMD released an update this week (4/10/18): https://www.amd.com/en/corporate/security-updates As @speed47 noted, IBPB is getting enabled in the vanilla kernel even with retpoline in place. If it's getting used by the kernel for protection then I definitely think that it's something to allways warn about when it's not there. That's why I agree with @speed47 's above reasoning to allways include a warning, but only change the toplevel status when "paranoid". I think that's a good compromise, similar to informing when the performance impact of PTI will be not be reduced. While I'd like to see the availability of IBRS and IBPB geting noted because of all their various uses (like mitigating firmware speculation or with KVM) those are things that may or may not fall under "core of the system" protection. |
One other thing to look into is an article I just found going through the various AMD mitigation strategies: What might be relavent to this this thread is that it's stated that AMD has mitigations you can implement that are software+hardware OR software only, though I've not seen any speak of software only naywhere else, and the link to their release in my previous comment would seem to contradict that, considering they speak exclusively of a combined approach. The witepaper they give is probably worth a look. |
@Matthew-Bradley nah, that's not true - they only discontinued further microcode development for very old CPUs. Lots of the modern CPUs are still getting microcode updates. |
One thing that I want to point out in closing - the Red Hat retpoline implementation is not extensive or thorough. It will not report "Full generic retpoline" or "Full AMD retpoline" on any CPU (it will report "Full retpoline" when new microcode is available). I hope the scanner script refrains from putting too much faith in what I see to be code of questionable quality. The (very brief) port was entirely performed by Josh Poimboeuf:
Oracle's current implementation in their UEKr4 appears to be a much more extensive effort, involving more people. It does report the "Full" status where Red Hat will not, and it certainly appears from this changelog to be far more robust:
|
@speed47 I came across this which would seem to shed some light on some above things: https://lkml.org/lkml/2018/3/20/295 It seems to me that there is a little bit of work being done on "what do you do if you don't have updated microcode" but reading all the messages it seems to me it's from the standpoint of "IBPB/IBRS are slow" Still, you might draw your own conclusions. At any rate, it's something I found that seems interesting. |
If/when I have control over a critical application, I will sink it into a chroot() if possible, either explicitly or through systemd-nspawn. This is obviously a challenge with an Oracle database instance, but it can be done. I wish the browsers would do it. POSIX chroot() has been with us since Bell Labs version 7. Everyone whines that it's too hard, which should stop. I wonder what Tails does. |
Looking WAAAY back: https://lkml.org/lkml/2018/1/22/598 So now I wonder how true this still is (that IBPB is needed for complete retpoline), especially if you've got a bunch of userspace stuff compiled with retpoline. |
BTI (CVE-2017-5715) mitigation: retpoline alone, without IBPB might be acceptable
... at least on some CPUs
in particular:$ cat /sys/devices/system/cpu/vulnerabilities/spectre_v2
... for some CPUs, acceptable mitigation (with minimal overhead penalties) can be had from an updated compiler, kernel, and system-wide userland rebuild to enable retpoline.
(and/or update all premade binary packages, as appropriate)
IBPB is painfully slow, and in cases where intel's testing hasn't shown it to be required, IBPB is a senseless penalty for little (if any) gains compared to a retpoline-only BTI mitigation.
This is a secondary source (not directly from intel):
https://github.com/marcan/speculation-bugs/blob/master/README.md#cpu-vendor-response
I tried tracking down the specific quote where intel made the recommendation to use retpoline without IBPB for performance reasons, but gave up after 30 minutes of digging.
Update:
Sorry. I didn't explicitly say which version of retpoline:
I was only thinking about upstream / kernel.org support (for example, on 4.15.x series kernels) and completely forgot that several distros have a patchset for older kernel branches. That flavor of retpoline-type mitigation is primarily developed by the distro's kernel maintainer(s).
The text was updated successfully, but these errors were encountered: