Microcode update from AMD available - possible to include in coreboot build?

[fhloston]

No description provided.

[fhloston]

There is updated microcode from AMD and it can be found in the wild:

https://github.com/platomav/CPUMicrocodes/blob/master/AMD/cpu00730F01_ver07030106_2018-02-09_88EDFAA0.bin

I have successfully loaded that microcode in my debian installation.

Can you include that into the coreboot builds, please?

[pietrushnic]

@fhloston thanks for this information. We will try to address that in next release although I have to consult if using microcode published "in the wild" would be ok. Have you got any pointers, if anyone tried it and can confirm that issues were mitigated?

[fhloston]

@pietrushnic I tried loading it during initrd and in a live system myself and it does contain some microcode enhancements which weren't there before:

• Indirect Branch Prediction Barrier (IBPB)
  • PRED_CMD MSR is available: YES
  • CPU indicates IBPB capability: YES (IBPB_SUPPORT feature bit)

Can't you just request the microcode from AMD?

[fhloston]

I should add, the raw microcode needs to be formatted properly into microcode_amd_fam16h.bin

https://github.com/groeck/amd-ucodegen

(link fixed)

[gretel]

@fhloston could you possibly post the commands .. script you are using to load the firmware at runtime? regards

[fhloston]

@gretel easy:

amd-ucodegen cpu00730F01_ver07030106_2018-02-09_88EDFAA0.bin -o microcode_amd_fam16h.bin

backup old file in /lib/firmware/amd-ucode (linux) or /usr/local/share/cpucontrol (freebsd)
put new file there
echo 1> /sys/devices/system/cpu/microcode/reload (linux)
cpucontrol -v -u /dev/cpuctl[0-3] (freebsd)

you might have to install devcpu-data on freebsd first, and activate the respective service

[gretel]

@fhloston thanks! for freebsd this is what i did:

$ git clone --depth 1 https://github.com/groeck/amd-ucodegen
$ cd amd-ucodegen
$ make
$ curl -O https://github.com/platomav/CPUMicrocodes/blob/master/AMD/cpu00730F01_ver07030106_2018-02-09_88EDFAA0.bin
$ ./amd-ucodegen cpu00730F01_ver07030106_2018-02-09_88EDFAA0.bin -o microcode_amd_fam16h.bin

[pietrushnic]

@fhloston decision was made to not include unofficial microcode binaries in official PC Engines firmware releases. I believe we can prepare unofficial firmware release based on v4.6.9 and/or v4.0.17 if there is interest in that.

Meanwhile, we are in contact with AMD and trying to obtain official microcode release.

[gretel]

@pietrushnic thanks - i am interested.

[fhloston]

@pietrushnic thanks for the update! also interested!

[pietrushnic]

@miczyg1 let's prepare unofficial releases version with new microcode. @fhloston @gretel I think, because of long weekend in Poland, it would be ready before end of next week. Sorry for delay.

[gretel]

@pietrushnic cool.

[miczyg1]

@fhloston we have been able to include microcode updates in coreboot, yet we still need some information about new microcode features You listed:

Indirect Branch Prediction Barrier (IBPB)

• PRED_CMD MSR is available: YES
• CPU indicates IBPB capability: YES (IBPB_SUPPORT feature bit)

Where did You get these information from? Is there any possibility to verify these features?

[fhloston]

@miczyg1 I loaded the updated microcode on a Debian Stretch Installation and used spectre-meltdown-checker to verify changes. It is also packaged in stretch-backports.

https://github.com/speed47/spectre-meltdown-checker

[pietrushnic]

@miczyg1 please publish here links to custom binaries tomorrow EOD.

[miczyg1]

@fhloston @gretel sorry for the delay. Binaries are available here. Keep in mind that the microcode comes from wild source. We will not take any responsibility for any damage these binaries will cause to Your boards. We are in contact with AMD and will try to obtain official microcode patch.

[gretel]

@miczyg1 thanks for getting back to us! going to be back from travels on friday and report back.

[nicklowe]

The latest AMD microcode/ucode is available from a reputable source:

https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amd-ucode

https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/log/amd-ucode

[nicklowe]

Could this be built in to 4.6.10?

[miczyg1]

@nicklowe unfortunately linux demands microcode in special container designed for kernel. We could use only raw binaries delivered by AMD. Second thing is we finished development cycle for v4.6.10 last Friday, the binaries should be available soon. Depending on result of our negotiations with AMD we may obtain the raw microcode binary, however we cannot assure the microcode will be included in next release.

[gretel]

still didnt get a chance to uprade mine :(

[nicklowe]

Couldn't you use the appropriate file from linux-firmware.git and use https://github.com/platomav/MCExtractor to strip the container?

[fhloston]

guys, what is the issue with the container?

@gretel posted on April 24th how that container is built from raw microcode.

[nicklowe]

I flashed the APU2 4.6.9 ucode image supplied by @miczyg1 and observed the following:

root@OpenWrt:~# dmesg | grep microcode
[ 1.541701] microcode: CPU0: patch_level=0x07030105
[ 1.558224] microcode: CPU1: patch_level=0x07030105
[ 1.570174] microcode: CPU2: patch_level=0x07030105
[ 1.575078] microcode: CPU3: patch_level=0x07030105
[ 1.580070] microcode: Microcode Update Driver: v2.2.

I don't think this is working currently therefore.

[miczyg1]

@nicklowe it may be that kernel is overriding the microcode patchlevel. I am 100% sure that MSR register responsible for storing current patch level reported 0x07030106 in coreboot. I would have to investigate it.

[nicklowe]

Hmm, well, OpenWRT's amd64-microcode package is not installed so there would be nothing to downgrade it with.

[paravoid]

That git log from linux-firmware above shows commits back in May 2018 with microcode updates, but if you look closer these were just for families 15h/17h. The log for 16h (which the APU2 uses) is:
https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/log/amd-ucode/microcode_amd_fam16h.bin
…and that was last updated in November 2014.

@miczyg1 if you're in contact with AMD, it might be worth to ask them to also update linux-firmware like they've done with the other families and/or get the person who made those commits there (who seems to have an @amd.com email address :) in the loop. Thanks for your responsiveness here and your efforts in general regardless!

[miczyg1]

@kolargol the microcode update has been changed since it did not patch all cores previously. Current microcode update procedure works well for all cores. The guide is available in https://github.com/pcengines/apu2-documentation/blob/master/docs/microcode_patching.md

Please do not follow @Firefishy guide since it is wrong way to do that. There is a special Kconfig option under microcode inclusion menu:
Add microcode patch for AMD fam16h (EXPERIMENTAL). It will correctly include the microcode binary into coreboot. Any other method will patch only 1 core and lead to platform instability.

[Firefishy]

@miczyg1 Try follow https://github.com/pcengines/apu2-documentation/blob/master/docs/microcode_patching.md with v4.9.0.2. The Add microcode patch for AMD fam16h (EXPERIMENTAL) option does not display because Include CPU microcode in CBFS is hidden because SUPPORT_CPU_UCODE_IN_CBFS is set to false.

[kolargol]

@miczyg1 the menu option you mention vanished from v4.9.0.2 and @Firefishy seems right. I am building firmware right now and will let you know if all cores are patched.

[miczyg1]

Something may have changed in upstream repository and probably we did not notice. I will check that.

[kolargol]

I confirm correct microcode loaded after @Firefishy suggested fix:

FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,IBPB,XSAVEOPT

on all 4 cores

I have pushed bin if someone want to test: https://github.com/kolargol/apu2_firmware

[miczyg1]

Microcode inclusion Kconfig settings changed upstream, which led to unavailability to patch the microcode for apu. Fix will be included in v4.9.0.3: pcengines/coreboot#271

[doktornotor]

I have pushed bin if someone want to test: https://github.com/kolargol/apu2_firmware

It works... On FreeBSD, I get this additional line in CPU features:

AMD Extended Feature Extensions ID EBX=0x1000

and

# x86info -a | grep "Microcode patch level"
Microcode patch level: 0x7030106

[miczyg1]

If anybody wants to build fresh v4.9.0.3 with microcode update, it is now possible. Fix for menuconfig has been applied.

https://pcengines.github.io

[kolargol]

yes i confirm it works - i have uploaded bin's for testing. You can close this issue.

[soder10]

What is the drawback or consequence if the microcode is not updated via the BIOS (via coreboot binary), but rather updated via the OS / kernel during boot or after boot?

[miczyg1]

@soder10 it depends on Your threat model. The earlier the microcode is updated the better. If You are afraid of attacks during boot time, then the consequences may be high. Let's say somebody has installed malicious service on Your system which executes before the microcode updater does its job. Then it could use the vulnerability to read personal data and leak it via network etc.

You may also find my blog post interesting:
https://blog.3mdeb.com/2019/2019-05-29-spectre-and-meltdown-on-apu2/
Since the mitigation highly depends on the kernel mitigations implemented, the microcode seems to not influence much on the security. What is more, microcode enables only 1 security feature, while AMD stated that there are 3 features that can be enabled via microcode update (Mitigation V2-4: https://developer.amd.com/wp-content/resources/Managing-Speculation-on-AMD-Processors.pdf). But there is also the AMD retpoline which is told to mitigate the variant 2 (which microcode update is responsible for). Mitigation based only on the hardware/firmware level won't prevent the attacks.

[soder10]

@miczyg1: well, I am not having any exotic threat model. I am more worried about the difference between early microcode update (before the kernel boots) vs late microcode update (after the kernel has loaded). Others have posted earlier, that the results are not exactly the same in the 2 methods. Because the kernel does not dynamically re-detect for example the CPU flags after the boot, the output of dmesg differs. And god knows what else will behave differently as well.

[miczyg1]

@soder10 the kernel is much more complicated than firmware and this is the problem here.

I also would like to say that we have sent a request to AMD in order to encourage them to publish the newest microcode. Although it has been over a month and processing from AMD side is slow, we constantly push them to make up a decision. When the microcode will be public, we will include it natively in our builds.

[soder10]

@miczyg1: Thanks very much to keep pushing AMD in this topic. Even if its very unlikely they change their minds. It would be the most clean solution to have the microcode update included in the firmware/bios, and no need to implement it into OS-dependent solutions.

What I dont really understand, why their stubborn secrecy and restriction? There are many public instances of their binary blob downloadable on the internet (e.g. freebsd port "devcpu-data" contains a combined intel+amd microcode binary blob).

[miczyg1]

@soder10 on the other side linux-firmware has not the latest microcode present for family 16h. AFAIK FreeBSD grabs the same package as Linux:
https://reviews.freebsd.org/D15523
And the latest microcode there is: https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/commit/amd-ucode/microcode_amd_fam16h.bin?id=8ac569dd3ca3ca685bd47ee86c1eeb6050864db3
from year 2014... This is not the same binary as https://github.com/platomav/CPUMicrocodes/blob/master/AMD/cpu00730F01_ver07030106_2018-02-09_88EDFAA0.bin from 2018. As long as this binary is not officially pushed to Linux/FreeBSD by AMD or to coreboot, we have to wait for the decision.

[gretel]

@miczyg1: well, I am not having any exotic threat model. I am more worried about the difference between early microcode update (before the kernel boots) vs late microcode update (after the kernel has loaded). Others have posted earlier, that the results are not exactly the same in the 2 methods. Because the kernel does not dynamically re-detect for example the CPU flags after the boot, the output of dmesg differs. And god knows what else will behave differently as well.

personally, i would not like to let the firmware update application code. so i dislike letting application code update firmware, regardless of any threat model.

both approaches should render the same result but actually they do not. from my experience i feel something dangerous lurking there.

@soder10 having current microcode being part of the actual firmware would get closer to my requirements.

[nicklowe]

@miczyg1 Pragmatically, as this is still outstanding from the official firmware in 2021, can two firmware files be produced for each release, one including the microcode alongside a disclaimer and explanation, one not including it.

It seems a rather ridiculous and silly situation to still be in. The microcode file itself should already be internally signed by AMD with an integrity check present which would be tested for before the CPU will accept it. An arbitrarily modified microcode should not practically be feasible and it seems, on balance of the probabilities, also infeasible that the files we have have been tampered with given the distribution sources we already have.

[miczyg1]

@nicklowe it is not about being signed by AMD or not being possible to tamper with. These files from the repo have been extracted from firmware images of firmware updates most likely. These microcode files have been distributed to firmware vendors under a certain license which probably does not allow redistribution. As a company that cooperates with AMD, we cannot integrate it in our public builds as there is a risk of violating such a license.

I will try to gain permission to the microcode update file. AMD is doing some moves into open-source, so maybe they will reconsider it. However, the platform is so old that the matter has a very low priority to AMD.

[soder10]

@miczyg1 these AMD SoCs are still not end-of-life, if I can trust this page:
https://www.amd.com/system/files/documents/g-series-soc-product-brief.pdf

"AMD’s commitment to long-term availability and support (5-10 years) maximizes ROI Footnote7"
--> where footnote number7 says:
"5-year, 7-year, and 10-year support offered, depending upon the AMD product. Please contact your AMD representative for more details." --> Well, we dont know how many years support contract PCEngines bought from AMD, its near expiration most probably. But PCEngines still selling APU2-3-4-5-6 even today, so that should not be a dead product with dead support from AMD.

[miczyg1]

@soder10 I did not say end-of-life, but a low priority. By low priority, it means the support is probably just "best effort". Issuing a license or permission on any component involves many legal issues and a huge process even for a small thing like a microcode file. Please understand it is a high cost for a relatively small matter. Also the support means how long the product is being manufactured. Thanks to that PC Engines is still able to sell their apu products today.

[pietrushnic]

@soder10 there are a couple of issues:

1. Low volume and stable business would always be treated by a sales-driven silicon vendor - this is true not only for AMD, as OSF consultants we saw that many times in practice
2. Contract between PC Engines and AMD is just theory, personally never heard about something like this. It is also weird to me that there is a need for any contract if e.g. I would like to sell your products I would obviously want an enforceable long-term availability document, but paid support is rather an additional thing and needed only if I cannot deal with your product myself. In many cases, producers have to dump pricing for good resellers, but if the market forces are on the producer side they start to do weird things e.g. you have to buy an NXP Code Warrior license to create memory initialization code for new Layerscape. The product is definitely not dead, but support is the best effort as @miczyg1 said, why? Mostly because AMD doing well with Ryzen and Epyc, what you can read from media and their financial results. All resources are in new processors nobody cares about old stuff especially inquired by 3mdeb or PC Engines. All requests made by the community to 3mdeb or PC Engines were communicated to AMD, but none were resolved.

Feel free to jump on the 3mdeb vPub event or some conferences where we are available and I would tell more stories about brain damage in the industry we are in.

[nicklowe]

Thanks very much for the explanation!

Surely there is, in practice, close to zero chance that AMD will or might conceivably care about its stable and for a long time available microcode being included in a BIOS image for these to better address a well publicised class of security vulnerabilities?

This seems a far far too hypothetical, abstract and honestly rather incredulous and nebulous concern and worry to have, a storm in a tea cup to use a British expression.

What another company may have technically agreed to or not around the distribution for a well publicised security fix with AMD is surely not directly a PC Engines or 3mdeb direct legal concern? The reality is these other companies have distributed and AMD do not seem in practice to have cared, it is still available and distributed publicly, It is far more likely they have not given it any thought or attention, and have no real concern about it at all.

The bad publicity that could also flow from being seen to restrict distribution of a well publicised security fix is not territory any reasonable company such as AMD would engage in or actually want to go anywhere near. Again, to use a another British expression, they would not touch this situation with a barge pole.

[soder10]

@soder10 there are a couple of issues:

1. Low volume and stable business would always be treated by a sales-driven silicon vendor - this is true not only for AMD, as OSF consultants we saw that many times in practice

2. Contract between PC Engines and AMD is just theory, personally never heard about something like this. It is also weird to me that there is a need for any contract if e.g. I would like to sell your products I would obviously want an enforceable long-term availability document, but paid support is rather an additional thing and needed only if I cannot deal with your product myself. In many cases, producers have to dump pricing for good resellers, but if the market forces are on the producer side they start to do weird things e.g. you have to buy an NXP Code Warrior license to create memory initialization code for new Layerscape. The product is definitely not dead, but support is the best effort as @miczyg1 said, why? Mostly because AMD doing well with Ryzen and Epyc, what you can read from media and their financial results. All resources are in new processors nobody cares about old stuff especially inquired by 3mdeb or PC Engines. All requests made by the community to 3mdeb or PC Engines were communicated to AMD, but none were resolved.

Feel free to jump on the 3mdeb vPub event or some conferences where we are available and I would tell more stories about brain damage in the industry we are in.

Do you have any such event in the near future? Probably virtual one, I cannot travel...

[soder10]

Thanks very much for the explanation!

Surely there is, in practice, close to zero chance that AMD will or might conceivably care about its stable and for a long time available microcode being included in a BIOS image for these to better address a well publicised class of security vulnerabilities?

This seems a far far too hypothetical, abstract and honestly rather incredulous and nebulous concern and worry to have, a storm in a tea cup to use a British expression.

What another company may have technically agreed to or not around the distribution for a well publicised security fix with AMD is surely not directly a PC Engines or 3mdeb direct legal concern? The reality is these other companies have distributed and AMD do not seem in practice to have cared, it is still available and distributed publicly, It is far more likely they have not given it any thought or attention, and have no real concern about it at all.

The bad publicity that could also flow from being seen to restrict distribution of a well publicised security fix is not territory any reasonable company such as AMD would engage in or actually want to go anywhere near. Again, to use a another British expression, they would not touch this situation with a barge pole.

I dont think AMD cares at all. Its sad to see, that the Rzyen product became so popular and hyped by the media. On the other hand, less popular / less known niche products like a lame SoC, gets near 0 treatment from AMD. Thatswhy its no logic behind praising AMD, favoring them in contrast with Intel. These 2 companies work with the exact same principles behind the curtains. Like any country with only 2 political parties: no matter if a political puppet person belongs to the left-wing or the right-wing party. They both are same kind of criminals, who should all sit in prison, life sentenced.

[pietrushnic]

Do you have any such event in the near future? Probably virtual one, I cannot travel...

Definitely a virtual event. review of the last one is here. Please watch our social media. We working on announcing the date of the next event.

[pietrushnic]

Feel invited to an upcoming event, where you can hear more horror stories from 3mdeb: https://twitter.com/3mdeb_com/status/1387017457118875651