VMware by Broadcom Missing PK-signed KEK

[jamesaepp]

I write this in my personal capacity and NOT as part of my employment/employer.

The core questions: Does VMware not have PKs for VMs created before ESXi 9? Does Microsoft recommend that VMware customers be installing a Microsoft PK?

I'm very confused by VMware's guidance documented in the below KBs (in particular the second one). I'm sure I can't be the only one.

• https://knowledge.broadcom.com/external/article/423893/secure-boot-certificate-expirations-and.html

• https://knowledge.broadcom.com/external/article/423919/manual-update-of-secure-boot-variables-i.html

KB423893 states:

On virtual machines created on ESXi versions earlier than 9.0, the PK is configured with a NULL signature by default due to previous design considerations. In this case, the platform cannot authorize updates to the KEK, which in turn prevents subsequent DB and DBX updates.

and:

There is no automated resolution available at this time. In coordination with Microsoft, Broadcom Engineering Team is actively working towards implementing an automated solution in a future release to update the Platform Key (PK) on the affected VMs which will facilitate the certificate rollout...

KB423919 describes the process to install both the PK and 2023KEK. What I find very peculiar however is that the KB recommends administrators to install PreSignedObjects/PK/Certificate/WindowsOEMDevicesPK.der as the PK, and not some Broadcom-specific PK.

All these facts together, I launch into questions I'm hoping a combination of ESXi9 users, Microsoft staff, and Broadcom staff can help with:

1. Who maintains the private key to the WindowsOEMDevicesPK certificate?

2. What happens if the private key to the WindowsOEMDevicesPK certificate is compromised? What safeguards are in place to prevent this?

3. What are Microsoft's recommendations for using WindowsOEMDevicesPK as a trust anchor/platform key? When should administrators not install trust in that key?

4. Is this type of arrangement (an OEM without a PK on secureboot-capable devices) ... normal? Best practice? Or does VMware stand alone amongst its peers?

5. What PK is present on ESXi9 VMs? Is it the WindowsOEMDevicesPK cert? If not and it's a VMware-specific PK, can that PK be back-ported to VMs on older ESXi versions?

6. What will the "automated solution" look like and when will administrators get it? Automating via Windows (the guest) should be impossible as the KEK2023 update wouldn't be authenticated (signed) by anything. It could be feasible via an ESXi (8 only) update but there's zero indication what that will look like.

7. Whatever this "automated solution" is, will administrators need to take some manual action on their VMs to apply the updates beyond AvailableUpdates (Windows land) or fwupd (Linux land)?

8. If VMware takes the approach of back-porting a PK to ESXi8 (or earlier) systems, will they also sign Microsoft's 2023 KEK and be uploaded to this repository like any other OEM?

---

Tagging a couple people who I hope might have the expertise to add commentary:

• @haz-ard-9 - who generously created a PowerShell script to help administrators with installation of the WindowsOEMDevicesPK PK.

• @plankers - who needs no introduction.

[haz-ard-9]

Chiming in as the author of the PowerShell script referenced above, speaking from a administrator perspective with some additional context from digging into Broadcom's documentation, Microsoft's documentation/guidance, and this repo. Here are some of the questions that I can provide answers or context to.

---

3. What are Microsoft's recommendations for using WindowsOEMDevicesPK as a trust anchor/platform key? When should administrators not install trust in that key?

4. Is this type of arrangement (an OEM without a PK on secureboot-capable devices) ... normal? Best practice? Or does VMware stand alone amongst its peers?

Looking at the PostSignedObjects folder, the normal OEM flow is pretty clear. OEMs maintain their own PK, sign Microsoft's KEK with it, and submit that PK-signed KEK to this repo where it gets distributed via Windows Update. Adlink did exactly this 5 days ago. That's how the trust chain is supposed to work. The OEM owns the PK, Microsoft's KEK sits below it, and Windows Update handles the rest.

Broadcom recommending administrators install the WindowsOEMDevicesPK as the PK on ESXi VMs is the unusual part here. That cert is Microsoft's own PK for devices where Microsoft is the platform owner, not a generic fallback for hypervisors. It works for the immediate certificate expiry problem but it shifts the Secure Boot root of trust from Broadcom to Microsoft, which isn't the intended arrangement. Administrators following KB 423919 should understand that they're temporarily putting Microsoft in the role of platform owner for their VMs, with the expectation that Broadcom will eventually provide a proper replacement.

---

6. What will the "automated solution" look like and when will administrators get it? Automating via Windows (the guest) should be impossible as the KEK2023 update wouldn't be authenticated (signed) by anything. It could be feasible via an ESXi (8 only) update but there's zero indication what that will look like.

Based on my understanding of the situation from Broadcom's guidance, the NVRAM rename approach that my script originally used is no longer recommended by Broadcom. Their current documented guidance in KB 423919 describes a manual process involving booting the VM into UEFI SetupMode to enroll the PK and KEK directly, without renaming the NVRAM file. From community testing of both approaches, the manual SetupMode path that Broadcom documents does work and avoids the NVRAM rename entirely for VMs where the KEK is already present in NVRAM.

Automating a fix from the Windows guest side doesn't seem feasible for the reason you described. There's nothing to authenticate the KEK update against without a valid PK first. An ESXi host update that writes the correct PK into existing VM NVRAM seems like the only viable automated path, which is presumably what Broadcom means by "future release."

---

7. Whatever this "automated solution" is, will administrators need to take some manual action on their VMs to apply the updates beyond AvailableUpdates (Windows land) or fwupd (Linux land)?

Once the PK and KEK are in place, the Windows-side AvailableUpdates/scheduled task path has handled everything automatically across Windows Server 2019, 2022, and 2025 VMs in the community's and my own testing, without any additional manual steps beyond triggering the task once.

---

8. If VMware takes the approach of back-porting a PK to ESXi8 (or earlier) systems, will they also sign Microsoft's 2023 KEK and be uploaded to this repository like any other OEM?

Based on my understanding of how this repo works and Broadcom's stated direction, the path forward seems straightforward given the existing OEM precedent here. They would maintain their own VMware specific PK, sign Microsoft's 2023 KEK with it, and submit that to this repo for distribution via Windows Update. That would resolve the trust chain concern entirely and align with how every other OEM handles this. Whether they backport a mechanism to install that PK on existing ESXi 8.x VMs alongside the 9.0 fix is the open question.

[Flickdm]

For context I'm a UEFI firmware engineer - but I likely don't have all the context / information being asked for.

Broadcom recommending administrators install the WindowsOEMDevicesPK as the PK on ESXi VMs is the unusual part here. That cert is Microsoft's own PK for devices where Microsoft is the platform owner, not a generic fallback for hypervisors. It works for the immediate certificate expiry problem but it shifts the Secure Boot root of trust from Broadcom to Microsoft, which isn't the intended arrangement. Administrators following KB 423919 should understand that they're temporarily putting Microsoft in the role of platform owner for their VMs, with the expectation that Broadcom will eventually provide a proper replacement.

So for some context here, We [Microsoft] actually have been proposing this. PKI / Certificate Management is expensive and requires a lot of technical experience and we've offered to let OEMs who wish to leave it to Microsoft to simply use our public certificates. Its a choice they can make but not a requirement. Without getting too far into it (simply because I don't know that much) we have entire teams dedicated to managing certificates and get audited to ensure they're in compliance with regulations.

In practice, when the PC is in "Setup Mode" you don't need the keys being signed you can install them in reverse order and by specification it should work. Now if the PC is in "User Mode" that means secure boot is enabled. That's when you need signed updates. I will say if you can update the PK through an out of band mechanism - theirs no reason you can't update other things like the KEK. Let me look into if we plan on signing a KEK update with our PK - in case that's needed.

Automating a fix from the Windows guest side doesn't seem feasible for the reason you described. There's nothing to authenticate the KEK update against without a valid PK first. An ESXi host update that writes the correct PK into existing VM NVRAM seems like the only viable automated path, which is presumably what Broadcom means by "future release."

This is correct, we expect the firmware to implement certain APIs and be able to validate. Without those APIs being correctly implemented from Windows theres not much we can do.

Does this help?

[jamesaepp]

Does this help?

Somewhat. I'd ideally like someone with a bit more ""authority"" over the MS PK to comment. Broadcom/VMware is the first example I've seen recommending this.

I will say if you can update the PK through an out of band mechanism - theirs no reason you can't update other things like the KEK

I'm still in my testing but it's definitely possible to install the KEK to VMs on ESXi (8) even with the PK in its "null" state (whatever that means, exactly).

I don't have direct access to ESXi 9 yet so can't comment on what VMs created on ESXi 9 look like (what PK Broadcom/VMware is using for those).

[Flickdm]

So again, I'm not a Broadcom engineer so I cannot speak for Broadcom but I am a UEFI firmware engineer with a deep understanding of Secure Boot.

Does this help?

Somewhat. I'd ideally like someone with a bit more ""authority"" over the MS PK to comment. Broadcom/VMware is the first example I've seen recommending this.

So I'm not what sure you mean by "authority". I along with a coworker announced this to our OEM partners in 2023 and this might be the only public information I'm aware of.. Most of the time we recommend it directly to OEMs but again its completely optional. However I can see if we can get more public facing documentation made.

https://youtu.be/o7kg1gX-KNc?t=1460

For reference, the PKpriv is stored in a HSM managed by our PRSS team that routinely gets audited for compliance. In order for someone to sign with it they need multiple people to sign off with it from our security team that is responsible for Secure Boot.

All of this is to say - you're free to use your own PK.

I will say if you can update the PK through an out of band mechanism - theirs no reason you can't update other things like the KEK

I'm still in my testing but it's definitely possible to install the KEK to VMs on ESXi (8) even with the PK in its "null" state (whatever that means, exactly).

I don't have direct access to ESXi 9 yet so can't comment on what VMs created on ESXi 9 look like (what PK Broadcom/VMware is using for those).

I don't know what a "NULL" PK means but generally if the PK is not defined, you are allowed to enroll whatever certificates you want. Generally Secure Boot is not enforced until the PK is set.

[haz-ard-9]

The NULL PK explanation actually clarifies something we've observed with the NVRAM rename approach. When ESXi regenerates the NVRAM file it creates exactly that state, no PK defined and Secure Boot not yet enforced, which is why cert enrollment works cleanly after the rename. The VM is effectively placed back into Setup Mode by the regeneration, allowing the KEK and DB to be written without needing authorization from an existing PK.

The confirmation that the WindowsOEMDevicesPK arrangement is a formally offered and HSM-backed option that Microsoft actively proposed to OEMs in 2023 is also reassuring for administrators who have already followed KB 423919 and enrolled it. It's not just a stopgap workaround. It's a documented, audited path that other OEMs have the option to take.

Still interested in your follow-up on whether Microsoft plans to sign a KEK update with the WindowsOEMDevicesPK PK. If that happens and gets submitted to this repo it would mean VMs running with the MS PK enrolled would receive future KEK updates through Windows Update automatically without any further manual steps, which would be a clean long-term solution for ESXi admins on 8.x.

[Flickdm]

The NULL PK explanation actually clarifies something we've observed with the NVRAM rename approach. When ESXi regenerates the NVRAM file it creates exactly that state, no PK defined and Secure Boot not yet enforced, which is why cert enrollment works cleanly after the rename. The VM is effectively placed back into Setup Mode by the regeneration, allowing the KEK and DB to be written without needing authorization from an existing PK.

So the reason I said "Generally" is that virtual machine firmware often does things weird. However in practice if you can get a machine into Setup Mode then you can use a script like InstallSecureBootKeys.ps1 from the OS to install keys.

Usually installing the PK will enable secure boot - but there are some known cases where an OEM decided to diverge from the UEFI specification.

The confirmation that the WindowsOEMDevicesPK arrangement is a formally offered and HSM-backed option that Microsoft actively proposed to OEMs in 2023 is also reassuring for administrators who have already followed KB 423919 and enrolled it. It's not just a stopgap workaround. It's a documented, audited path that other OEMs have the option to take.

Yes, it was also provided as a solution to PKFail, and I know a few other OEMs are chosing to go through this route because as you mention, it puts the work on us instead of them.

Still interested in your follow-up on whether Microsoft plans to sign a KEK update with the WindowsOEMDevicesPK PK. If that happens and gets submitted to this repo it would mean VMs running with the MS PK enrolled would receive future KEK updates through Windows Update automatically without any further manual steps, which would be a clean long-term solution for ESXi admins on 8.x.

So actually I forgot but I was just reminded that I already uploaded a public version of that very KEK update

[Secure Boot KEK Update] Microsoft PK-Signed KEK Update - Windows OEM Devices PK

So yes, if you use our PK, we will attempt to service it with our signed payload

[haz-ard-9]

So the reason I said "Generally" is that virtual machine firmware often does things weird. However in practice if you can get a machine into Setup Mode then you can use a script like InstallSecureBootKeys.ps1 from the OS to install keys. Usually installing the PK will enable secure boot - but there are some known cases where an OEM decided to diverge from the UEFI specification.

Good to know virtual machine firmware can behave differently here. The NVRAM regeneration approach has worked consistently in our testing but it's useful to understand that it's not guaranteed by the spec.

---

Yes, it was also provided as a solution to PKFail, and I know a few other OEMs are choosing to go through this route because as you mention, it puts the work on us instead of them.

The PKFail context also makes the WindowsOEMDevicesPK arrangement make a lot more sense from a broader industry perspective. This additional info lets us know that Broadcom isn't the only one taking this route.

---

So actually I forgot but I was just reminded that I already uploaded a public version of that very KEK update. So yes, if you use our PK, we will attempt to service it with our signed payload.

That closes the loop nicely on the KEK servicing question. For ESXi administrators who have enrolled the WindowsOEMDevicesPK following Broadcom KB 423919, this means future KEK updates will be handled automatically through Windows Update with no additional manual steps needed going forward. Really good news for the community.

Thank you for all of the additional context and clarification!

[Flickdm]

Of course! Let me know if there are any other questions!