Enable Multi-core based lazy-accept for TDVF #3813
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4172 TdxMailboxLib once was designed to be used in DXE phase. But now it is going to be used in SEC/PEI phase (in the following patches). Global variables are not allowed. The library is refactored after those global variables are deleted. Cc: Erdem Aktas <erdemaktas@google.com> Cc: Gerd Hoffmann <kraxel@redhat.com> Cc: James Bottomley <jejb@linux.ibm.com> Cc: Jiewen Yao <jiewen.yao@intel.com> Cc: Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by: Min Xu <min.m.xu@intel.com> Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4172 TdxMailboxLib is designed only for TDX guest which arch is X64. This patch set the VALID_ARCHITECTURES of TdxMailboxLib as X64. Because in the following patches TdxMailboxLib will be included in PlatformInitLib. While PlatformInitLib is imported by some X64 platforms (for example AmdSevX64.dsc). So we need a NULL instance of TdxMailboxLib which VALID_ARCHITECTURES is X64 as well. Based on this consideration we design TdxMailboxLibNull. Cc: Erdem Aktas <erdemaktas@google.com> Cc: Gerd Hoffmann <kraxel@redhat.com> Cc: James Bottomley <jejb@linux.ibm.com> Cc: Jiewen Yao <jiewen.yao@intel.com> Cc: Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by: Min Xu <min.m.xu@intel.com> Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4172 In the following patches TdxMailboxLib will be included in PlatformInitLib. While PlatformInitLib is imported by some IA32/X64 platforms (for example AmdSevX64.dsc). So TdxMailboxLibNull is added in those platforms which don't support TDX feature. Cc: Erdem Aktas <erdemaktas@google.com> Cc: Gerd Hoffmann <kraxel@redhat.com> Cc: James Bottomley <jejb@linux.ibm.com> Cc: Jiewen Yao <jiewen.yao@intel.com> Cc: Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by: Min Xu <min.m.xu@intel.com> Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4172 This patch moves the TDX APs nasm code from SecEntry.nasm to IntelTdxAPs.nasm. IntelTdxX64 and OvmfPkgX64 use the same nasm so that it can be easier to be managed. In the following patch there will be AcceptMemory related changes in IntelTdxAPs.nasm. Cc: Erdem Aktas <erdemaktas@google.com> Cc: Gerd Hoffmann <kraxel@redhat.com> Cc: James Bottomley <jejb@linux.ibm.com> Cc: Jiewen Yao <jiewen.yao@intel.com> Cc: Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by: Min Xu <min.m.xu@intel.com> Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4172 TDVF APs once did nothing but spin around to wait for the Wakeup command. This patch enables APs to handle the AcceptPages command. Once APs find the AcceptPages command, it set its stack and jump to the function of ApAcceptMemoryResourceRange (which will be introduced in the following patch). Cc: Erdem Aktas <erdemaktas@google.com> Cc: Gerd Hoffmann <kraxel@redhat.com> Cc: James Bottomley <jejb@linux.ibm.com> Cc: Jiewen Yao <jiewen.yao@intel.com> Cc: Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by: Min Xu <min.m.xu@intel.com> Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4172 TDVF once accepts memory only by BSP. To improve the boot performance this patch introduce the multi-core accpet memory. Multi-core means BSP and APs work together to accept memory. TDVF leverages mailbox to wake up APs. It is not enabled in MpInitLib (Which requires SIPI). So multi-core accept memory cannot leverages MpInitLib to coordinate BSP and APs to work together. So TDVF split the accept memory into 2 phases. - AcceptMemoryForAPsStack: BSP accepts a small piece of memory which is then used by APs to setup stack. We assign a 16KB stack for each AP. So a td-guest with 256 vCPU requires 255*16KB = 4080KB. - AcceptMemory: After above small piece of memory is accepted, BSP commands APs to accept memory by sending AcceptPages command in td-mailbox. Together with the command and accpet-function, the APsStack address is send as well. APs then set the stack and jump to accept-function to accept memory. AcceptMemoryForAPsStack accepts as small memory as possible and then jump to AcceptMemory. It fully takes advantage of BSP/APs to work together. After accept memory is done, the memory region for APsStack is not used anymore. It can be used as other private memory. Because accept-memory is in the very beginning of boot process and it will not impact other phases. Cc: Erdem Aktas <erdemaktas@google.com> Cc: Gerd Hoffmann <kraxel@redhat.com> Cc: James Bottomley <jejb@linux.ibm.com> Cc: Jiewen Yao <jiewen.yao@intel.com> Cc: Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by: Min Xu <min.m.xu@intel.com> Reviewed-by: Jiewen Yao <jiewen.yao@intel.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4172
TDVF once accepts memory only by BSP. To improve the boot performance
this patch-set introduce the multi-core accpet memory. Multi-core means
BSP and APs work together to accept memory.
TDVF leverages mailbox to wake up APs. It is not enabled in MpInitLib
(Which requires SIPI). So multi-core accept memory cannot leverages
MpInitLib to coordinate BSP and APs to work together.
So TDVF split the accept memory into 2 phases.
BSP accepts a small piece of memory which is then used by APs to setup
stack. We assign a 16KB stack for each AP. So a td-guest with 256 vCPU
requires 255*16KB = 4080KB.
After above small piece of memory is accepted, BSP commands APs to
accept memory by sending AcceptPages command in td-mailbox. Together
with the command and accpet-function, the APsStack address is send
as well. APs then set the stack and jump to accept-function to accept
memory.
Patch 1-3:
TdxMailboxLib is refactored to help multi-core lazy accept.
Patch 4-5:
Refactor APs' logical to support accpet page.
Patch 6:
Implement multi-core accept in TDVF.
Code: https://github.com/mxu9/edk2/tree/Multi-Core-Lazy-Accept.v2
v2 changes:
Cc: Erdem Aktas erdemaktas@google.com
Cc: Gerd Hoffmann kraxel@redhat.com
Cc: James Bottomley jejb@linux.ibm.com
Cc: Jiewen Yao jiewen.yao@intel.com
Cc: Tom Lendacky thomas.lendacky@amd.com
Signed-off-by: Min Xu min.m.xu@intel.com
Min M Xu (6):
OvmfPkg/TdxMailboxLib: Delete global variables
OvmfPkg/TdxMailboxLib: Add NULL instance of TdxMailboxLib
OvmfPkg: Add TdxMailboxLibNull in some platform dsc
OvmfPkg/Sec: Move TDX APs related nasm code to IntelTdxAPs.nasm
OvmfPkg: Enable APs to accept memory for TDVF
OvmfPkg/PlatformInitLib: Implement multi-core accept memory for TDVF
OvmfPkg/AmdSev/AmdSevX64.dsc | 1 +
OvmfPkg/CloudHv/CloudHvX64.dsc | 1 +
OvmfPkg/Include/TdxCommondefs.inc | 4 +-
OvmfPkg/IntelTdx/Sec/X64/IntelTdxAPs.nasm | 119 +++++
OvmfPkg/IntelTdx/Sec/X64/SecEntry.nasm | 58 +--
OvmfPkg/Library/PlatformInitLib/IntelTdx.c | 411 +++++++++++++++---
.../PlatformInitLib/PlatformInitLib.inf | 1 +
OvmfPkg/Library/TdxMailboxLib/TdxMailbox.c | 22 +-
.../Library/TdxMailboxLib/TdxMailboxLib.inf | 7 +-
.../TdxMailboxLib/TdxMailboxLibNull.inf | 34 ++
OvmfPkg/Microvm/MicrovmX64.dsc | 1 +
OvmfPkg/OvmfPkgIa32.dsc | 1 +
OvmfPkg/OvmfPkgIa32X64.dsc | 2 +-
OvmfPkg/Sec/X64/SecEntry.nasm | 58 +--
14 files changed, 545 insertions(+), 175 deletions(-)
create mode 100644 OvmfPkg/IntelTdx/Sec/X64/IntelTdxAPs.nasm
create mode 100644 OvmfPkg/Library/TdxMailboxLib/TdxMailboxLibNull.inf