[AMDGPU] Pick available high VGPR for CSR SGPR spilling #78669
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@llvm/pr-subscribers-backend-amdgpu Author: Christudasan Devadasan (cdevadas) ChangesCSR SGPR spilling currently uses the early available physical VGPRs. It currently imposes a high register pressure while trying to allocate large VGPR tuples within the default register budget. This patch changes the spilling strategy by picking the VGPRs in the reverse order, the highest available VGPR first and later after regalloc shift them back to the lowest available range. With that, the initial VGPRs would be available for allocation and possibility Patch is 696.26 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/78669.diff 31 Files Affected:
diff --git a/llvm/lib/Target/AMDGPU/SIFrameLowering.cpp b/llvm/lib/Target/AMDGPU/SIFrameLowering.cpp
index a02c2a46590822..5a7eb89b7a51ac 100644
--- a/llvm/lib/Target/AMDGPU/SIFrameLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIFrameLowering.cpp
@@ -95,7 +95,8 @@ static void getVGPRSpillLaneOrTempRegister(
TargetStackID::SGPRSpill);
if (TRI->spillSGPRToVGPR() &&
- MFI->allocateSGPRSpillToVGPRLane(MF, FI, /* IsPrologEpilog */ true)) {
+ MFI->allocateSGPRSpillToVGPRLane(MF, FI, /* SpillToPhysVGPRLane */ true,
+ /* IsPrologEpilog */ true)) {
// 2: There's no free lane to spill, and no free register to save the
// SGPR, so we're forced to take another VGPR to use for the spill.
MFI->addToPrologEpilogSGPRSpills(
@@ -1560,6 +1561,8 @@ void SIFrameLowering::determineCalleeSaves(MachineFunction &MF,
if (MFI->isChainFunction() && !MF.getFrameInfo().hasTailCall())
return;
+ MFI->shiftSpillPhysVGPRsToLowestRange(MF);
+
TargetFrameLowering::determineCalleeSaves(MF, SavedVGPRs, RS);
if (MFI->isEntryFunction())
return;
diff --git a/llvm/lib/Target/AMDGPU/SILowerSGPRSpills.cpp b/llvm/lib/Target/AMDGPU/SILowerSGPRSpills.cpp
index 0ba7792ac436d4..49c471395afe0e 100644
--- a/llvm/lib/Target/AMDGPU/SILowerSGPRSpills.cpp
+++ b/llvm/lib/Target/AMDGPU/SILowerSGPRSpills.cpp
@@ -369,7 +369,8 @@ bool SILowerSGPRSpills::runOnMachineFunction(MachineFunction &MF) {
// regalloc aware CFI generation to insert new CFIs along with the
// intermediate spills is implemented. There is no such support
// currently exist in the LLVM compiler.
- if (FuncInfo->allocateSGPRSpillToVGPRLane(MF, FI, true)) {
+ if (FuncInfo->allocateSGPRSpillToVGPRLane(
+ MF, FI, /* SpillToPhysVGPRLane */ true)) {
NewReservedRegs = true;
bool Spilled = TRI->eliminateSGPRToVGPRSpillFrameIndex(
MI, FI, nullptr, Indexes, LIS, true);
diff --git a/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp b/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp
index e8142244b7db69..f1d176d16601e3 100644
--- a/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.cpp
@@ -312,6 +312,35 @@ bool SIMachineFunctionInfo::isCalleeSavedReg(const MCPhysReg *CSRegs,
return false;
}
+void SIMachineFunctionInfo::shiftSpillPhysVGPRsToLowestRange(
+ MachineFunction &MF) {
+ for (unsigned I = 0, E = SpillPhysVGPRs.size(); I < E; ++I) {
+ Register Reg = SpillPhysVGPRs[I];
+ const SIRegisterInfo *TRI =
+ MF.getSubtarget<GCNSubtarget>().getRegisterInfo();
+ MachineRegisterInfo &MRI = MF.getRegInfo();
+ Register NewReg =
+ TRI->findUnusedRegister(MRI, &AMDGPU::VGPR_32RegClass, MF);
+ if (!NewReg || NewReg >= Reg)
+ continue;
+
+ MRI.replaceRegWith(Reg, NewReg);
+
+ // Update various tables with the new VGPR.
+ SpillPhysVGPRs[I] = NewReg;
+ WWMReservedRegs.remove(Reg);
+ WWMReservedRegs.insert(NewReg);
+ WWMSpills.insert(std::make_pair(NewReg, WWMSpills[Reg]));
+ WWMSpills.erase(Reg);
+
+ for (MachineBasicBlock &MBB : MF) {
+ MBB.removeLiveIn(Reg);
+ MBB.addLiveIn(NewReg);
+ MBB.sortUniqueLiveIns();
+ }
+ }
+}
+
bool SIMachineFunctionInfo::allocateVirtualVGPRForSGPRSpills(
MachineFunction &MF, int FI, unsigned LaneIndex) {
MachineRegisterInfo &MRI = MF.getRegInfo();
@@ -329,13 +358,17 @@ bool SIMachineFunctionInfo::allocateVirtualVGPRForSGPRSpills(
}
bool SIMachineFunctionInfo::allocatePhysicalVGPRForSGPRSpills(
- MachineFunction &MF, int FI, unsigned LaneIndex) {
+ MachineFunction &MF, int FI, unsigned LaneIndex, bool IsPrologEpilog) {
const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>();
const SIRegisterInfo *TRI = ST.getRegisterInfo();
MachineRegisterInfo &MRI = MF.getRegInfo();
Register LaneVGPR;
if (!LaneIndex) {
- LaneVGPR = TRI->findUnusedRegister(MRI, &AMDGPU::VGPR_32RegClass, MF);
+ // Find the highest available register if called before RA to ensure the
+ // lowest registers are available for allocation. The LaneVGPR, in that
+ // case, will be shifted back to the lowest range after VGPR allocation.
+ LaneVGPR = TRI->findUnusedRegister(MRI, &AMDGPU::VGPR_32RegClass, MF,
+ !IsPrologEpilog);
if (LaneVGPR == AMDGPU::NoRegister) {
// We have no VGPRs left for spilling SGPRs. Reset because we will not
// partially spill the SGPR to VGPRs.
@@ -359,12 +392,12 @@ bool SIMachineFunctionInfo::allocatePhysicalVGPRForSGPRSpills(
return true;
}
-bool SIMachineFunctionInfo::allocateSGPRSpillToVGPRLane(MachineFunction &MF,
- int FI,
- bool IsPrologEpilog) {
+bool SIMachineFunctionInfo::allocateSGPRSpillToVGPRLane(
+ MachineFunction &MF, int FI, bool SpillToPhysVGPRLane,
+ bool IsPrologEpilog) {
std::vector<SIRegisterInfo::SpilledReg> &SpillLanes =
- IsPrologEpilog ? SGPRSpillsToPhysicalVGPRLanes[FI]
- : SGPRSpillsToVirtualVGPRLanes[FI];
+ SpillToPhysVGPRLane ? SGPRSpillsToPhysicalVGPRLanes[FI]
+ : SGPRSpillsToVirtualVGPRLanes[FI];
// This has already been allocated.
if (!SpillLanes.empty())
@@ -384,14 +417,15 @@ bool SIMachineFunctionInfo::allocateSGPRSpillToVGPRLane(MachineFunction &MF,
assert(ST.getRegisterInfo()->spillSGPRToVGPR() &&
"not spilling SGPRs to VGPRs");
- unsigned &NumSpillLanes =
- IsPrologEpilog ? NumPhysicalVGPRSpillLanes : NumVirtualVGPRSpillLanes;
+ unsigned &NumSpillLanes = SpillToPhysVGPRLane ? NumPhysicalVGPRSpillLanes
+ : NumVirtualVGPRSpillLanes;
for (unsigned I = 0; I < NumLanes; ++I, ++NumSpillLanes) {
unsigned LaneIndex = (NumSpillLanes % WaveSize);
- bool Allocated = IsPrologEpilog
- ? allocatePhysicalVGPRForSGPRSpills(MF, FI, LaneIndex)
+ bool Allocated = SpillToPhysVGPRLane
+ ? allocatePhysicalVGPRForSGPRSpills(MF, FI, LaneIndex,
+ IsPrologEpilog)
: allocateVirtualVGPRForSGPRSpills(MF, FI, LaneIndex);
if (!Allocated) {
NumSpillLanes -= I;
diff --git a/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h b/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h
index dc63ae44c528db..6c593699b994e9 100644
--- a/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h
+++ b/llvm/lib/Target/AMDGPU/SIMachineFunctionInfo.h
@@ -548,7 +548,8 @@ class SIMachineFunctionInfo final : public AMDGPUMachineFunction,
bool allocateVirtualVGPRForSGPRSpills(MachineFunction &MF, int FI,
unsigned LaneIndex);
bool allocatePhysicalVGPRForSGPRSpills(MachineFunction &MF, int FI,
- unsigned LaneIndex);
+ unsigned LaneIndex,
+ bool IsPrologEpilog);
public:
Register getVGPRForAGPRCopy() const {
@@ -588,6 +589,7 @@ class SIMachineFunctionInfo final : public AMDGPUMachineFunction,
}
ArrayRef<Register> getSGPRSpillVGPRs() const { return SpillVGPRs; }
+
const WWMSpillsMap &getWWMSpills() const { return WWMSpills; }
const ReservedRegSet &getWWMReservedRegs() const { return WWMReservedRegs; }
@@ -702,7 +704,12 @@ class SIMachineFunctionInfo final : public AMDGPUMachineFunction,
I->second.IsDead = true;
}
+ // To bring the Physical VGPRs in the highest range allocated for CSR SGPR
+ // spilling into the lowest available range.
+ void shiftSpillPhysVGPRsToLowestRange(MachineFunction &MF);
+
bool allocateSGPRSpillToVGPRLane(MachineFunction &MF, int FI,
+ bool SpillToPhysVGPRLane = false,
bool IsPrologEpilog = false);
bool allocateVGPRSpillToAGPR(MachineFunction &MF, int FI, bool isAGPRtoVGPR);
diff --git a/llvm/test/CodeGen/AMDGPU/amdgpu-simplify-libcall-pow-codegen.ll b/llvm/test/CodeGen/AMDGPU/amdgpu-simplify-libcall-pow-codegen.ll
index e65eca78106105..bdd7ff11fde634 100644
--- a/llvm/test/CodeGen/AMDGPU/amdgpu-simplify-libcall-pow-codegen.ll
+++ b/llvm/test/CodeGen/AMDGPU/amdgpu-simplify-libcall-pow-codegen.ll
@@ -116,38 +116,38 @@ define double @test_pow_fast_f64__integral_y(double %x, i32 %y.i) {
; CHECK-NEXT: s_mov_b32 s16, s33
; CHECK-NEXT: s_mov_b32 s33, s32
; CHECK-NEXT: s_or_saveexec_b64 s[18:19], -1
-; CHECK-NEXT: buffer_store_dword v40, off, s[0:3], s33 offset:12 ; 4-byte Folded Spill
+; CHECK-NEXT: buffer_store_dword v43, off, s[0:3], s33 offset:12 ; 4-byte Folded Spill
; CHECK-NEXT: s_mov_b64 exec, s[18:19]
-; CHECK-NEXT: v_writelane_b32 v40, s16, 14
-; CHECK-NEXT: v_writelane_b32 v40, s30, 0
-; CHECK-NEXT: v_writelane_b32 v40, s31, 1
-; CHECK-NEXT: v_writelane_b32 v40, s34, 2
-; CHECK-NEXT: v_writelane_b32 v40, s35, 3
-; CHECK-NEXT: v_writelane_b32 v40, s36, 4
-; CHECK-NEXT: v_writelane_b32 v40, s37, 5
-; CHECK-NEXT: v_writelane_b32 v40, s38, 6
-; CHECK-NEXT: v_writelane_b32 v40, s39, 7
+; CHECK-NEXT: v_writelane_b32 v43, s16, 14
+; CHECK-NEXT: v_writelane_b32 v43, s30, 0
+; CHECK-NEXT: v_writelane_b32 v43, s31, 1
+; CHECK-NEXT: v_writelane_b32 v43, s34, 2
+; CHECK-NEXT: v_writelane_b32 v43, s35, 3
+; CHECK-NEXT: v_writelane_b32 v43, s36, 4
+; CHECK-NEXT: v_writelane_b32 v43, s37, 5
+; CHECK-NEXT: v_writelane_b32 v43, s38, 6
+; CHECK-NEXT: v_writelane_b32 v43, s39, 7
; CHECK-NEXT: s_addk_i32 s32, 0x800
-; CHECK-NEXT: v_writelane_b32 v40, s40, 8
-; CHECK-NEXT: v_writelane_b32 v40, s41, 9
+; CHECK-NEXT: v_writelane_b32 v43, s40, 8
+; CHECK-NEXT: v_writelane_b32 v43, s41, 9
; CHECK-NEXT: s_mov_b64 s[40:41], s[4:5]
; CHECK-NEXT: s_getpc_b64 s[4:5]
; CHECK-NEXT: s_add_u32 s4, s4, _Z4log2d@gotpcrel32@lo+4
; CHECK-NEXT: s_addc_u32 s5, s5, _Z4log2d@gotpcrel32@hi+12
; CHECK-NEXT: s_load_dwordx2 s[16:17], s[4:5], 0x0
-; CHECK-NEXT: v_writelane_b32 v40, s42, 10
-; CHECK-NEXT: buffer_store_dword v41, off, s[0:3], s33 offset:8 ; 4-byte Folded Spill
-; CHECK-NEXT: buffer_store_dword v42, off, s[0:3], s33 offset:4 ; 4-byte Folded Spill
-; CHECK-NEXT: buffer_store_dword v43, off, s[0:3], s33 ; 4-byte Folded Spill
-; CHECK-NEXT: v_writelane_b32 v40, s43, 11
-; CHECK-NEXT: v_mov_b32_e32 v43, v1
-; CHECK-NEXT: v_writelane_b32 v40, s44, 12
-; CHECK-NEXT: v_and_b32_e32 v1, 0x7fffffff, v43
+; CHECK-NEXT: v_writelane_b32 v43, s42, 10
+; CHECK-NEXT: buffer_store_dword v40, off, s[0:3], s33 offset:8 ; 4-byte Folded Spill
+; CHECK-NEXT: buffer_store_dword v41, off, s[0:3], s33 offset:4 ; 4-byte Folded Spill
+; CHECK-NEXT: buffer_store_dword v42, off, s[0:3], s33 ; 4-byte Folded Spill
+; CHECK-NEXT: v_writelane_b32 v43, s43, 11
+; CHECK-NEXT: v_mov_b32_e32 v42, v1
+; CHECK-NEXT: v_writelane_b32 v43, s44, 12
+; CHECK-NEXT: v_and_b32_e32 v1, 0x7fffffff, v42
; CHECK-NEXT: s_mov_b64 s[4:5], s[40:41]
-; CHECK-NEXT: v_writelane_b32 v40, s45, 13
-; CHECK-NEXT: v_mov_b32_e32 v41, v31
+; CHECK-NEXT: v_writelane_b32 v43, s45, 13
+; CHECK-NEXT: v_mov_b32_e32 v40, v31
; CHECK-NEXT: s_mov_b64 s[34:35], s[6:7]
-; CHECK-NEXT: v_mov_b32_e32 v42, v2
+; CHECK-NEXT: v_mov_b32_e32 v41, v2
; CHECK-NEXT: s_mov_b32 s42, s15
; CHECK-NEXT: s_mov_b32 s43, s14
; CHECK-NEXT: s_mov_b32 s44, s13
@@ -156,7 +156,7 @@ define double @test_pow_fast_f64__integral_y(double %x, i32 %y.i) {
; CHECK-NEXT: s_mov_b64 s[38:39], s[8:9]
; CHECK-NEXT: s_waitcnt lgkmcnt(0)
; CHECK-NEXT: s_swappc_b64 s[30:31], s[16:17]
-; CHECK-NEXT: v_cvt_f64_i32_e32 v[2:3], v42
+; CHECK-NEXT: v_cvt_f64_i32_e32 v[2:3], v41
; CHECK-NEXT: s_getpc_b64 s[4:5]
; CHECK-NEXT: s_add_u32 s4, s4, _Z4exp2d@gotpcrel32@lo+4
; CHECK-NEXT: s_addc_u32 s5, s5, _Z4exp2d@gotpcrel32@hi+12
@@ -170,32 +170,32 @@ define double @test_pow_fast_f64__integral_y(double %x, i32 %y.i) {
; CHECK-NEXT: s_mov_b32 s13, s44
; CHECK-NEXT: s_mov_b32 s14, s43
; CHECK-NEXT: s_mov_b32 s15, s42
-; CHECK-NEXT: v_mov_b32_e32 v31, v41
+; CHECK-NEXT: v_mov_b32_e32 v31, v40
; CHECK-NEXT: s_waitcnt lgkmcnt(0)
; CHECK-NEXT: s_swappc_b64 s[30:31], s[16:17]
-; CHECK-NEXT: v_lshlrev_b32_e32 v2, 31, v42
-; CHECK-NEXT: v_and_b32_e32 v2, v2, v43
-; CHECK-NEXT: buffer_load_dword v43, off, s[0:3], s33 ; 4-byte Folded Reload
-; CHECK-NEXT: buffer_load_dword v42, off, s[0:3], s33 offset:4 ; 4-byte Folded Reload
-; CHECK-NEXT: buffer_load_dword v41, off, s[0:3], s33 offset:8 ; 4-byte Folded Reload
+; CHECK-NEXT: v_lshlrev_b32_e32 v2, 31, v41
+; CHECK-NEXT: v_and_b32_e32 v2, v2, v42
+; CHECK-NEXT: buffer_load_dword v42, off, s[0:3], s33 ; 4-byte Folded Reload
+; CHECK-NEXT: buffer_load_dword v41, off, s[0:3], s33 offset:4 ; 4-byte Folded Reload
+; CHECK-NEXT: buffer_load_dword v40, off, s[0:3], s33 offset:8 ; 4-byte Folded Reload
; CHECK-NEXT: v_or_b32_e32 v1, v2, v1
-; CHECK-NEXT: v_readlane_b32 s45, v40, 13
-; CHECK-NEXT: v_readlane_b32 s44, v40, 12
-; CHECK-NEXT: v_readlane_b32 s43, v40, 11
-; CHECK-NEXT: v_readlane_b32 s42, v40, 10
-; CHECK-NEXT: v_readlane_b32 s41, v40, 9
-; CHECK-NEXT: v_readlane_b32 s40, v40, 8
-; CHECK-NEXT: v_readlane_b32 s39, v40, 7
-; CHECK-NEXT: v_readlane_b32 s38, v40, 6
-; CHECK-NEXT: v_readlane_b32 s37, v40, 5
-; CHECK-NEXT: v_readlane_b32 s36, v40, 4
-; CHECK-NEXT: v_readlane_b32 s35, v40, 3
-; CHECK-NEXT: v_readlane_b32 s34, v40, 2
-; CHECK-NEXT: v_readlane_b32 s31, v40, 1
-; CHECK-NEXT: v_readlane_b32 s30, v40, 0
-; CHECK-NEXT: v_readlane_b32 s4, v40, 14
+; CHECK-NEXT: v_readlane_b32 s45, v43, 13
+; CHECK-NEXT: v_readlane_b32 s44, v43, 12
+; CHECK-NEXT: v_readlane_b32 s43, v43, 11
+; CHECK-NEXT: v_readlane_b32 s42, v43, 10
+; CHECK-NEXT: v_readlane_b32 s41, v43, 9
+; CHECK-NEXT: v_readlane_b32 s40, v43, 8
+; CHECK-NEXT: v_readlane_b32 s39, v43, 7
+; CHECK-NEXT: v_readlane_b32 s38, v43, 6
+; CHECK-NEXT: v_readlane_b32 s37, v43, 5
+; CHECK-NEXT: v_readlane_b32 s36, v43, 4
+; CHECK-NEXT: v_readlane_b32 s35, v43, 3
+; CHECK-NEXT: v_readlane_b32 s34, v43, 2
+; CHECK-NEXT: v_readlane_b32 s31, v43, 1
+; CHECK-NEXT: v_readlane_b32 s30, v43, 0
+; CHECK-NEXT: v_readlane_b32 s4, v43, 14
; CHECK-NEXT: s_or_saveexec_b64 s[6:7], -1
-; CHECK-NEXT: buffer_load_dword v40, off, s[0:3], s33 offset:12 ; 4-byte Folded Reload
+; CHECK-NEXT: buffer_load_dword v43, off, s[0:3], s33 offset:12 ; 4-byte Folded Reload
; CHECK-NEXT: s_mov_b64 exec, s[6:7]
; CHECK-NEXT: s_addk_i32 s32, 0xf800
; CHECK-NEXT: s_mov_b32 s33, s4
@@ -257,37 +257,37 @@ define double @test_powr_fast_f64(double %x, double %y) {
; CHECK-NEXT: s_mov_b32 s16, s33
; CHECK-NEXT: s_mov_b32 s33, s32
; CHECK-NEXT: s_or_saveexec_b64 s[18:19], -1
-; CHECK-NEXT: buffer_store_dword v40, off, s[0:3], s33 offset:12 ; 4-byte Folded Spill
+; CHECK-NEXT: buffer_store_dword v43, off, s[0:3], s33 offset:12 ; 4-byte Folded Spill
; CHECK-NEXT: s_mov_b64 exec, s[18:19]
-; CHECK-NEXT: v_writelane_b32 v40, s16, 14
-; CHECK-NEXT: v_writelane_b32 v40, s30, 0
-; CHECK-NEXT: v_writelane_b32 v40, s31, 1
-; CHECK-NEXT: v_writelane_b32 v40, s34, 2
-; CHECK-NEXT: v_writelane_b32 v40, s35, 3
-; CHECK-NEXT: v_writelane_b32 v40, s36, 4
-; CHECK-NEXT: v_writelane_b32 v40, s37, 5
-; CHECK-NEXT: v_writelane_b32 v40, s38, 6
-; CHECK-NEXT: v_writelane_b32 v40, s39, 7
+; CHECK-NEXT: v_writelane_b32 v43, s16, 14
+; CHECK-NEXT: v_writelane_b32 v43, s30, 0
+; CHECK-NEXT: v_writelane_b32 v43, s31, 1
+; CHECK-NEXT: v_writelane_b32 v43, s34, 2
+; CHECK-NEXT: v_writelane_b32 v43, s35, 3
+; CHECK-NEXT: v_writelane_b32 v43, s36, 4
+; CHECK-NEXT: v_writelane_b32 v43, s37, 5
+; CHECK-NEXT: v_writelane_b32 v43, s38, 6
+; CHECK-NEXT: v_writelane_b32 v43, s39, 7
; CHECK-NEXT: s_addk_i32 s32, 0x800
-; CHECK-NEXT: v_writelane_b32 v40, s40, 8
-; CHECK-NEXT: v_writelane_b32 v40, s41, 9
+; CHECK-NEXT: v_writelane_b32 v43, s40, 8
+; CHECK-NEXT: v_writelane_b32 v43, s41, 9
; CHECK-NEXT: s_mov_b64 s[40:41], s[4:5]
; CHECK-NEXT: s_getpc_b64 s[4:5]
; CHECK-NEXT: s_add_u32 s4, s4, _Z4log2d@gotpcrel32@lo+4
; CHECK-NEXT: s_addc_u32 s5, s5, _Z4log2d@gotpcrel32@hi+12
; CHECK-NEXT: s_load_dwordx2 s[16:17], s[4:5], 0x0
-; CHECK-NEXT: v_writelane_b32 v40, s42, 10
-; CHECK-NEXT: v_writelane_b32 v40, s43, 11
-; CHECK-NEXT: v_writelane_b32 v40, s44, 12
+; CHECK-NEXT: v_writelane_b32 v43, s42, 10
+; CHECK-NEXT: v_writelane_b32 v43, s43, 11
+; CHECK-NEXT: v_writelane_b32 v43, s44, 12
; CHECK-NEXT: s_mov_b64 s[4:5], s[40:41]
-; CHECK-NEXT: buffer_store_dword v41, off, s[0:3], s33 offset:8 ; 4-byte Folded Spill
-; CHECK-NEXT: buffer_store_dword v42, off, s[0:3], s33 offset:4 ; 4-byte Folded Spill
-; CHECK-NEXT: buffer_store_dword v43, off, s[0:3], s33 ; 4-byte Folded Spill
-; CHECK-NEXT: v_writelane_b32 v40, s45, 13
-; CHECK-NEXT: v_mov_b32_e32 v43, v31
+; CHECK-NEXT: buffer_store_dword v40, off, s[0:3], s33 offset:8 ; 4-byte Folded Spill
+; CHECK-NEXT: buffer_store_dword v41, off, s[0:3], s33 offset:4 ; 4-byte Folded Spill
+; CHECK-NEXT: buffer_store_dword v42, off, s[0:3], s33 ; 4-byte Folded Spill
+; CHECK-NEXT: v_writelane_b32 v43, s45, 13
+; CHECK-NEXT: v_mov_b32_e32 v42, v31
; CHECK-NEXT: s_mov_b64 s[34:35], s[6:7]
-; CHECK-NEXT: v_mov_b32_e32 v42, v3
-; CHECK-NEXT: v_mov_b32_e32 v41, v2
+; CHECK-NEXT: v_mov_b32_e32 v41, v3
+; CHECK-NEXT: v_mov_b32_e32 v40, v2
; CHECK-NEXT: s_mov_b32 s42, s15
; CHECK-NEXT: s_mov_b32 s43, s14
; CHECK-NEXT: s_mov_b32 s44, s13
@@ -296,7 +296,7 @@ define double @test_powr_fast_f64(double %x, double %y) {
; CHECK-NEXT: s_mov_b64 s[38:39], s[8:9]
; CHECK-NEXT: s_waitcnt lgkmcnt(0)
; CHECK-NEXT: s_swappc_b64 s[30:31], s[16:17]
-; CHECK-NEXT: v_mul_f64 v[0:1], v[0:1], v[41:42]
+; CHECK-NEXT: v_mul_f64 v[0:1], v[0:1], v[40:41]
; CHECK-NEXT: s_getpc_b64 s[4:5]
; CHECK-NEXT: s_add_u32 s4, s4, _Z4exp2d@gotpcrel32@lo+4
; CHECK-NEXT: s_addc_u32 s5, s5, _Z4exp2d@gotpcrel32@hi+12
@@ -309,29 +309,29 @@ define double @test_powr_fast_f64(double %x, double %y) {
; CHECK-NEXT: s_mov_b32 s13, s44
; CHECK-NEXT: s_mov_b32 s14, s43
; CHECK-NEXT: s_mov_b32 s15, s42
-; CHECK-NEXT: v_mov_b32_e32 v31, v43
+; CHECK-NEXT: v_mov_b32_e32 v31, v42
; CHECK-NEXT: s_waitcnt lgkmcnt(0)
; CHECK-NEXT: s_swappc_b64 s[30:31], s[16:17]
-; CHECK-NEXT: buffer_load_dword v43, off, s[0:3], s33 ; 4-byte Folded Reload
-; CHECK-NEXT: buffer_load_dword v42, off, s[0:3], s33 offset:4 ; 4-byte Folded Reload
-; CHECK-NEXT: buffer_load_dword v41, off, s[0:3], s33 offset:8 ; 4-byte Folded Reload
-; CHECK-NEXT: v_readlane_b32 s45, v40, 13
-; CHECK-NEXT: v_readlane_b32 s44, v40, 12
-; CHECK-NEXT: v_readlane_b32 s43, v40, 11
-; CHECK-NEXT: v_readlane_b32 s42, v40, 10
-; CHECK-NEXT: v_readlane_b32 s41, v40, 9
-; CHECK-NEXT: v_readlane_b32 s40, v40, 8
-; CHECK-NEXT: v_readlane_b32 s39, v40, 7
-; CHECK-NEXT: v_readlane_b32 s38, v40, 6
-; CHECK-NEXT: v_readlane_b32 s37, v40, 5
-; CHECK-NEXT: v_readlane_b32 s36, v40, 4
-; CHECK-NEXT: v_readlane_b32 s35, v40, 3
-; CHECK-NEXT: v_readlane_b32 s34, v40, 2
-; CHECK-NEXT: v_readlane_b32 s31, v40, 1
-; CHECK-NEXT: v_readlane_b32 s30, v40, 0
-; CHECK-NEXT: v_readlane_b32 s4, v40, 14
+; CHECK-NEXT: buffer_load_dword v42, off, s[0:3], s33 ; 4-byte Folded Reload
+; CHECK-NEXT: buffer_load_dword v41, off, s[0:3], s33 offset:4 ; 4-byte Folded Reload
+; CHECK-NEXT: buffer_load_dword v40, off, s[0:3], s33 offset:8 ; 4-byte Folded Reload
+; CHECK-NEXT: v_readlane_b32 s45, v43, 13
+; CHECK-NEXT: v_readlane_b32 s44, v43, 12
+; CHECK-NEXT: v_readlane_b32 s43, v43, 11
+; CHECK-NEXT: v_readlane_b32 s42, v43, 10
+; CHECK-NEXT: v_readlane_b32 s41, v43, 9
+; CHECK-NEXT: v_readlane_b32 s40, v43, 8
+; CHECK-NEXT: v_readlane_b32 s39, v43, 7
+; CHECK-NEXT: ...
[truncated]
|
arsenm
reviewed
Jan 22, 2024
| @@ -95,7 +95,8 @@ static void getVGPRSpillLaneOrTempRegister( | |||
| TargetStackID::SGPRSpill); | |||
|
|
|||
| if (TRI->spillSGPRToVGPR() && | |||
| MFI->allocateSGPRSpillToVGPRLane(MF, FI, /* IsPrologEpilog */ true)) { | |||
| MFI->allocateSGPRSpillToVGPRLane(MF, FI, /* SpillToPhysVGPRLane */ true, | |||
| @@ -369,7 +369,8 @@ bool SILowerSGPRSpills::runOnMachineFunction(MachineFunction &MF) { | |||
| // regalloc aware CFI generation to insert new CFIs along with the | |||
| // intermediate spills is implemented. There is no such support | |||
| // currently exist in the LLVM compiler. | |||
| if (FuncInfo->allocateSGPRSpillToVGPRLane(MF, FI, true)) { | |||
| if (FuncInfo->allocateSGPRSpillToVGPRLane( | |||
| MF, FI, /* SpillToPhysVGPRLane */ true)) { | |||
Comment on lines
319
to
320
| const SIRegisterInfo *TRI = | ||
| MF.getSubtarget<GCNSubtarget>().getRegisterInfo(); |
Comment on lines
324
to
325
| if (!NewReg || NewReg >= Reg) | ||
| continue; |
There was a problem hiding this comment.
There's no point in continuing the loop if this failed once
| @@ -312,6 +312,35 @@ bool SIMachineFunctionInfo::isCalleeSavedReg(const MCPhysReg *CSRegs, | |||
| return false; | |||
| } | |||
|
|
|||
| void SIMachineFunctionInfo::shiftSpillPhysVGPRsToLowestRange( | |||
There was a problem hiding this comment.
Maybe this should be commoned with the code to compact the scratch registers
There was a problem hiding this comment.
I'm not sure that's a good choice. The only common part I see is the MRI.replaceRegWith.
Comment on lines
336
to
338
| for (MachineBasicBlock &MBB : MF) { | ||
| MBB.removeLiveIn(Reg); | ||
| MBB.addLiveIn(NewReg); | ||
| MBB.sortUniqueLiveIns(); | ||
| } |
There was a problem hiding this comment.
Are these registers still reserved? If so there's no need to do this
There was a problem hiding this comment.
Adding the NewReg here can be avoided as all wwm-registers are already added into BB LiveIns during SIFrameLowering::determineCalleeSaves. But Reg has to be removed from the BBLiveIns which was added earlier in SIMachineFunctionInfo::allocatePhysicalVGPRForSGPRSpills while spilling CSR SGPR into VGPR.
Adding them into the BBLiveIns is required for their spill stores and restores later inserted at respective prolog and epilog blocks to avoid the MIR verifier error.
CSR SGPR spilling currently uses the early available physical VGPRs. It currently imposes a high register pressure while trying to allocate large VGPR tuples within the default register budget. This patch changes the spilling strategy by picking the VGPRs in the reverse order, the highest available VGPR first and later after regalloc shift them back to the lowest available range. With that, the initial VGPRs would be available for allocation and possibility of finding large number of contiguous registers will be more.
cdevadas
force-pushed
the
high-vgprs-for-csr-sgpr-spilling
branch
from
fded655
to
db62f07
Compare
|
Rebase + Suggestions incorporated. |
arsenm
approved these changes
Jan 23, 2024
searlmc1
pushed a commit
to ROCm/llvm-project
that referenced
this pull request
Mar 28, 2024
CSR SGPR spilling currently uses the early available physical VGPRs. It currently imposes a high register pressure while trying to allocate large VGPR tuples within the default register budget. This patch changes the spilling strategy by picking the VGPRs in the reverse order, the highest available VGPR first and later after regalloc shift them back to the lowest available range. With that, the initial VGPRs would be available for allocation and possibility of finding large number of contiguous registers will be more. Change-Id: Ib22d02f3dd5255a5b9157099acb91b779d3d1fc0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CSR SGPR spilling currently uses the early available physical VGPRs. It currently imposes a high register pressure while trying to allocate large VGPR tuples within the default register budget.
This patch changes the spilling strategy by picking the VGPRs in the reverse order, the highest available VGPR first and later after regalloc shift them back to the lowest available range. With that, the initial VGPRs would be available for allocation and possibility
of finding large number of contiguous registers will be more.