Skip to content

[DRAFT][RegisterCoalescer] Enable non-trivial rematerialization #160153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[DRAFT][RegisterCoalescer] Enable non-trivial rematerialization #160153

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
37781c8
[CodeGen] Rework isReallyTriviallyReMaterializable API
preames Sep 19, 2025
9104743
[POC] Non trivial remat in register coalescer
preames Sep 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions llvm/include/llvm/CodeGen/TargetInstrInfo.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,14 @@ class LLVM_ABI TargetInstrInfo : public MCInstrInfo {
return (MI.getOpcode() == TargetOpcode::IMPLICIT_DEF &&
MI.getNumOperands() == 1) ||
(MI.getDesc().isRematerializable() &&
isReallyTriviallyReMaterializable(MI));
isReMaterializableImpl(MI, true));
}

bool isReMaterializable(const MachineInstr &MI) const {
return (MI.getOpcode() == TargetOpcode::IMPLICIT_DEF &&
MI.getNumOperands() == 1) ||
(MI.getDesc().isRematerializable() &&
isReMaterializableImpl(MI, false));
}

/// Given \p MO is a PhysReg use return if it can be ignored for the purpose
Expand All @@ -198,7 +205,8 @@ class LLVM_ABI TargetInstrInfo : public MCInstrInfo {
/// predicate must return false if the instruction has any side effects other
/// than producing a value, or if it requres any address registers that are
/// not always available.
virtual bool isReallyTriviallyReMaterializable(const MachineInstr &MI) const;
virtual bool isReMaterializableImpl(const MachineInstr &MI,
bool TrivialOnly) const;

/// This method commutes the operands of the given machine instruction MI.
/// The operands to be commuted are specified by their indices OpIdx1 and
Expand Down
2 changes: 1 addition & 1 deletion llvm/lib/CodeGen/RegisterCoalescer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1325,7 +1325,7 @@ bool RegisterCoalescer::reMaterializeTrivialDef(const CoalescerPair &CP,
if (!TII->isAsCheapAsAMove(*DefMI))
return false;

if (!TII->isTriviallyReMaterializable(*DefMI))
if (!TII->isReMaterializable(*DefMI))
return false;

if (!definesFullReg(*DefMI, SrcReg))
Expand Down
13 changes: 7 additions & 6 deletions llvm/lib/CodeGen/TargetInstrInfo.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1590,8 +1590,8 @@ MachineTraceStrategy TargetInstrInfo::getMachineCombinerTraceStrategy() const {
return MachineTraceStrategy::TS_MinInstrCount;
}

bool TargetInstrInfo::isReallyTriviallyReMaterializable(
const MachineInstr &MI) const {
bool TargetInstrInfo::isReMaterializableImpl(
const MachineInstr &MI, bool OnlyTrivial) const {
const MachineFunction &MF = *MI.getMF();
const MachineRegisterInfo &MRI = MF.getRegInfo();

Expand Down Expand Up @@ -1658,10 +1658,11 @@ bool TargetInstrInfo::isReallyTriviallyReMaterializable(
if (MO.isDef() && Reg != DefReg)
return false;

// Don't allow any virtual-register uses. Rematting an instruction with
// virtual register uses would length the live ranges of the uses, which
// is not necessarily a good idea, certainly not "trivial".
if (MO.isUse())
// If asked for trivial materialization, don't allow any virtual-register
// uses. Rematting an instruction with virtual register uses would length
// the live ranges of the uses, which means rematerialization must become
// a per-user query which many callers don't want.
Comment on lines +1662 to +1664
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the existing comment was accurate since we never lengthen live ranges when rematerializing. It's something we definitely need to consider if we ever do end up extending live ranges, but that's truly non-trivial and I don't know if that's on the roadmap any time soon

if (OnlyTrivial && MO.isUse())
return false;
}

Expand Down
7 changes: 4 additions & 3 deletions llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,8 @@ static bool canRemat(const MachineInstr &MI) {
return false;
}

bool SIInstrInfo::isReallyTriviallyReMaterializable(
const MachineInstr &MI) const {
bool SIInstrInfo::isReMaterializableImpl(const MachineInstr &MI,
bool OnlyTrivial) const {

if (canRemat(MI)) {
// Normally VALU use of exec would block the rematerialization, but that
Expand All @@ -139,13 +139,14 @@ bool SIInstrInfo::isReallyTriviallyReMaterializable(
// There is difference to generic method which does not allow
// rematerialization if there are virtual register uses. We allow this,
// therefore this method includes SOP instructions as well.
// FIXME: This should only be done if OnlyTrivial is setup.
if (!MI.hasImplicitDef() &&
MI.getNumImplicitOperands() == MI.getDesc().implicit_uses().size() &&
!MI.mayRaiseFPException())
return true;
}

return TargetInstrInfo::isReallyTriviallyReMaterializable(MI);
return TargetInstrInfo::isReMaterializableImpl(MI, OnlyTrivial);
}

// Returns true if the scalar result of a VALU instruction depends on exec.
Expand Down
3 changes: 2 additions & 1 deletion llvm/lib/Target/AMDGPU/SIInstrInfo.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,8 @@ class SIInstrInfo final : public AMDGPUGenInstrInfo {
return ST;
}

bool isReallyTriviallyReMaterializable(const MachineInstr &MI) const override;
bool isReMaterializableImpl(const MachineInstr &MI,
bool OnlyTrivial) const override;

bool isIgnorableUse(const MachineOperand &MO) const override;

Expand Down
6 changes: 3 additions & 3 deletions llvm/lib/Target/ARM/ARMBaseInstrInfo.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6510,14 +6510,14 @@ bool ARMBaseInstrInfo::shouldOutlineFromFunctionByDefault(
return Subtarget.isMClass() && MF.getFunction().hasMinSize();
}

bool ARMBaseInstrInfo::isReallyTriviallyReMaterializable(
const MachineInstr &MI) const {
bool ARMBaseInstrInfo::isReMaterializableImpl(const MachineInstr &MI,
bool TrivialOnly) const {
// Try hard to rematerialize any VCTPs because if we spill P0, it will block
// the tail predication conversion. This means that the element count
// register has to be live for longer, but that has to be better than
// spill/restore and VPT predication.
return (isVCTP(&MI) && !isPredicated(MI)) ||
TargetInstrInfo::isReallyTriviallyReMaterializable(MI);
TargetInstrInfo::isReMaterializableImpl(MI, TrivialOnly);
}

unsigned llvm::getBLXOpcode(const MachineFunction &MF) {
Expand Down
3 changes: 2 additions & 1 deletion llvm/lib/Target/ARM/ARMBaseInstrInfo.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -479,7 +479,8 @@ class ARMBaseInstrInfo : public ARMGenInstrInfo {
MachineInstr *canFoldIntoMOVCC(Register Reg, const MachineRegisterInfo &MRI,
const TargetInstrInfo *TII) const;

bool isReallyTriviallyReMaterializable(const MachineInstr &MI) const override;
bool isReMaterializableImpl(const MachineInstr &MI,
bool OnlyTrivial) const override;

private:
/// Modeling special VFP / NEON fp MLA / MLS hazards.
Expand Down
2 changes: 1 addition & 1 deletion llvm/lib/Target/LoongArch/LoongArchInstrInfo.td
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -943,7 +943,7 @@ let Predicates = [IsLA64] in {
def ADD_D : ALU_3R<0x00108000>;
def SUB_D : ALU_3R<0x00118000>;
// ADDI_D isn't always rematerializable, but isReMaterializable will be used as
// a hint which is verified in isReallyTriviallyReMaterializable.
// a hint which is verified in isReMaterializableImpl.
// See LoongArchInstrInfo::isAsCheapAsAMove for more details.
let isReMaterializable = 1, isAsCheapAsAMove = 1 in {
def ADDI_D : ALU_2RI12<0x02c00000, simm12_addlike>;
Expand Down
6 changes: 3 additions & 3 deletions llvm/lib/Target/PowerPC/PPCInstrInfo.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1075,8 +1075,8 @@ Register PPCInstrInfo::isLoadFromStackSlot(const MachineInstr &MI,

// For opcodes with the ReMaterializable flag set, this function is called to
// verify the instruction is really rematable.
bool PPCInstrInfo::isReallyTriviallyReMaterializable(
const MachineInstr &MI) const {
bool PPCInstrInfo::isReMaterializableImpl(const MachineInstr &MI,
bool OnlyTrivial) const {
switch (MI.getOpcode()) {
default:
// Let base implementaion decide.
Expand Down Expand Up @@ -1112,7 +1112,7 @@ bool PPCInstrInfo::isReallyTriviallyReMaterializable(
case PPC::DMXXSETACCZ:
return true;
}
return TargetInstrInfo::isReallyTriviallyReMaterializable(MI);
return TargetInstrInfo::isReMaterializableImpl(MI, OnlyTrivial);
}

Register PPCInstrInfo::isStoreToStackSlot(const MachineInstr &MI,
Expand Down
3 changes: 2 additions & 1 deletion llvm/lib/Target/PowerPC/PPCInstrInfo.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -530,7 +530,8 @@ class PPCInstrInfo : public PPCGenInstrInfo {
unsigned &SubIdx) const override;
Register isLoadFromStackSlot(const MachineInstr &MI,
int &FrameIndex) const override;
bool isReallyTriviallyReMaterializable(const MachineInstr &MI) const override;
bool isReMaterializableImpl(const MachineInstr &MI,
bool OnlyTrivial) const override;
Register isStoreToStackSlot(const MachineInstr &MI,
int &FrameIndex) const override;

Expand Down
8 changes: 5 additions & 3 deletions llvm/lib/Target/RISCV/RISCVInstrInfo.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,18 +232,20 @@ Register RISCVInstrInfo::isStoreToStackSlot(const MachineInstr &MI,
return 0;
}

bool RISCVInstrInfo::isReallyTriviallyReMaterializable(
const MachineInstr &MI) const {
bool RISCVInstrInfo::isReMaterializableImpl(const MachineInstr &MI,
bool OnlyTrivial) const {
switch (RISCV::getRVVMCOpcode(MI.getOpcode())) {
case RISCV::VMV_V_X:
case RISCV::VFMV_V_F:
case RISCV::VMV_V_I:
case RISCV::VMV_S_X:
case RISCV::VFMV_S_F:
case RISCV::VID_V:
// FIXME: the v.x and v.f forms are not 'trivial' in the meaning
// of this API. Split them!
return MI.getOperand(1).isUndef();
default:
return TargetInstrInfo::isReallyTriviallyReMaterializable(MI);
return TargetInstrInfo::isReMaterializableImpl(MI, OnlyTrivial);
}
}

Expand Down
3 changes: 2 additions & 1 deletion llvm/lib/Target/RISCV/RISCVInstrInfo.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,8 @@ class RISCVInstrInfo : public RISCVGenInstrInfo {
Register isStoreToStackSlot(const MachineInstr &MI, int &FrameIndex,
TypeSize &MemBytes) const override;

bool isReallyTriviallyReMaterializable(const MachineInstr &MI) const override;
bool isReMaterializableImpl(const MachineInstr &MI,
bool OnlyTrivial) const override;

bool shouldBreakCriticalEdgeToSink(MachineInstr &MI) const override {
return MI.getOpcode() == RISCV::ADDI && MI.getOperand(1).isReg() &&
Expand Down
2 changes: 1 addition & 1 deletion llvm/lib/Target/RISCV/RISCVInstrInfo.td
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -779,7 +779,7 @@ def SH : Store_rri<0b001, "sh">, Sched<[WriteSTH, ReadStoreData, ReadMemBase]>;
def SW : Store_rri<0b010, "sw">, Sched<[WriteSTW, ReadStoreData, ReadMemBase]>;

// ADDI isn't always rematerializable, but isReMaterializable will be used as
// a hint which is verified in isReallyTriviallyReMaterializable.
// a hint which is verified in isReMaterializableImpl.
let isReMaterializable = 1, isAsCheapAsAMove = 1 in
def ADDI : ALU_ri<0b000, "addi">;

Expand Down
8 changes: 4 additions & 4 deletions llvm/lib/Target/WebAssembly/WebAssemblyInstrInfo.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,18 +39,18 @@ WebAssemblyInstrInfo::WebAssemblyInstrInfo(const WebAssemblySubtarget &STI)
WebAssembly::CATCHRET),
RI(STI.getTargetTriple()) {}

bool WebAssemblyInstrInfo::isReallyTriviallyReMaterializable(
const MachineInstr &MI) const {
bool WebAssemblyInstrInfo::isReMaterializableImpl(const MachineInstr &MI,
bool OnlyTrivial) const {
switch (MI.getOpcode()) {
case WebAssembly::CONST_I32:
case WebAssembly::CONST_I64:
case WebAssembly::CONST_F32:
case WebAssembly::CONST_F64:
// TargetInstrInfo::isReallyTriviallyReMaterializable misses these
// TargetInstrInfo::isReMaterializableImpl misses these
// because of the ARGUMENTS implicit def, so we manualy override it here.
return true;
default:
return TargetInstrInfo::isReallyTriviallyReMaterializable(MI);
return TargetInstrInfo::isReMaterializableImpl(MI, OnlyTrivial);
}
}

Expand Down
3 changes: 2 additions & 1 deletion llvm/lib/Target/WebAssembly/WebAssemblyInstrInfo.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@ class WebAssemblyInstrInfo final : public WebAssemblyGenInstrInfo {

const WebAssemblyRegisterInfo &getRegisterInfo() const { return RI; }

bool isReallyTriviallyReMaterializable(const MachineInstr &MI) const override;
bool isReMaterializableImpl(const MachineInstr &MI,
bool OnlyTrivial) const override;

void copyPhysReg(MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
const DebugLoc &DL, Register DestReg, Register SrcReg,
Expand Down
6 changes: 3 additions & 3 deletions llvm/lib/Target/X86/X86InstrInfo.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -755,8 +755,8 @@ static bool regIsPICBase(Register BaseReg, const MachineRegisterInfo &MRI) {
return isPICBase;
}

bool X86InstrInfo::isReallyTriviallyReMaterializable(
const MachineInstr &MI) const {
bool X86InstrInfo::isReMaterializableImpl(const MachineInstr &MI,
bool OnlyTrivial) const {
switch (MI.getOpcode()) {
default:
// This function should only be called for opcodes with the ReMaterializable
Expand Down Expand Up @@ -951,7 +951,7 @@ bool X86InstrInfo::isReallyTriviallyReMaterializable(
break;
}
}
return TargetInstrInfo::isReallyTriviallyReMaterializable(MI);
return TargetInstrInfo::isReMaterializableImpl(MI, OnlyTrivial);
}

void X86InstrInfo::reMaterialize(MachineBasicBlock &MBB,
Expand Down
3 changes: 2 additions & 1 deletion llvm/lib/Target/X86/X86InstrInfo.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -340,7 +340,8 @@ class X86InstrInfo final : public X86GenInstrInfo {
Register isStoreToStackSlotPostFE(const MachineInstr &MI,
int &FrameIndex) const override;

bool isReallyTriviallyReMaterializable(const MachineInstr &MI) const override;
bool isReMaterializableImpl(const MachineInstr &MI,
bool OnlyTrivial) const override;
void reMaterialize(MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
Register DestReg, unsigned SubIdx,
const MachineInstr &Orig,
Expand Down
12 changes: 6 additions & 6 deletions llvm/test/CodeGen/AArch64/aarch64-matrix-umull-smull.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,7 @@ define void @larger_smull(ptr nocapture noundef readonly %x, i16 noundef %y, ptr
; CHECK-SD-NEXT: and x9, x8, #0xfffffff0
; CHECK-SD-NEXT: add x10, x2, #32
; CHECK-SD-NEXT: add x11, x0, #16
; CHECK-SD-NEXT: mov x12, x9
; CHECK-SD-NEXT: and x12, x8, #0xfffffff0
; CHECK-SD-NEXT: .LBB3_4: // %vector.body
; CHECK-SD-NEXT: // =>This Inner Loop Header: Depth=1
; CHECK-SD-NEXT: ldp q1, q2, [x11, #-16]
Expand Down Expand Up @@ -313,7 +313,7 @@ define void @larger_smull(ptr nocapture noundef readonly %x, i16 noundef %y, ptr
; CHECK-GI-NEXT: and x10, x9, #0xfffffff0
; CHECK-GI-NEXT: add x11, x2, #32
; CHECK-GI-NEXT: add x12, x0, #16
; CHECK-GI-NEXT: mov x13, x10
; CHECK-GI-NEXT: and x13, x9, #0xfffffff0
; CHECK-GI-NEXT: xtn v0.4h, v0.4s
; CHECK-GI-NEXT: .LBB3_3: // %vector.body
; CHECK-GI-NEXT: // =>This Inner Loop Header: Depth=1
Expand Down Expand Up @@ -428,7 +428,7 @@ define void @larger_umull(ptr nocapture noundef readonly %x, i16 noundef %y, ptr
; CHECK-SD-NEXT: and x9, x8, #0xfffffff0
; CHECK-SD-NEXT: add x10, x2, #32
; CHECK-SD-NEXT: add x11, x0, #16
; CHECK-SD-NEXT: mov x12, x9
; CHECK-SD-NEXT: and x12, x8, #0xfffffff0
; CHECK-SD-NEXT: .LBB4_4: // %vector.body
; CHECK-SD-NEXT: // =>This Inner Loop Header: Depth=1
; CHECK-SD-NEXT: ldp q1, q2, [x11, #-16]
Expand Down Expand Up @@ -472,7 +472,7 @@ define void @larger_umull(ptr nocapture noundef readonly %x, i16 noundef %y, ptr
; CHECK-GI-NEXT: and x8, x9, #0xfffffff0
; CHECK-GI-NEXT: add x10, x2, #32
; CHECK-GI-NEXT: add x11, x0, #16
; CHECK-GI-NEXT: mov x12, x8
; CHECK-GI-NEXT: and x12, x9, #0xfffffff0
; CHECK-GI-NEXT: .LBB4_3: // %vector.body
; CHECK-GI-NEXT: // =>This Inner Loop Header: Depth=1
; CHECK-GI-NEXT: and w13, w1, #0xffff
Expand Down Expand Up @@ -596,7 +596,7 @@ define i16 @red_mla_dup_ext_u8_s8_s16(ptr noalias nocapture noundef readonly %A,
; CHECK-SD-NEXT: and x11, x10, #0xfffffff0
; CHECK-SD-NEXT: fmov s2, w9
; CHECK-SD-NEXT: add x8, x0, #8
; CHECK-SD-NEXT: mov x12, x11
; CHECK-SD-NEXT: and x12, x10, #0xfffffff0
; CHECK-SD-NEXT: .LBB5_5: // %vector.body
; CHECK-SD-NEXT: // =>This Inner Loop Header: Depth=1
; CHECK-SD-NEXT: ldp d3, d4, [x8, #-8]
Expand Down Expand Up @@ -646,10 +646,10 @@ define i16 @red_mla_dup_ext_u8_s8_s16(ptr noalias nocapture noundef readonly %A,
; CHECK-GI-NEXT: movi v0.2d, #0000000000000000
; CHECK-GI-NEXT: movi v1.2d, #0000000000000000
; CHECK-GI-NEXT: add x10, x0, #8
; CHECK-GI-NEXT: and x11, x8, #0xfffffff0
; CHECK-GI-NEXT: sbfx w9, w9, #8, #8
; CHECK-GI-NEXT: dup v2.8h, w9
; CHECK-GI-NEXT: and x9, x8, #0xfffffff0
; CHECK-GI-NEXT: mov x11, x9
; CHECK-GI-NEXT: .LBB5_5: // %vector.body
; CHECK-GI-NEXT: // =>This Inner Loop Header: Depth=1
; CHECK-GI-NEXT: ldp d3, d4, [x10, #-8]
Expand Down
2 changes: 1 addition & 1 deletion llvm/test/CodeGen/AArch64/machine-combiner-copy.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ define void @fma_dup_f16(ptr noalias nocapture noundef readonly %A, half noundef
; CHECK-NEXT: and x9, x8, #0xfffffff0
; CHECK-NEXT: add x10, x1, #16
; CHECK-NEXT: add x11, x0, #16
; CHECK-NEXT: mov x12, x9
; CHECK-NEXT: and x12, x8, #0xfffffff0
; CHECK-NEXT: .LBB0_4: // %vector.body
; CHECK-NEXT: // =>This Inner Loop Header: Depth=1
; CHECK-NEXT: ldp q1, q4, [x10, #-16]
Expand Down
Loading
Loading