Skip to content

Commit 1023dda

Browse files
shiltianshiltian
committed
[LLVM] Add the support for fmax and fmin in atomicrmw instruction
This patch adds the support for `fmax` and `fmin` operations in `atomicrmw` instruction. For now (at least in this patch), the instruction will be expanded to CAS loop. There are already a couple of targets supporting the feature. I'll create another patch(es) to enable them accordingly. Reviewed By: arsenm Differential Revision: https://reviews.llvm.org/D127041
1 parent f8e0264 commit 1023dda

36 files changed

+230
-9
lines changed

‎llvm/docs/GlobalISel/GenericOpcode.rst

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -727,7 +727,10 @@ G_ATOMIC_CMPXCHG
727727
Generic atomic cmpxchg. Expects a MachineMemOperand in addition to explicit
728728
operands.
729729

730-
G_ATOMICRMW_XCHG, G_ATOMICRMW_ADD, G_ATOMICRMW_SUB, G_ATOMICRMW_AND, G_ATOMICRMW_NAND, G_ATOMICRMW_OR, G_ATOMICRMW_XOR, G_ATOMICRMW_MAX, G_ATOMICRMW_MIN, G_ATOMICRMW_UMAX, G_ATOMICRMW_UMIN, G_ATOMICRMW_FADD, G_ATOMICRMW_FSUB
730+
G_ATOMICRMW_XCHG, G_ATOMICRMW_ADD, G_ATOMICRMW_SUB, G_ATOMICRMW_AND,
731+
G_ATOMICRMW_NAND, G_ATOMICRMW_OR, G_ATOMICRMW_XOR, G_ATOMICRMW_MAX,
732+
G_ATOMICRMW_MIN, G_ATOMICRMW_UMAX, G_ATOMICRMW_UMIN, G_ATOMICRMW_FADD,
733+
G_ATOMICRMW_FSUB, G_ATOMICRMW_FMAX, G_ATOMICRMW_FMIN
731734
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
732735

733736
Generic atomicrmw. Expects a MachineMemOperand in addition to explicit

‎llvm/docs/LangRef.rst

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10330,12 +10330,14 @@ operation. The operation must be one of the following keywords:
1033010330
- umin
1033110331
- fadd
1033210332
- fsub
10333+
- fmax
10334+
- fmin
1033310335

1033410336
For most of these operations, the type of '<value>' must be an integer
1033510337
type whose bit width is a power of two greater than or equal to eight
1033610338
and less than or equal to a target-specific size limit. For xchg, this
1033710339
may also be a floating point or a pointer type with the same size constraints
10338-
as integers. For fadd/fsub, this must be a floating point type. The
10340+
as integers. For fadd/fsub/fmax/fmin, this must be a floating point type. The
1033910341
type of the '``<pointer>``' operand must be a pointer to that type. If
1034010342
the ``atomicrmw`` is marked as ``volatile``, then the optimizer is not
1034110343
allowed to modify the number or order of execution of this
@@ -10372,6 +10374,8 @@ operation argument:
1037210374
- umin: ``*ptr = *ptr < val ? *ptr : val`` (using an unsigned comparison)
1037310375
- fadd: ``*ptr = *ptr + val`` (using floating point arithmetic)
1037410376
- fsub: ``*ptr = *ptr - val`` (using floating point arithmetic)
10377+
- fmax: ``*ptr = maxnum(*ptr, val)`` (match the `llvm.maxnum.*`` intrinsic)
10378+
- fmin: ``*ptr = minnum(*ptr, val)`` (match the `llvm.minnum.*`` intrinsic)
1037510379

1037610380
Example:
1037710381
""""""""

‎llvm/docs/ReleaseNotes.rst

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,9 @@ Changes to the LLVM IR
7676
* ``sdiv``
7777
* ``urem``
7878
* ``srem``
79+
* Added the support for ``fmax`` and ``fmin`` in ``atomicrmw`` instruction. The
80+
comparison is expected to match the behavior of ``llvm.maxnum.*`` and
81+
``llvm.minnum.*`` respectively.
7982

8083
Changes to building LLVM
8184
------------------------

‎llvm/include/llvm-c/Core.h

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -381,8 +381,14 @@ typedef enum {
381381
the old one */
382382
LLVMAtomicRMWBinOpFAdd, /**< Add a floating point value and return the
383383
old one */
384-
LLVMAtomicRMWBinOpFSub /**< Subtract a floating point value and return the
385-
old one */
384+
LLVMAtomicRMWBinOpFSub, /**< Subtract a floating point value and return the
385+
old one */
386+
LLVMAtomicRMWBinOpFMax, /**< Sets the value if it's greater than the
387+
original using an floating point comparison and
388+
return the old one */
389+
LLVMAtomicRMWBinOpFMin, /**< Sets the value if it's smaller than the
390+
original using an floating point comparison and
391+
return the old one */
386392
} LLVMAtomicRMWBinOp;
387393

388394
typedef enum {

‎llvm/include/llvm/AsmParser/LLToken.h

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,8 @@ enum Kind {
221221
kw_min,
222222
kw_umax,
223223
kw_umin,
224+
kw_fmax,
225+
kw_fmin,
224226

225227
// Instruction Opcodes (Opcode in UIntVal).
226228
kw_fneg,

‎llvm/include/llvm/Bitcode/LLVMBitCodes.h

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -458,7 +458,9 @@ enum RMWOperations {
458458
RMW_UMAX = 9,
459459
RMW_UMIN = 10,
460460
RMW_FADD = 11,
461-
RMW_FSUB = 12
461+
RMW_FSUB = 12,
462+
RMW_FMAX = 13,
463+
RMW_FMIN = 14
462464
};
463465

464466
/// OverflowingBinaryOperatorOptionalFlags - Flags for serializing

‎llvm/include/llvm/CodeGen/GlobalISel/MachineIRBuilder.h

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1401,6 +1401,40 @@ class MachineIRBuilder {
14011401
const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
14021402
MachineMemOperand &MMO);
14031403

1404+
/// Build and insert `OldValRes<def> = G_ATOMICRMW_FMAX Addr, Val, MMO`.
1405+
///
1406+
/// Atomically replace the value at \p Addr with the floating point maximum of
1407+
/// \p Val and the original value. Puts the original value from \p Addr in \p
1408+
/// OldValRes.
1409+
///
1410+
/// \pre setBasicBlock or setMI must have been called.
1411+
/// \pre \p OldValRes must be a generic virtual register.
1412+
/// \pre \p Addr must be a generic virtual register with pointer type.
1413+
/// \pre \p OldValRes, and \p Val must be generic virtual registers of the
1414+
/// same type.
1415+
///
1416+
/// \return a MachineInstrBuilder for the newly created instruction.
1417+
MachineInstrBuilder buildAtomicRMWFMax(
1418+
const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
1419+
MachineMemOperand &MMO);
1420+
1421+
/// Build and insert `OldValRes<def> = G_ATOMICRMW_FMIN Addr, Val, MMO`.
1422+
///
1423+
/// Atomically replace the value at \p Addr with the floating point minimum of
1424+
/// \p Val and the original value. Puts the original value from \p Addr in \p
1425+
/// OldValRes.
1426+
///
1427+
/// \pre setBasicBlock or setMI must have been called.
1428+
/// \pre \p OldValRes must be a generic virtual register.
1429+
/// \pre \p Addr must be a generic virtual register with pointer type.
1430+
/// \pre \p OldValRes, and \p Val must be generic virtual registers of the
1431+
/// same type.
1432+
///
1433+
/// \return a MachineInstrBuilder for the newly created instruction.
1434+
MachineInstrBuilder buildAtomicRMWFMin(
1435+
const DstOp &OldValRes, const SrcOp &Addr, const SrcOp &Val,
1436+
MachineMemOperand &MMO);
1437+
14041438
/// Build and insert `G_FENCE Ordering, Scope`.
14051439
MachineInstrBuilder buildFence(unsigned Ordering, unsigned Scope);
14061440

‎llvm/include/llvm/CodeGen/ISDOpcodes.h

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1195,6 +1195,8 @@ enum NodeType {
11951195
ATOMIC_LOAD_UMAX,
11961196
ATOMIC_LOAD_FADD,
11971197
ATOMIC_LOAD_FSUB,
1198+
ATOMIC_LOAD_FMAX,
1199+
ATOMIC_LOAD_FMIN,
11981200

11991201
// Masked load and store - consecutive vector load and store operations
12001202
// with additional mask operand that prevents memory accesses to the

‎llvm/include/llvm/CodeGen/SelectionDAGNodes.h

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1403,6 +1403,8 @@ class MemSDNode : public SDNode {
14031403
case ISD::ATOMIC_LOAD_UMAX:
14041404
case ISD::ATOMIC_LOAD_FADD:
14051405
case ISD::ATOMIC_LOAD_FSUB:
1406+
case ISD::ATOMIC_LOAD_FMAX:
1407+
case ISD::ATOMIC_LOAD_FMIN:
14061408
case ISD::ATOMIC_LOAD:
14071409
case ISD::ATOMIC_STORE:
14081410
case ISD::MLOAD:
@@ -1468,6 +1470,8 @@ class AtomicSDNode : public MemSDNode {
14681470
N->getOpcode() == ISD::ATOMIC_LOAD_UMAX ||
14691471
N->getOpcode() == ISD::ATOMIC_LOAD_FADD ||
14701472
N->getOpcode() == ISD::ATOMIC_LOAD_FSUB ||
1473+
N->getOpcode() == ISD::ATOMIC_LOAD_FMAX ||
1474+
N->getOpcode() == ISD::ATOMIC_LOAD_FMIN ||
14711475
N->getOpcode() == ISD::ATOMIC_LOAD ||
14721476
N->getOpcode() == ISD::ATOMIC_STORE;
14731477
}

‎llvm/include/llvm/IR/Instructions.h

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -753,8 +753,16 @@ class AtomicRMWInst : public Instruction {
753753
/// *p = old - v
754754
FSub,
755755

756+
/// *p = maxnum(old, v)
757+
/// \p maxnum matches the behavior of \p llvm.maxnum.*.
758+
FMax,
759+
760+
/// *p = minnum(old, v)
761+
/// \p minnum matches the behavior of \p llvm.minnum.*.
762+
FMin,
763+
756764
FIRST_BINOP = Xchg,
757-
LAST_BINOP = FSub,
765+
LAST_BINOP = FMin,
758766
BAD_BINOP
759767
};
760768

@@ -797,6 +805,8 @@ class AtomicRMWInst : public Instruction {
797805
switch (Op) {
798806
case AtomicRMWInst::FAdd:
799807
case AtomicRMWInst::FSub:
808+
case AtomicRMWInst::FMax:
809+
case AtomicRMWInst::FMin:
800810
return true;
801811
default:
802812
return false;

‎llvm/include/llvm/Support/TargetOpcodes.def

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -384,6 +384,8 @@ HANDLE_TARGET_OPCODE(G_ATOMICRMW_UMAX)
384384
HANDLE_TARGET_OPCODE(G_ATOMICRMW_UMIN)
385385
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FADD)
386386
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FSUB)
387+
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMAX)
388+
HANDLE_TARGET_OPCODE(G_ATOMICRMW_FMIN)
387389

388390
// Generic atomic fence
389391
HANDLE_TARGET_OPCODE(G_FENCE)

‎llvm/include/llvm/Target/GenericOpcodes.td

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1126,6 +1126,8 @@ def G_ATOMICRMW_UMAX : G_ATOMICRMW_OP;
11261126
def G_ATOMICRMW_UMIN : G_ATOMICRMW_OP;
11271127
def G_ATOMICRMW_FADD : G_ATOMICRMW_OP;
11281128
def G_ATOMICRMW_FSUB : G_ATOMICRMW_OP;
1129+
def G_ATOMICRMW_FMAX : G_ATOMICRMW_OP;
1130+
def G_ATOMICRMW_FMIN : G_ATOMICRMW_OP;
11291131

11301132
def G_FENCE : GenericInstruction {
11311133
let OutOperandList = (outs);

‎llvm/include/llvm/Target/GlobalISel/SelectionDAGCompat.td

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,8 @@ def : GINodeEquiv<G_ATOMICRMW_UMIN, atomic_load_umin>;
206206
def : GINodeEquiv<G_ATOMICRMW_UMAX, atomic_load_umax>;
207207
def : GINodeEquiv<G_ATOMICRMW_FADD, atomic_load_fadd>;
208208
def : GINodeEquiv<G_ATOMICRMW_FSUB, atomic_load_fsub>;
209+
def : GINodeEquiv<G_ATOMICRMW_FMAX, atomic_load_fmax>;
210+
def : GINodeEquiv<G_ATOMICRMW_FMIN, atomic_load_fmin>;
209211
def : GINodeEquiv<G_FENCE, atomic_fence>;
210212

211213
// Specifies the GlobalISel equivalents for SelectionDAG's ComplexPattern.

‎llvm/include/llvm/Target/TargetSelectionDAG.td

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -651,6 +651,10 @@ def atomic_load_fadd : SDNode<"ISD::ATOMIC_LOAD_FADD" , SDTFPAtomic2,
651651
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
652652
def atomic_load_fsub : SDNode<"ISD::ATOMIC_LOAD_FSUB" , SDTFPAtomic2,
653653
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
654+
def atomic_load_fmax : SDNode<"ISD::ATOMIC_LOAD_FMAX", SDTFPAtomic2,
655+
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
656+
def atomic_load_fmin : SDNode<"ISD::ATOMIC_LOAD_FMIN", SDTFPAtomic2,
657+
[SDNPHasChain, SDNPMayStore, SDNPMayLoad, SDNPMemOperand]>;
654658

655659
def atomic_load : SDNode<"ISD::ATOMIC_LOAD", SDTAtomicLoad,
656660
[SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;

‎llvm/lib/AsmParser/LLLexer.cpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -661,7 +661,7 @@ lltok::Kind LLLexer::LexIdentifier() {
661661
KEYWORD(oge); KEYWORD(ord); KEYWORD(uno); KEYWORD(ueq); KEYWORD(une);
662662

663663
KEYWORD(xchg); KEYWORD(nand); KEYWORD(max); KEYWORD(min); KEYWORD(umax);
664-
KEYWORD(umin);
664+
KEYWORD(umin); KEYWORD(fmax); KEYWORD(fmin);
665665

666666
KEYWORD(vscale);
667667
KEYWORD(x);

‎llvm/lib/AsmParser/LLParser.cpp

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7463,6 +7463,14 @@ int LLParser::parseAtomicRMW(Instruction *&Inst, PerFunctionState &PFS) {
74637463
Operation = AtomicRMWInst::FSub;
74647464
IsFP = true;
74657465
break;
7466+
case lltok::kw_fmax:
7467+
Operation = AtomicRMWInst::FMax;
7468+
IsFP = true;
7469+
break;
7470+
case lltok::kw_fmin:
7471+
Operation = AtomicRMWInst::FMin;
7472+
IsFP = true;
7473+
break;
74667474
}
74677475
Lex.Lex(); // Eat the operation.
74687476

‎llvm/lib/Bitcode/Reader/BitcodeReader.cpp

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1243,6 +1243,8 @@ static AtomicRMWInst::BinOp getDecodedRMWOperation(unsigned Val) {
12431243
case bitc::RMW_UMIN: return AtomicRMWInst::UMin;
12441244
case bitc::RMW_FADD: return AtomicRMWInst::FAdd;
12451245
case bitc::RMW_FSUB: return AtomicRMWInst::FSub;
1246+
case bitc::RMW_FMAX: return AtomicRMWInst::FMax;
1247+
case bitc::RMW_FMIN: return AtomicRMWInst::FMin;
12461248
}
12471249
}
12481250

‎llvm/lib/Bitcode/Writer/BitcodeWriter.cpp

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -577,6 +577,8 @@ static unsigned getEncodedRMWOperation(AtomicRMWInst::BinOp Op) {
577577
case AtomicRMWInst::UMin: return bitc::RMW_UMIN;
578578
case AtomicRMWInst::FAdd: return bitc::RMW_FADD;
579579
case AtomicRMWInst::FSub: return bitc::RMW_FSUB;
580+
case AtomicRMWInst::FMax: return bitc::RMW_FMAX;
581+
case AtomicRMWInst::FMin: return bitc::RMW_FMIN;
580582
}
581583
}
582584

‎llvm/lib/CodeGen/AtomicExpandPass.cpp

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1646,6 +1646,8 @@ static ArrayRef<RTLIB::Libcall> GetRMWLibcall(AtomicRMWInst::BinOp Op) {
16461646
case AtomicRMWInst::Min:
16471647
case AtomicRMWInst::UMax:
16481648
case AtomicRMWInst::UMin:
1649+
case AtomicRMWInst::FMax:
1650+
case AtomicRMWInst::FMin:
16491651
case AtomicRMWInst::FAdd:
16501652
case AtomicRMWInst::FSub:
16511653
// No atomic libcalls are available for max/min/umax/umin.

‎llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2883,6 +2883,12 @@ bool IRTranslator::translateAtomicRMW(const User &U,
28832883
case AtomicRMWInst::FSub:
28842884
Opcode = TargetOpcode::G_ATOMICRMW_FSUB;
28852885
break;
2886+
case AtomicRMWInst::FMax:
2887+
Opcode = TargetOpcode::G_ATOMICRMW_FMAX;
2888+
break;
2889+
case AtomicRMWInst::FMin:
2890+
Opcode = TargetOpcode::G_ATOMICRMW_FMIN;
2891+
break;
28862892
}
28872893

28882894
MIRBuilder.buildAtomicRMW(

‎llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -937,6 +937,20 @@ MachineIRBuilder::buildAtomicRMWFSub(const DstOp &OldValRes, const SrcOp &Addr,
937937
MMO);
938938
}
939939

940+
MachineInstrBuilder
941+
MachineIRBuilder::buildAtomicRMWFMax(const DstOp &OldValRes, const SrcOp &Addr,
942+
const SrcOp &Val, MachineMemOperand &MMO) {
943+
return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMAX, OldValRes, Addr, Val,
944+
MMO);
945+
}
946+
947+
MachineInstrBuilder
948+
MachineIRBuilder::buildAtomicRMWFMin(const DstOp &OldValRes, const SrcOp &Addr,
949+
const SrcOp &Val, MachineMemOperand &MMO) {
950+
return buildAtomicRMW(TargetOpcode::G_ATOMICRMW_FMIN, OldValRes, Addr, Val,
951+
MMO);
952+
}
953+
940954
MachineInstrBuilder
941955
MachineIRBuilder::buildFence(unsigned Ordering, unsigned Scope) {
942956
return buildInstr(TargetOpcode::G_FENCE)

‎llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7505,6 +7505,8 @@ SDValue SelectionDAG::getAtomic(unsigned Opcode, const SDLoc &dl, EVT MemVT,
75057505
Opcode == ISD::ATOMIC_LOAD_UMAX ||
75067506
Opcode == ISD::ATOMIC_LOAD_FADD ||
75077507
Opcode == ISD::ATOMIC_LOAD_FSUB ||
7508+
Opcode == ISD::ATOMIC_LOAD_FMAX ||
7509+
Opcode == ISD::ATOMIC_LOAD_FMIN ||
75087510
Opcode == ISD::ATOMIC_SWAP ||
75097511
Opcode == ISD::ATOMIC_STORE) &&
75107512
"Invalid Atomic Op");

‎llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4611,6 +4611,8 @@ void SelectionDAGBuilder::visitAtomicRMW(const AtomicRMWInst &I) {
46114611
case AtomicRMWInst::UMin: NT = ISD::ATOMIC_LOAD_UMIN; break;
46124612
case AtomicRMWInst::FAdd: NT = ISD::ATOMIC_LOAD_FADD; break;
46134613
case AtomicRMWInst::FSub: NT = ISD::ATOMIC_LOAD_FSUB; break;
4614+
case AtomicRMWInst::FMax: NT = ISD::ATOMIC_LOAD_FMAX; break;
4615+
case AtomicRMWInst::FMin: NT = ISD::ATOMIC_LOAD_FMIN; break;
46144616
}
46154617
AtomicOrdering Ordering = I.getOrdering();
46164618
SyncScope::ID SSID = I.getSyncScopeID();

‎llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3962,6 +3962,8 @@ Value *OpenMPIRBuilder::emitRMWOpAsInstruction(Value *Src1, Value *Src2,
39623962
case AtomicRMWInst::Min:
39633963
case AtomicRMWInst::UMax:
39643964
case AtomicRMWInst::UMin:
3965+
case AtomicRMWInst::FMax:
3966+
case AtomicRMWInst::FMin:
39653967
llvm_unreachable("Unsupported atomic update operation");
39663968
}
39673969
llvm_unreachable("Unsupported atomic update operation");

‎llvm/lib/IR/Core.cpp

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3620,6 +3620,8 @@ static AtomicRMWInst::BinOp mapFromLLVMRMWBinOp(LLVMAtomicRMWBinOp BinOp) {
36203620
case LLVMAtomicRMWBinOpUMin: return AtomicRMWInst::UMin;
36213621
case LLVMAtomicRMWBinOpFAdd: return AtomicRMWInst::FAdd;
36223622
case LLVMAtomicRMWBinOpFSub: return AtomicRMWInst::FSub;
3623+
case LLVMAtomicRMWBinOpFMax: return AtomicRMWInst::FMax;
3624+
case LLVMAtomicRMWBinOpFMin: return AtomicRMWInst::FMin;
36233625
}
36243626

36253627
llvm_unreachable("Invalid LLVMAtomicRMWBinOp value!");
@@ -3640,6 +3642,8 @@ static LLVMAtomicRMWBinOp mapToLLVMRMWBinOp(AtomicRMWInst::BinOp BinOp) {
36403642
case AtomicRMWInst::UMin: return LLVMAtomicRMWBinOpUMin;
36413643
case AtomicRMWInst::FAdd: return LLVMAtomicRMWBinOpFAdd;
36423644
case AtomicRMWInst::FSub: return LLVMAtomicRMWBinOpFSub;
3645+
case AtomicRMWInst::FMax: return LLVMAtomicRMWBinOpFMax;
3646+
case AtomicRMWInst::FMin: return LLVMAtomicRMWBinOpFMin;
36433647
default: break;
36443648
}
36453649

‎llvm/lib/IR/Instructions.cpp

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1696,6 +1696,10 @@ StringRef AtomicRMWInst::getOperationName(BinOp Op) {
16961696
return "fadd";
16971697
case AtomicRMWInst::FSub:
16981698
return "fsub";
1699+
case AtomicRMWInst::FMax:
1700+
return "fmax";
1701+
case AtomicRMWInst::FMin:
1702+
return "fmin";
16991703
case AtomicRMWInst::BAD_BINOP:
17001704
return "<invalid operation>";
17011705
}

‎llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4803,6 +4803,8 @@ AMDGPUTargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *RMW) const {
48034803
case AtomicRMWInst::Nand:
48044804
case AtomicRMWInst::FAdd:
48054805
case AtomicRMWInst::FSub:
4806+
case AtomicRMWInst::FMax:
4807+
case AtomicRMWInst::FMin:
48064808
return AtomicExpansionKind::CmpXChg;
48074809
default:
48084810
return AtomicExpansionKind::None;

‎llvm/lib/Target/DirectX/DXILWriter/DXILBitcodeWriter.cpp

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -595,6 +595,10 @@ unsigned DXILBitcodeWriter::getEncodedRMWOperation(AtomicRMWInst::BinOp Op) {
595595
return bitc::RMW_FADD;
596596
case AtomicRMWInst::FSub:
597597
return bitc::RMW_FSUB;
598+
case AtomicRMWInst::FMax:
599+
return bitc::RMW_FMAX;
600+
case AtomicRMWInst::FMin:
601+
return bitc::RMW_FMIN;
598602
}
599603
}
600604

‎llvm/lib/Target/X86/X86ISelLowering.cpp

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30800,6 +30800,8 @@ X86TargetLowering::shouldExpandAtomicRMWInIR(AtomicRMWInst *AI) const {
3080030800
case AtomicRMWInst::UMin:
3080130801
case AtomicRMWInst::FAdd:
3080230802
case AtomicRMWInst::FSub:
30803+
case AtomicRMWInst::FMax:
30804+
case AtomicRMWInst::FMin:
3080330805
// These always require a non-trivial set of data operations on x86. We must
3080430806
// use a cmpxchg loop.
3080530807
return AtomicExpansionKind::CmpXChg;

0 commit comments

Comments
 (0)