[mlir][amdgpu] Expose waitcnt bitpacking infra #172313
Merged
+219
−222
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
@krzysz00 I'm trying to expose AMDGPU backend waitcnt bitpacking info so we can get rid of our copy in |
krzysz00
reviewed
Dec 15, 2025
Contributor
krzysz00
left a comment
krzysz00
left a comment
There was a problem hiding this comment.
I'd let the LLVM forks comment on this one too
Member
|
@llvm/pr-subscribers-llvm-support @llvm/pr-subscribers-backend-amdgpu Author: Ivan Butygin (Hardcode84) ChangesPatch is 20.00 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/172313.diff 3 Files Affected:
diff --git a/llvm/include/llvm/TargetParser/TargetParser.h b/llvm/include/llvm/TargetParser/TargetParser.h
index 9dfa50c1ad1ba..263f4036d9b2f 100644
--- a/llvm/include/llvm/TargetParser/TargetParser.h
+++ b/llvm/include/llvm/TargetParser/TargetParser.h
@@ -192,6 +192,192 @@ LLVM_ABI IsaVersion getIsaVersion(StringRef GPU);
/// default target features with entries overridden by \p Features.
LLVM_ABI std::pair<FeatureError, StringRef>
fillAMDGPUFeatureMap(StringRef GPU, const Triple &T, StringMap<bool> &Features);
+
+/// Represents the counter values to wait for in an s_waitcnt instruction.
+///
+/// Large values (including the maximum possible integer) can be used to
+/// represent "don't care" waits.
+struct Waitcnt {
+ unsigned LoadCnt = ~0u; // Corresponds to Vmcnt prior to gfx12.
+ unsigned ExpCnt = ~0u;
+ unsigned DsCnt = ~0u; // Corresponds to LGKMcnt prior to gfx12.
+ unsigned StoreCnt = ~0u; // Corresponds to VScnt on gfx10/gfx11.
+ unsigned SampleCnt = ~0u; // gfx12+ only.
+ unsigned BvhCnt = ~0u; // gfx12+ only.
+ unsigned KmCnt = ~0u; // gfx12+ only.
+ unsigned XCnt = ~0u; // gfx1250.
+
+ Waitcnt() = default;
+ // Pre-gfx12 constructor.
+ Waitcnt(unsigned VmCnt, unsigned ExpCnt, unsigned LgkmCnt, unsigned VsCnt)
+ : LoadCnt(VmCnt), ExpCnt(ExpCnt), DsCnt(LgkmCnt), StoreCnt(VsCnt) {}
+
+ // gfx12+ constructor.
+ Waitcnt(unsigned LoadCnt, unsigned ExpCnt, unsigned DsCnt, unsigned StoreCnt,
+ unsigned SampleCnt, unsigned BvhCnt, unsigned KmCnt, unsigned XCnt)
+ : LoadCnt(LoadCnt), ExpCnt(ExpCnt), DsCnt(DsCnt), StoreCnt(StoreCnt),
+ SampleCnt(SampleCnt), BvhCnt(BvhCnt), KmCnt(KmCnt), XCnt(XCnt) {}
+
+ bool hasWait() const { return StoreCnt != ~0u || hasWaitExceptStoreCnt(); }
+
+ bool hasWaitExceptStoreCnt() const {
+ return LoadCnt != ~0u || ExpCnt != ~0u || DsCnt != ~0u ||
+ SampleCnt != ~0u || BvhCnt != ~0u || KmCnt != ~0u || XCnt != ~0u;
+ }
+
+ bool hasWaitStoreCnt() const { return StoreCnt != ~0u; }
+
+ Waitcnt combined(const Waitcnt &Other) const {
+ // Does the right thing provided self and Other are either both pre-gfx12
+ // or both gfx12+.
+ return Waitcnt(
+ std::min(LoadCnt, Other.LoadCnt), std::min(ExpCnt, Other.ExpCnt),
+ std::min(DsCnt, Other.DsCnt), std::min(StoreCnt, Other.StoreCnt),
+ std::min(SampleCnt, Other.SampleCnt), std::min(BvhCnt, Other.BvhCnt),
+ std::min(KmCnt, Other.KmCnt), std::min(XCnt, Other.XCnt));
+ }
+
+ friend raw_ostream &operator<<(raw_ostream &OS, const AMDGPU::Waitcnt &Wait);
+};
+
+// The following methods are only meaningful on targets that support
+// S_WAITCNT.
+
+/// \returns Vmcnt bit mask for given isa \p Version.
+LLVM_ABI unsigned getVmcntBitMask(const IsaVersion &Version);
+
+/// \returns Expcnt bit mask for given isa \p Version.
+LLVM_ABI unsigned getExpcntBitMask(const IsaVersion &Version);
+
+/// \returns Lgkmcnt bit mask for given isa \p Version.
+LLVM_ABI unsigned getLgkmcntBitMask(const IsaVersion &Version);
+
+/// \returns Waitcnt bit mask for given isa \p Version.
+LLVM_ABI unsigned getWaitcntBitMask(const IsaVersion &Version);
+
+/// \returns Decoded Vmcnt from given \p Waitcnt for given isa \p Version.
+LLVM_ABI unsigned decodeVmcnt(const IsaVersion &Version, unsigned Waitcnt);
+
+/// \returns Decoded Expcnt from given \p Waitcnt for given isa \p Version.
+LLVM_ABI unsigned decodeExpcnt(const IsaVersion &Version, unsigned Waitcnt);
+
+/// \returns Decoded Lgkmcnt from given \p Waitcnt for given isa \p Version.
+LLVM_ABI unsigned decodeLgkmcnt(const IsaVersion &Version, unsigned Waitcnt);
+
+/// Decodes Vmcnt, Expcnt and Lgkmcnt from given \p Waitcnt for given isa
+/// \p Version, and writes decoded values into \p Vmcnt, \p Expcnt and
+/// \p Lgkmcnt respectively. Should not be used on gfx12+, the instruction
+/// which needs it is deprecated
+///
+/// \details \p Vmcnt, \p Expcnt and \p Lgkmcnt are decoded as follows:
+/// \p Vmcnt = \p Waitcnt[3:0] (pre-gfx9)
+/// \p Vmcnt = \p Waitcnt[15:14,3:0] (gfx9,10)
+/// \p Vmcnt = \p Waitcnt[15:10] (gfx11)
+/// \p Expcnt = \p Waitcnt[6:4] (pre-gfx11)
+/// \p Expcnt = \p Waitcnt[2:0] (gfx11)
+/// \p Lgkmcnt = \p Waitcnt[11:8] (pre-gfx10)
+/// \p Lgkmcnt = \p Waitcnt[13:8] (gfx10)
+/// \p Lgkmcnt = \p Waitcnt[9:4] (gfx11)
+///
+LLVM_ABI void decodeWaitcnt(const IsaVersion &Version, unsigned Waitcnt,
+ unsigned &Vmcnt, unsigned &Expcnt,
+ unsigned &Lgkmcnt);
+
+LLVM_ABI Waitcnt decodeWaitcnt(const IsaVersion &Version, unsigned Encoded);
+
+/// \returns \p Waitcnt with encoded \p Vmcnt for given isa \p Version.
+LLVM_ABI unsigned encodeVmcnt(const IsaVersion &Version, unsigned Waitcnt,
+ unsigned Vmcnt);
+
+/// \returns \p Waitcnt with encoded \p Expcnt for given isa \p Version.
+LLVM_ABI unsigned encodeExpcnt(const IsaVersion &Version, unsigned Waitcnt,
+ unsigned Expcnt);
+
+/// \returns \p Waitcnt with encoded \p Lgkmcnt for given isa \p Version.
+LLVM_ABI unsigned encodeLgkmcnt(const IsaVersion &Version, unsigned Waitcnt,
+ unsigned Lgkmcnt);
+
+/// Encodes \p Vmcnt, \p Expcnt and \p Lgkmcnt into Waitcnt for given isa
+/// \p Version. Should not be used on gfx12+, the instruction which needs
+/// it is deprecated
+///
+/// \details \p Vmcnt, \p Expcnt and \p Lgkmcnt are encoded as follows:
+/// Waitcnt[2:0] = \p Expcnt (gfx11+)
+/// Waitcnt[3:0] = \p Vmcnt (pre-gfx9)
+/// Waitcnt[3:0] = \p Vmcnt[3:0] (gfx9,10)
+/// Waitcnt[6:4] = \p Expcnt (pre-gfx11)
+/// Waitcnt[9:4] = \p Lgkmcnt (gfx11)
+/// Waitcnt[11:8] = \p Lgkmcnt (pre-gfx10)
+/// Waitcnt[13:8] = \p Lgkmcnt (gfx10)
+/// Waitcnt[15:10] = \p Vmcnt (gfx11)
+/// Waitcnt[15:14] = \p Vmcnt[5:4] (gfx9,10)
+///
+/// \returns Waitcnt with encoded \p Vmcnt, \p Expcnt and \p Lgkmcnt for given
+/// isa \p Version.
+///
+LLVM_ABI unsigned encodeWaitcnt(const IsaVersion &Version, unsigned Vmcnt,
+ unsigned Expcnt, unsigned Lgkmcnt);
+
+LLVM_ABI unsigned encodeWaitcnt(const IsaVersion &Version,
+ const Waitcnt &Decoded);
+
+// The following methods are only meaningful on targets that support
+// S_WAIT_*CNT, introduced with gfx12.
+
+/// \returns Loadcnt bit mask for given isa \p Version.
+/// Returns 0 for versions that do not support LOADcnt
+LLVM_ABI unsigned getLoadcntBitMask(const IsaVersion &Version);
+
+/// \returns Samplecnt bit mask for given isa \p Version.
+/// Returns 0 for versions that do not support SAMPLEcnt
+LLVM_ABI unsigned getSamplecntBitMask(const IsaVersion &Version);
+
+/// \returns Bvhcnt bit mask for given isa \p Version.
+/// Returns 0 for versions that do not support BVHcnt
+LLVM_ABI unsigned getBvhcntBitMask(const IsaVersion &Version);
+
+/// \returns Dscnt bit mask for given isa \p Version.
+/// Returns 0 for versions that do not support DScnt
+LLVM_ABI unsigned getDscntBitMask(const IsaVersion &Version);
+
+/// \returns Dscnt bit mask for given isa \p Version.
+/// Returns 0 for versions that do not support KMcnt
+LLVM_ABI unsigned getKmcntBitMask(const IsaVersion &Version);
+
+/// \returns Xcnt bit mask for given isa \p Version.
+/// Returns 0 for versions that do not support Xcnt.
+LLVM_ABI unsigned getXcntBitMask(const IsaVersion &Version);
+
+/// \return STOREcnt or VScnt bit mask for given isa \p Version.
+/// returns 0 for versions that do not support STOREcnt or VScnt.
+/// STOREcnt and VScnt are the same counter, the name used
+/// depends on the ISA version.
+LLVM_ABI unsigned getStorecntBitMask(const IsaVersion &Version);
+
+// The following are only meaningful on targets that support
+// S_WAIT_LOADCNT_DSCNT and S_WAIT_STORECNT_DSCNT.
+
+/// \returns Decoded Waitcnt structure from given \p LoadcntDscnt for given
+/// isa \p Version.
+LLVM_ABI Waitcnt decodeLoadcntDscnt(const IsaVersion &Version,
+ unsigned LoadcntDscnt);
+
+/// \returns Decoded Waitcnt structure from given \p StorecntDscnt for given
+/// isa \p Version.
+LLVM_ABI Waitcnt decodeStorecntDscnt(const IsaVersion &Version,
+ unsigned StorecntDscnt);
+
+/// \returns \p Loadcnt and \p Dscnt components of \p Decoded encoded as an
+/// immediate that can be used with S_WAIT_LOADCNT_DSCNT for given isa
+/// \p Version.
+LLVM_ABI unsigned encodeLoadcntDscnt(const IsaVersion &Version,
+ const Waitcnt &Decoded);
+
+/// \returns \p Storecnt and \p Dscnt components of \p Decoded encoded as an
+/// immediate that can be used with S_WAIT_STORECNT_DSCNT for given isa
+/// \p Version.
+LLVM_ABI unsigned encodeStorecntDscnt(const IsaVersion &Version,
+ const Waitcnt &Decoded);
} // namespace AMDGPU
struct BasicSubtargetFeatureKV {
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
index 75db58a292c13..2022a3d32b5da 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
@@ -1076,186 +1076,6 @@ getIntegerVecAttribute(const Function &F, StringRef Name, unsigned Size);
/// Checks if \p Val is inside \p MD, a !range-like metadata.
bool hasValueInRangeLikeMetadata(const MDNode &MD, int64_t Val);
-/// Represents the counter values to wait for in an s_waitcnt instruction.
-///
-/// Large values (including the maximum possible integer) can be used to
-/// represent "don't care" waits.
-struct Waitcnt {
- unsigned LoadCnt = ~0u; // Corresponds to Vmcnt prior to gfx12.
- unsigned ExpCnt = ~0u;
- unsigned DsCnt = ~0u; // Corresponds to LGKMcnt prior to gfx12.
- unsigned StoreCnt = ~0u; // Corresponds to VScnt on gfx10/gfx11.
- unsigned SampleCnt = ~0u; // gfx12+ only.
- unsigned BvhCnt = ~0u; // gfx12+ only.
- unsigned KmCnt = ~0u; // gfx12+ only.
- unsigned XCnt = ~0u; // gfx1250.
-
- Waitcnt() = default;
- // Pre-gfx12 constructor.
- Waitcnt(unsigned VmCnt, unsigned ExpCnt, unsigned LgkmCnt, unsigned VsCnt)
- : LoadCnt(VmCnt), ExpCnt(ExpCnt), DsCnt(LgkmCnt), StoreCnt(VsCnt) {}
-
- // gfx12+ constructor.
- Waitcnt(unsigned LoadCnt, unsigned ExpCnt, unsigned DsCnt, unsigned StoreCnt,
- unsigned SampleCnt, unsigned BvhCnt, unsigned KmCnt, unsigned XCnt)
- : LoadCnt(LoadCnt), ExpCnt(ExpCnt), DsCnt(DsCnt), StoreCnt(StoreCnt),
- SampleCnt(SampleCnt), BvhCnt(BvhCnt), KmCnt(KmCnt), XCnt(XCnt) {}
-
- bool hasWait() const { return StoreCnt != ~0u || hasWaitExceptStoreCnt(); }
-
- bool hasWaitExceptStoreCnt() const {
- return LoadCnt != ~0u || ExpCnt != ~0u || DsCnt != ~0u ||
- SampleCnt != ~0u || BvhCnt != ~0u || KmCnt != ~0u || XCnt != ~0u;
- }
-
- bool hasWaitStoreCnt() const { return StoreCnt != ~0u; }
-
- Waitcnt combined(const Waitcnt &Other) const {
- // Does the right thing provided self and Other are either both pre-gfx12
- // or both gfx12+.
- return Waitcnt(
- std::min(LoadCnt, Other.LoadCnt), std::min(ExpCnt, Other.ExpCnt),
- std::min(DsCnt, Other.DsCnt), std::min(StoreCnt, Other.StoreCnt),
- std::min(SampleCnt, Other.SampleCnt), std::min(BvhCnt, Other.BvhCnt),
- std::min(KmCnt, Other.KmCnt), std::min(XCnt, Other.XCnt));
- }
-
- friend raw_ostream &operator<<(raw_ostream &OS, const AMDGPU::Waitcnt &Wait);
-};
-
-// The following methods are only meaningful on targets that support
-// S_WAITCNT.
-
-/// \returns Vmcnt bit mask for given isa \p Version.
-unsigned getVmcntBitMask(const IsaVersion &Version);
-
-/// \returns Expcnt bit mask for given isa \p Version.
-unsigned getExpcntBitMask(const IsaVersion &Version);
-
-/// \returns Lgkmcnt bit mask for given isa \p Version.
-unsigned getLgkmcntBitMask(const IsaVersion &Version);
-
-/// \returns Waitcnt bit mask for given isa \p Version.
-unsigned getWaitcntBitMask(const IsaVersion &Version);
-
-/// \returns Decoded Vmcnt from given \p Waitcnt for given isa \p Version.
-unsigned decodeVmcnt(const IsaVersion &Version, unsigned Waitcnt);
-
-/// \returns Decoded Expcnt from given \p Waitcnt for given isa \p Version.
-unsigned decodeExpcnt(const IsaVersion &Version, unsigned Waitcnt);
-
-/// \returns Decoded Lgkmcnt from given \p Waitcnt for given isa \p Version.
-unsigned decodeLgkmcnt(const IsaVersion &Version, unsigned Waitcnt);
-
-/// Decodes Vmcnt, Expcnt and Lgkmcnt from given \p Waitcnt for given isa
-/// \p Version, and writes decoded values into \p Vmcnt, \p Expcnt and
-/// \p Lgkmcnt respectively. Should not be used on gfx12+, the instruction
-/// which needs it is deprecated
-///
-/// \details \p Vmcnt, \p Expcnt and \p Lgkmcnt are decoded as follows:
-/// \p Vmcnt = \p Waitcnt[3:0] (pre-gfx9)
-/// \p Vmcnt = \p Waitcnt[15:14,3:0] (gfx9,10)
-/// \p Vmcnt = \p Waitcnt[15:10] (gfx11)
-/// \p Expcnt = \p Waitcnt[6:4] (pre-gfx11)
-/// \p Expcnt = \p Waitcnt[2:0] (gfx11)
-/// \p Lgkmcnt = \p Waitcnt[11:8] (pre-gfx10)
-/// \p Lgkmcnt = \p Waitcnt[13:8] (gfx10)
-/// \p Lgkmcnt = \p Waitcnt[9:4] (gfx11)
-///
-void decodeWaitcnt(const IsaVersion &Version, unsigned Waitcnt, unsigned &Vmcnt,
- unsigned &Expcnt, unsigned &Lgkmcnt);
-
-Waitcnt decodeWaitcnt(const IsaVersion &Version, unsigned Encoded);
-
-/// \returns \p Waitcnt with encoded \p Vmcnt for given isa \p Version.
-unsigned encodeVmcnt(const IsaVersion &Version, unsigned Waitcnt,
- unsigned Vmcnt);
-
-/// \returns \p Waitcnt with encoded \p Expcnt for given isa \p Version.
-unsigned encodeExpcnt(const IsaVersion &Version, unsigned Waitcnt,
- unsigned Expcnt);
-
-/// \returns \p Waitcnt with encoded \p Lgkmcnt for given isa \p Version.
-unsigned encodeLgkmcnt(const IsaVersion &Version, unsigned Waitcnt,
- unsigned Lgkmcnt);
-
-/// Encodes \p Vmcnt, \p Expcnt and \p Lgkmcnt into Waitcnt for given isa
-/// \p Version. Should not be used on gfx12+, the instruction which needs
-/// it is deprecated
-///
-/// \details \p Vmcnt, \p Expcnt and \p Lgkmcnt are encoded as follows:
-/// Waitcnt[2:0] = \p Expcnt (gfx11+)
-/// Waitcnt[3:0] = \p Vmcnt (pre-gfx9)
-/// Waitcnt[3:0] = \p Vmcnt[3:0] (gfx9,10)
-/// Waitcnt[6:4] = \p Expcnt (pre-gfx11)
-/// Waitcnt[9:4] = \p Lgkmcnt (gfx11)
-/// Waitcnt[11:8] = \p Lgkmcnt (pre-gfx10)
-/// Waitcnt[13:8] = \p Lgkmcnt (gfx10)
-/// Waitcnt[15:10] = \p Vmcnt (gfx11)
-/// Waitcnt[15:14] = \p Vmcnt[5:4] (gfx9,10)
-///
-/// \returns Waitcnt with encoded \p Vmcnt, \p Expcnt and \p Lgkmcnt for given
-/// isa \p Version.
-///
-unsigned encodeWaitcnt(const IsaVersion &Version, unsigned Vmcnt,
- unsigned Expcnt, unsigned Lgkmcnt);
-
-unsigned encodeWaitcnt(const IsaVersion &Version, const Waitcnt &Decoded);
-
-// The following methods are only meaningful on targets that support
-// S_WAIT_*CNT, introduced with gfx12.
-
-/// \returns Loadcnt bit mask for given isa \p Version.
-/// Returns 0 for versions that do not support LOADcnt
-unsigned getLoadcntBitMask(const IsaVersion &Version);
-
-/// \returns Samplecnt bit mask for given isa \p Version.
-/// Returns 0 for versions that do not support SAMPLEcnt
-unsigned getSamplecntBitMask(const IsaVersion &Version);
-
-/// \returns Bvhcnt bit mask for given isa \p Version.
-/// Returns 0 for versions that do not support BVHcnt
-unsigned getBvhcntBitMask(const IsaVersion &Version);
-
-/// \returns Dscnt bit mask for given isa \p Version.
-/// Returns 0 for versions that do not support DScnt
-unsigned getDscntBitMask(const IsaVersion &Version);
-
-/// \returns Dscnt bit mask for given isa \p Version.
-/// Returns 0 for versions that do not support KMcnt
-unsigned getKmcntBitMask(const IsaVersion &Version);
-
-/// \returns Xcnt bit mask for given isa \p Version.
-/// Returns 0 for versions that do not support Xcnt.
-unsigned getXcntBitMask(const IsaVersion &Version);
-
-/// \return STOREcnt or VScnt bit mask for given isa \p Version.
-/// returns 0 for versions that do not support STOREcnt or VScnt.
-/// STOREcnt and VScnt are the same counter, the name used
-/// depends on the ISA version.
-unsigned getStorecntBitMask(const IsaVersion &Version);
-
-// The following are only meaningful on targets that support
-// S_WAIT_LOADCNT_DSCNT and S_WAIT_STORECNT_DSCNT.
-
-/// \returns Decoded Waitcnt structure from given \p LoadcntDscnt for given
-/// isa \p Version.
-Waitcnt decodeLoadcntDscnt(const IsaVersion &Version, unsigned LoadcntDscnt);
-
-/// \returns Decoded Waitcnt structure from given \p StorecntDscnt for given
-/// isa \p Version.
-Waitcnt decodeStorecntDscnt(const IsaVersion &Version, unsigned StorecntDscnt);
-
-/// \returns \p Loadcnt and \p Dscnt components of \p Decoded encoded as an
-/// immediate that can be used with S_WAIT_LOADCNT_DSCNT for given isa
-/// \p Version.
-unsigned encodeLoadcntDscnt(const IsaVersion &Version, const Waitcnt &Decoded);
-
-/// \returns \p Storecnt and \p Dscnt components of \p Decoded encoded as an
-/// immediate that can be used with S_WAIT_STORECNT_DSCNT for given isa
-/// \p Version.
-unsigned encodeStorecntDscnt(const IsaVersion &Version, const Waitcnt &Decoded);
-
namespace Hwreg {
using HwregId = EncodingField<5, 0>;
diff --git a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
index af825ad42dd82..ba8ce999eee17 100644
--- a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
+++ b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
@@ -30,6 +30,7 @@
#include "llvm/ADT/TypeSwitch.h"
#include "llvm/Support/Casting.h"
#include "llvm/Support/ErrorHandling.h"
+#include "llvm/TargetParser/TargetParser.h"
#include <optional>
namespace mlir {
@@ -435,50 +436,18 @@ struct RawBufferOpLowering : public ConvertOpToLLVMPattern<GpuOp> {
}
};
-// TODO: AMDGPU backend already have all this bitpacking logic, we should move
-// it to some common place.
-/// Vmcnt, Expcnt and Lgkmcnt are decoded as follows:
-/// Vmcnt = Waitcnt[3:0] (pre-gfx9)
-/// Vmcnt = Waitcnt[15:14,3:0] (gfx9,10)
-/// Vmcnt = Waitcnt[15:10] (gfx11)
-/// Expcnt = Waitcnt[6:4] (pre-gfx11)
-/// Expcnt = Waitcnt[2:0] (gfx11)
-/// Lgkmcnt = Waitcnt[11:8] (pre-gfx10)
-/// Lgkmcnt = Waitcnt[13:8] (gfx10)
-/// Lgkmcnt = Waitcnt[9:4] (gfx11)
static FailureOr<unsigned> encodeWaitcnt(Chipset chipset, unsigned vmcnt,
unsigned expcnt, unsigned lgkmcnt) {
- if (chipset.majorVersion < 9) {
- vmcnt = std::min(15u, vmcnt);
- expcnt = std::min(7u, expcnt);
- lgkmcnt = std::min(15u, lgkmcnt);
- return vmcnt | (expcnt << 4) | (lgkmcnt << 8);
- }
- if (chipset.majorVersion == 9) {
- vmcnt = std::min(63u, vmcnt);
- expcnt = std::min(7u, expcnt);
- lgkmcnt = std::min(15u, lgkmcnt);
- unsigned lowBits = vmcnt & 0xF;
- unsigned highBits = (vmcnt >> 4) << 14;
- unsigned otherCnts = (expcnt << 4) | (lgkmcnt << 8);
- return lowBits | highBits | otherCnts;
- }
- if (chipset.majorVersion == 10) {
- vmcnt = std::min(63u, vmcnt);
- expcnt = std::min(7u, expcnt);
- lgkmcnt = std::min(63u, lgkmcnt);
- unsigned lowBits = vmcnt & 0xF;
- unsigned highBits = (vmcnt >> 4) << 14;
- unsigned otherCnts = (expcnt << 4) | (lgkmcnt << 8);
- return lowBits | highBits | otherCnts;
- }
- if (chipset.majorVersion == 11) {
- vmcnt = std::min(63u, vmcnt);
- expcnt = std::min(7u, expcnt);
- lgkmcnt = std::min(63u, lgkmcnt);
- return (vmcnt << 10) | expcnt | (lgkmcnt << 4);
- }
- return failure();
+ if (chipset.majorVersion >= 12)
+ return failure();
+
+ llvm::AMDGPU::IsaVersion isaVersion{
+ chipset.majorVersion, chipset.minorVersion, chipset.steppingVersion};
+ vmcnt = std::min(vmcnt, llvm::AMDGPU::getVmcntBitMask(isaVersion));
+ expcnt = std::min(expcnt, llvm::AMDGPU::getExpcntBitMask(isaVersion));
+ lgkmcnt = std::min(lgkmcnt, llvm::AMDGPU::getLgkmcntBitMask(isaVersion));
+
+ return llvm::AMDGPU::encodeWaitcnt(isaVersion, vmcnt, expcnt, lgkmcnt);
}
struct MemoryCounterWaitOpLoweri...
[truncated]
|
Member
|
@llvm/pr-subscribers-mlir-gpu Author: Ivan Butygin (Hardcode84) ChangesPatch is 20.00 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/172313.diff 3 Files Affected:
diff --git a/llvm/include/llvm/TargetParser/TargetParser.h b/llvm/include/llvm/TargetParser/TargetParser.h
index 9dfa50c1ad1ba..263f4036d9b2f 100644
--- a/llvm/include/llvm/TargetParser/TargetParser.h
+++ b/llvm/include/llvm/TargetParser/TargetParser.h
@@ -192,6 +192,192 @@ LLVM_ABI IsaVersion getIsaVersion(StringRef GPU);
/// default target features with entries overridden by \p Features.
LLVM_ABI std::pair<FeatureError, StringRef>
fillAMDGPUFeatureMap(StringRef GPU, const Triple &T, StringMap<bool> &Features);
+
+/// Represents the counter values to wait for in an s_waitcnt instruction.
+///
+/// Large values (including the maximum possible integer) can be used to
+/// represent "don't care" waits.
+struct Waitcnt {
+ unsigned LoadCnt = ~0u; // Corresponds to Vmcnt prior to gfx12.
+ unsigned ExpCnt = ~0u;
+ unsigned DsCnt = ~0u; // Corresponds to LGKMcnt prior to gfx12.
+ unsigned StoreCnt = ~0u; // Corresponds to VScnt on gfx10/gfx11.
+ unsigned SampleCnt = ~0u; // gfx12+ only.
+ unsigned BvhCnt = ~0u; // gfx12+ only.
+ unsigned KmCnt = ~0u; // gfx12+ only.
+ unsigned XCnt = ~0u; // gfx1250.
+
+ Waitcnt() = default;
+ // Pre-gfx12 constructor.
+ Waitcnt(unsigned VmCnt, unsigned ExpCnt, unsigned LgkmCnt, unsigned VsCnt)
+ : LoadCnt(VmCnt), ExpCnt(ExpCnt), DsCnt(LgkmCnt), StoreCnt(VsCnt) {}
+
+ // gfx12+ constructor.
+ Waitcnt(unsigned LoadCnt, unsigned ExpCnt, unsigned DsCnt, unsigned StoreCnt,
+ unsigned SampleCnt, unsigned BvhCnt, unsigned KmCnt, unsigned XCnt)
+ : LoadCnt(LoadCnt), ExpCnt(ExpCnt), DsCnt(DsCnt), StoreCnt(StoreCnt),
+ SampleCnt(SampleCnt), BvhCnt(BvhCnt), KmCnt(KmCnt), XCnt(XCnt) {}
+
+ bool hasWait() const { return StoreCnt != ~0u || hasWaitExceptStoreCnt(); }
+
+ bool hasWaitExceptStoreCnt() const {
+ return LoadCnt != ~0u || ExpCnt != ~0u || DsCnt != ~0u ||
+ SampleCnt != ~0u || BvhCnt != ~0u || KmCnt != ~0u || XCnt != ~0u;
+ }
+
+ bool hasWaitStoreCnt() const { return StoreCnt != ~0u; }
+
+ Waitcnt combined(const Waitcnt &Other) const {
+ // Does the right thing provided self and Other are either both pre-gfx12
+ // or both gfx12+.
+ return Waitcnt(
+ std::min(LoadCnt, Other.LoadCnt), std::min(ExpCnt, Other.ExpCnt),
+ std::min(DsCnt, Other.DsCnt), std::min(StoreCnt, Other.StoreCnt),
+ std::min(SampleCnt, Other.SampleCnt), std::min(BvhCnt, Other.BvhCnt),
+ std::min(KmCnt, Other.KmCnt), std::min(XCnt, Other.XCnt));
+ }
+
+ friend raw_ostream &operator<<(raw_ostream &OS, const AMDGPU::Waitcnt &Wait);
+};
+
+// The following methods are only meaningful on targets that support
+// S_WAITCNT.
+
+/// \returns Vmcnt bit mask for given isa \p Version.
+LLVM_ABI unsigned getVmcntBitMask(const IsaVersion &Version);
+
+/// \returns Expcnt bit mask for given isa \p Version.
+LLVM_ABI unsigned getExpcntBitMask(const IsaVersion &Version);
+
+/// \returns Lgkmcnt bit mask for given isa \p Version.
+LLVM_ABI unsigned getLgkmcntBitMask(const IsaVersion &Version);
+
+/// \returns Waitcnt bit mask for given isa \p Version.
+LLVM_ABI unsigned getWaitcntBitMask(const IsaVersion &Version);
+
+/// \returns Decoded Vmcnt from given \p Waitcnt for given isa \p Version.
+LLVM_ABI unsigned decodeVmcnt(const IsaVersion &Version, unsigned Waitcnt);
+
+/// \returns Decoded Expcnt from given \p Waitcnt for given isa \p Version.
+LLVM_ABI unsigned decodeExpcnt(const IsaVersion &Version, unsigned Waitcnt);
+
+/// \returns Decoded Lgkmcnt from given \p Waitcnt for given isa \p Version.
+LLVM_ABI unsigned decodeLgkmcnt(const IsaVersion &Version, unsigned Waitcnt);
+
+/// Decodes Vmcnt, Expcnt and Lgkmcnt from given \p Waitcnt for given isa
+/// \p Version, and writes decoded values into \p Vmcnt, \p Expcnt and
+/// \p Lgkmcnt respectively. Should not be used on gfx12+, the instruction
+/// which needs it is deprecated
+///
+/// \details \p Vmcnt, \p Expcnt and \p Lgkmcnt are decoded as follows:
+/// \p Vmcnt = \p Waitcnt[3:0] (pre-gfx9)
+/// \p Vmcnt = \p Waitcnt[15:14,3:0] (gfx9,10)
+/// \p Vmcnt = \p Waitcnt[15:10] (gfx11)
+/// \p Expcnt = \p Waitcnt[6:4] (pre-gfx11)
+/// \p Expcnt = \p Waitcnt[2:0] (gfx11)
+/// \p Lgkmcnt = \p Waitcnt[11:8] (pre-gfx10)
+/// \p Lgkmcnt = \p Waitcnt[13:8] (gfx10)
+/// \p Lgkmcnt = \p Waitcnt[9:4] (gfx11)
+///
+LLVM_ABI void decodeWaitcnt(const IsaVersion &Version, unsigned Waitcnt,
+ unsigned &Vmcnt, unsigned &Expcnt,
+ unsigned &Lgkmcnt);
+
+LLVM_ABI Waitcnt decodeWaitcnt(const IsaVersion &Version, unsigned Encoded);
+
+/// \returns \p Waitcnt with encoded \p Vmcnt for given isa \p Version.
+LLVM_ABI unsigned encodeVmcnt(const IsaVersion &Version, unsigned Waitcnt,
+ unsigned Vmcnt);
+
+/// \returns \p Waitcnt with encoded \p Expcnt for given isa \p Version.
+LLVM_ABI unsigned encodeExpcnt(const IsaVersion &Version, unsigned Waitcnt,
+ unsigned Expcnt);
+
+/// \returns \p Waitcnt with encoded \p Lgkmcnt for given isa \p Version.
+LLVM_ABI unsigned encodeLgkmcnt(const IsaVersion &Version, unsigned Waitcnt,
+ unsigned Lgkmcnt);
+
+/// Encodes \p Vmcnt, \p Expcnt and \p Lgkmcnt into Waitcnt for given isa
+/// \p Version. Should not be used on gfx12+, the instruction which needs
+/// it is deprecated
+///
+/// \details \p Vmcnt, \p Expcnt and \p Lgkmcnt are encoded as follows:
+/// Waitcnt[2:0] = \p Expcnt (gfx11+)
+/// Waitcnt[3:0] = \p Vmcnt (pre-gfx9)
+/// Waitcnt[3:0] = \p Vmcnt[3:0] (gfx9,10)
+/// Waitcnt[6:4] = \p Expcnt (pre-gfx11)
+/// Waitcnt[9:4] = \p Lgkmcnt (gfx11)
+/// Waitcnt[11:8] = \p Lgkmcnt (pre-gfx10)
+/// Waitcnt[13:8] = \p Lgkmcnt (gfx10)
+/// Waitcnt[15:10] = \p Vmcnt (gfx11)
+/// Waitcnt[15:14] = \p Vmcnt[5:4] (gfx9,10)
+///
+/// \returns Waitcnt with encoded \p Vmcnt, \p Expcnt and \p Lgkmcnt for given
+/// isa \p Version.
+///
+LLVM_ABI unsigned encodeWaitcnt(const IsaVersion &Version, unsigned Vmcnt,
+ unsigned Expcnt, unsigned Lgkmcnt);
+
+LLVM_ABI unsigned encodeWaitcnt(const IsaVersion &Version,
+ const Waitcnt &Decoded);
+
+// The following methods are only meaningful on targets that support
+// S_WAIT_*CNT, introduced with gfx12.
+
+/// \returns Loadcnt bit mask for given isa \p Version.
+/// Returns 0 for versions that do not support LOADcnt
+LLVM_ABI unsigned getLoadcntBitMask(const IsaVersion &Version);
+
+/// \returns Samplecnt bit mask for given isa \p Version.
+/// Returns 0 for versions that do not support SAMPLEcnt
+LLVM_ABI unsigned getSamplecntBitMask(const IsaVersion &Version);
+
+/// \returns Bvhcnt bit mask for given isa \p Version.
+/// Returns 0 for versions that do not support BVHcnt
+LLVM_ABI unsigned getBvhcntBitMask(const IsaVersion &Version);
+
+/// \returns Dscnt bit mask for given isa \p Version.
+/// Returns 0 for versions that do not support DScnt
+LLVM_ABI unsigned getDscntBitMask(const IsaVersion &Version);
+
+/// \returns Dscnt bit mask for given isa \p Version.
+/// Returns 0 for versions that do not support KMcnt
+LLVM_ABI unsigned getKmcntBitMask(const IsaVersion &Version);
+
+/// \returns Xcnt bit mask for given isa \p Version.
+/// Returns 0 for versions that do not support Xcnt.
+LLVM_ABI unsigned getXcntBitMask(const IsaVersion &Version);
+
+/// \return STOREcnt or VScnt bit mask for given isa \p Version.
+/// returns 0 for versions that do not support STOREcnt or VScnt.
+/// STOREcnt and VScnt are the same counter, the name used
+/// depends on the ISA version.
+LLVM_ABI unsigned getStorecntBitMask(const IsaVersion &Version);
+
+// The following are only meaningful on targets that support
+// S_WAIT_LOADCNT_DSCNT and S_WAIT_STORECNT_DSCNT.
+
+/// \returns Decoded Waitcnt structure from given \p LoadcntDscnt for given
+/// isa \p Version.
+LLVM_ABI Waitcnt decodeLoadcntDscnt(const IsaVersion &Version,
+ unsigned LoadcntDscnt);
+
+/// \returns Decoded Waitcnt structure from given \p StorecntDscnt for given
+/// isa \p Version.
+LLVM_ABI Waitcnt decodeStorecntDscnt(const IsaVersion &Version,
+ unsigned StorecntDscnt);
+
+/// \returns \p Loadcnt and \p Dscnt components of \p Decoded encoded as an
+/// immediate that can be used with S_WAIT_LOADCNT_DSCNT for given isa
+/// \p Version.
+LLVM_ABI unsigned encodeLoadcntDscnt(const IsaVersion &Version,
+ const Waitcnt &Decoded);
+
+/// \returns \p Storecnt and \p Dscnt components of \p Decoded encoded as an
+/// immediate that can be used with S_WAIT_STORECNT_DSCNT for given isa
+/// \p Version.
+LLVM_ABI unsigned encodeStorecntDscnt(const IsaVersion &Version,
+ const Waitcnt &Decoded);
} // namespace AMDGPU
struct BasicSubtargetFeatureKV {
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
index 75db58a292c13..2022a3d32b5da 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.h
@@ -1076,186 +1076,6 @@ getIntegerVecAttribute(const Function &F, StringRef Name, unsigned Size);
/// Checks if \p Val is inside \p MD, a !range-like metadata.
bool hasValueInRangeLikeMetadata(const MDNode &MD, int64_t Val);
-/// Represents the counter values to wait for in an s_waitcnt instruction.
-///
-/// Large values (including the maximum possible integer) can be used to
-/// represent "don't care" waits.
-struct Waitcnt {
- unsigned LoadCnt = ~0u; // Corresponds to Vmcnt prior to gfx12.
- unsigned ExpCnt = ~0u;
- unsigned DsCnt = ~0u; // Corresponds to LGKMcnt prior to gfx12.
- unsigned StoreCnt = ~0u; // Corresponds to VScnt on gfx10/gfx11.
- unsigned SampleCnt = ~0u; // gfx12+ only.
- unsigned BvhCnt = ~0u; // gfx12+ only.
- unsigned KmCnt = ~0u; // gfx12+ only.
- unsigned XCnt = ~0u; // gfx1250.
-
- Waitcnt() = default;
- // Pre-gfx12 constructor.
- Waitcnt(unsigned VmCnt, unsigned ExpCnt, unsigned LgkmCnt, unsigned VsCnt)
- : LoadCnt(VmCnt), ExpCnt(ExpCnt), DsCnt(LgkmCnt), StoreCnt(VsCnt) {}
-
- // gfx12+ constructor.
- Waitcnt(unsigned LoadCnt, unsigned ExpCnt, unsigned DsCnt, unsigned StoreCnt,
- unsigned SampleCnt, unsigned BvhCnt, unsigned KmCnt, unsigned XCnt)
- : LoadCnt(LoadCnt), ExpCnt(ExpCnt), DsCnt(DsCnt), StoreCnt(StoreCnt),
- SampleCnt(SampleCnt), BvhCnt(BvhCnt), KmCnt(KmCnt), XCnt(XCnt) {}
-
- bool hasWait() const { return StoreCnt != ~0u || hasWaitExceptStoreCnt(); }
-
- bool hasWaitExceptStoreCnt() const {
- return LoadCnt != ~0u || ExpCnt != ~0u || DsCnt != ~0u ||
- SampleCnt != ~0u || BvhCnt != ~0u || KmCnt != ~0u || XCnt != ~0u;
- }
-
- bool hasWaitStoreCnt() const { return StoreCnt != ~0u; }
-
- Waitcnt combined(const Waitcnt &Other) const {
- // Does the right thing provided self and Other are either both pre-gfx12
- // or both gfx12+.
- return Waitcnt(
- std::min(LoadCnt, Other.LoadCnt), std::min(ExpCnt, Other.ExpCnt),
- std::min(DsCnt, Other.DsCnt), std::min(StoreCnt, Other.StoreCnt),
- std::min(SampleCnt, Other.SampleCnt), std::min(BvhCnt, Other.BvhCnt),
- std::min(KmCnt, Other.KmCnt), std::min(XCnt, Other.XCnt));
- }
-
- friend raw_ostream &operator<<(raw_ostream &OS, const AMDGPU::Waitcnt &Wait);
-};
-
-// The following methods are only meaningful on targets that support
-// S_WAITCNT.
-
-/// \returns Vmcnt bit mask for given isa \p Version.
-unsigned getVmcntBitMask(const IsaVersion &Version);
-
-/// \returns Expcnt bit mask for given isa \p Version.
-unsigned getExpcntBitMask(const IsaVersion &Version);
-
-/// \returns Lgkmcnt bit mask for given isa \p Version.
-unsigned getLgkmcntBitMask(const IsaVersion &Version);
-
-/// \returns Waitcnt bit mask for given isa \p Version.
-unsigned getWaitcntBitMask(const IsaVersion &Version);
-
-/// \returns Decoded Vmcnt from given \p Waitcnt for given isa \p Version.
-unsigned decodeVmcnt(const IsaVersion &Version, unsigned Waitcnt);
-
-/// \returns Decoded Expcnt from given \p Waitcnt for given isa \p Version.
-unsigned decodeExpcnt(const IsaVersion &Version, unsigned Waitcnt);
-
-/// \returns Decoded Lgkmcnt from given \p Waitcnt for given isa \p Version.
-unsigned decodeLgkmcnt(const IsaVersion &Version, unsigned Waitcnt);
-
-/// Decodes Vmcnt, Expcnt and Lgkmcnt from given \p Waitcnt for given isa
-/// \p Version, and writes decoded values into \p Vmcnt, \p Expcnt and
-/// \p Lgkmcnt respectively. Should not be used on gfx12+, the instruction
-/// which needs it is deprecated
-///
-/// \details \p Vmcnt, \p Expcnt and \p Lgkmcnt are decoded as follows:
-/// \p Vmcnt = \p Waitcnt[3:0] (pre-gfx9)
-/// \p Vmcnt = \p Waitcnt[15:14,3:0] (gfx9,10)
-/// \p Vmcnt = \p Waitcnt[15:10] (gfx11)
-/// \p Expcnt = \p Waitcnt[6:4] (pre-gfx11)
-/// \p Expcnt = \p Waitcnt[2:0] (gfx11)
-/// \p Lgkmcnt = \p Waitcnt[11:8] (pre-gfx10)
-/// \p Lgkmcnt = \p Waitcnt[13:8] (gfx10)
-/// \p Lgkmcnt = \p Waitcnt[9:4] (gfx11)
-///
-void decodeWaitcnt(const IsaVersion &Version, unsigned Waitcnt, unsigned &Vmcnt,
- unsigned &Expcnt, unsigned &Lgkmcnt);
-
-Waitcnt decodeWaitcnt(const IsaVersion &Version, unsigned Encoded);
-
-/// \returns \p Waitcnt with encoded \p Vmcnt for given isa \p Version.
-unsigned encodeVmcnt(const IsaVersion &Version, unsigned Waitcnt,
- unsigned Vmcnt);
-
-/// \returns \p Waitcnt with encoded \p Expcnt for given isa \p Version.
-unsigned encodeExpcnt(const IsaVersion &Version, unsigned Waitcnt,
- unsigned Expcnt);
-
-/// \returns \p Waitcnt with encoded \p Lgkmcnt for given isa \p Version.
-unsigned encodeLgkmcnt(const IsaVersion &Version, unsigned Waitcnt,
- unsigned Lgkmcnt);
-
-/// Encodes \p Vmcnt, \p Expcnt and \p Lgkmcnt into Waitcnt for given isa
-/// \p Version. Should not be used on gfx12+, the instruction which needs
-/// it is deprecated
-///
-/// \details \p Vmcnt, \p Expcnt and \p Lgkmcnt are encoded as follows:
-/// Waitcnt[2:0] = \p Expcnt (gfx11+)
-/// Waitcnt[3:0] = \p Vmcnt (pre-gfx9)
-/// Waitcnt[3:0] = \p Vmcnt[3:0] (gfx9,10)
-/// Waitcnt[6:4] = \p Expcnt (pre-gfx11)
-/// Waitcnt[9:4] = \p Lgkmcnt (gfx11)
-/// Waitcnt[11:8] = \p Lgkmcnt (pre-gfx10)
-/// Waitcnt[13:8] = \p Lgkmcnt (gfx10)
-/// Waitcnt[15:10] = \p Vmcnt (gfx11)
-/// Waitcnt[15:14] = \p Vmcnt[5:4] (gfx9,10)
-///
-/// \returns Waitcnt with encoded \p Vmcnt, \p Expcnt and \p Lgkmcnt for given
-/// isa \p Version.
-///
-unsigned encodeWaitcnt(const IsaVersion &Version, unsigned Vmcnt,
- unsigned Expcnt, unsigned Lgkmcnt);
-
-unsigned encodeWaitcnt(const IsaVersion &Version, const Waitcnt &Decoded);
-
-// The following methods are only meaningful on targets that support
-// S_WAIT_*CNT, introduced with gfx12.
-
-/// \returns Loadcnt bit mask for given isa \p Version.
-/// Returns 0 for versions that do not support LOADcnt
-unsigned getLoadcntBitMask(const IsaVersion &Version);
-
-/// \returns Samplecnt bit mask for given isa \p Version.
-/// Returns 0 for versions that do not support SAMPLEcnt
-unsigned getSamplecntBitMask(const IsaVersion &Version);
-
-/// \returns Bvhcnt bit mask for given isa \p Version.
-/// Returns 0 for versions that do not support BVHcnt
-unsigned getBvhcntBitMask(const IsaVersion &Version);
-
-/// \returns Dscnt bit mask for given isa \p Version.
-/// Returns 0 for versions that do not support DScnt
-unsigned getDscntBitMask(const IsaVersion &Version);
-
-/// \returns Dscnt bit mask for given isa \p Version.
-/// Returns 0 for versions that do not support KMcnt
-unsigned getKmcntBitMask(const IsaVersion &Version);
-
-/// \returns Xcnt bit mask for given isa \p Version.
-/// Returns 0 for versions that do not support Xcnt.
-unsigned getXcntBitMask(const IsaVersion &Version);
-
-/// \return STOREcnt or VScnt bit mask for given isa \p Version.
-/// returns 0 for versions that do not support STOREcnt or VScnt.
-/// STOREcnt and VScnt are the same counter, the name used
-/// depends on the ISA version.
-unsigned getStorecntBitMask(const IsaVersion &Version);
-
-// The following are only meaningful on targets that support
-// S_WAIT_LOADCNT_DSCNT and S_WAIT_STORECNT_DSCNT.
-
-/// \returns Decoded Waitcnt structure from given \p LoadcntDscnt for given
-/// isa \p Version.
-Waitcnt decodeLoadcntDscnt(const IsaVersion &Version, unsigned LoadcntDscnt);
-
-/// \returns Decoded Waitcnt structure from given \p StorecntDscnt for given
-/// isa \p Version.
-Waitcnt decodeStorecntDscnt(const IsaVersion &Version, unsigned StorecntDscnt);
-
-/// \returns \p Loadcnt and \p Dscnt components of \p Decoded encoded as an
-/// immediate that can be used with S_WAIT_LOADCNT_DSCNT for given isa
-/// \p Version.
-unsigned encodeLoadcntDscnt(const IsaVersion &Version, const Waitcnt &Decoded);
-
-/// \returns \p Storecnt and \p Dscnt components of \p Decoded encoded as an
-/// immediate that can be used with S_WAIT_STORECNT_DSCNT for given isa
-/// \p Version.
-unsigned encodeStorecntDscnt(const IsaVersion &Version, const Waitcnt &Decoded);
-
namespace Hwreg {
using HwregId = EncodingField<5, 0>;
diff --git a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
index af825ad42dd82..ba8ce999eee17 100644
--- a/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
+++ b/mlir/lib/Conversion/AMDGPUToROCDL/AMDGPUToROCDL.cpp
@@ -30,6 +30,7 @@
#include "llvm/ADT/TypeSwitch.h"
#include "llvm/Support/Casting.h"
#include "llvm/Support/ErrorHandling.h"
+#include "llvm/TargetParser/TargetParser.h"
#include <optional>
namespace mlir {
@@ -435,50 +436,18 @@ struct RawBufferOpLowering : public ConvertOpToLLVMPattern<GpuOp> {
}
};
-// TODO: AMDGPU backend already have all this bitpacking logic, we should move
-// it to some common place.
-/// Vmcnt, Expcnt and Lgkmcnt are decoded as follows:
-/// Vmcnt = Waitcnt[3:0] (pre-gfx9)
-/// Vmcnt = Waitcnt[15:14,3:0] (gfx9,10)
-/// Vmcnt = Waitcnt[15:10] (gfx11)
-/// Expcnt = Waitcnt[6:4] (pre-gfx11)
-/// Expcnt = Waitcnt[2:0] (gfx11)
-/// Lgkmcnt = Waitcnt[11:8] (pre-gfx10)
-/// Lgkmcnt = Waitcnt[13:8] (gfx10)
-/// Lgkmcnt = Waitcnt[9:4] (gfx11)
static FailureOr<unsigned> encodeWaitcnt(Chipset chipset, unsigned vmcnt,
unsigned expcnt, unsigned lgkmcnt) {
- if (chipset.majorVersion < 9) {
- vmcnt = std::min(15u, vmcnt);
- expcnt = std::min(7u, expcnt);
- lgkmcnt = std::min(15u, lgkmcnt);
- return vmcnt | (expcnt << 4) | (lgkmcnt << 8);
- }
- if (chipset.majorVersion == 9) {
- vmcnt = std::min(63u, vmcnt);
- expcnt = std::min(7u, expcnt);
- lgkmcnt = std::min(15u, lgkmcnt);
- unsigned lowBits = vmcnt & 0xF;
- unsigned highBits = (vmcnt >> 4) << 14;
- unsigned otherCnts = (expcnt << 4) | (lgkmcnt << 8);
- return lowBits | highBits | otherCnts;
- }
- if (chipset.majorVersion == 10) {
- vmcnt = std::min(63u, vmcnt);
- expcnt = std::min(7u, expcnt);
- lgkmcnt = std::min(63u, lgkmcnt);
- unsigned lowBits = vmcnt & 0xF;
- unsigned highBits = (vmcnt >> 4) << 14;
- unsigned otherCnts = (expcnt << 4) | (lgkmcnt << 8);
- return lowBits | highBits | otherCnts;
- }
- if (chipset.majorVersion == 11) {
- vmcnt = std::min(63u, vmcnt);
- expcnt = std::min(7u, expcnt);
- lgkmcnt = std::min(63u, lgkmcnt);
- return (vmcnt << 10) | expcnt | (lgkmcnt << 4);
- }
- return failure();
+ if (chipset.majorVersion >= 12)
+ return failure();
+
+ llvm::AMDGPU::IsaVersion isaVersion{
+ chipset.majorVersion, chipset.minorVersion, chipset.steppingVersion};
+ vmcnt = std::min(vmcnt, llvm::AMDGPU::getVmcntBitMask(isaVersion));
+ expcnt = std::min(expcnt, llvm::AMDGPU::getExpcntBitMask(isaVersion));
+ lgkmcnt = std::min(lgkmcnt, llvm::AMDGPU::getLgkmcntBitMask(isaVersion));
+
+ return llvm::AMDGPU::encodeWaitcnt(isaVersion, vmcnt, expcnt, lgkmcnt);
}
struct MemoryCounterWaitOpLoweri...
[truncated]
|
Contributor
Author
|
I should probably move it into separate header. |
kuhar
approved these changes
Dec 15, 2025
Member
kuhar
left a comment
kuhar
left a comment
There was a problem hiding this comment.
+1 but please wait for an approval from some backend folks
Contributor
Author
|
So, what will be the good place for header in llvm tree for this? |
Contributor
Previously we've added separate headers with "AMDGPU" in the name like AMDGPUAddrSpace.h: #73944 |
Hardcode84
force-pushed
the
waitcnt-api
branch
from
7b010aa to
007a852
Compare
Contributor
Author
|
moved to dedicated header |
kuhar
approved these changes
Dec 16, 2025
Hardcode84
added a commit
to Hardcode84/llvm-project
that referenced
this pull request
Dec 17, 2025
This reverts commit 9301381.
Collaborator
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/203/builds/32064 Here is the relevant piece of the build log for the reference |
Collaborator
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/205/builds/30853 Here is the relevant piece of the build log for the reference |
Collaborator
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/204/builds/30874 Here is the relevant piece of the build log for the reference |
Collaborator
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/137/builds/30762 Here is the relevant piece of the build log for the reference |
Hardcode84
added a commit
that referenced
this pull request
Dec 17, 2025
Collaborator
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/129/builds/35139 Here is the relevant piece of the build log for the reference |
llvm-sync bot
pushed a commit
to arm/arm-toolchain
that referenced
this pull request
Dec 17, 2025
…72313)" (#172636) This reverts commit 9301381. Revert llvm/llvm-project#172313 Missing libraries, again
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So we can get rid of our copy in
AMDGPUToROCDL.