-
Notifications
You must be signed in to change notification settings - Fork 11k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add support for TFE/LWE in image intrinsics
TFE and LWE support requires extra result registers that are written in the
event of a failure in order to detect that failure case.
The specific use-case that initiated these changes is sparse texture support.
This means that if image intrinsics are used with either option turned on, the
programmer must ensure that the return type can contain all of the expected
results. This can result in redundant registers since the vector size must be a
power-of-2.
This change takes roughly 6 parts:
1. Modify the instruction defs in tablegen to add new instruction variants that
can accomodate the extra return values.
2. Updates to lowerImage in SIISelLowering.cpp to accomodate setting TFE or LWE
(where the bulk of the work for these instruction types is now done)
3. Extra verification code to catch cases where intrinsics have been used but
insufficient return registers are used.
4. Modification to the adjustWritemask optimisation to account for TFE/LWE being
enabled (requires extra registers to be maintained for error return value).
5. An extra pass to zero initialize the error value return - this is because if
the error does not occur, the register is not written and thus must be zeroed
before use. Also added a new (on by default) option to ensure ALL return values
are zero-initialized that is required for sparse texture support.
6. Disable the inst_combine optimization in the presence of tfe/lwe (later TODO
for this to re-enable and handle correctly).
There's an additional fix now to avoid a dmask=0
For an image intrinsic with tfe where all result channels except tfe
were unused, I was getting an image instruction with dmask=0 and only a
single vgpr result for tfe. That is incorrect because the hardware
assumes there is at least one vgpr result, plus the one for tfe.
Fixed by forcing dmask to 1, which gives the desired two vgpr result
with tfe in the second one.
The TFE or LWE result is returned from the intrinsics using an aggregate
type. Look in the test code provided to see how this works, but in essence IR
code to invoke the intrinsic looks as follows:
%v = call {<4 x float>,i32} @llvm.amdgcn.image.load.1d.v4f32i32.i32(i32 15,
i32 %s, <8 x i32> %rsrc, i32 1, i32 0)
%v.vec = extractvalue {<4 x float>, i32} %v, 0
%v.err = extractvalue {<4 x float>, i32} %v, 1
Differential revision: https://reviews.llvm.org/D48826
Change-Id: If222bc03642e76cf98059a6bef5d5bffeda38dda
llvm-svn: 347871- Loading branch information
Showing
20 changed files
with
1,275 additions
and
76 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,181 @@ | ||
| //===-- SIAddIMGInit.cpp - Add any required IMG inits ---------------------===// | ||
| // | ||
| // The LLVM Compiler Infrastructure | ||
| // | ||
| // This file is distributed under the University of Illinois Open Source | ||
| // License. See LICENSE.TXT for details. | ||
| // | ||
| //===----------------------------------------------------------------------===// | ||
| // | ||
| /// \file | ||
| /// Any MIMG instructions that use tfe or lwe require an initialization of the | ||
| /// result register that will be written in the case of a memory access failure | ||
| /// The required code is also added to tie this init code to the result of the | ||
| /// img instruction | ||
| /// | ||
| //===----------------------------------------------------------------------===// | ||
| // | ||
|
|
||
| #include "AMDGPU.h" | ||
| #include "AMDGPUSubtarget.h" | ||
| #include "MCTargetDesc/AMDGPUMCTargetDesc.h" | ||
| #include "SIInstrInfo.h" | ||
| #include "llvm/CodeGen/MachineFunctionPass.h" | ||
| #include "llvm/CodeGen/MachineInstrBuilder.h" | ||
| #include "llvm/CodeGen/MachineRegisterInfo.h" | ||
| #include "llvm/IR/Function.h" | ||
| #include "llvm/Support/Debug.h" | ||
| #include "llvm/Target/TargetMachine.h" | ||
|
|
||
| #define DEBUG_TYPE "si-img-init" | ||
|
|
||
| using namespace llvm; | ||
|
|
||
| namespace { | ||
|
|
||
| class SIAddIMGInit : public MachineFunctionPass { | ||
| public: | ||
| static char ID; | ||
|
|
||
| public: | ||
| SIAddIMGInit() : MachineFunctionPass(ID) { | ||
| initializeSIAddIMGInitPass(*PassRegistry::getPassRegistry()); | ||
| } | ||
|
|
||
| bool runOnMachineFunction(MachineFunction &MF) override; | ||
|
|
||
| void getAnalysisUsage(AnalysisUsage &AU) const override { | ||
| AU.setPreservesCFG(); | ||
| MachineFunctionPass::getAnalysisUsage(AU); | ||
| } | ||
| }; | ||
|
|
||
| } // End anonymous namespace. | ||
|
|
||
| INITIALIZE_PASS(SIAddIMGInit, DEBUG_TYPE, "SI Add IMG Init", false, false) | ||
|
|
||
| char SIAddIMGInit::ID = 0; | ||
|
|
||
| char &llvm::SIAddIMGInitID = SIAddIMGInit::ID; | ||
|
|
||
| FunctionPass *llvm::createSIAddIMGInitPass() { return new SIAddIMGInit(); } | ||
|
|
||
| bool SIAddIMGInit::runOnMachineFunction(MachineFunction &MF) { | ||
| MachineRegisterInfo &MRI = MF.getRegInfo(); | ||
| const GCNSubtarget &ST = MF.getSubtarget<GCNSubtarget>(); | ||
| const SIInstrInfo *TII = ST.getInstrInfo(); | ||
| const SIRegisterInfo *RI = ST.getRegisterInfo(); | ||
| bool Changed = false; | ||
|
|
||
| for (MachineFunction::iterator BI = MF.begin(), BE = MF.end(); BI != BE; | ||
| ++BI) { | ||
| MachineBasicBlock &MBB = *BI; | ||
| MachineBasicBlock::iterator I, Next; | ||
| for (I = MBB.begin(); I != MBB.end(); I = Next) { | ||
| Next = std::next(I); | ||
| MachineInstr &MI = *I; | ||
|
|
||
| auto Opcode = MI.getOpcode(); | ||
| if (TII->isMIMG(Opcode) && !MI.mayStore()) { | ||
| MachineOperand *TFE = TII->getNamedOperand(MI, AMDGPU::OpName::tfe); | ||
| MachineOperand *LWE = TII->getNamedOperand(MI, AMDGPU::OpName::lwe); | ||
| MachineOperand *D16 = TII->getNamedOperand(MI, AMDGPU::OpName::d16); | ||
|
|
||
| // Check for instructions that don't have tfe or lwe fields | ||
| // There shouldn't be any at this point. | ||
| assert( (TFE && LWE) && "Expected tfe and lwe operands in instruction"); | ||
|
|
||
| unsigned TFEVal = TFE->getImm(); | ||
| unsigned LWEVal = LWE->getImm(); | ||
| unsigned D16Val = D16 ? D16->getImm() : 0; | ||
|
|
||
| if (TFEVal || LWEVal) { | ||
| // At least one of TFE or LWE are non-zero | ||
| // We have to insert a suitable initialization of the result value and | ||
| // tie this to the dest of the image instruction. | ||
|
|
||
| const DebugLoc &DL = MI.getDebugLoc(); | ||
|
|
||
| int DstIdx = | ||
| AMDGPU::getNamedOperandIdx(MI.getOpcode(), AMDGPU::OpName::vdata); | ||
|
|
||
| // Calculate which dword we have to initialize to 0. | ||
| MachineOperand *MO_Dmask = | ||
| TII->getNamedOperand(MI, AMDGPU::OpName::dmask); | ||
|
|
||
| // check that dmask operand is found. | ||
| assert(MO_Dmask && "Expected dmask operand in instruction"); | ||
|
|
||
| unsigned dmask = MO_Dmask->getImm(); | ||
| // Determine the number of active lanes taking into account the | ||
| // Gather4 special case | ||
| unsigned ActiveLanes = | ||
| TII->isGather4(Opcode) ? 4 : countPopulation(dmask); | ||
|
|
||
| // Subreg indices are counted from 1 | ||
| // When D16 then we want next whole VGPR after write data. | ||
| static_assert(AMDGPU::sub0 == 1 && AMDGPU::sub4 == 5, "Subreg indices different from expected"); | ||
|
|
||
| bool Packed = !ST.hasUnpackedD16VMem(); | ||
|
|
||
| unsigned InitIdx = | ||
| D16Val && Packed ? ((ActiveLanes + 1) >> 1) + 1 : ActiveLanes + 1; | ||
|
|
||
| // Abandon attempt if the dst size isn't large enough | ||
| // - this is in fact an error but this is picked up elsewhere and | ||
| // reported correctly. | ||
| uint32_t DstSize = | ||
| RI->getRegSizeInBits(*TII->getOpRegClass(MI, DstIdx)) / 32; | ||
| if (DstSize < InitIdx) | ||
| continue; | ||
|
|
||
| // Create a register for the intialization value. | ||
| unsigned PrevDst = | ||
| MRI.createVirtualRegister(TII->getOpRegClass(MI, DstIdx)); | ||
| unsigned NewDst = 0; // Final initialized value will be in here | ||
|
|
||
| // If PRTStrictNull feature is enabled (the default) then initialize | ||
| // all the result registers to 0, otherwise just the error indication | ||
| // register (VGPRn+1) | ||
| unsigned SizeLeft = ST.usePRTStrictNull() ? InitIdx : 1; | ||
| unsigned CurrIdx = ST.usePRTStrictNull() ? 1 : InitIdx; | ||
|
|
||
| if (DstSize == 1) { | ||
| // In this case we can just initialize the result directly | ||
| BuildMI(MBB, MI, DL, TII->get(AMDGPU::V_MOV_B32_e32), PrevDst) | ||
| .addImm(0); | ||
| NewDst = PrevDst; | ||
| } else { | ||
| BuildMI(MBB, MI, DL, TII->get(AMDGPU::IMPLICIT_DEF), PrevDst); | ||
| for (; SizeLeft; SizeLeft--, CurrIdx++) { | ||
| NewDst = | ||
| MRI.createVirtualRegister(TII->getOpRegClass(MI, DstIdx)); | ||
| // Initialize dword | ||
| unsigned SubReg = | ||
| MRI.createVirtualRegister(&AMDGPU::VGPR_32RegClass); | ||
| BuildMI(MBB, MI, DL, TII->get(AMDGPU::V_MOV_B32_e32), SubReg) | ||
| .addImm(0); | ||
| // Insert into the super-reg | ||
| BuildMI(MBB, I, DL, TII->get(TargetOpcode::INSERT_SUBREG), NewDst) | ||
| .addReg(PrevDst) | ||
| .addReg(SubReg) | ||
| .addImm(CurrIdx); | ||
|
|
||
| PrevDst = NewDst; | ||
| } | ||
| } | ||
|
|
||
| // Add as an implicit operand | ||
| MachineInstrBuilder(MF, MI).addReg(NewDst, RegState::Implicit); | ||
|
|
||
| // Tie the just added implicit operand to the dst | ||
| MI.tieOperands(DstIdx, MI.getNumOperands() - 1); | ||
|
|
||
| Changed = true; | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| return Changed; | ||
| } |
Oops, something went wrong.