Skip to content

[AMDGPU] Support true16 spill restore with sram-ecc #165320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 29, 2025
Merged

[AMDGPU] Support true16 spill restore with sram-ecc #165320

merged 1 commit into from
Oct 29, 2025
+348 −1

Conversation

@rampitec
Copy link
Collaborator

No description provided.

@rampitec Graphite App
Copy link
Collaborator Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@rampitec rampitec marked this pull request as ready for review October 27, 2025 21:32
@llvmbot
Copy link
Member

llvmbot commented Oct 27, 2025

@llvm/pr-subscribers-backend-amdgpu

Author: Stanislav Mekhanoshin (rampitec)

Changes

Patch is 27.66 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/165320.diff

4 Files Affected:

  • (modified) llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp (+22-1)
  • (modified) llvm/test/CodeGen/AMDGPU/spill_kill_v16.mir (+67)
  • (modified) llvm/test/CodeGen/AMDGPU/spillv16.ll (+235)
  • (modified) llvm/test/CodeGen/AMDGPU/spillv16.mir (+22)
diff --git a/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp b/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
index d80a6f339c8f6..2cb7d8ff1a532 100644
--- a/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp
@@ -1823,6 +1823,16 @@ void SIRegisterInfo::buildSpillLoadStore(
       }
     }
 
+    Register FinalValueReg = ValueReg;
+    if (LoadStoreOp == AMDGPU::SCRATCH_LOAD_UBYTE_SADDR) {
+      // If we are loading 16-bit value with SRAMECC endabled we need a temp
+      // 32-bit VGPR to load and extract 16-bits into the final register.
+      ValueReg = RS->scavengeRegisterBackwards(AMDGPU::VGPR_32RegClass, MI,
+                                               false, 0, false);
+      SubReg = ValueReg;
+      IsKill = false;
+    }
+
     MachinePointerInfo PInfo = BasePtrInfo.getWithOffset(RegOffset);
     MachineMemOperand *NewMMO =
         MF->getMachineMemOperand(PInfo, MMO->getFlags(), RemEltSize,
@@ -1863,6 +1873,15 @@ void SIRegisterInfo::buildSpillLoadStore(
       MIB.addImm(0); // swz
     MIB.addMemOperand(NewMMO);
 
+    if (FinalValueReg != ValueReg) {
+      // Extract 16-bit from the loaded 32-bit value.
+      ValueReg = getSubReg(ValueReg, AMDGPU::lo16);
+      MIB = BuildMI(MBB, MI, DL, TII->get(AMDGPU::V_MOV_B16_t16_e32))
+                .addReg(FinalValueReg, getDefRegState(true))
+                .addReg(ValueReg, getKillRegState(true));
+      ValueReg = FinalValueReg;
+    }
+
     if (!IsAGPR && NeedSuperRegDef)
       MIB.addReg(ValueReg, RegState::ImplicitDefine);
 
@@ -2505,7 +2524,9 @@ bool SIRegisterInfo::eliminateFrameIndex(MachineBasicBlock::iterator MI,
       unsigned Opc;
       if (MI->getOpcode() == AMDGPU::SI_SPILL_V16_RESTORE) {
         assert(ST.enableFlatScratch() && "Flat Scratch is not enabled!");
-        Opc = AMDGPU::SCRATCH_LOAD_SHORT_D16_SADDR_t16;
+        Opc = ST.d16PreservesUnusedBits()
+                  ? AMDGPU::SCRATCH_LOAD_SHORT_D16_SADDR_t16
+                  : AMDGPU::SCRATCH_LOAD_UBYTE_SADDR;
       } else {
         Opc = MI->getOpcode() == AMDGPU::SI_BLOCK_SPILL_V1024_RESTORE
                   ? AMDGPU::SCRATCH_LOAD_BLOCK_SADDR
diff --git a/llvm/test/CodeGen/AMDGPU/spill_kill_v16.mir b/llvm/test/CodeGen/AMDGPU/spill_kill_v16.mir
index 0c694d9f49e18..e88e5748be6c4 100644
--- a/llvm/test/CodeGen/AMDGPU/spill_kill_v16.mir
+++ b/llvm/test/CodeGen/AMDGPU/spill_kill_v16.mir
@@ -1,5 +1,6 @@
 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -march=amdgcn -verify-machineinstrs -mcpu=gfx1100 -mattr=+real-true16 -run-pass=prologepilog -o - %s | FileCheck -check-prefix=EXPANDED %s
+# RUN: llc -march=amdgcn -verify-machineinstrs -mcpu=gfx1250 -mattr=+real-true16 -run-pass=prologepilog -o - %s | FileCheck -check-prefix=SRAMECC-EXPANDED %s
 
 ---
 name: spill_restore_vgpr16
@@ -31,6 +32,28 @@ body: |
   ; EXPANDED-NEXT:   $vgpr0_lo16 = SCRATCH_LOAD_SHORT_D16_SADDR_t16 $sgpr32, 0, 0, implicit $exec, implicit $flat_scr :: (load (s16) from %stack.0, align 4, addrspace 5)
   ; EXPANDED-NEXT:   $vgpr0_hi16 = SCRATCH_LOAD_SHORT_D16_SADDR_t16 $sgpr32, 4, 0, implicit $exec, implicit $flat_scr :: (load (s16) from %stack.1, align 4, addrspace 5)
   ; EXPANDED-NEXT:   S_NOP 0, implicit killed renamable $vgpr0_lo16, implicit killed renamable $vgpr0_hi16
+  ;
+  ; SRAMECC-EXPANDED-LABEL: name: spill_restore_vgpr16
+  ; SRAMECC-EXPANDED: bb.0:
+  ; SRAMECC-EXPANDED-NEXT:   successors: %bb.1(0x80000000)
+  ; SRAMECC-EXPANDED-NEXT: {{  $}}
+  ; SRAMECC-EXPANDED-NEXT:   S_NOP 0, implicit-def renamable $vgpr0_lo16, implicit-def renamable $vgpr0_hi16
+  ; SRAMECC-EXPANDED-NEXT:   SCRATCH_STORE_SHORT_SADDR_t16 killed $vgpr0_hi16, $sgpr32, 4, 0, implicit $exec, implicit $flat_scr :: (store (s16) into %stack.1, align 4, addrspace 5)
+  ; SRAMECC-EXPANDED-NEXT:   S_NOP 0, implicit renamable $vgpr0_lo16
+  ; SRAMECC-EXPANDED-NEXT:   SCRATCH_STORE_SHORT_SADDR_t16 killed $vgpr0_lo16, $sgpr32, 0, 0, implicit $exec, implicit $flat_scr :: (store (s16) into %stack.0, align 4, addrspace 5)
+  ; SRAMECC-EXPANDED-NEXT:   S_CBRANCH_SCC1 %bb.1, implicit undef $scc
+  ; SRAMECC-EXPANDED-NEXT: {{  $}}
+  ; SRAMECC-EXPANDED-NEXT: bb.1:
+  ; SRAMECC-EXPANDED-NEXT:   successors: %bb.2(0x80000000)
+  ; SRAMECC-EXPANDED-NEXT: {{  $}}
+  ; SRAMECC-EXPANDED-NEXT:   S_NOP 1
+  ; SRAMECC-EXPANDED-NEXT: {{  $}}
+  ; SRAMECC-EXPANDED-NEXT: bb.2:
+  ; SRAMECC-EXPANDED-NEXT:   $vgpr1 = SCRATCH_LOAD_UBYTE_SADDR $sgpr32, 0, 0, implicit $exec, implicit $flat_scr :: (load (s16) from %stack.0, align 4, addrspace 5)
+  ; SRAMECC-EXPANDED-NEXT:   $vgpr0_lo16 = V_MOV_B16_t16_e32 killed $vgpr1_lo16, implicit $exec
+  ; SRAMECC-EXPANDED-NEXT:   $vgpr1 = SCRATCH_LOAD_UBYTE_SADDR $sgpr32, 4, 0, implicit $exec, implicit $flat_scr :: (load (s16) from %stack.1, align 4, addrspace 5)
+  ; SRAMECC-EXPANDED-NEXT:   $vgpr0_hi16 = V_MOV_B16_t16_e32 killed $vgpr1_lo16, implicit $exec
+  ; SRAMECC-EXPANDED-NEXT:   S_NOP 0, implicit killed renamable $vgpr0_lo16, implicit killed renamable $vgpr0_hi16
    bb.0:
      successors: %bb.1(0x80000000)
      S_NOP 0, implicit-def renamable $vgpr0_lo16, implicit-def renamable $vgpr0_hi16
@@ -78,6 +101,29 @@ body: |
   ; EXPANDED-NEXT:   $vgpr0_lo16 = SCRATCH_LOAD_SHORT_D16_SADDR_t16 $sgpr32, 0, 0, implicit $exec, implicit $flat_scr :: (load (s16) from %stack.0, align 4, addrspace 5)
   ; EXPANDED-NEXT:   $vgpr0_hi16 = SCRATCH_LOAD_SHORT_D16_SADDR_t16 $sgpr32, 4, 0, implicit $exec, implicit $flat_scr :: (load (s16) from %stack.1, align 4, addrspace 5)
   ; EXPANDED-NEXT:   S_NOP 0, implicit killed renamable $vgpr0_lo16, implicit killed renamable $vgpr0_hi16
+  ;
+  ; SRAMECC-EXPANDED-LABEL: name: spill_restore_vgpr16_middle_of_block
+  ; SRAMECC-EXPANDED: bb.0:
+  ; SRAMECC-EXPANDED-NEXT:   successors: %bb.1(0x80000000)
+  ; SRAMECC-EXPANDED-NEXT: {{  $}}
+  ; SRAMECC-EXPANDED-NEXT:   S_NOP 0, implicit-def renamable $vgpr0_lo16, implicit-def renamable $vgpr0_hi16
+  ; SRAMECC-EXPANDED-NEXT:   SCRATCH_STORE_SHORT_SADDR_t16 killed $vgpr0_hi16, $sgpr32, 4, 0, implicit $exec, implicit $flat_scr :: (store (s16) into %stack.1, align 4, addrspace 5)
+  ; SRAMECC-EXPANDED-NEXT:   S_NOP 0, implicit renamable $vgpr0_lo16
+  ; SRAMECC-EXPANDED-NEXT:   SCRATCH_STORE_SHORT_SADDR_t16 killed $vgpr0_lo16, $sgpr32, 0, 0, implicit $exec, implicit $flat_scr :: (store (s16) into %stack.0, align 4, addrspace 5)
+  ; SRAMECC-EXPANDED-NEXT:   S_CBRANCH_SCC1 %bb.1, implicit undef $scc
+  ; SRAMECC-EXPANDED-NEXT: {{  $}}
+  ; SRAMECC-EXPANDED-NEXT: bb.1:
+  ; SRAMECC-EXPANDED-NEXT:   successors: %bb.2(0x80000000)
+  ; SRAMECC-EXPANDED-NEXT: {{  $}}
+  ; SRAMECC-EXPANDED-NEXT:   S_NOP 1
+  ; SRAMECC-EXPANDED-NEXT: {{  $}}
+  ; SRAMECC-EXPANDED-NEXT: bb.2:
+  ; SRAMECC-EXPANDED-NEXT:   S_NOP 1
+  ; SRAMECC-EXPANDED-NEXT:   $vgpr1 = SCRATCH_LOAD_UBYTE_SADDR $sgpr32, 0, 0, implicit $exec, implicit $flat_scr :: (load (s16) from %stack.0, align 4, addrspace 5)
+  ; SRAMECC-EXPANDED-NEXT:   $vgpr0_lo16 = V_MOV_B16_t16_e32 killed $vgpr1_lo16, implicit $exec
+  ; SRAMECC-EXPANDED-NEXT:   $vgpr1 = SCRATCH_LOAD_UBYTE_SADDR $sgpr32, 4, 0, implicit $exec, implicit $flat_scr :: (load (s16) from %stack.1, align 4, addrspace 5)
+  ; SRAMECC-EXPANDED-NEXT:   $vgpr0_hi16 = V_MOV_B16_t16_e32 killed $vgpr1_lo16, implicit $exec
+  ; SRAMECC-EXPANDED-NEXT:   S_NOP 0, implicit killed renamable $vgpr0_lo16, implicit killed renamable $vgpr0_hi16
    bb.0:
      successors: %bb.1(0x80000000)
      S_NOP 0, implicit-def renamable $vgpr0_lo16, implicit-def renamable $vgpr0_hi16
@@ -124,6 +170,27 @@ body: |
   ; EXPANDED-NEXT: bb.2:
   ; EXPANDED-NEXT:   $vgpr0_lo16 = SCRATCH_LOAD_SHORT_D16_SADDR_t16 $sgpr32, 0, 0, implicit $exec, implicit $flat_scr :: (load (s16) from %stack.0, align 4, addrspace 5)
   ; EXPANDED-NEXT:   $vgpr0_hi16 = SCRATCH_LOAD_SHORT_D16_SADDR_t16 $sgpr32, 4, 0, implicit $exec, implicit $flat_scr :: (load (s16) from %stack.1, align 4, addrspace 5)
+  ;
+  ; SRAMECC-EXPANDED-LABEL: name: spill_restore_vgpr16_end_of_block
+  ; SRAMECC-EXPANDED: bb.0:
+  ; SRAMECC-EXPANDED-NEXT:   successors: %bb.1(0x80000000)
+  ; SRAMECC-EXPANDED-NEXT: {{  $}}
+  ; SRAMECC-EXPANDED-NEXT:   S_NOP 0, implicit-def renamable $vgpr0_lo16, implicit-def renamable $vgpr0_hi16
+  ; SRAMECC-EXPANDED-NEXT:   SCRATCH_STORE_SHORT_SADDR_t16 killed $vgpr0_hi16, $sgpr32, 4, 0, implicit $exec, implicit $flat_scr :: (store (s16) into %stack.1, align 4, addrspace 5)
+  ; SRAMECC-EXPANDED-NEXT:   S_NOP 0, implicit renamable $vgpr0_lo16
+  ; SRAMECC-EXPANDED-NEXT:   SCRATCH_STORE_SHORT_SADDR_t16 killed $vgpr0_lo16, $sgpr32, 0, 0, implicit $exec, implicit $flat_scr :: (store (s16) into %stack.0, align 4, addrspace 5)
+  ; SRAMECC-EXPANDED-NEXT:   S_CBRANCH_SCC1 %bb.1, implicit undef $scc
+  ; SRAMECC-EXPANDED-NEXT: {{  $}}
+  ; SRAMECC-EXPANDED-NEXT: bb.1:
+  ; SRAMECC-EXPANDED-NEXT:   successors: %bb.2(0x80000000)
+  ; SRAMECC-EXPANDED-NEXT: {{  $}}
+  ; SRAMECC-EXPANDED-NEXT:   S_NOP 1
+  ; SRAMECC-EXPANDED-NEXT: {{  $}}
+  ; SRAMECC-EXPANDED-NEXT: bb.2:
+  ; SRAMECC-EXPANDED-NEXT:   $vgpr1 = SCRATCH_LOAD_UBYTE_SADDR $sgpr32, 0, 0, implicit $exec, implicit $flat_scr :: (load (s16) from %stack.0, align 4, addrspace 5)
+  ; SRAMECC-EXPANDED-NEXT:   $vgpr0_lo16 = V_MOV_B16_t16_e32 killed $vgpr1_lo16, implicit $exec
+  ; SRAMECC-EXPANDED-NEXT:   $vgpr1 = SCRATCH_LOAD_UBYTE_SADDR $sgpr32, 4, 0, implicit $exec, implicit $flat_scr :: (load (s16) from %stack.1, align 4, addrspace 5)
+  ; SRAMECC-EXPANDED-NEXT:   $vgpr0_hi16 = V_MOV_B16_t16_e32 killed $vgpr1_lo16, implicit $exec
    bb.0:
      successors: %bb.1(0x80000000)
      S_NOP 0, implicit-def renamable $vgpr0_lo16, implicit-def renamable $vgpr0_hi16
diff --git a/llvm/test/CodeGen/AMDGPU/spillv16.ll b/llvm/test/CodeGen/AMDGPU/spillv16.ll
index 0e45df223465d..21059f7ef4a9d 100644
--- a/llvm/test/CodeGen/AMDGPU/spillv16.ll
+++ b/llvm/test/CodeGen/AMDGPU/spillv16.ll
@@ -1,6 +1,8 @@
 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --version 2
 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1100 -mattr=+real-true16 -enable-misched=0 -post-RA-scheduler=0 -stress-regalloc=8 < %s | FileCheck %s -check-prefixes=GCN,GCN-TRUE16
 ; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1100 -mattr=-real-true16 -enable-misched=0 -post-RA-scheduler=0 -stress-regalloc=8 < %s | FileCheck %s -check-prefixes=GCN,GCN-FAKE16
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1250 -mattr=+real-true16 -enable-misched=0 -post-RA-scheduler=0 -stress-regalloc=8 < %s | FileCheck %s -check-prefixes=GFX1250,GFX1250-TRUE16
+; RUN: llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx1250 -mattr=-real-true16 -enable-misched=0 -post-RA-scheduler=0 -stress-regalloc=8 < %s | FileCheck %s -check-prefixes=GFX1250,GFX1250-FAKE16
 
 define void @spill_i16_alu() {
 ; GCN-TRUE16-LABEL: spill_i16_alu:
@@ -32,6 +34,41 @@ define void @spill_i16_alu() {
 ; GCN-FAKE16-NEXT:    scratch_store_b16 off, v0, s32 dlc
 ; GCN-FAKE16-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GCN-FAKE16-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX1250-TRUE16-LABEL: spill_i16_alu:
+; GFX1250-TRUE16:       ; %bb.0: ; %entry
+; GFX1250-TRUE16-NEXT:    s_wait_loadcnt_dscnt 0x0
+; GFX1250-TRUE16-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-TRUE16-NEXT:    scratch_load_u16 v0, off, s32 scope:SCOPE_SYS
+; GFX1250-TRUE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-TRUE16-NEXT:    v_add_nc_u16 v0.l, 0x7b, v0.l
+; GFX1250-TRUE16-NEXT:    scratch_store_b16 off, v0, s32 offset:2 ; 2-byte Folded Spill
+; GFX1250-TRUE16-NEXT:    s_wait_xcnt 0x0
+; GFX1250-TRUE16-NEXT:    ;;#ASMSTART
+; GFX1250-TRUE16-NEXT:    ;;#ASMEND
+; GFX1250-TRUE16-NEXT:    scratch_load_u8 v1, off, s32 offset:2 th:TH_LOAD_LU ; 2-byte Folded Reload
+; GFX1250-TRUE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-TRUE16-NEXT:    v_mov_b16_e32 v0.l, v1.l
+; GFX1250-TRUE16-NEXT:    scratch_store_b16 off, v0, s32 scope:SCOPE_SYS
+; GFX1250-TRUE16-NEXT:    s_wait_storecnt 0x0
+; GFX1250-TRUE16-NEXT:    s_set_pc_i64 s[30:31]
+;
+; GFX1250-FAKE16-LABEL: spill_i16_alu:
+; GFX1250-FAKE16:       ; %bb.0: ; %entry
+; GFX1250-FAKE16-NEXT:    s_wait_loadcnt_dscnt 0x0
+; GFX1250-FAKE16-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-FAKE16-NEXT:    scratch_load_u16 v0, off, s32 scope:SCOPE_SYS
+; GFX1250-FAKE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-FAKE16-NEXT:    v_add_nc_u16 v0, 0x7b, v0
+; GFX1250-FAKE16-NEXT:    scratch_store_b32 off, v0, s32 offset:4 ; 4-byte Folded Spill
+; GFX1250-FAKE16-NEXT:    s_wait_xcnt 0x0
+; GFX1250-FAKE16-NEXT:    ;;#ASMSTART
+; GFX1250-FAKE16-NEXT:    ;;#ASMEND
+; GFX1250-FAKE16-NEXT:    scratch_load_b32 v0, off, s32 offset:4 th:TH_LOAD_LU ; 4-byte Folded Reload
+; GFX1250-FAKE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-FAKE16-NEXT:    scratch_store_b16 off, v0, s32 scope:SCOPE_SYS
+; GFX1250-FAKE16-NEXT:    s_wait_storecnt 0x0
+; GFX1250-FAKE16-NEXT:    s_set_pc_i64 s[30:31]
 entry:
   %alloca = alloca i16, i32 1, align 4, addrspace(5)
 
@@ -88,6 +125,51 @@ define void @spill_i16_alu_two_vals() {
 ; GCN-FAKE16-NEXT:    scratch_store_b16 off, v0, s32 offset:4 dlc
 ; GCN-FAKE16-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GCN-FAKE16-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX1250-TRUE16-LABEL: spill_i16_alu_two_vals:
+; GFX1250-TRUE16:       ; %bb.0: ; %entry
+; GFX1250-TRUE16-NEXT:    s_wait_loadcnt_dscnt 0x0
+; GFX1250-TRUE16-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-TRUE16-NEXT:    scratch_load_u16 v0, off, s32 scope:SCOPE_SYS
+; GFX1250-TRUE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-TRUE16-NEXT:    v_add_nc_u16 v0.l, 0x7b, v0.l
+; GFX1250-TRUE16-NEXT:    scratch_store_b16 off, v0, s32 offset:6 ; 2-byte Folded Spill
+; GFX1250-TRUE16-NEXT:    s_wait_xcnt 0x0
+; GFX1250-TRUE16-NEXT:    ;;#ASMSTART
+; GFX1250-TRUE16-NEXT:    ;;#ASMEND
+; GFX1250-TRUE16-NEXT:    scratch_load_u16 v0, off, s32 offset:4 scope:SCOPE_SYS
+; GFX1250-TRUE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-TRUE16-NEXT:    scratch_load_u8 v1, off, s32 offset:6 th:TH_LOAD_LU ; 2-byte Folded Reload
+; GFX1250-TRUE16-NEXT:    v_add_nc_u16 v0.l, 0x7b, v0.l
+; GFX1250-TRUE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-TRUE16-NEXT:    v_mov_b16_e32 v0.h, v1.l
+; GFX1250-TRUE16-NEXT:    scratch_store_d16_hi_b16 off, v0, s32 scope:SCOPE_SYS
+; GFX1250-TRUE16-NEXT:    s_wait_storecnt 0x0
+; GFX1250-TRUE16-NEXT:    scratch_store_b16 off, v0, s32 offset:4 scope:SCOPE_SYS
+; GFX1250-TRUE16-NEXT:    s_wait_storecnt 0x0
+; GFX1250-TRUE16-NEXT:    s_set_pc_i64 s[30:31]
+;
+; GFX1250-FAKE16-LABEL: spill_i16_alu_two_vals:
+; GFX1250-FAKE16:       ; %bb.0: ; %entry
+; GFX1250-FAKE16-NEXT:    s_wait_loadcnt_dscnt 0x0
+; GFX1250-FAKE16-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-FAKE16-NEXT:    scratch_load_u16 v0, off, s32 scope:SCOPE_SYS
+; GFX1250-FAKE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-FAKE16-NEXT:    v_add_nc_u16 v0, 0x7b, v0
+; GFX1250-FAKE16-NEXT:    scratch_store_b32 off, v0, s32 offset:8 ; 4-byte Folded Spill
+; GFX1250-FAKE16-NEXT:    s_wait_xcnt 0x0
+; GFX1250-FAKE16-NEXT:    ;;#ASMSTART
+; GFX1250-FAKE16-NEXT:    ;;#ASMEND
+; GFX1250-FAKE16-NEXT:    scratch_load_u16 v0, off, s32 offset:4 scope:SCOPE_SYS
+; GFX1250-FAKE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-FAKE16-NEXT:    scratch_load_b32 v1, off, s32 offset:8 th:TH_LOAD_LU ; 4-byte Folded Reload
+; GFX1250-FAKE16-NEXT:    v_add_nc_u16 v0, 0x7b, v0
+; GFX1250-FAKE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-FAKE16-NEXT:    scratch_store_b16 off, v1, s32 scope:SCOPE_SYS
+; GFX1250-FAKE16-NEXT:    s_wait_storecnt 0x0
+; GFX1250-FAKE16-NEXT:    scratch_store_b16 off, v0, s32 offset:4 scope:SCOPE_SYS
+; GFX1250-FAKE16-NEXT:    s_wait_storecnt 0x0
+; GFX1250-FAKE16-NEXT:    s_set_pc_i64 s[30:31]
 entry:
   %alloca = alloca i16, i32 1, align 4, addrspace(5)
   %alloca2 = alloca i16, i32 1, align 4, addrspace(5)
@@ -140,6 +222,22 @@ define void @spill_i16() {
 ; GCN-FAKE16-NEXT:    scratch_store_b16 off, v0, s32 dlc
 ; GCN-FAKE16-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GCN-FAKE16-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX1250-LABEL: spill_i16:
+; GFX1250:       ; %bb.0: ; %entry
+; GFX1250-NEXT:    s_wait_loadcnt_dscnt 0x0
+; GFX1250-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-NEXT:    scratch_load_u16 v0, off, s32 scope:SCOPE_SYS
+; GFX1250-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-NEXT:    scratch_store_b32 off, v0, s32 offset:4 ; 4-byte Folded Spill
+; GFX1250-NEXT:    s_wait_xcnt 0x0
+; GFX1250-NEXT:    ;;#ASMSTART
+; GFX1250-NEXT:    ;;#ASMEND
+; GFX1250-NEXT:    scratch_load_b32 v0, off, s32 offset:4 th:TH_LOAD_LU ; 4-byte Folded Reload
+; GFX1250-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-NEXT:    scratch_store_b16 off, v0, s32 scope:SCOPE_SYS
+; GFX1250-NEXT:    s_wait_storecnt 0x0
+; GFX1250-NEXT:    s_set_pc_i64 s[30:31]
 entry:
   %alloca = alloca i16, i32 1, align 4, addrspace(5)
 
@@ -183,6 +281,22 @@ define void @spill_half() {
 ; GCN-FAKE16-NEXT:    scratch_store_b16 off, v0, s32 dlc
 ; GCN-FAKE16-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GCN-FAKE16-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX1250-LABEL: spill_half:
+; GFX1250:       ; %bb.0: ; %entry
+; GFX1250-NEXT:    s_wait_loadcnt_dscnt 0x0
+; GFX1250-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-NEXT:    scratch_load_u16 v0, off, s32 scope:SCOPE_SYS
+; GFX1250-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-NEXT:    scratch_store_b32 off, v0, s32 offset:4 ; 4-byte Folded Spill
+; GFX1250-NEXT:    s_wait_xcnt 0x0
+; GFX1250-NEXT:    ;;#ASMSTART
+; GFX1250-NEXT:    ;;#ASMEND
+; GFX1250-NEXT:    scratch_load_b32 v0, off, s32 offset:4 th:TH_LOAD_LU ; 4-byte Folded Reload
+; GFX1250-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-NEXT:    scratch_store_b16 off, v0, s32 scope:SCOPE_SYS
+; GFX1250-NEXT:    s_wait_storecnt 0x0
+; GFX1250-NEXT:    s_set_pc_i64 s[30:31]
 entry:
   %alloca = alloca half, i32 1, align 4, addrspace(5)
 
@@ -226,6 +340,22 @@ define void @spill_i16_from_v2i16() {
 ; GCN-FAKE16-NEXT:    scratch_store_b16 off, v0, s32 offset:2 dlc
 ; GCN-FAKE16-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GCN-FAKE16-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX1250-LABEL: spill_i16_from_v2i16:
+; GFX1250:       ; %bb.0: ; %entry
+; GFX1250-NEXT:    s_wait_loadcnt_dscnt 0x0
+; GFX1250-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-NEXT:    scratch_load_u16 v0, off, s32 offset:2 scope:SCOPE_SYS
+; GFX1250-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-NEXT:    scratch_store_b32 off, v0, s32 offset:8 ; 4-byte Folded Spill
+; GFX1250-NEXT:    s_wait_xcnt 0x0
+; GFX1250-NEXT:    ;;#ASMSTART
+; GFX1250-NEXT:    ;;#ASMEND
+; GFX1250-NEXT:    scratch_load_b32 v0, off, s32 offset:8 th:TH_LOAD_LU ; 4-byte Folded Reload
+; GFX1250-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-NEXT:    scratch_store_b16 off, v0, s32 offset:2 scope:SCOPE_SYS
+; GFX1250-NEXT:    s_wait_storecnt 0x0
+; GFX1250-NEXT:    s_set_pc_i64 s[30:31]
 entry:
   %alloca = alloca <2 x i16>, i32 2, align 1, addrspace(5)
 
@@ -283,6 +413,54 @@ define void @spill_2xi16_from_v2i16() {
 ; GCN-FAKE16-NEXT:    scratch_store_b16 off, v0, s32 dlc
 ; GCN-FAKE16-NEXT:    s_waitcnt_vscnt null, 0x0
 ; GCN-FAKE16-NEXT:    s_setpc_b64 s[30:31]
+;
+; GFX1250-TRUE16-LABEL: spill_2xi16_from_v2i16:
+; GFX1250-TRUE16:       ; %bb.0: ; %entry
+; GFX1250-TRUE16-NEXT:    s_wait_loadcnt_dscnt 0x0
+; GFX1250-TRUE16-NEXT:    s_wait_kmcnt 0x0
+; GFX1250-TRUE16-NEXT:    scratch_load_u16 v0, off, s32 offset:2 scope:SCOPE_SYS
+; GFX1250-TRUE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-TRUE16-NEXT:    s_clause 0x1
+; GFX1250-TRUE16-NEXT:    scratch_store_b32 off, v0, s32 offset:12
+; GFX1250-TRUE16-NEXT:    scratch_load_u16 v0, off, s32 scope:SCOPE_SYS
+; GFX1250-TRUE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-TRUE16-NEXT:    scratch_store_b32 off, v0, s32 offset:8 ; 4-byte Folded Spill
+; GFX1250-TRUE16-NEXT:    s_wait_xcnt 0x0
+; GFX1250-TRUE16-NEXT:    ;;#ASMSTART
+; GFX1250-TRUE16-NEXT:    ;;#ASMEND
+; GFX1250-TRUE16-NEXT:    scratch_load_b32 v0, off, s32 offset:12 th:TH_LOAD_LU ; 4-byte Folded Reload
+; GFX1250-TRUE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-TRUE16-NEXT:    scratch_store_b16 off, v0, s32 offset:2 scope:SCOPE_SYS
+; GFX1250-TRUE16-NEXT:    s_wait_storecnt 0x0
+; GFX1250-TRUE16-NEXT:    scratch_load_b32 v0, off, s32 offset:8 th:TH_LOAD_LU ; 4-byte Folded Reload
+; GFX1250-TRUE16-NEXT:    s_wait_loadcnt 0x0
+; GFX1250-TRUE16-NEXT:    scratch_store_b16 off, v0, s32 scope:SCOPE_SYS
+; GFX1250-TRUE16-NEXT:    s_wait_storecnt 0x0
+; GFX1250-TRUE16-NEXT:    s_set_pc_i64 s[30:31]
+;
+; GFX1250-FAKE16-LABEL: spill_2xi16_from_v2i16:
+; GFX1250-FAKE16:       ; %bb.0: ; %entry
+; GFX1250-FAKE16-NEXT:    s_wait_loadcnt_dscnt 0x0
+; GFX1250-FAKE16-NEXT:  ...
[truncated]

jayfoad
jayfoad reviewed Oct 28, 2025
View reviewed changes
@rampitec rampitec force-pushed the users/rampitec/10-27-_amdgpu_support_true16_spill_restore_with_sram-ecc branch from 3fe3338 to 1a75d94 Compare October 28, 2025 18:47
rampitec
rampitec commented Oct 29, 2025
View reviewed changes
llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp

if (FinalValueReg != ValueReg) {
// Extract 16-bit from the loaded 32-bit value.
ValueReg = getSubReg(ValueReg, AMDGPU::lo16);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically I can even use the same SCRATCH_LOAD_SHORT_D16_SADDR_t16 as w/o sramecc, but I would need to chose lo16 or hi16 here. I do not think this is really needed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. It is simpler the way you have it now.

jayfoad
jayfoad reviewed Oct 29, 2025
View reviewed changes
@rampitec rampitec force-pushed the users/rampitec/10-27-_amdgpu_support_true16_spill_restore_with_sram-ecc branch from 1a75d94 to c5b3f1b Compare October 29, 2025 10:16
Sisyph
Sisyph reviewed Oct 29, 2025
View reviewed changes
@rampitec rampitec force-pushed the users/rampitec/10-27-_amdgpu_support_true16_spill_restore_with_sram-ecc branch from c5b3f1b to f0a9a9d Compare October 29, 2025 18:48
Sisyph
Sisyph approved these changes Oct 29, 2025
View reviewed changes
Copy link
Contributor

@Sisyph Sisyph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

llvm/test/CodeGen/AMDGPU/spillv16.ll
; GFX1250-TRUE16-NEXT: s_wait_xcnt 0x0
; GFX1250-TRUE16-NEXT: ;;#ASMSTART
; GFX1250-TRUE16-NEXT: ;;#ASMEND
; GFX1250-TRUE16-NEXT: v_mov_b16_e32 v0.l, v7.l
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A little unfortunate we can't fold this mov away since it gets inserted late.

@rampitec rampitec merged commit 5f1813e into main Oct 29, 2025
10 checks passed
@rampitec rampitec deleted the users/rampitec/10-27-_amdgpu_support_true16_spill_restore_with_sram-ecc branch October 29, 2025 19:35
aokblast pushed a commit to aokblast/llvm-project that referenced this pull request Oct 30, 2025
@rampitec @aokblast
[AMDGPU] Support true16 spill restore with sram-ecc (llvm#165320)
de69e7f
arsenm
arsenm reviewed Oct 30, 2025
View reviewed changes
llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp

Register FinalValueReg = ValueReg;
if (LoadStoreOp == AMDGPU::SCRATCH_LOAD_USHORT_SADDR) {
// If we are loading 16-bit value with SRAMECC endabled we need a temp
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no check for sramecc enabled?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SCRATCH_LOAD_USHORT_SADDR is only used with sramecc, that was checked earlier.

llvm/lib/Target/AMDGPU/SIRegisterInfo.cpp

Register FinalValueReg = ValueReg;
if (LoadStoreOp == AMDGPU::SCRATCH_LOAD_USHORT_SADDR) {
// If we are loading 16-bit value with SRAMECC endabled we need a temp
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this also an issue for the 8-bit cases?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not have V8 spills.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jayfoad jayfoad jayfoad left review comments

@arsenm arsenm arsenm left review comments

@Sisyph Sisyph Sisyph approved these changes

@broxigarchen broxigarchen Awaiting requested review from broxigarchen

Assignees

No one assigned

Labels

backend:AMDGPU

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@rampitec @llvmbot @jayfoad @arsenm @Sisyph