[AMDGPU][True16] si-fix-sgpr-copies: invalid sgpr_lo16 copy destination

[frederik-h]

Running pass si-fix-sgpr-copies on the following machine ir changes the %7:vgpr_16 operand of the second V_CNDMASK_B16_t16_e64 into a register of type sgpr_lo16 which is invalid.

test.mir

---
name: sgpr_copy_invalid_type
tracksRegLiveness: true
body:             |
  bb.0:
    %0:vgpr_32 = IMPLICIT_DEF
    %1:sreg_32 = IMPLICIT_DEF
    %2:sreg_32_xm0_xexec = IMPLICIT_DEF

    %3:sgpr_lo16 = COPY undef %0.lo16
    %4:sreg_32 = COPY undef %3
    %5:sreg_32 = S_AND_B32 undef %4, killed undef %1, implicit-def dead $scc
    %6:sgpr_32 = S_CVT_F32_F16 killed undef %5, implicit $mode

    %7:vgpr_16 = COPY undef %3
    %8:vgpr_16 = V_CNDMASK_B16_t16_e64 0, undef %7, 0, 1, killed undef %2, 0, implicit $exec

    S_ENDPGM 0

  bb.2:
    successors: %bb.0(0x80000000)

    S_CMP_LG_U32 killed undef %4, killed undef %4, implicit-def $scc
    S_CBRANCH_SCC1 %bb.0, implicit undef $scc
    S_ENDPGM 0
...

llc invocation

llc -mtriple=amdgcn -mcpu=gfx1150 -mattr=+real-true16 -print-changed=cdiff -run-pass=si-fix-sgpr-copies -debug-only=si-fix-sgpr-copies -verify-machineinstrs test.mir

Error message from machine verifier

*** Bad machine code: Illegal virtual register for instruction ***
- function:    sgpr_copy_invalid_type
- basic block: %bb.0  (0x557e9fcd29b8)
- instruction: %8:vgpr_16 = V_CNDMASK_B16_t16_e64 0, undef %7:sgpr_lo16, 0, 1, killed undef %2:sreg_32_xm0_xexec, 0, implicit $exec
- operand 2:   undef %7:sgpr_lo16
Expected a VS_16 register, but got a SGPR_LO16 register

Debug output from si-fix-sgpr-copies

V2S copy %3:sgpr_lo16 = COPY undef %0.lo16:vgpr_32
 is being turned to v_readfirstlane_b32 Score: 3
*** IR Dump After SI Fix SGPR copies (si-fix-sgpr-copies) on sgpr_copy_invalid_type ***
 # Machine code for function sgpr_copy_invalid_type: IsSSA, NoPHIs, TracksLiveness

 bb.0:
 ; predecessors: %bb.1

   %0:vgpr_32 = IMPLICIT_DEF
   %1:sreg_32 = IMPLICIT_DEF
   %2:sreg_32_xm0_xexec = IMPLICIT_DEF
-  %3:sgpr_lo16 = COPY undef %0.lo16:vgpr_32
-  %4:sreg_32 = COPY undef %3:sgpr_lo16
+  %10:vgpr_16 = IMPLICIT_DEF
+  %9:vgpr_32 = REG_SEQUENCE %0.lo16:vgpr_32, %subreg.lo16, %10:vgpr_16, %subreg.hi16
+  %3:sreg_32_xm0 = V_READFIRSTLANE_B32 %9:vgpr_32, implicit $exec
+  %4:sreg_32 = COPY undef %3:sreg_32_xm0
   %5:sreg_32 = S_AND_B32 undef %4:sreg_32, killed undef %1:sreg_32, implicit-def dead $scc
   %6:sgpr_32 = S_CVT_F32_F16 killed undef %5:sreg_32, implicit $mode
-  %7:vgpr_16 = COPY undef %3:sgpr_lo16
-  %8:vgpr_16 = V_CNDMASK_B16_t16_e64 0, undef %7:vgpr_16, 0, 1, killed undef %2:sreg_32_xm0_xexec, 0, implicit $exec
+  %7:sgpr_lo16 = COPY undef %3:sreg_32_xm0
+  %8:vgpr_16 = V_CNDMASK_B16_t16_e64 0, undef %7:sgpr_lo16, 0, 1, killed undef %2:sreg_32_xm0_xexec, 0, implicit $exec
   S_ENDPGM 0

 bb.1:
   successors: %bb.0(0x80000000); %bb.0(100.00%)

   S_CMP_LG_U32 killed undef %4:sreg_32, killed undef %4:sreg_32, implicit-def $scc
   S_CBRANCH_SCC1 %bb.0, implicit undef $scc
   S_ENDPGM 0

 # End machine code for function sgpr_copy_invalid_type.

Related commit

[AMDGPU][True16][CodeGen] readfirstlane for vgpr16 copy to sgpr32 by @broxigarchen

[llvmbot]

@llvm/issue-subscribers-backend-amdgpu

Author: Frederik Harwath (frederik-h)

Running pass `si-fix-sgpr-copies` on the following machine ir changes the `%7:vgpr_16` operand of the second ` V_CNDMASK_B16_t16_e64 ` into a register of type `sgpr_lo16` which is invalid.

test.mir

---
name: sgpr_copy_invalid_type
tracksRegLiveness: true
body:             |
  bb.0:
    %0:vgpr_32 = IMPLICIT_DEF
    %1:sreg_32 = IMPLICIT_DEF
    %2:sreg_32_xm0_xexec = IMPLICIT_DEF

    %3:sgpr_lo16 = COPY undef %0.lo16
    %4:sreg_32 = COPY undef %3
    %5:sreg_32 = S_AND_B32 undef %4, killed undef %1, implicit-def dead $scc
    %6:sgpr_32 = S_CVT_F32_F16 killed undef %5, implicit $mode

    %7:vgpr_16 = COPY undef %3
    %8:vgpr_16 = V_CNDMASK_B16_t16_e64 0, undef %7, 0, 1, killed undef %2, 0, implicit $exec

    S_ENDPGM 0

  bb.2:
    successors: %bb.0(0x80000000)

    S_CMP_LG_U32 killed undef %4, killed undef %4, implicit-def $scc
    S_CBRANCH_SCC1 %bb.0, implicit undef $scc
    S_ENDPGM 0
...

llc invocation

llc -mtriple=amdgcn -mcpu=gfx1150 -mattr=+real-true16 -print-changed=cdiff -run-pass=si-fix-sgpr-copies -debug-only=si-fix-sgpr-copies -verify-machineinstrs test.mir

Error message from machine verifier

*** Bad machine code: Illegal virtual register for instruction ***
- function:    sgpr_copy_invalid_type
- basic block: %bb.0  (0x557e9fcd29b8)
- instruction: %8:vgpr_16 = V_CNDMASK_B16_t16_e64 0, undef %7:sgpr_lo16, 0, 1, killed undef %2:sreg_32_xm0_xexec, 0, implicit $exec
- operand 2:   undef %7:sgpr_lo16
Expected a VS_16 register, but got a SGPR_LO16 register

Debug output from si-fix-sgpr-copies

V2S copy %3:sgpr_lo16 = COPY undef %0.lo16:vgpr_32
 is being turned to v_readfirstlane_b32 Score: 3
*** IR Dump After SI Fix SGPR copies (si-fix-sgpr-copies) on sgpr_copy_invalid_type ***
 # Machine code for function sgpr_copy_invalid_type: IsSSA, NoPHIs, TracksLiveness

 bb.0:
 ; predecessors: %bb.1

   %0:vgpr_32 = IMPLICIT_DEF
   %1:sreg_32 = IMPLICIT_DEF
   %2:sreg_32_xm0_xexec = IMPLICIT_DEF
-  %3:sgpr_lo16 = COPY undef %0.lo16:vgpr_32
-  %4:sreg_32 = COPY undef %3:sgpr_lo16
+  %10:vgpr_16 = IMPLICIT_DEF
+  %9:vgpr_32 = REG_SEQUENCE %0.lo16:vgpr_32, %subreg.lo16, %10:vgpr_16, %subreg.hi16
+  %3:sreg_32_xm0 = V_READFIRSTLANE_B32 %9:vgpr_32, implicit $exec
+  %4:sreg_32 = COPY undef %3:sreg_32_xm0
   %5:sreg_32 = S_AND_B32 undef %4:sreg_32, killed undef %1:sreg_32, implicit-def dead $scc
   %6:sgpr_32 = S_CVT_F32_F16 killed undef %5:sreg_32, implicit $mode
-  %7:vgpr_16 = COPY undef %3:sgpr_lo16
-  %8:vgpr_16 = V_CNDMASK_B16_t16_e64 0, undef %7:vgpr_16, 0, 1, killed undef %2:sreg_32_xm0_xexec, 0, implicit $exec
+  %7:sgpr_lo16 = COPY undef %3:sreg_32_xm0
+  %8:vgpr_16 = V_CNDMASK_B16_t16_e64 0, undef %7:sgpr_lo16, 0, 1, killed undef %2:sreg_32_xm0_xexec, 0, implicit $exec
   S_ENDPGM 0

 bb.1:
   successors: %bb.0(0x80000000); %bb.0(100.00%)

   S_CMP_LG_U32 killed undef %4:sreg_32, killed undef %4:sreg_32, implicit-def $scc
   S_CBRANCH_SCC1 %bb.0, implicit undef $scc
   S_ENDPGM 0

 # End machine code for function sgpr_copy_invalid_type.

Related commit

[AMDGPU][True16][CodeGen] readfirstlane for vgpr16 copy to sgpr32 by @broxigarchen

[Sisyph]

Thanks for the report! IIUC, is is only an emitted code issue at O0, because si-fold-operands fixes up the illegal registers. It's still an important issue to fix.
We well have to see if it makes sense to fix it in si-fix-sgpr-copies, or perhaps earlier. The presence of sgpr_lo16 registers in the input mir is questionable.

[frederik-h]

The presence of sgpr_lo16 registers in the input mir is questionable.

Here is an llvm ir reproducer:

define void @widget(i1 %arg) {
bb:
  %load = load <2 x half>, ptr addrspace(1) null, align 8
  %extractelement = extractelement <2 x half> %load, i64 0
  %call = call half @llvm.fabs.f16(half %extractelement)
  %fpext = fpext half %call to float
  %fcmp = fcmp ogt float %fpext, 0.000000e+00
  br i1 %fcmp, label %bb1, label %bb2

bb1:
  ret void

bb2:
  %select = select i1 %arg, half %extractelement, half 0xH0000
  %insertelement = insertelement <2 x half> zeroinitializer, half %select, i64 0
  store <2 x half> %insertelement, ptr addrspace(1) null, align 8

  br label %bb1
}

Compiling with llc -mtriple=amdgcn -mcpu=gfx1150 -mattr=+real-true16 -verify-machineinstrs should lead to the same error.

[broxigarchen]

Yeah Thanks for the report! I think this also happens in O3 since there is no -O0 in the llc cmd line.

The problem is from this copy between sgpr_lo16 and vgpr16 while the 16bit v2s copy is lowered to a 32bit readfirstlane, and we didn't legalize the use MI after we change the size of the register

%3:sgpr_lo16 = COPY undef %0.lo16
%7:vgpr_16 = COPY undef %3
%8:vgpr_16 = V_CNDMASK_B16_t16_e64 0, undef %7, 0, 1, killed undef %2, 0, implicit $exec