[RISCV] The LLVM JIT appears to be broken on RISC-V

[andreas-schwab]

Running the testsuite of perl-OpenGL, the test is hanging in an infinite loop inside JIT code. In the JIT code, all subroutine calls are turned into a auipc ra,0; jalr ra sequence, so the actual subroutine addresses are not properly resolved.

This is the start of the JIT code generated by llvmpipe of Mesa (called by llvm_pipeline_generic):

(gdb) x/100i $t1
   0x3ff74d5000:        addi    sp,sp,-1344
   0x3ff74d5004:        sd      ra,1336(sp)
   0x3ff74d5008:        sd      s0,1328(sp)
   0x3ff74d500c:        sd      s1,1320(sp)
   0x3ff74d5010:        sd      s2,1312(sp)
   0x3ff74d5014:        sd      s3,1304(sp)
   0x3ff74d5018:        sd      s4,1296(sp)
   0x3ff74d501c:        sd      s5,1288(sp)
   0x3ff74d5020:        sd      s6,1280(sp)
   0x3ff74d5024:        sd      s7,1272(sp)
   0x3ff74d5028:        sd      s8,1264(sp)
   0x3ff74d502c:        sd      s9,1256(sp)
   0x3ff74d5030:        sd      s10,1248(sp)
   0x3ff74d5034:        sd      s11,1240(sp)
   0x3ff74d5038:        addi    s0,sp,1344
   0x3ff74d503c:        andi    sp,sp,-32
   0x3ff74d5040:        lw      a7,8(a2)
   0x3ff74d5044:        lw      a5,4(a6)
   0x3ff74d5048:        sd      a4,24(sp)
   0x3ff74d504c:        sd      a0,1176(sp)
   0x3ff74d5050:        sd      zero,120(sp)
   0x3ff74d5054:        addiw   t0,a7,-11
   0x3ff74d5058:        sltu    a4,a7,t0
   0x3ff74d505c:        subw    a0,t0,a5
   0x3ff74d5060:        sltu    a7,t0,a0
   0x3ff74d5064:        or      a4,a4,a7
   0x3ff74d5068:        addi    a7,sp,1184
   0x3ff74d506c:        sd      a7,112(sp)
   0x3ff74d5070:        addi    a7,sp,1184
   0x3ff74d5074:        sd      a7,104(sp)
   0x3ff74d5078:        bnez    a4,0x3ff74d508c
   0x3ff74d507c:        ld      a4,0(a2)
   0x3ff74d5080:        add     a4,a4,a5
   0x3ff74d5084:        sd      a4,104(sp)
   0x3ff74d5088:        sd      a0,120(sp)
   0x3ff74d508c:        lw      a0,24(a2)
   0x3ff74d5090:        sd      zero,96(sp)
   0x3ff74d5094:        lw      a4,20(a6)
   0x3ff74d5098:        ld      t0,16(a2)
   0x3ff74d509c:        addiw   a5,a0,-11
   0x3ff74d50a0:        sltu    a7,a0,a5
   0x3ff74d50a4:        subw    a2,a5,a4
   0x3ff74d50a8:        sltu    a5,a5,a2
   0x3ff74d50ac:        or      a5,a7,a5
   0x3ff74d50b0:        addi    a7,sp,1184
   0x3ff74d50b4:        sd      a7,88(sp)
   0x3ff74d50b8:        bnez    a5,0x3ff74d50c8
   0x3ff74d50bc:        add     a5,t0,a4
   0x3ff74d50c0:        sd      a5,88(sp)
   0x3ff74d50c4:        sd      a2,96(sp)
   0x3ff74d50c8:        sd      a1,72(sp)
   0x3ff74d50cc:        sd      zero,80(sp)
   0x3ff74d50d0:        lhu     a1,16(a6)
   0x3ff74d50d4:        sd      a1,64(sp)
   0x3ff74d50d8:        addiw   a2,a4,16
   0x3ff74d50dc:        addiw   a4,a0,-3
   0x3ff74d50e0:        sltu    a5,a0,a4
   0x3ff74d50e4:        subw    a0,a4,a2
   0x3ff74d50e8:        sltu    a4,a4,a0
   0x3ff74d50ec:        or      a4,a5,a4
   0x3ff74d50f0:        addiw   s1,a3,-1
   0x3ff74d50f4:        bnez    a4,0x3ff74d5104
   0x3ff74d50f8:        add     a1,t0,a2
   0x3ff74d50fc:        sd      a1,112(sp)
   0x3ff74d5100:        sd      a0,80(sp)
   0x3ff74d5104:        li      s2,0
   0x3ff74d5108:        sd      zero,968(sp)
   0x3ff74d510c:        sd      zero,960(sp)
   0x3ff74d5110:        sd      zero,952(sp)
   0x3ff74d5114:        sd      zero,944(sp)
   0x3ff74d5118:        ld      s3,16(s0)
   0x3ff74d511c:        lhu     a0,0(a6)
   0x3ff74d5120:        sd      a0,40(sp)
   0x3ff74d5124:        sext.w  a0,a3
   0x3ff74d5128:        sd      a0,32(sp)
   0x3ff74d512c:        sd      s1,56(sp)
   0x3ff74d5130:        sd      s3,48(sp)
   0x3ff74d5134:        j       0x3ff74d5a94
   0x3ff74d5138:        sd      a1,976(sp)
   0x3ff74d513c:        li      a1,68
   0x3ff74d5140:        ld      a0,1016(sp)
   0x3ff74d5144:        auipc   ra,0x0
   0x3ff74d5148:        jalr    ra
   0x3ff74d514c:        add     a2,s2,a0
   0x3ff74d5150:        sd      a2,904(sp)
   0x3ff74d5154:        ld      s7,1072(sp)
   0x3ff74d5158:        sw      s7,16(a2)
   0x3ff74d515c:        ld      a0,1104(sp)
   0x3ff74d5160:        sw      a0,12(a2)
   0x3ff74d5164:        ld      s4,1120(sp)
   0x3ff74d5168:        sw      s4,8(a2)
   0x3ff74d516c:        ld      a0,1168(sp)
   0x3ff74d5170:        sw      a0,4(a2)
   0x3ff74d5174:        ld      a2,1096(sp)
   0x3ff74d5178:        sw      a2,80(s5)
   0x3ff74d517c:        ld      s10,560(sp)
   0x3ff74d5180:        sw      s10,84(s5)
   0x3ff74d5184:        ld      s3,1160(sp)
   0x3ff74d5188:        sw      s3,72(s5)
   0x3ff74d518c:        ld      a2,1136(sp)

[felixonmars]

Hi. I am not an expert here, but did you happen to take a look at the orcjit port at https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17801 ?

IIRC mesa currently uses mcjit which is deprecated and doesn't support new architectures.

[llvmbot]

@llvm/issue-subscribers-backend-risc-v

[llvmbot]

@llvm/issue-subscribers-orcjit

[inclyc]

CC @alexfanqi

[alexfanqi]

I would also suggest trying out the mesa orcjit patch felix mentioned, but using my wip branch https://gitlab.freedesktop.org/alexfanqi/mesa/-/commits/wip, which fixes some mistakes in the original MR. It runs successfully with opengl games currently I tested, although it is missing disk cache capability and my cpp skills aren't really great. I am still working to polish it up and get merged. I also would really appreciate if people interested and familiar with cpp can offer me some help.

[andreas-schwab]

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17801

I cannot get it to compile.

../src/gallium/auxiliary/gallivm/lp_bld_init_orc.cpp: In function ‘LLVMOpaqueError* {anonymous}::module_transform(void*, LLVMModuleRef)’:
../src/gallium/auxiliary/gallivm/lp_bld_init_orc.cpp:137:33: error: expected primary-expression before ‘,’ token
  137 |    LLVMRunPasses(mod, passes, tm, opts);
      |                                 ^
../src/gallium/auxiliary/gallivm/lp_bld_init_orc.cpp:144:33: error: expected primary-expression before ‘,’ token
  144 |    LLVMRunPasses(mod, passes, tm, opts);
      |                                 ^
../src/gallium/auxiliary/gallivm/lp_bld_init_orc.cpp:241:27: error: ‘passmgr’ was not declared in this scope
  241 |    LLVMDisposePassManager(passmgr);
      |                           ^~~~~~~
../src/gallium/auxiliary/gallivm/lp_bld_init_orc.cpp: In static member function ‘static void* {anonymous}::LPJit::lookup_in_jd(const char*, LLVMOrcJITDylibRef)’:
../src/gallium/auxiliary/gallivm/lp_bld_init_orc.cpp:351:28: error: ‘class llvm::orc::ExecutorAddr’ has no member named ‘getAddress’
  351 |       return (void *)(func.getAddress());
      |                            ^~~~~~~~~~

[alexfanqi]

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17801

I cannot get it to compile.

Can you try out https://gitlab.freedesktop.org/alexfanqi/mesa/-/tree/wip ? please make sure you use llvm 14 patched with https://reviews.llvm.org/D120001 or llvm 15+.

I also feel this is not a llvm orcjit issue, but an issue of mesa llvmpipe that it should transition into orcjit from mcjit. Maybe should move discussion somewhere else? but I am not sure a better place.

[andreas-schwab]

../src/gallium/auxiliary/gallivm/lp_bld_init_orc.cpp: In function ‘LLVMOpaqueError* {anonymous}::module_transform(void*, LLVMModuleRef)’: ../src/gallium/auxiliary/gallivm/lp_bld_init_orc.cpp:130:30: error: ‘LPJit’ was not declared in this scope 130 | LLVMTargetMachineRef tm = LPJit->get_instance()->tm; | ^~~~~

[alexfanqi]

@andreas-schwab Can you try again with the wip branch? I have pushed fix and updated with latest upstream.

[jrtc27]

Please take this discussion thread elsewhere, it's about Mesa development not LLVM development