JDK21 compiler debug build fails due to assertion omr/compiler/x/codegen/OMRRegisterDependency.hpp:208: newCursor <= _numPostConditions

[cjjdespres]

As in #19836, I tried building a JDK with the JIT in debug mode on x86_64 linux with the following flags:

export cflags="-Og -ggdb3 -fno-inline -DDEBUG"
export EXTRA_CMAKE_ARGS="-DJ9JIT_EXTRA_CFLAGS=\"$cflags\" -DJ9JIT_EXTRA_CXXFLAGS=\"$cflags\""

Note that I had to apply the openj9 change I suggested in #19836 (comment) in order to get past the assert in #19836. With that done, the full JDK now fails to build with the following assert:

Optimizing the exploded image
Assertion failed at /root/hostdir/foo/openj9-openjdk-jdk21/omr/compiler/x/codegen/OMRRegisterDependency.hpp:208: newCursor <= _numPostConditions
VMState: 0x0005ff04
	Too many dependencies
compiling jdk/internal/org/objectweb/asm/MethodWriter.<init>(Ljdk/internal/org/objectweb/asm/SymbolTable;ILjava/lang/String;Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;I)V at level: cold

JIT: crashed while compiling jdk/internal/org/objectweb/asm/MethodWriter.<init>(Ljdk/internal/org/objectweb/asm/SymbolTable;ILjava/lang/String;Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;I)V (recoverable 0)

Unhandled exception
Type=Unhandled trap vmState=0x0005ff04
J9Generic_Signal_Number=00000108 Signal_Number=00000005 Error_Value=00000000 Signal_Code=fffffffa
Handler1=00007FC3B5C44400 Handler2=00007FC3B5EC8B40
RDI=0000000000037E3B RSI=0000000000037E42 RAX=0000000000000000 RBX=00007FC3AE2DC640
RCX=00007FC3B60F59FC RDX=0000000000000005 R8=00007FC3B51A9880 R9=000000007FFFFFFF
R10=0000000000000001 R11=0000000000000246 R12=0000000000000005 R13=0000000000000016
R14=00007FC3B51A97EE R15=00007FC3AD33EF20
RIP=00007FC3B60F59FC GS=0000 FS=0000 RSP=00007FC3AE2D54F0
EFlags=0000000000000246 CS=0033 RBP=0000000000037E42 ERR=0000000000000000
TRAPNO=0000000000000000 OLDMASK=0000000000000000 CR2=0000000000000000
xmm0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm1=0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm2=000000000000ff00 (f: 65280.000000, d: 3.225261e-319)
xmm3=00007fc3ae2d4f70 (f: 2922204928.000000, d: 6.940556e-310)
xmm4=00007fc3a8009e8d (f: 2818612992.000000, d: 6.940551e-310)
xmm5=6775626564776f6c (f: 1685548928.000000, d: 2.381955e+190)
xmm6=6975622f31326b64 (f: 825387904.000000, d: 1.023002e+200)
xmm7=74736f682f746f6f (f: 796159872.000000, d: 8.905638e+252)
xmm8=5f2f706d742f000a (f: 1949237248.000000, d: 3.216021e+150)
xmm9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/lib/x86_64-linux-gnu/libc.so.6
Module_base_address=00007FC3B605F000 Symbol=pthread_kill
Symbol_address=00007FC3B60F58D0

Method_being_compiled=jdk/internal/org/objectweb/asm/MethodWriter.<init>(Ljdk/internal/org/objectweb/asm/SymbolTable;ILjava/lang/String;Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;I)V
Target=2_90_20240709_000000 (Linux 5.14.0-427.22.1.el9_4.x86_64)
CPU=amd64 (8 logical CPUs) (0x3d781b000 RAM)
----------- Stack Backtrace -----------

pthread_kill+0x12c (0x00007FC3B60F59FC [libc.so.6+0x969fc])
raise+0x16 (0x00007FC3B60A1476 [libc.so.6+0x42476])
_ZN2TR4trapEv+0x4d (0x00007FC3B4DC5C95 [libj9jit29.so+0x5c5c95])
_ZN2TR15fatal_assertionEPKciS1_S1_z+0x0 (0x00007FC3B4DC5EFB [libj9jit29.so+0x5c5efb])
_ZN2TR9assertionEPKciS1_S1_z+0xd0 (0x00007FC3B4DC6118 [libj9jit29.so+0x5c6118])
_ZN3OMR3X8628RegisterDependencyConditions18unionPostConditionEPN2TR8RegisterENS_12RealRegister6RegNumEPNS2_13CodeGeneratorEhb+0x79 (0x00007FC3B4D0DB37 [libj9jit29.so+0x50db37])
_ZN3OMR3X8628RegisterDependencyConditions23unionNoRegPostConditionEPN2TR8RegisterEPNS2_13CodeGeneratorE+0x21 (0x00007FC3B50A239F [libj9jit29.so+0x8a239f])
_ZN25TR_ScratchRegisterManager35addScratchRegistersToDependencyListEPN2TR28RegisterDependencyConditionsE+0x39 (0x00007FC3B4D38A81 [libj9jit29.so+0x538a81])
_ZN2J93X8613TreeEvaluator14VMnewEvaluatorEPN2TR4NodeEPNS2_13CodeGeneratorE+0xfc4 (0x00007FC3B4D06D78 [libj9jit29.so+0x506d78])
_ZN2J93X8613TreeEvaluator12newEvaluatorEPN2TR4NodeEPNS2_13CodeGeneratorE+0x77 (0x00007FC3B4D0708D [libj9jit29.so+0x50708d])
_ZN3OMR13CodeGenerator8evaluateEPN2TR4NodeE+0x129 (0x00007FC3B4D3C1A5 [libj9jit29.so+0x53c1a5])
_ZN3OMR3X8613TreeEvaluator16treetopEvaluatorEPN2TR4NodeEPNS2_13CodeGeneratorE+0x1e (0x00007FC3B506485A [libj9jit29.so+0x86485a])
_ZN3OMR13CodeGenerator8evaluateEPN2TR4NodeE+0x129 (0x00007FC3B4D3C1A5 [libj9jit29.so+0x53c1a5])
_ZN2J913CodeGenerator22doInstructionSelectionEv+0x218c (0x00007FC3B4903B66 [libj9jit29.so+0x103b66])
_ZN3OMR12CodeGenPhase32performInstructionSelectionPhaseEPN2TR13CodeGeneratorEPNS1_12CodeGenPhaseE+0x9e (0x00007FC3B4D47E08 [libj9jit29.so+0x547e08])
_ZN3OMR12CodeGenPhase10performAllEv+0xc0 (0x00007FC3B4D48426 [libj9jit29.so+0x548426])
_ZN3OMR13CodeGenerator12generateCodeEv+0x8e (0x00007FC3B4D44A7A [libj9jit29.so+0x544a7a])
_ZN3OMR11Compilation7compileEv+0xa70 (0x00007FC3B4D65F1C [libj9jit29.so+0x565f1c])
_ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0xae7 (0x00007FC3B493F707 [libj9jit29.so+0x13f707])
_ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0xad6 (0x00007FC3B4940992 [libj9jit29.so+0x140992])
omrsig_protect+0x241 (0x00007FC3B5EC97F1 [libj9prt29.so+0x2a7f1])
_ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x614 (0x00007FC3B493DB3E [libj9jit29.so+0x13db3e])
_ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x2e1 (0x00007FC3B493E1D1 [libj9jit29.so+0x13e1d1])
_ZN2TR24CompilationInfoPerThread14processEntriesEv+0x159 (0x00007FC3B493C70B [libj9jit29.so+0x13c70b])
_ZN2TR24CompilationInfoPerThread3runEv+0x2d (0x00007FC3B493CE6D [libj9jit29.so+0x13ce6d])
_Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x97 (0x00007FC3B493CF67 [libj9jit29.so+0x13cf67])
omrsig_protect+0x241 (0x00007FC3B5EC97F1 [libj9prt29.so+0x2a7f1])
_Z21compilationThreadProcPv+0x1c0 (0x00007FC3B493D370 [libj9jit29.so+0x13d370])
thread_wrapper+0x187 (0x00007FC3B5F42C07 [libj9thr29.so+0xbc07])
 (0x00007FC3B60F3AC3 [libc.so.6+0x94ac3])
 (0x00007FC3B6185850 [libc.so.6+0x126850])

As in #19836, there were a lot of "Unhandled exception/Stack Backtrace" blocks interleaved in the build log, but I think what I have above is correct.

[cjjdespres]

I thought I'd investigate this a little, and I noticed that PR #19514 changes J9TreeEvaluator.cpp a bit.

1. If I try to build a debug JDK (JDK21, on xlinux) with relatively recent commits (d2b02cf and eclipse/omr@330a128) and with the change in Update _unionPropertyA assert #19856 cherry-picked, the build fails with the assert I reported in this issue.
2. If I then revert 204fc71...b767300, I still get the assert.
3. If I then revert 204fc71 itself, the assert disappears and the debug build succeeds in building.

So, I think it might be 204fc71 that's the issue. I don't know much about the dependency tracking that's done here, but I wonder if @0xdaryl (or someone else) might be able to see what's going wrong.

(EDIT: That commit 204fc71 was responsible for removing two ¬ characters in J9TreeEvaluator.cpp introduced in an earlier commit in that PR, so I did also have to delete them manually for the build to succeed in 3.)