Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testJITServer_0 crash vmState=0x0005ffff #10215

Closed
pshipton opened this issue Jul 22, 2020 · 7 comments
Closed

testJITServer_0 crash vmState=0x0005ffff #10215

pshipton opened this issue Jul 22, 2020 · 7 comments
Labels
comp:jitserver Artifacts related to JIT-as-a-Service project segfault Issues that describe segfaults / JVM crashes
Projects
JIT as a Service

Comments

@pshipton
Copy link
Member

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/57
testJITServer_0

////////////////////////////////////////////////////////////////////////
//// JITServer is currently a technology preview. Its use is not yet supported
//// JIT: env var TR_Options is set to verbose={compileEnd|JITServer|heartbeat},vlog=testServerUnreachableForAWhile.client.jitverboselog.out
//// [IncludeExcludeTestAnnotationTransformer] [INFO] exclude file is /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdk-tests/TKG/../TestConfig/resources/excludes/latest_exclude_8.txt
//// ...
//// ... TestNG 6.14.2 by Cédric Beust (cedric@beust.com)
//// ...
//// 
//// [ZipTester] [INFO] -loopforever was specified; test will run continuously until the JVM is terminated.
//// #0: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x8a32c5) [0x7f517d75e2c5]
//// #1: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x8add10) [0x7f517d768d10]
//// #2: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x162a9e) [0x7f517d01da9e]
//// #3: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9prt29.so(+0x1b36a) [0x7f517e9b236a]
//// #4: /lib64/libpthread.so.0(+0xf630) [0x7f5180d13630]
//// #5: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x96f325) [0x7f517d82a325]
//// #6: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x96f429) [0x7f517d82a429]
//// #7: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x96f459) [0x7f517d82a459]
//// #8: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x308215) [0x7f517d1c3215]
//// #9: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x2c6091) [0x7f517d181091]
//// #10: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x2d4c4a) [0x7f517d18fc4a]
//// #11: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x175671) [0x7f517d030671]
//// #12: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x176431) [0x7f517d031431]
//// #13: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9prt29.so(+0x1bea3) [0x7f517e9b2ea3]
//// #14: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x1781f5) [0x7f517d0331f5]
//// #15: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x1787a8) [0x7f517d0337a8]
//// #16: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x1740db) [0x7f517d02f0db]
//// #17: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x1745a2) [0x7f517d02f5a2]
//// #18: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x17464a) [0x7f517d02f64a]
//// #19: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9prt29.so(+0x1bea3) [0x7f517e9b2ea3]
//// #20: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so(+0x174aa4) [0x7f517d02faa4]
//// #21: /home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9thr29.so(+0xe326) [0x7f517ee24326]
//// #22: /lib64/libpthread.so.0(+0x7ea5) [0x7f5180d0bea5]
//// #23: function clone+0x6d [0x7f51806188dd]
//// Unhandled exception
//// Type=Segmentation error vmState=0x0005ffff
//// J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
//// Handler1=00007F517F0C5CE0 Handler2=00007F517E9B2140 InaccessibleAddress=00007F51A74C53F2
//// RDI=00007F515EB702AA RSI=000000004820E0C1 RAX=00000000742F7473 RBX=00007F51A74C53F2
//// RCX=0000000000000000 RDX=0000000000000003 R8=0000000000000001 R9=0000000000000000
//// R10=000000000005FFFF R11=00007F51806A8710 R12=00007F515EB702AA R13=0000000000000000
//// R14=00007F515EE5FB50 R15=0000000000000000
//// RIP=00007F517D82A325 GS=0000 FS=0000 RSP=00007F515EE5F280
//// EFlags=0000000000010206 CS=0033 RBP=00007F515EB702AA ERR=0000000000000004
//// TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=00007F51A74C53F2
//// xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
//// xmm1 747269562f656c62 (f: 795176064.000000, d: 8.436550e+252)
//// xmm2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
//// xmm3 0000000001204800 (f: 18892800.000000, d: 9.334283e-317)
//// xmm4 1800800000001500 (f: 5376.000000, d: 4.520607e-193)
//// xmm5 0d01000000806800 (f: 8415232.000000, d: 4.862756e-246)
//// xmm6 0000000d04890000 (f: 76087296.000000, d: 2.762354e-313)
//// xmm7 00007fa9d7370ab8 (f: 3610708736.000000, d: 6.935073e-310)
//// xmm8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
//// xmm9 ffff37358d48005a (f: 2370306048.000000, d: -nan)
//// xmm10 ffffffffffffffff (f: 4294967296.000000, d: -nan)
//// xmm11 0000000049d70a38 (f: 1238829568.000000, d: 6.120632e-315)
//// xmm12 000000004689a022 (f: 1183424512.000000, d: 5.846894e-315)
//// xmm13 0000000047ac082f (f: 1202456576.000000, d: 5.940925e-315)
//// xmm14 0000000048650dc0 (f: 1214582272.000000, d: 6.000833e-315)
//// xmm15 0000000046b73e38 (f: 1186414080.000000, d: 5.861665e-315)
//// Module=/home/jenkins/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_OMR_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so
//// Module_base_address=00007F517CEBB000
//// 
//// Method_being_compiled=jit/test/tr/chtable/VirtualGuardTest.setUp()V
//// Target=2_90_20200721_669 (Linux 3.10.0-1127.13.1.el7.x86_64)
//// CPU=amd64 (4 logical CPUs) (0x1e8cf4000 RAM)
//// ----------- Stack Backtrace -----------
//// (0x00007F517D82A325 [libj9jit29.so+0x96f325])
//// (0x00007F517D82A429 [libj9jit29.so+0x96f429])
//// (0x00007F517D82A459 [libj9jit29.so+0x96f459])
//// (0x00007F517D1C3215 [libj9jit29.so+0x308215])
//// (0x00007F517D181091 [libj9jit29.so+0x2c6091])
//// (0x00007F517D18FC4A [libj9jit29.so+0x2d4c4a])
//// (0x00007F517D030671 [libj9jit29.so+0x175671])
//// (0x00007F517D031431 [libj9jit29.so+0x176431])
//// (0x00007F517E9B2EA3 [libj9prt29.so+0x1bea3])
//// (0x00007F517D0331F5 [libj9jit29.so+0x1781f5])
//// (0x00007F517D0337A8 [libj9jit29.so+0x1787a8])
//// (0x00007F517D02F0DB [libj9jit29.so+0x1740db])
//// (0x00007F517D02F5A2 [libj9jit29.so+0x1745a2])
//// (0x00007F517D02F64A [libj9jit29.so+0x17464a])
//// (0x00007F517E9B2EA3 [libj9prt29.so+0x1bea3])
//// (0x00007F517D02FAA4 [libj9jit29.so+0x174aa4])
//// (0x00007F517EE24326 [libj9thr29.so+0xe326])
//// (0x00007F5180D0BEA5 [libpthread.so.0+0x7ea5])
//// clone+0x6d (0x00007F51806188DD [libc.so.6+0xfe8dd])
//// ---------------------------------------
@pshipton pshipton added comp:jitserver Artifacts related to JIT-as-a-Service project segfault Issues that describe segfaults / JVM crashes labels Jul 22, 2020
@pshipton
Copy link
Member Author

@mpirvu

@pshipton
Copy link
Member Author

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_x86-64_linux_Nightly_testList_0/55

@dmitry-ten
Copy link
Contributor

I was able to reproduce the crash on my machine with this stacktrace:

#13 0x00007fa0e5b66246 in TR_J9VM::getBaseComponentClass(TR_OpaqueClassBlock*, int&) ()
   from /home/dmitry-ten/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so
#14 0x00007fa0e5cae57a in JITServerHelpers::packRemoteROMClassInfo[abi:cxx11](J9Class*, J9VMThread*, TR_Memory*, bool) ()
   from /home/dmitry-ten/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so
#15 0x00007fa0e5c6c69e in handleServerMessage(JITServer::ClientStream*, TR_J9VM*, JITServer::MessageType&) ()
   from /home/dmitry-ten/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so
#16 0x00007fa0e5c7adaa in remoteCompile(J9VMThread*, TR::Compilation*, TR_ResolvedMethod*, J9Method*, TR::IlGeneratorMethodDetails&, TR::CompilationInfoPerThreadBase*) ()
   from /home/dmitry-ten/src/openj9-openjdk-jdk8/build/linux-x86_64-normal-server-release/images/j2sdk-image/jre/lib/amd64/compressedrefs/libj9jit29.so

So it seems that ROM class caching is to blame. Out of recent commits, #9955 is most likely to be the problem, especially
since this crash only occurs when the server is unreachable for a while and then comes back on. The above PR added some client-side tracking of which ROM classes were cached by the server so it seems likely to be related.
I will investigate this issue further. @EmanElsaban fyi.

@dmitry-ten
Copy link
Contributor

Although with the above change seemingly disabled, I still get a crash. So it might not be the root cause.

@pshipton
Copy link
Member Author

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_x86-64_linux_cm_Nightly_testList_1/59
testJITServer_1

@dmitry-ten dmitry-ten mentioned this issue Aug 17, 2020
Closed
@dmitry-ten
Copy link
Contributor

This should be closed now, fixed by #10408.
The test passes now.

@dmitry-ten dmitry-ten mentioned this issue Aug 20, 2020
Closed
@mpirvu mpirvu added this to To do in JIT as a Service via automation Aug 21, 2020
@mpirvu
Copy link
Contributor

mpirvu commented Aug 21, 2020

Closing based on the comment above

@mpirvu mpirvu closed this as completed Aug 21, 2020
JIT as a Service automation moved this from To do to Done Aug 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
comp:jitserver Artifacts related to JIT-as-a-Service project segfault Issues that describe segfaults / JVM crashes
Projects
JIT as a Service
  
Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dmitry-ten @pshipton @mpirvu