Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sanity.system_x86-64_windows TestJlmRemoteMemoryNoAuth_0 crash vmState=0x0002000f #7156

Closed
pshipton opened this issue Sep 19, 2019 · 11 comments

Comments

@pshipton
Copy link
Member

https://ci.eclipse.org/openj9/job/Test_openjdk13_j9_sanity.system_x86-64_windows_Nightly/43
TestJlmRemoteMemoryNoAuth_0
core file in the artifacts https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk13_j9_sanity.system_x86-64_windows_Nightly/43/system_test_output.tar.gz

Unhandled exception
Type=Segmentation error vmState=0x0002000f
Windows_ExceptionCode=c0000005 J9Generic_Signal=00000004 ExceptionAddress=00007FF8D78F050D ContextFlags=0010005f
Handler1=00007FF8D847CF60 Handler2=00007FF8D838EF90 InaccessibleReadAddress=0000000000000000
RDI=000000507E91F601 RSI=0000000000000000 RAX=0000000000000000 RBX=000000507D3365E8
RCX=0000000099669966 RDX=0000000000000000 R8=0000000000698320 R9=0000000000691780
R10=000000507E91F600 R11=000000507E91F3E0 R12=0000000000000000 R13=000000505F725CB0
R14=000000507E91FA00 R15=0000000000000000
RIP=00007FF8D78F050D RSP=000000507E91F370 RBP=000000507E91F3E0 GS=002B
FS=0053 ES=002B DS=002B
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM2 3fe3180000000000 (f: 0.000000, d: 5.966797e-01)
XMM3 3fd0055c8dd20ccc (f: 2379353344.000000, d: 2.503272e-01)
XMM4 3d9f59e80eb72315 (f: 246883088.000000, d: 7.128437e-12)
XMM5 bf5ac633f80644ee (f: 4161160448.000000, d: -1.634169e-03)
XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=C:\Users\jenkins\workspace\Test_openjdk13_j9_sanity.system_x86-64_windows_Nightly\openjdkbinary\j2sdk-image\bin\compressedrefs\j9gc29.dll
Module_base_address=00007FF8D7780000 Offset_in_DLL=000000000017050d
Target=2_90_20190918_66 (Windows Server 2012 R2 6.3 build 9600)
CPU=amd64 (8 logical CPUs) (0x1ffb9c000 RAM)
----------- Stack Backtrace -----------
J9VMDllMain+0x16f4fd (0x00007FF8D78F050D [j9gc29+0x17050d])
J9VMDllMain+0x1598a (0x00007FF8D779699A [j9gc29+0x1699a])
J9_GetInterface+0x28fc9 (0x00007FF8D84B31D9 [j9vm29+0xb31d9])
J9_GetInterface+0x28591 (0x00007FF8D84B27A1 [j9vm29+0xb27a1])
J9_GetInterface+0x28f4a (0x00007FF8D84B315A [j9vm29+0xb315a])
J9_GetInterface+0x2811f (0x00007FF8D84B232F [j9vm29+0xb232f])
J9VMDllMain+0x15945 (0x00007FF8D7796955 [j9gc29+0x16955])
J9VMDllMain+0x126be (0x00007FF8D77936CE [j9gc29+0x136ce])
J9VMDllMain+0x129c3 (0x00007FF8D77939D3 [j9gc29+0x139d3])
J9VMDllMain+0x1184d (0x00007FF8D779285D [j9gc29+0x1285d])
J9VMDllMain+0x1656c3 (0x00007FF8D78E66D3 [j9gc29+0x1666d3])
J9VMDllMain+0x10e553 (0x00007FF8D788F563 [j9gc29+0x10f563])
J9VMDllMain+0x10f2e5 (0x00007FF8D78902F5 [j9gc29+0x1102f5])
j9port_isCompatible+0x1e3ff (0x00007FF8D839110F [j9prt29+0x2110f])
J9VMDllMain+0x10f438 (0x00007FF8D7890448 [j9gc29+0x110448])
omrthread_get_category+0xa42 (0x00007FF8D83E4452 [J9THR29+0x4452])
_o_strcat_s+0x5e (0x00007FF8D870C1AE [ucrtbase+0x1c1ae])
BaseThreadInitThunk+0x22 (0x00007FF8E7ED13D2 [KERNEL32+0x13d2])
RtlUserThreadStart+0x34 (0x00007FF8EA8554F4 [ntdll+0x154f4])
06:01:06  For example, to rebuild the failed tests in <Jenkins URL>=https://ci.adoptopenjdk.net/job/Grinder, use the following links:
06:01:06  https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=13&JDK_IMPL=openj9&BUILD_LIST=system&JenkinsFile=openjdk_x86-64_windows&TARGET=TestJlmRemoteMemoryNoAuth_0
@pshipton
Copy link
Member Author

@dmitripivkine

@dmitripivkine
Copy link
Contributor

Bad o-slot pointer in Bytecode frame:

<5dd500> Bytecode frame: bp = 0x0000000000698318, sp = 0x0000000000698308, pc = 0x000000506968C78C, cp = 0x0000000000150C70, arg0EA = 0x0000000000698338, flags = 0x0000000000000000
<5dd500> 	Method: java/lang/invoke/ArgumentConversionHandle.<init>(Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;B)V !j9method 0x00000000001506B0
<5dd500> 	Bytecode index = 8
<5dd500> 	Using local mapper
<5dd500> 	Locals starting at 0x0000000000698338 for 0x0000000000000004 slots
<5dd500> 		O-Slot: a0[0x0000000000698338] = 0x00000000FFAB1018
<5dd500> 		O-Slot: a1[0x0000000000698330] = 0x0000000000698320
<5dd500> 		O-Slot: a2[0x0000000000698328] = 0x00000000F03B6840
<5dd500> 		I-Slot: a3[0x0000000000698320] = 0x000000000000000D

O-Slot: a1[0x0000000000698330] = 0x0000000000698320 points to the same frame stack but there is no stack allocated object:

0x00698320 :  000000000000000d 00000000f03b6840 [ ........@h;..... ]
0x00698330 :  0000000000698320 00000000ffab1018 [  .i............. ]
0x00698340 :  0000000000058190 00000050694e5285 [ .........RNiP... ]

@DanHeidinga
Copy link
Member

That O-Slot is an argument to the method and should be a MethodHandle object. Do you have the caller's frame as well?

@dmitripivkine
Copy link
Contributor

dmitripivkine commented Sep 20, 2019

<5dd500> Bytecode frame: bp = 0x0000000000698318, sp = 0x0000000000698308, pc = 0x000000506968C78C, cp = 0x0000000000150C70, arg0EA = 0x0000000000698338, flags = 0x0000000000000000
<5dd500> 	Method: java/lang/invoke/ArgumentConversionHandle.<init>(Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;B)V !j9method 0x00000000001506B0
<5dd500> 	Bytecode index = 8
<5dd500> 	Using local mapper
<5dd500> 	Locals starting at 0x0000000000698338 for 0x0000000000000004 slots
<5dd500> 		O-Slot: a0[0x0000000000698338] = 0x00000000FFAB1018
<5dd500> 		O-Slot: a1[0x0000000000698330] = 0x0000000000698320
<5dd500> 		O-Slot: a2[0x0000000000698328] = 0x00000000F03B6840
<5dd500> 		I-Slot: a3[0x0000000000698320] = 0x000000000000000D
<5dd500> Bytecode frame: bp = 0x0000000000698350, sp = 0x0000000000698340, pc = 0x000000506968C37D, cp = 0x0000000000151380, arg0EA = 0x0000000000698368, flags = 0x0000000000000000
<5dd500> 	Method: java/lang/invoke/AsTypeHandle.<init>(Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)V !j9method 0x00000000001514D8
<5dd500> 	Bytecode index = 5
<5dd500> 	Using local mapper
<5dd500> 	Locals starting at 0x0000000000698368 for 0x0000000000000003 slots
<5dd500> 		I-Slot: a0[0x0000000000698368] = 0x00000000FFAB1018
<5dd500> 		I-Slot: a1[0x0000000000698360] = 0x0000000000698320
<5dd500> 		I-Slot: a2[0x0000000000698358] = 0x00000000F03B6840
<5dd500> Bytecode frame: bp = 0x0000000000698388, sp = 0x0000000000698370, pc = 0x00000050694E5285, cp = 0x00000000000564B0, arg0EA = 0x00000000006983C0, flags = 0x0000000000000000
<5dd500> 	Method: java/lang/invoke/MethodHandle.asType(Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/MethodHandle; !j9method 0x0000000000058190
<5dd500> 	Bytecode index = 121
<5dd500> 	Using local mapper
<5dd500> 	Locals starting at 0x00000000006983C0 for 0x0000000000000007 slots
<5dd500> 		O-Slot: a0[0x00000000006983C0] = 0x00000000F0A1AF98
<5dd500> 		I-Slot: a1[0x00000000006983B8] = 0x00000000F03B6840
<5dd500> 		I-Slot: t2[0x00000000006983B0] = 0x0000000000000000
<5dd500> 		I-Slot: t3[0x00000000006983A8] = 0x0000000000698320
<5dd500> 		I-Slot: t4[0x00000000006983A0] = 0x0000000000000000
<5dd500> 		I-Slot: t5[0x0000000000698398] = 0x0000000000000000
<5dd500> 		I-Slot: t6[0x0000000000698390] = 0x0000000000000000
<5dd500> 	Pending stack starting at 0x0000000000698370 for UDATA(0x0000000000000001) slots
<5dd500> 		O-Slot: p0[0x0000000000698370] = 0x00000000FFAB1018
<5dd500> J2I frame: bp = 0x0000000000698448, sp = 0x00000000006983C8, pc = 0x000000507D5885A8, cp = 0x0000000000556320, arg0EA = 0x0000000000698460, flags = 0x0000000010000000
<5dd500> 	Method: net/adoptopenjdk/test/lang/invoke/AsTypeTest.cast(Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/MethodHandle; !j9method 0x0000000000557EF0
<5dd500> 	Bytecode index = 28
<5dd500> 	Using local mapper
<5dd500> 	Locals starting at 0x0000000000698460 for 0x0000000000000003 slots
<5dd500> 		I-Slot: a0[0x0000000000698460] = 0x00000000FFE95F08
<5dd500> 		I-Slot: a1[0x0000000000698458] = 0x00000000F0A1AF98
<5dd500> 		I-Slot: a2[0x0000000000698450] = 0x00000000F03B6840
<5dd500> 	JIT-J2I-RegisterMap[0x00000000006983F0] = UDATA(0x00000000FFE95F08) (jit_rbx)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698428] = UDATA(0x00000000F03B6840) (jit_r9)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698420] = UDATA(0x0000000000698320) (jit_r10)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698418] = UDATA(0x0000000000000000) (jit_r11)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698410] = UDATA(0x00000000FFDD0B40) (jit_r12)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698408] = UDATA(0x0000000000698320) (jit_r13)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698400] = UDATA(0x0000000000000000) (jit_r14)
<5dd500> 	JIT-J2I-RegisterMap[0x00000000006983F8] = UDATA(0x00000000F06EF338) (jit_r15)
<5dd500> JIT frame: bp = 0x00000000006984C8, pc = 0x0000000055474965, unwindSP = 0x0000000000698450, cp = 0x0000000000556320, arg0EA = 0x00000000006984D0, jitInfo = 0x0000005002A30090
<5dd500> 	Method: net/adoptopenjdk/test/lang/invoke/AsTypeTest.testReference()V !j9method 0x0000000000558310
<5dd500> 	Bytecode index = 988, inlineDepth = 0, PC offset = 0x0000000000000B05
<5dd500> 	stackMap=0x0000005002A30C66, slots=I16(0x0001) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0001), localBaseOffset=I16(0xFFC8)
<5dd500> 	Described JIT args starting at 0x00000000006984D0 for U16(0x0001) slots
<5dd500> 		O-Slot: : a0[0x00000000006984D0] = 0x00000000FFE95F08
<5dd500> 	Described JIT temps starting at 0x0000000000698490 for IDATA(0x0000000000000007) slots
<5dd500> 		I-Slot: : t6[0x0000000000698490] = 0x00000000F052CCA0
<5dd500> 		O-Slot: : t5[0x0000000000698498] = 0x00000000F0A1AF98
<5dd500> 		O-Slot: : t4[0x00000000006984A0] = 0x00000000F03B6840
<5dd500> 		I-Slot: : t3[0x00000000006984A8] = 0x00000000FFDCC858
<5dd500> 		O-Slot: : t2[0x00000000006984B0] = 0x00000000F0A1AF98
<5dd500> 		I-Slot: : t1[0x00000000006984B8] = 0x00000000F0385278
<5dd500> 		I-Slot: : t0[0x00000000006984C0] = 0x00000000F02CE7D0
<5dd500> 	JIT-RegisterMap = UDATA(0x0000000000000202)
<5dd500> 		JIT-RegisterMap-O-Slot[0x00000000006983F0] = 0x00000000FFE95F08 (jit_rbx)
<5dd500> 		JIT-RegisterMap-O-Slot[0x0000000000698428] = 0x00000000F03B6840 (jit_r9)
<5dd500> 		JIT-RegisterMap-I-Slot[0x0000000000698420] = UDATA(0x0000000000698320) (jit_r10)
<5dd500> 		JIT-RegisterMap-I-Slot[0x0000000000698418] = UDATA(0x0000000000000000) (jit_r11)
<5dd500> 		JIT-RegisterMap-I-Slot[0x0000000000698410] = UDATA(0x00000000FFDD0B40) (jit_r12)
<5dd500> 		JIT-RegisterMap-I-Slot[0x0000000000698408] = UDATA(0x0000000000698320) (jit_r13)
<5dd500> 		JIT-RegisterMap-I-Slot[0x0000000000698400] = UDATA(0x0000000000000000) (jit_r14)
<5dd500> 		JIT-RegisterMap-I-Slot[0x00000000006983F8] = UDATA(0x00000000F06EF338) (jit_r15)
<5dd500> 	JIT-Frame-RegisterMap[0x0000000000698478] = UDATA(0x00000000005E3000) (jit_rbx)
<5dd500> 	JIT-Frame-RegisterMap[0x0000000000698480] = UDATA(0x00000000F02D4778) (jit_r9)
<5dd500> 	JIT-Frame-RegisterMap[0x0000000000698420] = UDATA(0x0000000000698320) (jit_r10)
<5dd500> 	JIT-Frame-RegisterMap[0x0000000000698418] = UDATA(0x0000000000000000) (jit_r11)
<5dd500> 	JIT-Frame-RegisterMap[0x0000000000698410] = UDATA(0x00000000FFDD0B40) (jit_r12)
<5dd500> 	JIT-Frame-RegisterMap[0x0000000000698408] = UDATA(0x0000000000698320) (jit_r13)
<5dd500> 	JIT-Frame-RegisterMap[0x0000000000698400] = UDATA(0x0000000000000000) (jit_r14)
<5dd500> 	JIT-Frame-RegisterMap[0x00000000006983F8] = UDATA(0x00000000F06EF338) (jit_r15)
<5dd500> I2J values: PC = 0x000000507D25621C, A0 = 0x0000000000698570, walkSP = 0x00000000006984D8, literals = 0x00000000005E32D8, JIT PC = 0x0000000069142550, pcAddress = 0x00000000006983E0, decomp = 0x0000000000000000
<5dd500> J2I frame: bp = 0x0000000000698558, sp = 0x00000000006984D8, pc = 0x000000507D25621C, cp = 0x00000000005E31B0, arg0EA = 0x0000000000698570, flags = 0x0000000010000000
<5dd500> 	Method: jdk/internal/reflect/GeneratedMethodAccessor30.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x00000000005E32D8
<5dd500> 	Bytecode index = 36
<5dd500> 	Using local mapper
<5dd500> 	Locals starting at 0x0000000000698570 for 0x0000000000000003 slots
<5dd500> 		I-Slot: a0[0x0000000000698570] = 0x00000000F0B7BC10
<5dd500> 		I-Slot: a1[0x0000000000698568] = 0x00000000FFDC3908
<5dd500> 		I-Slot: a2[0x0000000000698560] = 0x00000000F030AF08
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698500] = UDATA(0x00000000005E3000) (jit_rbx)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698538] = UDATA(0x00000000F02D4778) (jit_r9)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698530] = UDATA(0x0000000000000001) (jit_r10)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698528] = UDATA(0x00000050007FF790) (jit_r11)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698520] = UDATA(0x00000000FFDB2CA8) (jit_r12)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698518] = UDATA(0x00000000FFDC3918) (jit_r13)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698510] = UDATA(0x000000000000000A) (jit_r14)
<5dd500> 	JIT-J2I-RegisterMap[0x0000000000698508] = UDATA(0x00000000F03AB470) (jit_r15)
<5dd500> JIT inline frame: bp = 0x00000000006985E8, pc = 0x000000005593F3FA, unwindSP = 0x0000000000698560, cp = 0x0000000000149800, arg0EA = 0x0000000000000000, jitInfo = 0x000000507CC7FDE8
<5dd500> 	Method: jdk/internal/reflect/DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x0000000000149AE0

@dmitripivkine
Copy link
Contributor

you can find system core in my team folder under 7156/

@DanHeidinga
Copy link
Member

It looks like the bad value 698320 can traced back to this frame:

<5dd500> JIT frame: bp = 0x00000000006984C8, pc = 0x0000000055474965, unwindSP = 0x0000000000698450, cp = 0x0000000000556320, arg0EA = 0x00000000006984D0, jitInfo = 0x0000005002A30090
<5dd500> 	Method: net/adoptopenjdk/test/lang/invoke/AsTypeTest.testReference()V !j9method 0x0000000000558310

@andrewcraik Can you have someone take a look at this?

@andrewcraik
Copy link
Contributor

@liqunl this looks like it could be a handle problem - could you have a look?

@pshipton
Copy link
Member Author

Given this has only happened once in the nightly builds, seems unlikely to be fixed in the 0.17 release, except maybe under a different issue. Moving this one to the next release. Note there was another issue for the same vmstate, but it was closed for lack of reproduction. #5764
@liqunl if you want the core for future reference, please archive it before it expires on artifactory.

@andrewcraik
Copy link
Contributor

if there have been no further reproductions and no further progress has been made by @liqunl I think we should either move this out or close it. Any update @liqunl ?

@liqunl
Copy link
Contributor

liqunl commented Nov 27, 2019

@andrewcraik I haven't been able to reproduce the failure, no progress has been made so far.

@pshipton
Copy link
Member Author

Closing until the problem is seen again.

mpirvu added a commit to mpirvu/openj9 that referenced this issue Oct 27, 2023
By default, OpenJ9 uses an "implicit" shared class cache (SCC)
that is used to cache "bootstrap" classes and methods.
Currently, methods whose class is not present in the SCC
receive a high initial invocation count (3000/3000 for
loopless/loopy method) in order to improve steady state
throughput. However, this could affect start-up time of
applications because methods are interpreted for too long
before being JIT compiled.
The presence of `-Xshareclasses` on the command line is a sign
that the user cares about start-up time and OpenJ9 automatically
lowers the invocations counts to 1000/250. However, with the
implicit SCC (no -Xshareclasses command line option), application
classes are not cached and methods belonging to those classes
will receive a large initial invocation count (3000/3000). Since
application classes are likely important, we would like to use
lower counts even when the implict SCC is used.

This commit lowers the initial invocation counts for methods
whose class is not in SCC under the following conditions:
- SCC is used. Rationale: -Xshareclasses:none may be a sign that
user does not care about start-up that much.
- -Xtune:throughput is not used. Rationale: this option is a
clear signal that the user cares more about throughput than startup.
- The user has not provided a specific count on the command line.
- The JVM is in start-up mode. Rationale: this feature only seeks
to improve start-up time.
- The JVM runs on 4 or more vCPUs. Rationale: lower counts will
generate more compilations which will compete with application

Depends on eclipse/omr/eclipse-openj9#7156

Signed-off-by: Marius <mpirvu@ca.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants