Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JDK15 Windows : ClassLoadingTest_special Segmentation error vmState=0x00020002 #10249

Closed
JasonFengJ9 opened this issue Jul 24, 2020 · 17 comments · Fixed by #10300
Closed

JDK15 Windows : ClassLoadingTest_special Segmentation error vmState=0x00020002 #10249

JasonFengJ9 opened this issue Jul 24, 2020 · 17 comments · Fixed by #10300
Labels
comp:gc comp:vm jdk15 jdk16 test failure
Milestone
Release 0.22 (Jav...

Comments

@JasonFengJ9
Copy link
Member

Failure link

From an internal build Test_openjdk15_j9_special.system_x86-64_windows_Nightly_otherLoadTest/3:

01:04:53  openjdk version "15-internal" 2020-09-15
01:04:53  OpenJDK Runtime Environment (build 15-internal+0-adhoc.jenkins.BuildJDK15x86-64windowsNightly)
01:04:53  Eclipse OpenJ9 VM (build master-f60babb2cfa, JRE 15 Windows Server 2016 amd64-64-Bit Compressed References 20200723_24 (JIT enabled, AOT enabled)
01:04:53  OpenJ9   - f60babb2cfa
01:04:53  OMR      - 025d6b2b62b
01:04:53  JCL      - 3604912533c based on jdk-15+32)

Optional info

Failure output (captured from console output)

CLT stderr Type=Segmentation error vmState=0x00020002
CLT stderr Windows_ExceptionCode=c0000005 J9Generic_Signal=00000004 ExceptionAddress=00007FFC5AFF5C60 ContextFlags=0010005f
CLT stderr Handler1=00007FFC5C1605F0 Handler2=00007FFC625EAC30 InaccessibleReadAddress=00007FF7E502A920
CLT stderr RDI=000001F76386ABC8 RSI=000000000001FFFF RAX=00007FF7E4CB0000 RBX=000000070DEA48D0
CLT stderr RCX=0000000000000009 RDX=000001F744C9C8A0 R8=000000000006F524 R9=00007FF7E502A920
CLT stderr R10=0000000004000000 R11=0000004BAF2FF600 R12=0000000000000001 R13=000000000000003C
CLT stderr R14=0000000000000000 R15=00000000001FFFF1
CLT stderr RIP=00007FFC5AFF5C60 RSP=0000004BAF2FF470 RBP=000001F744CA3770 GS=002B
CLT stderr FS=0053 ES=002B DS=002B
CLT stderr XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
CLT stderr Module=C:\Users\jenkins\workspace\Test_openjdk15_j9_special.system_x86-64_windows_Nightly_otherLoadTest\openjdkbinary\j2sdk-image\bin\compressedrefs\j9gc29.dll
CLT stderr Module_base_address=00007FFC5AF30000 Offset_in_DLL=00000000000c5c60
CLT stderr Target=2_90_20200723_24 (Windows Server 2016 10.0 build 10240)
CLT stderr CPU=amd64 (8 logical CPUs) (0x3fff7c000 RAM)
CLT stderr ----------- Stack Backtrace -----------
CLT stderr J9VMDllMain+0xc4c50 (0x00007FFC5AFF5C60 [j9gc29+0xc5c60])
CLT stderr J9VMDllMain+0xc4d70 (0x00007FFC5AFF5D80 [j9gc29+0xc5d80])
CLT stderr J9VMDllMain+0xc452b (0x00007FFC5AFF553B [j9gc29+0xc553b])
CLT stderr J9VMDllMain+0xc8ee4 (0x00007FFC5AFF9EF4 [j9gc29+0xc9ef4])
CLT stderr J9VMDllMain+0xc1748 (0x00007FFC5AFF2758 [j9gc29+0xc2758])
CLT stderr J9VMDllMain+0x10e513 (0x00007FFC5B03F523 [j9gc29+0x10f523])
CLT stderr J9VMDllMain+0x10f2a5 (0x00007FFC5B0402B5 [j9gc29+0x1102b5])
CLT stderr j9port_isCompatible+0x1a09f (0x00007FFC625ECDAF [j9prt29+0x1cdaf])
CLT stderr J9VMDllMain+0x10f3f8 (0x00007FFC5B040408 [j9gc29+0x110408])
CLT stderr omrthread_get_category+0xa42 (0x00007FFC64844452 [J9THR29+0x4452])
CLT stderr _o__realloc_base+0x60 (0x00007FFC6AACFB80 [ucrtbase+0x1fb80])
CLT stderr BaseThreadInitThunk+0x14 (0x00007FFC6DB784D4 [KERNEL32+0x84d4])
CLT stderr RtlUserThreadStart+0x21 (0x00007FFC6E51E871 [ntdll+0x6e871])

ClassLoadingTest_special_14_FAILED

For example, to rebuild the failed tests in =https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder, use the following links:
01:35:46 https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/parambuild/?JDK_VERSION=15&JDK_IMPL=openj9&BUILD_LIST=system/otherLoadTest&PLATFORM=x86-64_windows&TARGET=ClassLoadingTest_special_14
@JasonFengJ9
Copy link
Member Author

Another similar failure with different vmState at Test_openjdk15_j9_special.system_x86-64_windows_Nightly_lambdaLoadTest/3

LT  stderr Type=Segmentation error vmState=0x0002000f
LT  stderr Windows_ExceptionCode=c0000005 J9Generic_Signal=00000004 ExceptionAddress=00007FF99F3967FD ContextFlags=0010005f
LT  stderr Handler1=00007FF9A29F05F0 Handler2=00007FF9A7B4AC30 InaccessibleReadAddress=0000000000000000
LT  stderr RDI=00000007211C7858 RSI=000002213F3E58E0 RAX=0000000099669966 RBX=000002213FCB65B0
LT  stderr RCX=0000000000000000 RDX=00000221401DF060 R8=000002215F0543E0 R9=000000000000022F
LT  stderr R10=0000000745FBDC30 R11=00000000005F1800 R12=00007FF99F542BB0 R13=00007FF99F55BF00
LT  stderr R14=0000000745FBDC30 R15=00007FF99F4F3B10
LT  stderr RIP=00007FF99F3967FD RSP=0000002D8D5FF380 RBP=000002215E912B18 GS=002B
LT  stderr FS=0053 ES=002B DS=002B
LT  stderr XMM0 42d96f248beb6200 (f: 2347459072.000000, d: 1.118606e+14)
LT  stderr XMM1 4040000000000000 (f: 0.000000, d: 3.200000e+01)
LT  stderr XMM2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr Module=C:\Users\jenkins\workspace\Test_openjdk15_j9_special.system_x86-64_windows_Nightly_lambdaLoadTest\openjdkbinary\j2sdk-image\bin\compressedrefs\j9gc29.dll
LT  stderr Module_base_address=00007FF99F300000 Offset_in_DLL=00000000000967fd
LT  stderr Target=2_90_20200723_24 (Windows Server 2016 10.0 build 10240)
LT  stderr CPU=amd64 (8 logical CPUs) (0x3fff7c000 RAM)
LT  stderr ----------- Stack Backtrace -----------
LT  stderr J9VMDllMain+0x957ed (0x00007FF99F3967FD [j9gc29+0x967fd])
LT  stderr J9VMDllMain+0xa52be (0x00007FF99F3A62CE [j9gc29+0xa62ce])
LT  stderr J9VMDllMain+0xa282c (0x00007FF99F3A383C [j9gc29+0xa383c])
LT  stderr J9VMDllMain+0xaafca (0x00007FF99F3ABFDA [j9gc29+0xabfda])
LT  stderr J9VMDllMain+0x10e513 (0x00007FF99F40F523 [j9gc29+0x10f523])
LT  stderr J9VMDllMain+0x10f2a5 (0x00007FF99F4102B5 [j9gc29+0x1102b5])
LT  stderr j9port_isCompatible+0x1a09f (0x00007FF9A7B4CDAF [j9prt29+0x1cdaf])
LT  stderr J9VMDllMain+0x10f3f8 (0x00007FF99F410408 [j9gc29+0x110408])
LT  stderr omrthread_get_category+0xa42 (0x00007FF9A7CA4452 [J9THR29+0x4452])
LT  stderr _o__realloc_base+0x60 (0x00007FF9B756FB80 [ucrtbase+0x1fb80])
LT  stderr BaseThreadInitThunk+0x14 (0x00007FF9B77784D4 [KERNEL32+0x84d4])
LT  stderr RtlUserThreadStart+0x21 (0x00007FF9BA12E871 [ntdll+0x6e871])

@pshipton pshipton mentioned this issue Jul 24, 2020
Closed
@pshipton pshipton added the jdk15 label Jul 24, 2020
@pshipton pshipton added this to the Release 0.22 (Java 15) milestone Jul 24, 2020
@dmitripivkine
Copy link
Contributor

dmitripivkine commented Jul 27, 2020 •
edited
Loading

From failure for ClassLoadingTest_special_14_FAILED:
There is number of classes with class object collected:

  <gc check (1): from debugger: CLASS HEAP: slots slot 5f0900(5f0930) -> 703e73ed8: object not in an object region>
  <gc check (2): from debugger: CLASS HEAP: slots slot 583c00(583c30) -> 700d84b30: corrupt data exception>
  <gc check (3): from debugger: CLASS HEAP: slots slot 583900(583930) -> 700d848f8: class pointer is null>
  <gc check (4): from debugger: CLASS HEAP: slots slot 583300(583330) -> 700d84848: class pointer is null>
  <gc check (5): from debugger: CLASS HEAP: slots slot 503300(503330) -> 700d5dd10: corrupt data exception>
  <gc check (6): from debugger: CLASS HEAP: slots slot 502800(502830) -> 700d55a58: class pointer not in a class segment>
  <gc check (7): from debugger: CLASS HEAP: slots slot 502500(502530) -> 700d55980: class header invalid>
  <gc check (8): from debugger: CLASS HEAP: slots slot 502200(502230) -> 700d48808: class pointer not in a class segment>
  <gc check (9): from debugger: CLASS HEAP: slots slot 4ffa00(4ffa30) -> 700ccbcc0: class header invalid>
  <gc check (10): from debugger: CLASS HEAP: slots slot 4ff700(4ff730) -> 700ce4f50: class pointer not in a class segment>
  <gc check (11): from debugger: CLASS HEAP: slots slot 4ff400(4ff430) -> 700cc3bb8: corrupt data exception>

All of these classes have an Anonymous flag set 0x24: U32 classFlags = 0x00000040 (64) but seems not listed in InternalAnonymousClassLoader:

> !classloaderssummary

[Classloaders Summary]
<num, Classloader name                                          , # Classloaders, # Loaded Classes, Total Segments, Segments memory>
  0)  jdk/internal/loader/ClassLoaders$AppClassLoader           , 1             , 651             , 59            , 2697392
  1)  jdk/internal/reflect/DelegatingClassLoader                , 13            , 13              , 26            , 28696
  2)  net/adoptopenjdk/test/classloading/deadlock/MyClassLoader2, 1             , 2               , 4             , 6408
  3)  jdk/internal/loader/ClassLoaders$PlatformClassLoader      , 1             , 48              , 14            , 220944
  4)  net/adoptopenjdk/test/classloading/deadlock/MyClassLoader1, 1             , 2               , 4             , 6456
  5)  *System*                                                  , 1             , 2920            , 272           , 5091192
  6)  java/lang/InternalAnonymousClassLoader                    , 1             , 0               , 0             , 0 <------------------------

<# Loaded Classes, # Classloaders, Classloader Names>
2920             , 1             , *System*
651              , 1             , jdk/internal/loader/ClassLoaders$AppClassLoader
48               , 1             , jdk/internal/loader/ClassLoaders$PlatformClassLoader
13               , 13            , jdk/internal/reflect/DelegatingClassLoader
2                , 2             , net/adoptopenjdk/test/classloading/deadlock/MyClassLoader2, net/adoptopenjdk/test/classloading/deadlock/MyClassLoader1
0                , 1             , java/lang/InternalAnonymousClassLoader <------------------------

Number of Classloaders: 19

Segment Totals:

<# RAM Segments, RAM Segments memory, # ROM Segments, ROM Segments memory, Total Segments, Segments memory>
 212           , 6247208            , 167           , 1803880            , 379           , 8051088

@pshipton @hangshao0 Do we have new code in JVM does not respect GC classes handling convention? Is it code for JEP 371 Hidden Classes perhaps #9328? ... Or just Anonymous classes creation code is broken and allowed GC occur and collect class object before real class instantiation.

There is the similar picture for second failure... All troubles start with collecting of class objects from classes with Anonymous flag set.

@pshipton
Copy link
Member

Perhaps related to #10207

@hangshao0
Copy link
Contributor

All of these classes have an Anonymous flag set 0x24: U32 classFlags = 0x00000040 (64) but seems not listed in InternalAnonymousClassLoader:

After #10207, there are classes that has anonymous flag set are not loaded by InternalAnonymousClassLoader. Is GC checking this flag ?

@dmitripivkine
Copy link
Contributor

dmitripivkine commented Jul 27, 2020 •
edited
Loading

All of these classes have an Anonymous flag set 0x24: U32 classFlags = 0x00000040 (64) but seems not listed in InternalAnonymousClassLoader:

After #10207, there are classes that has anonymous flag set are not loaded by InternalAnonymousClassLoader. Is GC checking this flag ?

"Checking" is a weak statement here. Yes, there are an exceptional handling of Anonymous classes across GC policies required for individual unloading. It might include assumption that Anonymous class must be discovered in Anonymous classloader. I believe convention for Anonymous classes handling is not correct for Balanced any more

This was referenced Jul 27, 2020
Closed
Closed
@hangshao0
Copy link
Contributor

Currently all the hidden classes has anonymous flag set. I will try setting anonymous flag only on hidden class that needs to be individual unloaded.

@dmitripivkine
Copy link
Contributor

Currently all the hidden classes has anonymous flag set. I will try setting anonymous flag only on hidden class that needs to be individual unloaded.

How this fix the problem except you are going to keep such classes in Anonymous classloader?

@dmitripivkine
Copy link
Contributor

I obviously do not understand specific details regarding Hidden classes and related requirements. However there are two models of class handling (not just for unloading purpose but for general roots discovering) in GC: regular classes and Anonymous classes. Hidden classes should be undistinguished from these two types (compatible) otherwise special handling in GC code is required to be designed.

@hangshao0
Copy link
Contributor

There are 2 types of hidden classes:

  1. Defined without ClassOption.STRONG - it is unloaded individually, before its class loader is GC'ed
  2. Defined with ClassOption.STRONG - it is unloaded when its class loader is GC'ed.

So I think we can construct the 1) the same as Anonymous classes, as mentioned in #9328 (comment).
It is in its own rom segment, own ram segment, and its loader is VM->anonClassLoader.

For 2), it will be construced as normal class from GC point of view - it won't be in its own rom segment, won't be in its own ram segment, its loader its hostclass's loader.

Do I miss anything ?
The above is done by the last commit here: https://github.com/hangshao0/openj9/commits/JEP371_Test
The current code in openj9 master stream does not distinguish between 1) and 2) yet.

@dmitripivkine
Copy link
Contributor

There are 2 types of hidden classes:

  1. Defined without ClassOption.STRONG - it is unloaded individually, before its class loader is GC'ed
  2. Defined with ClassOption.STRONG - it is unloaded when its class loader is GC'ed.

So I think we can construct the 1) the same as Anonymous classes, as mentioned in #9328 (comment).
It is in its own rom segment, own ram segment, and its loader is VM->anonClassLoader.

For 2), it will be construced as normal class from GC point of view - it won't be in its own rom segment, won't be in its own ram segment, its loader its hostclass's loader.

Do I miss anything ?
The above is done by the last commit here: https://github.com/hangshao0/openj9/commits/JEP371_Test
The current code in openj9 master stream does not distinguish between 1) and 2) yet.

It should work from GC point of view (plus J9ClassIsAnonymous flag should not be used for classes defined with ClassOption.STRONG). I hope this approach would not create conceptual problem from VM side.

@hangshao0
Copy link
Contributor

hangshao0 commented Jul 27, 2020 •
edited
Loading

plus J9ClassIsAnonymous flag should not be used for classes defined with ClassOption.STRONG.

J9ClassIsAnonymous won't be used for hidden classes defined with ClassOption.STRONG.

From Test_openjdk15_j9_special.system_x86-64_windows_Personal_otherLoadTest/2/console:
I am able to pass ClassLoadingTest_special_14 with my latest change in https://github.com/hangshao0/openj9/commits/JEP371_Test.
14:45:22 ClassLoadingTest_special_14_PASSED

However, I am getting an assertion failure in ClassLoadingTest_special_15
14:46:49 CLT stderr 18:46:20.681 0x193f00 j9mm.107 * ** ASSERTION FAILED ** at CopyForwardScheme.cpp:819: ((false && (newRegion->getReferenceObjectList()->isSoftListEmpty())))

@dmitripivkine
Copy link
Contributor

For the record: I understand mechanism of failure for Balanced. Every classloader should have a knowledge which regions associated roots are located. It is implemented by using Classloader Remembered Set. For Anonymous classes however this information should be available for each class to have ability for individual unloading. For each class with J9ClassIsAnonymous flag set such information would be reordered but never requested because class can not be discovered by scanning internal Anonymous classloader

@dmitripivkine
Copy link
Contributor

However, I am getting an assertion failure in ClassLoadingTest_special_15
14:46:49 CLT stderr 18:46:20.681 0x193f00 j9mm.107 * ** ASSERTION FAILED ** at CopyForwardScheme.cpp:819: ((false && (newRegion->getReferenceObjectList()->isSoftListEmpty())))

Do you have link to failed job with stored artifacts?

@hangshao0
Copy link
Contributor

Shared the link to the job via private message.

@hangshao0
Copy link
Contributor

A bit more information on the hidden classes:

For both 1) and 2) mentioned in #10249 (comment):
They are not added into hashClassTable.
They have the same class name format as anon classes, which is [className]/[ROMClassAddress].

@hangshao0
Copy link
Contributor

Had a discussion with Dmitri, we know what is going on here. But I will leave it to him to comment on what he has found out on the GC side.

@dmitripivkine
Copy link
Contributor

So we figured out what is wrong now. While Standard Collectors scanning Classloader's RAM Segments and Hash Table both Balanced has been optimized to scan Hash Table only for all classloaders except System and internal Anonymous. But hidden classes never added to hash table, so they are still be invisible and missed for scanning

@gacholio gacholio mentioned this issue Jul 28, 2020
Merged
@hangshao0 hangshao0 mentioned this issue Jul 30, 2020
Merged
@hangshao0 hangshao0 mentioned this issue Aug 6, 2020
Closed
hangshao0 added a commit to hangshao0/openj9 that referenced this issue Aug 6, 2020
@hangshao0
JEP 371 implementation part 3
9906e91 
This change adds code to handle ClassOption.STRONG
1) For hidden class defined without ClassOption.STRONG, 
set its classLoader to anon classLoader. J9_FINDCLASS_FLAG_ANON is 
set for such classes. 
2) For hidden class defined with ClassOption.STRONG, set its 
classLoader to its hostclass's classLoader. J9_FINDCLASS_FLAG_ANON 
is not set for such classes.
3) With 1) and 2) the field that I added in my previous change
ROMClassSegmentAllocationStrategy::__allocNewSeg is not necessary
anymore. Remove this field.
4) Add hidden class in 2) to the classTable of its hostclass's 
ClassLoader. In this way, the current balanced GC code is able to 
find such class when iterating through the classTable. 

Adding hidden classes to the classTable makes it not sharable via 
shared classes cache, as its name won't be unique across JVMs. The 
change in 4) is temporary if we finally decide to change balance 
GC to scan the ram segment like other collectors. A flag can be 
used to control whether to skip hidden class when iterating the 
class table. 

Issue eclipse-openj9#9328
Fixes eclipse-openj9#10249

Signed-off-by: Hang Shao <hangshao@ca.ibm.com>
@dmitripivkine dmitripivkine mentioned this issue Aug 7, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
comp:gc comp:vm jdk15 jdk16 test failure
Projects
None yet
Milestone
Release 0.22 (Java 15)
Development

Successfully merging a pull request may close this issue.

JEP 371 implementation part 3 hangshao0/openj9
4 participants
@JasonFengJ9 @hangshao0 @pshipton @dmitripivkine