Add support for persistent SCC on z/OS #17073

hangshao0 · 2023-03-30T15:10:07Z

Use mmap implementation of SCC on z/OS which can be enabled by
"persistent" sub-option.
Flush the change of the mapped region to the SCC file on VM exit.
Use map64 by default on Java 11 and up. Add new sub-option "map31"
which maps SCC below the 2G addressing range.
Add test cases
Remove the temporary define of OMRPORT_MMAP_FLAG_ZOS_READ_MAPFILE

Depends on eclipse/omr#6940
Depends on eclipse/omr#6989

Signed-off-by: Hang Shao hangshao@ca.ibm.com

hangshao0 · 2023-03-30T15:10:50Z

@pshipton

runtime/nls/shrc/j9shr.nls

runtime/oti/j9port.h

runtime/bcutil/jimagereader.c

runtime/shared_common/CompositeCache.cpp

test/functional/cmdLineTests/shareClassTests/SCCMLTests/ShareClassesCMLTests-1.xml

pshipton · 2023-03-31T13:58:30Z

What are the concerns with making map64 the default?

The testing is light. Have you tried making a JVM with persistent as the default and running all the functional tests?

pshipton · 2023-03-31T14:52:11Z

running all the functional tests

Actually the system tests as well, there are some class loading and shared cache related system tests.

hangshao0 · 2023-03-31T14:54:25Z

What are the concerns with making map64 the default?

__MAP_64 is only supported on z/OS version 2.5 and some z/OS version 2.4. The 64 bit mapping is not on by default on z/OS mmap() where it is supported. Both @r30shah and I have seen __MAP_64 is easy to cause out of memory error (ENOMEM) on mmap(). The ENOMEM goes away if a smaller cache size is used. There might be some machine configurations that could control this. Not sure if @r30shah has found out something on this.

The testing is light. Have you tried making a JVM with persistent as the default and running all the functional tests?

Not yet.

hangshao0 · 2023-03-31T15:19:56Z

I've also added change regarding:
eclipse/omr#6940 (comment).

pshipton · 2023-03-31T16:01:41Z

I've also added change regarding:
eclipse/omr#6940 (comment).

I think you misunderstood. This change doesn't add any value to this PR since the OMR PR must be merged first. I want a separate PR from this that adds that code and the changes for mmapflag |= OMRPORT_MMAP_FLAG_ZOS_READ_MAPFILE that can be merged now before the OMR change, so merging the OMR change doesn't break OpenJ9.

hangshao0 · 2023-03-31T16:17:32Z

PR for the OMR flag #17083

hangshao0 · 2023-03-31T20:13:55Z

As #17083 is merged, I pushed to resolve the merge conflicts.

pshipton · 2023-03-31T20:18:02Z

Now this PR can be updated to remove #define OMRPORT_MMAP_FLAG_ZOS_READ_MAPFILE 0 because this won't be merged until after the OMR change is merged.

hangshao0 · 2023-03-31T20:21:39Z

Now this PR can be updated to remove #define OMRPORT_MMAP_FLAG_ZOS_READ_MAPFILE 0

Yes, it can be done.

buildspecs/j9.flags

runtime/shared_common/CompositeCache.cpp

runtime/shared_common/OSCachemmap.cpp

runtime/shared_common/shrinit.cpp

runtime/tests/shared/AttachedDataTest.cpp

test/functional/cmdLineTests/shareClassTests/SCCMLTests/ShareClassesCMLTests-1.xml

hangshao0 · 2023-04-06T15:18:58Z

Have you tried making a JVM with persistent as the default and running all the functional tests?
Actually the system tests as well, there are some class loading and shared cache related system tests.

The heavy class loading tests like SharedClasses.SCM01.SingleCL, SharedClasses.SCM01.MultiThreadMultiCL are in extended.system.
But there are issues with Jenkins machines, no persistent SCC > 10m are allowed to create on these machine. I get:

JVMSHRC336E Port layer error code = -155
JVMSHRC337E Platform error message: EDC5124I Too many open files.
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

This means mmap() failed with EMFILE. Most SCC related tests directly fail on this error.

I tried another build with the default cache size reduced to 10m, most of these SCC class loading tests passed. There are much fewer tests failed with the error above.

I guess there are machine settings that is related to this error. Probably things like MAXFILEPROC, MAXMMAPAREA, MAXSHAREPAGES. Could you check these settings on internal Jenkins z/OS machine, like fyrec51n.vmec.svl.ibm.com ? @jdekonin @vsebe

jdekonin · 2023-04-06T17:32:10Z

bash-4.3$ oeconsol "D VS,HVSHARE"
IAR019I  10.28.37 DISPLAY VIRTSTOR 870
 SOURCE = DEFAULT
 TOTAL SHARED = 522240G
 SHARED RANGE = 2048G-524288G
 SHARED ALLOCATED = 1050446M
bash-4.3$ oeconsol "D OMVS,OPTIONS"
BPXO043I 10.27.08 DISPLAY OMVS 866
OMVS     000F ACTIVE             OMVS=(00)
CURRENT UNIX CONFIGURATION SETTINGS:
MAXPROCSYS      =        900    MAXPROCUSER     =        100
MAXFILEPROC     =      10000    MAXFILESIZE     = NOLIMIT
MAXCPUTIME      =      10800    MAXUIDS         =         50
MAXPTYS         =        800    MAXIOBUFUSER    =       2048
MAXMMAPAREA     =       4096    MAXASSIZE       = 2147483647
MAXTHREADS      =      10000    MAXTHREADTASKS  =      10000
MAXCORESIZE     =    4194304
IPCMSGQBYTES    = 2147483647    IPCMSGQMNUM     =      10000
IPCMSGNIDS      =        500    IPCSEMNIDS      =        500
IPCSEMNOPS      =         25    IPCSEMNSEMS     =       1000
IPCSHMMPAGES    =      25600    IPCSHMNIDS      =        500
IPCSHMNSEGS     =        500    IPCSHMSPAGES    =     262144
SUPERUSER       = BPXROOT       FORKCOPY        = COPY
STEPLIBLIST     =
USERIDALIASTABLE=
PRIORITYPG VALUES: NONE
PRIORITYGOAL VALUES: NONE
MAXQUEUEDSIGS   =       1000    SHRLIBRGNSIZE   =  157286400
SHRLIBMAXPAGES  =       4096    VERSION         = /       ,N
SYSCALL COUNTS  = NO            TTYGROUP        = TTY
SYSPLEX         = NO            BRLM SERVER     = N/A
LIMMSG          = SYSTEM        AUTOCVT         = OFF
RESOLVER PROC   = DEFAULT       LOSTMSG         = ON
AUTHPGMLIST     = NONE
SWA             = BELOW         NONEMPTYMOUNTPT = NOWARN
SERV_LINKLIB    =
SERV_LPALIB     =
MAXUSERMOUNTSYS =          0    MAXUSERMOUNTUSER=          0
MAXPIPEUSER     =       8730    PWT             = ENV
KERNELSTACKS    = ABOVE         UMASK           = NONE

hangshao0 · 2023-04-06T17:51:56Z

I notice that I can create persistent SCC with a much larger size 60m on VM farm machine stlab92.svl.ibm.com:

java -Xshareclasses:name=C1,persistent,reset -Xscmx60m -version
JVMJ9VM082E Unable to switch to IFA processor - issue "extattr +a libj9ifa29.so"
JVMSHRC023E Cache does not exist
java version "1.8.0_361"
Java(TM) SE Runtime Environment (build 8.0.8.0 - pmz6480sr8-20230208_02(SR8))
IBM J9 VM (build 2.9, JRE 1.8.0 z/OS s390x-64-Bit Compressed References 20230321_47772 (JIT enabled, AOT enabled)
OpenJ9   - cd369c6
OMR      - 0d8205c
IBM      - 103e724)
JCL - 20230208_01 based on Oracle jdk8u361-b09

Could you check the settings of stlab92.svl.ibm.com so that we can compare them against #17073 (comment) ? @jdekonin

jdekonin · 2023-04-06T18:56:58Z

The 2 systems are not the same release. fyrec51n is 2.5, whereas stlab92 is 2.1. Note the older system has a MAXSHAREPAGES env var.

MAXPROCSYS      =       1000    MAXPROCUSER     =        500 
MAXFILEPROC     =      64000    MAXFILESIZE     = NOLIMIT    
MAXCPUTIME      =     100000    MAXUIDS         =        250 
MAXPTYS         =        512    MAXIOBUFUSER    =       2048 
MAXMMAPAREA     =      40960    MAXASSIZE       = 2147483647 
MAXTHREADS      =       1000    MAXTHREADTASKS  =       1000 
MAXCORESIZE     =    4194304    MAXSHAREPAGES   =     131072 
IPCMSGQBYTES    = 2147483647    IPCMSGQMNUM     =      10000 
IPCMSGNIDS      =        500    IPCSEMNIDS      =        500 
IPCSEMNOPS      =         25    IPCSEMNSEMS     =       1000 
IPCSHMMPAGES    =      25600    IPCSHMNIDS      =        500 
IPCSHMNSEGS     =        500    IPCSHMSPAGES    =     262144

hangshao0 · 2023-04-06T19:06:58Z

Looks like it is MAXMMAPAREA that is making a difference here:

On fyrec51n.vmec.svl.ibm.com, it is 4096, which is equivalent to 16MB
On stlab92.svl.ibm.com, it is 40960, which is equivalent to 160MB.

stlab92.svl.ibm.com can allow SCC with 160m, but fail with 161m.

java -Xshareclasses:name=C1,persistent,reset -Xscmx160m -version
JVMJ9VM082E Unable to switch to IFA processor - issue "extattr +a libj9ifa29.so"
JVMSHRC023E Cache does not exist
java version "1.8.0_361"
Java(TM) SE Runtime Environment (build 8.0.8.0 - pmz6480sr8-20230208_02(SR8))
IBM J9 VM (build 2.9, JRE 1.8.0 z/OS s390x-64-Bit Compressed References 20230321_47772 (JIT enabled, AOT enabled)
OpenJ9   - cd369c6
OMR      - 0d8205c
IBM      - 103e724)

java -Xshareclasses:name=C1,persistent,reset -Xscmx161m -version
JVMJ9VM082E Unable to switch to IFA processor - issue "extattr +a libj9ifa29.so"
JVMSHRC256I Persistent shared cache "C1" has been destroyed
JVMSHRC245E Error mapping shared class cache file
JVMSHRC336E Port layer error code = -155
JVMSHRC337E Platform error message: EDC5124I Too many open files. (errno2=0x07360344)
JVMSHRC840E Failed to start up the shared cache.
JVMJ9VM015W Initialization error for library j9shr29(11): JVMJ9VM009E J9VMDllMain failed
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

hangshao0 · 2023-04-06T19:31:23Z

Seems that the MAXMMAPAREA value is too small on both VM farm and Jenkins z/OS test machines:

I just randomly tried on a VM farm machine fyrec51a.vmec.svl.ibm.com, 16m is allowed but 17m is too big:

java -Xshareclasses:name=C1,persistent,reset -Xscmx16m -version
JVMJ9VM082E Unable to switch to IFA processor - issue "extattr +a libj9ifa29.so"
JVMSHRC256I Persistent shared cache "C1" has been destroyed
java version "1.8.0_361"
Java(TM) SE Runtime Environment (build 8.0.8.0 - pmz6480sr8-20230208_02(SR8))
IBM J9 VM (build 2.9, JRE 1.8.0 z/OS s390x-64-Bit Compressed References 20230321_47772 (JIT enabled, AOT enabled)
OpenJ9   - cd369c6
OMR      - 0d8205c
IBM      - 103e724)
JCL - 20230208_01 based on Oracle jdk8u361-b09

java -Xshareclasses:name=C1,persistent,reset -Xscmx17m -version
JVMJ9VM082E Unable to switch to IFA processor - issue "extattr +a libj9ifa29.so"
JVMSHRC256I Persistent shared cache "C1" has been destroyed
JVMSHRC245E Error mapping shared class cache file
JVMSHRC336E Port layer error code = -155
JVMSHRC337E Platform error message: EDC5124I Too many open files. (errno2=0x07360344)
JVMSHRC840E Failed to start up the shared cache.

I guess we need to increase the MAXMMAPAREA value on z/OS machines. Maybe set it to 262,144 (==1GB) or 131,072 (==512MB) @jdekonin

mmap limits are a subset of the more general shared memory limits. May need to update them as well on some machines.

runtime/shared_common/OSCachemmap.cpp

When OMRPORT_MMAP_FLAG_ZOS_64BIT is passed in, we should always set flag __MAP_64 Related to eclipse-openj9/openj9#17073 Signed-off-by: Hang Shao <hangshao@ca.ibm.com>

pshipton · 2023-05-12T19:13:22Z

@keithc-ca this seems ready for another review. I want a change, but it's on the OMR side, and isn't a prereq for this.

When OMRPORT_MMAP_FLAG_ZOS_64BIT is passed in, we should always set flag __MAP_64 Related to eclipse-openj9/openj9#17073 Signed-off-by: Hang Shao <hangshao@ca.ibm.com>

When OMRPORT_MMAP_FLAG_ZOS_64BIT is passed in, we set flag __MAP_64 on z/OS version 2.4 and up. Related to eclipse-openj9/openj9#17073 Signed-off-by: Hang Shao <hangshao@ca.ibm.com>

hangshao0 · 2023-05-12T19:38:59Z

The implementation of zos_version_at_least() is in OMR code, not in OpenJ9. I have removed its declaration from OpenJ9 header util_api.h in this PR and added it into OMR header omrutil.h in eclipse/omr#6989

hangshao0 · 2023-05-12T19:56:08Z

I have removed its declaration from OpenJ9 header util_api.h in this PR and added it into OMR header omrutil.h in eclipse/omr#6989

This creates dependency between the OMR and OpenJ9 PR.

runtime/shared_common/CompositeCache.cpp

runtime/shared_common/CompositeCacheImpl.hpp

runtime/shared_common/OSCachemmap.cpp

runtime/shared_common/shrinit.cpp

runtime/tests/port/j9mmapTest.c

When OMRPORT_MMAP_FLAG_ZOS_64BIT is passed in, we set flag __MAP_64 on z/OS version 2.4 and up. Related to eclipse-openj9/openj9#17073 Signed-off-by: Hang Shao <hangshao@ca.ibm.com>

1. Use mmap implementation of SCC on z/OS which can be enabled by "persistent" sub-option. 2. Flush the change of the mapped region to the SCC file on VM exit. 3. Use map64 by default on Java 11 and up. Add new sub-option "map31" which maps SCC below the 2G addressing range. 4. Add test cases 5. Remove the temporary define of OMRPORT_MMAP_FLAG_ZOS_READ_MAPFILE Signed-off-by: Hang Shao <hangshao@ca.ibm.com>

hangshao0 · 2023-05-15T18:41:59Z

All existing review comments addressed.

keithc-ca · 2023-05-16T13:13:02Z

Jenkins test sanity zlinux,win32 jdk8 depends eclipse/omr#6989

keithc-ca

This will need to wait until eclipse/omr#6989 is promoted to the openj9 branch of OMR.

pshipton · 2023-05-29T14:14:44Z

I've promoted eclipse/omr#6989

When OMRPORT_MMAP_FLAG_ZOS_64BIT is passed in, we set flag __MAP_64 on z/OS version 2.4 and up. Related to eclipse-openj9/openj9#17073 Signed-off-by: Hang Shao <hangshao@ca.ibm.com>

hangshao0 added comp:vm os:zos depends:omr labels Mar 30, 2023

pshipton mentioned this pull request Mar 30, 2023

Implement omrmmap_map_file() using mmap() on z/OS eclipse/omr#6940

Merged

keithc-ca reviewed Mar 30, 2023

View reviewed changes

runtime/nls/shrc/j9shr.nls Outdated Show resolved Hide resolved

runtime/oti/j9port.h Show resolved Hide resolved

pshipton requested changes Mar 31, 2023

View reviewed changes

runtime/bcutil/jimagereader.c Outdated Show resolved Hide resolved

runtime/shared_common/CompositeCache.cpp Show resolved Hide resolved

test/functional/cmdLineTests/shareClassTests/SCCMLTests/ShareClassesCMLTests-1.xml Outdated Show resolved Hide resolved

hangshao0 force-pushed the FixSCC branch 2 times, most recently from a37bfd4 to 9ea5c3e Compare March 31, 2023 15:11

pshipton mentioned this pull request Mar 31, 2023

Add temporary define of OMRPORT_MMAP_FLAG_ZOS_READ_MAPFILE #17083

Merged

hangshao0 force-pushed the FixSCC branch 2 times, most recently from 67cae6a to 9459a71 Compare March 31, 2023 20:13

hangshao0 force-pushed the FixSCC branch from 9459a71 to 5416d1a Compare March 31, 2023 20:23

keithc-ca requested changes Apr 3, 2023

View reviewed changes

hangshao0 force-pushed the FixSCC branch from 5416d1a to 576872b Compare April 4, 2023 19:26

hangshao0 force-pushed the FixSCC branch 2 times, most recently from b63442d to 54193eb Compare May 11, 2023 21:12

hangshao0 mentioned this pull request May 12, 2023

Persistent Shared Cache support on z/OS eclipse-openj9/openj9-docs#1097

Closed

pshipton reviewed May 12, 2023

View reviewed changes

runtime/shared_common/OSCachemmap.cpp Show resolved Hide resolved

hangshao0 added a commit to hangshao0/omr that referenced this pull request May 12, 2023

Always define __MAP_64

03b8fd0

When OMRPORT_MMAP_FLAG_ZOS_64BIT is passed in, we should always set flag __MAP_64 Related to eclipse-openj9/openj9#17073 Signed-off-by: Hang Shao <hangshao@ca.ibm.com>

hangshao0 mentioned this pull request May 12, 2023

Always define __MAP_64 eclipse/omr#6989

Merged

hangshao0 added a commit to hangshao0/omr that referenced this pull request May 12, 2023

Always define __MAP_64

8102104

When OMRPORT_MMAP_FLAG_ZOS_64BIT is passed in, we should always set flag __MAP_64 Related to eclipse-openj9/openj9#17073 Signed-off-by: Hang Shao <hangshao@ca.ibm.com>

hangshao0 force-pushed the FixSCC branch from 54193eb to 80c6ec3 Compare May 12, 2023 19:33

keithc-ca requested changes May 12, 2023

View reviewed changes

hangshao0 force-pushed the FixSCC branch from 80c6ec3 to 254fa20 Compare May 12, 2023 20:09

hangshao0 force-pushed the FixSCC branch from 254fa20 to 40075eb Compare May 15, 2023 18:36

keithc-ca approved these changes May 16, 2023

View reviewed changes

keithc-ca merged commit 3b029b0 into eclipse-openj9:master May 29, 2023
7 checks passed

pshipton added the depends:doc label Jun 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for persistent SCC on z/OS #17073

Add support for persistent SCC on z/OS #17073

hangshao0 commented Mar 30, 2023 •

edited

hangshao0 commented Mar 30, 2023

pshipton commented Mar 31, 2023

pshipton commented Mar 31, 2023

hangshao0 commented Mar 31, 2023

hangshao0 commented Mar 31, 2023

pshipton commented Mar 31, 2023

hangshao0 commented Mar 31, 2023

hangshao0 commented Mar 31, 2023

pshipton commented Mar 31, 2023

hangshao0 commented Mar 31, 2023

hangshao0 commented Apr 6, 2023

jdekonin commented Apr 6, 2023

hangshao0 commented Apr 6, 2023 •

edited

jdekonin commented Apr 6, 2023

hangshao0 commented Apr 6, 2023

hangshao0 commented Apr 6, 2023 •

edited

pshipton commented May 12, 2023

hangshao0 commented May 12, 2023

hangshao0 commented May 12, 2023

hangshao0 commented May 15, 2023

keithc-ca commented May 16, 2023

keithc-ca left a comment

pshipton commented May 29, 2023

Add support for persistent SCC on z/OS #17073

Add support for persistent SCC on z/OS #17073

Conversation

hangshao0 commented Mar 30, 2023 • edited

hangshao0 commented Mar 30, 2023

pshipton commented Mar 31, 2023

pshipton commented Mar 31, 2023

hangshao0 commented Mar 31, 2023

hangshao0 commented Mar 31, 2023

pshipton commented Mar 31, 2023

hangshao0 commented Mar 31, 2023

hangshao0 commented Mar 31, 2023

pshipton commented Mar 31, 2023

hangshao0 commented Mar 31, 2023

hangshao0 commented Apr 6, 2023

jdekonin commented Apr 6, 2023

hangshao0 commented Apr 6, 2023 • edited

jdekonin commented Apr 6, 2023

hangshao0 commented Apr 6, 2023

hangshao0 commented Apr 6, 2023 • edited

pshipton commented May 12, 2023

hangshao0 commented May 12, 2023

hangshao0 commented May 12, 2023

hangshao0 commented May 15, 2023

keithc-ca commented May 16, 2023

keithc-ca left a comment

Choose a reason for hiding this comment

pshipton commented May 29, 2023

hangshao0 commented Mar 30, 2023 •

edited

hangshao0 commented Apr 6, 2023 •

edited

hangshao0 commented Apr 6, 2023 •

edited