Change default cache directory to user's home

[hangshao0]

Issue or pull request number:
eclipse-openj9/openj9#2862

Overview:
1. The default shared cache and cache snapshot directory is changed to user's home on OpenJ9 Java 11 and up on non-Windows platforms if -Xshareclasses:groupAccess is not used. The default shared cache and snapshot directory is still /tmp/javasharedresources/ if "groupAccess" is used.
2. New sub-options -Xshareclasses:bootClassesOnly and -Xshareclasses:fatal

Applies to the following JDK versions:
1. Java 11 and later for the change of default cache directory
2. Java 8 and up for the new bootClassesOnly and fatal option

Applies to the following platforms:
1. AIX, Linux X|P|Z, z/OS, for the change of default cache directory
2. All platforms, for bootClassesOnly and fatal option

For new command line options:

• Option name and syntax: -Xshareclasses:bootClassesOnly and -Xshareclasses:fatal
• If values are set:
  N.A.
• Purpose:
  Option "bootClassesOnly": Disable caching of classes loaded by the non-bootstrap class loaders. This option also turns on option "nonfatal".
  Option "fatal": Do not start up the shared cache if an error occurs. This option is turned on by default except when "bootClassesOnly" is used. User can use -Xshareclasses:bootClassesOnly,fatal if they want the shared cache error to be fatal.
• Restrictions:
  With -Xshareclasses, if we cannot user user's home as the cache directory (e.g. home is on a NFS or an error occurs when getting the home directory, etc.):

1. If "bootClassesOnly" or "nonfatal" is used, the JVM will continue to start up without the shared cache.
2. Otherwise JVM won't start up. There will be an error message. User can use -Xshareclasses:cacheDir= to specify the new cache directory to start up the JVM. Or change the home directory if it is on a NFS.

Migration:

1. Users that always use "groupAccess" or "cacheDir=" in the CML won't be affected by this change.

2. Shared caches and cache snapshots created in the obsolete default directory (/tmp/javasharedresources/) by an old JDK11 without this change cannot be started up by the new JDK11. User can use "-Xshareclasses:cacheDir=/tmp/javasharedresources/,listAllCaches" ("cacheDir=/tmp" for nonpersistent cache and snapshot) to find the caches (and cache snapshots) in the obsolete default directory. If an old shared cache is not useful anymore, use "-Xshareclasses:cacheDir=/tmp/javasharedresources/,name=cacheName,destroy" to remove it, For cache snapshot, use "-Xshareclasses:cacheDir=/tmp,name=snapshotName,destroySnapshot" to remove it.

3. If user does not destroy the old shared cache or cache snapshot in the obsolete default directory, and the new JDK 11 is creating a new shared cache or cache snapshot with "-Xshareclasses:name=existingCacheName,cacheDir=/tmp/javasharedresources/" ("cacheDir=/tmp" for nonpersistent cache and snapshot), the old shared cache or cache snapshot will be automatically deleted and a new one will be created.

4. If "cacheDir=" or "groupAccess" is not used in the CML, please make sure the user's home directory is not on a NFS.

[pshipton]

We're going to need additional details about "default shared cache directory is changed to user's home". There are some cases where this does not occur.

[hangshao0]

Added more content in the "Restrictions" part.

[pshipton]

and what about group access?

[pshipton]

I think this should be split into two parts since there are two PRs for the changes which can be delivered separately and in different releases. This one should cover the new options, and the other should cover moving the home directory.

[pshipton]

Sorry scratch that, I guess the single PR enables caches in the home directory.

[hangshao0]

and what about group access?

Added "The default shared cache and snapshot directory is still /tmp/javasharedresources/ if "groupAccess" is used"

[pshipton]

Please add a migration story, what users should do if they have created a Java 11 shared cache before this was introduced.

[hangshao0]

Added a migration section.

[pshipton]

Snapshots should be mentioned as well since they are affected.

[hangshao0]

Added cache snapshot in the migration part.

[doveye]

A few questions for starters (feel free to point out that the answers will be obvious to Java users if that's the case!):

1. Why is the default shared class cache dir changing?
2. Why/when would a user want to use the bootClassesOnly sub-option?
3. I guess the fatal sub-option is there to enable better troubleshooting, ie find any issues with cache dirs up front? So why does the bootClassesOnly sub-option turn on nonfatal by default?
4. There will be an error message. (From Restrictions point 2.) What is the error message please?

[pshipton]

Why is the default shared class cache dir changing?

To put the default cache in a location that is not writable by all users.

Why/when would a user want to use the bootClassesOnly sub-option?

Mostly for debugging pr service purposes. These changes are a pre-cursor to enabling bootClassesOnly by default. Having the option allows additional command line options to be enabled at the same time, such as the verbose options. It is also possible a user may want to enable shared classes only for bootstrap classes, perhaps to reduce the risk of using shared classes, although no one has ever requested this.

I guess the fatal sub-option is there to enable better troubleshooting, ie find any issues with cache dirs up front? So why does the bootClassesOnly sub-option turn on nonfatal by default?

See the answer above. bootClassesOnly will be enabled by default at a future time, and by default we want nonfatal as well. Having bootClassesOnly enable it avoid mistakes. Adding fatal allows control over this behavior for debugging or service, or by some chance a user wants to enable bootClassesOnly but have the JVM fail to start if something is wrong with the shared cache.

[pshipton]

@hangshao0 please answer 4.

[doveye]

Thanks Peter. Another question - there are statements like

please make sure the user's home directory is not on a NFS.

but currently the docs say that's a restriction only for persistent caches on AIX - is that still the case?

For AIX, the directory must not be on an NFS mount for persistent caches.

[pshipton]

The AIX part is misleading, it is a restriction for persistent caches on all Windows, Linux and AIX (z/OS doesn't support persistent caches).

[doveye]

Hi @hangshao0 and @pshipton . I've made the doc updates (inadvertently skipped the staging part, oops). They are visible here:

• In the -Xshareclasses topic (https://www.eclipse.org/openj9/docs/xshareclasses/#cachedir, https://www.eclipse.org/openj9/docs/xshareclasses/#bootclassesonly, https://www.eclipse.org/openj9/docs/xshareclasses/#fatal)
• What's New (draft) - see the sections "Changes to the location of the default shared cache and cache snapshot directory" and "New class data sharing suboptions" in https://github.ibm.com/Sue-Chaplain/sues-test-repo/blob/master/version0.11.md#changes-to-the-location-of-the-default-shared-cache-and-cache-snapshot-directory

I've removed the AIX mention in the existing NFS sentence too, thanks Peter for clarifying that.

[pshipton]

lgtm except the following in the What's New draft is not entirely accurate.

Caches cannot be shared across different Java releases, so you cannot re-use a cache that was created by a previous level of Java 11; if you use the name and cacheDir suboptions to specify an existing cache, the cache is deleted and a new one created.

The JVM attempts to delete the obsolete cache and create a new one. Unlike on Linux/AIX, on Windows a file cannot be deleted if it is in use. In this case the new JVM will continue to use the older cache if the older cache cannot be deleted.

I missed a point in #103 (comment). The bootClassesOnly option is also useful if you want this behavior but need to change the default cacheDir location.

[hangshao0]

4. There will be an error message. (From Restrictions point 2.) What is the error message please?

There will be an NLS saying what stopped shared class from starting up. The message is different depending on what the error condition is.

[doveye]

Re #103 (comment):
Thanks Peter. How about the following replacement sentence:

In general, caches cannot be shared across different Java releases, so you cannot re-use a cache that was created by a previous level of Java 11; if you use the name and cacheDir suboptions to specify an existing cache, the VM attempts to delete the cache and create a new one. However, on Windows, the cache cannot be deleted if it is in use, in which case the VM continues to use the existing cache.

I'm not sure what you mean by "this behavior" in this sentence?

The bootClassesOnly option is also useful if you want this behavior but need to change the default cacheDir location

[pshipton]

lgtm.

I'm not sure what you mean by "this behavior" in this sentence?

The bootClassesOnly option is useful if you want to have the equivalent of that is going to be enabled by default, but change the default cacheDir. I don't expect this needs to be called out in the doc, its just a point I missed mentioning earlier.

[doveye]

Ok, thanks for the clarification :)

[doveye]

Think we can close this now.