Working around Java 16/17 internals compatibility issues

[chadlwilson]

Background

In Java 16 and 17 access to the internals of the Java Virtual Machine (JVM) via reflection were restricted by default as part of JEP 403.

Why might this effect GoCD?

• GoCD is a Java-based product 🙂
• As of GoCD 22.1.0 the Server still uses a couple of libraries that rely on reflective access to JVM internals. This is unfortunate, but these are difficult to remove.
• As of GoCD 21.4.0 the Agent (and its bootstrapper) is not believed to be dependent on access to JVM internals at all.
• It is possible that there may be some server paths that our regression tests do not cover
• Plugin code runs on both server and agent. It is possible (but unlikely) that some plugin code may depend on access to some JVM internals

What is the purpose of this documentation?

If you encounter such issues; to allow you to workaround the issue before reporting it, so it does not block your upgrade to resolve other security issues in GoCD < 22.1.0, nor require a downgrade.

How do I know if I have a JVM internals access issue affecting my GoCD?

You will see errors in your logs such as:

Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private java.lang.String java.util.regex.Pattern.pattern accessible: module java.base does not "opens java.util.regex" to unnamed module

How can I workaround this?

To workaround the issue, the important part in the error is module ${module} does not "opens ${package}" to unnamed module.

1. Construct a special argument using ${module} and ${package} as follows:
   • --add-opens=${module}/${package}=ALL-UNNAMED
   • e.g --add-opens=java.base/java.util.regex=ALL-UNNAMED
2. We need to add this as a JVM option for either the server or the agent depending on where the error is being experienced.
3. If this was found in your server logs
   • Normal installs: Find the location of the wrapper-config/wrapper-properties.conf for your install type and add a value like wrapper.java.additional.100=--add-opens=java.base/java.util.regex=ALL-UNNAMED.
     • You can add multiple values with different indexes (101,102 etc) if necessary.
   • Docker installs: Add the --add-opens argument above to your GOCD_SERVER_JVM_OPTS, per docs here
   • Helm installs: Add the --add-opens argument above to your server.env.goServerJvmOpts values file or --set args, per docs here
4. If this was found in your agent logs
   • Normal installs: Find the location of the wrapper-config/wrapper-properties.conf for your install type and add a value like set.AGENT_STARTUP_ARGS=-Xms128m -Xmx256m --add-opens=java.base/java.util.regex=ALL-UNNAMED.
     • You can add multiple values if necessary, and the order does not matter. The memory values above are the defaults for agents. If you have overridden these you may need to adjust them.
   • Docker installs: Add the --add-opens argument above to your GOCD_AGENT_JVM_OPTS, per docs here
   • Helm installs: Add the --add-opens argument above to your server.env.goAgentJvmOpts values file or --set args, per docs here
5. Restart your server or agent. If doing so for a server, consider taking a backup and putting the server into maintenance mode before shutting down.
6. If there are more than one issues, you may experience a different error after restart, for a different module or package. You may need to add additional options if you experience more.
7. Please report the issue, with the full stack trace, so we can see where it is coming from (a particular plugin? a part of the server or agent?)

I tried to downgrade my server and I am getting these errors on agents. Help!

If you experience other issues and downgrade your server to <= 21.3.0 while leaving agents running under a 22.1.0+ bootstrapper and Java 17 you are likely to get errors like these in your agents logs.

When you downgrade your server, the agents also downgrade, however the Java Virtual Machine they are running in doesn't change. Agent versions <= 21.3.0 aren't compatible out-of-the-box with Java 16/17.

You have two options:

1. Downgrade all the agent bootstrappers back to <= 21.4.0 and/or Java 15.
2. Or Follow the agent instructions above, to configure --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED, which should be sufficient to get old agent versions working OK on Java 16/17.

Is this a security concern?

I don't believe so.

It's more of a reliability/maintainability concern for GoCD developers. If certain things are changed within the JVM, parts of GoCD may stop working in strange ways.

What opens are added by default?

At time of writing, the default 22.1.0 installers add --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED for the server only.

For the latest state look at getJvmModuleOpensArgs here.