Update OpenSSL version to 1.1.1 on Eclipse PR Jenkins build and test machines

[a7ehuo]

Some PPC linux JITServer PR builds and tests #8117 [1] fail due to the lower versions of the OpenSSL installed on the system. The JIT is built against 1.1.1d fetched into /home/jenkins/workspace/Build_JDK8_ppc64le_linux_jit_Personal/openssl. During the runtime, it would link to the system installed OpenSSL which is 1.0.2g on those machines.

Could all the PPC build and test machines have OpenSSL 1.1.1 installed? Thanks!

18:30:15  + bash ./get_source.sh --openssl-version=1.1.1d
...

18:33:36  + bash configure --with-freemarker-jar=/home/jenkins/freemarker.jar --with-boot-jdk=/usr/lib/jvm/adoptojdk-java-80 --enable-cuda --with-cuda=/usr/local/cuda-9.0 --with-openssl=fetched --enable-jitserver=yes

18:57:19  JVMJ9VM011W Unable to load j9jit29: libssl.so.1.1: cannot open shared object file: No such file or directory

---

https://ci.eclipse.org/openj9/job/Build_JDK11_ppc64le_linux_jit_Personal/1/console
https://ci.eclipse.org/openj9/job/Build_JDK8_ppc64le_linux_jit_Personal/4/console
https://ci.eclipse.org/openj9/job/Test_openjdk11_j9_sanity.functional_ppc64le_linux_Personal/421/console
https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_ppc64le_linux_Personal/434/console

[pshipton]

Does JIT server have a hard requirement to use 1.1.1? I think you'll be ruling out running JIT server on some Linux OSes, which don't provide openssl 1.1.1 libraries.

The openssl support in the class libraries works with 1.0.x or 1.1.x, even though we build it against 1.1.1.

@mpirvu

[pshipton]

fyi the JCL support for loading libcrypto
https://github.com/ibmruntimes/openj9-openjdk-jdk11/pull/196/files

[mpirvu]

JITServer can use either 1.0.x or 1.1.x. As I understand, the problem is that if we build with 1.1.x, at runtime it needs to find 1.1.x installed on the system.

[pshipton]

@mpirvu please take a look at the JCL code and how it works to load what is available, the JIT should be able to do the same.
https://github.com/ibmruntimes/openj9-openjdk-jdk11/pull/196/files

[AdamBrousseau]

if we build with 1.1.x, at runtime it needs to find 1.1.x installed on the system.

Then should we not be building with the lowest supported version? Similar to how glibc works.

[pshipton]

@AdamBrousseau Marius didn't write it down here, but the plan is to modify the code to load and link dynamically with libssl, similar to what the JCL code does for libcyrpto.

Then should we not be building with the lowest supported version? Similar to how glibc works.

It doesn't work the same way. As long as the APIs are the same between versions, it doesn't matter which version is used to compile against.

[mpirvu]

It doesn't work the same way. As long as the APIs are the same between versions, it doesn't matter which version is used to compile against.

Correct. We are working towards an implementation that would eliminate the above restriction and hope to have a prototype by tomorrow EOD for review.

[a7ehuo]

@pshipton @AdamBrousseau @mpirvu I’m experimenting JITServe code to dynamically load OpenSSL. The proto code is in #8145. I found a few issues when replacing the OpenSSL macros:

1. If the code is built against with 1.1.0, SSLv23_client_method() is defined as TLS_client_method.
   If the target machine has 1.0.2 installed, TLS_client_method doesn’t exist. Replacing APIs based on versions might get around this issue but there are subversions such as 1.0.2g, 1.0.2t, etc, it's hard to track down each macro on which subversion has the definition and which one doesn't.

2. Another issue is that if we replace an external API X (macro) with an internal Y (function), down the road, if X’s implementation changes, we won’t be able to get that. There are about 12 macros used in JITServer at least.

3. There could be a chain of macros/calls inside an OpenSSL macro: such as: sk_X509_INFO_pop_free => SKM_sk_pop_free => sk_pop_free. Different versions of OpenSSL could have different chains of macros/calls. It also means we are starting to hard code OpenSSL internal implementations into our code ...

With the issues found so far, just wondering if we should consider the following two alternatives:

1. As @AdamBrousseau suggested, shouldn’t we build with the lowest supported OpenSSL version?
2. Add (back) enable/disable building OpenSSL flag in JITServer.

[pshipton]

If the code is compiled against 1.0.2, does it still work on 1.1.1? There could be similar problems to what you describe doing it that way as well.

1.0.2[a-z] are security updates, I wouldn't expect any incompatibilities, but who knows.

I'll create a PR and try compiling against 1.0.2. I'm not sure if this has been tried for the JCL support so I can't say if it works or not.
#8155

[a7ehuo]

If the code is compiled against 1.0.2, does it still work on 1.1.1?

I'd assume it should be backward compatible but not 100% sure. Looks like there are failures in building 1.0.2 OpenSSL in #8155. Should we consider option 2?

2. Add (back) enable/disable building OpenSSL flag in JITServer.

There was an env flag JITSERVER_ENABLE_SSL that guarded OpenSSL code in JITServer, which is removed recently in #8039.

[pshipton]

I'll defer to @mpirvu about option 2.

For supporting 1.0.x and 1.1.1, are there APIs which can be used in place of the macros? I'm not that familiar with the JCL crypto implementation, but it loads all the APIs dynamically and manages to work with both versions. Perhaps it's a simpler usage though.
https://github.com/ibmruntimes/openj9-openjdk-jdk8/blob/b2a1abf9261a42a96ca250812fe1004e40440855/closed/adds/jdk/src/share/native/jdk/crypto/jniprovider/NativeCrypto.c#L217-L279

[mpirvu]

The behavior that I would like to see is the following:

1. openj9 with default options should run regardless of the OpenSSL version installed on the machine
2. openj9 in client mode (-XX:+UseJITServer) but with no encryption should run regardless of the OpenSSL version installed
3. openj9 in client mode (-XX:+UseJITserver) WITH encryption specified on the command line should run with OpenSSL 1.1.x. in this case I am ok with the restriction that older OpenSSl will not be accepted.

[mpirvu]

Add (back) enable/disable building OpenSSL flag in JITServer.

The disadvantage of this solution is that the default is going to be to build without OpenSSl and this is what we are going to force on the liberty/openshift users. Anybody wanting to use JITServer with encryption would have to build their own openj9 SDK

[a7ehuo]

#8182 eliminates the dependency on the version of the OpenSSL during the build time. Therefore close this issue.