Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CORDA-2050 Upgrade Corda to Java 11 (compatibility mode) #5356

Merged
merged 206 commits into from Sep 18, 2019

Conversation

@josecoll
Copy link
Contributor

josecoll commented Aug 6, 2019

Summary

This PR contains the first phase migration of Corda to build and execute on a Java 11 VM.
The codebase maintains 100% source code compatibility with Java 8 (ie. changes do not use any of the new Java 9+ JDK libraries). This compatibility mode entails all code executes within the global UNNAMED module space (eg. the equivalent of the Java 8 global classpath), so there is no effective true module isolation per se.

We use the Gradle sourceCompatibility and targetCompatibility flags to specify compatibility modes, such that:

  • sourceCompatibility = VERSION_1_8
  • targetCompatibility = VERSION_1_8 or VERSION_11 (dependent on JDK used to execute gradle)

Note: a specific patched version of Java 11 is being used due to an officially reported OpenJDK bug related to cryptographic libraries used by Corda. The official OpenJDK 11 patch is expected on 15th October 2019.

Please see JDK 11 Migration for further background and detailed migration steps completed.

Dependencies

The Java 11 build of Corda requires:

  • an upgraded version of Quasar built for Java 11.
    The version of Quasar to use may be driven from system properties (the default is to use the quasarVersion specified in constants.properties)
    For Java 11, these must be set as follows: -DquasarVersion=0.8.0 -DquasarClassifier
  • Corda Gradle plugins upgraded to use aforementioned version of Quasar - see CORDA-2924
  • Gradle 5: existing master codebase has already been upgraded to use this - see CORDA-2893 (and an associated task to upgrade the capsule plugin to use Gradle 5: CORDA-2895).
  • a patched version of the official OpenJDK 11 (note: we initially patched the OpenJDK in-house, subsequently reported and obtained an official patch from our JDK supplier, and are now awaiting an official patch due on or around the 15th October 2019 in line with Oracle's Critical Patch Update quarterly release). The OpenJDK reported bug in question can be found here.

Constraints

The Kotlin compiler does not support usage of Java 9 module directives (eg. to specify that private internal JDK packages may still be used in the global UNNAMED module space). It was necessary to re-write the SharedMemoryIncremental Kotlin class in Java to be able to specify the use of a private internal package at compile time as follows:

        if (JavaVersion.current() == JavaVersion.VERSION_11)
            options.compilerArgs = [
                    '--add-exports', 'java.base/sun.nio.ch=ALL-UNNAMED'
            ]

Corda Java 11 docker images included in this PR (eg. see Dockerfile11) temporarily use an in-house dockerized patched JDK 11 image published to an internal R3 docker registry.

DJVM and SGX compatibility

The modules core, core-deterministic, serialization, serialization-deterministic strictly target Java 8 JVM bytecode to support deployment and execution of contract verification logic in an SGX hosted deterministic JVM.

Testing

  • All Unit, Integration and Smoke tests have been executed in a Continuous Integration environment (as per Corda master).
  • Basic samples (Bank Of Corda) and compatibility tests (eg. a transacting Behave scenario combining Corda nodes built with Java 8 and 11).
  • Basic performance testing using the R3 performance cluster (no specific Java 11 JVM / GC tuning applied)
  • Specific tools and samples require additional testing.

Outstanding issues

Further work

This PR is part of a larger Epic described here, and which includes future activities such as:

  • true (Java 9) modularisation (inclusion of module-info.java and explicit imports/exports)
  • adoption of new Java 11 JDK libraries and language features - CORDA-2051
  • use JLink to create optimised runtime images - CORDA-2754
  • improve startup and footprint using Application Class Data Sharing (AppCDS) - CORDA-2755
josecoll and others added 30 commits Apr 9, 2019
…anges to relevant modules (explorer, demobench, client/jfx).
…only use for JDK 1.8 due to Proguard version not supporting JDK 11)
…ts (re-instate with module add-exports directive)
…4993)

The test is currently disabled till we move to Java 11 (or beyond) when TLS 1.3 becomes available as part of JDK.
Local testing been performed with Open JDK 12 (12+33) and the test is passing.
…t test (was causing ASM compilation failure).
…t test (was causing ASM compilation failure).
…assloading error (REVISIT)

* What went wrong:
Execution failed for task ':core:test'.
> failed to read class file /Users/josecoll/IdeaProjects/corda-jdk11/core/build/classes/java/test/net/corda/core/flows/FlowsInJavaTest$PrimitiveReceiveFlow.class
…assloading error (REVISIT)

* What went wrong:
Caused by: org.gradle.api.GradleException: failed to read class file /Users/josecoll/IdeaProjects/corda-jdk11/node/build/classes/java/test/net/corda/node/services/events/FlowLogicRefFromJavaTest$JavaNoArgFlowLogic.class
josecoll added 20 commits Sep 10, 2019
…signed shrunk contracts jar)
…dule (following tests decoupling from node-api).
…ure in "KotlinIntegrationTestingTutorial.alice bob cash exchange example"
…n CI (many introduced since Java 9).
…pe (required by Artemis JAAS login).
…private internal JDK packages)
Copy link
Contributor

rick-r3 left a comment

Can you work with @r3domfox team to create a PR gate for JDK 11 compatibility ? I guess up to them whether they run per PR or we have something like the Windows build, but the latter isn't good at keeping things green. Create a JIRA to track if we don't have one.

@josecoll

This comment has been minimized.

Copy link
Contributor Author

josecoll commented Sep 17, 2019

Yes - I created a JIRA to create a JDK11 compilation-only check gate: https://r3-cev.atlassian.net/browse/DEVOPS-1367
I propose we only run tests overnight as part of a snapshot build.

@josecoll josecoll merged commit 3fafbe5 into release/os/4.3 Sep 18, 2019
6 checks passed
6 checks passed
API Stability (Pull Requests) TeamCity build finished
Details
API Stability - Plugin Test WIP (Pull Requests) TeamCity build finished
Details
Code Checks (Pull Requests) TeamCity build finished
Details
Integration Tests (Pull Requests) TeamCity build finished
Details
Unit Tests (Pull Requests) TeamCity build finished
Details
continuous-integration/jenkins/pr-merge This commit looks good
Details
roastario added a commit that referenced this pull request Sep 20, 2019
* NOTICK: Corda 4.3-RC01

Created first release candidate of Corda 4.3 - RC01.

* CORDA-3141: Add GracefulReconnect callbacks which allow logic to be performed when RPC disconnects unexpectedly (#5430)

Also removed potential for growing stack trace on reconnects.

* CORDA-2050 Upgrade Corda to Java 11 (compatibility mode) (#5356)

Upgrade Corda to run with Java 11 (compatibility mode) - see #5356

* ENT-4198 Adding legal text

Signed-off-by: Ed Prosser <edward.prosser@r3.com>

* TM-29 new baseline for 4.3 since new debt has been added with the last few commits (#5487)

* TM-23 compileAll task to compile all code (#5490)

* Add simple compileAll task to be used by warning check

* lazy configure compileAll

* TM-32 Merge OS 4.3 into 4.4

* TM-32 fixed detekt issue

* Downgrade Dokka back to 0.9.17 due to failing docs_builder.

* add ability to group test types together (#5459)

* add ability to group test types together

* add ability to specify podCount for use in parallel testing

* remove compiler xml

* add Jenkinsfile to enable scanning

* trigger build

* add ability to specify what docker tag to use from outside of the build

* fix docker work dir

* fix pipeline syntax issues

* use environment rather than `def`

* move agent restrictor outside of stages block

* use steps block

* more pipeline syntax fixes

* even more pipeline syntax fixes

* even more pipeline syntax fixes

* add kubenetize as property to image build

* move clear of docker image to end of build rather than start to prevent colocated builds

* escape dollar on docker image remove command

* attempt to kill all existing jobs

* fix compile issue due to killall_jobs

* fix compile issue due to killall_jobs pt2

* fix spelling

* make all variables environment variables

* add logic to delete images locally after pushing

* wrap testing phase with try / finally so that junit reports are always evaluated

* change the behaviour around post build actions

* break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image

* prepend registry name to provided tag

* allow tasks to specify whether they wish to stream output from containers

* add timestamps directive to Jenkinsfile to have timing info on output

* make KubesTest resilient against transient pod failures in k8s

* increase CPU request

* add logic to allow specifying container resource requests

* attempt to run unit and integration tests in parallel

* change unit tests to use 3 cores to allow co-location on 8c machines

* join grouped tests together to give pod meaningful name

* add step to renew token with GKE

* change renew step to use pods instead of nodes

* fix bug where memory request is not correctly passed to pod

* disable unit tests for now

* [CORDA-2368] Added exception handling for missing files that displays appropriate messages rather than defaulting to file names. (#5472)

* NOTIK Minor adjustments to Detekt rules to reflect current working practises  (#5498)

* Minor adjustments to rules to reflect current working practises (including IntelliJ code style alignment)

* Adjust another rule in line with existing code style.

* rebaseline with changed detekt ruleset

* rebaseline with NodeStartup changes
josecoll added a commit that referenced this pull request Sep 26, 2019
Upgrade Corda to run with Java 11 (compatibility mode) - see #5356
fenryka added a commit that referenced this pull request Oct 19, 2019
* NOTICK: Corda 4.3-RC01

Created first release candidate of Corda 4.3 - RC01.

* CORDA-3141: Add GracefulReconnect callbacks which allow logic to be performed when RPC disconnects unexpectedly (#5430)

Also removed potential for growing stack trace on reconnects.

* CORDA-2050 Upgrade Corda to Java 11 (compatibility mode) (#5356)

Upgrade Corda to run with Java 11 (compatibility mode) - see #5356

* ENT-4198 Adding legal text

Signed-off-by: Ed Prosser <edward.prosser@r3.com>

* TM-29 new baseline for 4.3 since new debt has been added with the last few commits (#5487)

* TM-23 compileAll task to compile all code (#5490)

* Add simple compileAll task to be used by warning check

* lazy configure compileAll

* TM-32 Merge OS 4.3 into 4.4

* TM-32 fixed detekt issue

* Downgrade Dokka back to 0.9.17 due to failing docs_builder.

* add ability to group test types together (#5459)

* add ability to group test types together

* add ability to specify podCount for use in parallel testing

* remove compiler xml

* add Jenkinsfile to enable scanning

* trigger build

* add ability to specify what docker tag to use from outside of the build

* fix docker work dir

* fix pipeline syntax issues

* use environment rather than `def`

* move agent restrictor outside of stages block

* use steps block

* more pipeline syntax fixes

* even more pipeline syntax fixes

* even more pipeline syntax fixes

* add kubenetize as property to image build

* move clear of docker image to end of build rather than start to prevent colocated builds

* escape dollar on docker image remove command

* attempt to kill all existing jobs

* fix compile issue due to killall_jobs

* fix compile issue due to killall_jobs pt2

* fix spelling

* make all variables environment variables

* add logic to delete images locally after pushing

* wrap testing phase with try / finally so that junit reports are always evaluated

* change the behaviour around post build actions

* break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image

* prepend registry name to provided tag

* allow tasks to specify whether they wish to stream output from containers

* add timestamps directive to Jenkinsfile to have timing info on output

* make KubesTest resilient against transient pod failures in k8s

* increase CPU request

* add logic to allow specifying container resource requests

* attempt to run unit and integration tests in parallel

* change unit tests to use 3 cores to allow co-location on 8c machines

* join grouped tests together to give pod meaningful name

* add step to renew token with GKE

* change renew step to use pods instead of nodes

* fix bug where memory request is not correctly passed to pod

* disable unit tests for now

* [CORDA-2368] Added exception handling for missing files that displays appropriate messages rather than defaulting to file names. (#5472)

* NOTIK Minor adjustments to Detekt rules to reflect current working practises  (#5498)

* Minor adjustments to rules to reflect current working practises (including IntelliJ code style alignment)

* Adjust another rule in line with existing code style.

* rebaseline with changed detekt ruleset

* rebaseline with NodeStartup changes
fenryka added a commit that referenced this pull request Oct 20, 2019
* NOTICK: Corda 4.3-RC01

Created first release candidate of Corda 4.3 - RC01.

* CORDA-3141: Add GracefulReconnect callbacks which allow logic to be performed when RPC disconnects unexpectedly (#5430)

Also removed potential for growing stack trace on reconnects.

* CORDA-2050 Upgrade Corda to Java 11 (compatibility mode) (#5356)

Upgrade Corda to run with Java 11 (compatibility mode) - see #5356

* ENT-4198 Adding legal text

Signed-off-by: Ed Prosser <edward.prosser@r3.com>

* TM-29 new baseline for 4.3 since new debt has been added with the last few commits (#5487)

* TM-23 compileAll task to compile all code (#5490)

* Add simple compileAll task to be used by warning check

* lazy configure compileAll

* TM-32 Merge OS 4.3 into 4.4

* TM-32 fixed detekt issue

* Downgrade Dokka back to 0.9.17 due to failing docs_builder.

* add ability to group test types together (#5459)

* add ability to group test types together

* add ability to specify podCount for use in parallel testing

* remove compiler xml

* add Jenkinsfile to enable scanning

* trigger build

* add ability to specify what docker tag to use from outside of the build

* fix docker work dir

* fix pipeline syntax issues

* use environment rather than `def`

* move agent restrictor outside of stages block

* use steps block

* more pipeline syntax fixes

* even more pipeline syntax fixes

* even more pipeline syntax fixes

* add kubenetize as property to image build

* move clear of docker image to end of build rather than start to prevent colocated builds

* escape dollar on docker image remove command

* attempt to kill all existing jobs

* fix compile issue due to killall_jobs

* fix compile issue due to killall_jobs pt2

* fix spelling

* make all variables environment variables

* add logic to delete images locally after pushing

* wrap testing phase with try / finally so that junit reports are always evaluated

* change the behaviour around post build actions

* break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image

* prepend registry name to provided tag

* allow tasks to specify whether they wish to stream output from containers

* add timestamps directive to Jenkinsfile to have timing info on output

* make KubesTest resilient against transient pod failures in k8s

* increase CPU request

* add logic to allow specifying container resource requests

* attempt to run unit and integration tests in parallel

* change unit tests to use 3 cores to allow co-location on 8c machines

* join grouped tests together to give pod meaningful name

* add step to renew token with GKE

* change renew step to use pods instead of nodes

* fix bug where memory request is not correctly passed to pod

* disable unit tests for now

* [CORDA-2368] Added exception handling for missing files that displays appropriate messages rather than defaulting to file names. (#5472)

* NOTIK Minor adjustments to Detekt rules to reflect current working practises  (#5498)

* Minor adjustments to rules to reflect current working practises (including IntelliJ code style alignment)

* Adjust another rule in line with existing code style.

* rebaseline with changed detekt ruleset

* rebaseline with NodeStartup changes
lankydan added a commit that referenced this pull request Nov 1, 2019
…ade Corda to run with Java 11 (compatibility mode) - see #5356 - 3fafbe5

Reapply change that was lost during merge - Adjust resolution of byteman jar to use java 11 compatible mechanism. - a107709

Manual cherry pick of these changes (a107709 + 3fafbe5)
lankydan added a commit that referenced this pull request Nov 1, 2019
…ade Corda to run with Java 11 (compatibility mode) - see #5356 - 3fafbe5

Reapply change that was lost during merge - Adjust resolution of byteman jar to use java 11 compatible mechanism. - a107709

Manual cherry pick of these changes (a107709 + 3fafbe5)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.