* CORDA-2876: Migrate DJVM serialization modules into Corda.

* Pre-generate Corda classes for DJVM sandbox when node boots in production mode.
* Ensure that all DJVM test contract CorDapps are signed.
* Test examining attachments within DJVM sandbox.
* Test Contract.verify() using cryptographic verify function.
* Add test cases for more non-determinism in Contract.verify().
* Update node-driver to support testing nodes with DJVM support.
* Modify Node to allow alternative DJVM configurations for testing.
* Refactor DeterministicVerifierFactoryService for default use-case.
* Small whitespace and code-style refactors.
* Create and activate a DJVM execution profile for the Node.
* Revert making Verifier implement AutoCloseable.
* Allow the node to cache sandboxed Corda byte-code for reuse.
* Use updated Quasar agent that knows not to touch DJVM classloaders.
* Fix Quasar's package exclusions globs for DJVM.
* Deserialise LedgerTransaction into the sandbox for Contract.verify().
* Add the DJVM's serialisation modules to the Corda node.
* Update the node for the latest DJVM API, and preserve the ConstructorForDeserialization annotation on user contract classes.
* Add corda-dev to repositories while DJVM is SNAPSHOT.
* Migrate DJVM specialisation into AbstractNode's ServiceHubInternalImpl.
* Exclude sandbox.** and shaded djvm.** classes from Quasar agent.
* Add the corda-dev repository to :node for the deterministic runtime.
* Turn Verifier into an abstract base class that is specialised by BasicVerifier and DeterministicVerifier.
* Add the Corda deterministic libraries to the Node, and split the DJVM sandbox across two SandboxClassLoader instances.
* Add DJVM to contract verification path inside Corda Node.
* Minor lambda simplifications and removing unused import.
* CORDA-2871: Remove @CordaSerializable from LedgerTransaction.
* CORDA-2871: Add a callback to ServicesForResolution to allow the Node to modify a LedgerTransaction object.
* CORDA-2871: Refactor the contract verification code into a separate class,
 and allow LedgerTransaction to choose different Verifier objects.
* Update DJVM to use Corda 4.4-SNAPSHOT. (#95)
* CORDA-3330: Allow DJVM to preload / pregenerate classes from selected jars. (#92)
* Add support for SourceClassLoader.getResources() to DJVM.
* Allow a SandboxConfiguration to preload sandbox byte-code for all classes inside jars containing META-INF/DJVM-preload.
* CORDA-3309: Remove explicit try-catch in favour of UncaughtExceptionHandler. (#91)
* CORDA-3309: Install UncaughtExceptionHandler for DJVM tasks. (#88)
* Fix tests broken by Windows line endings. (#82)
* CORDA-3292: Reimplement ExecutionProfile as a data class. (#80)
* CORDA-2877: Refactor how we create child SandboxConfiguration objects. (#76)
* CORDA-2877: Load bytecode from a persistent cache to prevent repeated rewriting. (#75)
* Refactor byte-code cache to SandboxConfiguration instead of AnalysisConfiguration. We cannot "mix and match" byte-code generated by different sets of rules.
* CORDA-3137: Enhance annotation handling so that we can allow some annotations to be mapped into the sandbox without also needing to be stitched. (#72)
* CORDA-2871: Minor cosmetic fixes. (#69)
* CORDA-3218: Align DJVM with internal Corda Serialisation API. (#68)
* Ensure we get the latest SNAPSHOT of the serialisation code.
* CORDA-2871: Refactor SourceClassLoader to define source classes. (#66)
* Rewrite SourceClassLoader to support parent/child relationships.
* Revert catching TypNotPresebtException - it was a symptom of a bigger problem.
* Remove AutoCloseable from AnalysisConfiguration and SourceClassLoader.
* SourceClassLoader.getResource() must delegate to its parent first.
* CORDA-2871: Ensure ClassLoader.loadClass() throws ClassNotFoundException for all cases where the class cannot be found. (#64)
* CORDA-2871: Modify sandbox tasks to implement both java.Function and sandbox.Function (#62)
* Make TaskExecutors implement BiFunction to make them composable.
* Create ImportTask to wrap a java.Function inside a sandbox.Function.
* Add createExecutor() and createRawExecutor() APIs to SandboxClassLoader.
* Update serialization to use SandboxClassLoader.toSandboxClass().
* Remove a layer of lambdas from the serialisation code.
* Update SandboxExecutor and SandboxRawExecutor.
* Rename Executor to TaskFactory.
* Rename dangling executor -> taskFactory.
* CORDA-2871: Sanity fixes! (#63)
* Improve message for SandboxClassLoadingException.
* Fix serialisation API for using sandboxed environment.
* CORDA-3174: Extend serialisation to include InputStream and OpaqueBytesSubSequence. (#60)
* Update DJVM Example project for serialisation.
* Add serializers for InputStream and OpaqueBytesSubSequence.
* Support ZIP Inflater and CRC32 inside the sandbox.
* Allow the DJVM to wrap java.io.InputStream as sandbox.java.io.InputStream.
* Configure tests also to preserve @DeprecatedConstructorForDeserialization.
* CORDA-3174: Implement Corda serialization modules. (#59)
* Create DJVM serialization modules.
* Create test cases for Array<T>, List<T> and List<Array<T>>.
* Refactor SandboxPrimiveSerializer for all primitive types.
* Implement SandboxCollectionSerializer to support Collection types.
* Implement SandboxMapSerializer to support Map types.
* Attempt to fix infinite loop when computing Collection and Map fingerprints.
* Apply special handling when deserialising sandbox.java.lang.Character.
* Remap Java primitive types to sandbox Java object types to deter evolution.
* Use Class.getPackage().getName() to determine sandbox package name.
* Implement SandboxEnumSerializer to support Enum types.
* Implement SandboxPublicKeySerializer to support Java security keys.
* Add serialization projects to the composite example project.
* Implement serializers for BigInteger, BigDecimal, Currency and StringBuffer.
* Test that deserialising does not instantiate the untrusted user classes.
* Implement serializers for java.time.* types.
* Add serialiser for BitSet - currently disabled until BitSet itself is supported.
* Add serialisers for EnumSet and Class.
* Include support for EnumMap in the SandboxMapSerializer.
* Ensure the DJVM Example project's tests preserve @CordaSerializable.
* Add support for UUID as a primitive type.
* Use common abortReadOnly() method for declaring serialization as unsupported.
* Streamline the API for deserialising into the sandbox.
* Add preliminary support for deserialising X.509 certificates.
* Implement serializer for java.util.Optional.
* Refactor configuration of the sandbox serialization scheme.
* Add tests for deserialising arrays of basic types.
* Include method annotations in annotation stitching. This ensures that `@ConstructorForDeserialization` is not dropped.
* Enable test for SandboxBitSetSerializer.
* Enable tests for X.509 serializers.
* Implement serializers for ProtonJ primitive types.
* Serialize java.util.Date as a primitive type.
* Add the bintray Gradle plugin to the serialisation modules.
* Do not publish serialisation modules - they will become part of Corda itself.

build.gradle

@@ -76,6 +76,7 @@ buildscript {
     ext.disruptor_version = constants.getProperty("disruptorVersion")
     ext.metrics_version = constants.getProperty("metricsVersion")
     ext.metrics_new_relic_version = constants.getProperty("metricsNewRelicVersion")
+    ext.djvm_version = constants.getProperty("djvmVersion")
     ext.okhttp_version = '3.14.2'
     ext.netty_version = '4.1.22.Final'
     ext.typesafe_config_version = constants.getProperty("typesafeConfigVersion")
@@ -335,6 +336,7 @@ allprojects {
         jcenter()
         maven { url "$artifactory_contextUrl/corda-dependencies" }
         maven { url 'https://repo.gradle.org/gradle/libs-releases' }
+        maven { url "$artifactory_contextUrl/corda-dev" }
     }
 
     configurations {
@@ -481,11 +483,13 @@ bintrayConfig {
             'corda-core',
             'corda-core-deterministic',
             'corda-deterministic-verifier',
+            'corda-deserializers-djvm',
             'corda',
             'corda-finance-workflows',
             'corda-finance-contracts',
             'corda-node',
             'corda-node-api',
+            'corda-node-djvm',
             'corda-test-common',
             'corda-test-utils',
             'corda-test-db',
@@ -498,6 +502,7 @@ bintrayConfig {
             'corda-tools-shell-cli',
             'corda-serialization',
             'corda-serialization-deterministic',
+            'corda-serialization-djvm',
             'corda-tools-blob-inspector',
             'corda-tools-explorer',
             'corda-tools-network-bootstrapper',

constants.properties

@@ -13,11 +13,10 @@ java8MinUpdateVersion=171
 platformVersion=5
 guavaVersion=28.0-jre
 # Quasar version to use with Java 8:
-quasarVersion=0.7.10
+quasarVersion=0.7.12_r3
 quasarClassifier=jdk8
 # Quasar version to use with Java 11:
-quasarVersion11=0.8.0
-# Specify a classifier for Java 11 built artifacts
+quasarVersion11=0.8.0_r3_rc1
 jdkClassifier11=jdk11
 proguardVersion=6.1.1
 bouncycastleVersion=1.60
@@ -30,6 +29,7 @@ snakeYamlVersion=1.19
 caffeineVersion=2.7.0
 metricsVersion=4.1.0
 metricsNewRelicVersion=1.1.1
+djvmVersion=1.0-SNAPSHOT
 openSourceBranch=https://github.com/corda/corda/blob/master
 openSourceSamplesBranch=https://github.com/corda/samples/blob/master
 jolokiaAgentVersion=1.6.1

core-deterministic/build.gradle

@@ -56,6 +56,7 @@ task patchCore(type: Zip, dependsOn: coreJarTask) {
     archiveExtension = 'jar'
 
     from(compileKotlin)
+    from(processResources)
     from(zipTree(originalJar)) {
         exclude 'net/corda/core/internal/*ToggleField*.class'
         exclude 'net/corda/core/serialization/*SerializationFactory*.class'

core-tests/src/test/kotlin/net/corda/coretests/crypto/PartialMerkleTreeTest.kt

@@ -8,7 +8,6 @@ import net.corda.core.crypto.*
 import net.corda.core.crypto.SecureHash.Companion.zeroHash
 import net.corda.core.identity.CordaX500Name
 import net.corda.core.identity.Party
-import net.corda.core.internal.accessLeafIndex
 import net.corda.core.node.NotaryInfo
 import net.corda.core.node.services.IdentityService
 import net.corda.core.serialization.deserialize
@@ -307,37 +306,37 @@ class PartialMerkleTreeTest {
 
         val pmt = PartialMerkleTree.build(merkleTree, listOf<SecureHash>(SecureHash.sha256("1"), SecureHash.sha256("5"), SecureHash.sha256("0"), SecureHash.sha256("19")))
         // First leaf.
-        assertEquals(0, pmt.accessLeafIndex(SecureHash.sha256("0")))
+        assertEquals(0, pmt.leafIndex(SecureHash.sha256("0")))
         // Second leaf.
-        assertEquals(1, pmt.accessLeafIndex(SecureHash.sha256("1")))
+        assertEquals(1, pmt.leafIndex(SecureHash.sha256("1")))
         // A random leaf.
-        assertEquals(5, pmt.accessLeafIndex(SecureHash.sha256("5")))
+        assertEquals(5, pmt.leafIndex(SecureHash.sha256("5")))
         // The last leaf.
-        assertEquals(19, pmt.accessLeafIndex(SecureHash.sha256("19")))
+        assertEquals(19, pmt.leafIndex(SecureHash.sha256("19")))
         // The provided hash is not in the tree.
-        assertFailsWith<MerkleTreeException> { pmt.accessLeafIndex(SecureHash.sha256("10")) }
+        assertFailsWith<MerkleTreeException> { pmt.leafIndex(SecureHash.sha256("10")) }
         // The provided hash is not in the tree (using a leaf that didn't exist in the original Merkle tree).
-        assertFailsWith<MerkleTreeException> { pmt.accessLeafIndex(SecureHash.sha256("30")) }
+        assertFailsWith<MerkleTreeException> { pmt.leafIndex(SecureHash.sha256("30")) }
 
         val pmtFirstElementOnly = PartialMerkleTree.build(merkleTree, listOf<SecureHash>(SecureHash.sha256("0")))
-        assertEquals(0, pmtFirstElementOnly.accessLeafIndex(SecureHash.sha256("0")))
+        assertEquals(0, pmtFirstElementOnly.leafIndex(SecureHash.sha256("0")))
         // The provided hash is not in the tree.
-        assertFailsWith<MerkleTreeException> { pmtFirstElementOnly.accessLeafIndex(SecureHash.sha256("10")) }
+        assertFailsWith<MerkleTreeException> { pmtFirstElementOnly.leafIndex(SecureHash.sha256("10")) }
 
         val pmtLastElementOnly = PartialMerkleTree.build(merkleTree, listOf<SecureHash>(SecureHash.sha256("19")))
-        assertEquals(19, pmtLastElementOnly.accessLeafIndex(SecureHash.sha256("19")))
+        assertEquals(19, pmtLastElementOnly.leafIndex(SecureHash.sha256("19")))
         // The provided hash is not in the tree.
-        assertFailsWith<MerkleTreeException> { pmtLastElementOnly.accessLeafIndex(SecureHash.sha256("10")) }
+        assertFailsWith<MerkleTreeException> { pmtLastElementOnly.leafIndex(SecureHash.sha256("10")) }
 
         val pmtOneElement = PartialMerkleTree.build(merkleTree, listOf<SecureHash>(SecureHash.sha256("5")))
-        assertEquals(5, pmtOneElement.accessLeafIndex(SecureHash.sha256("5")))
+        assertEquals(5, pmtOneElement.leafIndex(SecureHash.sha256("5")))
         // The provided hash is not in the tree.
-        assertFailsWith<MerkleTreeException> { pmtOneElement.accessLeafIndex(SecureHash.sha256("10")) }
+        assertFailsWith<MerkleTreeException> { pmtOneElement.leafIndex(SecureHash.sha256("10")) }
 
         val pmtAllIncluded = PartialMerkleTree.build(merkleTree, sampleLeaves)
-        for (i in 0..19) assertEquals(i, pmtAllIncluded.accessLeafIndex(SecureHash.sha256(i.toString())))
+        for (i in 0..19) assertEquals(i, pmtAllIncluded.leafIndex(SecureHash.sha256(i.toString())))
         // The provided hash is not in the tree (using a leaf that didn't exist in the original Merkle tree).
-        assertFailsWith<MerkleTreeException> { pmtAllIncluded.accessLeafIndex(SecureHash.sha256("30")) }
+        assertFailsWith<MerkleTreeException> { pmtAllIncluded.leafIndex(SecureHash.sha256("30")) }
     }
 
     @Test

core/src/main/kotlin/net/corda/core/crypto/PartialMerkleTree.kt

@@ -1,6 +1,7 @@
 package net.corda.core.crypto
 
 import net.corda.core.CordaException
+import net.corda.core.CordaInternal
 import net.corda.core.KeepForDJVM
 import net.corda.core.crypto.SecureHash.Companion.zeroHash
 import net.corda.core.serialization.CordaSerializable
@@ -169,8 +170,9 @@ class PartialMerkleTree(val root: PartialTree) {
      * @return leaf-index of this component (starting from zero).
      * @throws MerkleTreeException if the provided hash is not in the tree.
      */
+    @CordaInternal
     @Throws(MerkleTreeException::class)
-    internal fun leafIndex(leaf: SecureHash): Int {
+    fun leafIndex(leaf: SecureHash): Int {
         // Special handling if the tree consists of one node only.
         if (root is PartialTree.IncludedLeaf && root.hash == leaf) return 0
         val flagPath = mutableListOf<Boolean>()

core/src/main/kotlin/net/corda/core/internal/TransactionVerifierServiceInternal.kt

@@ -1,12 +1,15 @@
 package net.corda.core.internal
 
 import net.corda.core.DeleteForDJVM
+import net.corda.core.KeepForDJVM
 import net.corda.core.concurrent.CordaFuture
 import net.corda.core.contracts.*
 import net.corda.core.contracts.TransactionVerificationException.TransactionContractConflictException
+import net.corda.core.crypto.SecureHash
 import net.corda.core.internal.rules.StateContractValidationEnforcementRule
 import net.corda.core.transactions.LedgerTransaction
 import net.corda.core.utilities.contextLogger
+import java.util.function.Function
 
 @DeleteForDJVM
 interface TransactionVerifierServiceInternal {
@@ -25,15 +28,13 @@ fun LedgerTransaction.prepareVerify(extraAttachments: List<Attachment>) = this.i
 /**
  * Because we create a separate [LedgerTransaction] onto which we need to perform verification, it becomes important we don't verify the
  * wrong object instance. This class helps avoid that.
- *
- * @param inputVersions A map linking each contract class name to the advertised version of the JAR that defines it. Used for downgrade protection.
  */
-class Verifier(val ltx: LedgerTransaction, private val transactionClassLoader: ClassLoader) {
+abstract class Verifier(val ltx: LedgerTransaction, protected val transactionClassLoader: ClassLoader) {
     private val inputStates: List<TransactionState<*>> = ltx.inputs.map { it.state }
     private val allStates: List<TransactionState<*>> = inputStates + ltx.references.map { it.state } + ltx.outputs
 
     companion object {
-        private val logger = contextLogger()
+        val logger = contextLogger()
     }
 
     /**
@@ -140,7 +141,7 @@ class Verifier(val ltx: LedgerTransaction, private val transactionClassLoader: C
                 .withIndex()
                 .filter { it.value.encumbrance != null }
                 .map { Pair(it.index, it.value.encumbrance!!) }
-        if (!statesAndEncumbrance.isEmpty()) {
+        if (statesAndEncumbrance.isNotEmpty()) {
             checkBidirectionalOutputEncumbrances(statesAndEncumbrance)
             checkNotariesOutputEncumbrance(statesAndEncumbrance)
         }
@@ -343,28 +344,55 @@ class Verifier(val ltx: LedgerTransaction, private val transactionClassLoader: C
         }
     }
 
+    /**
+     * Placeholder function for the contract verification logic.
+     */
+    abstract fun verifyContracts()
+}
+
+class BasicVerifier(ltx: LedgerTransaction, transactionClassLoader: ClassLoader) : Verifier(ltx, transactionClassLoader) {
     /**
      * Check the transaction is contract-valid by running the verify() for each input and output state contract.
      * If any contract fails to verify, the whole transaction is considered to be invalid.
      *
      * Note: Reference states are not verified.
      */
-    private fun verifyContracts() {
+    override fun verifyContracts() {
+        try {
+            ContractVerifier(transactionClassLoader).apply(ltx)
+        } catch (e: TransactionVerificationException.ContractRejection) {
+            logger.error("Error validating transaction ${ltx.id}.", e.cause)
+            throw e
+        }
+    }
+}
 
-        // Loads the contract class from the transactionClassLoader.
-        fun contractClassFor(className: ContractClassName) = try {
-            transactionClassLoader.loadClass(className).asSubclass(Contract::class.java)
+/**
+ * Verify all of the contracts on the given [LedgerTransaction].
+ */
+@KeepForDJVM
+class ContractVerifier(private val transactionClassLoader: ClassLoader) : Function<LedgerTransaction, Unit> {
+    // This constructor is used inside the DJVM's sandbox.
+    @Suppress("unused")
+    constructor() : this(ClassLoader.getSystemClassLoader())
+
+    // Loads the contract class from the transactionClassLoader.
+    private fun createContractClass(id: SecureHash, contractClassName: ContractClassName): Class<out Contract> {
+        return try {
+            Class.forName(contractClassName, false, transactionClassLoader).asSubclass(Contract::class.java)
         } catch (e: Exception) {
-            throw TransactionVerificationException.ContractCreationError(ltx.id, className, e)
+            throw TransactionVerificationException.ContractCreationError(id, contractClassName, e)
         }
+    }
 
-        val contractClasses: Map<ContractClassName, Class<out Contract>> = (inputStates + ltx.outputs)
-                .map { it.contract }
-                .toSet()
-                .map { contract -> contract to contractClassFor(contract) }
-                .toMap()
+    override fun apply(ltx: LedgerTransaction) {
+        val contractClassNames = (ltx.inputs.map(StateAndRef<ContractState>::state) + ltx.outputs)
+            .map(TransactionState<*>::contract)
+            .toSet()
 
-        val contractInstances: List<Contract> = contractClasses.map { (contractClassName, contractClass) ->
+        contractClassNames.associateBy(
+            { it }, { createContractClass(ltx.id, it) }
+        ).map { (contractClassName, contractClass) ->
             try {
                 /**
                  * This function must execute with the DJVM's sandbox, which does not
@@ -377,13 +405,10 @@ class Verifier(val ltx: LedgerTransaction, private val transactionClassLoader: C
             } catch (e: Exception) {
                 throw TransactionVerificationException.ContractCreationError(ltx.id, contractClassName, e)
             }
-        }
-
-        contractInstances.forEach { contract ->
+        }.forEach { contract ->
             try {
                 contract.verify(ltx)
             } catch (e: Exception) {
-                logger.error("Error validating transaction ${ltx.id}.", e)
                 throw TransactionVerificationException.ContractRejection(ltx.id, contract, e)
             }
         }

core/src/main/kotlin/net/corda/core/node/ServiceHub.kt

@@ -10,6 +10,7 @@ import net.corda.core.flows.ContractUpgradeFlow
 import net.corda.core.node.services.*
 import net.corda.core.serialization.SerializeAsToken
 import net.corda.core.transactions.FilteredTransaction
+import net.corda.core.transactions.LedgerTransaction
 import net.corda.core.transactions.SignedTransaction
 import net.corda.core.transactions.TransactionBuilder
 import java.security.PublicKey
@@ -71,6 +72,12 @@ interface ServicesForResolution {
      */
     @Throws(TransactionResolutionException::class, AttachmentResolutionException::class)
     fun loadContractAttachment(stateRef: StateRef): Attachment
+
+    /**
+     * Provides a callback for the Node to customise the [LedgerTransaction].
+     */
+    @JvmDefault
+    fun specialise(ltx: LedgerTransaction): LedgerTransaction = ltx
 }
 
 /**

core/src/main/kotlin/net/corda/core/node/services/AttachmentStorage.kt

@@ -1,6 +1,8 @@
+@file:KeepForDJVM
 package net.corda.core.node.services
 
 import net.corda.core.DoNotImplement
+import net.corda.core.KeepForDJVM
 import net.corda.core.contracts.Attachment
 import net.corda.core.crypto.SecureHash
 import net.corda.core.internal.cordapp.CordappImpl.Companion.DEFAULT_CORDAPP_VERSION

core/src/main/kotlin/net/corda/core/serialization/internal/AttachmentsClassLoader.kt

@@ -19,7 +19,6 @@ import java.io.IOException
 import java.io.InputStream
 import java.net.*
 import java.util.*
-import java.util.jar.JarInputStream
 
 /**
  * A custom ClassLoader that knows how to load classes from a set of attachments. The attachments themselves only
@@ -318,14 +317,14 @@ object AttachmentsClassLoaderBuilder {
                                               isAttachmentTrusted: (Attachment) -> Boolean,
                                               parent: ClassLoader = ClassLoader.getSystemClassLoader(),
                                               block: (ClassLoader) -> T): T {
-        val attachmentIds = attachments.map { it.id }.toSet()
+        val attachmentIds = attachments.map(Attachment::id).toSet()
 
         val serializationContext = cache.computeIfAbsent(Key(attachmentIds, params)) {
             // Create classloader and load serializers, whitelisted classes
             val transactionClassLoader = AttachmentsClassLoader(attachments, params, txId, isAttachmentTrusted, parent)
             val serializers = createInstancesOfClassesImplementing(transactionClassLoader, SerializationCustomSerializer::class.java)
             val whitelistedClasses = ServiceLoader.load(SerializationWhitelist::class.java, transactionClassLoader)
-                    .flatMap { it.whitelist }
+                    .flatMap(SerializationWhitelist::whitelist)
                     .toList()
 
             // Create a new serializationContext for the current transaction. In this context we will forbid