Serialization depends on reference identity

[ulrikrasmussen]

Consider the following program:

import net.corda.core.serialization.serialize
import net.corda.testing.node.internal.setDriverSerialization

fun main(args: Array<String>) {
    setDriverSerialization(ClassLoader.getSystemClassLoader())
    val x = Pair("foo", "bar")
    val list1 = listOf(x, x.copy())
    val list2 = listOf(x, x)
    println("list1 == list2: ${list1 == list2}")
    println("list1.serialize() == list2.serialize(): ${list1.serialize() == list2.serialize()}")
}

This prints:

list1 == list2: true
list1.serialize() == list2.serialize(): false

Serialization is not bijective!

This is because Corda automatically de-duplicates multiple occurrences of objects in the serialized data using an IdentityHashMap, i.e. by reference equality (see objectHistory in SerializationOutput.kt). Supposedly this is done to reduce the size of serialized data in some cases.

The behavior is a bit unfortunate because transaction hashes now depend on the degree of sharing in the heap. Reference equality is irrelevant from a semantic perspective for all data classes, so two transactions can be semantically equivalent but still have different serialized representations. This doesn't cause any problems when transactions are created by a central party and distributed from there, but consider a use case where a transaction contains fairly large data objects which are already known by all parties in a flow that need to sign the transaction. In that case, the parties can just exchange information such as the privacy nonce and deterministically derive the transaction locally, thereby avoiding transmitting large objects to each other. This approach however becomes quite brittle when reference identity of data (which is often non-deterministic, e.g. in the case of a cache eviction) affects the transaction hash.

I think that this behavior should be disabled or at least be optional.

[r3jirabot]

Automatically created Jira issue: CORDA-4260