[Java] COMPATIBLE mode: deserializing an older-schema payload poisons subsequent serializations on the same Fory instance (silent field loss / unreadable output)

[bank-ro]

Search before asking

• I had searched in the issues and found no similar issues.

Version

1.x.x - verified present in 1.3.0, 1.4.0 and 1.6.0 (not a regression; long-standing)

Component(s)

Java

Describe the bug

In CompatibleMode.COMPATIBLE, what a Fory instance writes depends on what it has previously
read. After deserializing a payload whose embedded TypeDef differs from the local class - the
ordinary schema-evolution case COMPATIBLE mode exists for - subsequent serializations on that same
instance embed the remote (stale) TypeDef while the data section is written by the current
class's serializer.

Depending on the class shape the result is either silent data loss (new fields dropped, no
exception) or a self-inconsistent stream that no reader can deserialize - not even a pristine
instance with the writer's own registrations.

This is severe for any read-modify-write service: reading an old record and re-saving it silently
persists corrupt or lossy bytes. In our production a routine one-field schema addition corrupted
~10,000 database records within one minute of deploying to a single node. The records were
unreadable by every binary including the one that wrote them; symptoms were
DeserializationException, IndexOutOfBoundsException,
IllegalArgumentException: UTF-16 byte size N is not aligned to element size 2, and NPEs inside
TypeResolver.readTypeInfo, at offsets that varied per record.

Mechanism

1. A codec's read and write paths share one TypeInfoHolder - e.g.
   ArraySerializers$ObjectArraySerializer.elementTypeInfoHolder is used by both
   writeArrayPayload and readArrayElements (1.4.0 ArraySerializers.java:108,122,134;
   unchanged in 1.6.0 :112,126,134).
2. Reading a payload whose TypeDef differs from the local class builds a meta-shared TypeInfo
   carrying the remote TypeDef (TypeResolver.buildCheckedMetaSharedTypeInfo →
   getMetaSharedTypeInfo, which sets typeInfo.typeDef = typeDef) and publishes it into that
   holder at the tail of readTypeInfo.
3. A later serialization resolves the type through the same holder, and
   writeSharedClassMeta emits exactly that stale def:

   TypeDef typeDef = typeInfo.typeDef;
   if (typeDef == null) { typeDef = buildTypeDef(typeInfo); }
   buffer.writeBytes(typeDef.getEncoded());

   (1.3.0 TypeResolver.java:586-605, 1.4.0 :631-651, identical in 1.6.0.)
4. The data, however, is written by the local class's serializer - CompatibleSerializer.write
   delegates to a serializer created from type, the local class (1.4.0
   CompatibleSerializer.java:217-225) - so def and data disagree.

The read path is immune because it validates the cached holder against the TypeDef id carried by
the stream (readSharedClassTypeInfo: a field-local cache hit is accepted only when the cached
TypeDef id equals the id read from the buffer). The write path has no external truth to validate
against, which is why only writes are poisoned.

Minimal reproduce step

Single file, fory-core only. ItemV1/BoxV1 model an older binary's schema; ItemV2/BoxV2 the
current one:

import org.apache.fory.*;
import org.apache.fory.config.*;
import java.util.Arrays;

public class ForyStaleTypeDefRepro {
    public static class ItemV1 { public String name; public ItemV1() {} }                     // old schema
    public static class ItemV2 { public String name; public String tag; public ItemV2() {} }  // current schema
    public static class BoxV1 { public ItemV1[] items; public BoxV1() {} }
    public static class BoxV2 { public ItemV2[] items; public BoxV2() {} }

    static ThreadSafeFory build(Class<?> item, Class<?> arr, Class<?> box) {
        ThreadSafeFory f = Fory.builder()
                .withLanguage(Language.JAVA)
                .withRefTracking(false)
                .withCompatibleMode(CompatibleMode.COMPATIBLE)
                .requireClassRegistration(true)
                .withAsyncCompilation(false)
                .buildThreadSafeFory();
        f.register(item, 100); f.register(arr, 101); f.register(box, 102);
        return f;
    }

    public static void main(String[] a) {
        // an older binary wrote this record
        ThreadSafeFory oldBinary = build(ItemV1.class, ItemV1[].class, BoxV1.class);
        BoxV1 ob = new BoxV1();
        ob.items = new ItemV1[]{new ItemV1(), new ItemV1()};
        ob.items[0].name = "a"; ob.items[1].name = "b";
        byte[] oldBytes = oldBinary.serialize(ob);

        // the current binary
        ThreadSafeFory victim = build(ItemV2.class, ItemV2[].class, BoxV2.class);
        BoxV2 box = new BoxV2();
        box.items = new ItemV2[]{new ItemV2(), new ItemV2()};
        box.items[0].name = "a"; box.items[0].tag = "x";
        box.items[1].name = "b"; box.items[1].tag = "y";

        byte[] s1 = victim.serialize(box);   // baseline write
        victim.deserialize(oldBytes);        // read ONE older-schema payload
        byte[] s2 = victim.serialize(box);   // same object, same instance, immediately after

        System.out.println("identical=" + Arrays.equals(s1, s2));
        ThreadSafeFory pristine = build(ItemV2.class, ItemV2[].class, BoxV2.class);
        System.out.println("s1 tag0=" + ((BoxV2) pristine.deserialize(s1)).items[0].tag);
        System.out.println("s2 tag0=" + ((BoxV2) pristine.deserialize(s2)).items[0].tag);
    }
}

Output on 1.4.0 (withRefTracking(false)):

identical=false
s1 tag0=x
s2 tag0=null      <-- the new field is silently dropped from s2

Output with .withRefTracking(true).withRefCopy(true) - same bug, harsher failure:

identical=false
s1 reads: tag0=x
s2 FAILS to deserialize: java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 3

With @ForyField(id = …)-annotated classes (our production shape) the stream is likewise
unreadable rather than lossy. In every variant, one unrelated read changed what a later
serialization produced.

Expected behavior

Serialization output must depend only on the object and the local class schema, never on what
the instance previously deserialized. writeSharedClassMeta should source the TypeDef from the
local class - e.g. getTypeDef(cls) / typeDefMap, which is populated exclusively from local
class builds - rather than from the holder-resolved typeInfo.typeDef; or read and write paths
should not share TypeInfoHolder state.

Environment

• fory-core 1.4.0 (mechanism verified byte-identical in 1.3.0 and 1.6.0)
• JDK 26 (also seen on JDK 25); Windows and Linux
• CompatibleMode.COMPATIBLE, requireClassRegistration(true), withAsyncCompilation(false),
  ThreadSafeFory. Reproduces with refTracking both false and true. Poisoning is per pooled
  instance; Shareable serializers registered across a ThreadSafeFory pool widen the blast
  radius.

Workaround

Two ThreadSafeFory instances with identical registrations: one used only for
deserialization, one used only for serialization. The write-only instance's holders can never
acquire a remote def, so its output is always self-consistent. Verified effective under the
production workload that triggered the corruption.

Possibly related

#3926 is a different primary bug (read-side
registration check while decoding an embedded TypeDef, and a 1.4.0 regression - this one is
present in 1.3.0). But its reporter also mentions, on a larger production corpus, "occasional
MemoryBuffer bounds errors on 1.3.0 bytes" and NPEs at TypeResolver.getMetaSharedTypeInfo, which
match this bug's signature exactly. If any node in that pipeline deserializes 1.3.0 payloads and
re-serializes them, this bug would produce precisely those symptoms - worth checking separately
from the registration-check issue.