[Java] ObjectStreamSerializer async meta-shared layer JIT fails for TreeSet/TreeMap subclasses

[mandrean]

Search before asking

• I had searched in the issues and found no similar issues.

Version

v0.16.0

Probably introduced via 924279c in v0.14.0

Component(s)

Java

Minimal reproduce step

This looks related to #3337 (fixed in #3342) and #3343 (fixed in #3344), but specifically for the .withAsyncCompilation(true) path.

A minimal reproducer is:

package org.apache.fory.serializer;

import static org.testng.Assert.assertEquals;

import java.util.TreeSet;
import org.apache.fory.Fory;
import org.apache.fory.builder.Generated;
import org.apache.fory.config.CompatibleMode;
import org.apache.fory.config.Language;
import org.apache.fory.reflect.ReflectionUtils;
import org.apache.fory.serializer.Serializer;
import org.testng.Assert;
import org.testng.annotations.Test;

public class AsyncObjectStreamTreeSetReproTest {
  public static class ChildTreeSet extends TreeSet<String> {
    public ChildTreeSet() {}
  }

  @Test(timeOut = 60000)
  public void testAsyncCompilationTreeSetSubclassObjectStreamSerializer()
      throws InterruptedException {
    Fory fory =
        Fory.builder()
            .withLanguage(Language.JAVA)
            .requireClassRegistration(false)
            .withRefTracking(true)
            .withCodegen(true)
            .withCompatibleMode(CompatibleMode.COMPATIBLE)
            .withAsyncCompilation(true)
            .build();

    fory.registerSerializer(
        ChildTreeSet.class, new ObjectStreamSerializer(fory, ChildTreeSet.class));

    ChildTreeSet values = new ChildTreeSet();
    values.add("one");
    values.add("two");

    assertEquals(fory.deserialize(fory.serialize(values)), values);

    waitForGeneratedLayerSerializer(fory, ChildTreeSet.class);

    assertEquals(fory.deserialize(fory.serialize(values)), values);
  }

  private void waitForGeneratedLayerSerializer(Fory fory, Class<?> type)
      throws InterruptedException {
    long deadline = System.currentTimeMillis() + 30_000;
    while (System.currentTimeMillis() < deadline) {
      if (hasGeneratedLayerSerializer(fory, type)) {
        return;
      }
      Thread.sleep(10);
    }
    Assert.fail("Timed out waiting for generated layer serializer for " + type.getName());
  }

  private boolean hasGeneratedLayerSerializer(Fory fory, Class<?> type) {
    Serializer<?> serializer = fory.getTypeResolver().getSerializer(type);
    if (!(serializer instanceof ObjectStreamSerializer)) {
      return false;
    }
    Object[] slotsInfos = (Object[]) ReflectionUtils.getObjectFieldValue(serializer, "slotsInfos");
    if (slotsInfos.length == 0) {
      return false;
    }
    for (Object slotsInfo : slotsInfos) {
      Object slotsSerializer = ReflectionUtils.getObjectFieldValue(slotsInfo, "slotsSerializer");
      if (!(slotsSerializer instanceof Generated)) {
        return false;
      }
    }
    return true;
  }
}

What did you expect to see?

ObjectStreamSerializer should successfully round-trip TreeSet/TreeMap subclasses with .withAsyncCompilation(true), and after async JIT completes it should switch to generated layer serializers without errors.

What did you see instead?

The initial interpreter-mode round trip works, but once async JIT kicks in the layer serializer bootstrap fails.

Typical failures include:

java.lang.NoSuchMethodException: no such constructor: ...ForyRefCodecMetaSharedLayer...<init>()void/newInvokeSpecial

and:

java.lang.ClassCastException: class ...ForyRefCodec_0 cannot be cast to class org.apache.fory.serializer.MetaSharedLayerSerializerBase

In a deterministic test like the one above, this usually shows up as a timeout waiting for the generated layer serializer to become available.

Anything Else?

• Setting .withAsyncCompilation(false) makes the same reproducer pass.
• A TreeSet subclass is the smallest reproducer I found.
• The same broken async layer bootstrap also shows up for related TreeMap subclasses and containers that route through ObjectStreamSerializer.

Are you willing to submit a PR?

• I'm willing to submit a PR!