Fury Java: fails to decode a `scala.collection.BitSet` (NPE)

[andyczerwonka]

Fury is configured as follows:

    val fury = Fury
      .builder()
      .withLanguage(Language.JAVA)
      .requireClassRegistration(false)
      .withRefTracking(true)
      .withStringCompressed(true)
      .withLongCompressed(true)
      .withIntCompressed(true)
      .withAsyncCompilation(true)
      .buildThreadSafeFury()

After successful encoding, taking the encoded value and decoding fails with the following exception.

java.lang.NullPointerException:
  at scala.collection.mutable.BitSet$.fromBitMaskNoCopy(BitSet.scala:383)
  at scala.collection.mutable.BitSet$SerializationProxy.readResolve(BitSet.scala:390)
  at io.fury.serializer.ReplaceResolveSerializer.readObject(ReplaceResolveSerializer.java:307)
  at io.fury.serializer.ReplaceResolveSerializer.read(ReplaceResolveSerializer.java:292)
  at io.fury.Fury.readData(Fury.java:812)
  at io.fury.serializer.ReplaceResolveSerializer.read(ReplaceResolveSerializer.java:271)
  at io.citrine.lolo.trees.splits.CategoricalSplitFuryRefCodec_1343252559_1426407511_823874944.read(CategoricalSplitFuryRefCodec_1343252559_1426407511_823874944.java:84)
  at io.citrine.lolo.trees.InternalModelNodeFuryRefCodec_1343252559_1426407511_1409680642.readFields$(InternalModelNodeFuryRefCodec_1343252559_1426407511_1409680642.java:147)
  at io.citrine.lolo.trees.InternalModelNodeFuryRefCodec_1343252559_1426407511_1409680642.read(InternalModelNodeFuryRefCodec_1343252559_1426407511_1409680642.java:176)
  at io.citrine.lolo.trees.InternalModelNodeFuryRefCodec_1343252559_1426407511_1409680642.readFields$(InternalModelNodeFuryRefCodec_1343252559_1426407511_1409680642.java:131)

https://github.com/alipay/fury/blob/releases-0.2.1/java/fury-core/src/main/java/io/fury/Fury.java#L812
https://github.com/alipay/fury/blob/releases-0.2.1/java/fury-core/src/main/java/io/fury/serializer/ReplaceResolveSerializer.java#L307

[chaokunyang]

@andyczerwonka Would you mind sharing reproduceable code here? It's not easy to know what happened without the code

[andyczerwonka]

@andyczerwonka Would you mind sharing reproduceable code here? It's not easy to know what happened without the code

Unfortunately I can't given it's a proprietary trained model from a customer. I'm trying to pare down the data structure to help isolate.

Is there no way to hypothesize as to where the NPE would come from? It looks to be in the JDKReplaceResolveMethodInfoCache.

[andyczerwonka]

If the NPE is happening here, it looks to me like it's on access to function application apply(o) call given there is null protection on both jdkMethodInfoCache and readResolveMethod.

[andyczerwonka]

@chaokunyang I was able to reproduce. Fury can't handle scala.collection.BitSet. The following test fails with an NPE.

  test("reproduce Fury defect") {
    import FuryEncoder._
    val encoded = fury.serialize(BitSet(1, 2, 3, 4))
    val decoded = fury.deserialize(encoded).asInstanceOf[BitSet]
    decoded.toList should contain theSameElementsAs List(1, 2, 3, 4)
  }

I've renamed the ticket.

[andyczerwonka]

Interestingly, I added a custom serializer, and I'm stiff getting an NPE.

    class BitSetSerializer(fury: Fury) extends Serializer(fury, classOf[BitSet]) {

      override def write(buffer: MemoryBuffer, value: BitSet) = {
        val converted = value.map(_.toByte).toArray
        buffer.writeBytesWithSizeEmbedded(converted)
      }

      override def read(buffer: MemoryBuffer): BitSet = {
        val bytes = buffer.readBytesAlignedSizeEmbedded
        val ints = bytes.map(_.toInt)
        BitSet.fromSpecific(ints)
      }

    }

I am registering the serializer via

fury.registerSerializer(classOf[BitSet], new BitSetSerializer(fury))

yet it doesn't see to call it when I run the test. 🤔

[knutwannheden]

I had a similar issue, where the problem was that the serializer must be registered for the concrete implementation class.

[chaokunyang]

Hi, @andyczerwonka @knutwannheden , I fixed this bug in #1030:

Please check it using latest snapshot jar and reopen this issue if it doesn't work. Thanks for finding this bug and creating an issue for fury. Such issue will make fury a better project

[andyczerwonka]

@chaokunyang I can confirm that the latest snapshot does in fact solve this issue. Can you give us some insight into when you'd cut a release? We have policy that will unfortunately not allow us to use nightly snapshots in production.

[chaokunyang]

@andyczerwonka New version will be released by the end of next week