Fury fails when deserializing from InputStream

[Munoon]

Search before asking

• I had searched in the issues and found no similar issues.

Version

0.5.0-SNAPSHOT, 0.4.1

Component(s)

Java

Minimal reproduce step

import org.apache.fury.Fury;
import org.apache.fury.config.Language;

import java.io.BufferedInputStream;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.util.Arrays;

public class Reproducer {
    public static void main(String[] args) {
        Fury fury = Fury.builder()
                .withLanguage(Language.JAVA)
                .requireClassRegistration(false)
                .withMetaContextShare(false)
                .build();

        byte[] data = fury.serializeJavaObject(new TestRecord("Some string", 123));

        try {
            TestRecord result = fury.deserializeJavaObject(Arrays.copyOf(data, data.length), TestRecord.class);
            System.out.println("Byte array result: " + result);
        } catch (Exception e) {
            System.out.println("Byte array result: " + e.getClass().getSimpleName());
            e.printStackTrace();
        }

        ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(Arrays.copyOf(data, data.length));
        try {
            TestRecord result = fury.deserializeJavaObject(byteArrayInputStream, TestRecord.class);
            System.out.println("ByteArrayInputStream result: " + result);
        } catch (Exception e) {
            System.out.println("ByteArrayInputStream result: " + e.getClass().getSimpleName());
            e.printStackTrace();
        }

        // hack: Fury have an optimisation for ByteArrayInputStream, which we want to avoid
        InputStream inputStream = new BufferedInputStream(new ByteArrayInputStream(Arrays.copyOf(data, data.length)));
        try {
            TestRecord result = fury.deserializeJavaObject(inputStream, TestRecord.class);
            System.out.println("InputStream result: " + result);
        } catch (Exception e) {
            System.out.println("InputStream result: " + e.getClass().getSimpleName());
            e.printStackTrace();
        }
    }

    public static class TestRecord {
        public final String a;
        public final int b;

        public TestRecord(String a, int b) {
            this.a = a;
            this.b = b;
        }

        @Override
        public String toString() {
            return "TestRecord{" +
                    "a='" + a + '\'' +
                    ", b=" + b +
                    '}';
        }
    }
}

What did you expect to see?

No exception to be throwed, the data should be deserialized correctly.

What did you see instead?

java.lang.UnsupportedOperationException: Unsupported coder 111 when deserializing with ByteArrayInputStream and java.lang.IllegalArgumentException when deserializing with any other InputStream.

Anything Else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[Munoon]

After hours of coding I finally came up that I have to manually prefix the byte array with it's original size. However I wasn't able to find any information about this on the documentation.
I think, either documentation have to be updated, either InputStream size read should be refactored with available() method, either this size should be written by Fury method.

[chaokunyang]

Hi @Munoon, fury.deserializeJavaObject(intputstream, class) should only be used with fury.serializeJavaObject(outputstream, object). You can't use it to deserialize the data from fury.serializeJavaObject(object).

Our document didn't pointed out this, sorry for the inconvenience. We do have plans to align the implementation between those methods, but that may take a little time to implement.

[Munoon]

Hi @Munoon, fury.deserializeJavaObject(intputstream, class) should only be used with fury.serializeJavaObject(outputstream, object). You can't use it to deserialize the data from fury.serializeJavaObject(object).

Can you please explain what's the benefit of writing this size?
In my app I am using Fury to write the data in to the file (currently it is implemented using Files.writeAllBytes(fury.deserializeJavaObject(obj))) and then read it later.
So I'm trying to understand which case suits better here.

Also, I found out that the fastest way to read the data from the file with Fury is by passing the FileChannel -> ByteBuffer -> MemoryBuffer wrapper. Since, I guess, this is a very popular use case in using Fury, such API may be added.

[chaokunyang]

Currently fury doesn't support streaming read, so we prefix the size in fury.serializeJavaObject(outputstream, object. Then when deserialization, we can read size first and read binary from the stream to deserialize the object like you did manually. If you want to fury do this for you, you must use fury.serializeJavaObject(outputstream, object) for serialization.

[chaokunyang]

It is possible to support streaming natively. We can make MemoryBuffer hold a read channel Or input stream object, and read data only in checkReadableBytes method.

I prefer using channel since it provide opportunities for zero copy.

It's feasible, but we don't have time for this currently.

[Munoon]

Fury seems like an interesting project to me, so I'd like to contribute, actually.
If you think, that this feature worth implementing - I'd like to make a try.

Also, I think that MemoryBuffer could become interface now, where different implementations holds different data sources (native address, byte array, InputStream, channel, etc)

[chaokunyang]

Fury seems like an interesting project to me, so I'd like to contribute, actually. If you think, that this feature worth implementing - I'd like to make a try.

Also, I think that MemoryBuffer could become interface now, where different implementations holds different data sources (native address, byte array, InputStream, channel, etc)

That would be really great, @Munoon .

I try to implement the native streaming deserialization in #1451 . It's a little complicated due to we don't employ a polymorphic design for MemoryBuffer. The streaming reading for InputStream are basiclly finished, but the support for java.nio.Channel are not included in that PR. Maybe it can be taken as an example to implement the streaming channel deserializaiton.

I thought about make MemoryBuffer polymorphic when I create fury several years ago, but polymorphism will make the invocation and other performance optimization such as bound checks elimination infeasible.
And considering many cases are small object deserializaiton, the streaming are unnecessary, so I made it as a final classes in the end.

[chaokunyang]

In those days, objects are getting bigger and bigger, the steaming is important to reduce latency for big objects, especially for big data and tensor computing.