Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Android] BigInteger #55

Closed
nicoruti opened this issue Feb 6, 2015 · 7 comments
Closed

[Android] BigInteger #55

nicoruti opened this issue Feb 6, 2015 · 7 comments

Comments

@nicoruti
Copy link

nicoruti commented Feb 6, 2015

Sorry for spamming you with nasty exceptions of Android incompatibilities. Here's the next one:

java.lang.RuntimeException: java.io.IOException: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:122) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTConfiguration.asObject(FSTConfiguration.java:876) ~[fst-2.22.jar:na]
    at org.hive2hive.core.security.FSTSerializer.deserialize(FSTSerializer.java:103) ~[classes/:na]
    at org.hive2hive.core.network.messages.MessageReplyHandler.reply(MessageReplyHandler.java:92) ~[classes/:na]
    at net.tomp2p.rpc.DirectDataRPC.handleResponse(DirectDataRPC.java:164) ~[tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at net.tomp2p.rpc.DispatchHandler.forwardMessage(DispatchHandler.java:166) ~[tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at net.tomp2p.connection.Dispatcher.channelRead0(Dispatcher.java:178) [tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at net.tomp2p.connection.Dispatcher.channelRead0(Dispatcher.java:59) [tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at net.tomp2p.message.TomP2PCumulationTCP.decoding(TomP2PCumulationTCP.java:71) [tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at net.tomp2p.message.TomP2PCumulationTCP.channelRead(TomP2PCumulationTCP.java:47) [tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at net.tomp2p.connection.IdleStateHandlerTomP2P.channelRead(IdleStateHandlerTomP2P.java:112) [tomp2p-core-5.0-Beta3-SNAPSHOT.jar:na]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [netty-transport-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [netty-common-4.0.25.Final.jar:4.0.25.Final]
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [netty-common-4.0.25.Final.jar:4.0.25.Final]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.io.IOException: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:240) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTConfiguration.asObject(FSTConfiguration.java:874) ~[fst-2.22.jar:na]
    ... 31 common frames omitted
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:122) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:561) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectCompatible(FSTObjectInput.java:523) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:508) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:356) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:657) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:515) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:356) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectInternal(FSTObjectInput.java:323) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:304) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObject(FSTObjectInput.java:238) ~[fst-2.22.jar:na]
    ... 32 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_71]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_71]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
    at org.nustaq.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:558) ~[fst-2.22.jar:na]
    ... 41 common frames omitted
Caused by: java.io.IOException: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.nustaq.serialization.FSTObjectInput$2.defaultReadObject(FSTObjectInput.java:988) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput$MyObjectStream.defaultReadObject(FSTObjectInput.java:1268) ~[fst-2.22.jar:na]
    at org.bouncycastle.jcajce.provider.asymmetric.rsa.BCRSAPublicKey.readObject(Unknown Source) ~[bcprov-jdk15on-1.51.jar:1.51.0]
    ... 46 common frames omitted
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.nustaq.serialization.util.FSTUtil.rethrow(FSTUtil.java:122) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:561) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectCompatible(FSTObjectInput.java:523) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.instantiateAndReadNoSer(FSTObjectInput.java:508) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:356) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectFields(FSTObjectInput.java:657) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput$2.defaultReadObject(FSTObjectInput.java:978) ~[fst-2.22.jar:na]
    ... 48 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
    at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
    at org.nustaq.serialization.FSTObjectInput.readObjectCompatibleRecursive(FSTObjectInput.java:558) ~[fst-2.22.jar:na]
    ... 53 common frames omitted
Caused by: java.lang.RuntimeException: unknown object tag -128
    at org.nustaq.serialization.FSTObjectInput.instantiateSpecialTag(FSTObjectInput.java:402) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readObjectWithHeader(FSTObjectInput.java:349) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput.readCompatibleObjectFields(FSTObjectInput.java:782) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput$2.readFields(FSTObjectInput.java:1001) ~[fst-2.22.jar:na]
    at org.nustaq.serialization.FSTObjectInput$MyObjectStream.readFields(FSTObjectInput.java:1273) ~[fst-2.22.jar:na]
    at java.math.BigInteger.readObject(BigInteger.java:3096) ~[na:1.7.0_71]
    ... 57 common frames omitted

As soon as I've more time, I'll look deeper into the FST code to probably be able to fix them myself.
@RuedigerMoeller
Copy link
Owner

:) that's not spamming, its actually helpful. man i did not imagine so many issues with android .. Was this fst 2.22 ?

@nicoruti
Copy link
Author

nicoruti commented Feb 7, 2015

Was this fst 2.22 ?

Yes.

@nicoruti
Copy link
Author

nicoruti commented Feb 7, 2015

The following custom serializer worked for me:

import java.io.IOException;
import java.math.BigInteger;

import org.nustaq.serialization.FSTBasicObjectSerializer;
import org.nustaq.serialization.FSTClazzInfo;
import org.nustaq.serialization.FSTClazzInfo.FSTFieldInfo;
import org.nustaq.serialization.FSTObjectInput;
import org.nustaq.serialization.FSTObjectOutput;

@SuppressWarnings("rawtypes")
public class FSTBigIntegerSerializer extends FSTBasicObjectSerializer {

    @Override
    public void writeObject(FSTObjectOutput out, Object toWrite, FSTClazzInfo clzInfo, FSTFieldInfo referencedBy,
            int streamPosition) throws IOException {
        byte[] value = ((BigInteger) toWrite).toByteArray();
        out.writeInt(value.length);
        out.write(value);
    }

    @Override
    public Object instantiate(Class objectClass, FSTObjectInput in, FSTClazzInfo serializationInfo, FSTFieldInfo referencee,
            int streamPositioin) throws IOException, ClassNotFoundException, InstantiationException, IllegalAccessException {
        byte[] buf = new byte[in.readInt()];
        in.read(buf);
        return new BigInteger(buf);
    }
}

@RuedigerMoeller
Copy link
Owner

Cool ! I thought about registering custom serializers for BigDecimal and BigInteger by default as the JDK emulation for putFields stuff is slowish.
I think I'll first find a generic fix, then add your custom serializer + maybe BigDec custom for better performance.

@RuedigerMoeller
Copy link
Owner

Evaluated the issue and found its impossible for fast-ser to catch this by "emulating" JDK except by adding a huge overhead to all classes using JDK-ser methods (e.g. readObject).
Detailed evaluation see https://github.com/RuedigerMoeller/fast-serialization/blob/master/src/test/ser/androidbitset/AndroidBitSetTest.java line ~40.

So only solution is adding your custom serializer to default init of FSTConfiguration. Watch out for / avoid double custom serializer registration as you update to 2.23 :-).

I really hope there won't be too many classes having similar issues. However even adding a handful of custom serializers is advantageous to dropping performance in half (+much larger size on wire).

Fixed with 2.23 (custom ser addition)

@nicoruti
Copy link
Author

nicoruti commented Feb 8, 2015

Yes, your argumentation makes sense and I see the point. There's no need to decrease overall performance just to support some crappy Android classes.

Thanks for adding the custom serializer to the library. The 'registerObject' line was missing in instantiate. Not sure what it does, but you have it in your other serializers, too. Here's the better version:

@Override
public Object instantiate(Class objectClass, FSTObjectInput in, FSTClazzInfo serializationInfo, FSTFieldInfo referencee,
    int streamPosition) throws IOException, ClassNotFoundException, InstantiationException, IllegalAccessException {
    byte[] buf = new byte[in.readInt()];
    in.read(buf);
    BigInteger result = new BigInteger(buf);
    in.registerObject(result, streamPosition, serializationInfo, referencee);
    return result;
}

@RuedigerMoeller
Copy link
Owner

The registerObject ensures the object is only written once in case its
referenced from other fields of an object graph. It needs to be done before
actually reading the object in case e.g. of collections containing
themselfes.

Thanks for your report+fix and keep the bugs coming :-)

2015-02-08 21:11 GMT+01:00 Nico Rutishauser notifications@github.com:

Yes, your argumentation makes sense and I see the point. There's no need
to decrease overall performance just to support some crappy Android classes.

Thanks for adding the custom serializer to the library. The
'registerObject' line was missing in instantiate. Not sure what it does,
but you have it in your other serializers, too. Here's the better version:

@OverRide
public Object instantiate(Class objectClass, FSTObjectInput in, FSTClazzInfo serializationInfo, FSTFieldInfo referencee,
int streamPosition) throws IOException, ClassNotFoundException, InstantiationException, IllegalAccessException {
byte[] buf = new byte[in.readInt()];
in.read(buf);
BigInteger result = new BigInteger(buf);
in.registerObject(result, streamPosition, serializationInfo, referencee);
return result;
}

—
Reply to this email directly or view it on GitHub
#55 (comment)
.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nicoruti @RuedigerMoeller