Buffer overflow in TcpConnection.send

[amcelroy]

Howdy,

Love the library, it has been such an easy networking client to work with, and I've used it in multiple projects.

For one particular project, I have a client connected to a server streaming frames from a camera. Everything works great, but as the connection time gets longer, in the 10's of hours range, there is eventually a buffer overflow on the server side which causes the Server send thread to shut down.

The client and server have write and object buffer sizes 50,000, which is sufficient (camera frames are small).

I've tracked the issue down to the TcpConnection.send BufferOverflow, but I'm not sure if it is the writebuffer or the serialization object.

I would like to kick around some ideas on how to fix this.

• Put a try / catch around writebuffer and serialization object to catch buffer overflows. Perhaps the buffer could be flushed and a notification sent that the buffer is corrupted and to resend data. This would be preferable to a server crash, at least for me.
• Determine why the write buffer is not being emptied?
• Other?

[amcelroy]

This seems to be associated always with the KeepAlive object, I was previously sending one out every 50 ms, which may be an issue. Still trying to deduce more information on the problem.

[RobertZenz]

Can you provide a stacktrace and maybe an example (if possible)?

[amcelroy]

Argh, didn't have my server debugger running at the time, but this is a stack trace of the error, after about 12 hours:

Exception in thread "Timer-0" Exception in thread "Server" Exception in thread "main" java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:244)
at com.esotericsoftware.kryonet.TcpConnection.send(TcpConnection.java:188)
at com.esotericsoftware.kryonet.Connection.sendTCP(Connection.java:59)
at com.esotericsoftware.kryonet.Server.update(Server.java:352)
at com.esotericsoftware.kryonet.Server.run(Server.java:372)
at java.lang.Thread.run(Thread.java:745)
java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:244)
at com.esotericsoftware.kryonet.TcpConnection.send(TcpConnection.java:188)
at com.esotericsoftware.kryonet.Connection.sendTCP(Connection.java:59)
at com.esotericsoftware.kryonet.Server.sendToAllTCP(Server.java:451)
at tahi.Lepton.Server.LeptonServer.sendMessage(LeptonServer.java:393)
at tahi.Lepton.Server.LeptonServer.run(LeptonServer.java:334)
at tahi.Lepton.Server.LeptonServer.main(LeptonServer.java:88)
com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0, required: 4
Serialization trace:
Frame (tahi.Lepton.Server.LeptonFrame)
at com.esotericsoftware.kryo.io.ByteBufferOutput.require(ByteBufferOutput.java:211)
at com.esotericsoftware.kryo.io.ByteBufferOutput.writeFloat(ByteBufferOutput.java:584)
at com.esotericsoftware.kryo.serializers.DefaultSerializers$FloatSerializer.write(DefaultSerializers.java:164)
at com.esotericsoftware.kryo.serializers.DefaultSerializers$FloatSerializer.write(DefaultSerializers.java:158)
at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:629)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:334)
at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:303)
at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:629)
at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:87)
at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:505)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
at com.esotericsoftware.kryonet.KryoSerialization.write(KryoSerialization.java:48)
at com.esotericsoftware.kryonet.TcpConnection.send(TcpConnection.java:192)
at com.esotericsoftware.kryonet.Connection.sendTCP(Connection.java:59)
at com.esotericsoftware.kryonet.Server.sendToAllTCP(Server.java:451)
at tahi.Lepton.Server.LeptonServer.sendFrame(LeptonServer.java:384)
at tahi.Lepton.Server.LeptonServer.access$0(LeptonServer.java:380)
at tahi.Lepton.Server.LeptonServer$2.run(LeptonServer.java:278)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

I can add some example code this evening.

[amcelroy]

Server:

private Server KyroServer;

public static void main(String[] args){
KryoServer = new Server((int)2e6, (int)5e5);
KryoServer.start();
KryoServer.bind(90, 91);
KryoServer.addListener(new Listener(){
@Override
        public void received(Connection c, Object o) {
		//Handle objects here 
        }
        	
        @Override
        public void connected(Connection c) {
        	c.setKeepAliveTCP(8000);
        	c.setTimeout(120000);
                //Send over the config file ArrayList<LeptonROI>
        	c.sendTCP(LeptonConfig.getROIs());
        }
});
KryoServer.getKryo().register(LeptonROI.class);
KryoServer.getKryo().register(ArrayList.class);
KryoServer.getKryo().register(LeptonFrame.class);
KryoServer.getKryo().register(Float[].class);
KryoServer.getKryo().register(Point.class);

//Take images and stream to client
KryoServer.sendToAllTCP(LeptonFrame);
}

Client:

private Client KryoClient;

String ip = 192.168.1.210; //IP set by user elsewhere, example here

KryoClient = new Client((int)2e6, (int)5e5);
KryoClient.start();
KryoClient.addListener(new Listener(){
	@Override
	public void disconnected(Connection c) { }
			
	@Override
	public void idle(Connection arg0) { }
			
	@Override
	public void received(Connection c, Object o) {
		//Handle receipt of Objects
	}	
});
		
KryoClient.getKryo().register(LeptonROI.class);
KryoClient.getKryo().register(ArrayList.class);
KryoClient.getKryo().register(LeptonFrame.class);
KryoClient.getKryo().register(Float[].class);
KryoClient.getKryo().register(Point.class);
		
new Thread(new Runnable(){
	@Override
	public void run() {
		try{
			KryoClient.connect(5000, ip, 90, 91);
		}catch(Exception e){
			e.printStackTrace();
		}
	}
}).start();

//Do other client stuff here

LeptonFrame, Point, and LeptonROI contain all basic primitives (int and float), except for one Float[] object with the pixels.

[crykn]

In the crash log it says:

at java.nio.Buffer.position(Buffer.java:244)
at com.esotericsoftware.kryonet.TcpConnection.send(TcpConnection.java:188)

but line 188 is not the send()-Method and doesn't call Buffer.position(). What code base are you currently using?

Two other things:

• To prevent problems with the serializer you could switch to one of the forks that added Kryo 4.0.0
• You should call KryoServer.bind(...) after KryoServer.getKryo().register(...) and KryoClient.addListener(...), because sometimes data was already sent when those methods get called

[amcelroy]

Thanks for looking into this. I am using the releases, specifically the production one jar file. Is there a specific fork that you recommend?

[crykn]

I made a fork myself a few days ago and that is probably the most actively maintained one right now: github.com/crykn/kryonet.
But you could also use this fork by Pyeroh, on which I based my fork.

[amcelroy]

Ok, thanks very much for everyone's help. I'll mark this as closed as an issue regarding an older version of Kryo.

[i3130002]

As @crykn wrote

I made a fork myself a few days ago and that is probably the most actively maintained one right now: github.com/crykn/kryonet.
But you could also use this fork by Pyeroh, on which I based my fork.

I tried to use it for android development. But I have Compability problem:

Error:Error converting bytecode to dex:
Cause: Dex cannot parse version 52 byte code.
This is caused by library dependencies that have been compiled using Java 8 or above.
If you are using the 'java' gradle plugin in a library submodule add 
targetCompatibility = '1.7'
sourceCompatibility = '1.7'
to that submodule's build.gradle file.

Although I set this config:

repositories {
    maven { url 'https://jitpack.io' }
}
dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    androidTestCompile('com.android.support.test.espresso:espresso-core:2.2.2', {
        exclude group: 'com.android.support', module: 'support-annotations'
    })
    compile 'com.android.support:appcompat-v7:25.3.1'
    compile 'com.android.support.constraint:constraint-layout:1.0.0-alpha7'
    compile ('com.github.crykn:kryonet:2.22.1'){
        sourceCompatibility =1.7
        targetCompatibility =1.7
    }
    testCompile 'junit:junit:4.12'
}

Please help me out.

[i3130002]

My last Issue solved using Gradle 4 and Android Studio 3. Wish it work on previous versions.
But I'm still having terrible sending 30MB File over tcp.

W/System.err: com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0, required: 32984
I/log_me: client  incoming bytes : 65535
W/System.err:     at com.esotericsoftware.kryo.io.ByteBufferOutput.require(ByteBufferOutput.java:212)
W/System.err:     at com.esotericsoftware.kryo.io.ByteBufferOutput.writeBytes(ByteBufferOutput.java:312)
W/System.err:     at com.esotericsoftware.kryo.io.ByteBufferOutput.writeBytes(ByteBufferOutput.java:298)
W/System.err:     at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:49)
W/System.err:     at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:38)
W/System.err:     at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
W/System.err:     at com.esotericsoftware.kryonet.serialization.KryoSerializationFactory$KryoSerialization.write(KryoSerializationFactory.java:67)
W/System.err:     at com.esotericsoftware.kryonet.TcpConnection.send(TcpConnection.java:242)
W/System.err:     at com.esotericsoftware.kryonet.Connection.sendTCP(Connection.java:112)
W/System.err:     at com.esotericsoftware.kryonet.Server.sendToTCP(Server.java:638)
W/System.err:     at packagename.server_thread.server_listener(server_thread.java:113)
W/System.err:     at packagename.server_thread$1.received(server_thread.java:60)
W/System.err:     at com.esotericsoftware.kryonet.Server$1.received(Server.java:93)
W/System.err:     at com.esotericsoftware.kryonet.Connection.notifyReceived(Connection.java:369)
W/System.err:     at com.esotericsoftware.kryonet.Server.update(Server.java:308)
W/System.err:     at com.esotericsoftware.kryonet.Server.run(Server.java:528)
W/System.err:     at java.lang.Thread.run(Thread.java:761)

My server code is something like this.

 server.addListener(new Listener() {
                        public void received (Connection connection, Object object) {
                            server_listener(connection,object);
                    });

public void server_listener (Connection connection, Object object) {
        Log.i("log_me","server:  incomming request "+ object.toString());
        if (object instanceof String) {
            String request = (String)object;
            Log.i("log_me",request);
            if(request.equals("send file"))
            {
                connection.sendTCP("File is comming name:");
                connection.sendTCP(file.getPath().substring(file.getPath().lastIndexOf("/")+1));//
                final int BUFFER_SIZE = 64*1024; //this is actually bytes
                FileInputStream fis = null;
                try {
                    fis = new FileInputStream(path);
                    byte[] buffer = new byte[BUFFER_SIZE];
                    int read = 0;
                    while( ( read = fis.read( buffer ) ) > 0 ){
                        server.sendToTCP(connection.getID(),buffer);
                        buffer=null;
                        buffer = new byte[BUFFER_SIZE];
                    }
                    server.sendToTCP(connection.getID(),"end");
                    fis.close();
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }else{
                String  response = "Thanks";
                connection.sendTCP(response);
            }
        }
    }

Any suggestion to send large files 10GB and more using kryonet ?

[crykn]

Did you change the servers internal byte buffer size via the constructor (new Server(int writeBufferSize, int objectBufferSize))?

[i3130002]

How much should I put there ? It allocates memory and I cannot put big numbers like 1010241024*1024
What am I missing?

[crykn]

Your stack trace shows that the buffer requires a size of at least 65535 bytes.

[i3130002]

No matter what I put. For 60MB file it will finally crash

server=new Server(1024*1024,2024*1024);
 client=new Client(1024*1024,2024*1024);

nor

 server=new Server(1024*1024,1024*1024);
    client=new Client(1024*1024,1024*1024);

Nothing works. I think that 65535 bytes is amount of bytes it needs to send this chunk. and no matter what the next chunk will broke.
I attack my server and client code if you need to read

code.zip

[crykn]

You could try using the built-in InputStreamSender. Here is an example for using it. That should clear up whether it's your codes fault or kryonets.

[i3130002]

Dear @crykn you are right.
But can you please clarify for me that:
1-Why I get buffer overflow by sending large amount of sendTcp ?
2-How can I clear the buffer to prevent this error ?

[crykn]

I have no idea why the buffer is overlowing, normally it should flush itself. Did you try the InputStreamSender?

[i3130002]

Dear @cyrkn inputStreamSender works. As I noticed it will flush itself after the object completely sent. But it also buffers sending objects and this functionality overlfows. So I forced to limit my request and also check for the buffer size before adding an object to send.
I expected that it sendtcp synchronize but it's asynchronous so in my loop it overlfows the objts sending queue.

[faucct]

I have just ran into this error while repeatedly calling sendTCP for lots of small objects. If updating kryo helps, why is this repo still stuck at old version?

[i3130002]

Sorry @faucct It's been so long that I can not remember what are we talking about it here. Please apologize me for not being helpful and also not clear enough about what solved my problem in the first place.