OOM heap space exception during HttpDecoder.readHeaders

[cgmartin]

I unfortunately don't have any details yet on what the requests looked like, but wanted to start a dialog in the meantime...

Using [http-kit "2.1.5"], I'm seeing several of these exceptions in my logs, some that happen just after the app restarts:

Exception in thread "server-loop" java.lang.OutOfMemoryError: Java heap space
        at org.httpkit.server.HttpDecoder.readHeaders(HttpDecoder.java:173)
        at org.httpkit.server.HttpDecoder.decode(HttpDecoder.java:84)
        at org.httpkit.server.HttpServer.decodeHttp(HttpServer.java:80)
        at org.httpkit.server.HttpServer.doRead(HttpServer.java:141)
        at org.httpkit.server.HttpServer.run(HttpServer.java:245)
        at java.lang.Thread.run(Thread.java:722)

My suspicions are that this happened during a websocket connection, but I'll try to add more debugging to get a better idea of what's going on. I'll also try upgrading to 2.1.7.

Thanks!

[shenfeng]

According to the code: https://github.com/http-kit/http-kit/blob/master/src/java/org/httpkit/server/HttpDecoder.java?source=c#L172

It seems that it fails to allocate a big enough buffer.

According to the code: https://github.com/http-kit/http-kit/blob/master/src/java/org/httpkit/server/HttpDecoder.java?source=c#L196

http-kit will try to reject too large body. the default value of max-body is 8M.

Does you server really busy and the memory JVM has is not big enough?

After the exception, http-kit will not process any http request. It's a bug, I will fix it.

[shenfeng]

please try

[http-kit "2.1.8"]

It has this new commit: b54f860

When server is busy or something, the OOM happens, but surely, it's not the right time for the server to fail: there are so many customers.

[jeluard]

Once an OOM is thrown there's no much you can do beyond shutting down the VM.
Even if it was due to a wrong huge allocation that you would discard right away there could be a bunch of other concurrent operation failing at the same time (e.g. class loading). You cannot survive those in the general case.

There is probably 2 potential cause to this:

• some memory leak
• a misconfigured VM (like Xmx)

It would help to have a memory dump here. You could use Visual VM or the VM option -XX:+HeapDumpOnOutOfMemoryError for this.

[shenfeng]

Hi, @jeluard , I agree with you.

Since http-kit is just a very small part of the whole web application, it still a good idea to catch the exception and discard the wrong huge allocation, It's better just let that request fail, instead of the whole application fail.

Year, -XX:+HeapDumpOnOutOfMemoryError will help a lot.

[cgmartin]

Thanks very much for the quick responses! Here's some details on the server/vm setup:

It runs on a 1GB RAM/1 CPU virtual server with Ubuntu 13.04 x64 that I recently set up a few weeks ago. It doesn't have anything heavy running on it except for my http-kit app.

java version "1.7.0_21"
OpenJDK Runtime Environment (IcedTea 2.3.9) (7u21-2.3.9-1ubuntu1)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

I've been running it with the java command without any VM options: /usr/bin/java -jar my-app-standalone.jar
Here's a printout of the default settings: https://gist.github.com/cgmartin/6050892

Not experiencing any heavy traffic spikes, very low usage. The web pages it serves are cached behind a reverse proxy. It is primarily used as a WebSocket server and I don't see many concurrent connections. If curious, you can see it running here: http://cljwamp.us

I haven't been logging memory usage but whenever I spot check there's usually around 400-500 MB free. I unfortunately don't know what the memory situation was at the time of the stack traces. The app had been running without fail for 10 days straight with about 90 websocket clients that had connected over that time period. At the time of the stack traces only 1 client was connected. It logged 6 stack traces within a 10 minute timeframe.

I've restarted with the suggested HeapDump setting, and now using [http-kit "2.1.8"]:
/usr/bin/java -XX:+HeapDumpOnOutOfMemoryError -jar my-app-standalone.jar
(I'm a newbie with VM tuning, any suggestions would be appreciated.)

I'll close this until I can capture a heap dump and provide some better details.
Much thanks for all the help!

[jeluard]

VisualVM would probably help you understand what happens to your memory usage, especially its monitor section. It's relatively easy to install/use and you can monitor remote VM with it (might require some conf).

One option you might want to set to the VM is Xmx which defines the maximum usable memory: when this threshold is crossed you get OOM. Its default depends on OS/VM details.

[shenfeng]

As far as I can think of, http-kit uses few memory ( 64K buffer, shared by all connection, other book keeping stuff). For you server, it has quite a lot of memory (1G).

I suspect there are memory leaks somewhere. Use tools or just write some code to stress test the code may help.

Stress test it for sometime, inspect the memory usage.

There are another tool, http://www.eclipse.org/mat/ , maybe helpful

[cgmartin]

I've been meaning to get VisualVM set up for remote monitoring. I've used it locally to monitor one load scenario, which tests the messaging rate between many concurrent websocket connections (no memory leaks found in that test). I'll create another one that stresses the connection handling (random disconnects/reconnects) and let it run for a while. I haven't tried MAT before, I'll check that out. Thanks again for the tips!