Exception in thread "Thread-39"

[Kaos1337]

Hi,
I would report these:

2011-12-17 23:28:59 [SEVERE] Exception in thread "Thread-39"
2011-12-17 23:28:59 [SEVERE] java.lang.OutOfMemoryError: unable to create new native thread
2011-12-17 23:28:59 [SEVERE] at java.lang.Thread.start0(Native Method)
2011-12-17 23:28:59 [SEVERE] at java.lang.Thread.start(Thread.java:691)
2011-12-17 23:28:59 [SEVERE] at org.dynmap.web.HttpServer.run(HttpServer.java:80)

It say "out of memory" but at that time the server had 9gb of ram allocated with a total of 21gb, the OS is CentOS 6.1 64bit.
Sometimes happen that after 2-3 hours that the server is started the webmap doesn't load more and anyone can't connect to the server (I must restart it).
Regards.

[mikeprimm]

Unfortunately, that does say you're out of memory, but doesn't say who consumed it. What bukkit and dynmap are you running, and what other plugins?

[Kaos1337]

Craftbukkit #1597 and dynmap 0.27
I prefer not provide the plugin list in public, I sent it via bukkit.org

[mikeprimm]

What are you doing with dynmap at the time? Do you have fullrender active? Have you tried temporarily removing dynmap and confirmed that you don't see the problem?

[Kaos1337]

What are you doing with dynmap at the time?
I don't know, maybe me nothing, but usually there are others 50 players in game.

Do you have fullrender active?
How can I know if it is activated?

Have you tried temporarily removing dynmap and confirmed that you don't see the problem?
Uhm, no, I think that players kill me if I remove the dynmap for 1 second eheh.

However I use Java SE 7u2, is dynmap compatible?

[mikeprimm]

Java 7 is fine (and recommended).

Take a look at the output from these commands (post here, if you can):
/dynmap stats
/dynmap triggerstats

These will give you a good idea what is going on, and what has gone on, as far as our rendering engine.

[Kaos1337]

This is about /dynmap stats: http://i.imgur.com/bMHy8.png
and this about /dynmap triggerstats: http://i.imgur.com/Tcg3y.png
I sent to you via MP on bukkit.org my configuration.txt file.

[mikeprimm]

I'm not seeing anything particularly busy here. Only thing I noticed is that your chunk load time is pretty slow (3.5ms/chunk) and you've dialed down the chunks-per-tick to 75, but don't seem to have an issue with having a long update queue. What sort of headroom do you have on memory when you have peak player load (do you have LagMeter? If so, what do you get with 'mem'?)

[Kaos1337]

I have essentials that provide "mem".
This is the result: http://imgur.com/A86NU
When the server reach 70-80 (our peak) players online usually "allocated memory" is about 12gb

[mikeprimm]

With that big a population, along with (what I'd assume) a relatively large number of potential dynmap users, you may want to consider shifting us over to being supported by an external server (Apache). I'd suggest this (https://github.com/webbukkit/dynmap/wiki/Setting-up-Dynamic-Map-with-apache2-under-Debian) or even this (https://github.com/webbukkit/dynmap/wiki/Setting-up-without-the-Internal-Web-Server). The latter one will absolutely minimize the load on the bukkit server from any amount of web traffic (as well as insulating the server from any denial-of-service type traffic), while the former will shift the vast majority of the bandwidth (tile loading) off to the external server.

[Kaos1337]

Hi,
I use the 1st method...

[waffletastic]

Kaosvf I am having the same issue as you, but do not have Dynmap, add me on skype, let's find a solution. My skype name is shamwoow.

[mikeprimm]

There is an issue with what some folks refer to as the"NextTickList" problem - I've got a fix for it in a pull request on CraftBukkit-Bleeding, but it has yet to be accepted. In any case, on busy servers (especially one with lots of players) it can definitely be an issue - see Bukkit/CraftBukkit-Bleeding@master...TickListGrowth for the patch

[mikeprimm]

Just pinned down a slow memory leak on dynmap - fix has been released with 0.28 this morning. I'd suggest giving it a look.

[Kaos1337]

I'm trying it.

[Kaos1337]

Received again this:

2011-12-28 20:32:22 [SEVERE] Exception in thread "Listen thread"
2011-12-28 20:32:22 [SEVERE] java.lang.OutOfMemoryError: unable to create new native thread
2011-12-28 20:32:22 [SEVERE] at java.lang.Thread.start0(Native Method)
2011-12-28 20:32:22 [SEVERE] at java.lang.Thread.start(Thread.java:691)
2011-12-28 20:32:22 [SEVERE] at net.minecraft.server.NetworkManager.(NetworkManager.java:69)
2011-12-28 20:32:22 [SEVERE] at net.minecraft.server.NetLoginHandler.(NetLoginHandler.java:27)
2011-12-28 20:32:22 [SEVERE] at net.minecraft.server.NetworkAcceptThread.run(SourceFile:49)
2011-12-28 20:32:22 [SEVERE] Exception in thread "Thread-42"
2011-12-28 20:32:22 [SEVERE] java.lang.OutOfMemoryError: unable to create new native thread
2011-12-28 20:32:22 [SEVERE] at java.lang.Thread.start0(Native Method)
2011-12-28 20:32:22 [SEVERE] at java.lang.Thread.start(Thread.java:691)
2011-12-28 20:32:22 [SEVERE] at org.dynmap.web.HttpServer.run(HttpServer.java:80)

[mikeprimm]

OK - I'm still of the opinion that we're not the cause there (even the fixed issue would only yield a very slow memory leak that would generally take days to amount to something of interest, or perhaps a fullrender of a truly enormous world running for quite a while). I'd suggest comparing plugin lists with @waffletastic and see if something common comes up. If you're interested in a dev build of the CraftBukkit fix I've submitted for the NextTickList issue, I'll be happy to supply one (I'm a member of the Bukkit-Bleeding team - not the core Bukkit team, but kind of the next ring out of trusted contributors).

[Kaos1337]

Ok, thank you for your support, which dev build of craftbukkit has the fix?

[mikeprimm]

It's still waiting to be accepted - I can spin a build for you, but its not an official dev build

[Kaos1337]

Uhm, yes thank you, I'll try it for a test, I look forward.

[mikeprimm]

Here's a download link - http://mikeprimm.com/craftbukkit-1.0.1-R2-SNAPSHOT.jar

[Kaos1337]

Now I'm running it in my server, I'll report here if the problem still occur, thanks.

[Kaos1337]

This afternoon happened this: http://www.mediafire.com/?qmi12c4ic42aadt

[mikeprimm]

Out of things to try - I've looked at everything we do, and we're not leaking, even on a low memory server with huge renders of big maps, so unless you know for sure that disabling us fixes the problem, I'm out of places to look.

[Kaos1337]

hmm, I think that is JavaSE 7u2 on CentOS 6.2 (64bit), waffletastic suggested that he had the same problem and he solved changing os, I'll try to return to java6... maybe it will help me.
However thank you for your help.

[mikeprimm]

Please let me know if you get any results - I'm not going to close this issue at present: I'll keep it in mind in case anything else I come across may be relevant. On the java side, what command line args are you using with the J2SE? Are you using Oracle/Sun J2SE 7 or OpenJDK (I've seen multiple issues in the past that happened with OpenJDK and were resolved with the Oracle/Sun one).

[Kaos1337]

I use this: http://i.imgur.com/kFXX3.png (link: http://www.oracle.com/technetwork/java/javase/downloads/index.html)
This is the direct download: http://download.oracle.com/otn-pub/java/jdk/7u2-b13/jdk-7u2-linux-x64.rpm

This is the command that I use: java -Xincgc -Xmx18G -jar -Djline.terminal=jline.UnsupportedTerminal craftbukkit.jar

[mikeprimm]

Cool - just for reference, I don't use the -Xincgc - here's my command line:

-Xmx4096M -Xms1024M -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalPacing -XX:ParallelGCThreads=2 -XX:+AggressiveOpts

[Kaos1337]

Now I'm using this: java -Xms1G -Xmx18G -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalPacing -XX:ParallelGCThreads=2 -XX:+AggressiveOpts -jar -Djline.terminal=jline.UnsupportedTerminal craftbukkit.jar
I'll see the situation...

[mikeprimm]

Cool - please let me know if it has any impact, in any case - any information is useful here :)

[Kaos1337]

The problem happen again, I'll change to java 6 soon...

[Kaos1337]

The same problem with java6u30...

[Kaos1337]

Now I have reinstalled dynmap completely.. (deleting "dynmap" folder).

[mikeprimm]

Have you had the chance to confirm that you can only produce this with Dynmap present? 0.28 addressed all the other folks I'm aware of that had reported memory issues.

[Kaos1337]

Yes, before this version I had not this issue, I use dynmap since june 2011.

[mikeprimm]

Understood, but besides bukkit updates, I'm quite sure we're not the only plugin you've upgraded or added.

[Kaos1337]

I would say that in these 2 days the problem didn't appear again!
I solved adding the argument -Xms10G in the run command line... it's the only one solution that I have at the moment, with this it's impossibile that the server can't take more memory... it has got all since the start. In fact it can reach only 9gb and after these happened the error OutOfMemoryError.

[tonybruess]

Alright I'm having this same error. Going to try -Xms and will report back.

[tonybruess]

Adding Xms did not help: java -Xms2G -Xmx6G -jar craftbukkit-1.1-R1.jar

Any suggestions?

[Kaos1337]

which OS and java version do you have?

[tonybruess]

http://pastebin.com/JF4sMP13

[Kaos1337]

The same problem, but I have java SE7u2 and CentOS 6.2 (64bit), and the server can't go really "outofmemory", it has got 18gb of ram to run, this is the only way: https://bukkit.atlassian.net/secure/Dashboard.jspa
only a my curiosity, do you use RemoteToolkit?

[tonybruess]

I do not.

And I'm not sure of this is bukkit or plugin related.

[mikeprimm]

-Xms doesn't help anything: its just an initial memory allocation number (just saves the JVM from needing to grow on its own). -Xmx is the real limit. In any case, you absolutely want to have something other than the default garbage collector : at least -Xincgc , althought the concurrent mark-and-sweep collector is better.

[tonybruess]

I have found the solution to this problem. Here's my post on the forums about it:

http://forums.bukkit.org/posts/954147/

[mikeprimm]

WOW - that's quite an impressive bit of debugging :) Really glad to hear you tracked it down!