Possible Memory leak?

[artisticmathematician]

Hello,

first off, by far one of the most creative mods from the custom A.I.'s and the grow-able monster crystals to the excellent utility items like your crystal dust torches and the crystal pickaxes. I really look forward to your future updates, especially when things slide back more to a magical aesthetic.

Unfortunately, just for now, I had to remove this mod from my family's modpack because of a possible memory leak. At first, my server was acting up, the allocated memory was being consumed completely without any new world gen or increase in entities.

I moved all my mods to a single player server to try and find out the issue. I removed mods one by one until I got to this one. So, I ran a fresh server with just crystallogy loaded up and generated a 1500x1500 map. I left my game to idle for a few hours and when I came back, the memory had creeped up from when I left, around 50% memory used, to 93%. Again, this mod was running on a fresh save with only a chunk gen mod along side it.

I don't know if I set something up wrong, but here is my JVM arguments just in case:
-Xmx7G -XX:+UseConcMarkSweepGC -XX:-UseAdaptiveSizePolicy -Xmn896M

Thank you so much for your time and work on this amazing mod, it really stood out among other mods in my eyes.

[COM8]

Hi artisticmathematician,
first of all thanks for reporting!
I will have a look at this as fast as possible (and hopefully fix it soon ;D) because this could be a serious problem.

[COM8]

Which version of Crystallogy do you use and what version of Forge you are using?

[COM8]

I don't think it's a problem of Crystallogy.
I did extensive testing on multiple machines and with multiple instances with the following result:
After letting two instances of Minecraft run parallel for a couple of hours, one with Crystallogy and the other one only with forge I didn't see any difference. Yes, the left one (with Crystallogy installed) allocated more ram at the end, but both use up to 89% of the given ram after 3 hours and than drop down to about 75% once the garbage collector starts its job.

The left one has Crystallogy installed and the right one only runs with Forge.

So all in all I think it's a problem of Forge or even Minecraft.

I Used Crystallogy v.1.2.0.4 and Forge 1.10.2 - 12.18.3.2239 for my tests. All instances ran in a new generated world, that was generated with the same seed.

You said you also tested it. Did you tried letting only Forge run for a couple hours by it self?

[artisticmathematician]

I was using an older version of forge, I was using forge-1.10.2-12.18.3.2221 with the latest version of the mod as well. I will update and mess around with it today and let you know. I will try to get more details on how to reproduce the problem.

I will run forge for a few hours first. I think the problem occurs specifically during chunk generation. I will run minecraft, generate a map, and see what happens to the memory vs just sitting at spawn with a very small world map.

[artisticmathematician]

Alright, I ran a few fresh saves with an updated forge. It seems to be tied to chunk generation. I watched the garbage collector doing work, so I am not sure what is causing such a huge amount of memory to be allocated. Just for context, my entire modpack, with a generated circular map of 300 chunks (so about a 5000 block radius), only hits about 55% allocated along with 20-25% ram.

I ran cyrtallogy with the latest 1.10.2 forge and a chunk gen mod, before I could finish building a circular map with a radius of 100 chunks, the memory tapped out. I took a picture.

I left the game running for about an hour with no signs of improvment, the game actually crashed when I tried to log out, but that crash could be related to the custom AI feature I was working with right before closing the game:

---- Minecraft Crash Report ----
// I feel sad now :(

Time: 3/4/17 11:13 AM
Description: Ticking entity

java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:711)
at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:734)
at net.minecraft.entity.ai.EntityAITasks.func_75774_a(SourceFile:81)
at net.minecraft.entity.EntityLiving.func_70626_be(EntityLiving.java:773)
at net.minecraft.entity.EntityLivingBase.func_70636_d(EntityLivingBase.java:2183)
at net.minecraft.entity.EntityLiving.func_70636_d(EntityLiving.java:577)
at net.minecraft.entity.monster.EntityMob.func_70636_d(SourceFile:47)
at net.minecraft.entity.EntityLivingBase.func_70071_h_(EntityLivingBase.java:2013)
at net.minecraft.entity.EntityLiving.func_70071_h_(EntityLiving.java:295)
at net.minecraft.entity.monster.EntityMob.func_70071_h_(SourceFile:52)
at net.minecraft.entity.monster.EntityCreeper.func_70071_h_(SourceFile:157)
at net.minecraft.world.World.func_72866_a(World.java:1964)
at net.minecraft.world.WorldServer.func_72866_a(WorldServer.java:839)
at net.minecraft.world.World.func_72870_g(World.java:1934)
at net.minecraft.world.World.func_72939_s(World.java:1750)
at net.minecraft.world.WorldServer.func_72939_s(WorldServer.java:620)
at net.minecraft.server.MinecraftServer.func_71190_q(MinecraftServer.java:709)
at net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:613)
at net.minecraft.server.integrated.IntegratedServer.func_71217_p(IntegratedServer.java:149)
at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:471)
at java.lang.Thread.run(Thread.java:745)

A detailed walkthrough of the error, its code path and all known details is as follows:

-- Head --
Thread: Server thread
Stacktrace:
at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:711)
at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:734)
at net.minecraft.entity.ai.EntityAITasks.func_75774_a(SourceFile:81)
at net.minecraft.entity.EntityLiving.func_70626_be(EntityLiving.java:773)
at net.minecraft.entity.EntityLivingBase.func_70636_d(EntityLivingBase.java:2183)
at net.minecraft.entity.EntityLiving.func_70636_d(EntityLiving.java:577)
at net.minecraft.entity.monster.EntityMob.func_70636_d(SourceFile:47)
at net.minecraft.entity.EntityLivingBase.func_70071_h_(EntityLivingBase.java:2013)
at net.minecraft.entity.EntityLiving.func_70071_h_(EntityLiving.java:295)
at net.minecraft.entity.monster.EntityMob.func_70071_h_(SourceFile:52)
at net.minecraft.entity.monster.EntityCreeper.func_70071_h_(SourceFile:157)
at net.minecraft.world.World.func_72866_a(World.java:1964)
at net.minecraft.world.WorldServer.func_72866_a(WorldServer.java:839)
at net.minecraft.world.World.func_72870_g(World.java:1934)

-- Entity being ticked --
Details:
Entity Type: Creeper (net.minecraft.entity.monster.EntityCreeper)
Entity ID: 22327
Entity Name: Creeper
Entity's Exact location: -310.64, 63.00, 773.18
Entity's Block location: World: (-311,63,773), Chunk: (at 9,3,5 in -20,48; contains blocks -320,0,768 to -305,255,783), Region: (-1,1; contains chunks -32,32 to -1,63, blocks -512,0,512 to -1,255,1023)
Entity's Momentum: 0.04, -0.08, 0.06
Entity's Passengers: []
Entity's Vehicle: ERROR NullPointerException: null
Stacktrace:
at net.minecraft.world.World.func_72939_s(World.java:1750)
at net.minecraft.world.WorldServer.func_72939_s(WorldServer.java:620)

-- Affected level --
Details:
Level name: Copy of Copy of Xoth
All players: 1 total; [EntityPlayerMP['Lucius_Eventide'/199, l='Copy of Copy of Xoth', x=-310.38, y=63.00, z=774.37]]
Chunk stats: ServerChunkCache: 1660 Drop: 0
Level seed: -1510335333124012141
Level generator: ID 00 - default, ver 1. Features enabled: true
Level generator options:
Level spawn location: World: (40,64,256), Chunk: (at 8,4,0 in 2,16; contains blocks 32,0,256 to 47,255,271), Region: (0,0; contains chunks 0,0 to 31,31, blocks 0,0,0 to 511,255,511)
Level time: 22567 game time, 22567 day time
Level dimension: 0
Level storage version: 0x04ABD - Anvil
Level weather: Rain time: 122650 (now: false), thunder time: 99254 (now: false)
Level game mode: Game mode: survival (ID 0). Hardcore: false. Cheats: true
Stacktrace:
at net.minecraft.server.MinecraftServer.func_71190_q(MinecraftServer.java:709)
at net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:613)
at net.minecraft.server.integrated.IntegratedServer.func_71217_p(IntegratedServer.java:149)
at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:471)
at java.lang.Thread.run(Thread.java:745)

-- System Details --
Details:
Minecraft Version: 1.10.2
Operating System: Windows 10 (amd64) version 10.0
Java Version: 1.8.0_25, Oracle Corporation
Java VM Version: Java HotSpot(TM) 64-Bit Server VM (mixed mode), Oracle Corporation
Memory: 1092998432 bytes (1042 MB) / 5495447552 bytes (5240 MB) up to 7422279680 bytes (7078 MB)
JVM Flags: 5 total; -XX:HeapDumpPath=MojangTricksIntelDriversForPerformance_javaw.exe_minecraft.exe.heapdump -Xmx7G -XX:+UseConcMarkSweepGC -XX:-UseAdaptiveSizePolicy -Xmn896M
IntCache: cache: 1, tcache: 1, allocated: 12, tallocated: 94
FML: MCP 9.32 Powered by Forge 12.18.3.2239 6 mods loaded, 6 mods active
States: 'U' = Unloaded 'L' = Loaded 'C' = Constructed 'H' = Pre-initialized 'I' = Initialized 'J' = Post-initialized 'A' = Available 'D' = Disabled 'E' = Errored
UCHIJAAAA mcp{9.19} [Minecraft Coder Pack] (minecraft.jar)
UCHIJAAAA FML{8.0.99.99} [Forge Mod Loader] (forge-1.10.2-12.18.3.2239.jar)
UCHIJAAAA Forge{12.18.3.2239} [Minecraft Forge] (forge-1.10.2-12.18.3.2239.jar)
UCHIJAAAA mercurius_updater{1.0} [mercurius_updater] (MercuriusUpdater-1.10.2.jar)
UCHIJAAAA chunkgen{1.3} [ChunkGen] (chunkgen-1.10.2-1.5.0.jar)
UCHIJAAAA crystallogy{1.2.0.4} [Crystallogy] (crystallogy-1.10.2-1.2.0.4.jar)
Loaded coremods (and transformers):
GL info: ERROR RuntimeException: No OpenGL context found in the current thread.
Profiler Position: N/A (disabled)
Player Count: 1 / 8; [EntityPlayerMP['Lucius_Eventide'/199, l='Copy of Copy of Xoth', x=-310.38, y=63.00, z=774.37]]
Type: Integrated Server (map_client.txt)
Is Modded: Definitely; Client brand changed to 'fml,forge'

---

I also ran another test where I just left my character idle without generating any new chunks and the memory never increased. So, this seems to be related to generating chunk data. When ever new chunks are generated, the amount of memory being used seems to be a lot larger in contrast to my other big mods.

I am going to keep messing with things and see if I can't find out more about what is causing this memory issue. I know before I have tried disabling all four crystal spawns to see if that was impacting the amount of memory being used for chunks, I will do that again and get back to you.

[COM8]

Just a quick update:
I was able to track it down to a method that is called during crystal world generation.
Due to the fact that this is a really important method for world generation I've decided to refactor my current word generation code.

[COM8]

That's the current state of for this issue:
I'm currently unable to fix it, because I'm unable to find it and this is what I've found out about it until now:

• It is client side only.
• It's kind of connected to world generation of crystals.
• If you let run your world for several hours the leak fixes it self about 50% of the time.
• If you disable world generation (for all crystals and the crystorya plant) it does not fix it completely. Only for about 70-95%.
• If you run Javas (not Minecrafts) garbage collector it fixes it.
• Thanks to JProfiler I was able to figure out, that the general Minecraft hash map is growing and growing. This continues until you either run the Java garbage collector or just wait it out and the map shrinks down to a normal size again.
• It probably got introduced between 1.2 prev2 and 1.2 prev3.

So, to sum up how you can "fix" the leak:

• Disable world generation for crystals and the crystorya plant in the config.
• Update to version 1.2.0.6 of Crystallogy.

If you have any suggestions or tips how I can fix it, feel free to post them here.
Thanks for reading and stay tuned for further updates. The version for Minecraft 1.11.2 will be a complete rewrite and so hopefully fix this issue.