Java heap space crash

[weirdcrap]

Hey, love the tool it has been great for keeping my IMDB stuff up to date.

I requested an API key from TheTVDB just to play with your tool some more and once I added TVDB support it crashes with Java heap space errors every time it runs.

I'm running in a docker on UnRAID.

I didn't see any closed issues for this, sorry if this has been covered before.

updatetool.log

[mynttt]

Hi!

Thanks for bringing this up. I looked through the log. Interesting that there are tons of 405 errors regarding the TVDB API. I just checked one of those failed API calls and it worked on my side.

I guess it might have to do with the TVDB API itself. I hope those go away otherwise I would have to investigate that.

The crash seems to be memory related. Your TV Show library seems to have nearly 40000 items. I guess the memory hard limit of 256MB is not enough to handle such a large library. I will update the docker file tomorrow to allow setting the memory limit manually via an environment variable.

Once that is done I'll notify you in this issue and you can then try it with 512MB or if actually necessary 1024MB. At 40000 items there are tons of Strings loaded that can easily eat up a good portion of the available RAM.

[weirdcrap]

Haha yeah my TV show library is a bit large

I assumed the 405 errors were due to rate limiting as it seems to hit the API as fast as humanly possible, I did not try the API calls outside of the tool though.

A configurable variable would be nice for Java's memory, I already limit my RAM for my dockers using the docker "--Memory=2GB" parameter.

Thanks for looking into it!

[mynttt]

Alright I have updated the Dockerfile.

You can specify a custom max. heap via the environment variable JVM_MAX_HEAP. Maybe you try out 512m first and if it still crashes try 1g. Let me know if the error still persists!

I also changed the tool to request garbage collection after every run and tweaked the JVM startup settings so unused allocated heap space is released back to the OS at a certain threshold. So if a run requires 512MB the garbage collection will push it down to something like 12-20MB after the task has finished and the JVM will shrink to something like 20-40MB. So unused heap is not stolen from the OS anymore.

The 405 errors usually come in place when you request a resource with the wrong method i.e. GET where POST is required. I have no idea why TVDB would return that since GET is used on a GET resource. APIs are weird, their website does not mention that code in the documentation for that endpoint. No idea whats happening there tbh.

[weirdcrap]

Thanks for the update!

I set the variable to start at 512m and restarted the container. It does not appear to immediately execute a ratings sync anymore? Does it now only run when scheduled even at container start?

When I first started playing with the container every time it was restarted a sync would occur. I can wait for the next scheduled run, just wasn't sure if this was an issue on my end or a change in behavior.

[mynttt]

It should start immediately. I changed the backend so the logs will look differently now at startup. Can you upload the log to this issue?

[weirdcrap]

Sure, this is literally all that's been logged since I started the container about four hours ago:

updatetool.0.log

[mynttt]

Whatever happened there should not happen now anymore. Let me know if it persists.

[weirdcrap]

It seems to be working now! The process didn't get to finish because Comcast is crap, but that is an all to familiar story for most of us. Thanks for your help.

EDIT: I like that it was able to resume the run on its own even though my internet cut out. Ran successfully, no more JVM Heap Crash.