[SHIRO-668] Catch unexpected errors which can lead to oom

[cpetzka]

Unexpected errors in the "run" method of the validation ExecutorServiceSessionValidationScheduler can kill the validation thread an it will not be executed anymore (see StackOverflow.) This can lead to OOM's through too much sessions and can be a security risk by never ending sessions.
Catching "Throwable" is not the cleanest solution but it garant that the thread will be executed in the future.

[fpapon]

Retest this please

[asfgit]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Shiro-pr/47/

[fpapon]

retest this please

[asfgit]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Shiro-pr/48/

[fpapon]

@bdemers have you any objection for this PR?

[bdemers]

We might want to take another look at this one, catching Throwable might not be the best option.
@cpetzka if you are still following this (and no excuse for the delay) what type of exception were you seeing?

[cpetzka]

Sorry, I can't tell you what exception it was. I don't think it was logged. There were entries in the log every 5 minutes that the validation thread was running normally. Then the entries stopped and old sessions were no longer removed. So I didn't fix the reason for the expedition but built this workaround.

[fpapon]

May be we can use a UncaughtExceptionHandler in the Thread initialization to have more informations. If there is a problem with a particular Session restart a new thread will not fix the problem. I agree with @bdemers that it will be usefull to have more information about the actual exception and try to fix the origin.

[cpetzka]

If the problem is only a short disconnect from SessionStore/Cache, a simple reboot would be sufficient. However, if it is always the same session that triggers an error, the error would happen regularly but also be logged. With the help of log monitoring, the administrator can be informed about repeated errors.

[bdemers]

Thanks @cpetzka I can see that.

Maybe we should catch Runtimes, and re-throw everything else. That would allow temporary exceptions like connection issues to be caught and logged (and the thread to continue), while other system errors to bubble up.

@fpapon's idea for handling errors could be helpful too

[fpapon]

supercedes by #186