LoadingMessageSourceProvider can cause OOM errors

[jdoklovic]

LoadingMessageSourceProvider creates a thread factory which creates daemon threads with no way of shutting them down.

The comments explicitly states:
/*
* Use daemon threads. We don't give control to the user about the
* ExecutorService, and we don't have a reliable way to shut it down (a JVM
* shutdown hook does not get involved on a webapp shutdown, so we cannot
* use that...).
*/

Unfortunately, this can cause OutOfMemory errors, especially in an OSGi environment, when these threads stay alive across bundle restarts. This provider should expose a way to safely shutdown / manage the lifecycle of the executor service threads.

Also, json-schema-validator which consumes this library should also expose a way to shut this down.

[alexvetter]

Yep, I do have this problem. It occurs in our Vert.x environment.

[fge]

OK, I admit that this can be a problem; now there is the lifecycle to consider.

If this is the JVM itself then when it shuts down the problem is not there.

If this is a dynamically loaded jar, such as in a webapp, then a mechanism should exist that shuts down all generated ExecutorServices. In which case this will need to be documented.

[alexvetter]

It occurred in the vertx-json-schema-validator, which uses the json-schema-validator of yours. And here it's the problem that the LoadingConfigurationBuilder initializes a URIDownloadersRegistry (from json-schema-core). And the vertx-json-schema-validator calls LoadingConfiguration.newBuilder() everytime this Vert.x module is re-deployed.

[fge]

@alexvetter how is this related to this package then? The default loading configuration uses statically initialized message bundles; if this is not the case it means that you change the message bundle to a loading one.

Can you show some example code?

[alexvetter]

The msg-simple repository is a dependency of json-schema-core. The LoadingConfigurationBuilder initializes a URIDownloadersRegistery and this uses (way down) LoadingMessageSourceProvider.

[error] Caused by: java.lang.OutOfMemoryError: unable to create new native thread
[error]     at java.lang.Thread.start0(Native Method)
[error]     at java.lang.Thread.start(Thread.java:714)
[error]     at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
[error]     at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
[error]     at com.github.fge.msgsimple.provider.LoadingMessageSourceProvider.getMessageSource(LoadingMessageSourceProvider.java:190)
[error]     at com.github.fge.msgsimple.bundle.MessageBundle.getMessage(MessageBundle.java:122)
[error]     at com.github.fge.msgsimple.bundle.MessageBundle.printf(MessageBundle.java:170)
[error]     at com.github.fge.msgsimple.bundle.MessageBundle.printf(MessageBundle.java:190)
[error]     at com.github.fge.jsonschema.core.util.URIUtils$4.check(URIUtils.java:146)
[error]     at com.github.fge.jsonschema.core.util.URIUtils$4.check(URIUtils.java:142)
[error]     at com.github.fge.jsonschema.core.util.Registry.put(Registry.java:96)
[error]     at com.github.fge.jsonschema.core.util.Registry.putAll(Registry.java:122)
[error]     at com.github.fge.jsonschema.core.load.configuration.URIDownloadersRegistry.<init>(URIDownloadersRegistry.java:77)
[error]     at com.github.fge.jsonschema.core.load.configuration.LoadingConfigurationBuilder.<init>(LoadingConfigurationBuilder.java:76)
[error]     at com.github.fge.jsonschema.core.load.configuration.LoadingConfiguration.newBuilder(LoadingConfiguration.java:138)

[goldsam]

The fact that LoadingMessageSourceProvider creates/uses a thread factory of its choice prevents this library and all that depend on it (e.g. json-schema-core) from working in sandboxed environments which disallows creation of standard threads. Unfortunately, your libraries don't function on Google App Engine currently. Here is a stack trace from the GAE/J devserver when i invoke LoadingConfiguration.byDefault():

java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "modifyThreadGroup")
    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
    at java.security.AccessController.checkPermission(AccessController.java:884)
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
    at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:429)
    at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkAccess(DevAppServerFactory.java:454)
    at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
    at java.lang.Thread.init(Thread.java:391)
    at java.lang.Thread.init(Thread.java:349)
    at java.lang.Thread.<init>(Thread.java:675)
    at java.util.concurrent.Executors$DefaultThreadFactory.newThread(Executors.java:613)
    at com.github.fge.msgsimple.provider.LoadingMessageSourceProvider$1.newThread(LoadingMessageSourceProvider.java:91)
    at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:612)
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:925)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
    at com.github.fge.msgsimple.provider.LoadingMessageSourceProvider.getMessageSource(LoadingMessageSourceProvider.java:190)
    at com.github.fge.msgsimple.bundle.MessageBundle.getMessage(MessageBundle.java:122)
    at com.github.fge.msgsimple.bundle.MessageBundle.getMessage(MessageBundle.java:146)
    at com.github.fge.jackson.JsonNodeReader.readNode(JsonNodeReader.java:140)
    at com.github.fge.jackson.JsonNodeReader.fromInputStream(JsonNodeReader.java:103)
    at com.github.fge.jackson.JsonLoader.fromResource(JsonLoader.java:99)
    at com.github.fge.jsonschema.SchemaVersion.<init>(SchemaVersion.java:63)
    at com.github.fge.jsonschema.SchemaVersion.<clinit>(SchemaVersion.java:44)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at com.google.appengine.tools.development.agent.runtime.RuntimeHelper.checkRestricted(RuntimeHelper.java:70)
    at com.google.appengine.tools.development.agent.runtime.Runtime.checkRestricted(Runtime.java:65)
    at com.github.fge.jsonschema.core.load.configuration.LoadingConfigurationBuilder.<init>(LoadingConfigurationBuilder.java:117)
    at com.github.fge.jsonschema.core.load.configuration.LoadingConfiguration.byDefault(LoadingConfiguration.java:151)
...

[lukeu]

Could the hard-coded ExecutorService be set up to release threads after a time, and spin them up again as required? e.g. in LoadingMessageSourceProvider, change:

    = Executors.newFixedThreadPool(NTHREADS, THREAD_FACTORY);

to

    = new ThreadPoolExecutor(0, NTHREADS,
                             10L, TimeUnit.SECONDS,
                             new LinkedBlockingQueue<Runnable>(),
                             THREAD_FACTORY);

This is hitting a GUI app I work on. Even without the OSGI scenario, json-schema-core spins up 9 (=3x3) threads to access a few tiny configuration files.

For reasons (that I won't go into) we set the default per thread stack size to 4MB, and 36MB is a lot of extra RAM to consume when it is seemingly unused most of the time.