releaseExternalResources and akka.pattern.AskTimeoutException in multi-threaded environment

[sebastianthelen]

We are using the Spark micro framework (http://sparkjava.com/) to create a prototype of our new web application. Spark allows you to define callbacks for routes invoked by client requests, e.g.,

get("/foo", (request, response) -> {
    callback();
});

In one of the routes we employ parallec to concurrently retrieve the information (from another web service) we need to generate the HTTP response for a client request. We follow the procedure shown in the examples, i.e.,

ParallelTaskConfig config = new ParallelTaskConfig();
config.setActorMaxOperationTimeoutSec(35);
config.setTimeoutInManagerSec(900);
config.setTimeoutAskManagerSec(910);
 
ptb = pc.prepareHttpGet(String.format("/%s/$PARAMS", serviceConf.getServiceRoute())).saveResponseHeaders(respHeaders)
    .setResponseContext(responseContext).setHttpHeaders(new ParallecHeader()
                            .addPair("Accept-Encoding","gzip")
                            .addPair("Accept",serviceConf.getMimeType())
                            .addPair("Accept-Language",this.language))
   .setConfig(config)
   .setConcurrency(100)
   .setReplaceVarMapToSingleTargetSingleVar("PARAMS", identifierList, serviceConf.getServiceHost())
   .setHttpPort(Integer.parseInt(serviceConf.getServicePort()));

ptb.execute(new ParallelResponseHandler());
pc.releaseExternalResources();

We noticed that when querying our web application with multiple concurrent client requests, i.e., one process is spawned for each concurrent request, we often encounter the following error message

Line 267: 84269 [Thread-16] ERROR io.parallec.core.task.ParallelTaskManager  - Exception in sendTaskToExecutionManager akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://ParallecActorSystem/user/ExecutionManager-PT_14_20170110104948951_853663af-ce4#-964038860]] after [910000 ms] details akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://ParallecActorSystem/user/ExecutionManager-PT_14_20170110104948951_853663af-ce4#-964038860]] after [910000 ms]: 

As it turns out this issue arises only when calling pc.releaseExternalResources() at the end of the callback. Our suspicion is that calling pc.releaseExternalResources() releases external resources that are still required by objects in one of the other spawned processes.

Is there anything we can do to avoid getting this error message? Is parallec "thread safe" in the sense that it can be used as described above? What would be the effect of not calling pc.releaseExternalResources()? Would the external resources get garbage collected?

[jeffpeiyt]

@sebastianthelen thanks so much for trying Parallec. As mentioned in the javadoc please not to call this releaseExternalResources() until you stop using Parallec.

If you are on some web server (such as Tomcat) and runs parallec jobs as cron jobs. We do not need to call this line at all. We have several production application that runs as web server. We do not call it in our app.

The actor system and the threadpools are live so that it will keep reusing the same one. The resources used are not significant either. Feel free to use tools such as VirtualVM to track the heap usage etc.

releaseExternalResources
public void releaseExternalResources()
Releases the external resources that this object depends on. You should not call this method if you still want to use the external resources (e.g. akka system, async http client store, thread pool for SSH/TCP) are in use by other objects.

[sebastianthelen]

@jeffpeiyt thanks for your prompt reply.

If you are on some web server (such as Tomcat) and runs parallec jobs as cron jobs. We do not need to call this line at all.

That's exactly what we were not sure of. We have read the javadocs for releaseExternalResources() but were not sure if this also applies in the web server scenario.

The actor system and the threadpools are live so that it will keep reusing the same one.

Understood. We also tracked the heap usage with VirtualVM and everything looked ok.

Again, thanks for clarifying.

[jeffpeiyt]

@sebastianthelen Glad that it helped. Thanks for your questions, we will enhance the doc on this. Guess other people will have similar questions.