Moco is not stopped when run within container

[lordofthejars]

Hello I am developing the arquillian-extension-moco so we can run moco server internally from container automatically while running Arquillian tests instead of having to do manually.

It works great but there is one error when shutting down the Moco server. Let me show you the important code:

This method is called before executing any test and reads a passed json request/response Moco file and starts the runner

public void executeBeforeClass() {
//....
jsonRunner = JsonRunner.newJsonRunnerWithStreams(
                        streams, of(12306));                
jsonRunner.run();
}

and then to stop it:

if(jsonRunner != null) {
    jsonRunner.stop();
}

the problem is that when you run it inside a container and without debugging then an exception that a thread group could not be stopped in an amount of time is thrown. But the funny thing is that if I try to debug then all works as expected.

So the first question is, Am I doing something wrong in the way I start and stop Moco server?

[lordofthejars]

It is funny because if I add

if(jsonRunner != null) {
    jsonRunner.stop();
    Thread.sleep(3000);
}

then it works. It seems like Netty takes some time to kill the threads.

[dreamhead]

Could you please tell me which version you are using?

[lordofthejars]

The latest one the 0.9.1. For what I have seen it seems that the group of threads are not stopped immediately and synchronously because the server warns about the fact that some threads were created but not terminated.

[lordofthejars]

I have tried in another computer with the sleep to 3000 but the problem appears again:

SEVERE: The web application [/ba32e781-3a18-44b3-9547-7c26787f3fe7] appears to have started a thread named [pool-2-thread-1] but has failed to stop it. This is very likely to create a memory leak.
abr 08, 2014 10:29:06 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/ba32e781-3a18-44b3-9547-7c26787f3fe7] created a ThreadLocal with key of type [io.netty.util.internal.ThreadLocalRandom$2] (value [io.netty.util.internal.ThreadLocalRandom$2@77468cae]) and a value of type [io.netty.util.internal.ThreadLocalRandom] (value [io.netty.util.internal.ThreadLocalRandom@6cd3851]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
abr 08, 2014 10:29:06 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/ba32e781-3a18-44b3-9547-7c26787f3fe7] created a ThreadLocal with key of type [io.netty.util.Recycler$1] (value [io.netty.util.Recycler$1@93e33bd]) and a value of type [io.netty.util.Recycler.Stack] (value [io.netty.util.Recycler$Stack@464c47d9]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
abr 08, 2014 10:29:07 AM org.apache.openejb.arquillian.common.TomEEContainer undeploy
INFO: cleaning /var/folders/k7/5t5fmkj547315vzzv51wh0fh0000gn/T/arquillian-tomee-app-working-dir/0
abr 08, 2014 10:29:07 AM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
abr 08, 2014 10:29:07 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-49322"]
abr 08, 2014 10:29:07 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-8009"]
abr 08, 2014 10:29:07 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
abr 08, 2014 10:29:07 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-49322"]
abr 08, 2014 10:29:12 AM org.apache.tomcat.util.net.AbstractEndpoint shutdownExecutor
WARNING: The executor associated with thread pool [http-bio-49322] has not fully shutdown. Some application threads may still be running.

[lordofthejars]

BTW you can try it by yourself:

https://github.com/arquillian/arquillian-extension-moco

and the important class here is the https://github.com/arquillian/arquillian-extension-moco/blob/master/src/main/java/org/jboss/arquillian/moco/container/MocoLifecycleExecuter.java and the test https://github.com/arquillian/arquillian-extension-moco/blob/master/src/test/java/org/jboss/arquillian/moco/MocoExtensionTestCase.java

[dreamhead]

It's Netty which didn't shutdown server as expected. I actually put a latency in the code from 0.9.1. Maybe I'll dig more.

[lordofthejars]

Ok I think it would be great to make some researching on this problem. I am thinking to open an issue to Netty people to see if they can address the problem because at least adding latency is only a patch not the real solution. Thank you so much.

[lordofthejars]

I have found a solution to this problem. I have read on Netty documentation that shutdownGracefully returns a Future class which means that you can synchronize and wait until threads are completely closed. So I have tried this modification on MocoServer class:

WARNING: it is not suitable for production code:

if (bossGroup != null) {
            try {
                bossGroup.shutdownGracefully().get();
            } catch(InterruptedException e) {
            } catch(ExecutionException e) {
            }
            bossGroup = null;
        }


        if (workerGroup != null) {
            try {
                workerGroup.shutdownGracefully().get();
                workerGroup = null;
            } catch(InterruptedException e) {
            } catch(ExecutionException e) {
            }

        }

And now it works perfectly. Notice that I have added the get() method. So if you agree I can send you a pull request with the modification.

BTW it would be awesome if I could have this issue fixed before 25th of April, do you think it will be possible to have a 0.9.2 version released before that date?

Thank you so much.

[dreamhead]

It's really a good suggestion. In my current plan, the next release will in May 1st. I can publish snapshot version anytime. Can I know more about your release plan?

[lordofthejars]

Well my plan is to release this weekend the first alpha version of the extension (because there are some users that are waiting for start using it 😄 Then for two weeks I expect their feedback and what can be improved and then release the alpha2, and after some more testing (and I would like that you as a creator of Moco also takes a look so you can suggest any missing point), I will release the first stable version.

In parallel of this on 25th of April I am going to talk in Barcelona JUG about Arquillian, NoMocks movement, Stubbs and so on and I would like to present Moco (and the extension) to all of them so I think that an snapshot version of Moco published would be awesome.

If you want I send you a pull request no problem, as you can see the change is really easy.

Thank you so much.

[lordofthejars]

Also I would like to use this issue to comment one issue I have found:

I am using next json file as expectation:

[
    {
        "request": {
            "uri": "/currencies/exchange_rates"
        },
        "response": {
            "json": {"foo":"bar"}
        }
    }
]

And I start Moco with next code:

List<InputStream> streams = new ArrayList<InputStream>();
                streams.add(Resources.getResource(mocoFile).openStream());
                jsonRunner = JsonRunner.newJsonRunnerWithStreams(
                        streams, of(port));

And next exception is thrown:

[http-bio-50884-exec-3] INFO Unrecognized field: Unrecognized field "json" (class com.github.dreamhead.moco.parser.model.ResponseSetting), not marked as ignorable (9 known properties: "headers", "path_resource", "text", "proxy", "cookies", "status", "version", "file", "latency"])
at [Source: sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream@7df82b9f; line: 7, column: 19](through reference chain: com.github.dreamhead.moco.parser.model.SessionSetting["response"]->com.github.dreamhead.moco.parser.model.ResponseSetting["json"])

What I am doing wrong here? If instead of json type I add text then it works as expected.

Thank you so much.

[dreamhead]

json api in json will be published in the next version. If you really want to use it, you can build Moco by yourself.

[lordofthejars]

Ok I have already done 👍 but do you think you could publish a 0.9.2.Alpha1 version on maven central repo? This would help me in publishing the first alpha of Arquillian-extension-Moco.

Thank you so much for your help.

[lordofthejars]

Do you want I send you a pull request for shutting down Runner or you will implement it? I ask it to close this issue or attach a PR.

[dreamhead]

It would be better if you can create PR with test cases for that.