Dev server sometimes hangs after reload

[dwickern]

The dev server sometimes hangs after changing some code and reloading. For example in this case it hung after the first reload:

--- (Running the application, auto-reloading is enabled) ---

[info] p.c.s.NettyServer - Listening for HTTP on /0.0.0.0:9000

(Server started, use Enter to stop and go back to the console...)

[info] Compiling 1 Scala source to  ...
[info] Done compiling.

--- (RELOAD) ---

At this point Play stops responding to any network request until I kill the process and restart it. Hitting Enter does not stop the server either.

Here's a thread dump after it hangs:
https://gist.github.com/dwickern/6274289d30b54468787143a7fc2c61d7

I can see several threads blocked here:

Thread 95251: (state = BLOCKED)
 - play.core.server.DevServerStart$$anon$1.get() @bci=0, line=126 (Interpreted frame)
 - play.core.server.netty.PlayRequestHandler.modelConversion() @bci=11, line=70 (Interpreted frame)
 - play.core.server.netty.PlayRequestHandler.handle(io.netty.channel.Channel, io.netty.handler.codec.http.HttpRequest) @bci=22, line=81 (Interpreted frame)
 - play.core.server.netty.PlayRequestHandler.channelRead(io.netty.channel.ChannelHandlerContext, java.lang.Object) @bci=57, line=184 (Interpreted frame)
 - io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(java.lang.Object) @bci=16, line=362 (Compiled frame)
 - io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(io.netty.channel.AbstractChannelHandlerContext, java.lang.Object) @bci=31, line=348 (Compiled frame)
 - io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(java.lang.Object) @bci=5, line=340 (Compiled frame)

and another relevant thread:

Thread 79627: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=997 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1304 (Interpreted frame)
 - scala.concurrent.impl.Promise$DefaultPromise.tryAwait(scala.concurrent.duration.Duration) @bci=70, line=238 (Interpreted frame)
 - scala.concurrent.impl.Promise$DefaultPromise.ready(scala.concurrent.duration.Duration, scala.concurrent.CanAwait) @bci=2, line=254 (Interpreted frame)
 - scala.concurrent.impl.Promise$DefaultPromise.ready(scala.concurrent.duration.Duration, scala.concurrent.CanAwait) @bci=3, line=183 (Interpreted frame)
 - scala.concurrent.Await$.$anonfun$ready$1(scala.concurrent.Awaitable, scala.concurrent.duration.Duration) @bci=5, line=191 (Interpreted frame)
 - scala.concurrent.Await$$$Lambda$9172.apply() @bci=8 (Interpreted frame)
 - akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block() @bci=12, line=167 (Interpreted frame)
 - akka.dispatch.forkjoin.ForkJoinPool.managedBlock(akka.dispatch.forkjoin.ForkJoinPool$ManagedBlocker) @bci=145, line=3641 (Interpreted frame)
 - akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(scala.Function0, scala.concurrent.CanAwait) @bci=21, line=165 (Interpreted frame)
 - akka.dispatch.BatchingExecutor$BlockableBatch.blockOn(scala.Function0, scala.concurrent.CanAwait) @bci=22, line=106 (Interpreted frame)
 - scala.concurrent.Await$.ready(scala.concurrent.Awaitable, scala.concurrent.duration.Duration) @bci=26, line=142 (Interpreted frame)
 - play.api.Play$.$anonfun$stop$1(play.api.Application) @bci=15, line=145 (Interpreted frame)
 - play.api.Play$$$Lambda$9170.apply() @bci=4 (Interpreted frame)
 - play.utils.Threads$.withContextClassLoader(java.lang.ClassLoader, scala.Function0) @bci=16, line=21 (Interpreted frame)
 - play.api.Play$.stop(play.api.Application) @bci=19, line=145 (Interpreted frame)
 - play.core.server.DevServerStart$$anon$1.$anonfun$reload$1(play.api.Application) @bci=4, line=148 (Interpreted frame)
 - play.core.server.DevServerStart$$anon$1.$anonfun$reload$1$adapted(play.api.Application) @bci=1, line=148 (Interpreted frame)
 - play.core.server.DevServerStart$$anon$1$$Lambda$4238.apply(java.lang.Object) @bci=4 (Interpreted frame)
 - scala.util.Success.foreach(scala.Function1) @bci=5, line=249 (Interpreted frame)
 - play.core.server.DevServerStart$$anon$1.reload(java.lang.ClassLoader) @bci=50, line=148 (Interpreted frame)
 - play.core.server.DevServerStart$$anon$1.get() @bci=33, line=124 (Interpreted frame)
 - play.core.server.netty.PlayRequestHandler.resultUtils() @bci=11, line=68 (Interpreted frame)
 - play.core.server.netty.PlayRequestHandler.$anonfun$handleAction$5(play.core.server.netty.PlayRequestHandler, play.api.mvc.RequestHeader, io.netty.handler.codec.http.HttpRequest, scala.Option, akka.stream.Materializer, play.api.mvc.Result) @bci=1, line=289 (Interpreted frame)
 - play.core.server.netty.PlayRequestHandler$$Lambda$5572.apply(java.lang.Object) @bci=24 (Interpreted frame)
 - scala.concurrent.Future.$anonfun$flatMap$1(scala.concurrent.Future, scala.Function1, scala.util.Try) @bci=19, line=304 (Compiled frame)
 - scala.concurrent.Future$$Lambda$4545.apply(java.lang.Object) @bci=12 (Compiled frame)
 - scala.concurrent.impl.Promise.$anonfun$transformWith$1(scala.concurrent.impl.Promise, scala.Function1, scala.concurrent.impl.Promise$DefaultPromise, scala.util.Try) @bci=2, line=37 (Compiled frame)
 - scala.concurrent.impl.Promise$$Lambda$4546.apply(java.lang.Object) @bci=16 (Compiled frame)
 - scala.concurrent.impl.CallbackRunnable.run() @bci=26, line=60 (Compiled frame)
 - play.api.libs.streams.Execution$trampoline$.executeScheduled() @bci=53, line=109 (Interpreted frame)
 - play.api.libs.streams.Execution$trampoline$.execute(java.lang.Runnable) @bci=29, line=71 (Compiled frame)
 - scala.concurrent.impl.CallbackRunnable.executeWithValue(scala.util.Try) @bci=28, line=68 (Compiled frame)
 - scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(scala.util.Try, scala.concurrent.impl.CallbackRunnable) @bci=2, line=284 (Compiled frame)
 - scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(scala.util.Try, scala.concurrent.impl.CallbackRunnable) @bci=2, line=284 (Compiled frame)
 - scala.concurrent.impl.Promise$DefaultPromise.tryComplete(scala.util.Try) @bci=63, line=284 (Compiled frame)
 - scala.concurrent.Promise.complete(scala.util.Try) @bci=2, line=49 (Compiled frame)
 - scala.concurrent.Promise.complete$(scala.concurrent.Promise, scala.util.Try) @bci=2, line=48 (Compiled frame)
 - scala.concurrent.impl.Promise$DefaultPromise.complete(scala.util.Try) @bci=2, line=183 (Compiled frame)
 - scala.concurrent.impl.Promise.$anonfun$transformWith$1(scala.concurrent.impl.Promise, scala.Function1, scala.concurrent.impl.Promise$DefaultPromise, scala.util.Try) @bci=20, line=38 (Compiled frame)
 - scala.concurrent.impl.Promise$$Lambda$4546.apply(java.lang.Object) @bci=16 (Compiled frame)
 - scala.concurrent.impl.CallbackRunnable.run() @bci=26, line=60 (Compiled frame)
 - akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(akka.dispatch.BatchingExecutor$AbstractBatch) @bci=19, line=55 (Compiled frame)
 - akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(akka.dispatch.BatchingExecutor$BlockableBatch, boolean) @bci=2, line=91 (Compiled frame)
 - akka.dispatch.BatchingExecutor$BlockableBatch$$Lambda$4540.apply$mcV$sp() @bci=8 (Compiled frame)
 - scala.runtime.java8.JFunction0$mcV$sp.apply() @bci=1, line=12 (Compiled frame)
 - scala.concurrent.BlockContext$.withBlockContext(scala.concurrent.BlockContext, scala.Function0) @bci=20, line=81 (Compiled frame)
 - akka.dispatch.BatchingExecutor$BlockableBatch.run() @bci=96, line=91 (Compiled frame)
 - akka.dispatch.TaskInvocation.run() @bci=4, line=40 (Compiled frame)
 - akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec() @bci=4, line=43 (Compiled frame)
 - akka.dispatch.forkjoin.ForkJoinTask.doExec() @bci=10, line=260 (Compiled frame)
 - akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(akka.dispatch.forkjoin.ForkJoinTask) @bci=10, line=1339 (Compiled frame)
 - akka.dispatch.forkjoin.ForkJoinPool.runWorker(akka.dispatch.forkjoin.ForkJoinPool$WorkQueue) @bci=11, line=1979 (Interpreted frame)
 - akka.dispatch.forkjoin.ForkJoinWorkerThread.run() @bci=14, line=107 (Interpreted frame)

---

Play Version (2.5.x / etc)

2.6.7

API (Scala / Java / Neither / Both)

Scala

Operating System (Ubuntu 15.10 / MacOS 10.10 / Windows 10)

OSX

JDK (Oracle 1.8.0_72, OpenJDK 1.8.x, Azul Zing)

1.8u112