delay between tests that use DropwizardExtensionsSupport

[Athou]

Hey there,

I've been using dropwizard for CommaFeed for almost 10 years now, thank you for the great work, it's been a joy to work with! 😃

I updated to 4.0.5 from 2.1.6 and everything works fine. However, I just noticed that the integration tests take a lot more time now, there's a ~2s delay between tests during server shutdown that was not happening before:

INFO  [2024-01-05 21:25:35,322] org.eclipse.jetty.server.Server: Stopped Server@472dc215{STOPPING}[11.0.19,sto=30000]
INFO  [2024-01-05 21:25:35,322] org.eclipse.jetty.server.Server: Shutdown Server@472dc215{STOPPING}[11.0.19,sto=30000]
INFO  [2024-01-05 21:25:37,327] org.eclipse.jetty.server.AbstractConnector: Stopped application@696a6506{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
INFO  [2024-01-05 21:25:37,332] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@6ed238c2{/,null,STOPPED}

The regression is noticeable since this commit Athou/commafeed@fd0e542
The delay does not seem to happen when the app is shutdown when running normally, it only seem to happen when tests annotated with @ExtendWith(DropwizardExtensionsSupport.class) are running.

Is this something you noticed too? Do you have any pointers as to where to look or why this is happening?

Thanks!

[Athou]

Alright so I slept on it and enabled debugging:

INFO  [2024-01-06 07:26:44,205] org.eclipse.jetty.server.Server: Shutdown Server@a15e3c1{STOPPING}[11.0.19,sto=30000]
DEBUG [2024-01-06 07:26:44,206] org.eclipse.jetty.server.Server: graceful application@16448827{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
DEBUG [2024-01-06 07:26:44,206] org.eclipse.jetty.server.Server: graceful i.d.j.MutableServletContextHandler@6f1c3f18{/,null,AVAILABLE}
DEBUG [2024-01-06 07:26:44,206] org.eclipse.jetty.server.Server: graceful StatisticsHandler@608b74f3{STARTED,r=0,d=0}
DEBUG [2024-01-06 07:26:44,206] org.eclipse.jetty.server.Server: graceful SessionTracker@7ec49d9b{STARTED}
DEBUG [2024-01-06 07:26:44,207] org.eclipse.jetty.util.thread.QueuedThreadPool: ran pool-10-thread-12-acceptor-1@2ef4e7f3-application@16448827{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} in InstrumentedQueuedThreadPool[dw]@1ebcfcf1{STARTED,8<=14<=1024,i=1,r=-1,t=59527ms,q=0}[ReservedThreadExecutor@100d5018{reserved=1/20,pending=0}]
DEBUG [2024-01-06 07:26:44,207] org.eclipse.jetty.util.thread.QueuedThreadPool: ran pool-10-thread-11-acceptor-0@3bf47d1e-application@16448827{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} in InstrumentedQueuedThreadPool[dw]@1ebcfcf1{STARTED,8<=14<=1024,i=1,r=-1,t=59527ms,q=0}[ReservedThreadExecutor@100d5018{reserved=1/20,pending=0}]
DEBUG [2024-01-06 07:26:44,207] org.eclipse.jetty.util.thread.QueuedThreadPool: Evict check, period=60000ms InstrumentedQueuedThreadPool[dw]@1ebcfcf1{STARTED,8<=14<=1024,i=3,r=-1,t=59527ms,q=0}[ReservedThreadExecutor@100d5018{reserved=1/20,pending=0}]
DEBUG [2024-01-06 07:26:44,207] org.eclipse.jetty.util.thread.QueuedThreadPool: Evict check, period=60000ms InstrumentedQueuedThreadPool[dw]@1ebcfcf1{STARTED,8<=14<=1024,i=3,r=-1,t=59527ms,q=0}[ReservedThreadExecutor@100d5018{reserved=1/20,pending=0}]
DEBUG [2024-01-06 07:26:44,207] org.eclipse.jetty.util.thread.QueuedThreadPool: Evict skipped, threshold=119527ms in the future InstrumentedQueuedThreadPool[dw]@1ebcfcf1{STARTED,8<=14<=1024,i=3,r=-1,t=59527ms,q=0}[ReservedThreadExecutor@100d5018{reserved=1/20,pending=0}]
DEBUG [2024-01-06 07:26:44,207] org.eclipse.jetty.util.thread.QueuedThreadPool: Evict skipped, threshold=119527ms in the future InstrumentedQueuedThreadPool[dw]@1ebcfcf1{STARTED,8<=14<=1024,i=3,r=-1,t=59527ms,q=0}[ReservedThreadExecutor@100d5018{reserved=1/20,pending=0}]
DEBUG [2024-01-06 07:26:44,207] org.eclipse.jetty.io.IdleTimeout: Setting idle timeout 30000 -> 1000 on SocketChannelEndPoint@60b32495[{l=/127.0.0.1:8080,r=/127.0.0.1:58811,OPEN,fill=FI,flush=-,to=4/1000}{io=1/1,kio=1,kro=1}]->[HttpConnection@528ca39c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@7bb75668{s=START}]=>HttpChannelOverHttp@6e338fa8{s=HttpChannelState@9adf938{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=2,c=false/false,a=IDLE,uri=null,age=0}]
DEBUG [2024-01-06 07:26:44,207] org.eclipse.jetty.io.IdleTimeout: SocketChannelEndPoint@60b32495[{l=/127.0.0.1:8080,r=/127.0.0.1:58811,OPEN,fill=FI,flush=-,to=4/1000}{io=1/1,kio=1,kro=1}]->[HttpConnection@528ca39c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@7bb75668{s=START}]=>HttpChannelOverHttp@6e338fa8{s=HttpChannelState@9adf938{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=2,c=false/false,a=IDLE,uri=null,age=0}] idle timeout check, elapsed: 4 ms, remaining: 996 ms
DEBUG [2024-01-06 07:26:45,204] org.eclipse.jetty.io.IdleTimeout: SocketChannelEndPoint@60b32495[{l=/127.0.0.1:8080,r=/127.0.0.1:58811,OPEN,fill=FI,flush=-,to=1001/1000}{io=1/1,kio=1,kro=1}]->[HttpConnection@528ca39c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@7bb75668{s=START}]=>HttpChannelOverHttp@6e338fa8{s=HttpChannelState@9adf938{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=2,c=false/false,a=IDLE,uri=null,age=0}] idle timeout check, elapsed: 1001 ms, remaining: -1 ms
DEBUG [2024-01-06 07:26:45,204] org.eclipse.jetty.io.IdleTimeout: SocketChannelEndPoint@60b32495[{l=/127.0.0.1:8080,r=/127.0.0.1:58811,OPEN,fill=FI,flush=-,to=1001/1000}{io=1/1,kio=1,kro=1}]->[HttpConnection@528ca39c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@7bb75668{s=START}]=>HttpChannelOverHttp@6e338fa8{s=HttpChannelState@9adf938{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=2,c=false/false,a=IDLE,uri=null,age=0}] idle timeout expired
DEBUG [2024-01-06 07:26:45,204] org.eclipse.jetty.io.FillInterest: onFail FillInterest@7e88356{ReadCallback@242e49e4{HttpConnection@528ca39c::SocketChannelEndPoint@60b32495[{l=/127.0.0.1:8080,r=/127.0.0.1:58811,OPEN,fill=FI,flush=-,to=1002/1000}{io=1/1,kio=1,kro=1}]->[HttpConnection@528ca39c[p=HttpParser{s=START,0 of -1},g=HttpGenerator@7bb75668{s=START}]=>HttpChannelOverHttp@6e338fa8{s=HttpChannelState@9adf938{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=2,c=false/false,a=IDLE,uri=null,age=0}]}}
! java.util.concurrent.TimeoutException: Idle timeout expired: 1001/1000 ms
! at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:170)
! at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:112)
! at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
! at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
! at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
! at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
! at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
! at java.base/java.lang.Thread.run(Thread.java:833)

It seems that jetty is waiting for HTTP connections to close before shutdown. It happens twice in the log (1s for each connection so it explains my 2s delay).
I wrote a test that doesn't use the extension HTTP client and it shuts down immediately, so there seems to be dangling idle connections.

I added a cleanup method to close the HTTP client after each test and that solves the issue, the tests now run as fast as before.

@AfterEach
void cleanup() {
    this.client.close();
}

Am I doing something wrong in my tests? I don't think I'm leaving open connections.

Should the documentation be updated to include a cleanup method, or should DropwizardExtensionsSupport maybe be modified to close the HTTP client after each test?

[zUniQueX]

Hi @Athou. The problem you describe is related to the handling of HTTP connections. As the Mozilla docs state, the default value for the Connection header for HTTP/1.1 is keep-alive. So the connections won't be closed immediately. If you call client.close(), the client will terminate the open connections, so Jetty won't wait for expiration.

Regarding the exception in the Jetty debug logs: The Jetty idleTimeout property controls the keep-alive duration. This duration can be set in the connector properties. If you set the value to 1ms, the server will shut down immediately without the client having to be closed.

However, the much cleaner option for Dropwizard is to set this with a client property. Dropwizard 3.x and 4.x use the Apache5ConnectorProvider. For that provider there is the property Apache5ClientProperties.REUSE_STRATEGY that can be set to control the keep-alive behavior. When you add the following line to your test setup, the application should shut down immediately after finishing the test:

client.property(Apache5ClientProperties.REUSE_STRATEGY, (ConnectionReuseStrategy) (request, response, context) -> false);

That being said, the behavior to keep connections alive is common practice and shouldn't be changes for the testing package IMHO. With this viable workaround, that should be fine.