Allow the embedded web server to be shut down gracefully

We are using spring-boot and spring-cloud to realize a micro service archtecture. During load tests we are facing lots of errors such as "entitymanager closed and others" if we shut the micro service down during load.
We are wondering if there is an option to configure the embedded container to shut its service connector down and waits for an empty request queue before shutting down the complete application context.

If there is no such option, I did not find any, then it would be great to extend the shutdownhook of spring to respect such requirements.

We currently stop the application context and then stop the container. We did try to reverse this ordering but it had some unwanted side effects so we're stuck with it for now at least. That's the bad news.

The good news is that you can actually get a graceful shutdown yourself if you're happy to get your hands a bit dirty. The gist is that you need to pause Tomcat's connector and then wait for its thread pool to shutdown before allowing the destruction of the application context to proceed. It looks something like this:

package com.example;

import java.util.concurrent.Executor;
import java.util.concurrent.TimeUnit;

import org.apache.catalina.connector.Connector;
import org.apache.tomcat.util.threads.ThreadPoolExecutor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.embedded.ConfigurableEmbeddedServletContainer;
import org.springframework.boot.context.embedded.EmbeddedServletContainerCustomizer;
import org.springframework.boot.context.embedded.tomcat.TomcatConnectorCustomizer;
import org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory;
import org.springframework.context.ApplicationListener;
import org.springframework.context.annotation.Bean;
import org.springframework.context.event.ContextClosedEvent;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class Gh4657Application {

    public static void main(String[] args) {
        SpringApplication.run(Gh4657Application.class, args);
    }

    @RequestMapping("/pause")
    public String pause() throws InterruptedException {
        Thread.sleep(10000);
        return "Pause complete";
    }

    @Bean
    public GracefulShutdown gracefulShutdown() {
        return new GracefulShutdown();
    }

    @Bean
    public EmbeddedServletContainerCustomizer tomcatCustomizer() {
        return new EmbeddedServletContainerCustomizer() {

            @Override
            public void customize(ConfigurableEmbeddedServletContainer container) {
                if (container instanceof TomcatEmbeddedServletContainerFactory) {
                    ((TomcatEmbeddedServletContainerFactory) container)
                            .addConnectorCustomizers(gracefulShutdown());
                }

            }
        };
    }

    private static class GracefulShutdown implements TomcatConnectorCustomizer,
            ApplicationListener<ContextClosedEvent> {

        private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);

        private volatile Connector connector;

        @Override
        public void customize(Connector connector) {
            this.connector = connector;
        }

        @Override
        public void onApplicationEvent(ContextClosedEvent event) {
            this.connector.pause();
            Executor executor = this.connector.getProtocolHandler().getExecutor();
            if (executor instanceof ThreadPoolExecutor) {
                try {
                    ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
                    threadPoolExecutor.shutdown();
                    if (!threadPoolExecutor.awaitTermination(30, TimeUnit.SECONDS)) {
                        log.warn("Tomcat thread pool did not shut down gracefully within "
                                + "30 seconds. Proceeding with forceful shutdown");
                    }
                }
                catch (InterruptedException ex) {
                    Thread.currentThread().interrupt();
                }
            }
        }

    }

}

I think it makes sense to provide behaviour like this out of the box, or at least an option to enable it. However, that'll require a bit more work as it'll need to work with all of the embedded containers that we support (Jetty, Tomcat, and Undertow), cope with multiple connectors (HTTP and HTTPS, for example) and we'll need to think about the configuration options, if any, that we'd want to offer: switching it on or off, configuring the period that we wait for the thread pool to shutdown, etc.

Thank you for your advice.
We currently simply shut down the embedded container and the waiting a short time before closing the application context.

What do you thing, will it be possible in future to shut down a spring boot application more gracefully?

What do you thing, will it be possible in future to shut down a spring boot application more gracefully?

Yes. As I said above "I think it makes sense to provide behaviour like this out of the box, or at least an option to enable it".

I'm going to re-open this issue as we'll use it to track the possible enhancement.

@wilkinsona

+1 on this request. We ran into a similar problem when load testing and dropping a node from the test. @wilkinsona in your example, I was thinking of using an implementation of smartlifecylce so I can insure the connector is shutdown first. You said you ran into issues shutting down tomcat first?

I think the right order is:

pause the io endpoint

web container just pasue(deny new request come in),and RPC framework need notify client don't send new request and wait current request be proccessed
wait the request to be processed and response client
close the io endpoint
close spring service
close log system

so we do it as follows:

    @Override
    public void finished(ConfigurableApplicationContext context, Throwable exception) {
        if (exception == null) {
            //install UncaughtExceptionHandler
            UncaughtExceptionHandlerWrapper.install();
            //when system startup ,register shutdown hooks to clean resouces.
            Runtime.getRuntime().addShutdownHook(new Thread() {
                @Override
                public void run() {
                    //run all shutdown hooks before spring container avoid service dependency
                    ShutdownHooks.shutdownAll();
                    //close spring container
                    context.close();
                    shutdownLogSystem();
                }
            });
            //log startup info
            LoggerFactory.getLogger(YijiApplicationRunListener.class).info("启动成功: http://127.0.0.1:{}",
            context.getEnvironment().getProperty(Apps.HTTP_PORT));
        } else {
            ShutdownHooks.shutdownAll();
            shutdownLogSystem();
        }
    }

This issue is actually a bit more complicated.

I know that SmartLifeCycle can be used to set the order in which beans are notified of life cycle events. However, there is no generalized way for a service to know it should startup/shutdown before another service.

Consider the following:

A sprint boot application is running an embedded servlet container and is also producing/consuming JMS messages.

On the close event, the servlet container really needs to pause the connector first, process its remain working (any connections that have already been establish.)

We need to insure this is the FIRST thing that happens, prior to the JMS infrastructure shutting down because the work done inside tomcat may rely on JMS.

The JMS infrastructure has a similar requirement: it must stop listening for messages and chew through any remaining messages it has already accepted.

I can certainly implement a SmartLifeCycle class that sets the phase "very high"....and I could even create two instances, one for embedded servlet container and one for JMS and insure the correct order.

But in the spirit of Spring Boot, if I add a tomcat container, its my expectation that when the application shuts down, it will gracefully stop accepting connections, process remaining work, and exit.

It would be helpful if there was a mechanism to allow ordering to be expressed relative to other services "JMS must start before tomcat", "tomcat must close before JMS". This would be similar to the AutoConfigureBefore/AutoConfigureAfter annotations that are used in Spring boot.

One way to approach this might be to create an enum for the generalized services (This is not ideal, but I can't think of another way without introducing artificial, compile time dependencies.):

EMBEDDED_CONTAINER
JMS
DISCOVERY_CLIENT
.
.
The annotations could leverage the enums to order the life cycle listeners.

For now, its up to the developer to explicitly define the shutdown order of services via "SmartLifeCycle" instances...which can get a bit messy and seems like extra work for something that should really work out of the box.

@tkvangorder

@tkvangorder You don't need to use SmartLifecycleto gracefully shut down Tomcat as shown in my comment above. It happens in response to a ContextClosedEvent which is fired at the very beginning of the context's close processing before any of the beans in the context are processed.

Beyond this, Spring Framework already has support for closing things down in the correct order. For example you can implement DisposableBean. When the container disposes of a bean, it will dispose of anything that depends on it first.

@wilkinsona

@wilkinsona is the order of shutdown anyway guaranteed ? Is spring boot always going to shutdown application context first? we are trying to implement what you said, but was not sure if this will change in the future.

Is spring boot always going to shutdown application context first?

Yes. When you're using an embedded container, it's the application context being shut down that triggers the shutdown of the container.

When Kubernetes is taking a pod out of service it will send the SIGTERM signal to it. This will cause a Spring Boot application to start shutting down. At this point, the liveness and readiness probes are no longer called.

In parallel, Kubernetes will also initiate the sequence of events that results in requests no longer being routed to the application. The parallel and eventual nature of this means that there is a period of time when the application has received SIGTERM but requests are still routed to it. Ideally, these requests should be handled so that clients do not become aware of the application being shut down.

The recommended way to minimise or perhaps even eliminate the window where requests are still received after SIGTERM is to sleep in a pre-stop hook. In the event of the window only being minimised, continuing to accept new requests during graceful shutdown should reduce the number of errors that are seen by clients. Such errors cannot be eliminated entirely as, if there are no active requests, the graceful shutdown may complete and stop accepting new connections while Kubernetes contains to route requests to the application.

This is correct. Its also documented in the book : Kubernetes in Action (ISBN: 9781617293726), under Chapter 17. Best practices for developing apps.

I think the sleep based configuration would be good enough in our case (infact we did the same thing in our application). I expect that the configuration server.shutdown.grace-period will be well-documented so that users will know what exactly will happen when its configured.

Thanks for taking this up.

@wilkinsona

Hi, I have a spring-boot app with HTTP endpoints and Kafka consumers. I have tested this graceful shutdown implementation and it looks like the whole shutdown is delayed waiting for HTTP with Thread.sleep(), so no other components are notified. This means I need 2*n graceful period if I wanted to give HTTP and Kafka n seconds to finish their thing.

@wilkinsona I saw your comment about not needing SmartLifecycle just for tomcat, but have you considered other components needing graceful shutdown as well? In my case, I would like to leave some time for both @controllers and Kafka Listeners to be able to finish already started processing -- which in some cases can make external calls with exponential backoff, so I would also like to be able to check in retry loops if shutdown was initiated.

I'm thinking about such logic:

shutdown initiated
components notified about shutdown, should not block, clock starts
- HTTP connections no longer accepted
- kafka consumers stop polling new messages
- retry loops can early exit, throwing "retry-able" exception
when time runs out, interrupt processing threads
finish shutdown, cleanup

Would you consider changing the above logic, or recommend a custom implementation instead?

@wilkinsona

Hi, I have a spring-boot app with HTTP endpoints and Kafka consumers. I have tested this graceful shutdown implementation and it looks like the whole shutdown is delayed waiting for HTTP with Thread.sleep(), so no other components are notified. This means I need 2*n graceful period if I wanted to give HTTP and Kafka n seconds to finish their thing.

@wilkinsona I saw your comment about not needing SmartLifecycle just for tomcat, but have you considered other components needing graceful shutdown as well? In my case, I would like to leave some time for both @controllers and Kafka Listeners to be able to finish already started processing -- which in some cases can make external calls with exponential backoff, so I would also like to be able to check in retry loops if shutdown was initiated.

I'm thinking about such logic:

shutdown initiated

components notified about shutdown, should not block, clock starts

HTTP connections no longer accepted

kafka consumers stop polling new messages

retry loops can early exit, throwing "retry-able" exception

when time runs out, interrupt processing threads

finish shutdown, cleanup

Would you consider changing the above logic, or recommend a custom implementation instead?

@gnagy Even in our deployment, we have HTTP endpoints and Kafka consumers. So we want to stop Kafka consumers from consuming the messages and processes already consumed messaged within the grace period.
Did you end up implementing a custom method? Or overriding GracefulShutdown implementation?

@gnagy

@gnagy Even in our deployment, we have HTTP endpoints and Kafka consumers. So we want to stop Kafka consumers from consuming the messages and processes already consumed messaged within the grace period.
Did you end up implementing a custom method? Or overriding GracefulShutdown implementation?

I no longer have access to that project but if I remember correctly Spring Kafka integration already supported stop()-ing the listener container when the application context is shutting down, we just had to make sure our custom RetryTemplates in the consumers got notified as well.

@gnagy

@gnagy Thanks. I saw spring-Kafka has stop() method on application context shutdown.
I think it would be nice to test rolling upgrade under load to see if any messages are lost during shutdown

Now there is the graceful shutdown feature (https://docs.spring.io/spring-boot/docs/current/reference/html/web.html#web.graceful-shutdown), but this feature uses a timeout, and only waits within the timeout. But sometimes I want to wait till all the requests are processed, without a timeout. Could you please support this?

@zdaccount

@zdaccount what would happen if one of those outstanding requests is blocked and never finishes? The application instance would then stay up for ever?

Yes, sometimes I want that. If the application instance hangs, then I may force kill it, but sometimes I don't want the instance itself to do this.

I think that this is like how the web server processes requests. When the web server process a request, there is no timeout on the process logic (I assume this, but I may be wrong, since I am new to Spring Boot). If this is the case, then it is also reasonable to have no timeout for the last requests.

I don’t think it’s reasonable to have no timeout at all. For one, it is highly unlikely that any network connection will survive indefinitely so the server will be unable to send a response.

Instead, I would recommend configuring a timeout that’s slightly greater than the maximum time that you would wait before manually killing the application instance.

I don’t think it’s reasonable to have no timeout at all. For one, it is highly unlikely that any network connection will survive indefinitely so the server will be unable to send a response.

Instead, I would recommend configuring a timeout that’s slightly greater than the maximum time that you would wait before manually killing the application instance.

But there is no timeout on the ordinary processing of requests, right? If this is the case, then it seems also reasonable to have no timeout on the processing of last requests.

Requests that have the potential to be long-running should be async and will then be subject to the async request timeout. This timeout is in addition to timeouts at the network level.

In a Spring MVC app, a request can be made async by returning DeferredResult, among others, from a controller.

When I am testing, I am seeing the container (Tomcat) getting shutdown before the application context / beans have shutdown. From logs, the order is:

Shutdown hooks (Runtime.addShutdownHook)
Graceful container shutdown (Tomcat)
Application context destroyed (ServletContextListener.contextDestroyed)
Spring application @PreDestroy

Which is problematic for me: I have some async threads running some computations, and the server is publishing metrics/health of that computation via its API. When shutdown is triggered, I want the computations to gracefully exit, but keep the server running and accepting connections to have metrics/health during that shutdown period. I can wait for the computation to finish in Application context or Predestroy hooks, but the tomcat server shuts down anyways. Any solution that would allow the server to stay up while the async thread is finishing its computation?

You'll have to implement SmartLifecycle with an appropriate phase so that you can participate in the lifecycle. Your implementation can then wait for the async tasks to complete. You'll want it to be called to stop before WebServerStartStopLifecycle is called. This ordering is controlled by the phase. You may also want to consider the phase of WebServerGracefulShutdownLifecycle.

If you have any further questions, please follow up on Stack Overflow. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements.

Follow up, for those reading this thread in the future

Navigation Menu

Repository navigation

Allow the embedded web server to be shut down gracefully #4657

116 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allow the embedded web server to be shut down gracefully #4657

Description

Activity

wilkinsona commented on Dec 2, 2015

LutzStrobel commented on Dec 4, 2015

wilkinsona commented on Dec 4, 2015

tkvangorder commented on Jan 19, 2016

bohrqiu commented on Jan 19, 2016

tkvangorder commented on Jan 19, 2016

wilkinsona commented on Jan 19, 2016

thefallentree commented on Mar 8, 2016

wilkinsona commented on Mar 8, 2016

116 remaining items

pradeepanchan commented on Mar 13, 2020

gnagy commented on May 4, 2020

dheerajjoshim commented on Sep 6, 2021

gnagy commented on Sep 7, 2021

dheerajjoshim commented on Sep 7, 2021

zdaccount commented on Mar 18, 2022

bclozel commented on Mar 18, 2022

zdaccount commented on Mar 18, 2022

wilkinsona commented on Mar 18, 2022

zdaccount commented on Mar 18, 2022

wilkinsona commented on Mar 18, 2022

Azmisov commented on Apr 18, 2024

wilkinsona commented on Apr 18, 2024

Azmisov commented on Apr 18, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions