Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow the embedded web server to be shut down gracefully #4657

Closed
LutzStrobel opened this issue Dec 2, 2015 · 100 comments
Closed

Allow the embedded web server to be shut down gracefully #4657

LutzStrobel opened this issue Dec 2, 2015 · 100 comments

Comments

@LutzStrobel
Copy link

@LutzStrobel LutzStrobel commented Dec 2, 2015

We are using spring-boot and spring-cloud to realize a micro service archtecture. During load tests we are facing lots of errors such as "entitymanager closed and others" if we shut the micro service down during load.
We are wondering if there is an option to configure the embedded container to shut its service connector down and waits for an empty request queue before shutting down the complete application context.

If there is no such option, I did not find any, then it would be great to extend the shutdownhook of spring to respect such requirements.

@wilkinsona
Copy link
Member

@wilkinsona wilkinsona commented Dec 2, 2015

We currently stop the application context and then stop the container. We did try to reverse this ordering but it had some unwanted side effects so we're stuck with it for now at least. That's the bad news.

The good news is that you can actually get a graceful shutdown yourself if you're happy to get your hands a bit dirty. The gist is that you need to pause Tomcat's connector and then wait for its thread pool to shutdown before allowing the destruction of the application context to proceed. It looks something like this:

package com.example;

import java.util.concurrent.Executor;
import java.util.concurrent.TimeUnit;

import org.apache.catalina.connector.Connector;
import org.apache.tomcat.util.threads.ThreadPoolExecutor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.embedded.ConfigurableEmbeddedServletContainer;
import org.springframework.boot.context.embedded.EmbeddedServletContainerCustomizer;
import org.springframework.boot.context.embedded.tomcat.TomcatConnectorCustomizer;
import org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory;
import org.springframework.context.ApplicationListener;
import org.springframework.context.annotation.Bean;
import org.springframework.context.event.ContextClosedEvent;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class Gh4657Application {

    public static void main(String[] args) {
        SpringApplication.run(Gh4657Application.class, args);
    }

    @RequestMapping("/pause")
    public String pause() throws InterruptedException {
        Thread.sleep(10000);
        return "Pause complete";
    }

    @Bean
    public GracefulShutdown gracefulShutdown() {
        return new GracefulShutdown();
    }

    @Bean
    public EmbeddedServletContainerCustomizer tomcatCustomizer() {
        return new EmbeddedServletContainerCustomizer() {

            @Override
            public void customize(ConfigurableEmbeddedServletContainer container) {
                if (container instanceof TomcatEmbeddedServletContainerFactory) {
                    ((TomcatEmbeddedServletContainerFactory) container)
                            .addConnectorCustomizers(gracefulShutdown());
                }

            }
        };
    }

    private static class GracefulShutdown implements TomcatConnectorCustomizer,
            ApplicationListener<ContextClosedEvent> {

        private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);

        private volatile Connector connector;

        @Override
        public void customize(Connector connector) {
            this.connector = connector;
        }

        @Override
        public void onApplicationEvent(ContextClosedEvent event) {
            this.connector.pause();
            Executor executor = this.connector.getProtocolHandler().getExecutor();
            if (executor instanceof ThreadPoolExecutor) {
                try {
                    ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
                    threadPoolExecutor.shutdown();
                    if (!threadPoolExecutor.awaitTermination(30, TimeUnit.SECONDS)) {
                        log.warn("Tomcat thread pool did not shut down gracefully within "
                                + "30 seconds. Proceeding with forceful shutdown");
                    }
                }
                catch (InterruptedException ex) {
                    Thread.currentThread().interrupt();
                }
            }
        }

    }

}

I think it makes sense to provide behaviour like this out of the box, or at least an option to enable it. However, that'll require a bit more work as it'll need to work with all of the embedded containers that we support (Jetty, Tomcat, and Undertow), cope with multiple connectors (HTTP and HTTPS, for example) and we'll need to think about the configuration options, if any, that we'd want to offer: switching it on or off, configuring the period that we wait for the thread pool to shutdown, etc.

@wilkinsona wilkinsona changed the title graceful shutdown of embedded container should be possible Shut down embedded servlet container gracefully Dec 2, 2015
@LutzStrobel
Copy link
Author

@LutzStrobel LutzStrobel commented Dec 4, 2015

Thank you for your advice.
We currently simply shut down the embedded container and the waiting a short time before closing the application context.

What do you thing, will it be possible in future to shut down a spring boot application more gracefully?

@LutzStrobel LutzStrobel closed this Dec 4, 2015
@wilkinsona
Copy link
Member

@wilkinsona wilkinsona commented Dec 4, 2015

What do you thing, will it be possible in future to shut down a spring boot application more gracefully?

Yes. As I said above "I think it makes sense to provide behaviour like this out of the box, or at least an option to enable it".

I'm going to re-open this issue as we'll use it to track the possible enhancement.

@wilkinsona wilkinsona reopened this Dec 4, 2015
@tkvangorder
Copy link
Contributor

@tkvangorder tkvangorder commented Jan 19, 2016

+1 on this request. We ran into a similar problem when load testing and dropping a node from the test. @wilkinsona in your example, I was thinking of using an implementation of smartlifecylce so I can insure the connector is shutdown first. You said you ran into issues shutting down tomcat first?

@bohrqiu
Copy link
Contributor

@bohrqiu bohrqiu commented Jan 19, 2016

I think the right order is:

  1. pause the io endpoint

    web container just pasue(deny new request come in),and RPC framework need notify client don't send new request and wait current request be proccessed

  2. wait the request to be processed and response client

  3. close the io endpoint

  4. close spring service

  5. close log system

so we do it as follows:

    @Override
    public void finished(ConfigurableApplicationContext context, Throwable exception) {
        if (exception == null) {
            //install UncaughtExceptionHandler
            UncaughtExceptionHandlerWrapper.install();
            //when system startup ,register shutdown hooks to clean resouces.
            Runtime.getRuntime().addShutdownHook(new Thread() {
                @Override
                public void run() {
                    //run all shutdown hooks before spring container avoid service dependency
                    ShutdownHooks.shutdownAll();
                    //close spring container
                    context.close();
                    shutdownLogSystem();
                }
            });
            //log startup info
            LoggerFactory.getLogger(YijiApplicationRunListener.class).info("启动成功: http://127.0.0.1:{}",
            context.getEnvironment().getProperty(Apps.HTTP_PORT));
        } else {
            ShutdownHooks.shutdownAll();
            shutdownLogSystem();
        }
    }
@tkvangorder
Copy link
Contributor

@tkvangorder tkvangorder commented Jan 19, 2016

This issue is actually a bit more complicated.

I know that SmartLifeCycle can be used to set the order in which beans are notified of life cycle events. However, there is no generalized way for a service to know it should startup/shutdown before another service.

Consider the following:

A sprint boot application is running an embedded servlet container and is also producing/consuming JMS messages.

On the close event, the servlet container really needs to pause the connector first, process its remain working (any connections that have already been establish.)

We need to insure this is the FIRST thing that happens, prior to the JMS infrastructure shutting down because the work done inside tomcat may rely on JMS.

The JMS infrastructure has a similar requirement: it must stop listening for messages and chew through any remaining messages it has already accepted.

I can certainly implement a SmartLifeCycle class that sets the phase "very high"....and I could even create two instances, one for embedded servlet container and one for JMS and insure the correct order.

But in the spirit of Spring Boot, if I add a tomcat container, its my expectation that when the application shuts down, it will gracefully stop accepting connections, process remaining work, and exit.

It would be helpful if there was a mechanism to allow ordering to be expressed relative to other services "JMS must start before tomcat", "tomcat must close before JMS". This would be similar to the AutoConfigureBefore/AutoConfigureAfter annotations that are used in Spring boot.

One way to approach this might be to create an enum for the generalized services (This is not ideal, but I can't think of another way without introducing artificial, compile time dependencies.):

EMBEDDED_CONTAINER
JMS
DISCOVERY_CLIENT
.
.
The annotations could leverage the enums to order the life cycle listeners.

For now, its up to the developer to explicitly define the shutdown order of services via "SmartLifeCycle" instances...which can get a bit messy and seems like extra work for something that should really work out of the box.

@wilkinsona
Copy link
Member

@wilkinsona wilkinsona commented Jan 19, 2016

@tkvangorder You don't need to use SmartLifecycleto gracefully shut down Tomcat as shown in my comment above. It happens in response to a ContextClosedEvent which is fired at the very beginning of the context's close processing before any of the beans in the context are processed.

Beyond this, Spring Framework already has support for closing things down in the correct order. For example you can implement DisposableBean. When the container disposes of a bean, it will dispose of anything that depends on it first.

@thefallentree
Copy link

@thefallentree thefallentree commented Mar 8, 2016

@wilkinsona is the order of shutdown anyway guaranteed ? Is spring boot always going to shutdown application context first? we are trying to implement what you said, but was not sure if this will change in the future.

@wilkinsona
Copy link
Member

@wilkinsona wilkinsona commented Mar 8, 2016

Is spring boot always going to shutdown application context first?

Yes. When you're using an embedded container, it's the application context being shut down that triggers the shutdown of the container.

@thefallentree
Copy link

@thefallentree thefallentree commented Mar 12, 2016

I want to report back on how to setup this correctly using Jetty and springboot , use this class instead of EmbeddedServletAutoconfiguration and you are set.

public class HttpConfig {
    private static final Logger log = LoggerFactory.getLogger(HttpConfig.class);

    private static volatile Server server;

    // Jetty HTTP Server
    //
    // see [1] on how to implement graceful shutdown in springboot.
    // Note that since use jetty, we need to use server.stop(), also StatisticsHandler must be
    // configured, for jetty graceful shutdown to work.
    //
    // [1]: https://github.com/spring-projects/spring-boot/issues/4657
    @Bean
    @Autowired
    public JettyEmbeddedServletContainerFactory jettyEmbeddedServletContainerFactory(HttpSetting httpSetting) {
        JettyEmbeddedServletContainerFactory factory = new JettyEmbeddedServletContainerFactory();

        factory.setPort(httpSetting.getPort());
        log.info("Jetty configured on port: " + httpSetting.getPort());

        factory.addServerCustomizers(new JettyServerCustomizer() {
            @Override
            public void customize(Server server1) {
                server = server1;
                log.info("Jetty version: " + server.getVersion());

                // Configure shutdown wait time.
                if (httpSetting.getShutdownWaitTime() > 0) {
                    // Add StatsticsHandler, in order for graceful shutdown to work.
                    StatisticsHandler handler = new StatisticsHandler();
                    handler.setHandler(server.getHandler());
                    server.setHandler(handler);

                    log.info("Shutdown wait time: " + httpSetting.getShutdownWaitTime() + "s");
                    server.setStopTimeout(httpSetting.getShutdownWaitTime());

                    // We will stop it through JettyGracefulShutdown class.
                    server.setStopAtShutdown(false);
                }
            }
        });
        return factory;
    }

    @Bean
    public JettyGracefulShutdown jettyGracefulShutdown() { return new JettyGracefulShutdown(); }

    // Springboot closes application context before everything.
    private static class JettyGracefulShutdown implements ApplicationListener<ContextClosedEvent>{
        private static final Logger log = LoggerFactory.getLogger(JettyGracefulShutdown.class);

        @Override
        public void onApplicationEvent(ContextClosedEvent event) {
            if (server == null) {
                log.error("Jetty server variable is null, this should not happen!");
                return;
            }
            log.info("Entering shutdown for Jetty.");
            if (!(server.getHandler() instanceof StatisticsHandler)) {
                log.error("Root handler is not StatisticsHandler, graceful shutdown may not work at all!");
            } else {
                log.info("Active requests: " + ((StatisticsHandler) server.getHandler()).getRequestsActive());
            }
            try {
                long begin = System.currentTimeMillis();
                server.stop();
                log.info("Shutdown took " + (System.currentTimeMillis() - begin) + " ms.");
            } catch (Exception e) {
                log.error("Fail to shutdown gracefully.", e);
            }
        }
    }
}
@clanie
Copy link

@clanie clanie commented May 27, 2016

The tomcat version didn't work as-is for me because this.connector.getProtocolHandler().getExecutor() returned null. The protocolHandler was a Http11NioProtocol.
To make it work I let the customize(Connector connector) method also set an Executor on the connectors ProtocolHandler.

@ktracer-isp
Copy link

@ktracer-isp ktracer-isp commented Aug 4, 2016

@clanie: Can you post the code you did for that? We're running into a similar problem and I'd like to see a working solution if you have one.

@czyoshida-naoko
Copy link

@czyoshida-naoko czyoshida-naoko commented Aug 15, 2016

I found this article.
http://martin.podval.eu/2015/06/java-docker-spring-boot-and-signals.html
So I want to do exec java -jar even when I install my app as init.d service.
How can I pass exec to a launch.script?
or How can I use a customized launch.script than embedded launch.script?
Is there any solutions?

@robocide
Copy link

@robocide robocide commented Dec 7, 2016

Hi,

forgive me if i am mistaken.
but the the question stated the request to "waits for an empty request queue before shutting "

isnt it simpler... to just count the number of requests (From Filter) ? and then when the number is 0 just signal for grceful shutdown ?

Thanks!

@wilkinsona
Copy link
Member

@wilkinsona wilkinsona commented Dec 8, 2016

isnt it simpler... to just count the number of requests (From Filter) ? and then when the number is 0 just signal for grceful shutdown ?

No, not really. You need to stop accepting new requests, wait for any active requests to complete, and then gracefully shut everything down. If you just wait for the number of active requests to reach 0 there's then a race condition between new requests being accepted and the shutdown completing.

@robocide
Copy link

@robocide robocide commented Dec 8, 2016

@wilkinsona thanks for your response ! ... i would like to regard.

You said:

  1. "You need to stop accepting new requests," -> My Solution below handles it.
  2. "wait for any active requests to complete," -> My Solution below handles it.

Solution:

public class LimitFilter implements Filter {
    public static AtomicBoolean isRunning = true;
    public int count;
    private Object lock = new Object();

    public void doFilter(ServletRequest request, ServletResponse response,
            FilterChain chain) throws IOException, ServletException {
        try {           
            if (isRunning) {
                // let the request through and process as usual
                 synchronized (lock) {                
                      count++;
                 }
                try{
                chain.doFilter(request, response); 
                }finaly{
                       synchronized (lock) {                
                          count--;
                 }
                }
            } else {
               //return server unavailable http status 503 
            }
        } finally {
            synchronized (lock) {                
                   if(count==0 && !isRunning)
                       AppSingleton.signalSafeShutdown();
            }           
        }
    }

now from shutdown hook:

@PreDestroy public void cleanup(){ LimitFilter.isRunning=false; AppSingleton.waitSafeShutdown(); }

Description:
ok now the Filter's isRunning can be controlled from listening to contextCloseEvent .....
now when the contextClosedEvent is fired we will do 2 things:
1] set LimitFilter.isRunning=false in order not recieve new requests.
2] wait for the current counter to be 0 in order to wait for current requests to finish.

i would be happy if someone sees any problem with this kind of approach rather than trying complex solution of reverting the shutdown order and handling container specific complex code.... ?

thanks.
Avishay.

@wilkinsona
Copy link
Member

@wilkinsona wilkinsona commented Dec 8, 2016

@AvishayHirsh Thanks for your efforts, but you are over-simplifying the problem.

Firstly, your solution does not stop accepting requests, instead it accepts them and then responds with a 503. We need to pause the container's connectors so that they stop allowing clients to connect at all. Secondly, your mechanism for counting active requests means that all requests briefly become single threaded. That will have an unacceptable impact on performance.

@wilkinsona wilkinsona self-assigned this Jan 24, 2020
@wilkinsona wilkinsona changed the title Shut down embedded servlet container gracefully Shut down embedded web server gracefully Jan 27, 2020
@wilkinsona
Copy link
Member

@wilkinsona wilkinsona commented Mar 3, 2020

When using Kubernetes and HTTP endpoints for liveness and readiness checks, the readiness check needs to start failing when you initiate the graceful shutdown but the liveness check should continue to succeed, either until the app has stopped itself or until you're reading for Kubernetes to kill the pod.

I now believe this is incorrect.

When Kubernetes is taking a pod out of service it will send the SIGTERM signal to it. This will cause a Spring Boot application to start shutting down. At this point, the liveness and readiness probes are no longer called.

In parallel, Kubernetes will also initiate the sequence of events that results in requests no longer being routed to the application. The parallel and eventual nature of this means that there is a period of time when the application has received SIGTERM but requests are still routed to it. Ideally, these requests should be handled so that clients do not become aware of the application being shut down.

The recommended way to minimise or perhaps even eliminate the window where requests are still received after SIGTERM is to sleep in a pre-stop hook. In the event of the window only being minimised, continuing to accept new requests during graceful shutdown should reduce the number of errors that are seen by clients. Such errors cannot be eliminated entirely as, if there are no active requests, the graceful shutdown may complete and stop accepting new connections while Kubernetes contains to route requests to the application.

We may need to provide a configurable strategy for graceful shutdown. One that rejects new connections as soon as graceful shutdown begins and one that accepts them. The former could be used in environments where the routing change is guaranteed to happen before graceful shutdown begins and the latter in environments like Kubernetes where the routing change and graceful shutdown run in parallel.

@wilkinsona
Copy link
Member

@wilkinsona wilkinsona commented Mar 9, 2020

We discussed the need for different strategies for handling requests during graceful shutdown, and decided not to do anything at this time. As explained above, it is impossible to completely close the window on the application's side so it is hard to justify adding complexity that attempts to do so, particularly for a problem that is particular to Kubernetes. In that deployment environment, our recommendation will be to sleep for a period in a pre-stop hook. The length of that period will vary from deployment to deployment, depending on the needs of the application and the time it takes for Kubernetes to propagate the routing change.

@wilkinsona wilkinsona modified the milestones: 2.3.x, 2.3.0.M3 Mar 9, 2020
@wilkinsona wilkinsona closed this in 308e1d3 Mar 9, 2020
@wilkinsona
Copy link
Member

@wilkinsona wilkinsona commented Mar 9, 2020

308e1d3 adds built-in support for gracefully shutting down Jetty, Tomcat, Reactor Netty, and Undertow. Support is provided for both reactive and Servlet-based web applications and can be enabled by configuring the server.shutdown.grace-period property. The exact behaviour during graceful shutdown varies a little from container to container. If you are interested in this feature, we'd love to hear your feedback before 2.3 is released. The functionality will be available shortly in 2.3 snapshots and will also be in this week's 2.3.0.M3 release. Please take it for a spin and let us know if it meets your needs.

wilkinsona added a commit that referenced this issue Mar 9, 2020
Polish
See gh-4657
wilkinsona added a commit that referenced this issue Mar 9, 2020
See gh-4657
@wilkinsona
Copy link
Member

@wilkinsona wilkinsona commented Mar 9, 2020

A snapshot containing the new functionality is now available from https://repo.spring.io/snapshot.

@joedj
Copy link

@joedj joedj commented Mar 9, 2020

Thanks for working on this @wilkinsona

I find the documentation a bit confusing, specifically "Jetty, Reactor Netty, and Tomcat will stop accepting requests at the network layer."

  • Do you know what happens for new requests that arrive on existing idle connections, during graceful shutdown?
  • What about new requests that arrive on existing connections after the current in-flight request completes?
  • Would it be possible to also document this?
  • Do any/all of the implementations send Connection:close after processing an in-flight request, such that the connection is gracefully terminated?
  • Assuming that they do not, is this new functionality exposed in such a way that applications could implement this easily themselves (e.g. with a servlet Filter that checks if (iAmGracefullyShuttingDown) { response.setHeader("Connection", "close"); }

Most of our applications run on AWS Elastic Container Service (ECS). They are primarily behind two different types of load-balancers:

AWS Application Load Balancer (ALB):
With ALB, we can set a Deregistration Delay on the load-balancer.
As soon as the deregistration process starts for a container, the load-balancer will not send any new requests to the app - even on existing connections - because ALB is protocol-aware.
ECS will wait Deregistration Delay seconds before sending SIGTERM to the application.
This new graceful shutdown process in Spring does not really buy us anything new here, because we have already stopped receiving new requests/connections, and have been given time to handle any outstanding in-flight requests even before the application shutdown process begins.

AWS Network Load Balancer (NLB)
With NLB, we can set a Deregistration Delay on the load-balancer.
As soon as the deregistration process starts for a container, the load-balancer will not send any new connections to the app, but it will continue to send requests on existing connections.
ECS will wait Deregistration Delay seconds before sending SIGTERM to the application.
At this point, application shutdown begins. ECS gives the application a configurable amount of time to shut down, which defaults to 30sec, at which point it will send SIGKILL.
The app will continue to receive new requests on existing connections until it terminates or closes them.
The new graceful shutdown process in Spring could help us here, but only if it gives us the ability to gracefully close these existing connections after processing outstanding in-flight requests.

Right now, we use custom application logic (in a Filter) to discover when the LB deregistration process begins, and we start sending Connection:close after any in-flight requests complete. This gives us the ability to gracefully close connections before the application shutdown process even begins, but is cumbersome to implement, and adds significant delays to the shutdown process for our applications (which, in turn, adds delays to the overall release process).

Framework support for gracefully closing existing connections after in-flight requests complete (once the app receives SIGTERM) would be very helpful in this scenario.

wilkinsona added a commit that referenced this issue Mar 10, 2020
@wilkinsona
Copy link
Member

@wilkinsona wilkinsona commented Mar 10, 2020

Thanks for taking a look, @joedj.

Do you know what happens for new requests that arrive on existing idle connections, during graceful shutdown?

We're making as much use as possible of each container's built-in capabilities, which often aren't documented or consider the exact behaviour to be an implementation detail and this is one area where the containers vary. For example, Tomcat will close idle connections such that a new request cannot be made on it.

What about new requests that arrive on existing connections after the current in-flight request completes?

I don't believe that any of the containers distinguish between this scenario and the scenario above. As above, the exact behaviour varies from container to container.

Would it be possible to also document this?

There's an awkward balance for us to strike here. The currently described difference (rejected requests at the network layer versus them being accepted and then responded to with a 503) is within our control but, beyond that, the subtleties are container-specific and are often undocumented implementation details. I can only assume that's because the users of each container haven't needed to know the precise details to make use of the feature. If we documented things more precisely we'd be running the risk of our documentation becoming out of date as a container made a change to its behaviour. We could, perhaps, document what we do in terms of API calls as that is within our control.

Do any/all of the implementations send Connection:close after processing an in-flight request, such that the connection is gracefully terminated?

At the time of writing, Tomcat and Undertow do not. When an in-flight request completes successfully after graceful shutdown has begun, both respond with a Connection: keep-alive header. At this point, Tomcat won't deal with any more requests at the HTTP level. Undertow will deal with requests at the HTTP level and will respond with a 503. This response also has a Connection: keep-alive header. I haven't tested Jetty or Reactor Netty.

is this new functionality exposed in such a way that applications could implement this easily themselves

The container will be shutting down gracefully if you've configured server.shutdown.grace-period and the context closed event has been published. You could use this as a signal to set Connection: Close in any responses. That said, you may want to explore pushing this down into the container that you're using by requesting a change in behaviour or to understand why the container does not do this by default before doing so.

This new graceful shutdown process in Spring does not really buy us anything new here, because we have already stopped receiving new requests/connections, and have been given time to handle any outstanding in-flight requests even before the application shutdown process begins.

It sounds like the time that you've been given to handle outstanding in-flight requests is of fixed duration. For example, if you're being given 30 seconds to finish handling in-flight requests, shutdown will take at least 30 seconds even if those requests complete after 1 second. With graceful shutdown, you should be able to configure a deregistration delay of 0 seconds so that SIGTERM is sent as soon as the load-balancer has stopped sending requests to the app. SIGTERM results in the application context being closed. With server.shutdown.grace-period configured, this close processing will wait for up to the grace period for active requests to complete, before proceeding to shut down the app. In the scenario above where the active requests complete after 1 second, this would allow the shutdown to complete in a little over 1 second rather than a little over 30 seconds.

The new graceful shutdown process in Spring could help us here, but only if it gives us the ability to gracefully close these existing connections after processing outstanding in-flight requests.

While Tomcat will not send Connection: close, I believe that it will behave in a way that will work for you out of the box. Undertow, however, will not. If you have an opportunity to try things out in your environment with the container that you chose to use, it'd be great to know whether or not this is indeed the case.

Right now, we use custom application logic (in a Filter) to discover when the LB deregistration process begins, and we start sending Connection:close after any in-flight requests complete. This gives us the ability to gracefully close connections before the application shutdown process even begins, but is cumbersome to implement, and adds significant delays to the shutdown process for our applications (which, in turn, adds delays to the overall release process).

I'd like to understand why sending Connection:close adds significant delays to the shutdown process. Can you expand a bit on that please?

@joedj
Copy link

@joedj joedj commented Mar 10, 2020

Hey @wilkinsona, thanks for the response.

I'll just answer your last question, for now, until I hopefully get a chance to play around:

I'd like to understand why sending Connection:close adds significant delays to the shutdown process. Can you expand a bit on that please?

(This is specifically in relation to shutting down gracefully while using NLB, which will continue to send requests on existing connections up until they are closed)

Because we don't currently have a nice way to gracefully shut down the container upon receiving SIGTERM (and some other internal legacy reasons that mean we currently only have a max of 30sec after SIGTERM), we instead perform the graceful connection draining before the actual app shutdown process begins.

To do this, we use the LB Deregistration Delay functionality. For NLB, this controls the duration between when the app will stop receiving new TCP connections, and when the app will receive SIGTERM.

For our app to discover that it has changed state in the LB from "healthy" to "draining" (signalling that the Deregistration Delay period has started), it polls the AWS APIs. This happens once per minute - any more frequent than that, and we often hit API rate limits. So, the Deregistration Delay needs to be at least 1min or so, for us to be sure we have even seen the state change.

Once we notice the state change, there are additional delays:

  • wait up to the maximum idle timeout for any new requests to arrive on the existing connections (say, 60sec), then
  • wait up to the maximum request timeout for those new in-flight requests to complete (say, another 60sec)

...but those additional delays also exist if we start draining in response to SIGTERM, rather than in response to an LB state change.

So really, using the Spring support for graceful shutdown in response to SIGTERM would only allow us to save that initial 60sec delay while polling the LB state (edit: and also, as you accurately pointed out in your earlier response, it would also mean that shutdown completes when all the in-flight requests are finished, rather than waiting for the fixed Deregistration Delay period - in practice, this could reduce the container shutdown time from 3min+ to mere seconds (or 60sec, i.e. the idle timeout we need to wait for any new requests on existing connections, assuming the web container we're using supports this and doesn't close those idle connections immediately when graceful shutdown begins).

...and it would also allow us to delete all that supporting code and get rid of the unreliable scheduled polling.

wilkinsona added a commit that referenced this issue Mar 10, 2020
Standalone Jetty will, by default, include a Connection: close header
in all responses once graceful shutdown has begun. Previously, the
way in which we were shutting Jetty down did not cause this to happen.

This commit updates JettyGracefulShutdown to shut down each connector.
This causes Jetty to send the Connection: close header, aligning its
behaviour more closely with what it does when used standalone. The
tests have also been updated to verify this behaviour and to ensure
that the correct port is used for requests even once the connector is
no longer bound and the web server no longer knows its ephemeral port.

See gh-4657
wilkinsona added a commit that referenced this issue Mar 10, 2020
@ractive
Copy link
Contributor

@ractive ractive commented Mar 10, 2020

On kubernetes we have a different requirement when shutting down gracefully when doing rolling updates or scaling a deployment down.

According to this blog post and to our own experience you need to wait for a couple of seconds after receiving the SIGTERM from kubernetes before stopping to accept requests to actually still be able to handle stray connections that are coming in.
This is because the kubernetes endpoint object of this app has not yet been removed from the kuberenetes service via the endpoint-controller and kube-proxy although the SIGTERM already has been sent by the Kubelet, so requests are still routed to this endpoint. When we stop accepting connections immediately after receiving the SIGTERM (which we already do) we see "connection refused" errors from the clients. So another configuration option we'd like to see is something like server.shutdown.wait-for-seconds that would wait for this amount of seconds before actually pausing any connector and continuing with the shutdown.

@wilkinsona
Copy link
Member

@wilkinsona wilkinsona commented Mar 10, 2020

Thanks, @ractive. What you've described matches our experience too. Rather than implementing the wait in Spring Boot, our recommendation is to use a pre-stop hook configured to run something like sh -c sleep 10. We have some documentation updates planned that will cover this and more from a Kubernetes perspective.

wilkinsona added a commit that referenced this issue Mar 10, 2020
@wilkinsona wilkinsona changed the title Shut down embedded web server gracefully Allow the embedded web server to be shut down gracefully Mar 12, 2020
@pradeepanchan
Copy link

@pradeepanchan pradeepanchan commented Mar 13, 2020

When Kubernetes is taking a pod out of service it will send the SIGTERM signal to it. This will cause a Spring Boot application to start shutting down. At this point, the liveness and readiness probes are no longer called.

In parallel, Kubernetes will also initiate the sequence of events that results in requests no longer being routed to the application. The parallel and eventual nature of this means that there is a period of time when the application has received SIGTERM but requests are still routed to it. Ideally, these requests should be handled so that clients do not become aware of the application being shut down.

The recommended way to minimise or perhaps even eliminate the window where requests are still received after SIGTERM is to sleep in a pre-stop hook. In the event of the window only being minimised, continuing to accept new requests during graceful shutdown should reduce the number of errors that are seen by clients. Such errors cannot be eliminated entirely as, if there are no active requests, the graceful shutdown may complete and stop accepting new connections while Kubernetes contains to route requests to the application.

This is correct. Its also documented in the book : Kubernetes in Action (ISBN: 9781617293726), under Chapter 17. Best practices for developing apps.

I think the sleep based configuration would be good enough in our case (infact we did the same thing in our application). I expect that the configuration server.shutdown.grace-period will be well-documented so that users will know what exactly will happen when its configured.

Thanks for taking this up.

@gnagy
Copy link

@gnagy gnagy commented May 4, 2020

Hi, I have a spring-boot app with HTTP endpoints and Kafka consumers. I have tested this graceful shutdown implementation and it looks like the whole shutdown is delayed waiting for HTTP with Thread.sleep(), so no other components are notified. This means I need 2*n graceful period if I wanted to give HTTP and Kafka n seconds to finish their thing.

@wilkinsona I saw your comment about not needing SmartLifecycle just for tomcat, but have you considered other components needing graceful shutdown as well? In my case, I would like to leave some time for both @controllers and Kafka Listeners to be able to finish already started processing -- which in some cases can make external calls with exponential backoff, so I would also like to be able to check in retry loops if shutdown was initiated.

I'm thinking about such logic:

  • shutdown initiated
  • components notified about shutdown, should not block, clock starts
    • HTTP connections no longer accepted
    • kafka consumers stop polling new messages
    • retry loops can early exit, throwing "retry-able" exception
  • when time runs out, interrupt processing threads
  • finish shutdown, cleanup

Would you consider changing the above logic, or recommend a custom implementation instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.