New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shut down embedded servlet container gracefully #4657

Open
LutzStrobel opened this Issue Dec 2, 2015 · 65 comments

Comments

Projects
None yet
@LutzStrobel

LutzStrobel commented Dec 2, 2015

We are using spring-boot and spring-cloud to realize a micro service archtecture. During load tests we are facing lots of errors such as "entitymanager closed and others" if we shut the micro service down during load.
We are wondering if there is an option to configure the embedded container to shut its service connector down and waits for an empty request queue before shutting down the complete application context.

If there is no such option, I did not find any, then it would be great to extend the shutdownhook of spring to respect such requirements.

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona Dec 2, 2015

Member

We currently stop the application context and then stop the container. We did try to reverse this ordering but it had some unwanted side effects so we're stuck with it for now at least. That's the bad news.

The good news is that you can actually get a graceful shutdown yourself if you're happy to get your hands a bit dirty. The gist is that you need to pause Tomcat's connector and then wait for its thread pool to shutdown before allow the destruction of the application context to proceed. It looks something like this:

package com.example;

import java.util.concurrent.Executor;
import java.util.concurrent.TimeUnit;

import org.apache.catalina.connector.Connector;
import org.apache.tomcat.util.threads.ThreadPoolExecutor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.embedded.ConfigurableEmbeddedServletContainer;
import org.springframework.boot.context.embedded.EmbeddedServletContainerCustomizer;
import org.springframework.boot.context.embedded.tomcat.TomcatConnectorCustomizer;
import org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory;
import org.springframework.context.ApplicationListener;
import org.springframework.context.annotation.Bean;
import org.springframework.context.event.ContextClosedEvent;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class Gh4657Application {

    public static void main(String[] args) {
        SpringApplication.run(Gh4657Application.class, args);
    }

    @RequestMapping("/pause")
    public String pause() throws InterruptedException {
        Thread.sleep(10000);
        return "Pause complete";
    }

    @Bean
    public GracefulShutdown gracefulShutdown() {
        return new GracefulShutdown();
    }

    @Bean
    public EmbeddedServletContainerCustomizer tomcatCustomizer() {
        return new EmbeddedServletContainerCustomizer() {

            @Override
            public void customize(ConfigurableEmbeddedServletContainer container) {
                if (container instanceof TomcatEmbeddedServletContainerFactory) {
                    ((TomcatEmbeddedServletContainerFactory) container)
                            .addConnectorCustomizers(gracefulShutdown());
                }

            }
        };
    }

    private static class GracefulShutdown implements TomcatConnectorCustomizer,
            ApplicationListener<ContextClosedEvent> {

        private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);

        private volatile Connector connector;

        @Override
        public void customize(Connector connector) {
            this.connector = connector;
        }

        @Override
        public void onApplicationEvent(ContextClosedEvent event) {
            this.connector.pause();
            Executor executor = this.connector.getProtocolHandler().getExecutor();
            if (executor instanceof ThreadPoolExecutor) {
                try {
                    ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
                    threadPoolExecutor.shutdown();
                    if (!threadPoolExecutor.awaitTermination(30, TimeUnit.SECONDS)) {
                        log.warn("Tomcat thread pool did not shut down gracefully within "
                                + "30 seconds. Proceeding with forceful shutdown");
                    }
                }
                catch (InterruptedException ex) {
                    Thread.currentThread().interrupt();
                }
            }
        }

    }

}

I think it makes sense to provide behaviour like this out of the box, or at least an option to enable it. However, that'll require a bit more work as it'll need to work with all of the embedded containers that we support (Jetty, Tomcat, and Undertow), cope with multiple connectors (HTTP and HTTPS, for example) and we'll need to think about the configuration options, if any, that we'd want to offer: switching it on or off, configuring the period that we wait for the thread pool to shutdown, etc.

Member

wilkinsona commented Dec 2, 2015

We currently stop the application context and then stop the container. We did try to reverse this ordering but it had some unwanted side effects so we're stuck with it for now at least. That's the bad news.

The good news is that you can actually get a graceful shutdown yourself if you're happy to get your hands a bit dirty. The gist is that you need to pause Tomcat's connector and then wait for its thread pool to shutdown before allow the destruction of the application context to proceed. It looks something like this:

package com.example;

import java.util.concurrent.Executor;
import java.util.concurrent.TimeUnit;

import org.apache.catalina.connector.Connector;
import org.apache.tomcat.util.threads.ThreadPoolExecutor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.embedded.ConfigurableEmbeddedServletContainer;
import org.springframework.boot.context.embedded.EmbeddedServletContainerCustomizer;
import org.springframework.boot.context.embedded.tomcat.TomcatConnectorCustomizer;
import org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory;
import org.springframework.context.ApplicationListener;
import org.springframework.context.annotation.Bean;
import org.springframework.context.event.ContextClosedEvent;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class Gh4657Application {

    public static void main(String[] args) {
        SpringApplication.run(Gh4657Application.class, args);
    }

    @RequestMapping("/pause")
    public String pause() throws InterruptedException {
        Thread.sleep(10000);
        return "Pause complete";
    }

    @Bean
    public GracefulShutdown gracefulShutdown() {
        return new GracefulShutdown();
    }

    @Bean
    public EmbeddedServletContainerCustomizer tomcatCustomizer() {
        return new EmbeddedServletContainerCustomizer() {

            @Override
            public void customize(ConfigurableEmbeddedServletContainer container) {
                if (container instanceof TomcatEmbeddedServletContainerFactory) {
                    ((TomcatEmbeddedServletContainerFactory) container)
                            .addConnectorCustomizers(gracefulShutdown());
                }

            }
        };
    }

    private static class GracefulShutdown implements TomcatConnectorCustomizer,
            ApplicationListener<ContextClosedEvent> {

        private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);

        private volatile Connector connector;

        @Override
        public void customize(Connector connector) {
            this.connector = connector;
        }

        @Override
        public void onApplicationEvent(ContextClosedEvent event) {
            this.connector.pause();
            Executor executor = this.connector.getProtocolHandler().getExecutor();
            if (executor instanceof ThreadPoolExecutor) {
                try {
                    ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
                    threadPoolExecutor.shutdown();
                    if (!threadPoolExecutor.awaitTermination(30, TimeUnit.SECONDS)) {
                        log.warn("Tomcat thread pool did not shut down gracefully within "
                                + "30 seconds. Proceeding with forceful shutdown");
                    }
                }
                catch (InterruptedException ex) {
                    Thread.currentThread().interrupt();
                }
            }
        }

    }

}

I think it makes sense to provide behaviour like this out of the box, or at least an option to enable it. However, that'll require a bit more work as it'll need to work with all of the embedded containers that we support (Jetty, Tomcat, and Undertow), cope with multiple connectors (HTTP and HTTPS, for example) and we'll need to think about the configuration options, if any, that we'd want to offer: switching it on or off, configuring the period that we wait for the thread pool to shutdown, etc.

@wilkinsona wilkinsona changed the title from graceful shutdown of embedded container should be possible to Shut down embedded servlet container gracefully Dec 2, 2015

@LutzStrobel

This comment has been minimized.

Show comment
Hide comment
@LutzStrobel

LutzStrobel Dec 4, 2015

Thank you for your advice.
We currently simply shut down the embedded container and the waiting a short time before closing the application context.

What do you thing, will it be possible in future to shut down a spring boot application more gracefully?

LutzStrobel commented Dec 4, 2015

Thank you for your advice.
We currently simply shut down the embedded container and the waiting a short time before closing the application context.

What do you thing, will it be possible in future to shut down a spring boot application more gracefully?

@LutzStrobel LutzStrobel closed this Dec 4, 2015

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona Dec 4, 2015

Member

What do you thing, will it be possible in future to shut down a spring boot application more gracefully?

Yes. As I said above "I think it makes sense to provide behaviour like this out of the box, or at least an option to enable it".

I'm going to re-open this issue as we'll use it to track the possible enhancement.

Member

wilkinsona commented Dec 4, 2015

What do you thing, will it be possible in future to shut down a spring boot application more gracefully?

Yes. As I said above "I think it makes sense to provide behaviour like this out of the box, or at least an option to enable it".

I'm going to re-open this issue as we'll use it to track the possible enhancement.

@wilkinsona wilkinsona reopened this Dec 4, 2015

@tkvangorder

This comment has been minimized.

Show comment
Hide comment
@tkvangorder

tkvangorder Jan 19, 2016

+1 on this request. We ran into a similar problem when load testing and dropping a node from the test. @wilkinsona in your example, I was thinking of using an implementation of smartlifecylce so I can insure the connector is shutdown first. You said you ran into issues shutting down tomcat first?

tkvangorder commented Jan 19, 2016

+1 on this request. We ran into a similar problem when load testing and dropping a node from the test. @wilkinsona in your example, I was thinking of using an implementation of smartlifecylce so I can insure the connector is shutdown first. You said you ran into issues shutting down tomcat first?

@bohrqiu

This comment has been minimized.

Show comment
Hide comment
@bohrqiu

bohrqiu Jan 19, 2016

Contributor

I think the right order is:

  1. pause the io endpoint

    web container just pasue(deny new request come in),and RPC framework need notify client don't send new request and wait current request be proccessed

  2. wait the request to be processed and response client

  3. close the io endpoint

  4. close spring service

  5. close log system

so we do it as follows:

    @Override
    public void finished(ConfigurableApplicationContext context, Throwable exception) {
        if (exception == null) {
            //install UncaughtExceptionHandler
            UncaughtExceptionHandlerWrapper.install();
            //when system startup ,register shutdown hooks to clean resouces.
            Runtime.getRuntime().addShutdownHook(new Thread() {
                @Override
                public void run() {
                    //run all shutdown hooks before spring container avoid service dependency
                    ShutdownHooks.shutdownAll();
                    //close spring container
                    context.close();
                    shutdownLogSystem();
                }
            });
            //log startup info
            LoggerFactory.getLogger(YijiApplicationRunListener.class).info("启动成功: http://127.0.0.1:{}",
            context.getEnvironment().getProperty(Apps.HTTP_PORT));
        } else {
            ShutdownHooks.shutdownAll();
            shutdownLogSystem();
        }
    }
Contributor

bohrqiu commented Jan 19, 2016

I think the right order is:

  1. pause the io endpoint

    web container just pasue(deny new request come in),and RPC framework need notify client don't send new request and wait current request be proccessed

  2. wait the request to be processed and response client

  3. close the io endpoint

  4. close spring service

  5. close log system

so we do it as follows:

    @Override
    public void finished(ConfigurableApplicationContext context, Throwable exception) {
        if (exception == null) {
            //install UncaughtExceptionHandler
            UncaughtExceptionHandlerWrapper.install();
            //when system startup ,register shutdown hooks to clean resouces.
            Runtime.getRuntime().addShutdownHook(new Thread() {
                @Override
                public void run() {
                    //run all shutdown hooks before spring container avoid service dependency
                    ShutdownHooks.shutdownAll();
                    //close spring container
                    context.close();
                    shutdownLogSystem();
                }
            });
            //log startup info
            LoggerFactory.getLogger(YijiApplicationRunListener.class).info("启动成功: http://127.0.0.1:{}",
            context.getEnvironment().getProperty(Apps.HTTP_PORT));
        } else {
            ShutdownHooks.shutdownAll();
            shutdownLogSystem();
        }
    }
@tkvangorder

This comment has been minimized.

Show comment
Hide comment
@tkvangorder

tkvangorder Jan 19, 2016

This issue is actually a bit more complicated.

I know that SmartLifeCycle can be used to set the order in which beans are notified of life cycle events. However, there is no generalized way for a service to know it should startup/shutdown before another service.

Consider the following:

A sprint boot application is running an embedded servlet container and is also producing/consuming JMS messages.

On the close event, the servlet container really needs to pause the connector first, process its remain working (any connections that have already been establish.)

We need to insure this is the FIRST thing that happens, prior to the JMS infrastructure shutting down because the work done inside tomcat may rely on JMS.

The JMS infrastructure has a similar requirement: it must stop listening for messages and chew through any remaining messages it has already accepted.

I can certainly implement a SmartLifeCycle class that sets the phase "very high"....and I could even create two instances, one for embedded servlet container and one for JMS and insure the correct order.

But in the spirit of Spring Boot, if I add a tomcat container, its my expectation that when the application shuts down, it will gracefully stop accepting connections, process remaining work, and exit.

It would be helpful if there was a mechanism to allow ordering to be expressed relative to other services "JMS must start before tomcat", "tomcat must close before JMS". This would be similar to the AutoConfigureBefore/AutoConfigureAfter annotations that are used in Spring boot.

One way to approach this might be to create an enum for the generalized services (This is not ideal, but I can't think of another way without introducing artificial, compile time dependencies.):

EMBEDDED_CONTAINER
JMS
DISCOVERY_CLIENT
.
.
The annotations could leverage the enums to order the life cycle listeners.

For now, its up to the developer to explicitly define the shutdown order of services via "SmartLifeCycle" instances...which can get a bit messy and seems like extra work for something that should really work out of the box.

tkvangorder commented Jan 19, 2016

This issue is actually a bit more complicated.

I know that SmartLifeCycle can be used to set the order in which beans are notified of life cycle events. However, there is no generalized way for a service to know it should startup/shutdown before another service.

Consider the following:

A sprint boot application is running an embedded servlet container and is also producing/consuming JMS messages.

On the close event, the servlet container really needs to pause the connector first, process its remain working (any connections that have already been establish.)

We need to insure this is the FIRST thing that happens, prior to the JMS infrastructure shutting down because the work done inside tomcat may rely on JMS.

The JMS infrastructure has a similar requirement: it must stop listening for messages and chew through any remaining messages it has already accepted.

I can certainly implement a SmartLifeCycle class that sets the phase "very high"....and I could even create two instances, one for embedded servlet container and one for JMS and insure the correct order.

But in the spirit of Spring Boot, if I add a tomcat container, its my expectation that when the application shuts down, it will gracefully stop accepting connections, process remaining work, and exit.

It would be helpful if there was a mechanism to allow ordering to be expressed relative to other services "JMS must start before tomcat", "tomcat must close before JMS". This would be similar to the AutoConfigureBefore/AutoConfigureAfter annotations that are used in Spring boot.

One way to approach this might be to create an enum for the generalized services (This is not ideal, but I can't think of another way without introducing artificial, compile time dependencies.):

EMBEDDED_CONTAINER
JMS
DISCOVERY_CLIENT
.
.
The annotations could leverage the enums to order the life cycle listeners.

For now, its up to the developer to explicitly define the shutdown order of services via "SmartLifeCycle" instances...which can get a bit messy and seems like extra work for something that should really work out of the box.

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona Jan 19, 2016

Member

@tkvangorder You don't need to use SmartLifecycleto gracefully shut down Tomcat as shown in my comment above. It happens in response to a ContextClosedEvent which is fired at the very beginning of the context's close processing before any of the beans in the context are processed.

Beyond this, Spring Framework already has support for closing things down in the correct order. For example you can implement DisposableBean. When the container disposes of a bean, it will dispose of anything that depends on it first.

Member

wilkinsona commented Jan 19, 2016

@tkvangorder You don't need to use SmartLifecycleto gracefully shut down Tomcat as shown in my comment above. It happens in response to a ContextClosedEvent which is fired at the very beginning of the context's close processing before any of the beans in the context are processed.

Beyond this, Spring Framework already has support for closing things down in the correct order. For example you can implement DisposableBean. When the container disposes of a bean, it will dispose of anything that depends on it first.

@thefallentree

This comment has been minimized.

Show comment
Hide comment
@thefallentree

thefallentree Mar 8, 2016

@wilkinsona is the order of shutdown anyway guaranteed ? Is spring boot always going to shutdown application context first? we are trying to implement what you said, but was not sure if this will change in the future.

thefallentree commented Mar 8, 2016

@wilkinsona is the order of shutdown anyway guaranteed ? Is spring boot always going to shutdown application context first? we are trying to implement what you said, but was not sure if this will change in the future.

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona Mar 8, 2016

Member

Is spring boot always going to shutdown application context first?

Yes. When you're using an embedded container, it's the application context being shut down that triggers the shutdown of the container.

Member

wilkinsona commented Mar 8, 2016

Is spring boot always going to shutdown application context first?

Yes. When you're using an embedded container, it's the application context being shut down that triggers the shutdown of the container.

@thefallentree

This comment has been minimized.

Show comment
Hide comment
@thefallentree

thefallentree Mar 12, 2016

I want to report back on how to setup this correctly using Jetty and springboot , use this class instead of EmbeddedServletAutoconfiguration and you are set.

public class HttpConfig {
    private static final Logger log = LoggerFactory.getLogger(HttpConfig.class);

    private static volatile Server server;

    // Jetty HTTP Server
    //
    // see [1] on how to implement graceful shutdown in springboot.
    // Note that since use jetty, we need to use server.stop(), also StatisticsHandler must be
    // configured, for jetty graceful shutdown to work.
    //
    // [1]: https://github.com/spring-projects/spring-boot/issues/4657
    @Bean
    @Autowired
    public JettyEmbeddedServletContainerFactory jettyEmbeddedServletContainerFactory(HttpSetting httpSetting) {
        JettyEmbeddedServletContainerFactory factory = new JettyEmbeddedServletContainerFactory();

        factory.setPort(httpSetting.getPort());
        log.info("Jetty configured on port: " + httpSetting.getPort());

        factory.addServerCustomizers(new JettyServerCustomizer() {
            @Override
            public void customize(Server server1) {
                server = server1;
                log.info("Jetty version: " + server.getVersion());

                // Configure shutdown wait time.
                if (httpSetting.getShutdownWaitTime() > 0) {
                    // Add StatsticsHandler, in order for graceful shutdown to work.
                    StatisticsHandler handler = new StatisticsHandler();
                    handler.setHandler(server.getHandler());
                    server.setHandler(handler);

                    log.info("Shutdown wait time: " + httpSetting.getShutdownWaitTime() + "s");
                    server.setStopTimeout(httpSetting.getShutdownWaitTime());

                    // We will stop it through JettyGracefulShutdown class.
                    server.setStopAtShutdown(false);
                }
            }
        });
        return factory;
    }

    @Bean
    public JettyGracefulShutdown jettyGracefulShutdown() { return new JettyGracefulShutdown(); }

    // Springboot closes application context before everything.
    private static class JettyGracefulShutdown implements ApplicationListener<ContextClosedEvent>{
        private static final Logger log = LoggerFactory.getLogger(JettyGracefulShutdown.class);

        @Override
        public void onApplicationEvent(ContextClosedEvent event) {
            if (server == null) {
                log.error("Jetty server variable is null, this should not happen!");
                return;
            }
            log.info("Entering shutdown for Jetty.");
            if (!(server.getHandler() instanceof StatisticsHandler)) {
                log.error("Root handler is not StatisticsHandler, graceful shutdown may not work at all!");
            } else {
                log.info("Active requests: " + ((StatisticsHandler) server.getHandler()).getRequestsActive());
            }
            try {
                long begin = System.currentTimeMillis();
                server.stop();
                log.info("Shutdown took " + (System.currentTimeMillis() - begin) + " ms.");
            } catch (Exception e) {
                log.error("Fail to shutdown gracefully.", e);
            }
        }
    }
}

thefallentree commented Mar 12, 2016

I want to report back on how to setup this correctly using Jetty and springboot , use this class instead of EmbeddedServletAutoconfiguration and you are set.

public class HttpConfig {
    private static final Logger log = LoggerFactory.getLogger(HttpConfig.class);

    private static volatile Server server;

    // Jetty HTTP Server
    //
    // see [1] on how to implement graceful shutdown in springboot.
    // Note that since use jetty, we need to use server.stop(), also StatisticsHandler must be
    // configured, for jetty graceful shutdown to work.
    //
    // [1]: https://github.com/spring-projects/spring-boot/issues/4657
    @Bean
    @Autowired
    public JettyEmbeddedServletContainerFactory jettyEmbeddedServletContainerFactory(HttpSetting httpSetting) {
        JettyEmbeddedServletContainerFactory factory = new JettyEmbeddedServletContainerFactory();

        factory.setPort(httpSetting.getPort());
        log.info("Jetty configured on port: " + httpSetting.getPort());

        factory.addServerCustomizers(new JettyServerCustomizer() {
            @Override
            public void customize(Server server1) {
                server = server1;
                log.info("Jetty version: " + server.getVersion());

                // Configure shutdown wait time.
                if (httpSetting.getShutdownWaitTime() > 0) {
                    // Add StatsticsHandler, in order for graceful shutdown to work.
                    StatisticsHandler handler = new StatisticsHandler();
                    handler.setHandler(server.getHandler());
                    server.setHandler(handler);

                    log.info("Shutdown wait time: " + httpSetting.getShutdownWaitTime() + "s");
                    server.setStopTimeout(httpSetting.getShutdownWaitTime());

                    // We will stop it through JettyGracefulShutdown class.
                    server.setStopAtShutdown(false);
                }
            }
        });
        return factory;
    }

    @Bean
    public JettyGracefulShutdown jettyGracefulShutdown() { return new JettyGracefulShutdown(); }

    // Springboot closes application context before everything.
    private static class JettyGracefulShutdown implements ApplicationListener<ContextClosedEvent>{
        private static final Logger log = LoggerFactory.getLogger(JettyGracefulShutdown.class);

        @Override
        public void onApplicationEvent(ContextClosedEvent event) {
            if (server == null) {
                log.error("Jetty server variable is null, this should not happen!");
                return;
            }
            log.info("Entering shutdown for Jetty.");
            if (!(server.getHandler() instanceof StatisticsHandler)) {
                log.error("Root handler is not StatisticsHandler, graceful shutdown may not work at all!");
            } else {
                log.info("Active requests: " + ((StatisticsHandler) server.getHandler()).getRequestsActive());
            }
            try {
                long begin = System.currentTimeMillis();
                server.stop();
                log.info("Shutdown took " + (System.currentTimeMillis() - begin) + " ms.");
            } catch (Exception e) {
                log.error("Fail to shutdown gracefully.", e);
            }
        }
    }
}
@clanie

This comment has been minimized.

Show comment
Hide comment
@clanie

clanie May 27, 2016

The tomcat version didn't work as-is for me because this.connector.getProtocolHandler().getExecutor() returned null. The protocolHandler was a Http11NioProtocol.
To make it work I let the customize(Connector connector) method also set an Executor on the connectors ProtocolHandler.

clanie commented May 27, 2016

The tomcat version didn't work as-is for me because this.connector.getProtocolHandler().getExecutor() returned null. The protocolHandler was a Http11NioProtocol.
To make it work I let the customize(Connector connector) method also set an Executor on the connectors ProtocolHandler.

@ktracer-isp

This comment has been minimized.

Show comment
Hide comment
@ktracer-isp

ktracer-isp Aug 4, 2016

@clanie: Can you post the code you did for that? We're running into a similar problem and I'd like to see a working solution if you have one.

ktracer-isp commented Aug 4, 2016

@clanie: Can you post the code you did for that? We're running into a similar problem and I'd like to see a working solution if you have one.

@czyoshida-naoko

This comment has been minimized.

Show comment
Hide comment
@czyoshida-naoko

czyoshida-naoko Aug 15, 2016

I found this article.
http://martin.podval.eu/2015/06/java-docker-spring-boot-and-signals.html
So I want to do exec java -jar even when I install my app as init.d service.
How can I pass exec to a launch.script?
or How can I use a customized launch.script than embedded launch.script?
Is there any solutions?

czyoshida-naoko commented Aug 15, 2016

I found this article.
http://martin.podval.eu/2015/06/java-docker-spring-boot-and-signals.html
So I want to do exec java -jar even when I install my app as init.d service.
How can I pass exec to a launch.script?
or How can I use a customized launch.script than embedded launch.script?
Is there any solutions?

@robocide

This comment has been minimized.

Show comment
Hide comment
@robocide

robocide Dec 7, 2016

Hi,

forgive me if i am mistaken.
but the the question stated the request to "waits for an empty request queue before shutting "

isnt it simpler... to just count the number of requests (From Filter) ? and then when the number is 0 just signal for grceful shutdown ?

Thanks!

robocide commented Dec 7, 2016

Hi,

forgive me if i am mistaken.
but the the question stated the request to "waits for an empty request queue before shutting "

isnt it simpler... to just count the number of requests (From Filter) ? and then when the number is 0 just signal for grceful shutdown ?

Thanks!

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona Dec 8, 2016

Member

isnt it simpler... to just count the number of requests (From Filter) ? and then when the number is 0 just signal for grceful shutdown ?

No, not really. You need to stop accepting new requests, wait for any active requests to complete, and then gracefully shut everything down. If you just wait for the number of active requests to reach 0 there's then a race condition between new requests being accepted and the shutdown completing.

Member

wilkinsona commented Dec 8, 2016

isnt it simpler... to just count the number of requests (From Filter) ? and then when the number is 0 just signal for grceful shutdown ?

No, not really. You need to stop accepting new requests, wait for any active requests to complete, and then gracefully shut everything down. If you just wait for the number of active requests to reach 0 there's then a race condition between new requests being accepted and the shutdown completing.

@robocide

This comment has been minimized.

Show comment
Hide comment
@robocide

robocide Dec 8, 2016

@wilkinsona thanks for your response ! ... i would like to regard.

You said:

  1. "You need to stop accepting new requests," -> My Solution below handles it.
  2. "wait for any active requests to complete," -> My Solution below handles it.

Solution:

public class LimitFilter implements Filter {
    public static AtomicBoolean isRunning = true;
    public int count;
    private Object lock = new Object();

    public void doFilter(ServletRequest request, ServletResponse response,
            FilterChain chain) throws IOException, ServletException {
        try {           
            if (isRunning) {
                // let the request through and process as usual
                 synchronized (lock) {                
                      count++;
                 }
                try{
                chain.doFilter(request, response); 
                }finaly{
                       synchronized (lock) {                
                          count--;
                 }
                }
            } else {
               //return server unavailable http status 503 
            }
        } finally {
            synchronized (lock) {                
                   if(count==0 && !isRunning)
                       AppSingleton.signalSafeShutdown();
            }           
        }
    }

now from shutdown hook:

@PreDestroy public void cleanup(){ LimitFilter.isRunning=false; AppSingleton.waitSafeShutdown(); }

Description:
ok now the Filter's isRunning can be controlled from listening to contextCloseEvent .....
now when the contextClosedEvent is fired we will do 2 things:
1] set LimitFilter.isRunning=false in order not recieve new requests.
2] wait for the current counter to be 0 in order to wait for current requests to finish.

i would be happy if someone sees any problem with this kind of approach rather than trying complex solution of reverting the shutdown order and handling container specific complex code.... ?

thanks.
Avishay.

robocide commented Dec 8, 2016

@wilkinsona thanks for your response ! ... i would like to regard.

You said:

  1. "You need to stop accepting new requests," -> My Solution below handles it.
  2. "wait for any active requests to complete," -> My Solution below handles it.

Solution:

public class LimitFilter implements Filter {
    public static AtomicBoolean isRunning = true;
    public int count;
    private Object lock = new Object();

    public void doFilter(ServletRequest request, ServletResponse response,
            FilterChain chain) throws IOException, ServletException {
        try {           
            if (isRunning) {
                // let the request through and process as usual
                 synchronized (lock) {                
                      count++;
                 }
                try{
                chain.doFilter(request, response); 
                }finaly{
                       synchronized (lock) {                
                          count--;
                 }
                }
            } else {
               //return server unavailable http status 503 
            }
        } finally {
            synchronized (lock) {                
                   if(count==0 && !isRunning)
                       AppSingleton.signalSafeShutdown();
            }           
        }
    }

now from shutdown hook:

@PreDestroy public void cleanup(){ LimitFilter.isRunning=false; AppSingleton.waitSafeShutdown(); }

Description:
ok now the Filter's isRunning can be controlled from listening to contextCloseEvent .....
now when the contextClosedEvent is fired we will do 2 things:
1] set LimitFilter.isRunning=false in order not recieve new requests.
2] wait for the current counter to be 0 in order to wait for current requests to finish.

i would be happy if someone sees any problem with this kind of approach rather than trying complex solution of reverting the shutdown order and handling container specific complex code.... ?

thanks.
Avishay.

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona Dec 8, 2016

Member

@avishayhirsh Thanks for your efforts, but you are over-simplifying the problem.

Firstly, your solution does not stop accepting requests, instead it accepts them and then responds with a 503. We need to pause the container's connectors so that they stop allowing clients to connect at all. Secondly, your mechanism for counting active requests means that all requests briefly become single threaded. That will have an unacceptable impact on performance.

Member

wilkinsona commented Dec 8, 2016

@avishayhirsh Thanks for your efforts, but you are over-simplifying the problem.

Firstly, your solution does not stop accepting requests, instead it accepts them and then responds with a 503. We need to pause the container's connectors so that they stop allowing clients to connect at all. Secondly, your mechanism for counting active requests means that all requests briefly become single threaded. That will have an unacceptable impact on performance.

@ramato-procon

This comment has been minimized.

Show comment
Hide comment
@ramato-procon

ramato-procon Dec 5, 2017

@robocide Yes it is the same idea. The filter counts each running request (but with an AtomicFieldUpdater, which is better in performance than synchronize). It returns 503, when the shutdown process has been started:
https://github.com/ramato-procon/springboot-graceful-terminator/blob/master/src/main/java/de/ra/springboot/graceful/GracefulHttpFilter.java

The shutdown process is started by the GracefulShutdownHook, which first inits the shutdown process in the Filter and waits until all running requests have been processed.
In the next step the http connector will be stopped.
Afterwards the ApplicationContext will be closed: https://github.com/ramato-procon/springboot-graceful-terminator/blob/master/src/main/java/de/ra/springboot/graceful/GracefulShutdownHook.java

ramato-procon commented Dec 5, 2017

@robocide Yes it is the same idea. The filter counts each running request (but with an AtomicFieldUpdater, which is better in performance than synchronize). It returns 503, when the shutdown process has been started:
https://github.com/ramato-procon/springboot-graceful-terminator/blob/master/src/main/java/de/ra/springboot/graceful/GracefulHttpFilter.java

The shutdown process is started by the GracefulShutdownHook, which first inits the shutdown process in the Filter and waits until all running requests have been processed.
In the next step the http connector will be stopped.
Afterwards the ApplicationContext will be closed: https://github.com/ramato-procon/springboot-graceful-terminator/blob/master/src/main/java/de/ra/springboot/graceful/GracefulShutdownHook.java

@rabbitgyk

This comment has been minimized.

Show comment
Hide comment
@rabbitgyk

rabbitgyk May 20, 2018

This is a troublesome thing, waiting for shutdown.

rabbitgyk commented May 20, 2018

This is a troublesome thing, waiting for shutdown.

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona May 30, 2018

Member

Having discussed this with a couple of people at Spring IO last week, it occurred to me that this may be better handled as a Spring Framework feature or at least building upon a Spring Framework feature. There's nothing Boot-specific about allowing Spring-managed components to be shut down gracefully and the concept being part of the Framework would allow other projects in the portfolio to provide graceful shutdown integration without creating a circular dependency between those projects and Boot. @snicoll is going to raise it with the Framework team.

Member

wilkinsona commented May 30, 2018

Having discussed this with a couple of people at Spring IO last week, it occurred to me that this may be better handled as a Spring Framework feature or at least building upon a Spring Framework feature. There's nothing Boot-specific about allowing Spring-managed components to be shut down gracefully and the concept being part of the Framework would allow other projects in the portfolio to provide graceful shutdown integration without creating a circular dependency between those projects and Boot. @snicoll is going to raise it with the Framework team.

@RichardBradley

This comment has been minimized.

Show comment
Hide comment
@RichardBradley

RichardBradley May 30, 2018

FWIW, I think this is a feature specific to Spring Boot, and doesn't need any Spring Framework support.

The entity which is being "gracefully shut down" here is not necessarily the entire Spring app context, but could just be the Spring Boot webapp. What is needed is a way for the Spring boot app to ask its EmbeddedServletContainer to stop accepting new requests, then to trigger a normal Spring shutdown when all in-flight requests have completed.

Since the TomcatEmbeddedServletContainer lives in the org.springframework.boot:spring-boot jar, this seems like a Spring Boot and/or a Tomcat feature request, rather than a Spring Framework feature.

I don't think there is currently any way to either a) ask Tomcat to stop accepting new requests without cancelling in-progress requests ("Pause" has problems, see above) or b) to ask Tomcat if there are any requests in progress, is there?

The thread above has gotten sidetracked into discussions of Spring-managed components dependency orders etc., which I think is not necessary for the desired feature.

RichardBradley commented May 30, 2018

FWIW, I think this is a feature specific to Spring Boot, and doesn't need any Spring Framework support.

The entity which is being "gracefully shut down" here is not necessarily the entire Spring app context, but could just be the Spring Boot webapp. What is needed is a way for the Spring boot app to ask its EmbeddedServletContainer to stop accepting new requests, then to trigger a normal Spring shutdown when all in-flight requests have completed.

Since the TomcatEmbeddedServletContainer lives in the org.springframework.boot:spring-boot jar, this seems like a Spring Boot and/or a Tomcat feature request, rather than a Spring Framework feature.

I don't think there is currently any way to either a) ask Tomcat to stop accepting new requests without cancelling in-progress requests ("Pause" has problems, see above) or b) to ask Tomcat if there are any requests in progress, is there?

The thread above has gotten sidetracked into discussions of Spring-managed components dependency orders etc., which I think is not necessary for the desired feature.

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona May 30, 2018

Member

I don't think there is currently any way to either a) ask Tomcat to stop accepting new requests without cancelling in-progress requests or b) to ask Tomcat if there are any requests in progress, is there?

Both already possible, as shown by code that has already been shared above.

The thread above has gotten sidetracked into discussions of Spring-managed components dependency orders etc., which I think is not necessary for the desired feature.

We disagree. If we are to introduce the concept of graceful shutdown it's important that we introduce it in the right place. This applies irrespective of exactly what it is that's being gracefully shut down. We do not want to get into the situation where we introduce something that is specific to Spring Boot and its embedded container only to realise that a more general solution should have been introduced in Spring Framework and then built upon by Boot.

Member

wilkinsona commented May 30, 2018

I don't think there is currently any way to either a) ask Tomcat to stop accepting new requests without cancelling in-progress requests or b) to ask Tomcat if there are any requests in progress, is there?

Both already possible, as shown by code that has already been shared above.

The thread above has gotten sidetracked into discussions of Spring-managed components dependency orders etc., which I think is not necessary for the desired feature.

We disagree. If we are to introduce the concept of graceful shutdown it's important that we introduce it in the right place. This applies irrespective of exactly what it is that's being gracefully shut down. We do not want to get into the situation where we introduce something that is specific to Spring Boot and its embedded container only to realise that a more general solution should have been introduced in Spring Framework and then built upon by Boot.

@RichardBradley

This comment has been minimized.

Show comment
Hide comment
@RichardBradley

RichardBradley May 30, 2018

Both already possible, as shown by code that has already been shared above.

As per nithril's comment that's not a graceful shutdown, as the server continues to accept requests during the shutdown period, but stalls them and fails them all at once at the end, instead of unbinding from the socket which is what I understand by "graceful shutdown", like apache httpd does

Perhaps that is a Tomcat bug, rather than a Spring Boot one, though.

This applies irrespective of exactly what it is that's being gracefully shut down.

In my opinion, this is premature generalisation. There is a real and immediate need for Spring Boot webapps to have graceful HTTP shutdown, which it is not currently possible to meet, as seen in this thread. I don't see an immediate need for a generalised Spring Component shutdown ordering framework (and as noted above, there is already such a thing to some extent via DisposableBean). It seems like it might be easier to fix the current specific case before starting a big rethink about the proposed general case.

Regardless of whether we do a generalised shutdown framework or a specific HTTP one, there is some integration work to be done in adding a graceful "unbind" command to each of the implementations of EmbeddedServletContainer. If that work were done now, the specific case would be within easy reach. I think the general case is a distraction from that.

Of course, I am just an interested bystander, so it is up to the project maintainers where to take this :-)

RichardBradley commented May 30, 2018

Both already possible, as shown by code that has already been shared above.

As per nithril's comment that's not a graceful shutdown, as the server continues to accept requests during the shutdown period, but stalls them and fails them all at once at the end, instead of unbinding from the socket which is what I understand by "graceful shutdown", like apache httpd does

Perhaps that is a Tomcat bug, rather than a Spring Boot one, though.

This applies irrespective of exactly what it is that's being gracefully shut down.

In my opinion, this is premature generalisation. There is a real and immediate need for Spring Boot webapps to have graceful HTTP shutdown, which it is not currently possible to meet, as seen in this thread. I don't see an immediate need for a generalised Spring Component shutdown ordering framework (and as noted above, there is already such a thing to some extent via DisposableBean). It seems like it might be easier to fix the current specific case before starting a big rethink about the proposed general case.

Regardless of whether we do a generalised shutdown framework or a specific HTTP one, there is some integration work to be done in adding a graceful "unbind" command to each of the implementations of EmbeddedServletContainer. If that work were done now, the specific case would be within easy reach. I think the general case is a distraction from that.

Of course, I am just an interested bystander, so it is up to the project maintainers where to take this :-)

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona May 30, 2018

Member

that's not a graceful shutdown

That's a matter of opinion and is something that's already been raised and addressed above. We're just retreading old ground here without anything new being added to the discussion which is a waste of both my time and yours.

There is a real and immediate need for Spring Boot webapps to have graceful HTTP shutdown, which it is not currently possible to meet

The comments above show that's not the case. There are multiple users who have commented above with examples of something that works for their specific needs.

It seems like it might be easier to fix the current specific case before starting a big rethink about the proposed general case.

Being easier doesn't make it the right thing to do. If we forge ahead now with something that then doesn't fit well without something that's subsequently added to Spring Framework we'll be left with a mess that will be difficult to straighten out. We need to take the time now to think things through and give us the best possible chance to get things right first time.

Member

wilkinsona commented May 30, 2018

that's not a graceful shutdown

That's a matter of opinion and is something that's already been raised and addressed above. We're just retreading old ground here without anything new being added to the discussion which is a waste of both my time and yours.

There is a real and immediate need for Spring Boot webapps to have graceful HTTP shutdown, which it is not currently possible to meet

The comments above show that's not the case. There are multiple users who have commented above with examples of something that works for their specific needs.

It seems like it might be easier to fix the current specific case before starting a big rethink about the proposed general case.

Being easier doesn't make it the right thing to do. If we forge ahead now with something that then doesn't fit well without something that's subsequently added to Spring Framework we'll be left with a mess that will be difficult to straighten out. We need to take the time now to think things through and give us the best possible chance to get things right first time.

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona May 31, 2018

Member

With Jetty, I think things can be a bit simpler than the example shared above. All you need to do is to add the StatisticsHandler:

@Bean
public WebServerFactoryCustomizer<JettyServletWebServerFactory> jettyCustomizer() {
    return (factory) -> {
        factory.addServerCustomizers((server) -> {
            StatisticsHandler handler = new StatisticsHandler();
            handler.setHandler(server.getHandler());
            server.setHandler(handler);
        });
    };
}

This is sufficient to switch on Jetty's graceful shutdown with its default timeout period of 30 seconds. To customise the timeout, a call to server.setStopTimeout(timeInMillis) can be made in the same server customizer. Closing the application context will take care of calling Server.stop().

Jetty's behaviour during graceful shutdown is as follows:

  1. New connections are refused
  2. Wait for up to the shutdown timeout period for the number of active requests to drop to zero
  3. Drop the connections for any requests that are still active
Member

wilkinsona commented May 31, 2018

With Jetty, I think things can be a bit simpler than the example shared above. All you need to do is to add the StatisticsHandler:

@Bean
public WebServerFactoryCustomizer<JettyServletWebServerFactory> jettyCustomizer() {
    return (factory) -> {
        factory.addServerCustomizers((server) -> {
            StatisticsHandler handler = new StatisticsHandler();
            handler.setHandler(server.getHandler());
            server.setHandler(handler);
        });
    };
}

This is sufficient to switch on Jetty's graceful shutdown with its default timeout period of 30 seconds. To customise the timeout, a call to server.setStopTimeout(timeInMillis) can be made in the same server customizer. Closing the application context will take care of calling Server.stop().

Jetty's behaviour during graceful shutdown is as follows:

  1. New connections are refused
  2. Wait for up to the shutdown timeout period for the number of active requests to drop to zero
  3. Drop the connections for any requests that are still active
@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona May 31, 2018

Member

Here's the equivalent configuration for Undertow:

@Bean
public GracefulUndertowShutdown GracefulUndertowShutdown() {
    return new GracefulUndertowShutdown();
}

@Bean
public WebServerFactoryCustomizer<UndertowServletWebServerFactory> undertowCustomizer(
        GracefulUndertowShutdown gracefulUndertowShutdown) {
    return (factory) -> {
        factory.addDeploymentInfoCustomizers((builder) -> {
            builder.addInitialHandlerChainWrapper(gracefulUndertowShutdown);
        });
    };
}

private static class GracefulUndertowShutdown
        implements ApplicationListener<ContextClosedEvent>, HandlerWrapper {

    private volatile GracefulShutdownHandler handler;

    @Override
    public HttpHandler wrap(HttpHandler handler) {
        this.handler = new GracefulShutdownHandler(handler);
        return this.handler;
    }

    @Override
    public void onApplicationEvent(ContextClosedEvent event) {
        try {
            this.handler.shutdown();
            this.handler.awaitShutdown(30000);
        }
        catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
        }
    }

}

Undertow's behaviour during graceful shutdown is as follows:

  1. Respond to new requests with a 503
  2. Wait for up to the shutdown timeout period for the number of active requests to drop to zero
  3. Drop the connections for any requests that are still active
Member

wilkinsona commented May 31, 2018

Here's the equivalent configuration for Undertow:

@Bean
public GracefulUndertowShutdown GracefulUndertowShutdown() {
    return new GracefulUndertowShutdown();
}

@Bean
public WebServerFactoryCustomizer<UndertowServletWebServerFactory> undertowCustomizer(
        GracefulUndertowShutdown gracefulUndertowShutdown) {
    return (factory) -> {
        factory.addDeploymentInfoCustomizers((builder) -> {
            builder.addInitialHandlerChainWrapper(gracefulUndertowShutdown);
        });
    };
}

private static class GracefulUndertowShutdown
        implements ApplicationListener<ContextClosedEvent>, HandlerWrapper {

    private volatile GracefulShutdownHandler handler;

    @Override
    public HttpHandler wrap(HttpHandler handler) {
        this.handler = new GracefulShutdownHandler(handler);
        return this.handler;
    }

    @Override
    public void onApplicationEvent(ContextClosedEvent event) {
        try {
            this.handler.shutdown();
            this.handler.awaitShutdown(30000);
        }
        catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
        }
    }

}

Undertow's behaviour during graceful shutdown is as follows:

  1. Respond to new requests with a 503
  2. Wait for up to the shutdown timeout period for the number of active requests to drop to zero
  3. Drop the connections for any requests that are still active
@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona May 31, 2018

Member

With Jetty, I think things can be a bit simpler than the example shared above.

I take this back.

I said above that " closing the application context will take care of calling Server.stop()". That's true, but it'll get Jetty into a state where it rejects new connections after most of the context close processing has occurred. It needs to be happen earlier and when ContextRefreshedEvent is published is an ideal time to do so.

Having thought some more, I think it's actually a bit more complicated than the example shared above. We need to get Jetty into a state where it rejects new connections without actually calling Server.stop(). If we call Server.stop(), we may encounter problems that are similar to those that we encountered when we tried to invert the ordering of context close and server stop.

Member

wilkinsona commented May 31, 2018

With Jetty, I think things can be a bit simpler than the example shared above.

I take this back.

I said above that " closing the application context will take care of calling Server.stop()". That's true, but it'll get Jetty into a state where it rejects new connections after most of the context close processing has occurred. It needs to be happen earlier and when ContextRefreshedEvent is published is an ideal time to do so.

Having thought some more, I think it's actually a bit more complicated than the example shared above. We need to get Jetty into a state where it rejects new connections without actually calling Server.stop(). If we call Server.stop(), we may encounter problems that are similar to those that we encountered when we tried to invert the ordering of context close and server stop.

@ramato-procon

This comment has been minimized.

Show comment
Hide comment
@ramato-procon

ramato-procon Jun 1, 2018

In my opinion there is only one logic which should work for all kind of loadbalancers:

  1. Stop accepting new requests (new connections or channels are refused)
  2. Waiting until all requests have been processed or shutdown timeout
  3. Shutdown Spring context

Responding with 503 only works for "real" http loadbalancers. No iptables or tcp (e.g. for ssl passthrough) loadbalancing.

ramato-procon commented Jun 1, 2018

In my opinion there is only one logic which should work for all kind of loadbalancers:

  1. Stop accepting new requests (new connections or channels are refused)
  2. Waiting until all requests have been processed or shutdown timeout
  3. Shutdown Spring context

Responding with 503 only works for "real" http loadbalancers. No iptables or tcp (e.g. for ssl passthrough) loadbalancing.

@wilkinsona wilkinsona modified the milestones: Icebox, Backlog Jun 1, 2018

@pmhsfelix

This comment has been minimized.

Show comment
Hide comment
@pmhsfelix

pmhsfelix Jun 12, 2018

@wilkinsona: does the first example works correctly with asynchronous processing? There can be pending requests and no busy executor threads. So, unless there is special logic on the executor, the shutdown could succeed even with pending requests.

pmhsfelix commented Jun 12, 2018

@wilkinsona: does the first example works correctly with asynchronous processing? There can be pending requests and no busy executor threads. So, unless there is special logic on the executor, the shutdown could succeed even with pending requests.

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona Jun 13, 2018

Member

@pmhsfelix I don't know yet. @snicoll and I discussed it briefly last week and it's something that I need to investigate for all our supported containers.

Member

wilkinsona commented Jun 13, 2018

@pmhsfelix I don't know yet. @snicoll and I discussed it briefly last week and it's something that I need to investigate for all our supported containers.

@pdeva

This comment has been minimized.

Show comment
Hide comment
@pdeva

pdeva Jul 11, 2018

@wilkinsona could you update example for #4657 (comment) with code for Spring Boot 2?

it seems the relevant classes have been renamed and moved to different packages in Spring boot 2.

pdeva commented Jul 11, 2018

@wilkinsona could you update example for #4657 (comment) with code for Spring Boot 2?

it seems the relevant classes have been renamed and moved to different packages in Spring boot 2.

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona Jul 13, 2018

Member

Jetty does not consider asynchronous requests when shutting down gracefully. I've opened eclipse/jetty.project#2717.

Member

wilkinsona commented Jul 13, 2018

Jetty does not consider asynchronous requests when shutting down gracefully. I've opened eclipse/jetty.project#2717.

@buch11

This comment has been minimized.

Show comment
Hide comment
@buch11

buch11 Jul 16, 2018

Update : ---IGNORE THIS COMMENT---

I am not sure if this a widely faced problem or not but what I am facing is, we have a kafka stream processor running as a spring boot app(parent : spring-boot-starter-parent) inside a docker container.

As soon as we do a docker stop on container, we are observing 503s for some of the outbound calls to external REST api(using resttemplates).

Has there been any workarounds or plans to release a spring-boot version with a fix?

buch11 commented Jul 16, 2018

Update : ---IGNORE THIS COMMENT---

I am not sure if this a widely faced problem or not but what I am facing is, we have a kafka stream processor running as a spring boot app(parent : spring-boot-starter-parent) inside a docker container.

As soon as we do a docker stop on container, we are observing 503s for some of the outbound calls to external REST api(using resttemplates).

Has there been any workarounds or plans to release a spring-boot version with a fix?

@wilkinsona

This comment has been minimized.

Show comment
Hide comment
@wilkinsona

wilkinsona Jul 16, 2018

Member

@buch11 Please don’t drag this issue off-topic. Outbound HTTP calls in a Docker container are unrelated to graceful shutdown of the embedded container.

Member

wilkinsona commented Jul 16, 2018

@buch11 Please don’t drag this issue off-topic. Outbound HTTP calls in a Docker container are unrelated to graceful shutdown of the embedded container.

@kppullin

This comment has been minimized.

Show comment
Hide comment
@kppullin

kppullin Sep 18, 2018

@pdeva - Here's an updated snippet for Spring Boot 2:

    @Bean
    public WebServerFactoryCustomizer tomcatCustomizer() {
        return factory -> {
            if (factory instanceof TomcatServletWebServerFactory) {
                ((TomcatServletWebServerFactory) factory)
                        .addConnectorCustomizers(gracefulShutdown());
            }
        };
    }

    private static class GracefulShutdown implements TomcatConnectorCustomizer,
            ApplicationListener<ContextClosedEvent> {

        private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);

        private volatile Connector connector;

        @Override
        public void customize(Connector connector) {
            this.connector = connector;
        }

        @Override
        public void onApplicationEvent(ContextClosedEvent event) {
            Executor executor = this.connector.getProtocolHandler().getExecutor();
            if (executor instanceof ThreadPoolExecutor) {
                try {
                    ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
                    threadPoolExecutor.shutdown();
                    if (!threadPoolExecutor.awaitTermination(30, TimeUnit.SECONDS)) {
                        log.warn("Tomcat thread pool did not shut down gracefully within "
                                + "30 seconds. Proceeding with forceful shutdown");
                    }
                }
                catch (InterruptedException ex) {
                    Thread.currentThread().interrupt();
                }
            }
        }
    }

kppullin commented Sep 18, 2018

@pdeva - Here's an updated snippet for Spring Boot 2:

    @Bean
    public WebServerFactoryCustomizer tomcatCustomizer() {
        return factory -> {
            if (factory instanceof TomcatServletWebServerFactory) {
                ((TomcatServletWebServerFactory) factory)
                        .addConnectorCustomizers(gracefulShutdown());
            }
        };
    }

    private static class GracefulShutdown implements TomcatConnectorCustomizer,
            ApplicationListener<ContextClosedEvent> {

        private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);

        private volatile Connector connector;

        @Override
        public void customize(Connector connector) {
            this.connector = connector;
        }

        @Override
        public void onApplicationEvent(ContextClosedEvent event) {
            Executor executor = this.connector.getProtocolHandler().getExecutor();
            if (executor instanceof ThreadPoolExecutor) {
                try {
                    ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
                    threadPoolExecutor.shutdown();
                    if (!threadPoolExecutor.awaitTermination(30, TimeUnit.SECONDS)) {
                        log.warn("Tomcat thread pool did not shut down gracefully within "
                                + "30 seconds. Proceeding with forceful shutdown");
                    }
                }
                catch (InterruptedException ex) {
                    Thread.currentThread().interrupt();
                }
            }
        }
    }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment