net/http: server's Shutdown() method is not graceful enough

[chenjie199234]

Http server's registered func on shutdown is called in goroutine.
It will be interrupted by end of the process.

package main

import (
	"context"
	"fmt"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	var closing bool
	m := http.DefaultServeMux
	m.HandleFunc("/", root)
	s := &http.Server{
		Addr:    ":8080",
		Handler: m,
	}
	s.RegisterOnShutdown(shutdown_func1)
	s.RegisterOnShutdown(shutdown_func2)
	sigs := make(chan os.Signal, 1)
	signal.Notify(sigs, syscall.SIGINT)
	done := make(chan bool, 1)
	go func() {
		fmt.Println("start server")
		s.ListenAndServe()
		fmt.Println("end server")
		done <- true
	}()
	for {
		select {
		case <-sigs:
			if !closing {
				fmt.Println("start shutting down")
				closing = true
				s.Shutdown(context.Background())
				fmt.Println("end shutting down")
			}
		case <-done:
			return
		}
	}
}
func root(w http.ResponseWriter, r *http.Request) {
	fmt.Println("root")
}
func shutdown_func1() {
	time.Sleep(time.Second)
	fmt.Println("shutdown call back 1")
}
func shutdown_func2() {
	time.Sleep(time.Second)
	fmt.Println("shutdown call back 2")
}

[ALTree]

Hi @chenjie199234,

this is how Go works. When the main goroutine exists, the program is done. The main goroutine won't wait for other goroutines to end. Once you call return in the main function after <-done triggers, the main function will return without caring if other goroutines still have work to do (in this case, your callbacks).

You'll need some form of synchronization if you want to make main wait for the other goroutines to finish before exiting.

I'm closing this, since this is not a Go bug.

[alexedwards]

@ALTree I've just run into this same issue, and I'm not sure it should have been closed.

The problem is that http.Server.Shutdown() doesn't wait for the functions registered with http.Server.RegisterOnShutdown to complete before returning:

func (srv *Server) Shutdown(ctx context.Context) error {
	atomic.StoreInt32(&srv.inShutdown, 1)

	srv.mu.Lock()
	lnerr := srv.closeListenersLocked()
	srv.closeDoneChanLocked()
	for _, f := range srv.onShutdown {
		go f()
	}
	srv.mu.Unlock()

	ticker := time.NewTicker(shutdownPollInterval)
	defer ticker.Stop()
	for {
		if srv.closeIdleConns() {
			return lnerr
		}
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-ticker.C:
		}
	}
}

Even if this is intended behaviour of Shutdown() -- and I'm not sure it is -- then it's not intuitive and I think the documentation should be updated to warn that the functions you register with RegisterOnShutdown() may not complete before Shutdown() returns.

[ALTree]

Even if this is intended behaviour of Shutdown() -- and I'm not sure it is -- then it's not intuitive and I think the documentation should be updated to warn that the functions you register with RegisterOnShutdown() may not complete before Shutdown() returns.

Fair enough; I'm re-opening this for further investigation (and possibly to be converted to a doc issue).

[ALTree]

cc @bradfitz @tombergan

[acynothia]

for _, f := range srv.onShutdown {
	go f()
}

call onShutdown with goroutine, a WaitGroup maybe work. but it's not intuitive.

[rumsrami]

any update on this issue?

[rkuska]

Hey! I found this issue when browsing code in net/http. I gave it a try and wrote a quick fix, but it might be too complicated. Is there an interest to have this fixed and is this issue suitable for volunteers (newcomers)?

diff --git a/src/net/http/server.go b/src/net/http/server.go
index 58aff08424..e013e40d2f 100644
--- a/src/net/http/server.go
+++ b/src/net/http/server.go
@@ -2667,19 +2667,41 @@ var shutdownPollInterval = 500 * time.Millisecond
 // future calls to methods such as Serve will return ErrServerClosed.
 func (srv *Server) Shutdown(ctx context.Context) error {
        atomic.StoreInt32(&srv.inShutdown, 1)
+       var wg sync.WaitGroup
+       wgDone := make(chan struct{})

        srv.mu.Lock()
        lnerr := srv.closeListenersLocked()
        srv.closeDoneChanLocked()
+
+       wg.Add(len(srv.onShutdown))
        for _, f := range srv.onShutdown {
-               go f()
+               f := f
+               go func() {
+                       defer wg.Done()
+                       f()
+               }()
        }
        srv.mu.Unlock()

+       go func() {
+               defer close(wgDone)
+               wg.Wait()
+       }()
+
+       isOnShutdownDone := func() bool {
+               select {
+               case <-wgDone:
+                       return true
+               default:
+                       return false
+               }
+       }
+
        ticker := time.NewTicker(shutdownPollInterval)
        defer ticker.Stop()
        for {
-               if srv.closeIdleConns() {
+               if srv.closeIdleConns() && isOnShutdownDone() {
                        return lnerr
                }
                select {

[nhooyr-ts]

There's another issue here where if you call Shutdown twice, the callbacks will be invoked in a new goroutine twice...