Graceful Shutdown

Graceful shutdown

Run blocks until one of four things happens:

• SIGINT
• SIGTERM
• the context passed to RunContext is cancelled
• Stop() is called

Then:

listeners stopped        ← nothing new arrives
OnStop hooks             ← bounded by ShutdownTimeout
OnShutdown hooks         ← run even if OnStop failed
event bus closed
Run returns

Ordering, and why

Listeners first. Stopping them before running hooks means nothing new arrives while extensions are tearing down. Router Stop is idempotent, so a hook that stops a router again is harmless.

OnShutdown runs even if OnStop failed. Skipping it would leak whatever the remaining hooks were going to release. Errors from both are joined and returned from Run.

The event bus closes last, after both hook rounds, so a shutdown hook can still emit — and it closes even if a hook failed, because its workers are goroutines owned by the instance.

Exactly once

Stop(), a signal and a cancelled context can arrive together. The shutdown path is guarded, so hooks run once whichever combination fires. (Before this, a Stop() followed by SIGTERM ran every shutdown hook twice.)

Stop during startup

A Stop() arriving mid-startup waits for startup to finish before any stop hook runs. Without that, an extension's OnReady and OnStop could execute concurrently on the same instance.

The wait is bounded by ShutdownTimeout: a startup that hangs cannot make shutdown hang with it. If the bound is hit, the hooks run anyway and a warning is logged — leaking the process is worse than an overlapping hook.

Stop() on an instance that was never Run returns immediately. That is the common shape in tests (defer app.Stop() on an application built but never started) and a plausible one in an application that fails during construction.

Timeout

rex.WithConfig(&rex.Config{ShutdownTimeout: 30 * time.Second})

10 seconds by default. It bounds the hook rounds and the startup wait above.

Set it below your orchestrator's kill timeout. Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds, then SIGKILL — a ShutdownTimeout longer than that grace period means the process is killed mid-drain and the timeout never does anything.

In your own code

Respect the context your hooks are given:

func (e *Extension) OnStop(ctx context.Context, r rextension.Rex) error {
	select {
	case <-e.drained:
		return nil
	case <-ctx.Done():
		return fmt.Errorf("drain did not finish: %w", ctx.Err())
	}
}

A hook that blocks past the deadline delays every hook after it.

Handling the exit

if err := app.Run(); err != nil {
	app.Logger().WithError(err).Error("shutdown reported errors")
	os.Exit(1)
}

A non-nil return after a clean signal means a shutdown hook failed — the application still stopped, but something did not release cleanly, and that is worth a non-zero exit code.

Stopping from elsewhere

go func() {
	<-adminStopRequested
	if err := app.Stop(); err != nil {
		log.Printf("stop: %v", err)
	}
}()

Stop unblocks Run and runs the shutdown hooks, whether Run is waiting or not.