SIGTERM handling

[pmcgrath]

Hi,
I'm seeing issues with handling sigterm on a linux system, seeing similar behaviour to 516 but on linux

Environment info
OS: ubuntu 16.04 64bit
dotnet --version is 2.1.300

As the docs indicate WaitForShutdown should handle sigterm, sigint and ctrl-c
Can see it handling sigint and ctrl-c as expected, but sigterm is not behaving as documented for me
Since sigterm is what docker will send to a container being stopped this is pretty important for graceful shutdowns with log entries

I create a standard webapi project with dotnet new webapi and altered the Program.cs as follows

using System;
using System.Linq;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server.Features;

namespace shutdown
{
    public class Program
    {
        public static void Main(string[] args)
        {
            try
            {
                Console.WriteLine("About to build web host");
                using (var host = BuildWebHost(args))
                {
                    Console.WriteLine("About to start host");
                    host.Start();

                    var serverAddressFeature = host.ServerFeatures.Get<IServerAddressesFeature>();
                    Console.WriteLine($"Waiting for shutdown, listening on {String.Join(", ", serverAddressFeature.Addresses.ToArray())}");
                    host.WaitForShutdown();

                    Console.WriteLine("Shutting down");
                }
            }
            catch (Exception exception)
            {
                Console.WriteLine($"Exception encountered, about to die {exception}");
                throw;
            }
            finally
            {
                Console.WriteLine("Exiting finally");
            }
        }

        public static IWebHost BuildWebHost(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
                .UseStartup<Startup>()
                .Build();
    }
}

So ctlr-c and kill -INT pid are working as expected and the exiting log lines appear

But kill pid is just resulting in the process terminating with no exiting log lines, this I did not expect and is an issue with docker where I need it to handle a sigterm

Thanks in advance
Pat

[khellang]

The last time I checked, the recommendation was to hook up the AssemblyLoadContext.Default.Unloading event for SIGTERM, but that doesn't seem to happen:

https://github.com/aspnet/Hosting/blob/e54dcc023ed0b548fe318de72d2b27f40c427f03/src/Microsoft.AspNetCore.Hosting/WebHostExtensions.cs#L134-L162

I wonder if the AppDomain.CurrentDomain.ProcessExit event is meant to also catch that, or if it's missing altogether.

[khellang]

Ah, this comment clears up the confusion; https://github.com/dotnet/coreclr/issues/2688#issuecomment-394843595

[pmcgrath]

@khellang Looks like the docs are incorrect so ?

They indicate it handles a sigterm, when actaully the code seems to handle sigint and not sigterm

[khellang]

According to the comment I linked:

SIGTERM raises the AssemblyLoadContext.Default.Unloading and AppDomain.CurrentDomain.ProcessExit events.

So the documentation should be correct.

[pmcgrath]

Linked to wrong doc earlier, should have been WaitForShutdown that indicates it blocks for a ctrl-c or sigterm

What I am seeing with a sigterm, is that the code after WaitForShutdown is not executed, the process is actually terminated, but a sigint/ctrl-c does execute these lines and seems to have a grace period for inflight requests.

If I add a AppDomain.CurrentDomain.ProcessExit event handler as you indicated, then I could treat like a continuation I guess and do my cleanup there

[khellang]

Yeah, I think there's a difference in how Console.CancelKeyPress (which handles CTRL-C and SIGINT) and AppDomain.CurrentDomain.ProcessExit (which handles SIGTERM) are handled.

In the former case, if you set eventArgs.Cancel = true, it'll run the main thread to completion, i.e. executing the Console.WriteLine("Shutting down") call.

But in the latter case, it'll just wait for the event delegate to complete (which should still wait for in-flight requests to finish (I think it's 2 seconds?)), then shut down immediately.

[pmcgrath]

Doesn't wait to leave inflight requests complete with sigterm, but does seem to with sigint and ctrl-c, I have seen the 2 seconds mentioned in docs and have observed it allowing some time for them to finish

So it looks like I need to cater for this in the event handler as you said

Surprises me as this is going to be something anyone running on docker or kubernetes is going to have to deal with, would prefer to do so in Program.main as we may have other hosts i.e. GenericHost's running, don't want to have use IApplicationLifetime

[davidfowl]

IApplicationLifetime works with the generic host as well.

[pmcgrath]

FYI
My use case is for kubernetes\docker workloads and since aspnet.core seems to do a graceful shutdown for SIGINT and I did not want to have to write a AppDomain.CurrentDomain.ProcessExit event handler in my case

There are 2 options that I could see at this time
a) Use dockerfile STOPSIGNAL instruction
See https://docs.docker.com/engine/reference/builder/#stopsignal
Can get docker runtime to turn the default stop signal to be any signal, default is SIGTERM, but we can override in the dockerfile to send a SIGINT instead

b) Use k8s preStop lifecyle event
See https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
Can get this to send a SIGINT and allow some time for a graceful shutdown

Option a is the one I went with

[muratg]

Closing this as no further action is planned here.