Gracefully stop Octopus Server if running in terminal and process terminated

[zentron]

Whats missing?

When the Octopus Server is running as a windows service, we listen for the Stop signal and react by gracefully closing the web listeners and flushing all logs. Similarly when running in a terminal we listen on StdIn and perform the same shutdown if a newline is entered

If however, the user types Ctrl+C, closes the window, the machine shuts down or in the case of Octopus Docker images, docker stop is invoked, then none of graceful shutdown code gets a chance to run.

We plan to build on this shutdown process later by ensuring server node records are appropriately updated if the Octopus Server process knows it is shutting down.

[zentron]

NET472

There are various hooks available in .NET to detect different shutdown events. Given the following source code

	class Program
	{
		[DllImport("Kernel32")]
		internal static extern bool SetConsoleCtrlHandler(HandlerRoutine Handler, bool Add);

		[DllImport("msvcrt.dll", PreserveSig = true)]
		static extern SignalHandler signal(int sig, SignalHandler handler);

		internal delegate bool HandlerRoutine(CtrlTypes CtrlType);

		[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
		delegate void SignalHandler(int sig);

		const int SIGINT  = 2; // Ctrl-C
		const int SIGFPE  = 8;
		const int SIGTERM = 15; // process termination
		const int WM_CLOSE = 0x0010;

		internal enum CtrlTypes
		{
			CTRL_C_EVENT = 0,
			CTRL_BREAK_EVENT,
			CTRL_CLOSE_EVENT,
			CTRL_LOGOFF_EVENT = 5,
			CTRL_SHUTDOWN_EVENT
		}

		static void Main(string[] args)
		{
			Console.CancelKeyPress += (s, e) => Console.WriteLine($"Console.CancelKeyPress {e.SpecialKey}, thread {Thread.CurrentThread.ManagedThreadId}");
			AppDomain.CurrentDomain.ProcessExit += (s, e) => Console.WriteLine($"CurrentDomain.ProcessExit, thread {Thread.CurrentThread.ManagedThreadId}");
			AppDomain.CurrentDomain.DomainUnload += (s, e) => Console.WriteLine($"CurrentDomain.DomainUnload, thread {Thread.CurrentThread.ManagedThreadId}");
			AssemblyLoadContext.Default.Unloading += context => Console.WriteLine($"AssemblyLoadContext.Unload, thread {Thread.CurrentThread.ManagedThreadId}");
			var hr = new HandlerRoutine(type => { Console.WriteLine($"ConsoleCtrlHandler {type}, thread {Thread.CurrentThread.ManagedThreadId}"); return false; });
			var sh = new SignalHandler(sig => Console.WriteLine($"Got signal {sig} on thread {Thread.CurrentThread.ManagedThreadId}"));

			Console.WriteLine($"SetConsoleCtrlHandler returned {SetConsoleCtrlHandler(hr, true)}");
			Console.WriteLine($"signal returned null: {signal(SIGTERM, sh) == null}");

			Console.WriteLine($"just sitting around waiting, on thread {Thread.CurrentThread.ManagedThreadId}");
			while( true )
				Thread.Sleep(1000);
			Console.WriteLine("uh, we exited the loop, wat");

			GC.KeepAlive(hr);
			GC.KeepAlive(sh);
		}
	}

We can test and confirm the events that are raised in the following scenarios

Ctrl+C

• Console.CancelKeyPress ControlC
• ConsoleCtrlHandler CTRL_C_EVENT

Closing Window

• ConsoleCtrlHandler CTRL_CLOSE_EVENT

Docker Stop

• ConsoleCtrlHandler CTRL_SHUTDOWN_EVENT
• SystemEvents.SessionEnded SystemShutdown, thread 3

The ConsoleCtrlHandler seems like a good option to listen for. Granted it is not guarenteed that these events will be raised but we can react to them with a best effort.

netcoreapp2.1

Since we don't want to rely on the unmanaged code calls used in the above example for dotnetcore where it may not be running on windows, we need to test and rely on some different events.

class Program
	{
		static void Main(string[] args)
		{
			Console.CancelKeyPress += (s, e) => Console.WriteLine($"Console.CancelKeyPress {e.SpecialKey}, thread {Thread.CurrentThread.ManagedThreadId}");
			AppDomain.CurrentDomain.ProcessExit += (s, e) => Console.WriteLine($"CurrentDomain.ProcessExit, thread {Thread.CurrentThread.ManagedThreadId}");
			AppDomain.CurrentDomain.DomainUnload += (s, e) => Console.WriteLine($"CurrentDomain.DomainUnload, thread {Thread.CurrentThread.ManagedThreadId}");
			AssemblyLoadContext.Default.Unloading += context => {Thread.CurrentThread.ManagedThreadId}");
			while( true )
				Thread.Sleep(1000);
			Console.WriteLine("uh, we exited the loop, wat");
		}
	}

Testing this from linux we can confirm the following events are raised

SIGINT (also Ctrl+C)

• Console.CancelKeyPress ControlC

SIGQUIT

• Console.CancelKeyPress ControlBreak

SIGTERM (also Docker Stop)

• CurrentDomain.ProcessExit

In this case it seems reasonable to hook into the Console.CancelKeyPress event and CurrentDomain.ProcessExit. It's a little surprising that the ProcessExit event isn't fired for the other two signals but we can listen to as many as we need and simply use a mutex to ensure it is only invoked at most once.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you think you've found a related issue, please contact our support team so we can triage your issue, and make sure it's handled appropriately.