-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ConsoleLifetime doesn't allow full graceful shutdown for ProcessExit #35990
Comments
A couple other things that likely currently get lost: full disposing of custom service providers, full disposing of custom hosts. I ended up working around this with a wrapper around the entry point for now. Here's what it looks like: class Program
{
static void Main()
{
ProcessLifetimeHost.Run(MainCore);
}
static void MainCore(IProcessLifetime processLifetime)
{
new HostBuilder()
// Configure etc
.UseProcessLifetime(processLifetime)
.Build()
.Run();
}
} When run with the original set of things hooked up above, all of the cleanup code runs (including code outside the using block and setting the exit code). The IProcessLifetime interface looks like this: /// <summary>Defines a process lifetime.</summary>
interface IProcessLifetime
{
/// <summary>
/// Gets or sets the action to invoke to initiate graceful shutdown of currently-running application, if any.
/// </summary>
Action StopApplication { get; set; }
} If you're interested in going down this route further, let me know and I can provide the other details as well (UseProcessLifetime extension method and ProcessLifetimeHost.Run method). |
We'll likely need some corefx support on this one to give us an API to properly handle SIGTERM. The rough proposal we have right now is to have a new API in corefx to hook the It means that if the app doesn't gracefully shutdown, |
Unfortunately, I don't think it's possible for corefx to provide a cancellable API on Windows - the underlying signal sent by docker stop on Windows containers changed to CTRL_SHUTDOWN_EVENT, which can't be canceled. The MSDN docs could be a little clearer - they more say what can be done in this case (inline cleanup in the handler) rather than what can't be done (canceling the signal). See also a related moby issue for details. |
Good to know. I was referring to graceful termination on Linux which sends This will be a tricky one, we'll have to investigate some options. |
Hello! |
It is possible to achieve graceful shutdown in a Windows container, but it currently requires a bit of custom code to make it work. Specifically, it requires using P/Invoke to get a shutdown notification (see dotnet/extensions#2029 and moby/moby#25982). And then this signal cannot be canceled, so it requires different wiring to make it work with IHostBuilder/Host. The ProcessLifetimeHost snippet above shows how I'm currently doing that from a design standpoint (the implementation is rather complicated). So yes it's possible, but it requires custom code and is not easy. |
@davidmatson |
My hope with this bug was to get a solution into the official product, so I wasn't planning on publishing my implementation. I'd have to check, but I could probably paste it here as a temporary workaround if there's interest. |
@davidmatson |
Here's a zip of the source I'm using as a workaround. The usage is as described above. I'm using it with v2.2.0 of the hosting extensions package (it may work with a later version but I haven't tested that). Disclaimer: as-is; just a workaround; not an official solution; not offering support for this code; etc. |
This has cost me the better part of the day :-( Whenever I ran my application on the command line on Linux, CTRL+C would gracefully exit and dispose my Autofac container. When running as systemd service, that would just start the disposing work, but then stop in the middle of it. Since that service would normally only be stopped when my embedded instrument with a volatile system log would be shut down, I had to unwrap a lot of layers before this became clear. For the time being as a workaround, I have internalized the code for Is there a planned path forward already? I think this should be more than "Backlog"… |
Triage: The right approach here is twofold:
|
The above approach would also fix dotnet/extensions#1363 |
@anurse - the equivalent of SIGTERM on Windows/Docker (CTRL_SHUTDOWN_EVENT) can't be canceled. Thoughts? |
Sharing my view of dotnet/extensions#1363 (comment). ConsoleLifetime shouldn't be used by default. That avoids the issue because the Host no longer assumes it gets disposed on ProcessExit. |
The console host should be used by default when you call CreateDefaultBuilder, I'd be ok removing it when you new up the host.
It doesn't we still need to fix it. |
Why not? The issue is caused by ConsoleLifetime right? |
Because when you call CreateDefaultBuilder the problem still exists. |
I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label. |
Moving to 6.0 as it involves API work. |
@davidmatson I know this issue is old but how are you exiting the process here? AFAIK there's no way to make this work without runtime changes. Ideally, we want Main to run to completion before exiting for these un-cancellable signals. |
The repro steps at the top exit the process using
|
For me (#35990 (comment)), the issue became apparent through .NET's handling of
|
@jkotas - it's been awhile, but I believe the Environment.Exit was just a deterministic, programmatic way of simulating what happens when someone does Ctrl+C, docker stop, or similar. (I expect there are other ways to trigger this behavior as well, like an unhandled exception from another thread.) |
Normally, exiting the process by Ctrl+C or docker stop. |
Not exactly.
Ctrl-C and similar signals exit the process by default, but that behavior can be overriden. #50527 is about making the APIs for these signals better. This issue looks like duplicate of #50527 and can be closed. @davidfowl Do you agree?
Unhandled exception is abnormal exit. It does not follow the same path as |
Yeah, I'm not all that concerned with Environment.Exit per se as long as all the other cases (Ctrl+C, docker stop, system shutdown notification, and unhandled exceptions from other threads) are covered. I'm guessing the other issue was filed with this kind of scenario in mind; as long as we're covering the same cases there, closing this one in favor of that one sounds fine to me. |
Don't listen to ProcessExit on net6.0+ in Hosting anymore. This allows for Environment.Exit to not hang the app. Don't clobber ExitCode during ProcessExit now that SIGTERM is handled separately. For non-net6.0 targets, only wait for the shutdown timeout, so the process doesn't hang forever. Fix dotnet#55417 Fix dotnet#44086 Fix dotnet#50397 Fix dotnet#42224 Fix dotnet#35990
* Add NetCoreAppCurrent target to Microsoft.Extensions.Hosting * Handle SIGTERM in Hosting and handle just like SIGINT (CTRL+C) Don't listen to ProcessExit on net6.0+ in Hosting anymore. This allows for Environment.Exit to not hang the app. Don't clobber ExitCode during ProcessExit now that SIGTERM is handled separately. For non-net6.0 targets, only wait for the shutdown timeout, so the process doesn't hang forever. Fix #55417 Fix #44086 Fix #50397 Fix #42224 Fix #35990 * Remove _shutdownBlock on netcoreappcurrent, as this is no longer waited on * Change Console.CancelKeyPress to use PosixSignalRegistration SIGINT and SIGQUIT * Use a no-op lifetime on mobile platforms * Add docs for shutdown
@eerhardt - to clarify the status here: with the merged fix, does the repro at the top now work as in the Expected behavior section above (run all the cleanup steps listed and return an exit code of 123 from Main)? Thanks |
Also, does it work that way specifically on Windows? |
No. That is unreasonable expected behavior. See https://docs.microsoft.com/en-us/dotnet/api/system.environment.exit?view=net-5.0#remarks for what to expect with
When calling With the merged fix, Hosting no longer subscribes to In short, See https://github.com/dotnet/runtime/blob/main/src/libraries/Microsoft.Extensions.Hosting/docs/HostShutdown.md for more information. |
@eerhardt - sorry, I should clarify ignoring Environment.Exit to trigger process stop as in jkota's comment above. If, at that point in the code, CTRL+C or docker stop happens, will it now behave as in the Expected behavior section, even on Windows? Thanks |
I'm particularly interested in the docker stop case on Windows - at least last time I looked, docker stop used system shutdown, which is a non-cancellable event. |
Yes. When using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using System;
using System.Threading;
class Program
{
static int Main()
{
var builder = new HostBuilder()
.ConfigureLogging(l =>
{
l.AddConsole();
l.Services.AddSingleton<ILoggerProvider, CustomLoggerProvider>();
}).ConfigureServices(s =>
{
s.AddSingleton<OtherService>();
});
using (IHost host = builder.Build())
{
host.Services.GetRequiredService<OtherService>();
host.Start();
//Thread thread = new Thread(() => Environment.Exit(456));
//thread.Start();
host.WaitForShutdown();
Console.WriteLine("Ran cleanup code inside using host block.");
}
Console.WriteLine("Ran cleanup code outside using host block.");
Console.WriteLine("Returning exit code from Program.Main.");
return 123;
}
class CustomLoggerProvider : ILoggerProvider
{
public ILogger CreateLogger(string categoryName)
{
return NullLogger.Instance;
}
public void Dispose()
{
Console.WriteLine("Ran logger cleanup code.");
}
}
class OtherService : IDisposable
{
public void Dispose()
{
Console.WriteLine("Ran most service's cleanup code.");
}
}
}
Above I ran |
Excellent. Thanks, @eerhardt! |
Describe the bug
When ConsoleLifetime shuts down via the AppDomain.ProcessExit path, it skips a number of steps done in normal graceful program termination.
Related to dotnet/extensions#1363, but perhaps the opposite problem (in this case, it shuts down too soon rather than hanging/too late).
To Reproduce
Program.cs:
App.csproj:
Expected behavior
The same cleanup code runs when exiting via ProcessExit as it did with Ctrl+C:
Actual behavior
Only the following cleanup code runs:
Note that the return code for a normal ProcessExit case (rather than one simulated by Environment.Exit) would likely be 3221225786 (STATUS_CONTROL_C_EXIT) on Windows for v2.2.0 of this package and 0 for the latest (I think; based on ConsoleLifetime source).
Additional context
Trapping ProcessExit and then deciding how long to wait overall is rather problematic; it could wait too long/block, as in dotnet/extensions#1363, or too short, as here. The main options I can think of are:
Overall, the host itself doesn't own the entirety of Program.Main, so having it decide when to stop running is tricky.
I did find that at least the logger cleanup can likely be done better by changing the order of the disposables in the service container, including by adding a first constructor parameter to Host that gets registered first - currently, the IHostApplicationLifetime is after loggers in the list of things to dispose, and the logic to dispose in reverse order on the service provider/scope means loggers don't get to cleanup.
For application insights, for example, having the logger not get to have Dispose called means it can't call Flush, so logs currently get lost on shutdown.
The full list of things that currently get lost are:
The text was updated successfully, but these errors were encountered: