Unix socket not cleaned up on exit

[Daniel15]

Describe the bug

I hit a similar issue in 2015 (aspnet/KestrelHttpServer#419) and I think it's still lingering today. When I shutdown the systemd service for an ASP.NET site, it won't restart again as the UNIX socket is not deleted on shutdown. I can work around the issue by deleting the socket in Program.cs, but Kestrel should be doing this itself on shutdown.

To Reproduce

Steps to reproduce the behavior:

• Created a new site using the latest ASP.NET Core 3.0 preview
• Published from Visual Studio with "self-contained" deployment mode, targeting linux-x64 runtime
• Created systemd unit as per https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/linux-nginx?view=aspnetcore-2.2, /etc/systemd/system/test-site.service:

[Unit]
Description=Test ASP.NET Core Site

[Service]
WorkingDirectory=/var/www/test-site/
ExecStart=/var/www/test-site/Daniel15.Test
Restart=always
# Restart service after 10 seconds if the dotnet service crashes:
RestartSec=10
KillSignal=SIGINT
SyslogIdentifier=test-site
User=www-data
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=ASPNETCORE_URLS=http://unix:/tmp/test-site.sock

[Install]
WantedBy=multi-user.target

• Enable and start the unit:

sudo service test-site enable
sudo service test-site start

It works fine.

• Try to restart it:

sudo service test-site restart

It fails to start:

Sep 18 23:48:34 syd02.d.sb test-site[19486]: System.IO.IOException: Failed to bind to address http://unix:/tmp/test-site.sock: a
Sep 18 23:48:34 syd02.d.sb test-site[19486]:  ---> Microsoft.AspNetCore.Connections.AddressInUseException: Address already in use
Sep 18 23:48:34 syd02.d.sb test-site[19486]:  ---> System.Net.Sockets.SocketException (98): Address already in use
Sep 18 23:48:34 syd02.d.sb test-site[19486]:    at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketErr
Sep 18 23:48:34 syd02.d.sb test-site[19486]:    at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketA
Sep 18 23:48:34 syd02.d.sb test-site[19486]:    at System.Net.Sockets.Socket.Bind(EndPoint localEP)
Sep 18 23:48:34 syd02.d.sb test-site[19486]:    at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bi
Sep 18 23:48:34 syd02.d.sb test-site[19486]:    --- End of inner exception stack trace ---

Expected behavior

Should restart cleanly

Additional context

dotnet --info on my computer:

λ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview9-014004
 Commit:    8e7ef240a5

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18362
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.0.100-preview9-014004\

Host (useful for support):
  Version: 3.0.0-preview9-19423-09
  Commit:  2be172345a

.NET Core SDKs installed:
  1.0.0-preview2-003121 [C:\Program Files\dotnet\sdk]
  1.0.0-preview2-1-003177 [C:\Program Files\dotnet\sdk]
  1.0.0-rc4-004771 [C:\Program Files\dotnet\sdk]
  1.0.0 [C:\Program Files\dotnet\sdk]
  1.0.4 [C:\Program Files\dotnet\sdk]
  1.1.0 [C:\Program Files\dotnet\sdk]
  1.1.9 [C:\Program Files\dotnet\sdk]
  1.1.11 [C:\Program Files\dotnet\sdk]
  1.1.12 [C:\Program Files\dotnet\sdk]
  2.0.0 [C:\Program Files\dotnet\sdk]
  2.0.2 [C:\Program Files\dotnet\sdk]
  2.1.201 [C:\Program Files\dotnet\sdk]
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.503 [C:\Program Files\dotnet\sdk]
  2.1.504 [C:\Program Files\dotnet\sdk]
  2.1.602 [C:\Program Files\dotnet\sdk]
  2.2.103 [C:\Program Files\dotnet\sdk]
  2.2.202 [C:\Program Files\dotnet\sdk]
  3.0.100-preview9-014004 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0-preview9.19424.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0-preview9-19423-09 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0-preview9-19423-09 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

On the server, it's using a self-contained deployment so I don't think there's a dotnet ---info command I can run, but it's using 3.0.0-preview9-19423-09

[davidfowl]

We hit the same thing in our tests. It does feel strange that we don't explicitly create it yet we're responsible for deleting it. We do the same in our tests 😄 https://github.com/aspnet/AspNetCore/blob/master/src/Servers/Kestrel/test/FunctionalTests/UnixDomainSocketsTests.cs#L113 (maybe that should have been a sign).

This seems like something we should fix in 3.1 cc @anurse

[Daniel15]

Is it possible to mention this issue in the readme of ASP.NET Core 3.0, or perhaps just in the Kestrel docs? Anyone using it with UNIX sockets will hit this issue.

[analogrelay]

We should, at least, provide an option to control deletion behavior. If we're worried about the breaking change of always deleting, we can make the option off by default.

[Daniel15]

This is how I'm working around it for now:

		public static void Main(string[] args)
		{
			var host = CreateWebHostBuilder(args).Build();
			DeleteOldSocketIfExists(host);
			host.Run();
		}

		private static void DeleteOldSocketIfExists(IWebHost host)
		{
			// Delete UNIX pipe if it exists at startup (eg. previous process crashed before cleaning it up)
			// Workaround for https://github.com/aspnet/AspNetCore/issues/14134
			var addressFeature = host.ServerFeatures.Get<IServerAddressesFeature>();
			var url = BindingAddress.Parse(addressFeature.Addresses.First());
			if (url.IsUnixPipe && File.Exists(url.UnixPipePath))
			{
				Console.WriteLine("UNIX pipe {0} already existed, deleting it.", url.UnixPipePath);
				File.Delete(url.UnixPipePath);
			}
		}

[Yuki-Nagato]

While @Daniel15 provided a workaround, it may not help if using Generic Host in ASP.NET Core 3.0.

[sherlock1982]

BTW I advice to cleanup on startup rather than on shutdown because your app might crash and than socket will remain there anyway. This can be done in systemd hooks.

[Daniel15]

This can be done in systemd hooks.

Do you have an example of this?