kestrel - intermittent error - Unable to bind - EACCES permission denied

[Aardvark71]

We are running our asp.net core apps in production on Windows in IIS.
We are hosting out-of-process.

Now since a few weeks we have seen the following error in production :

Unable to bind to http://localhost:24834 on the IPv4 loopback interface: 'Error -4092 EACCES permission denied'.

(obviously, the exact port number differs between events)

It is clear that somewhere the assignment of the dynamic port went wrong. but I haven't found any info on how kestrel determines it's port to use.

Also clueless on how to troubleshoot this (when it occurs again).

I do know that the dynamic port range for tcp/ip has recently been changed by our webadmins.
( MaxUserPort increased from 16K to 32K, meaning that the dynamic port range is now 49152 to 65535 )
The intermittent errors have all happened after this change, but … port numbers that failed to bind are all still lower than 49152

So…. My question is… how does kestrel ‘get’ it’s port numer assigned and is it possible that our webadmins messed this up by adapting the MaxUserPort ?

[Aardvark71]

Just some additional info...

• Our intermittent error is the same as in this old kestrel issue :

• We are still on asp.net core 2.0

• And we are running on windows server 2008 r2

• We use this to fire up kestrel in the program class

            WebHost.CreateDefaultBuilder(args)
                .CaptureStartupErrors(true)
                .UseSetting("detailedErrors", "true")
                .UseStartup<CustomWebStartup>()
                .Build();

If I understand the documentation here correctly, then it is actually the ASP.NET Core Module in IIS that generates a dynamic port to assign to the backend process ?
So my question perhaps rather pertains the ASP.NET Core Module ?

[analogrelay]

how does kestrel ‘get’ it’s port numer assigned

When running out-of-process, IIS assigns a port number and passes it to Kestrel via an environment variable.

I can't see how changing MaxUserPort would cause this. Plus, you changed it by making the range larger rather than smaller, which should reduce the likelihood of this issue rather than increasing it.

How frequently do these issues occur? Do you have other applications installed on the server which may be using ports? You can use a tool like netstat to list all the applications using TCP ports on your system, which may help identify applications that are interfering with port binding.

• We are still on asp.net core 2.0

Just to make it clear, I'm happy to help out, but ASP.NET Core 2.0 is no longer supported. Any fixes we provide will be on 2.1.x or higher builds. I'd strongly recommend upgrading to 2.1 (the current Long-Term Servicing release) or a higher version. We've improved the ASP.NET Core Module quite a bit over these releases and you may find the issue resolved in the more recent releases.

[Aardvark71]

First of all : Thanks for the feedback !!

1. About port number assignment

• I understand that IIS (ASP.NET Core Module or ANCM) assigns the port number and passes this to kestrel.
• What I don't understand is why these port numbers have always been outside the Protocol tcp Dynamic Port Range (requested with the command netsh int ipv4 show dynamicport tcp) ? Start port was 49152 and each port kestrel binds to was below that..
• Actually... when we check the Protocol tcp Dynamic Port Range now... we see that our start port is 1025 .
  ( Yes, looks like increasing MaxUserPort from 16K to 32K, unexpectedly changed the dynamic port start port and our dynamic port range is now 1025 to 33793.. )
  == > could the change of dynamic port range cause collision with the way that ANCM determines 'free' port numbers to use ??

2. How frequently do these issues occur? Do you have other applications installed on the server?
   -- > Frequency : Not often... 4 issues in the last 3 weeks. (but each production issue gets blown up, especially since when kestrel does not bind properly it does not restart causing a prolonged outage)
   -- > other applications... yes, but pretty standard what belongs on windows server 2008 r2. I should note that we are migrating from asp.net mvc to asp.net core . At this moment about 10 apps have migrated (and all are on the same server with very high load).

3. ASP.NET Core 2.0 is no longer supported
   -- > I didn't realize this... we'll start upgrading all our .net core apps to 2.1 as soon as possible.

[analogrelay]

• could the change of dynamic port range cause collision with the way that ANCM determines 'free' port numbers to use

I don't think just expanding the range would cause this problem. However, if you expanded the range because you were running out of ports, it's possible you still don't have enough ports available to you.

1. At this moment about 10 apps have migrated (and all are on the same server with very high load).

Do these applications make outbound requests frequently? That could definitely cause port exhaustion issues like this. The number of outbound TCP connections you can have is finite (limited by the dynamic port range) and if you are finding that you need to increase it, it may be time to look at migrating applications to different servers to reduce the load on this finite resource (TCP connections).

3. ASP.NET Core 2.0 is no longer supported
-- > I didn't realize this...

Oof, sorry to be the bearer of bad news there :). Please let me know if you have ideas on how we could communicate this better. The page I linked to (.NET Core Support Policy) is a live updating page that will give you the breakdown of which versions are supported and when exactly they will go end-of-life. There is also a JSON file in the dotnet/core repo that is updated regularly as support information changes. You're welcome to build a little automation to poll that file for changes.

[analogrelay]

An update: This can occur if your dynamic port range overlaps with some of the excluded port range. If IIS chooses a port that's in this range it could cause access denied errors. The easy way to check is to run the below command on your server. If the port numbers that you're seeing in these errors are within these ranges, that might be the issue:

netsh interface ipv4 show excludedportrange protocol=tcp

[analogrelay]

Closing this as we haven't heard from you and generally close issues with no response after ~7 days. Please feel free to comment if you're able to get the information we're looking for and we can reopen the issue to investigate further!