Host thresholds exceeded: [Connections]

[forki]

Since a couple of days my function app is crashing after a while.

Error:

The function runtime is unable to start. Microsoft.Azure.WebJobs.Script.WebHost: Host thresholds exceeded: [Connections]. For more information, see https://aka.ms/functions-thresholds.
Session Id: 6db3b79aa6f24b22aa73895db2f87c04

Timestamp: 2018-11-15T08:26:33.200Z

I don't see that I have too many connections anywhere in the code. How can I troubleshoot this?

Investigative information

Please provide the following:

• Function App version (1.0 or 2.0): 2
• Function App name: ecarcache
• Function name(s) (as appropriate): MasterDataCache
• Region: Europe West

[forki]

all functions in that app look like the following:

so we have one connection to Redis and one connection to Azure Storage

[forki]

I moved some of the functions in that app to new functionapp ecarstats in order to see which half of the functions has that problem

[fabiocav]

/cc @brettsam

[dcarr42]

@forki I'm not clued up on f# async programming model but that blocking task is going to lead to thread pool issues. You end up using two threads for an operation that could be done with just one if written in a non blocking manner.

[forki]

@dallancarr believe me I'd love to remove that .wait but the runtime can't deal with Task<unit>yet. It's a reported bug and the azure-functions Team promised to look into that

[dcarr42]

I've seen examples in the docs using Async. Is this a recent bug or regression because they recommend this pattern.

[forki]

It's a regression in v2 runtime. It worked before. But AFAIK it's unrelated to this issue here.

[dcarr42]

Ignore me this must be the regression. :-)

[brettsam]

Could you give this a try with the latest Functions release 2.0.12210.0? We added a workaround that should help here.

[forki]

What would that workaround do? Am Di., 4. Dez. 2018, 16:51 hat Brett Samblanet <notifications@github.com> geschrieben:

…

Could you give this a try with the latest Functions release 2.0.12210.0 <https://github.com/Azure/azure-functions-host/releases/tag/v2.0.12210>? We added a workaround that should help here. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3781 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNGt5_S0UvuTY7c-RF90KgRxZuTJGks5u1pn3gaJpZM4YfUBo> .

[brettsam]

A little background -- .NET Core made the ServicePointManager a no-op when it comes to HttpClient usage. We used this in Functions v1 to keep a constraint on the number of connections a single HttpClient could create to a single host (using ServicePointManager.DefaultConnectionLimit). This now does nothing in .NET Core. This means that the various SDKs now need to give us a way to control this by passing an HttpClientHandler into them (or exposing some other means of controlling MaxConnectionsPerServer).

For Storage, specifically, we implemented a private-reflection workaround to change this setting back to 50, which is what it was in Functions v1: #3749. Our testing has shown this to work, but I'm trying to find all issues filed that look like this is the cause and see if we're missing any scenarios.

I don't see this error in your app for 2.0.12210, but that may not be related (you may be running with lower load, etc). Let me know if you're still seeing it, and I can take a deeper look.

[forki]

Well I changed a lot of code to do better caching myself, so that's probably the cause why it works right now... Am Di., 4. Dez. 2018, 17:12 hat Brett Samblanet <notifications@github.com> geschrieben:

…

A little background -- .NET Core made the ServicePointManager a no-op when it comes to HttpClient usage. We used this in Functions v1 to keep a constraint on the number of connections a single HttpClient could create to a single host (using ServicePointManager.DefaultConnectionLimit). This now does nothing in .NET Core. This means that the various SDKs now need to give us a way to control this by passing an HttpClientHandler into them (or exposing some other means of controlling MaxConnectionsPerServer <https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.maxconnectionsperserver?view=netcore-2.1> ). For Storage, specifically, we implemented a private-reflection workaround to change this setting back to 50, which is what it was in Functions v1: #3749 <#3749>. Our testing has shown this to work, but I'm trying to find all issues filed that look like this is the cause and see if we're missing any scenarios. I don't see this error in your app for 2.0.12210, but that may not be related (you may be running with lower load, etc). Let me know if you're still seeing it, and I can take a deeper look. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3781 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADgNCh3tX4U92_yYC0d3GsVQr_scxDiks5u1p77gaJpZM4YfUBo> .

[jeffhollan]

Closing as we both increased the connection limit (600 active, 1200 total. Up from 300 total), in addition to the fixes Brett mentioned. Ping if this should be reopened