RedisConnectionException in a tight loop (2.0.505)

[kierenj]

Hi! Having some good success with v2, thank you very much :)
However, a couple of times we've run into a situation where we get lots and lots of StackExchange.Redis.RedisConnectionException.

It's in a worker process loop, which is in prod in a tight while loop at present (my bad) - in a new version we use Polly to do some traffic calming.

The app is running on an Azure App Service and it's a Redis Cache on Azure too. Other apps in the same region, etc, can all connect to the server just fine. The exceptions are thrown basically until process restart.

Exception details here: https://gist.github.com/kierenj/2e4eb49b17f38596a0327b745d67dee6

Anything I can do to help diagnose?

[kierenj]

Very strange, this does seem to keep happening. A process restart lets it connect again - really seems like a client library error right now?

[kierenj]

The problem has kicked itself up a notch, and is happening more and more frequently. Should I expect to see some other exception prior to the one in the first post above?

It jumps straight in with:

No connection is available to service this operation: RPOPLPUSH Redactedredacted-Livesaff_wqitem_CTG_Production_cmd; SocketFailure on redacted.redis.cache.windows.net:6380/Subscription, origin: Error, outstanding: 0, last-read: 0s ago, last-write: 0s ago, unanswered-write: -1493811s ago, keep-alive: 60s, state: Connecting, mgr: 0 of 1 busy (10 max), last-heartbeat: never, last-mbeat: -1s ago, global: 0s ago; IOCP: (Busy=1,Free=999,Min=2,Max=1000), WORKER: (Busy=1,Free=32766,Min=2,Max=32767), Local-CPU: n/a

The only thing that varies between the messages is the number of busy/free threads: busy moves between 1, 2 and 3 (the 3 only appears once).

Maybe state: Connecting, mgr: 0 of 1 busy (10 max) means something - is that the connection manager, not connecting?

[kierenj]

It's not glamorous or particularly interesting, but is still happening repeatable for us, unfortunately. The client doesn't appear to attempt to reconnect. Anything I can do to help diagnose?

[kierenj]

Interestingly, we're also having HTTP requests fail on the server, with WinHttpException, message "Unknown error (0xffffffff)". I wonder if it's a wider issue with sockets/network.

I found: https://github.com/dotnet/corefx/issues/31037, https://github.com/dotnet/corefx/issues/25949, which talk about similar exceptions and suggest upgrading to .NET Core 2.1.

Is there any chance that that might overlap with this and explain something here? I'm not sure, since Redis is of course not HTTP..

[NickCraver]

@kierenj This sounds a lot like port exhaustion from the description, can you get a netstat dump when it happens?

[kierenj]

Ah, fantastic idea, I felt it might be something like that, out of handles, some fairly low level socket error. netstat isn't allowed on App Services but it looks like they have a feature for it, https://blogs.msdn.microsoft.com/appserviceteam/2018/01/18/exciting-new-features-in-app-service-diagnostics/ . I'll report back here when it next occurs. Sharing this link in case it helps another in the meantime.

[NickCraver]

@kierenj Did you have any lock here? Checking up on older issues.

[kierenj]

Hi, I’m not sure since we have an auto-restart thing going on now, so I’ll have to check the logs. I’ll be able to do that Monday. Thank you!

…

Sent from my iPhone

On 14 Mar 2020, at 18:46, Nick Craver ***@***.***> wrote:  @kierenj Did you have any lock here? Checking up on older issues. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.