Using StackExchange.Redis inside an AWS lambda

[ReturnOfSpotScarta]

I have an AWS ElasticCache Redis instance which I connect to using StackExchange.Redis. Everything mostly works fine, however, after about 7 minutes errors start randomly happening and then it works fine again. The errors are something like:

No connection is available to service this operation: DEL KEY_NAME_REMOVED; SocketClosed (WriteSocketError/ConnectionReset, last-recv: 0) on instance_id_removed.cache.amazonaws.com:6379/Interactive, Flushing/MarkProcessed, last: DEL, origin: ReadFromPipe, outstanding: 6, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, in-pipe: 0, out-pipe: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.0.601.3402; IOCP: (Busy=0,Free=1000,Min=50,Max=1000), WORKER: (Busy=0,Free=32767,Min=50,Max=32767), Local-CPU: n/a.

I've noticed the following pattern whenever the problem occurs:

• Lots of load from other services calling the lambda, which makes requests to redis
• The amount of load drops off
• Around 7 minutes later the load to the lambda increases again
• The first few requests throw the error above.

I think what is happening is that there are a number of lambda instances running. After the load decreases these lambdas are put into a frozen state. After the load increases they are unfrozen and perform the work which involves making requests to redis. However, during this time the TCP connections must've closed, dropped or got into some state that Redis.StackExchange was not expecting.

I don't believe this is an issue with StackExchange.Redis, but there must be a way to make this work reliably. Are there any suggestions for which configuration settings should be used to make this work? Is it possible to make StackExchange.Redis audit/check its connections after the lambda gets woken up?

I've tried setting the keep-alive to 3 minutes and 15 minutes and it has the same outcome.

The ConnectionMultiplexer is created once per lambda (i.e it's stored staticly), so that each call of lambda's Run method can reuse connections. I believe this is right, because the alternative would be to create the ConnectionMultiplexer each time, which means each run would have to open a new connection every time (and then surely exhaust all the connections in the underlying lambda container/ec2 instance).

Thanks for any help!

EDIT: I'm using StackExchange.Redis version 2.0.601 from nuget

[ReturnOfSpotScarta]

After asking AWS support for help with this, they seem to think that the library should be able to deal with the connections dropping and get back into a usable state after the lambda is frozen. Any thoughts?

[NickCraver]

Please see #1120 and more importantly #1374 - we have a fix for this pending which should resole the reconnect issue you're experiencing. Closing this out just to consolidate discussions all together.