-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NOAUTH Authentication required ERROR after ElastiCache patch #1273
Comments
Huh, that's odd. And the config (string or object) contains the auth password? This is confusing, because this is literally the first thing we send. I wonder whether elasticache works via a TCP proxy that changes the backend without severing the client connection (just: passing payloads to the new endpoint). That's the only way I can imagine of causing this... We don't currently expect NOAUTH in an established connection. |
Yes the config contains the password. That never changes. Currently I am building a solution that will essentially catch that error and then close and reopen the connection and then try again. But it is virtually impossible to test since we have to wait for an AWS patch to duplicate the error. private async Task HashSetAsync(string key, HashEntry[] hashEntries)
{
await Retry.Do(
() => _redisProvider.HashSetAsync(key, hashEntries),
() => _redisProvider.Reconnect(),
TimeSpan.FromSeconds(5));
}
public Task Reconnect()
{
if (connectionMultiplexer == null)
{
connectionMultiplexer = ConnectionMultiplexer.Connect(options);
}
else
{
connectionMultiplexer.Close();
connectionMultiplexer = ConnectionMultiplexer.Connect(options);
}
return null;
} |
Hi @bdlee420 , we are not experiencing an AWS patch but we receive NOAUTH Authentication required ERROR after one of our slave node restarts. |
+1 on this issue, we suffer from it as well, following a cluster update on AWS |
Can you please give this a try on the 2.1.0 release now on NuGet? |
We've also experienced this after performing a failover with our elasticache redis cluster, even after upgrading to StackExchange.Redis 2.1.28 The first errors we see are: Then after a couple of minutes, Redis Setup
Client
|
We also experience this NOAUTH error after Redis patch. We are using Redis AUTH + SSL. |
any update? we experience same issue
Redis Setup
Client
|
Hi @NickCraver , according to AWS this happens due to the restarted Redis engine accepting connections before its auth token is populated. Would it be possible to modify the client to reconnect once ping has auth issue, or not connect until the restarted redis is populated with auth token? Thanks! |
We can perhaps try, but the problem is: none of the cloud hosts have a
magic "now do crazy stuff in the background without severing connections"
button that we can test on demand to replicate and validate things. Which
makes this incredibly frustrating to develop.
…On Tue, 30 Jun 2020, 15:21 Tat Lee Tang, ***@***.***> wrote:
Hi @NickCraver <https://github.com/NickCraver> , according to AWS this
happens due to the restarted Redis engine accepting connections before its
auth token is populated. Would it be possible to modify the client to
reconnect once ping has auth issue, or not connect until the restarted
redis is populated with auth token? Thanks!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1273 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAEHMAPBW66UDCXWRFDH53RZHYGJANCNFSM4JNQGYFA>
.
|
Thanks to @TimLovellSmith's investigation here, this has bee fixed in the 2.2.50 release now on NuGet. It took some fresh eyes to see what was happening here in environments with some latency and races in play. Please grab 2.2.50 to eliminate this race! |
Redis Setup
Client
Occasionally there are AWS updates applied to our Redis instance for ElastiCache. When this updates applies, all of our clients that use the StackExchance.Redis library start to receive "StackExchange.Redis.RedisServerException: NOAUTH Authentication required." errors. We have to restart the clients and then everything works fine.
We have been unable to duplicate this issue outside of an ElastiCache service update.
Things we have tried to duplicate the issue:
In all these cases, the local client reconnected with no problems.
We are at a lose as to how to duplicate this issue and then how to resolve this issue. One thought I had was to create a heartbeat inside each client to try to connect and read a key, if it fails, then rebuild the connection to Redis, but we are blind since we can't duplicate it on demand. Any help would be greatly appreciated.
Our Redis provider constructor
Our method to get the Redis Database used in each call to Redis
The text was updated successfully, but these errors were encountered: