How to handle `RedisClient::CommandError: ERR IAM Authentication service is not available` error?

[emmanuel-chime]

For some context this is what our applications use:

• redis-rb v5.4.1
• redis-client v0.26.1
• Redis v7.1.0 on ElastiCache
• IAM authentication

We occasionally receive this error in our applications when trying to establish a connection with our Redis client:
redis_client/connection_mixin.rb:80:in `call_pipelined': ERR IAM Authentication service is not available. (rediss://master.xxx.xxx.use1.cache.amazonaws.com:6379) (RedisClient::CommandError)

This then results in a NOAUTH on all subsequent requests to this client:
redis_client/connection_mixin.rb:44:in `call': NOAUTH Authentication required. (rediss://master.xxx.xxx.use1.cache.amazonaws.com:6379) (RedisClient::CommandError)

I have reached out to AWS support about the IAM Authentication service is not available error and they said that it is a bug on their end that they are working to fix, but they do not have a timeline for the fix.

Is it possible to force a re-auth in the case of the IAM Authentication error or force closing the connection when receiving a NOAUTH Authentication required error? Closing the connection will force a re-auth on the next call.

[byroot]

Is it possible to force a re-auth in the case of the ...

Right now, no, there isn't a way to do that with redis-client, other than adding your own decorator to do it manually.

It might be worth looking if I could add something to the middleware API to allow adding this sort of behaviors. An event with the connection and the raised error would do I think.

[emmanuel-chime]

It might be worth looking if I could add something to the middleware API to allow adding this sort of behaviors. An event with the connection and the raised error would do I think.

It would be great if you're able to add something like this

[emmanuel-chime]

I came up a solution registering middleware that works for my use case for now.

Something like this:

def call(command, config)
  super
rescue RedisClient::CommandError => e
  handle_noauth_error(e, config)  # handles NOAUTH Authentication required error
end

def call_pipelined(commands, config)
  super
rescue RedisClient::CommandError => e
  handle_iam_auth_error(e, config)  # handles IAM Authentication service is not available error
end

private

def handle_noauth_error(error, config)
  return raise error unless error.message.match?(/NOAUTH Authentication/i)

  # Log the error and emit metric

  # Force close the connection so next call will reconnect with fresh auth
  @client.close

  # Re-raise the error so the application knows authentication failed
  raise error
end

def handle_iam_auth_error(error, config)
  return raise error unless error.message.match?(/IAM Authentication/i)

  # Log the error and emit metric

  # Force close the connection so next call will reconnect with fresh auth
  @client.close

  # Re-raise the error so the application knows authentication failed
  raise error
end

[byroot]

So I hadn't realized, but client is already available in middlewares, so you can already implement this:

module RedisIAMMiddleware
  def call_pipelined(_commands, _config)
    super
  rescue CommandError => error
    if error.message.match?(/IAM Authentication service/)
      client.close
    end
    raise
  end

  def call(_commands, _config)
    super
  rescue CommandError => error
    if error.message.match?(/IAM Authentication service/)
      client.close
    end
    raise
  end
end

Please let me know if I missed something.

[byroot]

As mentioned, I think it can already be done with a middleware, so I'll close for now. But I can reopen if it turns out I'm wrong.