unexpected used connections metric reported when using StreamingCredentialsProvider

[yolu16]

Issue tracker is used for reporting bugs and discussing new features. Please use
stackoverflow for supporting issues.

We're migrating from CredentialsProviderContext to StreamingCredentialsProvider for EntraID authentication using go-redis-entraid. A few hours after deploying this change, we observed a large increase in db.client.connections.usage in the "used" state, while the "idle" state remained stable. This behavior did not occur with CredentialsProviderContext, so we're unsure if it's related to the new provider.

Expected Behavior

We do not expect the db.client.connections.usage in the "used" state to have a sudden large increase.

Current Behavior

The number of db.client.connections.usage in "used" state is stable with value X and suddenly increases to 4,294,967,295+X. Not sure if it's a coincidence, 4,294,967,295 is the max value of uint32.
The db.client.connections.usage "idle" state remains stable during the sudden increase.
The metrics are set up with redisotel.InstrumentMetrics(client)

Graph of used and idle connections

Used connections data (4,294,967,295 + 1)

Idle connections data

Steps to Reproduce

Unfortunately, we did not see anything special that could have caused this state, it just seemed to occur spontaneously a few hours after deploying our service. We only create one RedisClient in our service code and use it for all our redis operations.
Here is a sample of our redis client configs:

type RedisConfig struct {
	Host          string
	Port          int
}

func NewRedisClient(redisConfig RedisConfig) (redis.UniversalClient, error) {
	address := fmt.Sprintf("%s:%d", redisConfig.Host, redisConfig.Port)

	provider, err := entraid.NewDefaultAzureCredentialsProvider(entraid.DefaultAzureCredentialsProviderOptions{
		DefaultAzureIdentityProviderOptions: identity.DefaultAzureIdentityProviderOptions{
			Scopes: []string{"https://redis.azure.com/.default"},
		},
	})
	if err != nil {
		return nil, err
	}

	opt := &redis.ClusterOptions{
		Addrs:                        []string{address},
		StreamingCredentialsProvider: provider,
		TLSConfig: &tls.Config{
			ServerName: redisConfig.Host,
			MinVersion: tls.VersionTLS12,
		},
	}

	client := redis.NewClusterClient(opt)

	if err := redisotel.InstrumentTracing(client); err != nil {
		return nil, err
	}

	if err := redisotel.InstrumentMetrics(client); err != nil {
		return nil, err
	}

	return client, nil
}

Context (Environment)

go 1.25.0
go-redis/v9 v9.14.0
go-redis/extra/redisotel/v9 v9.14.0
go-redis-entraid v1.0.6

Our service is deployed on AKS with three active instances running concurrently. We are using Azure Managed Redis and Managed identities for auth.

[ndyakov]

Hello @yolu16 , thank you for reporting this. I will investigate this. Maybe there is something related to redisotel and reporting of the pool stats. We did some changes on the Pool within the latest version of go-redis, can you verify this is not present with the CredentialsProviderContext and go-redis/v9 v9.14.9? Maybe it is not at all related to the credentials provider, but to the changes we did on the pool stats.

[yolu16]

thank you @ndyakov! This service deployed in our staging environment is still using CredentialsProviderContext with go-redis/v9 v9.14.0. In our metrics, everything looks pretty consistent, for example the last 7 days =>

For reference, here is our CredentialsProviderContext configuration:

func NewRedisClient(redisConfig RedisConfig) (redis.UniversalClient, error) {
	[...]
	opt := &redis.ClusterOptions{
		Addrs:                      []string{address},
		CredentialsProviderContext: redisCredentialProvider(redisConfig.AzureCredential),
		TLSConfig: &tls.Config{
			ServerName: redisConfig.Host,
			MinVersion: tls.VersionTLS12,
		},
	}
	[...]
}

func redisCredentialProvider(credential azcore.TokenCredential) func(context.Context) (string, string, error) {
	return func(ctx context.Context) (string, string, error) {
		// get an access token for Azure Cache for Redis
		tk, err := credential.GetToken(ctx, policy.TokenRequestOptions{
			// Azure Cache for Redis uses the same scope in all clouds
			Scopes: []string{"https://redis.azure.com/.default"},
		})
		if err != nil {
			return "", "", err
		}
		// the token is a JWT; get the principal's object ID from its payload
		parts := strings.Split(tk.Token, ".")
		if len(parts) != 3 {
			return "", "", errors.New(...)
		}

		payload, err := base64.RawURLEncoding.DecodeString(parts[1])
		if err != nil {
			return "", "", err
		}

		claims := struct {
			OID string `json:"oid"`
		}{}

		err = json.Unmarshal(payload, &claims)
		if err != nil {
			return "", "", err
		}

		if claims.OID == "" {
			return "", "", errors.New(...)
		}

		return claims.OID, tk.Token, nil
	}
}

[ndyakov]

@yolu16 looking at redis otel -

go-redis/extra/redisotel/metrics.go

Line 209 in 19fdc48

o.ObserveInt64(usage, int64(stats.TotalConns-stats.IdleConns), metric.WithAttributeSet(usedAttrs))

this probably overflows and that is why you are getting max uint32. I have to investigate why the total is not incremented, but the idle has been incremented.

Update:

I think I found the issue and it is actually something I stumbled upon in a feature I worked on for the next version that is beta and is released. @yolu16 would you mind testing (in non-critical env) with https://github.com/redis/go-redis/releases/tag/v9.16.0-beta.1 and StreamingCredentialsProvider?

[yolu16]

thank you @ndyakov it seems to fix the issue! We deployed our service in our dev environment yesterday around 10am with the suggested configs, i.e.

• go-redis/extra/redisotel/v9 v9.16.0-beta.1
• go-redis/v9 v9.16.0-beta.1
• StreamingCredentialsProvider

We've been monitoring since, and everything has been stable =>

[ndyakov]

@yolu16 glad to hear that, thank you for verifying. I don't advice to use this beta version in production at the moment. I will cherry-pick (or actually reimplement) the fix in a patch version for the current stable release early next week.

On the other hand, if you would like to help us stabilize this beta version, it would be great if you can use it on staging and get back to us with your feedback. The pool management should be a bit better with the beta version as well.