Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Weird issue with connection dropping with cluster pub/sub #1772

Open
argjent-dua opened this issue Jun 16, 2023 · 0 comments
Open

Weird issue with connection dropping with cluster pub/sub #1772

argjent-dua opened this issue Jun 16, 2023 · 0 comments

Comments

@argjent-dua
Copy link

argjent-dua commented Jun 16, 2023

Hello,

I'm using redis pub/sub to receive keyspace notifications for the expired keys. The solution works fine with the redis instance that doesn't have cluster mode enabled, but when switching to a cluster deployment it sometimes drops the connection and the subscriber won't reconnect.

I'm using Elasticache 7.0.7 Cluster Deployment with 2 shards & 2 nodes per shard.
The output of debug logging for the subscriber is the following

2023-06-15T19:45:22.150Z ioredis:cluster:subscriber selected a subscriber xxx.xx.xx.17:6379
2023-06-15T19:45:22.151Z ioredis:cluster:subscriber started
2023-06-15T19:45:22.255Z ioredis:cluster:subscriber started
2023-06-15T19:45:22.255Z ioredis:cluster:subscriber selected a subscriber xxx.xx.xx.152:6379
2023-06-15T19:58:25.747Z ioredis:cluster:subscriber subscriber has disconnected, but ClusterSubscriber is not started, so not reconnecting.
2023-06-15T19:58:25.744Z ioredis:cluster:subscriber subscriber has disconnected, but ClusterSubscriber is not started, so not reconnecting.
2023-06-15T19:58:25.743Z ioredis:cluster:subscriber stopped
2023-06-15T19:58:25.730Z ioredis:cluster:subscriber stopped

The pub/sub subscriber is in a fargate task inside the same VPC as the redis instance.

The subscriber connects the following way:

new IORedis.Cluster(
        [
          {
            host,
            port,
          },
        ],
        {
          enableOfflineQueue: false,
          lazyConnect: true,
          redisOptions: {
                connectTimeout: 1000,
                enableOfflineQueue: false,
                autoResubscribe: true,
                maxRetriesPerRequest: 1,
                lazyConnect: true,
                retryStrategy(times: number) {
                  logger.warn(`Retrying redis connection: Attempt ${times}`);
                  return Math.min(times * 500, 20 * 1000);
                },
              },
        },
      );
 pubSubClient.psubscribe('__keyspace@*__:*', () => {});
 pubSubClient.on('pmessage', () => {});

I've also checked the maintenance of elasticache if it caused this, but I didn't find anything that happened. The instance was working normally as always.
If this issue can't be solved how can I retry the whole subscription manually if this ever occurs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant