Repeated new connections with Elasticache Serverless #5589
Comments
|
How many master nodes does the cluster have? |
|
hey I'm having the same issue with version CLUSTER INFO: CLUSTER NODES: logs: |
Hi, as it is a serverless instance, it may have too many shards depending on the instance's lunched behind the scene and EPU usage and this is out of user's control. |
|
Which hostname did you define in Redisson config? |
|
@DemonDriverDan @mshahmoradi87 Can you try to define the same host twiсe? I know it sounds wired. |
I used |
|
when defining the same host twice, it ended up in some stacktrace: logs: please note I used |
|
Can you try attached version? redisson-3.26.1-SNAPSHOT.jar.zip UPDATE: specify hostname only one time in the configuration. |
|
hey I can confirm the lock issue reported by @mshahmoradi87 seems fixed, however we still see those jumps from node to node: Actually @mshahmoradi87 and I work together and our company would be ok providing temporary access to a AWS serverless redis to make debugging easier for you. Thanks a lot |
|
@mrniko I think the issue is that serverless master nodes reporting to have complete slot ranges [[0-16383]], while in cluster mode, depending on the number of shards, slots are reported. |
|
Can you check if a new ip is resolved for |
|
nslookup returns: and dig: |
|
Please execute the code below for both redis-host.cache.amazonaws.com and HOSTNAME.serverless.euw1.cache.amazonaws.com hostnames. |
|
just to clarify A bit of context of our setup and how I have been testing: For this last DNS test I ran it locally and in Fargate where I get different results: locally aws Fargate: Locally I get back 9 IPs, where in AWS I get only 3 |
@mrniko I think this is about topology scanning and slots, as a redis cluster master nodes has log likes below and serveless has the logs That is covering whole slots area for each master node. In the cluster mode, if there is a change in the shards, some slots get moved, so connection should be removed and reinitiated. Looks like in serverless mode, all master nodes have all the slots. |
|
It's because of multiple ips resolved by a single hostname in topology. |
Hi again, I have created a redis in cluster mode enabled with 3 shards, so 3 master and 3 replica IP's all resolved by a single hostname in topology, using redisson client, I don't see "master and related slaves: removed" and reconnecting. So I think the issue is not about "multiple ips resolved by a single hostname" but about all master nodes declaring to have all the slots. |
|
Redisson handles this case. Single master covering all slots works fine. |
That's good, just to make sure, this is not a single master, it is 3 masters all covering all slots. |
|
Just wanted to make sure that we are on the same page. Can you share the log output with |
|
mrniko
pushed a commit
that referenced
this issue
Jan 30, 2024
|
Fixed. Thanks for report |
|
I have tested and I see the master discovery is now stable, but slaves are still jumping from IP to IP: thank you |
mrniko
pushed a commit
that referenced
this issue
Jan 31, 2024
|
Can you try a new build with the latest commit? |
mrniko
pushed a commit
that referenced
this issue
Jan 31, 2024
|
new commit added |
mrniko
pushed a commit
that referenced
this issue
Jan 31, 2024
|
hi @mrniko it looks better now. with read from slaves, then i see connections to the slave too: so I guess it's working correctly now |
mrniko
pushed a commit
that referenced
this issue
Feb 1, 2024
|
Hi, Thanks for testing. It's better but not completely resolved. New master nodes shouldn't be added each time. Can you try build with the latest commit? Also please set |
|
hey I missed tell you that my application is instantiating 3 Redisson clients, or in other words, I call 3x I can see the lines only once per client creation. logs: |
|
Thank you! |
|
very welcome just to confirm, do you also think it is correctly working now? |
mrniko
pushed a commit
that referenced
this issue
Feb 1, 2024
|
Yes |
mrniko
pushed a commit
that referenced
this issue
Feb 1, 2024
|
@mrniko Can you please provide any ETA on when 3.26.1 will be released? Our company is using Redisson and we are looking to migrate from self hosted Redis to Serverless Elasticache Redis. |
mrniko
pushed a commit
that referenced
this issue
Feb 5, 2024
|
@mrniko As we are waiting for the release of 3.26.1 version, i built the latest 3.26.1-SNAPSHOT version and tried to test pointing to serverlesss elasticache, i still see Redisson making repeated connections from the logs, please check below sample logs. [2024-02-08 13:32:44,344][redisson-netty-1-21][o.r.c.DNSMonitor] -dd.trace_id= dd.span_id= [ INFO] Detected DNS change. Master rediss://testnet.serverless.use1.cache.amazonaws.com:6379 has changed ip from IP1 to IP2 can you please confirm is #5589 has been fixed in the 3.26.1-SNAPSHOT version or if the above is a different issue ? |
|
It's a different issue connected with DNSMonitor. |
tkatz77
pushed a commit
to tkatz77/redisson
that referenced
this issue
Feb 18, 2024
tkatz77
pushed a commit
to tkatz77/redisson
that referenced
this issue
Feb 18, 2024
tkatz77
pushed a commit
to tkatz77/redisson
that referenced
this issue
Feb 18, 2024
tkatz77
pushed a commit
to tkatz77/redisson
that referenced
this issue
Feb 18, 2024
tkatz77
pushed a commit
to tkatz77/redisson
that referenced
this issue
Feb 18, 2024
tkatz77
pushed a commit
to tkatz77/redisson
that referenced
this issue
Feb 18, 2024
tkatz77
pushed a commit
to tkatz77/redisson
that referenced
this issue
Feb 18, 2024
tkatz77
pushed a commit
to tkatz77/redisson
that referenced
this issue
Feb 18, 2024
tkatz77
pushed a commit
to tkatz77/redisson
that referenced
this issue
Feb 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have a serverless Elasticache cluster running Redis 7.1 and Java clients using Redisson 3.26.0.
Redisson config:
The client is continually reconnecting to the cluster with different IP addresses that the connection host resolves to. Running
hostcommand for the cluster returns:As the cluster is serverless I have no direct control or knowledge of the individual nodes, but I would guess it's a 3 node cluster across 3 AZs, giving the 9 different IPs.
Here's a log file with
org.redissonlogging set toDEBUG: redis-logs.txtFrom what I can tell Redisson is reconnecting each time the DNS resolution returns a different IP address.
Is there a recommended fix for this or config setting I need to enable?
The text was updated successfully, but these errors were encountered: