Redis cluster refresh of large clusters keeps I/O threads busy

[be-hase]

Bug Report

Current Behavior

For large redis cluster, using enablePeriodicRefresh causes serious performance problems.

The processing time required for DefaultClusterTopologyRefresh.getNodeSpecificViews increases in proportion to the cluster size.
Unfortunately, this is executed in the NIO event loop thread.

See the repository README below for details.
https://github.com/be-hase/lettuce-with-large-cluster

Input Code

I have prepared the code that can reproduce this problem.
https://github.com/be-hase/lettuce-with-large-cluster

Expected behavior/code

Even if we use enablePeriodicRefresh with large redis cluster, the performance will not deteriorate.

Environment

• Lettuce version(s): 6.1.6.RELEASE
• Redis version: 6.2.0

Possible Solution

Idea 1

Stop running DefaultClusterTopologyRefresh.getNodeSpecificViews on the NIO event loop.

I won't go into details, but if I customized DefaultClusterTopologyRefresh and ran it on another thread, the
performance improved.

My team is considering adopting this method as a workaround.

Idea 2

Looking at the previous framegraph, it seems that the overhead is large in the processing using BitSet.
Why not change to a more primitive method like boolean[] ?

However, I'm not sure if the performance will improve.

Additional context

NONE

[be-hase]

In my actual environment, I use both periodic refresh and adaptive refresh.

Should I use only adaptive refresh in a large cluster?
Are there any disadvantages when using only adaptive refresh ?

[be-hase]

Ah, it seems good to use dynamicRefreshSources(false) for a large cluster.

[mp911de]

BitSet has a much smaller memory footprint than boolean[] and it is easier in usage. Generally, if you increase the periodic refresh period and rely more on adaptive refresh, then you will likely experience less refreshes.

It would be possible to use a different scheduler in DefaultClusterTopologyRefresh.loadViews(…) as we're using future chaining. What is the underlying cause for the ticket, how did you find out about this issue? Was it motivated by an incident or something similar?

[be-hase]

I had this problem when I upgraded from lettuce5 to lettuce6.
We are using a large redis cluster(96 nodes).

After investigating the cause, it was determined that it was due to DefaultClusterTopologyRefresh.getNodeSpecificViews running on the NIO event loop thread.
Asynchronous Cluster Topology Refresh has appeared from lettuce6, so I think it is the influence.

[mp911de]

Feel free to submit a pull request to use ClientResources.eventExecutorGroup() in the future chaining in DefaultClusterTopologyRefresh.getNodeSpecificViews

[be-hase]

OK. I'll try to submit PR later.

[be-hase]

@mp911de
Submited PR. Please check 🙏
#2048