[Proposal] Read-Write splitting

[proost]

Summary

Adding Read-Write Splitting feature to cluster client to use Redis cluster more efficiently.

Problem Definition

Background

Someone who wants to use redis cluster more efficiently with rueidis, have to use two rueidis clients, that one is for primary and the other is replicas. Because rueidis doesn’t support Read-Write splitting.

Goal

• Engineers doesn’t need to wrap rueidis when using redis cluster.
• Read-Write Splitting can reduce load to primary server and optimize performance of overall redis cluster when redis cluster have high volume read operations compared to write operation.

Non-Goal

• In this proposal, I don’t add various read-write splitting strategies. Only add read to replicas & write to primary. I think various read-write splitting strategies handle additional PR.

Plan

Read-Write Splitting configuration is defined in ReadFrom struct. In this proposal, I defined only Replicas option which client use replicas for read operations.

// ReadFrom describe how client routes read operation to redis nodes.
type ReadFrom struct {
	// Replicas indicates that client will use replicas for read operations.
	Replicas bool
}

ReadFrom option is defined as ClientOption field so that user can configure Read-Write Splitting configuration.

type ClientOption struct {
	...

	// ReadFrom indicates that this client how to route read operations to redis nodes.
	ReadFrom *ReadFrom

	...

}

ReadFrom can’t work with ReplicaOnly option. because a client can send a command to primary node. so constructor should return error when ReadFrom is set & ReplicaOnly is true.

var ErrReadFromConflict = errors.New("ReadFrom and ReplicaOnly cannot be used together")

func newClusterClient(opt *ClientOption, connFn connFn) (client *clusterClient, err error) {
	...
	
	if opt.ReadFrom != nil && opt.ReplicaOnly {
		return nil, ErrReadFromConflict
	}
	
	...
}
	

Cluster client should have read connection & write connection per slot. If a read command is given, use read connection to send command to a slot. If a write command is given, use write connection to send command to a slot.

type clusterClient struct {
	...
	readConnsPerSlot [16384]conn // read connections per slot.
	writeConnsPerSlot [16384]conn // write connections per slot.
	..
}

readConnsPerSlot & writeConnsPerSlot are refreshed in a _refresh function.

Do, DoMulti , DoCache , DoMultiCache , Receive have to pick connection to send commands. All Completed type commands has IsReadOnly method. Using IsReadOnly method, pick can know whether command is read operation or not. so pick method changed like this.

func (c *clusterClient) pick(ctx context.Context, slot uint16, isReadOnlyCommand bool) (p conn, err error)

if ReadFrom is set, then choose read connection or write connection in pick method.

Pipelining method, like DoMulti , chooses each connection per command in _pickMulti method or _pickMultiCache

Currently Cacheable type doesn’t support IsReadOnly method. So define IsReadOnly method too.

[rueian]

Hi @proost,

Thank you for your proposal. It looks good generally, however, there are several things unclear:

1. The Replicas bool option seems to be less expressive. What if users sometimes want to send read commands to the primary?
2. How should this feature affect the Dedicate() and Dedicated() for acquiring a dedicated connection?

BTW, in my opinion, client-side caching should be a better choice for read-heavy applications. Have you tried that?

[proost]

Thank you for your feedback @rueian.

1. Yes, you are right. Replicas bool is less expressive. How about this?

// ReadFrom describe how client routes read operation to redis nodes.
type ReadFrom string

const (
	// ReadFromUnspecified is the default value of ClientOption.ReadFrom,
	// which means that client will use master for read operations.
	ReadFromUnspecified ReadFrom = ""
	// ReadFromReplicas indicates that client will use replicas for read operations.
	ReadFromReplicas ReadFrom = "replicas"
	// ReadFromMaster indicates that client will use master for read operations.
	ReadFromMaster ReadFrom = "master"
	// ReadFromAny indicates that client will use any node for read operations.
	ReadFromAny ReadFrom = "any"
)

var validReadFrom = map[ReadFrom]struct{}{
	ReadFromUnspecified: {},
	ReadFromReplicas:    {},
	ReadFromMaster:      {},
	ReadFromAny:         {},
}

type ClientOption struct {
         ...

   	// ReadFrom indicates that this client how to route read operations to redis nodes.
	ReadFrom ReadFrom

        ...
}

func newClusterClient(opt *ClientOption, connFn connFn) (client *clusterClient, err error) {
	...
	
	if opt.ReadFrom != nil && opt.ReplicaOnly {
		return nil, ErrReadFromConflict
	}
	
	if _, ok := validReadFrom[opt.ReadFrom]; !ok {
		return nil, ErrUnknownReadFrom
	}

	...
}

as i above said, if user want to send read operation to master, user can send read operation to primary if set ReadFromMaster or ReadFromAny, but In this proposal i only add ReadFromReplicas.

2. Dedicated or Dedicate method, only use primary node. acquire method have to call pick method with isReadOnlyCommand as false.

And i also use client-side caching. In my case, some read-operation use high cardinality & massive volume cached data, so can't use client-side cache. I wrap rueidis client to divide operation whether use client-side cache or not.

[rueian]

Would it be better if we exposed the option as a function? For example: SendToReplica: func(cmd Completed) bool { return true }. In this way, users can control how much traffic they want to send to replicas or they can always send some specific commands to the primary.

[proost]

Using function as options is nice! Anything else you can give me a review? If nothing left, i will send PR based on this discution.

[rueian]

One thing I want to add is related to the:

type clusterClient struct {
	...
	readConnsPerSlot [16384]conn // read connections per slot.
	writeConnsPerSlot [16384]conn // write connections per slot.
	..
}

I think we should allocate them only when necessary. So, this one might be better:

type clusterClient struct {
	...
	pslots [16384]conn // this is always allocated. Used when no `ReadFrom` or `ReplicaOnly=true` 
	rslots []conn // this is only allocated when `ReadFrom` is provided.
	..
}

[proost]

close: #402