Feature request: reads/writes routing with Redis.

[pavel-shpilev]

Hi.

A couple of projects I am currently working on would greatly benefit if twemproxy supported routing for reads and writes. I imagine this could be a quite useful feature for many others as well.

Let me explain what I mean. I have a Redis cluster with one master and a bunch of slaves. Reads to writes ratio in my scenario is at least 100:1. Immediate availability of writes across all the slaves isn't a concern, and Redis replication serves the purpose just fine. I want to have a single nutcracker setup that would look something like this:

alpha:
  auto_eject_hosts: true
  read_distribution: random
  write_distribution: one_at_a_time
  hash: murmur
  listen: 127.0.0.1:6379
  preconnect: true
  redis: true
  redis_db: 1
  server_failure_limit: 3
  server_retry_timeout: 5000
  timeout: 20000
  read_servers:
  - read-001.84omyv.0001.use1.cache.amazonaws.com:6379:1
  - read-002.84omyv.0001.use1.cache.amazonaws.com:6379:1
  - read-003.84omyv.0001.use1.cache.amazonaws.com:6379:1
  write_servers:
  - write-001.84omyv.0001.use1.cache.amazonaws.com:6379:1

Currently I do routing on application level by handling two connections: one direct to write node and one to nutcracker balancing the reads. This task, however, appears to me common enough to be implemented as twemproxy feature. It would allow to decouple infrastructure configuration and business logic.

I can start working on the feature myself, but before that I would like to know the maintainers opinion whether it worth looking at. Perhaps there are some obvious obstacles that defeat the purpose, that I am missing.

Thank you.

[andybrier]

Hi,
I need the write-read split feather too. Currently i am doing this by routing request on client side. so, i thinks this feather is very useful.

[rohitjoshi]

@pavel-shpilev did you add the support for write-read split?

[therealbill]

I have a Redis cluster with one master and a bunch of slaves. Reads to writes ratio in my scenario is at least 100:1. Immediate availability of writes across all the slaves isn't a concern, and Redis replication serves the purpose just fine.

Honestly you'd be better served by using Redis Sentinel as a discovery (and failover management) mechanism rather than twemproxy here. That scenario is specifically what sentinel is designed to do. All you'd need to do in the client code depends on the client library but basically it queries sentinel for current master (for writes) or current slave list (for reads).

This decouples infrastructure changes from the client because Sentinel knows what is where. It also would be faster due to the lack of an additional network hop. Another proxy based alternative is to use HAproxy and set up a listener for writes and a listener for reads. You can even combine this with the Sentinel route to have HAproxy's configuration manages by Sentinel. That way the clients don't need to know about Sentinel either.

While doing the routing in a proxy makes sense, doing it within Twemproxy is going to be much more extensive/invasive than you might think. The problem is the underlying purpose of Twemproxy is essentially to do hashing. In the case of a single write master it all hashes to one server - which is fine. However, in the read case once you have more than one read slave you will have to short-circuit all the work Twemproxy does to figure out which read server to route to. Doing so would break the moment a second write master is introduced and the data is hashed out over the pair.

At this point the read-slave pool does need to do hashing but it won't be able to just hash across the lis of servers. You would have to declare a read-pool for every master, and the code would have to hash, then balance across the resulting pool.

Thus if I were to implement this (read/write split with read-only load balancing) in C, I'd either modify HAproxy or write a module for Nginx. But ultimately I wouldn't do it in the server. I'd do as I currently do and have a read-connection pool and a write connection pool on the client, use Sentinel to manage it, and call the appropriate connection when writing or reading. That is the most robust route, and the least amount of customization of the endpoint. It also means you get the full array of Redis commands, whereas in Twemproxy you lose some.

[aijanai]

Hello,

how about if you can't have 2 distinct connection pools in the client?
Wouldn't it be much more elegant if there was just one access point that hides read slaves and write masters?

+1 for transparent traffic split