redis-trib: use pipeline to speed up moving slot #2711

Closed
wants to merge 1 commit into
from

Projects

None yet

3 participants

@HeartSaVioR
Contributor

Closes #2710

  • "keysinslot" now retrieves more keys (10 -> 10000) for every iteration
  • Use pipeline to run "migrate" commands much faster then current

Test environment is below,

  • Tested on my MacBook Pro Retina. (15 inch, Late 2013)
    • OSX 10.10.3
    • i7 2.3GHz, 16GB DDR3 1600 MHz, SSD 256GB
  • total 6 processes of redis-server (cluster mode)
    • 3 masters, 3 slaves
  • 1000001 keys in one slot (slot number 13784)
    • “{hashtag}0” to “{hashtag}1000000”
  • check elapsed seconds for only moving slot 13784

Test result is here,
a) verbose = true

  • origin (keysinslot 10, not using pipeline) : elapsed: 142.083674 secs
  • keysinslot 1000, using pipeline : elapsed: 59.248071 secs
  • keysinslot 10000, using pipeline : elapsed: 57.582782 secs

b) verbose = false

  • origin (keysinslot 10, not using pipeline) : elapsed: 115.745819 secs
  • keysinslot 1000, using pipeline : elapsed: 59.676552 secs
  • keysinslot 10000, using pipeline : elapsed: 57.819602 secs

Increasing fetch size of keysinslot and using pipeline seems to be 2 ~ 2.5 times faster than current.
I couldn't test it with multiple machines since I don't have three or more physical machines.

Btw, just modifying verbose option to 'false' also reduces more than 25 secs.

@HeartSaVioR HeartSaVioR Use pipeline while moving slot
* "keysinslot" now retrieves more keys (10 -> 10000) for every iteration
* Use pipeline to run "migrate" commands much faster then current
69105f2
@Calabor-Hoc

That's great! But it's a little disappointing that pipeline can only shorten the time for half. I thought the pipeline have huge potential...

@HeartSaVioR
Contributor

@Calabor-Hoc
I don't know about how redis-rb pipeline works, but in benchmark, Jedis pipelining is more than 15x faster than normal.
xetorthio/jedis#713 (comment)

Please note that it is just a benchmark, but I also expect more speedup than 2x.
Could you please test my changeset into your environment and compare? Thanks!

ps. Maybe we need to increase bulk size. (10000 could be small.)

@antirez
Owner
antirez commented Dec 11, 2015

The reason why the speedup is not the one expected is the way MIGRATE is implemented. It requires to chat with other nodes and get a reply. So instead of implementing pipelining, if we truly want much faster migration, we should pipeline MIGRATE itself internally. I'm trying to do this right now for 3.2 and possibly will back port to 3.0. Migrate in this use case will have a new option where it is possible to specify multiple keys, so that the source node will send all the keys and will later retrieve all the replies, making the process much faster. News soon, on it.

@antirez antirez added the cluster label Dec 11, 2015
@antirez
Owner
antirez commented Dec 11, 2015

Multi keys MIGRATE is now merged into unstable. I'm implementing it into redis-trib (trivial) to check the gain (harder, benchmarking is always hard).

@antirez antirez added a commit that closed this pull request Dec 11, 2015
@antirez Cluster: redis-trib: use variadic MIGRATE.
We use the new variadic/pipelined MIGRATE for faster migration.
Testing is not easy because to see the time it takes for a slot to be
migrated requires a very large data set, but even with all the overhead
of migrating multiple slots and to setup them properly, what used to
take 4 seconds (1 million keys, 200 slots migrated) is now 1.6 which is
a good improvement. However the improvement can be a lot larger if:

1. We use large datasets where a single slot has many keys.
2. By moving more than 10 keys per iteration, making this configurable,
   which is planned.

Close #2710
Close #2711
f0b168e
@antirez antirez closed this in f0b168e Dec 11, 2015
@antirez
Owner
antirez commented Dec 11, 2015

Just tested with a larger data set, from 20 to 5 seconds. If I migrate 100 keys per iteration, 2.7 seconds from 20. 💃

@antirez antirez added a commit that referenced this pull request Dec 13, 2015
@antirez Cluster: redis-trib: use variadic MIGRATE.
We use the new variadic/pipelined MIGRATE for faster migration.
Testing is not easy because to see the time it takes for a slot to be
migrated requires a very large data set, but even with all the overhead
of migrating multiple slots and to setup them properly, what used to
take 4 seconds (1 million keys, 200 slots migrated) is now 1.6 which is
a good improvement. However the improvement can be a lot larger if:

1. We use large datasets where a single slot has many keys.
2. By moving more than 10 keys per iteration, making this configurable,
   which is planned.

Close #2710
Close #2711
87615a8
@antirez antirez added a commit that referenced this pull request Dec 13, 2015
@antirez Cluster: redis-trib: use variadic MIGRATE.
We use the new variadic/pipelined MIGRATE for faster migration.
Testing is not easy because to see the time it takes for a slot to be
migrated requires a very large data set, but even with all the overhead
of migrating multiple slots and to setup them properly, what used to
take 4 seconds (1 million keys, 200 slots migrated) is now 1.6 which is
a good improvement. However the improvement can be a lot larger if:

1. We use large datasets where a single slot has many keys.
2. By moving more than 10 keys per iteration, making this configurable,
   which is planned.

Close #2710
Close #2711
ea89733
@HeartSaVioR
Contributor

@antirez 👍

@JackieXie168 JackieXie168 pushed a commit to JackieXie168/redis that referenced this pull request Aug 29, 2016
@antirez Cluster: redis-trib: use variadic MIGRATE.
We use the new variadic/pipelined MIGRATE for faster migration.
Testing is not easy because to see the time it takes for a slot to be
migrated requires a very large data set, but even with all the overhead
of migrating multiple slots and to setup them properly, what used to
take 4 seconds (1 million keys, 200 slots migrated) is now 1.6 which is
a good improvement. However the improvement can be a lot larger if:

1. We use large datasets where a single slot has many keys.
2. By moving more than 10 keys per iteration, making this configurable,
   which is planned.

Close #2710
Close #2711
6c4db4e
@JackieXie168 JackieXie168 pushed a commit to JackieXie168/redis that referenced this pull request Aug 29, 2016
@antirez Cluster: redis-trib: use variadic MIGRATE.
We use the new variadic/pipelined MIGRATE for faster migration.
Testing is not easy because to see the time it takes for a slot to be
migrated requires a very large data set, but even with all the overhead
of migrating multiple slots and to setup them properly, what used to
take 4 seconds (1 million keys, 200 slots migrated) is now 1.6 which is
a good improvement. However the improvement can be a lot larger if:

1. We use large datasets where a single slot has many keys.
2. By moving more than 10 keys per iteration, making this configurable,
   which is planned.

Close #2710
Close #2711
bc08452
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment