New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

redis-trib: use pipeline to speed up moving slot #2711

Closed
wants to merge 1 commit into
base: unstable
from

Conversation

Projects
None yet
3 participants
@HeartSaVioR
Contributor

HeartSaVioR commented Aug 3, 2015

Closes #2710

  • "keysinslot" now retrieves more keys (10 -> 10000) for every iteration
  • Use pipeline to run "migrate" commands much faster then current

Test environment is below,

  • Tested on my MacBook Pro Retina. (15 inch, Late 2013)
    • OSX 10.10.3
    • i7 2.3GHz, 16GB DDR3 1600 MHz, SSD 256GB
  • total 6 processes of redis-server (cluster mode)
    • 3 masters, 3 slaves
  • 1000001 keys in one slot (slot number 13784)
    • “{hashtag}0” to “{hashtag}1000000”
  • check elapsed seconds for only moving slot 13784

Test result is here,
a) verbose = true

  • origin (keysinslot 10, not using pipeline) : elapsed: 142.083674 secs
  • keysinslot 1000, using pipeline : elapsed: 59.248071 secs
  • keysinslot 10000, using pipeline : elapsed: 57.582782 secs

b) verbose = false

  • origin (keysinslot 10, not using pipeline) : elapsed: 115.745819 secs
  • keysinslot 1000, using pipeline : elapsed: 59.676552 secs
  • keysinslot 10000, using pipeline : elapsed: 57.819602 secs

Increasing fetch size of keysinslot and using pipeline seems to be 2 ~ 2.5 times faster than current.
I couldn't test it with multiple machines since I don't have three or more physical machines.

Btw, just modifying verbose option to 'false' also reduces more than 25 secs.

Use pipeline while moving slot
* "keysinslot" now retrieves more keys (10 -> 10000) for every iteration
* Use pipeline to run "migrate" commands much faster then current
@Calabor-Hoc

This comment has been minimized.

Show comment
Hide comment
@Calabor-Hoc

Calabor-Hoc Aug 7, 2015

That's great! But it's a little disappointing that pipeline can only shorten the time for half. I thought the pipeline have huge potential...

Calabor-Hoc commented Aug 7, 2015

That's great! But it's a little disappointing that pipeline can only shorten the time for half. I thought the pipeline have huge potential...

@HeartSaVioR

This comment has been minimized.

Show comment
Hide comment
@HeartSaVioR

HeartSaVioR Aug 7, 2015

Contributor

@Calabor-Hoc
I don't know about how redis-rb pipeline works, but in benchmark, Jedis pipelining is more than 15x faster than normal.
xetorthio/jedis#713 (comment)

Please note that it is just a benchmark, but I also expect more speedup than 2x.
Could you please test my changeset into your environment and compare? Thanks!

ps. Maybe we need to increase bulk size. (10000 could be small.)

Contributor

HeartSaVioR commented Aug 7, 2015

@Calabor-Hoc
I don't know about how redis-rb pipeline works, but in benchmark, Jedis pipelining is more than 15x faster than normal.
xetorthio/jedis#713 (comment)

Please note that it is just a benchmark, but I also expect more speedup than 2x.
Could you please test my changeset into your environment and compare? Thanks!

ps. Maybe we need to increase bulk size. (10000 could be small.)

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Dec 11, 2015

Owner

The reason why the speedup is not the one expected is the way MIGRATE is implemented. It requires to chat with other nodes and get a reply. So instead of implementing pipelining, if we truly want much faster migration, we should pipeline MIGRATE itself internally. I'm trying to do this right now for 3.2 and possibly will back port to 3.0. Migrate in this use case will have a new option where it is possible to specify multiple keys, so that the source node will send all the keys and will later retrieve all the replies, making the process much faster. News soon, on it.

Owner

antirez commented Dec 11, 2015

The reason why the speedup is not the one expected is the way MIGRATE is implemented. It requires to chat with other nodes and get a reply. So instead of implementing pipelining, if we truly want much faster migration, we should pipeline MIGRATE itself internally. I'm trying to do this right now for 3.2 and possibly will back port to 3.0. Migrate in this use case will have a new option where it is possible to specify multiple keys, so that the source node will send all the keys and will later retrieve all the replies, making the process much faster. News soon, on it.

@antirez antirez added the cluster label Dec 11, 2015

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Dec 11, 2015

Owner

Multi keys MIGRATE is now merged into unstable. I'm implementing it into redis-trib (trivial) to check the gain (harder, benchmarking is always hard).

Owner

antirez commented Dec 11, 2015

Multi keys MIGRATE is now merged into unstable. I'm implementing it into redis-trib (trivial) to check the gain (harder, benchmarking is always hard).

@antirez antirez closed this in f0b168e Dec 11, 2015

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Dec 11, 2015

Owner

Just tested with a larger data set, from 20 to 5 seconds. If I migrate 100 keys per iteration, 2.7 seconds from 20. 💃

Owner

antirez commented Dec 11, 2015

Just tested with a larger data set, from 20 to 5 seconds. If I migrate 100 keys per iteration, 2.7 seconds from 20. 💃

antirez added a commit that referenced this pull request Dec 13, 2015

Cluster: redis-trib: use variadic MIGRATE.
We use the new variadic/pipelined MIGRATE for faster migration.
Testing is not easy because to see the time it takes for a slot to be
migrated requires a very large data set, but even with all the overhead
of migrating multiple slots and to setup them properly, what used to
take 4 seconds (1 million keys, 200 slots migrated) is now 1.6 which is
a good improvement. However the improvement can be a lot larger if:

1. We use large datasets where a single slot has many keys.
2. By moving more than 10 keys per iteration, making this configurable,
   which is planned.

Close #2710
Close #2711

antirez added a commit that referenced this pull request Dec 13, 2015

Cluster: redis-trib: use variadic MIGRATE.
We use the new variadic/pipelined MIGRATE for faster migration.
Testing is not easy because to see the time it takes for a slot to be
migrated requires a very large data set, but even with all the overhead
of migrating multiple slots and to setup them properly, what used to
take 4 seconds (1 million keys, 200 slots migrated) is now 1.6 which is
a good improvement. However the improvement can be a lot larger if:

1. We use large datasets where a single slot has many keys.
2. By moving more than 10 keys per iteration, making this configurable,
   which is planned.

Close #2710
Close #2711
@HeartSaVioR

This comment has been minimized.

Show comment
Hide comment
@HeartSaVioR

HeartSaVioR Dec 14, 2015

Contributor

@antirez 👍

Contributor

HeartSaVioR commented Dec 14, 2015

@antirez 👍

JackieXie168 pushed a commit to JackieXie168/redis that referenced this pull request Aug 29, 2016

Cluster: redis-trib: use variadic MIGRATE.
We use the new variadic/pipelined MIGRATE for faster migration.
Testing is not easy because to see the time it takes for a slot to be
migrated requires a very large data set, but even with all the overhead
of migrating multiple slots and to setup them properly, what used to
take 4 seconds (1 million keys, 200 slots migrated) is now 1.6 which is
a good improvement. However the improvement can be a lot larger if:

1. We use large datasets where a single slot has many keys.
2. By moving more than 10 keys per iteration, making this configurable,
   which is planned.

Close #2710
Close #2711

JackieXie168 pushed a commit to JackieXie168/redis that referenced this pull request Aug 29, 2016

Cluster: redis-trib: use variadic MIGRATE.
We use the new variadic/pipelined MIGRATE for faster migration.
Testing is not easy because to see the time it takes for a slot to be
migrated requires a very large data set, but even with all the overhead
of migrating multiple slots and to setup them properly, what used to
take 4 seconds (1 million keys, 200 slots migrated) is now 1.6 which is
a good improvement. However the improvement can be a lot larger if:

1. We use large datasets where a single slot has many keys.
2. By moving more than 10 keys per iteration, making this configurable,
   which is planned.

Close #2710
Close #2711
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment