Skip to content
Branch: master
Go to file
Code

Latest commit

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

redis-transfer

Build Status

Concurrent bulk transfer of keys from one redis server to another.

What it does?

Parallelizes the transfer of redis keys from a source to a target redis server. You can specify what keys to transfer via regex or from an input file which contains a list of keys. Redis-transfer will then spawn N go routines (threads) which pull (dump) keys from the source redis server & immediately push (restore) those keys to the target redis. As a bonus, gives you a pretty progress bar.

Usage

 $ ./redis-transfer
usage: redis-transfer <from_redis> <to_redis> <regex_or_input_file> <concurrent_threads>

 * There are two redis connection formats to choose from:
   - host:port:[db[:password]]
   - redis://user:password@host:port?db=number

 * regex_or_input_file:
   - To transfer only those keys that match a regex pattern:
     some_key_prefix*
     *some_key_suffix
     prefix*something*suffix
     etc..

   - To transfer keys from an input file, simply list keys in a file, separated by newlines
     key1
     keyN

 * concurrent_threads:
   - This should be a number between 1 and (max_cpu's*10)
     You can play around with this to find the optimal setting. I generally use 50 on my 8 core box.

 * examples:
   redis-transfer localhost:6379 remotehost:6379:1:password "migrate:*" 50
   redis-transfer redis://localhost:6379 redis://user:password@remotehost?db=1 "migrate:*" 50

Example benchmark output

Note: Experiment performed in ~2013. Go has made significant improvements since then. Expect better results.

The example progress bar output below illustrates the difference we see specific to the concurrency level. Transferring 5 million keys using 40 concurrent threads between two servers (across the internet) took approximately 10 minutes in this experiment. Not surprisingly, using only one thread took approximately 40 minutes.

40 Threads

80973 / 5000000 [>--------------------------------------------------------------] 1.62 % 8066/s 10m9s

10 Threads

108367 / 5000000 [==>-----------------------------------------------------------] 2.17 % 6457/s 12m37

1 Thread

4124 / 5000000 [>---------------------------------------------------------------] 0.08 % 2059/s 40m26

Todo

  • Add optional params via flags.
  • With flags, add the ability to pass '--replace' which would replace existing keys.
  • Deeper refactor.

Contributing

Pull requests welcome!

See CONTRIBUTORS.

About

Concurrent bulk transfer of keys from one redis server to another. Written in Go / GoLang.

Resources

License

You can’t perform that action at this time.