[Feature Request] Add a way to set dest db as rmt option

[smaznet]

Thanks for this amazing project

Please add a way to set dest db as an argument

for example some times a user wants to migrate data from server 1 on db 20 to server 2 on db 2
i think its a good option

[smaznet]

Also i tried to set db as connection url but it doesnt work

redis://127.0.0.1:6379/5
and
redis://127.0.0.1:6379/?db=5

[leonchen83]

We can't use url like redis://127.0.0.1:6379/?db=5 or redis://127.0.0.1:6379/5.
rmt now can filter souce db using rmt -s redis://127.0.0.1:6379 -m redis://127.0.0.1:6380 -d 0 1 2 -r
if we add above way to set dest db. there are several scenarios to consider.
let set dest db use -x option.

rmt -s redis://127.0.0.1:6379 -m redis://127.0.0.1:6380 -x 3 means migrate all 6379 data to 6380 db 3

1. if every 6379 db has a same key same key with string type, but value different like value1, value2, value3
   the question is after migration what is the same key's value in dest 6380 db 3

2. if that key is not string type but list type. then in this scenario what is the value in 6380

3. if rmt does't specify --replace option. and dest db 3 also has a key same key. what is the final value in dest db 3

we can implement this feature after we covered above scenarios.

we can add an option in rdt command instead of rmt command
rdt -b redis://127.0.0.1:6379 -o dump-3.rdb -x 3 means backup data from 6379 and save the file dump-3.rdb to make sure saved data's db is 3.
after that user can use rmt -s dump-3.rdb -m redis://127.0.0.1:6380 -r to migrate data to dest db 3

[smaznet]

let me explain why i need that feature

i created many projects on 1 server and i increased max db count from 16 (which is default in redis)
now i want to move some projects to other server in i dont want modify redis settings
when i tried
rmt -s ./dump.rdb -m redis://127.0.0.1:6380 -d 31 and
rmt -s ./dump.rdb -m redis://127.0.0.1:6380 -d 21

all keys moved to db 0 (because i dont changed redis.conf to increase db counts and i dont want do that)

now i want move from db 21 to db 1 and from 31 to db 2

[smaznet]

We can't use url like redis://127.0.0.1:6379/?db=5 or redis://127.0.0.1:6379/5.
rmt now can filter souce db using rmt -s redis://127.0.0.1:6379 -m redis://127.0.0.1:6380 -d 0 1 2 -r
if we add above way to set dest db. there are several scenarios to consider.
let set dest db use -x option.

rmt -s redis://127.0.0.1:6379 -m redis://127.0.0.1:6380 -x 3 means migrate all 6379 data to 6380 db 3

1. if every 6379 db has a same key same key with string type, but value different like value1, value2, value3
   the question is after migration what is the same key's value in dest 6380 db 3
2. if that key is not string type but listtype. then in this scenario what is the value in6380`
3. if rmt does't specify --replace option. and dest db 3 also has a key same key. what is the final value in dest db 3

we can implement this feature after we covered above scenarios.

we can add an option in rdt command instead of rmt command
rdt -b redis://127.0.0.1:6379 -o dump-3.rdb -x 3 means backup data from 6379 and save the file dump-3.rdb to make sure saved data's db is 3.
after that user can use rmt -s dump-3.rdb -m redis://127.0.0.1:6380 -r to migrate data to dest db 3

Yes also adding this option to rdt is good idea

[leonchen83]

hi @smaznet
I create a commit that add option -g to rdt command
now you can convert db as following step
step 1 rdt -b redis://127.0.0.1:6379 -o ./dump-2.rdb -d 31 -g 2
step 2 rmt -s ./dump-2.rdb -m redis://127.0.0.1:6380 -r

but this feature not release yet. you can build a local release base on doc by yourself.

[leonchen83]

following are examples about above scenarios we discussed before.

scenario 1

127.0.0.1:6379>select 0
127.0.0.1:6379>set string_key value1
127.0.0.1:6379>select 1
127.0.0.1:6379>set string_key value2
127.0.0.1:6379>select 3
127.0.0.1:6379>set string_key value3

$ rdt -b redis://127.0.0.1:6379 -o ./dump.rdb -g 5

# string override by db3
$ rmt -s ./dump.rdb -m redis://127.0.0.1:6380 -r
127.0.0.1:6380>select 5
127.0.0.1:6380>get string_key 
"value3"

scenario 2

127.0.0.1:6379>select 0
127.0.0.1:6379>lpush list_key value1
127.0.0.1:6379>select 1
127.0.0.1:6379>lpush list_key value2
127.0.0.1:6379>select 3
127.0.0.1:6379>lpush list_key value3

$ rdt -b redis://127.0.0.1:6379 -o ./dump.rdb -g 5

# list override by db3
$ rmt -s ./dump.rdb -m redis://127.0.0.1:6380 -r
127.0.0.1:6380>select 5
127.0.0.1:6380>lrange list_key 0 -1
1) "value3"

scenario 3

127.0.0.1:6379>select 0
127.0.0.1:6379>set string_key value1
127.0.0.1:6379>select 1
127.0.0.1:6379>set string_key value2
127.0.0.1:6379>select 3
127.0.0.1:6379>set string_key value3

127.0.0.1:6380>select 5
127.0.0.1:6380>set string_key value5

$ rdt -b redis://127.0.0.1:6379 -o ./dump.rdb -g 5
# without -r option
$ rmt -s ./dump.rdb -m redis://127.0.0.1:6380
127.0.0.1:6380>select 5
127.0.0.1:6380>get string_key 
"value5"

# with -r option
$ rmt -s ./dump.rdb -m redis://127.0.0.1:6380 -r
127.0.0.1:6380>select 5
127.0.0.1:6380>get string_key 
"value3"

[smaznet]

Yes you right
but what about mapping 🤔

for example
specify -d like this
-d 31:0 -d 20:1 -d 17:2 -d 3

which will move
all db 31 keys to db 0
all db 20 keys to db 1
all db 17 keys to db 2
all db 3 keys to db 3

[leonchen83]

@smaznet
db mapping is a good idea. but not only rdt has -d option, but also rct, rmt, rst has -d option.
we want to provide a consistent command options to users. and db mapping could broke this consistence.

I suggest write a shell to handle db mapping and migration. pseudo code may following

# backup remote redis rdb only once
rdt -b redis://${source_host}:${source_port} -o ./dump.rdb

for (mapping) {
    # filter
    rdt -b ./dump.rdb -o ./dump-{mapping.target}.rdb -d ${mapping.source} -g ${mapping.target}
    # migration
    rmt -s ./dump-${mapping.target}.rdb -m redis://${target_host}:${target_port} -r
}

[smaznet]

It's also useful for rmt and rct and rct

[leonchen83]

I'm not sure for now how other command can add db mapping feature. especially rst is hard to add this feature.
so for now we only add -g option in rdt command. If I surely clear how implement this feature in all command. then I'll consider to add db mapping in that time.