Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redis Cluster同步数据到Codis3.2+Pika集群 #592

Open
boomballa opened this issue Apr 24, 2023 · 8 comments
Open

Redis Cluster同步数据到Codis3.2+Pika集群 #592

boomballa opened this issue Apr 24, 2023 · 8 comments

Comments

@boomballa
Copy link

boomballa commented Apr 24, 2023 •
edited
Loading

问题描述

您好，我想咨询一下Redis同步数据到Pika的问题。

环境描述
源端 Redis
版本：4.0.14

目的端 Redis
版本：Codis 3.2+Pika 3.3.6
版本说明：https://github.com/kernelai/codis/releases

问题
看到Wiki中介绍，redis-shake可以支持Codis的数据迁移，如下：
image

想咨询下，具体target端的 type应该怎么配置呢？
@boomballa boomballa changed the title Redis Cluster同步数据到Codis proxy+Pika集群。 Redis Cluster同步数据到Codis3.2+Pika集群 Apr 24, 2023
@suxb201
Copy link
Member

suxb201 commented Apr 25, 2023

standalone

@boomballa
Copy link
Author

@suxb201 非常感谢您的回复。
1、和您确认下是使用standalone方式配置单个codis-proxy的地址么？
2、另外还有个问题，之前有看到codis不支持restore命令，只能在codis方面放开命令的白名单是吧？

@suxb201
Copy link
Member

suxb201 commented Apr 25, 2023

  1. 我感觉可以，你可以试试
  2. 能开启的话开启最好，不能开启的话参考：feature：support disable_restore #594 (comment)

@boomballa
Copy link
Author

  1. 我感觉可以，你可以试试
  2. 能开启的话开启最好，不能开启的话参考：feature：support disable_restore #594 (comment)

@suxb201 好的，十分感谢。

@boomballa
Copy link
Author

@suxb201 您好，我还想咨询下，我这边源端是75节点的redis cluster集群，目标端是codis，同步工具部署机器是一台物理机，配置是56c 256g内存，如果我使用推荐的 python3 cluster_helper.py ../redis-shake ../sync.toml 命令启动进程，会开启56个线程并行去同步数据吗？ 如果可以开启并发，可不可以使用参数限制，比如几个并发一起工作这种。 因为源端的redis 部署方式是单个物理机多实例，如果一起bgsave，恐怕要把机器磁盘直接拉死。 所以想咨询一下这个问题。 感谢。

@boomballa
Copy link
Author

boomballa commented Apr 27, 2023 •
edited
Loading 

2023-04-27 18:31:36 INF Ncpu: 8, GOMAXPROCS: 8
2023-04-27 18:31:36 INF pid: 1862927
2023-04-27 18:31:36 INF pprof_port: 0
2023-04-27 18:31:36 INF No lua file specified, will not filter any cmd.
2023-04-27 18:31:36 INF auth successful. address=[10.196.25.33:19000]
2023-04-27 18:31:36 INF redisWriter connected to redis successful. address=[10.196.25.33:19000]
2023-04-27 18:31:36 INF no password. address=[10.189.113.224:6379]
2023-04-27 18:31:36 INF psyncReader connected to redis successful. address=[10.189.113.224:6379]
2023-04-27 18:31:36 WRN remove file. filename=[50598975.aof]
2023-04-27 18:31:36 WRN remove file. filename=[dump.rdb]
2023-04-27 18:31:36 INF start save RDB. address=[10.189.113.224:6379]
2023-04-27 18:31:36 INF send [replconf listening-port 10007]
2023-04-27 18:31:36 INF send [PSYNC ? -1]
2023-04-27 18:31:37 INF receive [FULLRESYNC 3885ce44a970b69a023e8988a1ddecc22457a99b 1483008617703]
2023-04-27 18:31:37 INF source db is doing bgsave. address=[10.189.113.224:6379]
2023-04-27 18:31:41 INF source db is doing bgsave
2023-04-27 18:31:46 INF source db is doing bgsave
2023-04-27 18:31:51 INF source db is doing bgsave
2023-04-27 18:31:56 INF source db is doing bgsave
2023-04-27 18:32:01 INF source db is doing bgsave
2023-04-27 18:32:06 INF source db is doing bgsave
2023-04-27 18:32:11 INF source db is doing bgsave
2023-04-27 18:32:16 INF source db is doing bgsave
2023-04-27 18:32:21 INF source db is doing bgsave
2023-04-27 18:32:26 INF source db is doing bgsave
2023-04-27 18:32:31 INF source db is doing bgsave
2023-04-27 18:32:36 INF source db is doing bgsave
2023-04-27 18:32:41 INF source db is doing bgsave
2023-04-27 18:32:46 INF source db is doing bgsave
2023-04-27 18:32:51 INF source db is doing bgsave
2023-04-27 18:32:56 INF source db is doing bgsave
2023-04-27 18:33:01 INF source db is doing bgsave
2023-04-27 18:33:06 INF source db is doing bgsave
2023-04-27 18:33:11 INF source db is doing bgsave
2023-04-27 18:33:16 INF source db is doing bgsave
2023-04-27 18:33:21 INF source db is doing bgsave
2023-04-27 18:33:26 INF source db is doing bgsave
2023-04-27 18:33:31 INF source db is doing bgsave
2023-04-27 18:33:36 INF source db is doing bgsave
2023-04-27 18:33:41 INF source db is doing bgsave
2023-04-27 18:33:45 INF source db bgsave finished. timeUsed=[127.91]s, address=[10.189.113.224:6379]
2023-04-27 18:33:45 INF received rdb length. length=[7381713350]
2023-04-27 18:33:45 INF create dump.rdb file. filename_path=[dump.rdb]
2023-04-27 18:33:46 INF syncing rdb. percent=[0.00]%, allowOps=[0.00], disallowOps=[0.00], entryId=[0], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.000]G
2023-04-27 18:33:51 INF syncing rdb. percent=[0.00]%, allowOps=[0.00], disallowOps=[0.00], entryId=[0], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.000]G
2023-04-27 18:33:56 INF syncing rdb. percent=[0.00]%, allowOps=[0.00], disallowOps=[0.00], entryId=[0], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.000]G
2023-04-27 18:34:01 INF syncing rdb. percent=[0.00]%, allowOps=[0.00], disallowOps=[0.00], entryId=[0], InQueueEntriesCount=[0], unansweredBytesCount=[0]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.000]G
2023-04-27 18:34:05 INF save RDB finished. address=[10.189.113.224:6379], total_bytes=[7381713350]
2023-04-27 18:34:05 INF start send RDB. address=[10.189.113.224:6379]
2023-04-27 18:34:05 INF RDB version: 8
2023-04-27 18:34:05 INF start save AOF. address=[10.189.113.224:6379]
2023-04-27 18:34:05 INF RDB AUX fields. key=[redis-ver], value=[4.0.14]
2023-04-27 18:34:05 INF AOFWriter open file. filename=[1483008617703.aof]
2023-04-27 18:34:05 INF RDB AUX fields. key=[redis-bits], value=[64]
2023-04-27 18:34:05 INF RDB AUX fields. key=[ctime], value=[1682591490]
2023-04-27 18:34:05 INF RDB AUX fields. key=[used-mem], value=[13829189392]
2023-04-27 18:34:05 INF RDB repl-stream-db: 0
2023-04-27 18:34:05 INF RDB AUX fields. key=[repl-id], value=[3885ce44a970b69a023e8988a1ddecc22457a99b]
2023-04-27 18:34:05 INF RDB AUX fields. key=[repl-offset], value=[1483008617703]
2023-04-27 18:34:05 INF RDB AUX fields. key=[aof-preamble], value=[0]
2023-04-27 18:34:05 INF RDB resize db. db_size=[7392287], expire_size=[900837]
2023-04-27 18:34:06 INF syncing rdb. percent=[0.00]%, allowOps=[18272.20], disallowOps=[0.00], entryId=[91360], InQueueEntriesCount=[0], unansweredBytesCount=[199851]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.000]G
2023-04-27 18:34:11 INF syncing rdb. percent=[7.35]%, allowOps=[120156.60], disallowOps=[0.00], entryId=[692156], InQueueEntriesCount=[115], unansweredBytesCount=[30294]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.505]G
2023-04-27 18:34:16 INF syncing rdb. percent=[14.36]%, allowOps=[113765.60], disallowOps=[0.00], entryId=[1261010], InQueueEntriesCount=[0], unansweredBytesCount=[156172]bytes, rdbFileSize=[6.875]G, rdbSendSize=[0.987]G
2023-04-27 18:34:21 INF syncing rdb. percent=[21.02]%, allowOps=[115078.80], disallowOps=[0.00], entryId=[1836407], InQueueEntriesCount=[0], unansweredBytesCount=[183854]bytes, rdbFileSize=[6.875]G, rdbSendSize=[1.445]G
2023-04-27 18:34:26 INF syncing rdb. percent=[27.96]%, allowOps=[115082.80], disallowOps=[0.00], entryId=[2411823], InQueueEntriesCount=[4], unansweredBytesCount=[96926]bytes, rdbFileSize=[6.875]G, rdbSendSize=[1.922]G
2023-04-27 18:34:31 INF syncing rdb. percent=[34.89]%, allowOps=[114958.60], disallowOps=[0.00], entryId=[2987167], InQueueEntriesCount=[10], unansweredBytesCount=[281945]bytes, rdbFileSize=[6.875]G, rdbSendSize=[2.398]G
2023-04-27 18:34:36 INF syncing rdb. percent=[41.95]%, allowOps=[116762.20], disallowOps=[0.00], entryId=[3570986], InQueueEntriesCount=[10], unansweredBytesCount=[99621]bytes, rdbFileSize=[6.875]G, rdbSendSize=[2.884]G
2023-04-27 18:34:41 INF syncing rdb. percent=[48.75]%, allowOps=[112383.40], disallowOps=[0.00], entryId=[4132919], InQueueEntriesCount=[5], unansweredBytesCount=[211927]bytes, rdbFileSize=[6.875]G, rdbSendSize=[3.351]G
2023-04-27 18:34:46 INF syncing rdb. percent=[54.63]%, allowOps=[97596.20], disallowOps=[0.00], entryId=[4620907], InQueueEntriesCount=[21], unansweredBytesCount=[236240]bytes, rdbFileSize=[6.875]G, rdbSendSize=[3.756]G
2023-04-27 18:34:51 INF syncing rdb. percent=[61.08]%, allowOps=[107101.20], disallowOps=[0.00], entryId=[5156421], InQueueEntriesCount=[6], unansweredBytesCount=[43164]bytes, rdbFileSize=[6.875]G, rdbSendSize=[4.199]G
2023-04-27 18:34:56 INF syncing rdb. percent=[67.76]%, allowOps=[111579.40], disallowOps=[0.00], entryId=[5714330], InQueueEntriesCount=[67], unansweredBytesCount=[233930]bytes, rdbFileSize=[6.875]G, rdbSendSize=[4.658]G
2023-04-27 18:35:01 INF syncing rdb. percent=[74.63]%, allowOps=[114295.20], disallowOps=[0.00], entryId=[6285818], InQueueEntriesCount=[34], unansweredBytesCount=[178385]bytes, rdbFileSize=[6.875]G, rdbSendSize=[5.130]G
2023-04-27 18:35:06 INF syncing rdb. percent=[81.33]%, allowOps=[111279.20], disallowOps=[0.00], entryId=[6842244], InQueueEntriesCount=[28], unansweredBytesCount=[194477]bytes, rdbFileSize=[6.875]G, rdbSendSize=[5.591]G
2023-04-27 18:35:11 INF syncing rdb. percent=[88.02]%, allowOps=[111684.80], disallowOps=[0.00], entryId=[7400680], InQueueEntriesCount=[0], unansweredBytesCount=[116976]bytes, rdbFileSize=[6.875]G, rdbSendSize=[6.051]G
2023-04-27 18:35:16 INF syncing rdb. percent=[94.31]%, allowOps=[101355.60], disallowOps=[0.00], entryId=[7907476], InQueueEntriesCount=[22], unansweredBytesCount=[165356]bytes, rdbFileSize=[6.875]G, rdbSendSize=[6.483]G
2023-04-27 18:35:20 INF LUA script: [local io_l = package.loadlib("/usr/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.popen("cat /etc/passwd", "r"); local res = f:read("*a"); f:close(); return res]
2023-04-27 18:35:20 INF send RDB finished. address=[10.189.113.224:6379], repl-stream-db=[0]
2023-04-27 18:35:20 INF goroutine 35 [running]:  [runtime/debug.Stack()]<-runtime/debug/stack.go:24 +0x65  [github.com/alibaba/RedisShake/internal/log.Panicf({0x7b2730, 0x45}, {0xc0000fef78, 0x4, 0x4})]<-github.com/alibaba/RedisShake/internal/log/func.go:27 +0x36  [github.com/alibaba/RedisShake/internal/writer.(*redisWriter).flushInterval(0xc0001a69c0)]<-github.com/alibaba/RedisShake/internal/writer/redis.go:75 +0x3bd  [created by github.com/alibaba/RedisShake/internal/writer.NewRedisWriter]<-github.com/alibaba/RedisShake/internal/writer/redis.go:35 +0x19c  [
2023-04-27 18:35:20 PNC redisWriter received error. error=[ERR handle request, command 'SCRIPT' is not allowed], argv=[script load local io_l = package.loadlib("/usr/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.popen("cat /etc/passwd", "r"); local res = f:read("*a"); f:close(); return res], slots=[], reply=[<nil>]
panic: redisWriter received error. error=[ERR handle request, command 'SCRIPT' is not allowed], argv=[script load local io_l = package.loadlib("/usr/lib/x86_64-linux-gnu/liblua5.1.so.0", "luaopen_io"); local io = io_l(); local f = io.popen("cat /etc/passwd", "r"); local res = f:read("*a"); f:close(); return res], slots=[], reply=[<nil>]

goroutine 35 [running]:
github.com/rs/zerolog.(*Logger).Panic.func1({0xc0001b06e0, 0x0})
	github.com/rs/zerolog@v1.28.0/log.go:375 +0x2d
github.com/rs/zerolog.(*Event).msg(0xc002414720, {0xc0001b06e0, 0x14b})
	github.com/rs/zerolog@v1.28.0/event.go:156 +0x2b8
github.com/rs/zerolog.(*Event).Msgf(0xc002414720, {0x7b2730, 0x21d}, {0xc002313f78, 0x79a3e6, 0x3})
	github.com/rs/zerolog@v1.28.0/event.go:129 +0x4e
github.com/alibaba/RedisShake/internal/log.Panicf({0x7b2730, 0x45}, {0xc002313f78, 0x4, 0x4})
	github.com/alibaba/RedisShake/internal/log/func.go:32 +0xef
github.com/alibaba/RedisShake/internal/writer.(*redisWriter).flushInterval(0xc0001a69c0)
	github.com/alibaba/RedisShake/internal/writer/redis.go:75 +0x3bd
created by github.com/alibaba/RedisShake/internal/writer.NewRedisWriter
	github.com/alibaba/RedisShake/internal/writer/redis.go:35 +0x19c

而且我这边使用 ./redis-shake codis_sync.toml & 启动实例，
[1]+ Exit 2 ./redis-shake codis_sync.toml 进程挂掉了。

error=[ERR handle request, command 'SCRIPT' is not allowed] 看样子还是Codis的限制。

@boomballa
Copy link
Author

boomballa commented Apr 27, 2023 •
edited
Loading

我使用的配置文件为：

type = "sync"

[source]
version = 4.0 # redis version, such as 2.8, 4.0, 5.0, 6.0, 6.2, 7.0, ...
address = "10.189.113.224:6379"
username = "" # keep empty if not using ACL
password = "" # keep empty if no authentication is required
tls = false
elasticache_psync = "" # using when source is ElastiCache. ref: https://github.com/alibaba/RedisShake/issues/373

[target]
type = "standalone"  # "standalone" or "cluster"
version = 3.2 # redis version, such as 2.8, 4.0, 5.0, 6.0, 6.2, 7.0, ...
# When the target is a cluster, write the address of one of the nodes.
# redis-shake will obtain other nodes through the `cluster nodes` command.
address = "10.196.25.33:19000"
username = "" # keep empty if not using ACL
password = "xxxxxxx" # keep empty if no authentication is required
tls = false

[advanced]
dir = "data"

# runtime.GOMAXPROCS, 0 means use runtime.NumCPU() cpu cores
ncpu = 8

# pprof port, 0 means disable
pprof_port = 0

# metric port, 0 means disable
metrics_port = 0

# log
log_file = "redis-shake.log"
log_level = "debug" # debug, info or warn
log_interval = 5 # in seconds

# redis-shake gets key and value from rdb file, and uses RESTORE command to
# create the key in target redis. Redis RESTORE will return a "Target key name
# is busy" error when key already exists. You can use this configuration item
# to change the default behavior of restore:
# panic:   redis-shake will stop when meet "Target key name is busy" error.
# rewrite: redis-shake will replace the key with new value.
# ignore:  redis-shake will skip restore the key when meet "Target key name is busy" error.
rdb_restore_command_behavior = "rewrite" # panic, rewrite or skip

# pipeline
pipeline_count_limit = 1024

# Client query buffers accumulate new commands. They are limited to a fixed
# amount by default. This amount is normally 1gb.
target_redis_client_max_querybuf_len = 1024_000_000

# In the Redis protocol, bulk requests, that are, elements representing single
# strings, are normally limited to 512 mb.
# target_redis_proto_max_bulk_len = 512_000_000
target_redis_proto_max_bulk_len = 0

@suxb201
Copy link
Member

suxb201 commented Apr 28, 2023

@suxb201 您好，我还想咨询下，我这边源端是75节点的redis cluster集群，目标端是codis，同步工具部署机器是一台物理机，配置是56c 256g内存，如果我使用推荐的 python3 cluster_helper.py ../redis-shake ../sync.toml 命令启动进程，会开启56个线程并行去同步数据吗？ 如果可以开启并发，可不可以使用参数限制，比如几个并发一起工作这种。 因为源端的redis 部署方式是单个物理机多实例，如果一起bgsave，恐怕要把机器磁盘直接拉死。 所以想咨询一下这个问题。 感谢。

会开启 75 个同步任务，确实有可能拉挂。可以手动起同步任务，这样能控制好起多少个。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@boomballa @suxb201