Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

重构数据同步迁移系统,剥离rsync #1393

Closed
4 tasks
wanghenshui opened this issue Apr 22, 2023 · 3 comments
Closed
4 tasks

重构数据同步迁移系统,剥离rsync #1393

wanghenshui opened this issue Apr 22, 2023 · 3 comments
Assignees

Comments

@wanghenshui
Copy link
Collaborator

当前pika的网络模型比较粗糙。由于文件组织和redis的rdb那种简单文件不同,同步也比较复杂,

当前的能力都是基于rsync同步目录,rsync不适合嵌入一个进程中来做长期任务,也不适合多副本同步

pika当前需要水平扩展能力,不局限于一个主从,拓展为三副本模式

如何在当前的基础上建立一套备份/同步系统,来支持主从 从从 之间的同步(多副本同步)?

需要的能力

  • POST命令或者定制一个redis SYNCTO命令触发,如果支持http 请求的接收,可以更方便接入基于web的管控层的开发,让管控层依赖redis命令有点不合适,但最好也支持redis命令,运维人员各取所需
  • 基于流的文件传输能力,以及文件校验 可参考brpc文件传输
  • 对账能力,副本互相验证数据一致性,这个是用key生成个hash tree互相校验,不是get key,学名叫啥记不太清了

不紧急的

  • pika server支持http server,可以把metric信息性能信息等等通过http导入,以及后续admin相关能力补充做铺垫
@Joeyzsy
Copy link

Joeyzsy commented May 6, 2023

Wanna Join

@luky116
Copy link
Collaborator

luky116 commented May 27, 2023

@Tangruilin

Tangruilin added a commit to Tangruilin/pika that referenced this issue Jun 16, 2023
Signed-off-by: reilly <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/pika that referenced this issue Jun 17, 2023
Signed-off-by: reilly <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/pika that referenced this issue Jun 26, 2023
Signed-off-by: reilly <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/pika that referenced this issue Jun 26, 2023
Signed-off-by: reilly <tang.ruilin@foxmail.com>
Tangruilin added a commit to Tangruilin/pika that referenced this issue Jun 30, 2023
Signed-off-by: reilly <tang.ruilin@foxmail.com>
@yaoyinnan
Copy link
Contributor

#1805

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants