Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

请求支持索引数据从DB中同步 #45

Open
ljynfo opened this issue Dec 16, 2022 · 0 comments
Open

请求支持索引数据从DB中同步 #45

ljynfo opened this issue Dec 16, 2022 · 0 comments

Comments

@ljynfo
Copy link

ljynfo commented Dec 16, 2022

请求支持场景:
业务数据使用MySQL存储,ES只做搜索,这样就需要支持MySQL数据同步到ES;
MySQL设计表结构一般遵循三范式、ES索引结构却支持多层级JSON,故一般SQL主表一条记录对应ES一个文档、SQL主表的关联表对应ES文档下的一个JSON,如:

create table student
(
    id   bigint unsigned auto_increment
        primary key,
    name varchar(32) not null comment '名称'
);
create table course
(
    id   bigint unsigned auto_increment
        primary key,
    name varchar(32) not null comment '名称'
);
create table student_course
(
    id         bigint unsigned auto_increment
        primary key,
    student_id bigint not null comment '学生id',
    course_id  bigint not null comment '课程id'
);

对应ES的数据应该是:

{
  "_id": 1,
  "name": "学生名称",
  "course": [
    {
      "course_id": 1,
      "name": "课程名称"
    }
  ]
}

请求支持此种数据同步,业界方案一般是使用读binlog、表和文档字段映射关系配置来实现,但自己实现起来会遇到各种问题,如:拓展性不够、容灾性不高、数据会丢失、同步会卡住、同步延迟大、SQL改动后会涉及到重建索引
一直没有一个好用的、开源的、易入手的框架工具来解决这个事情,看了作者大大的EE索引维护方案,希望作者大大能够考虑这种场景,出一个解决方案

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant