Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Database governance-Sharding CRD for sharding l | 数据库治理-分库分表 #26

Closed
mlycore opened this issue Aug 15, 2022 · 0 comments · Fixed by #40
Closed
Labels
area/database-governance Category issues or PRs related to database governance kind/spec-RFC Issues or PRs for OpenSergo spec RFC

Comments

@mlycore
Copy link
Contributor

mlycore commented Aug 15, 2022

domain: database

Umbrella issue: #15

Sharding defines a set of rules help declare how to implementing sharding.

(English version TBD...)


数据分片是基于数据属性一种扩展策略,对数据属性进行计算后将请求发往特定的数据后端,目前分为分片键分片和自动分片。其中分片键分片中需要指明需要分片的表、列、以及进行分片的算法。
配置数据分片首先需要确定是分片键分片还是自动分片,规则配置如下:
分片键分片的规则配置包括:

  • tables: # 数据分片规则配置
    • < logicalTableName>: # 逻辑表名称
      • actualDataNodes: # 由数据源名 + 表名组成(参考 Inline 语法规则)
      • databaseStrategy: # 分库策略,缺省表示使用默认分库策略,以下的切分策略只能选其一
      • tableStrategy: # 分表策略,同分库策略
      • keyGenerateStrategy: # 分布式序列策略
        自动分片的规则配置包括:
  • autoTables: # 自动分片表规则配置
    • t_order_auto: # 逻辑表名称
    • actualDataSources: # 数据源名称
      • shardingStrategy: # 切分策略

两种分片规则都需要根据场景配置绑定表和广播表策略,规则配置如下:

绑定表策略的规则配置包括:

  • bindingTables: # 绑定表规则列表
    • <logic_table_name_1, logic_table_name_2, ...>
    • <logic_table_name_1, logic_table_name_2, ...>

广播表策略的规则配置包括:

  • broadcastTables: # 广播表规则列表
    • <table_name>
    • <table_name>

除此之外,对于用户未指定的情况需要应用默认策略,规则配置包括:

  • defaultDatabaseStrategy: # 默认数据库分片策略
  • defaultTableStrategy: # 默认表分片策略
  • defaultKeyGenerateStrategy: # 默认的分布式序列策略
  • defaultShardingColumn: # 默认分片列名称

需要用到的策略包括切分策略、分布式序列策略,规则配置如下:

databaseStrategy 和 tableStrategy 都需要配置切分策略:

  • standard: # 用于单分片键的标准分片场景
    • shardingColumn: # 分片列名称
    • shardingAlgorithmName: # 自动分片算法名称
  • complex: # 用于多分片键的复合分片场景
    • shardingColumn: # 分片列名称,多个列以逗号分隔
    • shardingAlgorithm: # 分片算法名称
  • hint: # Hint 分片策略
    • shardingColumn: 分片列名
    • shardingAlgorithm: 分片算法
  • none: # 不填即为 none,表示不分片

分布式序列策略:

  • keyGenerateStrategy: # 分布式序列策略
    • column: # 自增列名称,缺省表示不使用自增主键生成器
    • keyGeneratorName: # 分布式序列算法名称

分片算法和分布式序列算法配置如下:

分片算法配置:

  • shardingAlgorithms:
    • : # 分片算法名称
      type: # 分片算法类型
      props: # 分片算法属性配置

分布式序列算法配置:

  • keyGenerators:
    • : # 分布式序列算法名称
      type: # 分布式序列算法类型
      props: # 分布式序列算法属性配置

分片键分片 YAML 示例:

apiVersion: database.opensergo.io/v1alpha1
kind: Sharding
metadata:
  name: sharding_db
spec:
  tables: # map[string]object 类型
    t_order:
      actualDataNodes: "ds_${0..1}.t_order_${0..1}"
      tableStrategy:
        standard:
          shardingColumn: "order_id"
          shardingAlgorithmName: "t_order_inline"
      keyGenerateStrategy:
        column: "order_id"
        keyGeneratorName: "snowflake"
    t_order_item:
      actualDataNodes: "ds_${0..1}.t_order_item_${0..1}"
      tableStrategy:
        standard:
          shardingColumn: "order_id"
          shardingAlgorithmName: "t_order_item_inline"
      keyGenerateStrategy:
        column: order_item_id
        keyGeneratorName: snowflake
  bindingTables:
  - "t_order,t_order_item"
  defaultDatabaseStrategy:
    standard:
     shardingColumn: "user_id"
     shardingAlgorithmName: "database_inline"
  # defaultTableStrategy: # 为空表示 none
  shardingAlgorithms: # map[string]object 类型
    database_inline:
      type: INLINE    
      props: # map[string]string 类型
        algorithm-expression: "ds_${user_id % 2}"
    t_order_inline:  
      type: INLINE    
      props:
        algorithm-expression: "d_order_${order_id % 2}"      
    t_order_item_inline:
      type: INLINE    
      props:
        algorithm-expression: "d_order_item_${order_id % 2}"
  keyGenerators: # map[string]object 类型
    snowflake:
      type: SNOWFLAKE

自动分片 YAML 示例:

apiVersion: database.opensergo.io/v1alpha1
kind: Sharding
metadata:
  name: sharding_db
spec:
  autoTables:
    t_order_auto:
      actualDataNodes: "ds_${0..1}.t_order_${0..1}"
      shardingStrategy:
        standard:
          shardingColumn: ""
          shardingAlgorithmName: ""
@sczyh30 sczyh30 added kind/spec-RFC Issues or PRs for OpenSergo spec RFC area/database-governance Category issues or PRs related to database governance labels Aug 16, 2022
@sczyh30 sczyh30 linked a pull request Sep 8, 2022 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/database-governance Category issues or PRs related to database governance kind/spec-RFC Issues or PRs for OpenSergo spec RFC
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants