From 0cbb1c099dc59c06d6804fb82ff470cce71cdb7f Mon Sep 17 00:00:00 2001 From: Zhigao Hong Date: Wed, 2 Sep 2026 21:15:55 +0800 Subject: [PATCH] [doc](bucket): Document distribution_hash_type --- .../partitioning-and-bucketing.mdx | 12 +++-- .../table-and-view/table/CREATE-TABLE.md | 6 +++ .../data-partitioning/data-bucketing.md | 49 +++++++++++++++++-- .../table-and-view/table/CREATE-TABLE.md | 6 +++ .../data-partitioning/data-bucketing.md | 49 +++++++++++++++++-- 5 files changed, 109 insertions(+), 13 deletions(-) diff --git a/docs/key-features/partitioning-and-bucketing.mdx b/docs/key-features/partitioning-and-bucketing.mdx index 7363d0214907b..aa1c7ef1069fe 100644 --- a/docs/key-features/partitioning-and-bucketing.mdx +++ b/docs/key-features/partitioning-and-bucketing.mdx @@ -8,11 +8,14 @@ keywords: - range partition - list partition - hash bucket + - distribution_hash_type + - crc32 hash + - identity hash - Apache Doris table design slug: /key-features/partitioning-and-bucketing image: /images/next/key-features/partitioning-and-bucketing.jpg last_update: - date: 2026-05-10 + date: 2026-09-02 author: Apache Doris featureCard: tags: @@ -44,7 +47,10 @@ Apache Doris partitioning and bucketing are the two-level physical layout for ev - **`Tablet`**: the physical data shard. The unit of replication, scheduling, and parallelism inside a BE. - **`PARTITION BY RANGE / LIST`**: explicit partitioning declared in DDL. Range fits time and numeric ranges; List fits enumerated dimensions like region or tenant. - **`AUTO PARTITION`**: partitions created on demand at write time, by `date_trunc(col, 'month')` for ranges or by enumerated value for lists. Replaces the static partition list. -- **`DISTRIBUTED BY HASH(col) BUCKETS N`**: hash-distributed shards. `crc32(col) % N` picks the bucket; equality predicates on `col` enable bucket pruning. +- **`DISTRIBUTED BY HASH(col) BUCKETS N`**: hash-distributed shards. `hash(col) % N` picks the bucket; equality predicates on `col` enable bucket pruning. The `hash` function supports a customizable hash algorithm by table's `distribution_hash_type` property: + - **`crc32`** (default): calculates CRC32 over the bucket key; + - **`identity`**: an identity hash that maps the bucket key directly without any hash, just like `col % N`. + - See [Choose the Hash Algorithm](../table-design/data-partitioning/data-bucketing#choose-the-hash-algorithm). - **`DISTRIBUTED BY RANDOM BUCKETS N`**: rows scattered across buckets without a key. Avoids skew, but no bucket pruning. Duplicate-Key tables only. - **`BUCKETS AUTO`**: the FE picks the bucket count from `estimate_partition_size`, BE count, and disk count. Set per partition. @@ -53,7 +59,7 @@ Apache Doris partitioning and bucketing are the two-level physical layout for ev Apache Doris partitioning and bucketing together route a row's path from `INSERT` to disk through two layers, partition selection and tablet selection. 1. **Map the row to a partition.** The planner evaluates the partition expression. Range partitions binary-search a sorted interval list; List partitions look up the value in a hash map. Auto Partition creates the partition on the fly if none matches. -2. **Map the row to a bucket.** Hash distribution computes `crc32(bucket_cols) % bucket_num`; random distribution picks a tablet round-robin (or sticks the whole batch on one tablet when `load_to_single_tablet = true`). +2. **Map the row to a bucket.** Hash distribution applies the table's `distribution_hash_type` (`crc32` by default) to the bucket columns and takes the result modulo `bucket_num`; random distribution picks a tablet round-robin (or sticks the whole batch on one tablet when `load_to_single_tablet = true`). 3. **Write to that tablet's replicas.** Each tablet has N replicas (default 3) on different BEs. The Coordinator streams the row to all of them. 4. **Prune at query time (FE).** `PruneOlapScanPartition` matches `WHERE` predicates against the partition tree; `PruneOlapScanTablet` extracts equality predicates on the bucket key and asks `HashDistributionPruner` which buckets they hash to. EXPLAIN shows the survivors as `partitions=1/365` and `tablets=1/32`. 5. **Scan the survivors in parallel (BE).** Each surviving tablet is a parallel scan unit. The pipeline engine fans them out across BE cores, so a query that touches 16 tablets on 4 BEs runs 16-way parallel without any session tuning. diff --git a/docs/sql-manual/sql-statements/table-and-view/table/CREATE-TABLE.md b/docs/sql-manual/sql-statements/table-and-view/table/CREATE-TABLE.md index 50825e8707d45..ad47b2752ffa1 100644 --- a/docs/sql-manual/sql-statements/table-and-view/table/CREATE-TABLE.md +++ b/docs/sql-manual/sql-statements/table-and-view/table/CREATE-TABLE.md @@ -231,6 +231,10 @@ CREATE TABLE LIKE > Bucketing columns and bucket counts. Detail model bucket columns can be any columns, aggregation model and primary key model bucket columns must be consistent with key columns. Bucket count is any positive integer. For details on bucketing, see the [Manual Bucketing](../../../../table-design/data-partitioning/data-bucketing#1-manually-set-the-number-of-buckets) and [Automatic Bucketing](../../../../table-design/data-partitioning/data-bucketing#2-automatically-set-the-number-of-buckets) sections. +**distribution_hash_type** + +> Optional Hash bucketing algorithm. Valid values are `crc32` and `identity`; the default is `crc32`. The algorithm is fixed after table creation, inherited by new partitions, and must be identical for all tables in the same Colocation Group. For the complete mapping rules, compatibility requirements, and selection guidance, see [Choose the Hash Algorithm](../../../../table-design/data-partitioning/data-bucketing#choose-the-hash-algorithm). + ### Column Default Value Related Parameters **[ GENERATED ALWAYS ] AS ()** @@ -353,6 +357,7 @@ The functionality of creating synchronized materialized views with rollup is lim | storage_medium | Declares the initial storage medium for table data. | | storage_cooldown_time | Sets the expiration time for the initial storage medium of the table data. After this time, it will automatically downgrade to the first-level storage medium. | | colocate_with | When the Colocation Join feature is needed, use this parameter to set the Colocation Group. | +| distribution_hash_type | Hash bucketing algorithm. Valid values are `crc32` (default) and `identity`. This property is valid only with `DISTRIBUTED BY HASH` and cannot be changed after table creation. | | bloom_filter_columns | A list of column names specified by the user that require the addition of a Bloom Filter index. Each column's Bloom Filter index is independent and not a composite index. For example: `"bloom_filter_columns" = "k1, k2, k3"` | | compression | The default compression method for Doris tables is LZ4. After version 1.1, support for specifying ZSTD as the compression method is available for higher compression ratios. | | function_column.sequence_col | When using the Unique Key model, you can specify a Sequence column. When Key columns are the same, REPLACE will be performed according to the Sequence column (the larger value replaces the smaller value; otherwise, it cannot be replaced). `function_column.sequence_col` is used to map the sequence column to a specific column in the table, which can be of integer or date/time types (`DATE`, `DATETIME`, `TIMESTAMP_NS`). The type of this column cannot be changed after creation. If `function_column.sequence_col` is set, `function_column.sequence_type` will be ignored. | @@ -406,6 +411,7 @@ The [user](../../../../admin-manual/auth/authentication-and-authorization.md) ex - A table must specify bucketing columns but can opt out of specifying partitions. For detailed information on partitioning and bucketing, refer to the [Data Partitioning](../../../../table-design/data-partitioning/data-bucketing.md) documentation. - Tables in Doris can be either partitioned or non-partitioned. This attribute is determined at table creation and cannot be changed afterward. That is, for partitioned tables, partitions can be added or removed in subsequent use, while non-partitioned tables cannot have partitions added later. - Partition and bucket columns cannot be altered after table creation; neither the types of partition and bucket columns can be changed nor can these columns be added or removed. + - The Hash bucketing algorithm cannot be altered after table creation. New partitions inherit the table-level `distribution_hash_type`. - Dynamic Partitioning - The dynamic partitioning feature is primarily used to help users manage partitions automatically. By setting certain rules, the Doris system periodically adds new partitions or removes old ones. For more assistance, refer to the [Dynamic Partitioning](../../../../table-design/data-partitioning/dynamic-partitioning.md) documentation. - Automatic Partitioning diff --git a/docs/table-design/data-partitioning/data-bucketing.md b/docs/table-design/data-partitioning/data-bucketing.md index a5b9653c00b2d..d25b635fb9f06 100644 --- a/docs/table-design/data-partitioning/data-bucketing.md +++ b/docs/table-design/data-partitioning/data-bucketing.md @@ -6,6 +6,7 @@ "keywords": [ "Doris data bucketing", "Hash bucketing", + "distribution_hash_type", "Random bucketing", "bucket key selection", "bucket number", @@ -30,8 +31,9 @@ When creating a table, you can complete the bucketing design in the following or |------|------|----------| | 1 | Choose the bucketing method | Whether there are high-frequency filter columns, whether the data is evenly distributed, and the table model | | 2 | Select the bucket key (Hash bucketing only) | Query filter conditions, column cardinality, query concurrency and throughput characteristics | -| 3 | Determine the number of buckets | Data size per Tablet, number of BEs, number of disks | -| 4 | Plan the bucket maintenance strategy | Data growth trend, whether dynamic partitioning is used | +| 3 | Select the hash algorithm (Hash bucketing only) | Choose an appropriate hash algorithm based on the bucket key's data distribution and business logic | +| 4 | Determine the number of buckets | Data size per Tablet, number of BEs, number of disks | +| 5 | Plan the bucket maintenance strategy | Data growth trend, whether dynamic partitioning is used | ## 1. Choose the Bucketing Method @@ -153,7 +155,43 @@ Based on business query characteristics, you can refer to the following principl | High-concurrency point query scenarios | Choose a single column or a small number of columns as the bucket key | A single query triggers a scan of only one bucket, reducing IO interference between queries | | High-throughput query scenarios | Choose multiple columns as the bucket key | Data is more evenly distributed; when the query conditions cannot fully match the equality conditions, overall throughput is improved | -## 3. Determine the Number of Buckets +## 3. Select the Hash Algorithm + +Hash-bucketed tables support the `distribution_hash_type` table property. It controls how Doris maps the bucket-key values to a bucket and accepts the following values: + +| Value | Default | Mapping | Recommended Use | +|---|---|---|---| +| `crc32` | Yes | Calculates CRC32 over the canonical bytes of all bucket-key values, then takes the remainder modulo the bucket number. | General-purpose Hash bucketing. Keep this default unless the application requires a predictable mapping. | +| `identity` | No | Interprets each value's canonical bytes as an unsigned little-endian integer, appends multiple columns in the declared order, and keeps the result modulo the bucket number. | Workloads that require a stable, directly derivable bucket mapping, such as pre-sharded or routing-aware data. | + +For a single non-negative integer bucket key, `identity` has the intuitive result `value % bucket_num`. The following table therefore maps `id` values `0`, `1`, `8`, and `9` to buckets `0`, `1`, `0`, and `1`: + +```sql +CREATE TABLE demo.identity_bucket_tbl ( + id BIGINT NOT NULL, + payload STRING +) +DUPLICATE KEY(id) +DISTRIBUTED BY HASH(id) BUCKETS 8 +PROPERTIES ( + "distribution_hash_type" = "identity", + "replication_num" = "1" +); +``` + +`identity` also supports multiple bucket columns, nullable columns, and all data types that are valid Hash bucket keys. For non-integer values, Doris uses the value's canonical binary representation. `NULL` is represented by four zero bytes. Negative integers use their unsigned two's-complement representation, so their bucket is not calculated with signed modulo. + +:::caution Compatibility and immutability + +- `crc32` is the default and preserves the bucket layout of existing tables. If the property is omitted, `SHOW CREATE TABLE` does not display it. +- The hash algorithm is fixed when the table is created and cannot be changed later. New partitions automatically inherit the table's hash algorithm, even when `ADD PARTITION` specifies a different bucket number. +- Every table in the same Colocation Group must use the same `distribution_hash_type`; Doris rejects a table whose algorithm differs from the group. + +::: + +**Best practices:** Use `crc32` for normal workloads because it mixes arbitrary input bytes and is generally less sensitive to patterns in the bucket key. Note that `crc32` may produce a certain degree of hash collision, so under some data distributions it may lead to bucket skew (uneven data across buckets). If you observe such skew with the default algorithm, consider switching to another hash algorithm. + +## 4. Determine the Number of Buckets In Doris, each Bucket is stored as a physical file (Tablet). The total number of Tablets in a table equals: @@ -248,7 +286,7 @@ The change only affects partitions **created after** the upgrade; the bucket num ::: -## 4. Maintain Data Bucketing +## 5. Maintain Data Bucketing :::tip Tip @@ -256,7 +294,8 @@ Currently, Doris only supports modifying the number of buckets for newly added p 1. Modifying the bucketing type is not supported. 2. Modifying the bucket key is not supported. -3. Modifying the number of buckets for already-created buckets is not supported. +3. Modifying the hash algorithm (`distribution_hash_type`) is not supported. +4. Modifying the number of buckets for already-created buckets is not supported. ::: diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/table-and-view/table/CREATE-TABLE.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/table-and-view/table/CREATE-TABLE.md index 4d8f28a118170..cd95854c5fa8c 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/table-and-view/table/CREATE-TABLE.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-statements/table-and-view/table/CREATE-TABLE.md @@ -233,6 +233,10 @@ CREATE TABLE LIKE > 分桶列和分桶数。明细模型的分桶列可以是任意的列，聚合模型和主键模型的分桶列必须和 key 列保持一致。分桶数是任意的正整数。有关分桶的详细信息，请参阅[手动分桶](../../../../table-design/data-partitioning/data-bucketing#手动设置分桶数)和[自动分桶](../../../../table-design/data-partitioning/data-bucketing#自动设置分桶数)章节。 +**distribution_hash_type** + +> 可选的 Hash 分桶算法。支持 `crc32` 和 `identity`，默认值为 `crc32`。算法在建表后不可修改，新增分区会继承该属性，同一 Colocation Group 中的所有表必须使用相同算法。完整映射规则、兼容性要求和选型建议请参阅[选择 Hash 算法](../../../../table-design/data-partitioning/data-bucketing#选择-hash-算法)。 + ### 列的默认值相关参数 **[ GENERATED ALWAYS ] AS ()** @@ -354,6 +358,7 @@ rollup 可以创建的同步物化视图功能有限。已不再推荐使用。 | storage_medium | 声明表数据的初始存储介质 | | storage_cooldown_time | 设定表数据的初始存储介质的到期时间。超过此时间后，会自动降级到第一级别的存储介质上。 | | colocate_with | 当需要使用 Colocation Join 功能时，使用这个参数设置 Colocation Group。 | +| distribution_hash_type | Hash 分桶算法。支持 `crc32`（默认）和 `identity`。该属性仅对 `DISTRIBUTED BY HASH` 有效，建表后不可修改。 | | bloom_filter_columns | 用户指定需要添加 Bloom Filter 索引的列名称列表。各个列的 Bloom Filter 索引是独立的，并不是组合索引。列如：`"bloom_filter_columns" = "k1, k2, k3"` | | compression | Doris 表的默认压缩方式是 LZ4。1.1 版本后，支持将压缩方式指定为 ZSTD 以获得更高的压缩比。 | | function_column.sequence_col | 当使用 Unique Key 模型时，可以指定一个 Sequence 列，当 Key 列相同时，将按照 Sequence 列进行 REPLACE(较大值替换较小值，否则无法替换) 。`function_column.sequence_col`用来指定 sequence 列到表中某一列的映射，该列可以为整型和时间类型（`DATE`、`DATETIME`、`TIMESTAMP_NS`），创建后不能更改该列的类型。如果设置了`function_column.sequence_col`, `function_column.sequence_type`将被忽略。 | @@ -406,6 +411,7 @@ rollup 可以创建的同步物化视图功能有限。已不再推荐使用。 - 一个表必须指定分桶列，但可以不指定分区。关于分区和分桶的具体介绍，可参阅 [数据划分](../../../../table-design/data-partitioning/auto-partitioning.md) 文档。 - Doris 中的表可以分为分区表和无分区的表。这个属性在建表时确定，之后不可更改。即对于分区表，可以在之后的使用过程中对分区进行增删操作，而对于无分区的表，之后不能再进行增加分区等操作。 - 分区列和分桶列在表创建之后不可更改，既不能更改分区和分桶列的类型，也不能对这些列进行任何增删操作。 + - Hash 分桶算法在建表后不可修改，新增分区会继承表级 `distribution_hash_type`。 - 动态分区 - 动态分区功能主要用于帮助用户自动的管理分区。通过设定一定的规则，Doris 系统定期增加新的分区或删除历史分区。可参阅 [动态分区](../../../../table-design/data-partitioning/dynamic-partitioning.md) 文档查看更多帮助。 - 自动分区 diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/data-partitioning/data-bucketing.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/data-partitioning/data-bucketing.md index d2a9c7e9c0d01..4c5b3ec7f9a8f 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/data-partitioning/data-bucketing.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/data-partitioning/data-bucketing.md @@ -6,6 +6,7 @@ "keywords": [ "Doris 数据分桶", "Hash 分桶", + "distribution_hash_type", "Random 分桶", "分桶键选择", "分桶数量", @@ -30,8 +31,9 @@ |------|------|----------| | 1 | 选择分桶方式 | 是否有高频过滤列、数据是否均匀、表模型 | | 2 | 选择分桶键（仅 Hash 分桶） | 查询过滤条件、列基数、查询并发与吞吐特征 | -| 3 | 确定分桶数量 | 单 Tablet 数据大小、BE 数量、磁盘数 | -| 4 | 规划分桶维护策略 | 数据量增长趋势、是否使用动态分区 | +| 3 | 选择 Hash 算法（仅 Hash 分桶） | 根据分桶键的数据分布与业务逻辑选择合适的 Hash 算法 | +| 4 | 确定分桶数量 | 单 Tablet 数据大小、BE 数量、磁盘数 | +| 5 | 规划分桶维护策略 | 数据量增长趋势、是否使用动态分区 | ## 一、选择分桶方式 @@ -153,7 +155,43 @@ DISTRIBUTED BY RANDOM BUCKETS 8; | 高并发点查场景 | 选择单列或较少列作为分桶键 | 单次查询仅触发一个分桶扫描，减少查询间 IO 影响 | | 大吞吐查询场景 | 选择多列作为分桶键 | 数据分布更均匀；当查询条件不能完全匹配等值条件时，能提升整体吞吐 | -## 三、确定分桶数量 +## 三、 选择 Hash 算法 + +Hash 分桶表可通过表属性 `distribution_hash_type` 指定分桶键到分桶的映射算法。该属性支持以下取值： + +| 取值 | 是否默认 | 映射方式 | 推荐场景 | +|---|---|---|---| +| `crc32` | 是 | 对所有分桶键值的规范字节计算 CRC32，再对分桶数取模。 | 通用 Hash 分桶。除非应用需要可预测的映射，否则建议保留默认值。 | +| `identity` | 否 | 将每个值的规范字节按小端序解释为无符号整数，多列按声明顺序拼接，并在计算过程中对分桶数取模。 | 需要稳定且可直接推导分桶映射的场景，例如数据已预分片或应用感知数据路由。 | + +当分桶键为单个非负整数时，`identity` 的结果可直观表示为 `value % bucket_num`。因此，下列表中 `id` 为 `0`、`1`、`8`、`9` 的数据会分别映射到 `0`、`1`、`0`、`1` 号分桶： + +```sql +CREATE TABLE demo.identity_bucket_tbl ( + id BIGINT NOT NULL, + payload STRING +) +DUPLICATE KEY(id) +DISTRIBUTED BY HASH(id) BUCKETS 8 +PROPERTIES ( + "distribution_hash_type" = "identity", + "replication_num" = "1" +); +``` + +`identity` 同样支持多列分桶键、可空列以及所有可用作 Hash 分桶键的数据类型。对于非整数值，Doris 使用该值的规范二进制表示；`NULL` 使用四个零字节表示。负整数使用其无符号补码表示，因此对应分桶不能通过有符号取模计算。 + +:::caution 兼容性与不可变性 + +- `crc32` 是默认值，并保持已有表的分桶布局。未显式设置该属性时，`SHOW CREATE TABLE` 不会输出该属性。 +- Hash 算法在建表时确定，建表后不能修改。新增分区会自动继承表的 Hash 算法，即使 `ADD PARTITION` 指定了不同的分桶数。 +- 同一 Colocation Group 中的所有表必须使用相同的 `distribution_hash_type`；如果新表使用的算法与 Group 不同，Doris 会拒绝建表。 + +::: + +**最佳实践：** 普通业务建议使用 `crc32`，因为它会混合任意输入字节，通常不易受分桶键自身模式影响。注意 `crc32` 可能产生一定程度的哈希冲突，因此在某些数据分布下可能导致分桶倾斜（各分桶数据不均）。若在默认算法下观察到此类倾斜，可考虑切换到其他 Hash 算法。 + +## 四、 确定分桶数量 在 Doris 中，一个 Bucket 会被存储为一个物理文件（Tablet）。一个表的 Tablet 数量等于： @@ -248,7 +286,7 @@ properties("estimate_partition_size" = "20G") ::: -## 四、维护数据分桶 +## 五、维护数据分桶 :::tip 提示 @@ -256,7 +294,8 @@ properties("estimate_partition_size" = "20G") 1. 不支持修改分桶类型； 2. 不支持修改分桶键； -3. 不支持修改已创建分桶的分桶数量。 +3. 不支持修改 Hash 算法（`distribution_hash_type`）； +4. 不支持修改已创建分桶的分桶数量。 :::