Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions docs/key-features/partitioning-and-bucketing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,14 @@ keywords:
- range partition
- list partition
- hash bucket
- distribution_hash_type
- crc32 hash
- identity hash
- Apache Doris table design
slug: /key-features/partitioning-and-bucketing
image: /images/next/key-features/partitioning-and-bucketing.jpg
last_update:
date: 2026-05-10
date: 2026-09-02
author: Apache Doris
featureCard:
tags:
Expand Down Expand Up @@ -44,7 +47,10 @@ Apache Doris partitioning and bucketing are the two-level physical layout for ev
- **`Tablet`**: the physical data shard. The unit of replication, scheduling, and parallelism inside a BE.
- **`PARTITION BY RANGE / LIST`**: explicit partitioning declared in DDL. Range fits time and numeric ranges; List fits enumerated dimensions like region or tenant.
- **`AUTO PARTITION`**: partitions created on demand at write time, by `date_trunc(col, 'month')` for ranges or by enumerated value for lists. Replaces the static partition list.
- **`DISTRIBUTED BY HASH(col) BUCKETS N`**: hash-distributed shards. `crc32(col) % N` picks the bucket; equality predicates on `col` enable bucket pruning.
- **`DISTRIBUTED BY HASH(col) BUCKETS N`**: hash-distributed shards. `hash(col) % N` picks the bucket; equality predicates on `col` enable bucket pruning. The `hash` function supports a customizable hash algorithm by table's `distribution_hash_type` property:
- **`crc32`** (default): calculates CRC32 over the bucket key;
- **`identity`**: an identity hash that maps the bucket key directly without any hash, just like `col % N`.
- See [Choose the Hash Algorithm](../table-design/data-partitioning/data-bucketing#choose-the-hash-algorithm).
- **`DISTRIBUTED BY RANDOM BUCKETS N`**: rows scattered across buckets without a key. Avoids skew, but no bucket pruning. Duplicate-Key tables only.
- **`BUCKETS AUTO`**: the FE picks the bucket count from `estimate_partition_size`, BE count, and disk count. Set per partition.

Expand All @@ -53,7 +59,7 @@ Apache Doris partitioning and bucketing are the two-level physical layout for ev
Apache Doris partitioning and bucketing together route a row's path from `INSERT` to disk through two layers, partition selection and tablet selection.

1. **Map the row to a partition.** The planner evaluates the partition expression. Range partitions binary-search a sorted interval list; List partitions look up the value in a hash map. Auto Partition creates the partition on the fly if none matches.
2. **Map the row to a bucket.** Hash distribution computes `crc32(bucket_cols) % bucket_num`; random distribution picks a tablet round-robin (or sticks the whole batch on one tablet when `load_to_single_tablet = true`).
2. **Map the row to a bucket.** Hash distribution applies the table's `distribution_hash_type` (`crc32` by default) to the bucket columns and takes the result modulo `bucket_num`; random distribution picks a tablet round-robin (or sticks the whole batch on one tablet when `load_to_single_tablet = true`).
3. **Write to that tablet's replicas.** Each tablet has N replicas (default 3) on different BEs. The Coordinator streams the row to all of them.
4. **Prune at query time (FE).** `PruneOlapScanPartition` matches `WHERE` predicates against the partition tree; `PruneOlapScanTablet` extracts equality predicates on the bucket key and asks `HashDistributionPruner` which buckets they hash to. EXPLAIN shows the survivors as `partitions=1/365` and `tablets=1/32`.
5. **Scan the survivors in parallel (BE).** Each surviving tablet is a parallel scan unit. The pipeline engine fans them out across BE cores, so a query that touches 16 tablets on 4 BEs runs 16-way parallel without any session tuning.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,10 @@ CREATE TABLE <new_table_name> LIKE <existing_table_name>

> Bucketing columns and bucket counts. Detail model bucket columns can be any columns, aggregation model and primary key model bucket columns must be consistent with key columns. Bucket count is any positive integer. For details on bucketing, see the [Manual Bucketing](../../../../table-design/data-partitioning/data-bucketing#1-manually-set-the-number-of-buckets) and [Automatic Bucketing](../../../../table-design/data-partitioning/data-bucketing#2-automatically-set-the-number-of-buckets) sections.

**distribution_hash_type**

> Optional Hash bucketing algorithm. Valid values are `crc32` and `identity`; the default is `crc32`. The algorithm is fixed after table creation, inherited by new partitions, and must be identical for all tables in the same Colocation Group. For the complete mapping rules, compatibility requirements, and selection guidance, see [Choose the Hash Algorithm](../../../../table-design/data-partitioning/data-bucketing#choose-the-hash-algorithm).

### Column Default Value Related Parameters

**[ GENERATED ALWAYS ] AS (<col_generate_expression>)**
Expand Down Expand Up @@ -353,6 +357,7 @@ The functionality of creating synchronized materialized views with rollup is lim
| storage_medium | Declares the initial storage medium for table data. |
| storage_cooldown_time | Sets the expiration time for the initial storage medium of the table data. After this time, it will automatically downgrade to the first-level storage medium. |
| colocate_with | When the Colocation Join feature is needed, use this parameter to set the Colocation Group. |
| distribution_hash_type | Hash bucketing algorithm. Valid values are `crc32` (default) and `identity`. This property is valid only with `DISTRIBUTED BY HASH` and cannot be changed after table creation. |
| bloom_filter_columns | A list of column names specified by the user that require the addition of a Bloom Filter index. Each column's Bloom Filter index is independent and not a composite index. For example: `"bloom_filter_columns" = "k1, k2, k3"` |
| compression | The default compression method for Doris tables is LZ4. After version 1.1, support for specifying ZSTD as the compression method is available for higher compression ratios. |
| function_column.sequence_col | When using the Unique Key model, you can specify a Sequence column. When Key columns are the same, REPLACE will be performed according to the Sequence column (the larger value replaces the smaller value; otherwise, it cannot be replaced). `function_column.sequence_col` is used to map the sequence column to a specific column in the table, which can be of integer or date/time types (`DATE`, `DATETIME`, `TIMESTAMP_NS`). The type of this column cannot be changed after creation. If `function_column.sequence_col` is set, `function_column.sequence_type` will be ignored. |
Expand Down Expand Up @@ -406,6 +411,7 @@ The [user](../../../../admin-manual/auth/authentication-and-authorization.md) ex
- A table must specify bucketing columns but can opt out of specifying partitions. For detailed information on partitioning and bucketing, refer to the [Data Partitioning](../../../../table-design/data-partitioning/data-bucketing.md) documentation.
- Tables in Doris can be either partitioned or non-partitioned. This attribute is determined at table creation and cannot be changed afterward. That is, for partitioned tables, partitions can be added or removed in subsequent use, while non-partitioned tables cannot have partitions added later.
- Partition and bucket columns cannot be altered after table creation; neither the types of partition and bucket columns can be changed nor can these columns be added or removed.
- The Hash bucketing algorithm cannot be altered after table creation. New partitions inherit the table-level `distribution_hash_type`.
- Dynamic Partitioning
- The dynamic partitioning feature is primarily used to help users manage partitions automatically. By setting certain rules, the Doris system periodically adds new partitions or removes old ones. For more assistance, refer to the [Dynamic Partitioning](../../../../table-design/data-partitioning/dynamic-partitioning.md) documentation.
- Automatic Partitioning
Expand Down
49 changes: 44 additions & 5 deletions docs/table-design/data-partitioning/data-bucketing.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"keywords": [
"Doris data bucketing",
"Hash bucketing",
"distribution_hash_type",
"Random bucketing",
"bucket key selection",
"bucket number",
Expand All @@ -30,8 +31,9 @@ When creating a table, you can complete the bucketing design in the following or
|------|------|----------|
| 1 | Choose the bucketing method | Whether there are high-frequency filter columns, whether the data is evenly distributed, and the table model |
| 2 | Select the bucket key (Hash bucketing only) | Query filter conditions, column cardinality, query concurrency and throughput characteristics |
| 3 | Determine the number of buckets | Data size per Tablet, number of BEs, number of disks |
| 4 | Plan the bucket maintenance strategy | Data growth trend, whether dynamic partitioning is used |
| 3 | Select the hash algorithm (Hash bucketing only) | Choose an appropriate hash algorithm based on the bucket key's data distribution and business logic |
| 4 | Determine the number of buckets | Data size per Tablet, number of BEs, number of disks |
| 5 | Plan the bucket maintenance strategy | Data growth trend, whether dynamic partitioning is used |

## 1. Choose the Bucketing Method

Expand Down Expand Up @@ -153,7 +155,43 @@ Based on business query characteristics, you can refer to the following principl
| High-concurrency point query scenarios | Choose a single column or a small number of columns as the bucket key | A single query triggers a scan of only one bucket, reducing IO interference between queries |
| High-throughput query scenarios | Choose multiple columns as the bucket key | Data is more evenly distributed; when the query conditions cannot fully match the equality conditions, overall throughput is improved |

## 3. Determine the Number of Buckets
## 3. Select the Hash Algorithm

Hash-bucketed tables support the `distribution_hash_type` table property. It controls how Doris maps the bucket-key values to a bucket and accepts the following values:

| Value | Default | Mapping | Recommended Use |
|---|---|---|---|
| `crc32` | Yes | Calculates CRC32 over the canonical bytes of all bucket-key values, then takes the remainder modulo the bucket number. | General-purpose Hash bucketing. Keep this default unless the application requires a predictable mapping. |
| `identity` | No | Interprets each value's canonical bytes as an unsigned little-endian integer, appends multiple columns in the declared order, and keeps the result modulo the bucket number. | Workloads that require a stable, directly derivable bucket mapping, such as pre-sharded or routing-aware data. |

For a single non-negative integer bucket key, `identity` has the intuitive result `value % bucket_num`. The following table therefore maps `id` values `0`, `1`, `8`, and `9` to buckets `0`, `1`, `0`, and `1`:

```sql
CREATE TABLE demo.identity_bucket_tbl (
id BIGINT NOT NULL,
payload STRING
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 8
PROPERTIES (
"distribution_hash_type" = "identity",
"replication_num" = "1"
);
```

`identity` also supports multiple bucket columns, nullable columns, and all data types that are valid Hash bucket keys. For non-integer values, Doris uses the value's canonical binary representation. `NULL` is represented by four zero bytes. Negative integers use their unsigned two's-complement representation, so their bucket is not calculated with signed modulo.

:::caution Compatibility and immutability

- `crc32` is the default and preserves the bucket layout of existing tables. If the property is omitted, `SHOW CREATE TABLE` does not display it.
- The hash algorithm is fixed when the table is created and cannot be changed later. New partitions automatically inherit the table's hash algorithm, even when `ADD PARTITION` specifies a different bucket number.
- Every table in the same Colocation Group must use the same `distribution_hash_type`; Doris rejects a table whose algorithm differs from the group.

:::

**Best practices:** Use `crc32` for normal workloads because it mixes arbitrary input bytes and is generally less sensitive to patterns in the bucket key. Note that `crc32` may produce a certain degree of hash collision, so under some data distributions it may lead to bucket skew (uneven data across buckets). If you observe such skew with the default algorithm, consider switching to another hash algorithm.

## 4. Determine the Number of Buckets

In Doris, each Bucket is stored as a physical file (Tablet). The total number of Tablets in a table equals:

Expand Down Expand Up @@ -248,15 +286,16 @@ The change only affects partitions **created after** the upgrade; the bucket num

:::

## 4. Maintain Data Bucketing
## 5. Maintain Data Bucketing

:::tip Tip

Currently, Doris only supports modifying the number of buckets for newly added partitions. The following operations are not supported:

1. Modifying the bucketing type is not supported.
2. Modifying the bucket key is not supported.
3. Modifying the number of buckets for already-created buckets is not supported.
3. Modifying the hash algorithm (`distribution_hash_type`) is not supported.
4. Modifying the number of buckets for already-created buckets is not supported.

:::

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,10 @@ CREATE TABLE <table_name> LIKE <source_table>

> 分桶列和分桶数。明细模型的分桶列可以是任意的列,聚合模型和主键模型的分桶列必须和 key 列保持一致。分桶数是任意的正整数。有关分桶的详细信息,请参阅[手动分桶](../../../../table-design/data-partitioning/data-bucketing#手动设置分桶数)和[自动分桶](../../../../table-design/data-partitioning/data-bucketing#自动设置分桶数)章节。

**distribution_hash_type**

> 可选的 Hash 分桶算法。支持 `crc32` 和 `identity`,默认值为 `crc32`。算法在建表后不可修改,新增分区会继承该属性,同一 Colocation Group 中的所有表必须使用相同算法。完整映射规则、兼容性要求和选型建议请参阅[选择 Hash 算法](../../../../table-design/data-partitioning/data-bucketing#选择-hash-算法)。

### 列的默认值相关参数

**[ GENERATED ALWAYS ] AS (<col_generate_expression>)**
Expand Down Expand Up @@ -354,6 +358,7 @@ rollup 可以创建的同步物化视图功能有限。已不再推荐使用。
| storage_medium | 声明表数据的初始存储介质 |
| storage_cooldown_time | 设定表数据的初始存储介质的到期时间。超过此时间后,会自动降级到第一级别的存储介质上。 |
| colocate_with | 当需要使用 Colocation Join 功能时,使用这个参数设置 Colocation Group。 |
| distribution_hash_type | Hash 分桶算法。支持 `crc32`(默认)和 `identity`。该属性仅对 `DISTRIBUTED BY HASH` 有效,建表后不可修改。 |
| bloom_filter_columns | 用户指定需要添加 Bloom Filter 索引的列名称列表。各个列的 Bloom Filter 索引是独立的,并不是组合索引。列如:`"bloom_filter_columns" = "k1, k2, k3"` |
| compression | Doris 表的默认压缩方式是 LZ4。1.1 版本后,支持将压缩方式指定为 ZSTD 以获得更高的压缩比。 |
| function_column.sequence_col | 当使用 Unique Key 模型时,可以指定一个 Sequence 列,当 Key 列相同时,将按照 Sequence 列进行 REPLACE(较大值替换较小值,否则无法替换) 。`function_column.sequence_col`用来指定 sequence 列到表中某一列的映射,该列可以为整型和时间类型(`DATE`、`DATETIME`、`TIMESTAMP_NS`),创建后不能更改该列的类型。如果设置了`function_column.sequence_col`, `function_column.sequence_type`将被忽略。 |
Expand Down Expand Up @@ -406,6 +411,7 @@ rollup 可以创建的同步物化视图功能有限。已不再推荐使用。
- 一个表必须指定分桶列,但可以不指定分区。关于分区和分桶的具体介绍,可参阅 [数据划分](../../../../table-design/data-partitioning/auto-partitioning.md) 文档。
- Doris 中的表可以分为分区表和无分区的表。这个属性在建表时确定,之后不可更改。即对于分区表,可以在之后的使用过程中对分区进行增删操作,而对于无分区的表,之后不能再进行增加分区等操作。
- 分区列和分桶列在表创建之后不可更改,既不能更改分区和分桶列的类型,也不能对这些列进行任何增删操作。
- Hash 分桶算法在建表后不可修改,新增分区会继承表级 `distribution_hash_type`。
- 动态分区
- 动态分区功能主要用于帮助用户自动的管理分区。通过设定一定的规则,Doris 系统定期增加新的分区或删除历史分区。可参阅 [动态分区](../../../../table-design/data-partitioning/dynamic-partitioning.md) 文档查看更多帮助。
- 自动分区
Expand Down
Loading