From 369d4374de8baa09ce3d57bb5ea4ba6ae943030a Mon Sep 17 00:00:00 2001 From: Samy OUBOUAZIZ Date: Thu, 16 Oct 2025 14:18:15 +0200 Subject: [PATCH 1/6] feat(dwh): update db and users mgmt pages for beta MTA-6431 --- .../how-to/manage-databases.mdx | 49 ++++++-- pages/data-warehouse/how-to/manage-users.mdx | 106 ++++++++++++++++-- 2 files changed, 136 insertions(+), 19 deletions(-) diff --git a/pages/data-warehouse/how-to/manage-databases.mdx b/pages/data-warehouse/how-to/manage-databases.mdx index 6d8e402582..b6f169c250 100644 --- a/pages/data-warehouse/how-to/manage-databases.mdx +++ b/pages/data-warehouse/how-to/manage-databases.mdx @@ -9,11 +9,7 @@ dates: import Requirements from '@macros/iam/requirements.mdx' -This page explains how to manage databases in your Data Warehouse for ClickHouse® deployment using the [Scaleway console](https://console.scaleway.com/). - - -During the private beta phase, database management can only be done with the **scwadmin** & via the ClickHouse® CLI, MySQL or HTTPS protocols. - +This page explains how to manage databases in your Data Warehouse for ClickHouse® deployment. @@ -24,14 +20,47 @@ During the private beta phase, database management can only be done with the **s ## How to manage databases using the Scaleway console -During the private beta phase, the management of databases in your Data Warehouse for ClickHouse® deployment is limited to viewing databases in your deployment. +The Scaleway console allows you to create and delete databases for your Data Warehouse for ClickHouse® deployment. + +### How to create a database using the Scaleway console + +1. Click **ClickHouse®** under **Data & Analytics** on the side menu. The Data Warehouse deployment page displays. + +2. Click the name of the desired Data Warehouse deployment. The **Overview** tab of the deployment displays. + +3. Click the **Databases** tab. A list of your current databases displays. + +4. Click **+ Create database**. A popup displays. + +5. Enter a name for your new database. It can can only contain alphanumeric characters, underscores, and dashes. + +6. Click **Create database** to confirm. + +Your new database appears in the list. + +### How to delete a database using the Scaleway console + + +The `default` database cannot be deleted. + + +1. Click **ClickHouse®** under **Data & Analytics** on the side menu. The Data Warehouse deployment page displays. + +2. Click the name of the desired Data Warehouse deployment. The **Overview** tab of the deployment displays. + +3. Click next to the name of the database you want to delete. A confirmation popup displays. + +4. Enter `DELETE`, then click **Delete database** to confirm. + +The deleted database no longer appears in the list. ## How to manage databases using frameworks -[Connect to your deployment](/data-warehouse/how-to/connect-applications/) using your preferred framework. - - You can also connect using the ClickHouse HTTP console from the **Overview** tab of your deployment. Make sure to enter valid credentials in the top-right fields. - +[Connect to your deployment](/data-warehouse/how-to/connect-applications/) using your preferred framework. + + +You can also connect using the ClickHouse HTTP console from the **Overview** tab of your deployment. Make sure to enter valid credentials in the top-right fields. + Once connected, you can run SQL queries to `CREATE/ALTER/DROP` your `DATABASE/TABLE`. diff --git a/pages/data-warehouse/how-to/manage-users.mdx b/pages/data-warehouse/how-to/manage-users.mdx index 90586137cc..3957f6233f 100644 --- a/pages/data-warehouse/how-to/manage-users.mdx +++ b/pages/data-warehouse/how-to/manage-users.mdx @@ -9,11 +9,7 @@ dates: import Requirements from '@macros/iam/requirements.mdx' -This page explains how to manage users in your Data Warehouse for ClickHouse® deployment using the [Scaleway console](https://console.scaleway.com/). - - -During the private beta phase, user management can only be done with the **scwadmin** user via the ClickHouse® CLI, MySQL or HTTPS protocols. - +This page explains how to manage users in your Data Warehouse for ClickHouse® deployment. @@ -24,9 +20,53 @@ During the private beta phase, user management can only be done with the **scwad ## How to manage users using the Scaleway console -During the private beta phase, the management of users for your Data Warehouse for ClickHouse® deployment is limited to: -- Viewing users in your deployment -- Changing the password of an existing user +The Scaleway console allows you to create users for your Data Warehouse for ClickHouse® deployment, to grant them admin rights, to update their password, and to delete them. + +### How to create a user using the Scaleway console + +1. Click **ClickHouse®** under **Data & Analytics** on the side menu. The Data Warehouse deployment page displays. + +2. Click the name of the desired Data Warehouse deployment. The **Overview** tab of the deployment displays. + +3. Click the **Users** tab. A list of your current users displays. + +4. Click **+ Create user**. A popup displays. + +5. Enter a username. It must be unique can contain up to 63 characters, and must start with a letter. Only alphanumeric characters, underscores, and dashes are accepted. + +6. Enter a password, or generate one automatically. + +7. Enable the toggle if you want to grant admin rights to the user. + +8. Click **Create user** to proceed. + +The newly created user appears in the list of your deployment's users. + +### How to update a user's password and admin rights + +1. Click **ClickHouse®** under **Data & Analytics** on the side menu. The Data Warehouse deployment page displays. + +2. Click the name of the desired Data Warehouse deployment. The **Overview** tab of the deployment displays. + +3. Click the **Users** tab. A list of your current users displays. + +4. Click next to the name of the user you want to update, then click ¨**Update user**. A popup displays. + +5. Update the user's password and/or admin rights, then click **Update user** to confirm. + +### How to delete a user using the Scaleway console + +1. Click **ClickHouse®** under **Data & Analytics** on the side menu. The Data Warehouse deployment page displays. + +2. Click the name of the desired Data Warehouse deployment. The **Overview** tab of the deployment displays. + +3. Click the **Users** tab. A list of your current users displays. + +4. Click next to the name of the user you want to delete. A confirmation popup displays. + +5. Enter `DELETE`, then click **Delete user** to confirm.`` + +The user is deleted and no longer appears in the list. ## How to manage users using frameworks @@ -37,4 +77,52 @@ During the private beta phase, the management of users for your Data Warehouse f Once connected, you can run SQL queries using `USER`, `ROLE`, and `ROW POLICY` to create new users and grant them the desired permissions via roles. -Refer to the official ClickHouse® documentation on [users and roles](https://clickhouse.com/docs/sql-reference/statements/create/user) for more information. \ No newline at end of file +Refer to the official ClickHouse® documentation on [users and roles](https://clickhouse.com/docs/sql-reference/statements/create/user) for more information. + +## Manage user permissions + +[Connect to your deployment](/data-warehouse/how-to/connect-applications/) using your preferred framework. + +Once connected, you can run SQL as shown below to grant the desired permissions to users. + +**Grant permission to create databases** + +```sql +GRANT CREATE DATABASE ON . TO user; +``` + +**Grant permission to create tables in any database** + +```sql +GRANT CREATE TABLE ON . TO user; +``` + +**Grant permission to create tables in a specific database** + +```sql +GRANT CREATE TABLE ON database_name.* TO user; +``` + +**Grant data manipulation permissions for any table in any database** + +```sql +GRANT INSERT, UPDATE, DELETE ON . TO user; +``` + +**Grant data manipulation permissions for any table in a specific database** + +```sql +GRANT INSERT, UPDATE, DELETE ON database_name.* TO user; +``` + +**Grant data manipulation permissions for a specific table in a specific database** + +```sql +GRANT INSERT, UPDATE, DELETE ON database_name.table_name TO user; +``` + +**Grant every permissions on every database** + +```sql +GRANT CREATE DATABASE, CREATE TABLE, INSERT, UPDATE, DELETE ON . TO user; +``` \ No newline at end of file From 397b728fb90a4bc604f2309b0e0b7a95eac3260b Mon Sep 17 00:00:00 2001 From: Samy OUBOUAZIZ Date: Thu, 16 Oct 2025 14:22:30 +0200 Subject: [PATCH 2/6] feat(dwh): update --- pages/data-warehouse/how-to/connect-applications.mdx | 1 - pages/data-warehouse/how-to/connect-bi-tools.mdx | 1 - pages/data-warehouse/how-to/create-deployment.mdx | 1 - pages/data-warehouse/how-to/edit-autoscaling.mdx | 2 +- pages/data-warehouse/how-to/import-data.mdx | 2 +- pages/data-warehouse/how-to/manage-databases.mdx | 1 - pages/data-warehouse/how-to/manage-delete-deployment.mdx | 1 - pages/data-warehouse/how-to/manage-users.mdx | 1 - pages/data-warehouse/quickstart.mdx | 1 - 9 files changed, 2 insertions(+), 9 deletions(-) diff --git a/pages/data-warehouse/how-to/connect-applications.mdx b/pages/data-warehouse/how-to/connect-applications.mdx index cde63f2949..31ab2ecd04 100644 --- a/pages/data-warehouse/how-to/connect-applications.mdx +++ b/pages/data-warehouse/how-to/connect-applications.mdx @@ -17,7 +17,6 @@ To connect your deployment with BI tools, refer to the [dedicated documentation] - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) 1. Click **ClickHouse®** under **Data & Analytics** on the side menu. The Data Warehouse deployment page displays. diff --git a/pages/data-warehouse/how-to/connect-bi-tools.mdx b/pages/data-warehouse/how-to/connect-bi-tools.mdx index cee52e3485..195c40b5ae 100644 --- a/pages/data-warehouse/how-to/connect-bi-tools.mdx +++ b/pages/data-warehouse/how-to/connect-bi-tools.mdx @@ -15,7 +15,6 @@ This page explains how to integrate your Data Warehouse for ClickHouse® deploym - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) ## Tableau diff --git a/pages/data-warehouse/how-to/create-deployment.mdx b/pages/data-warehouse/how-to/create-deployment.mdx index f921343806..381617df3f 100644 --- a/pages/data-warehouse/how-to/create-deployment.mdx +++ b/pages/data-warehouse/how-to/create-deployment.mdx @@ -15,7 +15,6 @@ A Data Warehouse is a centralized repository designed to store, manage, and anal - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. ## Creating a Data Warehouse Deployment diff --git a/pages/data-warehouse/how-to/edit-autoscaling.mdx b/pages/data-warehouse/how-to/edit-autoscaling.mdx index 345bf583d8..2713590b17 100644 --- a/pages/data-warehouse/how-to/edit-autoscaling.mdx +++ b/pages/data-warehouse/how-to/edit-autoscaling.mdx @@ -15,7 +15,7 @@ This page explains how to edit the autoscaling configuration of your Data Wareho - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. + - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) diff --git a/pages/data-warehouse/how-to/import-data.mdx b/pages/data-warehouse/how-to/import-data.mdx index bd154c0294..70ce718e58 100644 --- a/pages/data-warehouse/how-to/import-data.mdx +++ b/pages/data-warehouse/how-to/import-data.mdx @@ -17,7 +17,7 @@ Scaleway Data Warehouse for ClickHouse® allows you to quickly import any type o - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. + - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) ## How to import data using the ClickHouse® CLI diff --git a/pages/data-warehouse/how-to/manage-databases.mdx b/pages/data-warehouse/how-to/manage-databases.mdx index b6f169c250..b7a2bbe0ef 100644 --- a/pages/data-warehouse/how-to/manage-databases.mdx +++ b/pages/data-warehouse/how-to/manage-databases.mdx @@ -15,7 +15,6 @@ This page explains how to manage databases in your Data Warehouse for ClickHouse - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) ## How to manage databases using the Scaleway console diff --git a/pages/data-warehouse/how-to/manage-delete-deployment.mdx b/pages/data-warehouse/how-to/manage-delete-deployment.mdx index ceb5ae1771..496d84d680 100644 --- a/pages/data-warehouse/how-to/manage-delete-deployment.mdx +++ b/pages/data-warehouse/how-to/manage-delete-deployment.mdx @@ -15,7 +15,6 @@ This page explains how to manage and delete your Data Warehouse deployment. - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) ## How to manage a Data Warehouse deployment diff --git a/pages/data-warehouse/how-to/manage-users.mdx b/pages/data-warehouse/how-to/manage-users.mdx index 3957f6233f..7cca6c523d 100644 --- a/pages/data-warehouse/how-to/manage-users.mdx +++ b/pages/data-warehouse/how-to/manage-users.mdx @@ -15,7 +15,6 @@ This page explains how to manage users in your Data Warehouse for ClickHouse® d - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. - Created a [Data Warehouse deployment](/data-warehouse/how-to/create-deployment/) ## How to manage users using the Scaleway console diff --git a/pages/data-warehouse/quickstart.mdx b/pages/data-warehouse/quickstart.mdx index 9bcc2c4960..9a0123fb6f 100644 --- a/pages/data-warehouse/quickstart.mdx +++ b/pages/data-warehouse/quickstart.mdx @@ -15,7 +15,6 @@ Data Warehouse for ClickHouse® is a centralized repository to store large sets - A Scaleway account logged into the [console](https://console.scaleway.com) - [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization -- [Signed up to the private beta](https://www.scaleway.com/fr/betas/) and received a confirmation email. ## How to create a Data Warehouse for ClickHouse® deployment From 30af0506f4579b108791de105502d94bca42ad1b Mon Sep 17 00:00:00 2001 From: Samy OUBOUAZIZ Date: Thu, 16 Oct 2025 14:33:07 +0200 Subject: [PATCH 3/6] feat(dwh): update --- pages/data-warehouse/how-to/manage-storage.mdx | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 pages/data-warehouse/how-to/manage-storage.mdx diff --git a/pages/data-warehouse/how-to/manage-storage.mdx b/pages/data-warehouse/how-to/manage-storage.mdx new file mode 100644 index 0000000000..12bfeade32 --- /dev/null +++ b/pages/data-warehouse/how-to/manage-storage.mdx @@ -0,0 +1,9 @@ +--- +title: How to manage data storage in a Data Warehouse for ClickHouse® deployment +description: This page explains how to manage data storage in a Data Warehouse for ClickHouse® deployment using storage policies +tags: data storage policy strategy options object block +dates: + validation: 2025-10-16 + posted: 2025-10-16 +--- + From c2d3e1e91b5b173d9048107c74175edf37a02190 Mon Sep 17 00:00:00 2001 From: Samy OUBOUAZIZ Date: Thu, 16 Oct 2025 17:12:11 +0200 Subject: [PATCH 4/6] feat(dwh): update --- .../data-warehouse/how-to/manage-storage.mdx | 116 ++++++++++++++++++ pages/data-warehouse/menu.ts | 16 ++- 2 files changed, 126 insertions(+), 6 deletions(-) diff --git a/pages/data-warehouse/how-to/manage-storage.mdx b/pages/data-warehouse/how-to/manage-storage.mdx index 12bfeade32..65f8090eaa 100644 --- a/pages/data-warehouse/how-to/manage-storage.mdx +++ b/pages/data-warehouse/how-to/manage-storage.mdx @@ -7,3 +7,119 @@ dates: posted: 2025-10-16 --- +## Overview + +Scaleway Data Warehouse for ClickHouse® uses ClickHouse®'s native storage policies to decide where table data lives: on Scaleway Block storage (local PVC), and/or on Object Storage (Amazon S3). You can choose between three policies for your deployment: + +- `tiered` (default in Scaleway Data Warehouse for ClickHouse®) +- `s3_cache` (cache mode) +- `default` (local-only mode) + +The table below provides an overview of each storage policy's behavior and use case. + +| Storage policy | Behavior | When to use | +|------|---------|------------| +| `tiered` | Writes land on Block storage first (hot volume). When the local disk reaches ~90% fullness, parts are automatically moved to Object storage (cold volume). This is controlled by `move_factor: 0.1` (move when only ~10% free remains). | General purpose setting for fast local writes and transparent spillover to Object Storage as the dataset grows. | +| `s3_cache` | Data is mainly stored on Object Storage. A local cache layer (on Block storage) keeps frequently-read parts to accelerate repeated reads. | Large datasets that mostly live in Object Storage, with repeated reads on hot subsets that benefit from local caching. | +| `default` | Data is stored on Block storage only. | Small datasets where lowest latency for reads/writes on local disk is desired and capacity fits the PVC. | + + Under the hood, our ClickHouse config defines three disks (default = local PVC, s3 = object storage, s3_cache = S3 + local cache) and three policies (tiered, s3_cache, default). + Cluster default (for new tables when not specified): tiered. + +How clients choose a policy (per table) + +The storage policy is chosen at table creation time via SETTINGS storage_policy = '...'. + +## Tiered storage policy + +**Tiered** (Block Storage first, spill to Object Storage) is the cluster default policy. If you omit the `storage_policy` setting; ClickHouse® will automatically apply this policy. + + + + + Storage policy is explicitly defined as `tiered`. + + ```sql + CREATE TABLE tiered_table + ( + id UInt64, + ts DateTime, + value Float32 + ) + ENGINE = MergeTree + PARTITION BY toYYYYMM(ts) + ORDER BY id + SETTINGS storage_policy = 'tiered'; + ``` + + + + + Storage policy is not defined, ClickHouse® applies the `tiered` storage policy by default. + + ```sql + CREATE TABLE dw.tiered_table_implicit + ( + id UInt64, + ts DateTime, + value Float32 + ) + ENGINE = MergeTree + PARTITION BY toYYYYMM(ts) + ORDER BY id; + ``` + + + +S3 with local cache: + +```sql +CREATE TABLE cache_table +( + id UInt64, + ts DateTime, + value Float32 +) +ENGINE = MergeTree +PARTITION BY toYYYYMM(ts) +ORDER BY id +SETTINGS storage_policy = 's3_cache'; +``` + +Local-only: + +```sql +CREATE TABLE local_table +( + id UInt64, + ts DateTime, + value Float32 +) +ENGINE = MergeTree +PARTITION BY toYYYYMM(ts) +ORDER BY id +SETTINGS storage_policy = 'default'; +``` + +Moving data with the tiered policy + +With tiered, you can manually move partitions between volumes: + +Move specific partitions to cold (S3): + +```sql +ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'cold'; +``` + +For non-partitioned tables (tuple()): + +```sql +ALTER TABLE tiered_table_without_partitions MOVE PARTITION tuple() TO VOLUME 'cold'; +``` + +Move back to hot (local): + +```sql +ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'hot'; +``` + diff --git a/pages/data-warehouse/menu.ts b/pages/data-warehouse/menu.ts index bb830f4cab..16cffa3c4b 100644 --- a/pages/data-warehouse/menu.ts +++ b/pages/data-warehouse/menu.ts @@ -30,6 +30,14 @@ export const dataWarehouseMenu = { label: 'Connect to a deployment', slug: 'connect-applications', }, + { + label: 'Manage databases in a deployment', + slug: 'manage-databases', + }, + { + label: 'Manage users in a deployment', + slug: 'manage-users', + }, { label: 'Connect a deployment with BI tools', slug: 'connect-bi-tools', @@ -43,12 +51,8 @@ export const dataWarehouseMenu = { slug: 'edit-autoscaling', }, { - label: 'Manage databases in a deployment', - slug: 'manage-databases', - }, - { - label: 'Manage users in a deployment', - slug: 'manage-users', + label: 'Manage the storage policy of a deployment', + slug: 'manage-storage', }, ], label: 'How to', From e4c7217c9b37ea87564d9817fb437ebdbfbc594f Mon Sep 17 00:00:00 2001 From: Samy OUBOUAZIZ Date: Fri, 17 Oct 2025 14:18:10 +0200 Subject: [PATCH 5/6] feat(dwh): update --- .../data-warehouse/how-to/manage-storage.mdx | 86 +++++++++++-------- 1 file changed, 51 insertions(+), 35 deletions(-) diff --git a/pages/data-warehouse/how-to/manage-storage.mdx b/pages/data-warehouse/how-to/manage-storage.mdx index 65f8090eaa..668fc18fd7 100644 --- a/pages/data-warehouse/how-to/manage-storage.mdx +++ b/pages/data-warehouse/how-to/manage-storage.mdx @@ -7,9 +7,16 @@ dates: posted: 2025-10-16 --- +import Requirements from '@macros/iam/requirements.mdx' + + + +- A Scaleway account logged into the [console](https://console.scaleway.com) +- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization + ## Overview -Scaleway Data Warehouse for ClickHouse® uses ClickHouse®'s native storage policies to decide where table data lives: on Scaleway Block storage (local PVC), and/or on Object Storage (Amazon S3). You can choose between three policies for your deployment: +Scaleway Data Warehouse for ClickHouse® uses ClickHouse®'s native storage policies to decide where table data lives: on Scaleway Block storage, and/or on Object Storage (Amazon S3). Policies are applied at table-level during table creation, using the settings presented below. You can choose between three policies for your deployment: - `tiered` (default in Scaleway Data Warehouse for ClickHouse®) - `s3_cache` (cache mode) @@ -19,20 +26,17 @@ The table below provides an overview of each storage policy's behavior and use c | Storage policy | Behavior | When to use | |------|---------|------------| -| `tiered` | Writes land on Block storage first (hot volume). When the local disk reaches ~90% fullness, parts are automatically moved to Object storage (cold volume). This is controlled by `move_factor: 0.1` (move when only ~10% free remains). | General purpose setting for fast local writes and transparent spillover to Object Storage as the dataset grows. | -| `s3_cache` | Data is mainly stored on Object Storage. A local cache layer (on Block storage) keeps frequently-read parts to accelerate repeated reads. | Large datasets that mostly live in Object Storage, with repeated reads on hot subsets that benefit from local caching. | -| `default` | Data is stored on Block storage only. | Small datasets where lowest latency for reads/writes on local disk is desired and capacity fits the PVC. | +| `tiered` | Writes on Block Storage first, and parts are automatically moved to Object Storage when disk is 90% full. | General purpose setting for fast local writes and transparent spillover to Object Storage as the dataset grows. | +| `s3_cache` | Data is stored on Object Storage. A local Block Storage cache layer keeps frequently-read parts to accelerate repeated reads. | Large datasets that mostly live in Object Storage, with repeated reads on smaller subsets that benefit from local caching. | +| `default` | Data is stored on Block storage only. | Small datasets where lowest latency for reads/writes on local disk is desired and capacity fits the Block Storage volume. | - Under the hood, our ClickHouse config defines three disks (default = local PVC, s3 = object storage, s3_cache = S3 + local cache) and three policies (tiered, s3_cache, default). - Cluster default (for new tables when not specified): tiered. - -How clients choose a policy (per table) +## Tiered storage policy -The storage policy is chosen at table creation time via SETTINGS storage_policy = '...'. +### Applying the tiered policy -## Tiered storage policy +**Tiered** is the deployment's default policy. Data is stored on the `hot` volume (Block Storage) until it reaches 90% of its capacity, then data is moved to the `cold` volume. This mechanism is controlled by `move_factor:0.1`. -**Tiered** (Block Storage first, spill to Object Storage) is the cluster default policy. If you omit the `storage_policy` setting; ClickHouse® will automatically apply this policy. +[Connect to you deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to apply the `tiered` storage policy. @@ -71,8 +75,40 @@ The storage policy is chosen at table creation time via SETTINGS storage_policy -S3 with local cache: + +If you omit the `storage_policy` setting, ClickHouse® will automatically apply the `tiered` policy. + + +### Moving data with the tiered policy + +When using the `tiered` storage policy, you can manually move partitions between `hot` (Block Storage) and `cold` (Object Storage) volumes using the SQL queries below. + +- Moving specific partitions to the `cold` volume (Object Storage): + + ```sql + ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'cold'; + ``` + +- Moving a non-partitioned table (`tuple()`) to the `cold` volume (Object Storage): + + ```sql + ALTER TABLE tiered_table_without_partitions MOVE PARTITION tuple() TO VOLUME 'cold'; + ``` + +- Move a partition to the `hot` volume (Block Storage): + + ```sql + ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'hot'; + ``` +- Moving a non-partitioned table (`tuple()`) to the `hot` volume (Block Storage): + ```sql + ALTER TABLE tiered_table_without_partitions MOVE PARTITION tuple() TO VOLUME 'hot'; + ``` + +## S3 cache policy + +[Connect to you deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to create a new table with the `s3_cache` policy enabled. ```sql CREATE TABLE cache_table ( @@ -86,7 +122,9 @@ ORDER BY id SETTINGS storage_policy = 's3_cache'; ``` -Local-only: +## Local-only policy + +[Connect to you deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to create a new table with the `default` (Block Storage only) policy enabled. ```sql CREATE TABLE local_table @@ -101,25 +139,3 @@ ORDER BY id SETTINGS storage_policy = 'default'; ``` -Moving data with the tiered policy - -With tiered, you can manually move partitions between volumes: - -Move specific partitions to cold (S3): - -```sql -ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'cold'; -``` - -For non-partitioned tables (tuple()): - -```sql -ALTER TABLE tiered_table_without_partitions MOVE PARTITION tuple() TO VOLUME 'cold'; -``` - -Move back to hot (local): - -```sql -ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'hot'; -``` - From 8d1d7f7f852b9f9009bafde800c691d36d1c3913 Mon Sep 17 00:00:00 2001 From: SamyOubouaziz Date: Fri, 17 Oct 2025 15:15:13 +0200 Subject: [PATCH 6/6] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Néda <87707325+nerda-codes@users.noreply.github.com> --- pages/data-warehouse/how-to/manage-databases.mdx | 4 ++-- pages/data-warehouse/how-to/manage-storage.mdx | 6 +++--- pages/data-warehouse/how-to/manage-users.mdx | 8 ++++---- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/pages/data-warehouse/how-to/manage-databases.mdx b/pages/data-warehouse/how-to/manage-databases.mdx index b7a2bbe0ef..73d429ec01 100644 --- a/pages/data-warehouse/how-to/manage-databases.mdx +++ b/pages/data-warehouse/how-to/manage-databases.mdx @@ -29,7 +29,7 @@ The Scaleway console allows you to create and delete databases for your Data War 3. Click the **Databases** tab. A list of your current databases displays. -4. Click **+ Create database**. A popup displays. +4. Click **+ Create database**. A pop-up displays. 5. Enter a name for your new database. It can can only contain alphanumeric characters, underscores, and dashes. @@ -47,7 +47,7 @@ The `default` database cannot be deleted. 2. Click the name of the desired Data Warehouse deployment. The **Overview** tab of the deployment displays. -3. Click next to the name of the database you want to delete. A confirmation popup displays. +3. Click next to the name of the database you want to delete. A confirmation pop-up displays. 4. Enter `DELETE`, then click **Delete database** to confirm. diff --git a/pages/data-warehouse/how-to/manage-storage.mdx b/pages/data-warehouse/how-to/manage-storage.mdx index 668fc18fd7..01b7afa0f6 100644 --- a/pages/data-warehouse/how-to/manage-storage.mdx +++ b/pages/data-warehouse/how-to/manage-storage.mdx @@ -95,7 +95,7 @@ When using the `tiered` storage policy, you can manually move partitions between ALTER TABLE tiered_table_without_partitions MOVE PARTITION tuple() TO VOLUME 'cold'; ``` -- Move a partition to the `hot` volume (Block Storage): +- Moving a partition to the `hot` volume (Block Storage): ```sql ALTER TABLE tiered_table MOVE PARTITION '2024-08' TO VOLUME 'hot'; @@ -108,7 +108,7 @@ When using the `tiered` storage policy, you can manually move partitions between ## S3 cache policy -[Connect to you deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to create a new table with the `s3_cache` policy enabled. +[Connect to your deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to create a new table with the `s3_cache` policy enabled. ```sql CREATE TABLE cache_table ( @@ -124,7 +124,7 @@ SETTINGS storage_policy = 's3_cache'; ## Local-only policy -[Connect to you deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to create a new table with the `default` (Block Storage only) policy enabled. +[Connect to your deployment](/data-warehouse/how-to/connect-applications/), then run the SQL query below to create a new table with the `default` (Block Storage only) policy enabled. ```sql CREATE TABLE local_table diff --git a/pages/data-warehouse/how-to/manage-users.mdx b/pages/data-warehouse/how-to/manage-users.mdx index 7cca6c523d..2cc7e64799 100644 --- a/pages/data-warehouse/how-to/manage-users.mdx +++ b/pages/data-warehouse/how-to/manage-users.mdx @@ -29,9 +29,9 @@ The Scaleway console allows you to create users for your Data Warehouse for Clic 3. Click the **Users** tab. A list of your current users displays. -4. Click **+ Create user**. A popup displays. +4. Click **+ Create user**. A pop-up displays. -5. Enter a username. It must be unique can contain up to 63 characters, and must start with a letter. Only alphanumeric characters, underscores, and dashes are accepted. +5. Enter a username. It must be unique, contain up to 63 characters, and must start with a letter. Only alphanumeric characters, underscores, and dashes are accepted. 6. Enter a password, or generate one automatically. @@ -49,7 +49,7 @@ The newly created user appears in the list of your deployment's users. 3. Click the **Users** tab. A list of your current users displays. -4. Click next to the name of the user you want to update, then click ¨**Update user**. A popup displays. +4. Click next to the name of the user you want to update, then click **Update user**. A pop-up displays. 5. Update the user's password and/or admin rights, then click **Update user** to confirm. @@ -61,7 +61,7 @@ The newly created user appears in the list of your deployment's users. 3. Click the **Users** tab. A list of your current users displays. -4. Click next to the name of the user you want to delete. A confirmation popup displays. +4. Click next to the name of the user you want to delete. A confirmation pop-up displays. 5. Enter `DELETE`, then click **Delete user** to confirm.``