diff --git a/docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md b/docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md index 9a2657937b2..e90da00c552 100644 --- a/docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md +++ b/docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md @@ -8,7 +8,7 @@ doc_type: 'guide' --- import Image from '@theme/IdealImage'; -import secure_s3 from '@site/static/images/cloud/security/secures3.jpg'; +import secure_s3 from '@site/static/images/cloud/security/secures3.png'; import s3_info from '@site/static/images/cloud/security/secures3_arn.png'; import s3_output from '@site/static/images/cloud/security/secures3_output.jpg'; @@ -18,7 +18,7 @@ This article demonstrates how ClickHouse Cloud customers can leverage role-based Before diving into the setup for secure S3 access, it is important to understand how this works. Below is an overview of how ClickHouse services can access private S3 buckets by assuming into a role within customers' AWS account. -Overview of Secure S3 Access with ClickHouse +Overview of Secure S3 Access with ClickHouse This approach allows customers to manage all access to their S3 buckets in a single place (the IAM policy of the assumed-role) without having to go through all of their bucket policies to add or remove access. diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/07_OSS_to_Cloud/01_clickhouse-to-cloud.md b/docs/cloud/onboard/02_migrate/01_migration_guides/07_OSS_to_Cloud/01_clickhouse-to-cloud.md index 83bac46556e..9bd863522b0 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/07_OSS_to_Cloud/01_clickhouse-to-cloud.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/07_OSS_to_Cloud/01_clickhouse-to-cloud.md @@ -18,13 +18,13 @@ import self_managed_06 from '@site/static/images/integrations/migration/self-man # Migrating between self-managed ClickHouse and ClickHouse Cloud -Migrating Self-managed ClickHouse +Migrating Self-managed ClickHouse This guide will show how to migrate from a self-managed ClickHouse server to ClickHouse Cloud, and also how to migrate between ClickHouse Cloud services. The [`remoteSecure`](/sql-reference/table-functions/remote) function is used in `SELECT` and `INSERT` queries to allow access to remote ClickHouse servers, which makes migrating tables as simple as writing an `INSERT INTO` query with an embedded `SELECT`. ## Migrating from Self-managed ClickHouse to ClickHouse Cloud {#migrating-from-self-managed-clickhouse-to-clickhouse-cloud} -Migrating Self-managed ClickHouse +Migrating Self-managed ClickHouse :::note Regardless of if your source table is sharded and/or replicated, on ClickHouse Cloud you just create a destination table (you can leave out the Engine parameter for this table, it will be automatically a ReplicatedMergeTree table), @@ -85,7 +85,7 @@ CREATE TABLE db.table ... - Use the `remoteSecure` function to pull the data from the self-managed source -Migrating Self-managed ClickHouse +Migrating Self-managed ClickHouse ```sql INSERT INTO db.table SELECT * FROM @@ -98,7 +98,7 @@ If the source system is not available from outside networks then you can push th - Use the `remoteSecure` function to push the data to the ClickHouse Cloud service -Migrating Self-managed ClickHouse +Migrating Self-managed ClickHouse :::tip Add the remote system to your ClickHouse Cloud service IP Access List In order for the `remoteSecure` function to connect to your ClickHouse Cloud service the IP Address of the remote system will need to be allowed by the IP Access List. Expand **Manage your IP Access List** below this tip for more information. @@ -114,7 +114,7 @@ remoteSecure('HOSTNAME.clickhouse.cloud:9440', 'db.table', ## Migrating between ClickHouse Cloud services {#migrating-between-clickhouse-cloud-services} -Migrating Self-managed ClickHouse +Migrating Self-managed ClickHouse Some example uses for migrating data between ClickHouse Cloud services: - Migrating data from a restored backup @@ -122,7 +122,7 @@ Some example uses for migrating data between ClickHouse Cloud services: In this example there are two ClickHouse Cloud services, and they will be referred to as *source* and *destination*. The data will be pulled from the source to the destination. Although you could push if you like, pulling is shown as it uses a read-only user. -Migrating Self-managed ClickHouse +Migrating Self-managed ClickHouse There are a few steps in the migration: 1. Identify one ClickHouse Cloud service to be the *source*, and the other as the *destination* diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/01_clickhouse-local-etl.md b/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/01_clickhouse-local-etl.md index 7488206c5ae..0fd66a80504 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/01_clickhouse-local-etl.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/01_clickhouse-local-etl.md @@ -19,7 +19,7 @@ import ch_local_04 from '@site/static/images/integrations/migration/ch-local-04. # Migrating to ClickHouse using clickhouse-local -Migrating Self-managed ClickHouse +Migrating Self-managed ClickHouse You can use ClickHouse, or to be more specific,[`clickhouse-local`](/operations/utilities/clickhouse-local.md) as an ETL tool for migrating data from your current database system to ClickHouse Cloud, as long as for your current database system there is either a @@ -33,7 +33,7 @@ For all other popular database systems, there is JDBC driver or ODBC driver avai ## What is clickhouse-local? {#what-is-clickhouse-local} -Migrating Self-managed ClickHouse +Migrating Self-managed ClickHouse Typically, ClickHouse is run in the form of a cluster, where several instances of the ClickHouse database engine are running in a distributed fashion on different servers. @@ -92,7 +92,7 @@ In order for the `remoteSecure` function to connect to your ClickHouse Cloud ser We will use the [integration table engine](/engines/table-engines/integrations/mysql/) (created on-the-fly by the [mysql table function](/sql-reference/table-functions/mysql/)) for reading data from the source MySQL database and we will use the [remoteSecure table function](/sql-reference/table-functions/remote/) for writing the data into a destination table on your ClickHouse cloud service. -Migrating Self-managed ClickHouse +Migrating Self-managed ClickHouse ### On the destination ClickHouse Cloud service: {#on-the-destination-clickhouse-cloud-service} @@ -133,7 +133,7 @@ No data is stored locally on the `clickhouse-local` host machine. Instead, the d We will use the [JDBC integration table engine](/engines/table-engines/integrations/jdbc.md) (created on-the-fly by the [jdbc table function](/sql-reference/table-functions/jdbc.md)) together with the [ClickHouse JDBC Bridge](https://github.com/ClickHouse/clickhouse-jdbc-bridge) and the MySQL JDBC driver for reading data from the source MySQL database and we will use the [remoteSecure table function](/sql-reference/table-functions/remote.md) for writing the data into a destination table on your ClickHouse cloud service. -Migrating Self-managed ClickHouse +Migrating Self-managed ClickHouse ### On the destination ClickHouse Cloud service: {#on-the-destination-clickhouse-cloud-service-1} diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/02_etl-tool-to-clickhouse.md b/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/02_etl-tool-to-clickhouse.md index 6f33cf36666..a6b932f01b1 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/02_etl-tool-to-clickhouse.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/02_etl-tool-to-clickhouse.md @@ -3,17 +3,13 @@ sidebar_label: 'Using a third-party ETL tool' keywords: ['clickhouse', 'migrate', 'migration', 'migrating', 'data', 'etl', 'elt', 'clickhouse-local', 'clickhouse-client'] slug: /cloud/migration/etl-tool-to-clickhouse title: 'Using a third-party ETL Tool' -description: 'Page describing how to use a 3rd-party ETL tool with ClickHouse' +description: 'Page describing how to use a third-party ETL tool with ClickHouse' doc_type: 'guide' --- import Image from '@theme/IdealImage'; import third_party_01 from '@site/static/images/integrations/migration/third-party-01.png'; -# Using a 3rd-party ETL Tool - -Migrating Self-managed ClickHouse - A great option for moving data from an external data source into ClickHouse is to use one of the many popular ETL and ELT. We have docs that cover the following: - [Airbyte](/integrations/data-ingestion/etl-tools/airbyte-and-clickhouse.md) @@ -21,3 +17,5 @@ A great option for moving data from an external data source into ClickHouse is t - [Vector](/integrations/data-ingestion/etl-tools/vector-to-clickhouse.md) But there are many other ETL/ELT tools that integrate with ClickHouse, so check your favorite tool's documentation for details. + +Migrating Self-managed ClickHouse diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/03_object-storage-to-clickhouse.md b/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/03_object-storage-to-clickhouse.md index e647ecd8668..3cb2be34f85 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/03_object-storage-to-clickhouse.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/08_other_methods/03_object-storage-to-clickhouse.md @@ -11,7 +11,7 @@ import object_storage_01 from '@site/static/images/integrations/migration/object # Move data from cloud object storage to ClickHouse Cloud -Migrating Self-managed ClickHouse +Migrating Self-managed ClickHouse If you use a Cloud Object Storage as a data lake and wish to import this data into ClickHouse Cloud, or if your current database system is able to directly offload data into a Cloud Object Storage, then you can use one of the diff --git a/docs/integrations/data-ingestion/redshift/_snippets/_migration_guide.md b/docs/integrations/data-ingestion/redshift/_snippets/_migration_guide.md index 960120aa751..fa7291e6e87 100644 --- a/docs/integrations/data-ingestion/redshift/_snippets/_migration_guide.md +++ b/docs/integrations/data-ingestion/redshift/_snippets/_migration_guide.md @@ -10,7 +10,7 @@ import Image from '@theme/IdealImage'; [Amazon Redshift](https://aws.amazon.com/redshift/) is a popular cloud data warehousing solution that is part of the Amazon Web Services offerings. This guide presents different approaches to migrating data from a Redshift instance to ClickHouse. We will cover three options: -Redshift to ClickHouse Migration Options +Redshift to ClickHouse Migration Options From the ClickHouse instance standpoint, you can either: @@ -28,7 +28,7 @@ We used Redshift as a data source in this tutorial. However, the migration appro In the push scenario, the idea is to leverage a third-party tool or service (either custom code or an [ETL/ELT](https://en.wikipedia.org/wiki/Extract,_transform,_load#ETL_vs._ELT)) to send your data to your ClickHouse instance. For example, you can use a software like [Airbyte](https://www.airbyte.com/) to move data between your Redshift instance (as a source) and ClickHouse as a destination ([see our integration guide for Airbyte](/integrations/data-ingestion/etl-tools/airbyte-and-clickhouse.md)) -PUSH Redshift to ClickHouse +PUSH Redshift to ClickHouse ### Pros {#pros} @@ -45,7 +45,7 @@ In the push scenario, the idea is to leverage a third-party tool or service (eit In the pull scenario, the idea is to leverage the ClickHouse JDBC Bridge to connect to a Redshift cluster directly from a ClickHouse instance and perform `INSERT INTO ... SELECT` queries: -PULL from Redshift to ClickHouse +PULL from Redshift to ClickHouse ### Pros {#pros-1} @@ -177,7 +177,7 @@ Ok. In this scenario, we export data to S3 in an intermediary pivot format and, in a second step, load the data from S3 into ClickHouse. -PIVOT from Redshift using S3 +PIVOT from Redshift using S3 ### Pros {#pros-2} diff --git a/static/images/cloud/security/cloud-access-management/user_grant_permissions_options.png b/static/images/cloud/security/cloud-access-management/user_grant_permissions_options.png index 88c4f76622a..4be39c65a30 100644 Binary files a/static/images/cloud/security/cloud-access-management/user_grant_permissions_options.png and b/static/images/cloud/security/cloud-access-management/user_grant_permissions_options.png differ diff --git a/static/images/cloud/security/secures3.jpg b/static/images/cloud/security/secures3.jpg index 55311413e44..c7669178184 100644 Binary files a/static/images/cloud/security/secures3.jpg and b/static/images/cloud/security/secures3.jpg differ diff --git a/static/images/cloud/security/secures3.png b/static/images/cloud/security/secures3.png new file mode 100644 index 00000000000..fb3213c5d2e Binary files /dev/null and b/static/images/cloud/security/secures3.png differ diff --git a/static/images/integrations/data-ingestion/redshift/pivot.png b/static/images/integrations/data-ingestion/redshift/pivot.png index 6169d9ebfc1..b894d0820c9 100644 Binary files a/static/images/integrations/data-ingestion/redshift/pivot.png and b/static/images/integrations/data-ingestion/redshift/pivot.png differ diff --git a/static/images/integrations/data-ingestion/redshift/pull.png b/static/images/integrations/data-ingestion/redshift/pull.png index 616eb570643..dbb29392f16 100644 Binary files a/static/images/integrations/data-ingestion/redshift/pull.png and b/static/images/integrations/data-ingestion/redshift/pull.png differ diff --git a/static/images/integrations/data-ingestion/redshift/push.png b/static/images/integrations/data-ingestion/redshift/push.png index c3f375f0e93..66acf93d083 100644 Binary files a/static/images/integrations/data-ingestion/redshift/push.png and b/static/images/integrations/data-ingestion/redshift/push.png differ diff --git a/static/images/integrations/data-ingestion/redshift/redshift-to-clickhouse.png b/static/images/integrations/data-ingestion/redshift/redshift-to-clickhouse.png index 3e062438fb7..e78900d9d5d 100644 Binary files a/static/images/integrations/data-ingestion/redshift/redshift-to-clickhouse.png and b/static/images/integrations/data-ingestion/redshift/redshift-to-clickhouse.png differ diff --git a/static/images/integrations/migration/ch-local-01.png b/static/images/integrations/migration/ch-local-01.png index cec2d481dee..0f87ad4986f 100644 Binary files a/static/images/integrations/migration/ch-local-01.png and b/static/images/integrations/migration/ch-local-01.png differ diff --git a/static/images/integrations/migration/ch-local-02.png b/static/images/integrations/migration/ch-local-02.png index e95cae0916d..2411a295809 100644 Binary files a/static/images/integrations/migration/ch-local-02.png and b/static/images/integrations/migration/ch-local-02.png differ diff --git a/static/images/integrations/migration/ch-local-03.png b/static/images/integrations/migration/ch-local-03.png index 6525eccd67b..dbe728f4da4 100644 Binary files a/static/images/integrations/migration/ch-local-03.png and b/static/images/integrations/migration/ch-local-03.png differ diff --git a/static/images/integrations/migration/ch-local-04.png b/static/images/integrations/migration/ch-local-04.png index bab1739aee7..5effef4329b 100644 Binary files a/static/images/integrations/migration/ch-local-04.png and b/static/images/integrations/migration/ch-local-04.png differ diff --git a/static/images/integrations/migration/object-storage-01.png b/static/images/integrations/migration/object-storage-01.png index 6f6108ac334..d5102b49b32 100644 Binary files a/static/images/integrations/migration/object-storage-01.png and b/static/images/integrations/migration/object-storage-01.png differ diff --git a/static/images/integrations/migration/self-managed-01.png b/static/images/integrations/migration/self-managed-01.png index 8c3d1253fad..c64d52087c1 100644 Binary files a/static/images/integrations/migration/self-managed-01.png and b/static/images/integrations/migration/self-managed-01.png differ diff --git a/static/images/integrations/migration/self-managed-02.png b/static/images/integrations/migration/self-managed-02.png index 473d131bc3a..edd103d87db 100644 Binary files a/static/images/integrations/migration/self-managed-02.png and b/static/images/integrations/migration/self-managed-02.png differ diff --git a/static/images/integrations/migration/self-managed-03.png b/static/images/integrations/migration/self-managed-03.png index 8dd838958c4..090a715b835 100644 Binary files a/static/images/integrations/migration/self-managed-03.png and b/static/images/integrations/migration/self-managed-03.png differ diff --git a/static/images/integrations/migration/self-managed-04.png b/static/images/integrations/migration/self-managed-04.png index 8e0e740a280..cf74d7e0e51 100644 Binary files a/static/images/integrations/migration/self-managed-04.png and b/static/images/integrations/migration/self-managed-04.png differ diff --git a/static/images/integrations/migration/self-managed-05.png b/static/images/integrations/migration/self-managed-05.png index 651bc5993da..a8eb792db21 100644 Binary files a/static/images/integrations/migration/self-managed-05.png and b/static/images/integrations/migration/self-managed-05.png differ diff --git a/static/images/integrations/migration/self-managed-06.png b/static/images/integrations/migration/self-managed-06.png index b0c718157d1..d1b86cda708 100644 Binary files a/static/images/integrations/migration/self-managed-06.png and b/static/images/integrations/migration/self-managed-06.png differ diff --git a/static/images/integrations/migration/third-party-01.png b/static/images/integrations/migration/third-party-01.png index 420c980c9f8..a7c7ee07fcd 100644 Binary files a/static/images/integrations/migration/third-party-01.png and b/static/images/integrations/migration/third-party-01.png differ