diff --git a/TOC-tidb-cloud-essential.md b/TOC-tidb-cloud-essential.md index 94c83ebef5dad..8f62ad115ab71 100644 --- a/TOC-tidb-cloud-essential.md +++ b/TOC-tidb-cloud-essential.md @@ -226,7 +226,7 @@ - [Import Snapshot Files from Cloud Storage](/tidb-cloud/import-snapshot-files-serverless.md) - [Import with MySQL CLI](/tidb-cloud/import-with-mysql-cli-serverless.md) - Reference - - [Configure External Storage Access for TiDB Cloud](/tidb-cloud/serverless-external-storage.md) + - [Configure External Storage Access for TiDB Cloud](/tidb-cloud/configure-external-storage-access.md) - [Naming Conventions for Data Import](/tidb-cloud/naming-conventions-for-data-import.md) - [CSV Configurations for Importing Data](/tidb-cloud/csv-config-for-import-data.md) - [Troubleshoot Access Denied Errors during Data Import from Amazon S3](/tidb-cloud/troubleshoot-import-access-denied-error.md) diff --git a/TOC-tidb-cloud-premium.md b/TOC-tidb-cloud-premium.md index a2a4cde227b96..3bad77a9a6378 100644 --- a/TOC-tidb-cloud-premium.md +++ b/TOC-tidb-cloud-premium.md @@ -216,7 +216,7 @@ - [Import Snapshot Files from Cloud Storage](/tidb-cloud/import-snapshot-files-serverless.md) - [Import Data Using MySQL CLI](/tidb-cloud/premium/import-with-mysql-cli-premium.md) - Reference - - [Configure External Storage Access for TiDB Cloud](/tidb-cloud/serverless-external-storage.md) + - [Configure External Storage Access for TiDB Cloud](/tidb-cloud/configure-external-storage-access.md) - [Naming Conventions for Data Import](/tidb-cloud/naming-conventions-for-data-import.md) - [CSV Configurations for Importing Data](/tidb-cloud/csv-config-for-import-data.md) - [Troubleshoot Access Denied Errors during Data Import from Amazon S3](/tidb-cloud/troubleshoot-import-access-denied-error.md) diff --git a/TOC-tidb-cloud-starter.md b/TOC-tidb-cloud-starter.md index e3ecea449267a..5ce9245e2e71c 100644 --- a/TOC-tidb-cloud-starter.md +++ b/TOC-tidb-cloud-starter.md @@ -229,7 +229,7 @@ - [Import Snapshot Files from Cloud Storage](/tidb-cloud/import-snapshot-files-serverless.md) - [Import with MySQL CLI](/tidb-cloud/import-with-mysql-cli-serverless.md) - Reference - - [Configure External Storage Access for TiDB Cloud](/tidb-cloud/serverless-external-storage.md) + - [Configure External Storage Access for TiDB Cloud](/tidb-cloud/configure-external-storage-access.md) - [Naming Conventions for Data Import](/tidb-cloud/naming-conventions-for-data-import.md) - [CSV Configurations for Importing Data](/tidb-cloud/csv-config-for-import-data.md) - [Troubleshoot Access Denied Errors during Data Import from Amazon S3](/tidb-cloud/troubleshoot-import-access-denied-error.md) diff --git a/tidb-cloud/serverless-external-storage.md b/tidb-cloud/configure-external-storage-access.md similarity index 78% rename from tidb-cloud/serverless-external-storage.md rename to tidb-cloud/configure-external-storage-access.md index 6666b73e83cb4..413b8e36dec06 100644 --- a/tidb-cloud/serverless-external-storage.md +++ b/tidb-cloud/configure-external-storage-access.md @@ -1,17 +1,28 @@ --- -title: Configure External Storage Access for {{{ .starter }}} or Essential -summary: Learn how to configure Amazon Simple Storage Service (Amazon S3) access. +title: Configure External Storage Access +summary: Learn how to configure cross-account access to an external storage such as Amazon Simple Storage Service (Amazon S3). +aliases: ['/tidbcloud/serverless-external-storage'] --- -# Configure External Storage Access for {{{ .starter }}} or Essential +# Configure External Storage Access + + If you want to import data from or export data to an external storage in a TiDB Cloud cluster, you need to configure cross-account access. This document describes how to configure access to an external storage for {{{ .starter }}} and {{{ .essential }}} clusters. + + + + +If you want to import data from or export data to an external storage in a TiDB Cloud instance, you need to configure cross-account access. This document describes how to configure access to an external storage for {{{ .premium }}} instances. + + + If you need to configure these external storages for a TiDB Cloud Dedicated cluster, see [Configure External Storage Access for TiDB Cloud Dedicated](/tidb-cloud/dedicated-external-storage.md). ## Configure Amazon S3 access -To allow a TiDB Cloud cluster to access the source data in your Amazon S3 bucket, configure the bucket access for the cluster using either of the following methods: +To allow a TiDB Cloud clusterinstance to access the source data in your Amazon S3 bucket, configure the bucket access for the clusterinstance using either of the following methods: - [Use a Role ARN](#configure-amazon-s3-access-using-a-role-arn): use a Role ARN to access your Amazon S3 bucket. - [Use an AWS access key](#configure-amazon-s3-access-using-an-aws-access-key): use the access key of an IAM user to access your Amazon S3 bucket. @@ -22,13 +33,13 @@ It is recommended that you use [AWS CloudFormation](https://docs.aws.amazon.com/ > **Note:** > -> Role ARN access to Amazon S3 is only supported for clusters with AWS as the cloud provider. If you use a different cloud provider, use an AWS access key instead. For more information, see [Configure Amazon S3 access using an AWS access key](#configure-amazon-s3-access-using-an-aws-access-key). +> Role ARN access to Amazon S3 is only supported for clustersinstances with AWS as the cloud provider. If you use a different cloud provider, use an AWS access key instead. For more information, see [Configure Amazon S3 access using an AWS access key](#configure-amazon-s3-access-using-an-aws-access-key). -1. Open the **Import** page for your target cluster. +1. Open the **Import** page for your target clusterinstance. - 1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/project/clusters) page of your project. + 1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/project/clusters) page of your project.navigate to the [**TiDB Instances**](https://tidbcloud.com/tidbs) page. - 2. Click the name of your target cluster to go to its overview page, and then click **Data** > **Import** in the left navigation pane. + 2. Click the name of your target clusterinstance to go to its overview page, and then click **Data** > **Import** in the left navigation pane. 2. Open the **Add New ARN** dialog. @@ -40,7 +51,7 @@ It is recommended that you use [AWS CloudFormation](https://docs.aws.amazon.com/ - If you want to export data to Amazon S3, open the **Add New ARN** dialog as follows: - 1. Click **Export data to...** > **Amazon S3**. If your cluster has neither imported nor exported any data before, click **Click here to export data to...** > **Amazon S3** at the bottom of the page. + 1. Click **Export data to...** > **Amazon S3**. If your clusterinstance has neither imported nor exported any data before, click **Click here to export data to...** > **Amazon S3** at the bottom of the page. 2. Fill in the **Folder URI** field. 3. Choose **AWS Role ARN** and click **Click here to create new one with AWS CloudFormation**. @@ -56,7 +67,7 @@ It is recommended that you use [AWS CloudFormation](https://docs.aws.amazon.com/ 5. After the CloudFormation stack is executed, you can click the **Outputs** tab and find the Role ARN value in the **Value** column. - ![img.png](/media/tidb-cloud/serverless-external-storage/serverless-role-arn.png) + ![Role ARN](/media/tidb-cloud/serverless-external-storage/serverless-role-arn.png) If you have any trouble creating a role ARN with AWS CloudFormation, you can take the following steps to create one manually: @@ -79,10 +90,10 @@ If you have any trouble creating a role ARN with AWS CloudFormation, you can tak 4. On the **Create policy** page, click the **JSON** tab. - 5. Configure the policy in the policy text field according to your needs. The following is an example that you can use to export data from and import data into a TiDB Cloud cluster. + 5. Configure the policy in the policy text field according to your needs. The following is an example that you can use to export data from and import data into a TiDB Cloud clusterinstance. - - Exporting data from a TiDB Cloud cluster needs the **s3:PutObject** and **s3:ListBucket** permissions. - - Importing data into a TiDB Cloud cluster needs the **s3:GetObject**, **s3:GetObjectVersion**, and **s3:ListBucket** permissions. + - Exporting data from a TiDB Cloud clusterinstance needs the **s3:PutObject** and **s3:ListBucket** permissions. + - Importing data into a TiDB Cloud clusterinstance needs the **s3:GetObject**, **s3:GetObjectVersion**, and **s3:ListBucket** permissions. ```json { @@ -150,7 +161,7 @@ If you have any trouble creating a role ARN with AWS CloudFormation, you can tak - In **Trusted entity type**, select **AWS account**. - In **An AWS account**, select **Another AWS account**, and then paste the TiDB Cloud account ID to the **Account ID** field. - - In **Options**, click **Require external ID (Best practice when a third party will assume this role)**, and then paste the TiDB Cloud External ID to the **External ID** field. If the role is created without a Require external ID, once the configuration is done for one TiDB cluster in a project, all TiDB clusters in that project can use the same Role ARN to access your Amazon S3 bucket. If the role is created with the account ID and external ID, only the corresponding TiDB cluster can access the bucket. + - In **Options**, click **Require external ID (Best practice when a third party will assume this role)**, and then paste the TiDB Cloud External ID to the **External ID** field. If the role is created without a Require external ID, once the configuration is done for one TiDB cluster in a project, all TiDB clusters in that project can use the same Role ARN to access your Amazon S3 bucket. If the role is created with the account ID and external ID, only the corresponding TiDB cluster can access the bucket. 3. Click **Next** to open the policy list, choose the policy you just created, and then click **Next**. @@ -178,6 +189,8 @@ Take the following steps to configure an access key: > > TiDB Cloud does not store your access keys. It is recommended that you [delete the access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) after the import or export is complete. + + ## Configure GCS access To allow a TiDB Cloud cluster to access your GCS bucket, you need to configure the GCS access for the bucket. You can use a service account key to configure the bucket access: @@ -206,11 +219,15 @@ Take the following steps to configure a service account key: 3. Choose the default `JSON` key type, and then click **CREATE** to download the Google Cloud credentials file. The file contains the service account key that you need to use when configuring the GCS access for the TiDB Cloud cluster. + + + + ## Configure Azure Blob Storage access To allow TiDB Cloud to access your Azure Blob container, you need to create a service SAS token for the container. -You can create a SAS token either using an [Azure ARM template](https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/overview) (recommended) or manual configuration. +You can create a SAS token either using an [Azure ARM template](https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/overview) (recommended) or manual configuration. To create a SAS token using an Azure ARM template, take the following steps: @@ -221,15 +238,15 @@ To create a SAS token using an Azure ARM template, take the following steps: 2. Click the name of your target cluster to go to its overview page, and then click **Data** > **Import** in the left navigation pane. 2. Open the **Generate New SAS Token via ARM Template Deployment** dialog. - + 1. Click **Export data to...** > **Azure Blob Storage**. If your cluster has neither imported nor exported any data before, click **Click here to export data to...** > **Azure Blob Storage** at the bottom of the page. - - 2. Scroll down to the **Azure Blob Storage Settings** area, and then click **Click here to create a new one with Azure ARM template** under the SAS Token field. - + + 2. Scroll down to the **Azure Blob Storage Settings** area, and then click **Click here to create a new one with Azure ARM template** under the SAS Token field. + 3. Create a SAS token with the Azure ARM template. 1. In the **Generate New SAS Token via ARM Template Deployment** dialog, click **Click to open the Azure Portal with the pre-configured ARM template**. - + 2. After logging in to Azure, you will be redirected to the Azure **Custom deployment** page. 3. Fill in the **Resource group** and **Storage Account Name** in the **Custom deployment** page. You can get all the information from the storage account overview page where the container is located. @@ -237,7 +254,7 @@ To create a SAS token using an Azure ARM template, take the following steps: ![azure-storage-account-overview](/media/tidb-cloud/serverless-external-storage/azure-storage-account-overview.png) 4. Click **Review + create** or **Next** to review the deployment. Click **Create** to start the deployment. - + 5. After it completes, you will be redirected to the deployment overview page. Navigate to the **Outputs** section to get the SAS token. If you have any trouble creating a SAS token with the Azure ARM template, take the following steps to create one manually: @@ -246,7 +263,7 @@ If you have any trouble creating a SAS token with the Azure ARM template, take t Click here to see details 1. On the [Azure Storage account](https://portal.azure.com/#browse/Microsoft.Storage%2FStorageAccounts) page, click your storage account to which the container belongs. - + 2. On your **Storage account** page, click the **Security+network**, and then click **Shared access signature**. ![sas-position](/media/tidb-cloud/serverless-external-storage/azure-sas-position.png) @@ -269,6 +286,8 @@ If you have any trouble creating a SAS token with the Azure ARM template, take t + + ## Configure Alibaba Cloud Object Storage Service (OSS) access To allow TiDB Cloud to access your Alibaba Cloud OSS bucket, you need to create an AccessKey pair for the bucket. @@ -276,19 +295,19 @@ To allow TiDB Cloud to access your Alibaba Cloud OSS bucket, you need to create Take the following steps to configure an AccessKey pair: 1. Create a RAM user and get the AccessKey pair. For more information, see [Create a RAM user](https://www.alibabacloud.com/help/en/ram/user-guide/create-a-ram-user). - + In the **Access Mode** section, select **Using permanent AccessKey to access**. 2. Create a custom policy with the required permissions. For more information, see [Create custom policies](https://www.alibabacloud.com/help/en/ram/user-guide/create-a-custom-policy). - + - In the **Effect** section, select **Allow**. - In the **Service** section, select **Object Storage Service**. - In the **Action** section, select the permissions as needed. - - To import data into a TiDB Cloud cluster, grant **oss:GetObject**, **oss:GetBucketInfo**, and **oss:ListObjects** permissions. - To export data from a TiDB Cloud cluster, grant **oss:PutObject**, **oss:GetBucketInfo**, and **oss:ListBuckets** permissions. - + To import data into a TiDB Cloud clusterinstance, grant **oss:GetObject**, **oss:GetBucketInfo**, and **oss:ListObjects** permissions. + + To export data from a TiDB Cloud clusterinstance, grant **oss:PutObject**, **oss:GetBucketInfo**, and **oss:ListBuckets** permissions. + - In the **Resource** section, select the bucket and the objects in the bucket. 3. Attach the custom policies to the RAM user. For more information, see [Grant permissions to a RAM user](https://www.alibabacloud.com/help/en/ram/user-guide/grant-permissions-to-the-ram-user). diff --git a/tidb-cloud/dedicated-external-storage.md b/tidb-cloud/dedicated-external-storage.md index cc2bf955e0271..abcf80e4492b0 100644 --- a/tidb-cloud/dedicated-external-storage.md +++ b/tidb-cloud/dedicated-external-storage.md @@ -8,7 +8,7 @@ aliases: ['/tidb-cloud/config-s3-and-gcs-access'] If your source data is stored in Amazon S3 buckets, Azure Blob Storage containers, or Google Cloud Storage (GCS) buckets, before importing or migrating the data to TiDB Cloud, you need to configure cross-account access to the buckets. This document describes how to do this for TiDB Cloud Dedicated clusters. -If you need to configure these external storages for {{{ .starter }}} or {{{ .essential }}} clusters, see [Configure External Storage Access for {{{ .starter }}} or Essential](/tidb-cloud/serverless-external-storage.md). +If you need to configure these external storages for {{{ .starter }}} or {{{ .essential }}} clusters, see [Configure External Storage Access for {{{ .starter }}} or Essential](/tidb-cloud/configure-external-storage-access.md). ## Configure Amazon S3 access diff --git a/tidb-cloud/essential-database-audit-logging.md b/tidb-cloud/essential-database-audit-logging.md index f48132c3c4572..88be732f5063e 100644 --- a/tidb-cloud/essential-database-audit-logging.md +++ b/tidb-cloud/essential-database-audit-logging.md @@ -62,7 +62,7 @@ To store audit logs in Amazon S3, you need to provide the following information: - An [access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) with the `s3:PutObject` permission. - A [role ARN](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html) with the `s3:PutObject` permission. Only clusters hosted on AWS support using a role ARN. -For more information, see [Configure Amazon S3 access](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access). +For more information, see [Configure Amazon S3 access](/tidb-cloud/configure-external-storage-access.md#configure-amazon-s3-access). ### Google Cloud Storage @@ -71,7 +71,7 @@ To store audit logs in Google Cloud Storage, you need to provide the following i - URI: `gs:////` - Access credential: a [service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) with the `storage.objects.create` and `storage.objects.delete` permissions. -For more information, see [Configure GCS access](/tidb-cloud/serverless-external-storage.md#configure-gcs-access). +For more information, see [Configure GCS access](/tidb-cloud/configure-external-storage-access.md#configure-gcs-access). ### Azure Blob Storage @@ -80,7 +80,7 @@ To store audit logs in Azure Blob Storage, you need to provide the following inf - URI: `azure://.blob.core.windows.net///` or `https://.blob.core.windows.net///` - Access credential: a [shared access signature (SAS) token](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview) with `Read` and `Write` permissions on the `Container` and `Object` resources. -For more information, see [Configure Azure Blob Storage access](/tidb-cloud/serverless-external-storage.md#configure-azure-blob-storage-access). +For more information, see [Configure Azure Blob Storage access](/tidb-cloud/configure-external-storage-access.md#configure-azure-blob-storage-access). ### Alibaba Cloud OSS @@ -89,7 +89,7 @@ To store audit logs in Alibaba Cloud OSS, you need to provide the following info - URI: `oss:////` - Access credential: an [AccessKey pair](https://www.alibabacloud.com/help/en/ram/user-guide/create-an-accesskey-pair) with the `oss:PutObject` and `oss:GetBucketInfo` permissions to allow data export to the OSS bucket. -For more information, see [Configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/serverless-external-storage.md#configure-alibaba-cloud-object-storage-service-oss-access). +For more information, see [Configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/configure-external-storage-access.md#configure-alibaba-cloud-object-storage-service-oss-access). ## Audit logging filter rules diff --git a/tidb-cloud/import-csv-files-serverless.md b/tidb-cloud/import-csv-files-serverless.md index 0a8c4f0212dcd..80df511c07642 100644 --- a/tidb-cloud/import-csv-files-serverless.md +++ b/tidb-cloud/import-csv-files-serverless.md @@ -76,15 +76,15 @@ Because CSV files do not contain schema information, before importing data from To allow TiDB Cloud to access the CSV files in the Amazon S3, GCS, Azure Blob Storage, or Alibaba Cloud Object Storage Service bucket, do one of the following: -- If your CSV files are located in Amazon S3, [configure Amazon S3 access](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access) for your cluster. +- If your CSV files are located in Amazon S3, [configure Amazon S3 access](/tidb-cloud/configure-external-storage-access.md#configure-amazon-s3-access) for your cluster. You can use either an AWS access key or a Role ARN to access your bucket. Once finished, make a note of the access key (including the access key ID and secret access key) or the Role ARN value as you will need it in [Step 4](#step-4-import-csv-files). -- If your CSV files are located in GCS, [configure GCS access](/tidb-cloud/serverless-external-storage.md#configure-gcs-access) for your cluster. +- If your CSV files are located in GCS, [configure GCS access](/tidb-cloud/configure-external-storage-access.md#configure-gcs-access) for your cluster. -- If your CSV files are located in Azure Blob Storage, [configure Azure Blob Storage access](/tidb-cloud/serverless-external-storage.md#configure-azure-blob-storage-access) for your cluster. +- If your CSV files are located in Azure Blob Storage, [configure Azure Blob Storage access](/tidb-cloud/configure-external-storage-access.md#configure-azure-blob-storage-access) for your cluster. -- If your CSV files are located in Alibaba Cloud Object Storage Service (OSS), [configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/serverless-external-storage.md#configure-alibaba-cloud-object-storage-service-oss-access) for your cluster. +- If your CSV files are located in Alibaba Cloud Object Storage Service (OSS), [configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/configure-external-storage-access.md#configure-alibaba-cloud-object-storage-service-oss-access) for your cluster. ## Step 4. Import CSV files @@ -111,7 +111,7 @@ To import the CSV files to {{{ .starter }}} or {{{ .essential }}}, take the foll - **Source Files URI**: - When importing one file, enter the source file URI in the following format `s3://[bucket_name]/[data_source_folder]/[file_name].csv`. For example, `s3://sampledata/ingest/TableName.01.csv`. - When importing multiple files, enter the source folder URI in the following format `s3://[bucket_name]/[data_source_folder]/`. For example, `s3://sampledata/ingest/`. - - **Credential**: you can use either an AWS Role ARN or an AWS access key to access your bucket. For more information, see [Configure Amazon S3 access](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access). + - **Credential**: you can use either an AWS Role ARN or an AWS access key to access your bucket. For more information, see [Configure Amazon S3 access](/tidb-cloud/configure-external-storage-access.md#configure-amazon-s3-access). - **AWS Role ARN**: enter the AWS Role ARN value. - **AWS Access Key**: enter the AWS access key ID and AWS secret access key. @@ -164,7 +164,7 @@ To import the CSV files to {{{ .starter }}} or {{{ .essential }}}, take the foll - **Source Files URI**: - When importing one file, enter the source file URI in the following format `[gcs|gs]://[bucket_name]/[data_source_folder]/[file_name].csv`. For example, `[gcs|gs]://sampledata/ingest/TableName.01.csv`. - When importing multiple files, enter the source folder URI in the following format `[gcs|gs]://[bucket_name]/[data_source_folder]/`. For example, `[gcs|gs]://sampledata/ingest/`. - - **Credential**: you can use a GCS IAM Role Service Account key to access your bucket. For more information, see [Configure GCS access](/tidb-cloud/serverless-external-storage.md#configure-gcs-access). + - **Credential**: you can use a GCS IAM Role Service Account key to access your bucket. For more information, see [Configure GCS access](/tidb-cloud/configure-external-storage-access.md#configure-gcs-access). 4. Click **Next**. @@ -215,7 +215,7 @@ To import the CSV files to {{{ .starter }}} or {{{ .essential }}}, take the foll - **Source Files URI**: - When importing one file, enter the source file URI in the following format `[azure|https]://[bucket_name]/[data_source_folder]/[file_name].csv`. For example, `[azure|https]://sampledata/ingest/TableName.01.csv`. - When importing multiple files, enter the source folder URI in the following format `[azure|https]://[bucket_name]/[data_source_folder]/`. For example, `[azure|https]://sampledata/ingest/`. - - **Credential**: you can use a shared access signature (SAS) token to access your bucket. For more information, see [Configure Azure Blob Storage access](/tidb-cloud/serverless-external-storage.md#configure-azure-blob-storage-access). + - **Credential**: you can use a shared access signature (SAS) token to access your bucket. For more information, see [Configure Azure Blob Storage access](/tidb-cloud/configure-external-storage-access.md#configure-azure-blob-storage-access). 4. Click **Next**. @@ -266,7 +266,7 @@ To import the CSV files to {{{ .starter }}} or {{{ .essential }}}, take the foll - **Source Files URI**: - When importing one file, enter the source file URI in the following format `oss://[bucket_name]/[data_source_folder]/[file_name].csv`. For example, `oss://sampledata/ingest/TableName.01.csv`. - When importing multiple files, enter the source folder URI in the following format `oss://[bucket_name]/[data_source_folder]/`. For example, `oss://sampledata/ingest/`. - - **Credential**: you can use an AccessKey pair to access your bucket. For more information, see [Configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/serverless-external-storage.md#configure-alibaba-cloud-object-storage-service-oss-access). + - **Credential**: you can use an AccessKey pair to access your bucket. For more information, see [Configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/configure-external-storage-access.md#configure-alibaba-cloud-object-storage-service-oss-access). 4. Click **Next**. diff --git a/tidb-cloud/import-parquet-files-serverless.md b/tidb-cloud/import-parquet-files-serverless.md index c7bbab4895568..3068b4e50e9bc 100644 --- a/tidb-cloud/import-parquet-files-serverless.md +++ b/tidb-cloud/import-parquet-files-serverless.md @@ -81,15 +81,15 @@ Because Parquet files do not contain schema information, before importing data f To allow TiDB Cloud to access the Parquet files in the Amazon S3, GCS, Azure Blob Storage, or Alibaba Cloud Object Storage Service bucket, do one of the following: -- If your Parquet files are located in Amazon S3, [configure Amazon S3 access](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access) for your cluster. +- If your Parquet files are located in Amazon S3, [configure Amazon S3 access](/tidb-cloud/configure-external-storage-access.md#configure-amazon-s3-access) for your cluster. You can use either an AWS access key or a Role ARN to access your bucket. Once finished, make a note of the access key (including the access key ID and secret access key) or the Role ARN value as you will need it in [Step 4](#step-4-import-parquet-files). -- If your Parquet files are located in GCS, [configure GCS access](/tidb-cloud/serverless-external-storage.md#configure-gcs-access) for your cluster. +- If your Parquet files are located in GCS, [configure GCS access](/tidb-cloud/configure-external-storage-access.md#configure-gcs-access) for your cluster. -- If your Parquet files are located in Azure Blob Storage, [configure Azure Blob Storage access](/tidb-cloud/serverless-external-storage.md#configure-azure-blob-storage-access) for your cluster. +- If your Parquet files are located in Azure Blob Storage, [configure Azure Blob Storage access](/tidb-cloud/configure-external-storage-access.md#configure-azure-blob-storage-access) for your cluster. -- If your Parquet files are located in Alibaba Cloud Object Storage Service (OSS), [configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/serverless-external-storage.md#configure-alibaba-cloud-object-storage-service-oss-access) for your cluster. +- If your Parquet files are located in Alibaba Cloud Object Storage Service (OSS), [configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/configure-external-storage-access.md#configure-alibaba-cloud-object-storage-service-oss-access) for your cluster. ## Step 4. Import Parquet files @@ -116,7 +116,7 @@ To import the Parquet files to {{{ .starter }}} or {{{ .essential }}}, take the - **Source Files URI**: - When importing one file, enter the source file URI in the following format `s3://[bucket_name]/[data_source_folder]/[file_name].parquet`. For example, `s3://sampledata/ingest/TableName.01.parquet`. - When importing multiple files, enter the source folder URI in the following format `s3://[bucket_name]/[data_source_folder]/`. For example, `s3://sampledata/ingest/`. - - **Credential**: you can use either an AWS Role ARN or an AWS access key to access your bucket. For more information, see [Configure Amazon S3 access](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access). + - **Credential**: you can use either an AWS Role ARN or an AWS access key to access your bucket. For more information, see [Configure Amazon S3 access](/tidb-cloud/configure-external-storage-access.md#configure-amazon-s3-access). - **AWS Role ARN**: enter the AWS Role ARN value. - **AWS Access Key**: enter the AWS access key ID and AWS secret access key. @@ -169,7 +169,7 @@ To import the Parquet files to {{{ .starter }}} or {{{ .essential }}}, take the - **Source Files URI**: - When importing one file, enter the source file URI in the following format `[gcs|gs]://[bucket_name]/[data_source_folder]/[file_name].parquet`. For example, `[gcs|gs]://sampledata/ingest/TableName.01.parquet`. - When importing multiple files, enter the source folder URI in the following format `[gcs|gs]://[bucket_name]/[data_source_folder]/`. For example, `[gcs|gs]://sampledata/ingest/`. - - **Credential**: you can use a GCS IAM Role Service Account key to access your bucket. For more information, see [Configure GCS access](/tidb-cloud/serverless-external-storage.md#configure-gcs-access). + - **Credential**: you can use a GCS IAM Role Service Account key to access your bucket. For more information, see [Configure GCS access](/tidb-cloud/configure-external-storage-access.md#configure-gcs-access). 4. Click **Next**. @@ -220,7 +220,7 @@ To import the Parquet files to {{{ .starter }}} or {{{ .essential }}}, take the - **Source Files URI**: - When importing one file, enter the source file URI in the following format `[azure|https]://[bucket_name]/[data_source_folder]/[file_name].parquet`. For example, `[azure|https]://sampledata/ingest/TableName.01.parquet`. - When importing multiple files, enter the source folder URI in the following format `[azure|https]://[bucket_name]/[data_source_folder]/`. For example, `[azure|https]://sampledata/ingest/`. - - **Credential**: you can use a shared access signature (SAS) token to access your bucket. For more information, see [Configure Azure Blob Storage access](/tidb-cloud/serverless-external-storage.md#configure-azure-blob-storage-access). + - **Credential**: you can use a shared access signature (SAS) token to access your bucket. For more information, see [Configure Azure Blob Storage access](/tidb-cloud/configure-external-storage-access.md#configure-azure-blob-storage-access). 4. Click **Next**. @@ -271,7 +271,7 @@ To import the Parquet files to {{{ .starter }}} or {{{ .essential }}}, take the - **Source Files URI**: - When importing one file, enter the source file URI in the following format `oss://[bucket_name]/[data_source_folder]/[file_name].parquet`. For example, `oss://sampledata/ingest/TableName.01.parquet`. - When importing multiple files, enter the source folder URI in the following format `oss://[bucket_name]/[data_source_folder]/`. For example, `oss://sampledata/ingest/`. - - **Credential**: you can use an AccessKey pair to access your bucket. For more information, see [Configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/serverless-external-storage.md#configure-alibaba-cloud-object-storage-service-oss-access). + - **Credential**: you can use an AccessKey pair to access your bucket. For more information, see [Configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/configure-external-storage-access.md#configure-alibaba-cloud-object-storage-service-oss-access). 4. Click **Next**. diff --git a/tidb-cloud/migrate-from-op-tidb.md b/tidb-cloud/migrate-from-op-tidb.md index a104ed292ca21..d5452af1f15dd 100644 --- a/tidb-cloud/migrate-from-op-tidb.md +++ b/tidb-cloud/migrate-from-op-tidb.md @@ -204,7 +204,7 @@ After you export data from the TiDB Self-Managed cluster to Amazon S3, you need 1. In the [TiDB Cloud console](https://tidbcloud.com/), get the Account ID and External ID of your target cluster according to the following documentation: - For TiDB Cloud Dedicated clusters, see [Configure Amazon S3 access using a Role ARN](/tidb-cloud/dedicated-external-storage.md#configure-amazon-s3-access-using-a-role-arn). - - For {{{ .starter }}} or {{{ .essential }}} clusters, see [Configure Amazon S3 access using a Role ARN](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access-using-a-role-arn). + - For {{{ .starter }}} or {{{ .essential }}} clusters, see [Configure Amazon S3 access using a Role ARN](/tidb-cloud/configure-external-storage-access.md#configure-amazon-s3-access-using-a-role-arn). 2. Configure access permissions for Amazon S3. Usually you need the following read-only permissions: diff --git a/tidb-cloud/premium/import-csv-files-premium.md b/tidb-cloud/premium/import-csv-files-premium.md index 358157167ca4d..e17a169407dae 100644 --- a/tidb-cloud/premium/import-csv-files-premium.md +++ b/tidb-cloud/premium/import-csv-files-premium.md @@ -83,11 +83,11 @@ Because CSV files do not contain schema information, before importing data from To allow {{{ .premium }}} to access the CSV files in Amazon S3 or Alibaba Cloud Object Storage Service (OSS), do one of the following: -- If your CSV files are located in Amazon S3, [configure Amazon S3 access](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access) for your TiDB instance. +- If your CSV files are located in Amazon S3, [configure Amazon S3 access](/tidb-cloud/configure-external-storage-access.md#configure-amazon-s3-access) for your TiDB instance. You can use either an AWS access key or a Role ARN to access your bucket. Once finished, make a note of the access key (including the access key ID and secret access key) or the Role ARN value as you will need it in [Step 4](#step-4-import-csv-files). -- If your CSV files are located in Alibaba Cloud Object Storage Service (OSS), [configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/serverless-external-storage.md#configure-alibaba-cloud-object-storage-service-oss-access) for your TiDB instance. +- If your CSV files are located in Alibaba Cloud Object Storage Service (OSS), [configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/configure-external-storage-access.md#configure-alibaba-cloud-object-storage-service-oss-access) for your TiDB instance. ## Step 4. Import CSV files @@ -114,7 +114,7 @@ To import the CSV files to {{{ .premium }}}, take the following steps: - **Source Files URI**: - When importing one file, enter the source file URI in the following format `s3://[bucket_name]/[data_source_folder]/[file_name].csv`. For example, `s3://sampledata/ingest/TableName.01.csv`. - When importing multiple files, enter the source folder URI in the following format `s3://[bucket_name]/[data_source_folder]/`. For example, `s3://sampledata/ingest/`. - - **Credential**: you can use either an AWS Role ARN or an AWS access key to access your bucket. For more information, see [Configure Amazon S3 access](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access). + - **Credential**: you can use either an AWS Role ARN or an AWS access key to access your bucket. For more information, see [Configure Amazon S3 access](/tidb-cloud/configure-external-storage-access.md#configure-amazon-s3-access). - **AWS Role ARN**: enter the AWS Role ARN value. If you need to create a new role, click **Click here to create a new one with AWS CloudFormation** and follow the guided steps to launch the provided template, acknowledge the IAM warning, create the stack, and copy the generated ARN back into {{{ .premium }}}. - **AWS Access Key**: enter the AWS access key ID and AWS secret access key. - **Test Bucket Access**: click this button after the credentials are in place to confirm that {{{ .premium }}} can reach the bucket. @@ -168,7 +168,7 @@ To import the CSV files to {{{ .premium }}}, take the following steps: - **Source Files URI**: - When importing one file, enter the source file URI in the following format `oss://[bucket_name]/[data_source_folder]/[file_name].csv`. For example, `oss://sampledata/ingest/TableName.01.csv`. - When importing multiple files, enter the source folder URI in the following format `oss://[bucket_name]/[data_source_folder]/`. For example, `oss://sampledata/ingest/`. - - **Credential**: you can use an AccessKey pair to access your bucket. For more information, see [Configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/serverless-external-storage.md#configure-alibaba-cloud-object-storage-service-oss-access). + - **Credential**: you can use an AccessKey pair to access your bucket. For more information, see [Configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/configure-external-storage-access.md#configure-alibaba-cloud-object-storage-service-oss-access). - **Test Bucket Access**: click this button after the credentials are in place to confirm that {{{ .premium }}} can reach the bucket. - **Target Connection**: provide the TiDB username and password that will run the import. Optionally, click **Test Connection** to validate the credentials. diff --git a/tidb-cloud/release-notes-2024.md b/tidb-cloud/release-notes-2024.md index d204f7dd1f4cd..3122085ffb404 100644 --- a/tidb-cloud/release-notes-2024.md +++ b/tidb-cloud/release-notes-2024.md @@ -148,7 +148,7 @@ This page lists the release notes of [TiDB Cloud](https://www.pingcap.com/tidb-c Previously, TiDB Cloud only supported exporting data using the [TiDB Cloud CLI](/tidb-cloud/cli-reference.md). Now, you can easily export data from TiDB Cloud Serverless clusters to local files and Amazon S3 in the [TiDB Cloud console](https://tidbcloud.com/). - For more information, see [Export Data from TiDB Cloud Serverless](/tidb-cloud/serverless-export.md) and [Configure External Storage Access for TiDB Cloud Serverless](/tidb-cloud/serverless-external-storage.md). + For more information, see [Export Data from TiDB Cloud Serverless](/tidb-cloud/serverless-export.md) and [Configure External Storage Access for TiDB Cloud Serverless](/tidb-cloud/configure-external-storage-access.md). - Enhance the connection experience for [TiDB Cloud Dedicated](/tidb-cloud/select-cluster-tier.md#tidb-cloud-dedicated) clusters. diff --git a/tidb-cloud/serverless-export.md b/tidb-cloud/serverless-export.md index d3e72e59a6d67..085e802e0b051 100644 --- a/tidb-cloud/serverless-export.md +++ b/tidb-cloud/serverless-export.md @@ -52,7 +52,7 @@ To export data to Amazon S3, you need to provide the following information: - [An access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html): make sure the access key has the `s3:PutObject` and `s3:ListBucket` permissions. - [A role ARN](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html): make sure the role ARN (Amazon Resource Name) has the `s3:PutObject` and `s3:ListBucket` permissions. Note that only clusters hosted on AWS support the role ARN. -For more information, see [Configure Amazon S3 access](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access). +For more information, see [Configure Amazon S3 access](/tidb-cloud/configure-external-storage-access.md#configure-amazon-s3-access). ### Google Cloud Storage @@ -61,7 +61,7 @@ To export data to Google Cloud Storage, you need to provide the following inform - URI: `gs:////` - Access credential: a **base64 encoded** [service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) for your bucket. Make sure the service account key has the `storage.objects.create` permission. -For more information, see [Configure GCS access](/tidb-cloud/serverless-external-storage.md#configure-gcs-access). +For more information, see [Configure GCS access](/tidb-cloud/configure-external-storage-access.md#configure-gcs-access). ### Azure Blob Storage @@ -70,7 +70,7 @@ To export data to Azure Blob Storage, you need to provide the following informat - URI: `azure://.blob.core.windows.net///` or `https://.blob.core.windows.net///` - Access credential: a [shared access signature (SAS) token](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview) for your Azure Blob Storage container. Make sure the SAS token has the `Read` and `Write` permissions on the `Container` and `Object` resources. -For more information, see [Configure Azure Blob Storage access](/tidb-cloud/serverless-external-storage.md#configure-azure-blob-storage-access). +For more information, see [Configure Azure Blob Storage access](/tidb-cloud/configure-external-storage-access.md#configure-azure-blob-storage-access). ### Alibaba Cloud OSS @@ -79,7 +79,7 @@ To export data to Alibaba Cloud OSS, you need to provide the following informati - URI: `oss:////` - Access credential: An [AccessKey pair](https://www.alibabacloud.com/help/en/ram/user-guide/create-an-accesskey-pair) for your Alibaba Cloud account. Make sure the AccessKey pair has the `oss:PutObject`, `oss:ListBuckets` and `oss:GetBucketInfo` permissions to allow data export to the OSS bucket. -For more information, see [Configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/serverless-external-storage.md#configure-alibaba-cloud-object-storage-service-oss-access). +For more information, see [Configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/configure-external-storage-access.md#configure-alibaba-cloud-object-storage-service-oss-access). ## Export options @@ -242,7 +242,7 @@ When exporting data to the Parquet format, the data conversion between TiDB and - **Compression**: choose **Gzip**, **Snappy**, **Zstd**, or **None**. - **Folder URI**: enter the URI of the Amazon S3 with the `s3:////` format. - **Bucket Access**: choose one of the following access credentials and then fill in the credential information: - - **AWS Role ARN**: enter the role ARN that has the permission to access the bucket. It is recommended to create the role ARN with AWS CloudFormation. For more information, see [Configure Amazon S3 access](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access). + - **AWS Role ARN**: enter the role ARN that has the permission to access the bucket. It is recommended to create the role ARN with AWS CloudFormation. For more information, see [Configure Amazon S3 access](/tidb-cloud/configure-external-storage-access.md#configure-amazon-s3-access). - **AWS Access Key**: enter the access key ID and access key secret that have the permission to access the bucket. 4. Click **Export**. @@ -323,7 +323,7 @@ ticloud serverless export create -c --target-type GCS --gcs.uri .blob.core.windows.net///` format. - - **SAS Token**: enter the SAS token that has the permission to access the container. It is recommended to create a SAS token with the [Azure ARM template](https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/). For more information, see [Configure Azure Blob Storage access](/tidb-cloud/serverless-external-storage.md#configure-azure-blob-storage-access). + - **SAS Token**: enter the SAS token that has the permission to access the container. It is recommended to create a SAS token with the [Azure ARM template](https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/). For more information, see [Configure Azure Blob Storage access](/tidb-cloud/configure-external-storage-access.md#configure-azure-blob-storage-access). 4. Click **Export**. diff --git a/tidb-cloud/tidb-cloud-migration-overview.md b/tidb-cloud/tidb-cloud-migration-overview.md index c5daaa8038a71..10ccd0125bd36 100644 --- a/tidb-cloud/tidb-cloud-migration-overview.md +++ b/tidb-cloud/tidb-cloud-migration-overview.md @@ -55,7 +55,7 @@ If you have data files in SQL, CSV, Parquet, or Aurora Snapshot formats, you can ### Configure cloud storage access -If your source data is stored in Amazon S3, Google Cloud Storage (GCS) buckets, Azure Blob Storage containers, or Alibaba Cloud OSS buckets, before importing or migrating the data to TiDB Cloud, you need to configure access to the storage. For more information, see [Configure External Storage Access for {{{ .starter }}} or Essential](/tidb-cloud/serverless-external-storage.md) and [Configure External Storage Access for TiDB Cloud Dedicated](/tidb-cloud/dedicated-external-storage.md). +If your source data is stored in Amazon S3, Google Cloud Storage (GCS) buckets, Azure Blob Storage containers, or Alibaba Cloud OSS buckets, before importing or migrating the data to TiDB Cloud, you need to configure access to the storage. For more information, see [Configure External Storage Access for {{{ .starter }}} or Essential](/tidb-cloud/configure-external-storage-access.md) and [Configure External Storage Access for TiDB Cloud Dedicated](/tidb-cloud/dedicated-external-storage.md). ### Naming conventions for data import diff --git a/tidb-cloud/tidb-cloud-release-notes.md b/tidb-cloud/tidb-cloud-release-notes.md index cde6361bd8006..958a5a54db663 100644 --- a/tidb-cloud/tidb-cloud-release-notes.md +++ b/tidb-cloud/tidb-cloud-release-notes.md @@ -744,4 +744,4 @@ This page lists the release notes of [TiDB Cloud](https://www.pingcap.com/tidb-c - Support exporting data in Parquet files through the [TiDB Cloud console](https://tidbcloud.com/). - For more information, see [Export Data from TiDB Cloud Serverless](/tidb-cloud/serverless-export.md) and [Configure External Storage Access for TiDB Cloud Serverless](/tidb-cloud/serverless-external-storage.md). + For more information, see [Export Data from TiDB Cloud Serverless](/tidb-cloud/serverless-export.md) and [Configure External Storage Access for TiDB Cloud Serverless](/tidb-cloud/configure-external-storage-access.md).