diff --git a/content/embeds/rc-rdi-secrets-encryption-permissions.md b/content/embeds/rc-rdi-secrets-encryption-permissions.md new file mode 100644 index 0000000000..2b02248723 --- /dev/null +++ b/content/embeds/rc-rdi-secrets-encryption-permissions.md @@ -0,0 +1,5 @@ +* **Encryption key**: Select the [encryption key](#create-encryption-key) you created earlier. + +* **Resource permissions**: Add the following permissions to your secret to allow the Redis data pipeline to access your secret. Replace `` with the AWS account ID for the Redis Cloud cluster that you saved earlier. + +{{< embed-md "rc-rdi-secrets-permissions.md" >}} \ No newline at end of file diff --git a/content/embeds/rc-rdi-secrets-permissions.md b/content/embeds/rc-rdi-secrets-permissions.md new file mode 100644 index 0000000000..74867bd35d --- /dev/null +++ b/content/embeds/rc-rdi-secrets-permissions.md @@ -0,0 +1,17 @@ +```json +{ + "Version" : "2012-10-17", + "Statement" : [ { + "Sid" : "RedisDataIntegrationRoleAccess", + "Effect" : "Allow", + "Principal" : "*", + "Action" : [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ], + "Resource" : "*", + "Condition" : { + "StringLike" : { + "aws:PrincipalArn" : "arn:aws:iam:::role/redis-data-pipeline-secrets-role" + } + } + } ] +} +``` \ No newline at end of file diff --git a/content/operate/rc/databases/rdi/define.md b/content/operate/rc/databases/rdi/define.md index 5ee56e0c03..9ee648273f 100644 --- a/content/operate/rc/databases/rdi/define.md +++ b/content/operate/rc/databases/rdi/define.md @@ -30,7 +30,12 @@ After you have [prepared your source database]({{}}) in the **Source database secrets ARN** field. +1. Enter the ARN of your [database credentials secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Source database secrets ARN** field. +1. If your database requires TLS, select **Use TLS**. Enter the ARN of your [CA certificate secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **CA Cert Secret ARN** field. + {{The Source database connectivity section, with Use TLS selected and the CA Cert Secret ARN field.}} +1. If your database requires mTLS, select **Use mTLS**. Enter the ARN of your [Client certificate secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Client Certificate Secret ARN** field and the ARN of your [Client key secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Client Key Secret ARN** field. + {{The Source database connectivity section, with Use TLS selected and the Client Certificate Secret ARN and Client Key Secret ARN fields.}} +1. If your database requires mTLS with a client key passphrase, enter the ARN of your [Client key passphrase secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Please add a secret ARN for the password to use with the secret store** field. 1. Select **Start pipeline setup**. {{The start pipeline setup button.}} 1. Redis Cloud will attempt to connect to PrivateLink. If your PrivateLink does not allow automatic acceptance of incoming connections, accept the incoming connection on AWS PrivateLink to proceed. See [Accept or Reject PrivateLink connection requests](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests). @@ -53,21 +58,29 @@ After your pipeline is provisioned, you will be able to define your pipeline. Yo ### Configure a new pipeline -1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. If your pipeline is already provisioned, select **Complete setup** to go to the **Select data** section. +1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. If your pipeline is already provisioned, select **Complete setup** to go to the **Data modeling** section. {{The complete setup button.}} -1. Select the Schema and Tables you want to migrate to the target database from the **Source data selection** list. - {{The select source data section. }} +1. Select the Schema and Tables you want to migrate to the target database from the list. + {{The data modeling section. }} + + Select **Manage Columns** to choose which columns you want to import. + + {{The manage columns button.}} You can select any number of columns from a table. - {{The select source data section. A table is expanded with a few columns selected.}} + {{The manage columns screen, with a few columns selected from one table}} - If any tables are missing a unique constraint, the **Missing unique constraint** list will appear. Select the columns that define a unique constraint for those tables from the list. + If any tables are missing a unique constraint, a warning will appear in the **Data modeling** section. Select **Manage columns** to select the columns that define a unique constraint for those tables. {{The missing unique constraint list.}} {{The missing unique constraint list with columns selected.}} + Select **Save** to save your column changes and go back to schema selection. + + {{The save button.}} + Select **Add schema** to add more database schemas. {{The add schema button.}} @@ -80,19 +93,21 @@ After your pipeline is provisioned, you will be able to define your pipeline. Yo {{The continue button.}} -1. In the **Pipeline definition** section, select the Redis data type to write keys to the target. You can choose **Hash** or **JSON** if the target database supports JSON. +1. Select the Redis data type to write keys to the target. You can choose **Hash** or **JSON** if the target database supports JSON. {{The pipeline definition screen.}} You can also supply one or more [transformation job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) that specify how you want to transform the captured data before writing it to the target. Select **Upload jobs** to upload your job files. {{The transformation jobs section. Select Upload jobs to upload transformation jobs.}} + When you upload job files, Redis Cloud will validate the job files to check for errors. + Select **Continue**. {{The continue button.}} -1. Review the tables you selected in the **Summary**. If everything looks correct, select **Start ingest** to start ingesting data from your source database. +1. Review the tables you selected in the **Review and deploy** section. If everything looks correct, select **Confirm & Deploy** to start ingesting data from your source database. - {{The start ingest button.}} + {{The Confirm & Deploy button.}} At this point, the data pipeline will ingest data from the source database to your target Redis database. This process will take time, especially if you have a lot of records in your source database. diff --git a/content/operate/rc/databases/rdi/setup.md b/content/operate/rc/databases/rdi/setup.md index afd09417f5..0edcd752fa 100644 --- a/content/operate/rc/databases/rdi/setup.md +++ b/content/operate/rc/databases/rdi/setup.md @@ -198,7 +198,7 @@ You need to share your source database credentials and certificates in an Amazon To do this, you need to: 1. [Create an encryption key](#create-encryption-key) using AWS Key Management Service with the right permissions. -1. [Create a secret](#create-database-credentials-secret) containing the source database credentials encrypted using that key. +1. [Create secrets](#create-database-credentials-secrets) containing the source database credentials encrypted using that key. ### Create encryption key @@ -216,7 +216,27 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv Review the key policy and key settings, and then select **Finish** to create the key. -### Create database credentials secret +### Create database credentials secrets + +To let Redis Cloud access your source database, you need to create AWS secrets for the source database's credentials and certificates. + +The required secrets depend on your source database's security configuration. The following table shows the required secrets for each configuration: + +| Security configuration | Required secrets | +| :-- | :-- | +| Username and password only |
  • Credentials secret (username and password)
| +| TLS connection |
  • Credentials secret (username and password)
  • CA Certificate secret (server certificate)
| +| mTLS connection |
  • Credentials secret (username and password)
  • CA Certificate secret (server certificate)
  • Client certificate secret
  • Client key secret
| +| mTLS connection with client key passphrase |
  • Credentials secret (username and password)
  • CA Certificate secret (server certificate)
  • Client certificate secret
  • Client key secret
  • Client key passphrase secret
| + +Select a tab to learn how to create the required secret. + +{{< multitabs id="rdi-cloud-secrets" + tab1="Credentials secret" + tab2="CA Certificate secret" + tab3="Client certificate secret" + tab4="Client key secret" + tab5="Client key passphrase secret" >}} In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings: @@ -224,36 +244,62 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv - `username`: Database username - `password`: Database password - - `trust_certificate`: Server certificate in PEM format *(TLS only)* - - `client_public_key`: [X.509 client certificate](https://en.wikipedia.org/wiki/X.509) or chain in PEM format *(mTLS only)* - - `client_private_key`: Key for the client certificate or chain in PEM format *(mTLS only)* - - `client_private_key_passphrase`: Passphrase or password for the client certificate or chain in PEM format *(mTLS only)* - - {{}} -If your source database has TLS or mTLS enabled, we recommend that you enter the `trust_certificate`, `client_public_key`, and `client_private_key` into the secret editor using the **Key/Value** input method instead of the **JSON** input method. Pasting directly into the JSON editor may cause an error. - {{}} - -- **Encryption key**: Select the [encryption key](#create-encryption-key) you created earlier. - -- **Resource permissions**: Add the following permissions to your secret to allow the Redis data pipeline to access your secret. Replace `` with the AWS account ID for the Redis Cloud cluster that you saved earlier. - - ```json - { - "Version" : "2012-10-17", - "Statement" : [ { - "Sid" : "RedisDataIntegrationRoleAccess", - "Effect" : "Allow", - "Principal" : "*", - "Action" : [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ], - "Resource" : "*", - "Condition" : { - "StringLike" : { - "aws:PrincipalArn" : "arn:aws:iam:::role/redis-data-pipeline-secrets-role" - } - } - } ] - } - ``` + +{{< embed-md "rc-rdi-secrets-encryption-permissions.md" >}} + +--tab-sep-- + +In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings: + +- **Key/value pairs**: Select **Plaintext** and enter the server certificate. + +{{< embed-md "rc-rdi-secrets-encryption-permissions.md" >}} + +--tab-sep-- + +In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings: + +- **Key/value pairs**: Select **Plaintext** and enter the client certificate. + +{{< embed-md "rc-rdi-secrets-encryption-permissions.md" >}} + +--tab-sep-- + +Use the [AWS CLI create-secret command](https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/create-secret.html) or the [AWS CreateSecret API endpoint](https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_CreateSecret.html) to create a binary secret containing the client key. + +For example, using the AWS CLI, run the following command: + +```sh +aws secretsmanager create-secret \ + --name \ + --secret-binary fileb:// \ + --kms-key-id +``` + +Where: +- `` - Name of the secret +- `` - Path to the client key file +- `` - ARN of the [encryption key](#create-encryption-key) you created earlier + +After you create the secret, you need to add permissions to allow the data pipeline to access it. + +In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. Select the private key secret you just created and then select **Edit permissions**. + +Add the following permissions to your secret: + +{{< embed-md "rc-rdi-secrets-permissions.md" >}} + +Replace `` with the AWS account ID for the Redis Cloud cluster that you saved earlier. + +--tab-sep-- + +In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings: + +- **Key/value pairs**: Select **Plaintext** and enter the client key passphrase. + +{{< embed-md "rc-rdi-secrets-encryption-permissions.md" >}} + +{{< /multitabs >}} After you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. diff --git a/content/operate/rc/databases/rdi/view-edit.md b/content/operate/rc/databases/rdi/view-edit.md index 8ebeea0d01..72d6438eba 100644 --- a/content/operate/rc/databases/rdi/view-edit.md +++ b/content/operate/rc/databases/rdi/view-edit.md @@ -49,20 +49,27 @@ To change the data you want to ingest from the data pipeline: {{The edit pipeline button.}} -1. Select the schema and tables you want to migrate to the target database from the **Source data selection** list. +1. Select the Schema and Tables you want to migrate to the target database from the list. + {{The data modeling section. }} - {{The select source data section. }} + Select **Manage Columns** to choose which columns you want to import. + + {{The manage columns button.}} You can select any number of columns from a table. - {{The select source data section. A table is expanded with a few columns selected.}} + {{The manage columns screen, with a few columns selected from one table}} - If any tables are missing a unique constraint, the **Missing unique constraint** list will appear. Select the columns that define a unique constraint for those tables from the list. + If any tables are missing a unique constraint, a warning will appear in the **Data modeling** section. Select **Manage columns** to select the columns that define a unique constraint for those tables. {{The missing unique constraint list.}} {{The missing unique constraint list with columns selected.}} + Select **Save** to save your column changes and go back to schema selection. + + {{The save button.}} + Select **Add schema** to add more database schemas. {{The add schema button.}} @@ -74,16 +81,20 @@ To change the data you want to ingest from the data pipeline: After you've selected the schemas and tables you want to sync, select **Continue**. {{The continue button.}} + +1. Select the Redis data type to write keys to the target. You can choose **Hash** or **JSON** if the target database supports JSON. + {{The pipeline definition screen.}} -1. In the **Pipeline definition** section, select the Redis data type to write keys to the target. You can choose **Hash** or **JSON** if the target database supports JSON. + You can also supply one or more [transformation job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) that specify how you want to transform the captured data before writing it to the target. Select **Upload jobs** to upload your job files. + + {{The transformation jobs section. Select Upload jobs to upload transformation jobs.}} + + When you upload job files, Redis Cloud will validate the job files to check for errors. - {{The Pipeline definition screen. Configure a new pipeline is selected.}} - Select **Continue**. - {{The continue button.}} -1. Review the tables you selected in the **Summary** and select how you want to update the data pipeline: +1. Review the tables you selected in and select how you want to update the data pipeline: {{The Select update preferences section.}} diff --git a/static/images/rc/button-save.png b/static/images/rc/button-save.png new file mode 100644 index 0000000000..4af5f07ac2 Binary files /dev/null and b/static/images/rc/button-save.png differ diff --git a/static/images/rc/rdi/rdi-configure-new-pipeline.png b/static/images/rc/rdi/rdi-configure-new-pipeline.png index 4ceeb8fa25..8f04aa34f4 100644 Binary files a/static/images/rc/rdi/rdi-configure-new-pipeline.png and b/static/images/rc/rdi/rdi-configure-new-pipeline.png differ diff --git a/static/images/rc/rdi/rdi-start-ingest.png b/static/images/rc/rdi/rdi-confirm-deploy.png similarity index 63% rename from static/images/rc/rdi/rdi-start-ingest.png rename to static/images/rc/rdi/rdi-confirm-deploy.png index b23878a723..faeeaec467 100644 Binary files a/static/images/rc/rdi/rdi-start-ingest.png and b/static/images/rc/rdi/rdi-confirm-deploy.png differ diff --git a/static/images/rc/rdi/rdi-define-connectivity.png b/static/images/rc/rdi/rdi-define-connectivity.png index e0f297a5c4..9fc00e6521 100644 Binary files a/static/images/rc/rdi/rdi-define-connectivity.png and b/static/images/rc/rdi/rdi-define-connectivity.png differ diff --git a/static/images/rc/rdi/rdi-define-mtls.png b/static/images/rc/rdi/rdi-define-mtls.png new file mode 100644 index 0000000000..f660ad88e6 Binary files /dev/null and b/static/images/rc/rdi/rdi-define-mtls.png differ diff --git a/static/images/rc/rdi/rdi-define-pipeline-cidr.png b/static/images/rc/rdi/rdi-define-pipeline-cidr.png index d40ca25af4..f267880478 100644 Binary files a/static/images/rc/rdi/rdi-define-pipeline-cidr.png and b/static/images/rc/rdi/rdi-define-pipeline-cidr.png differ diff --git a/static/images/rc/rdi/rdi-define-tls.png b/static/images/rc/rdi/rdi-define-tls.png new file mode 100644 index 0000000000..63ea2601ae Binary files /dev/null and b/static/images/rc/rdi/rdi-define-tls.png differ diff --git a/static/images/rc/rdi/rdi-manage-columns.png b/static/images/rc/rdi/rdi-manage-columns.png new file mode 100644 index 0000000000..ec4b2931c2 Binary files /dev/null and b/static/images/rc/rdi/rdi-manage-columns.png differ diff --git a/static/images/rc/rdi/rdi-missing-unique-constraint.png b/static/images/rc/rdi/rdi-missing-unique-constraint.png index b1eb80b732..ddc7add526 100644 Binary files a/static/images/rc/rdi/rdi-missing-unique-constraint.png and b/static/images/rc/rdi/rdi-missing-unique-constraint.png differ diff --git a/static/images/rc/rdi/rdi-select-columns.png b/static/images/rc/rdi/rdi-select-columns.png index 23d0420555..625505654d 100644 Binary files a/static/images/rc/rdi/rdi-select-columns.png and b/static/images/rc/rdi/rdi-select-columns.png differ diff --git a/static/images/rc/rdi/rdi-select-constraints.png b/static/images/rc/rdi/rdi-select-constraints.png index 2ed7d04eb0..ed5f7e5ffb 100644 Binary files a/static/images/rc/rdi/rdi-select-constraints.png and b/static/images/rc/rdi/rdi-select-constraints.png differ diff --git a/static/images/rc/rdi/rdi-select-source-data.png b/static/images/rc/rdi/rdi-select-source-data.png index d19e5046de..cadd0e5220 100644 Binary files a/static/images/rc/rdi/rdi-select-source-data.png and b/static/images/rc/rdi/rdi-select-source-data.png differ diff --git a/static/images/rc/rdi/rdi-transformation-jobs.png b/static/images/rc/rdi/rdi-transformation-jobs.png index d8f7f3ae1a..c1c8bc2882 100644 Binary files a/static/images/rc/rdi/rdi-transformation-jobs.png and b/static/images/rc/rdi/rdi-transformation-jobs.png differ