Skip to content

Commit d56ef78

Browse files
Merge pull request #2366 from redis/DOC-5937-rdi-source-prep-checklists
DOC-5937 RDI source prep checklists
2 parents 00ed5d5 + ebb0e0a commit d56ef78

File tree

10 files changed

+339
-58
lines changed

10 files changed

+339
-58
lines changed

content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-mysql.md

Lines changed: 46 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,12 @@ Select the steps for your database type.
2525
tab1="AWS Aurora MySQL"
2626
tab2="AWS RDS MySQL" >}}
2727

28+
```checklist {id="auroramysql" nointeractive="true" }
29+
- [ ] [Add an Aurora reader node](#add-an-aurora-reader-node)
30+
- [ ] [Create and apply parameter group](#aurora-create-and-apply-parameter-group)
31+
- [ ] [Create Debezium user](#aurora-create-debezium-user)
32+
```
33+
2834
## Add an Aurora reader node
2935

3036
RDI requires that your Aurora MySQL database has at least one replica or reader node.
@@ -33,11 +39,19 @@ To add a reader node to an existing database, select **Add reader** from the **A
3339

3440
You can also create one during database creation by selecting **Create an Aurora Replica or Reader node in a different AZ (recommended for scaled availability)** under **Availability & durability > Multi-AZ deployment**.
3541

36-
## Create and apply parameter group
42+
## <a id="aurora-create-and-apply-parameter-group"></a>Create and apply parameter group
3743

3844
RDI requires some changes to database parameters. On AWS Aurora, you change these parameters via a parameter group.
3945

40-
1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.Creating.html) with the following settings:
46+
```checklist {id="auroramysql-param-group" nointeractive="true" }
47+
- [ ] [Create a parameter group](#aurora-create-a-parameter-group)
48+
- [ ] [Apply the parameter group](#aurora-apply-the-parameter-group)
49+
- [ ] [Apply the parameter group to the database](#aurora-apply-the-parameter-group-to-the-database)
50+
- [ ] [Reboot the database instance](#aurora-reboot-the-database-instance)
51+
```
52+
53+
1. <a id="aurora-create-a-parameter-group"></a>
54+
In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.Creating.html) with the following settings:
4155

4256
| Name | Value |
4357
| :-- | :-- |
@@ -49,7 +63,8 @@ RDI requires some changes to database parameters. On AWS Aurora, you change thes
4963

5064
Select **Create** to create the parameter group.
5165

52-
1. Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters:
66+
1. <a id="aurora-apply-the-parameter-group"></a>
67+
Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters:
5368

5469
| Name | Value |
5570
| :-- | :-- |
@@ -60,22 +75,38 @@ RDI requires some changes to database parameters. On AWS Aurora, you change thes
6075

6176
Select **Save Changes** to apply the changes to the parameter group.
6277

63-
1. Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.
78+
1. <a id="aurora-apply-the-parameter-group-to-the-database"></a>
79+
Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.
6480

6581
Select **Save changes** to apply the parameter group to the new database.
6682

67-
1. Reboot your database instance. See [Rebooting a DB instance within an Aurora cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-reboot-db-instance.html) for more information.
83+
1. <a id="aurora-reboot-the-database-instance"></a>
84+
Reboot your database instance. See [Rebooting a DB instance within an Aurora cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-reboot-db-instance.html) for more information.
6885

86+
<a id="aurora-create-debezium-user"></a>
6987

7088
{{< embed-md "aur-rds-mysql-create-debezium-user.md" >}}
7189

7290
-tab-sep-
7391

74-
## Create and apply parameter group
92+
```checklist {id="rds-mysql-list" nointeractive="true" }
93+
- [ ] [Create and apply parameter group](#rds-create-and-apply-parameter-group)
94+
- [ ] [Create Debezium user](#rds-create-debezium-user)
95+
```
96+
97+
## <a id="rds-create-and-apply-parameter-group"></a>Create and apply parameter group
7598

7699
RDI requires some changes to database parameters. On AWS RDS, you change these parameters via a parameter group.
77100

78-
1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.Creating.html) with the following settings:
101+
```checklist {id="rds-mysql-param-group" nointeractive="true" }
102+
- [ ] [Create a parameter group](#rds-create-a-parameter-group)
103+
- [ ] [Apply the parameter group](#rds-apply-the-parameter-group)
104+
- [ ] [Apply the parameter group to the database](#rds-apply-the-parameter-group-to-the-database)
105+
- [ ] [Reboot the database instance](#rds-reboot-the-database-instance)
106+
```
107+
108+
1. <a id="rds-create-a-parameter-group"></a>
109+
In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/),navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.Creating.html) with the following settings:
79110

80111
| Name | Value |
81112
| :-- | :-- |
@@ -86,7 +117,8 @@ RDI requires some changes to database parameters. On AWS RDS, you change these p
86117

87118
Select **Create** to create the parameter group.
88119

89-
1. Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters:
120+
1. <a id="rds-apply-the-parameter-group"></a>
121+
Navigate to **Parameter groups** in the console. Select the parameter group you have just created and then select **Edit**. Change the following parameters:
90122

91123
| Name | Value |
92124
| :-- | :-- |
@@ -95,11 +127,15 @@ RDI requires some changes to database parameters. On AWS RDS, you change these p
95127

96128
Select **Save Changes** to apply the changes to the parameter group.
97129

98-
1. Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.
130+
1. <a id="rds-apply-the-parameter-group-to-the-database"></a>
131+
Go back to your target database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.
99132

100133
Select **Save changes** to apply the parameter group to the new database.
101134

102-
1. Reboot your database instance. See [Rebooting a DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RebootInstance.html) for more information.
135+
1. <a id="rds-reboot-the-database-instance"></a>
136+
Reboot your database instance. See [Rebooting a DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RebootInstance.html) for more information.
137+
138+
<a id="rds-create-debezium-user"></a>
103139

104140
{{< embed-md "aur-rds-mysql-create-debezium-user.md" >}}
105141

content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-aur-pgsql.md

Lines changed: 32 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -23,11 +23,24 @@ Follow the steps in the sections below to prepare an
2323
[AWS Aurora PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_GettingStartedAurora.CreatingConnecting.AuroraPostgreSQL.html) or [AWS RDS PostgreSQL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html)
2424
database to work with RDI.
2525

26+
```checklist {id="aurorapostgresql" nointeractive="true" }
27+
- [ ] [Create and apply parameter group](#create-and-apply-parameter-group)
28+
- [ ] [Create Debezium user](#create-debezium-user)
29+
```
30+
2631
## Create and apply parameter group
2732

2833
RDI requires some changes to database parameters. On AWS RDS and AWS Aurora, you change these parameters via a parameter group.
2934

30-
1. In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/), navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings:
35+
```checklist {id="aurorapostgresql-param-group" nointeractive="true" }
36+
- [ ] [Create a parameter group](#create-a-parameter-group)
37+
- [ ] [Apply the parameter group](#apply-the-parameter-group)
38+
- [ ] [Apply the parameter group to the database](#apply-the-parameter-group-to-the-database)
39+
- [ ] [Reboot the database instance](#reboot-the-database-instance)
40+
```
41+
42+
1. <a id="create-a-parameter-group"></a>
43+
In the [Relational Database Service (RDS) console](https://console.aws.amazon.com/rds/), navigate to **Parameter groups > Create parameter group**. [Create a parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_WorkingWithParamGroups.CreatingCluster.html) with the following settings:
3144

3245
| Name | Value |
3346
| :-- | :-- |
@@ -38,43 +51,55 @@ RDI requires some changes to database parameters. On AWS RDS and AWS Aurora, you
3851

3952
Select **Create** to create the parameter group.
4053

41-
1. Navigate to **Parameter groups** in the console. Select the group you have just created and then select **Edit**. Change the following parameters:
54+
1. <a id="apply-the-parameter-group"></a>
55+
Navigate to **Parameter groups** in the console. Select the group you have just created and then select **Edit**. Change the following parameters:
4256

4357
| Name | Value |
4458
| :-- | :-- |
4559
| `rds.logical_replication` | `1` |
4660

4761
Select **Save Changes** to apply the changes to the parameter group.
4862

49-
1. Go back to your database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.
63+
1. <a id="apply-the-parameter-group-to-the-database"></a>
64+
Go back to your database on the RDS console, select **Modify** and then scroll down to **Additional Configuration**. Set the **DB Cluster Parameter Group** to the group you just created.
5065

5166
Select **Save changes** to apply the parameter group to your database.
5267

53-
1. Reboot your database instance. See [Rebooting a DB instance within an Aurora cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-reboot-db-instance.html) or [Rebooting a DB instance (RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RebootInstance.html) for more information.
68+
1. <a id="reboot-the-database-instance"></a>
69+
Reboot your database instance. See [Rebooting a DB instance within an Aurora cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-reboot-db-instance.html) or [Rebooting a DB instance (RDS)](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RebootInstance.html) for more information.
5470

5571
## Create Debezium user
5672

5773
The Debezium connector needs a user account to connect to PostgreSQL. This
5874
user must have appropriate permissions on all databases where you want Debezium
5975
to capture changes.
6076

61-
1. Connect to PostgreSQL as the `postgres` user and create a new user for the connector:
77+
```checklist {id="aurorapostgresql-create-debezium-user" nointeractive="true" }
78+
- [ ] [Connect to PostgreSQL as the `postgres` user](#connect-to-postgresql-as-the-postgres-user)
79+
- [ ] [Grant the user the necessary replication permissions](#grant-the-user-the-necessary-replication-permissions)
80+
- [ ] [Grant the user access to the database](#grant-the-user-access-to-the-database)
81+
```
82+
83+
1. <a id="connect-to-postgresql-as-the-postgres-user"></a>
84+
Connect to PostgreSQL as the `postgres` user and create a new user for the connector:
6285

6386
```sql
6487
CREATE ROLE <username> WITH LOGIN PASSWORD '<password>' VALID UNTIL 'infinity';
6588
```
6689

6790
Replace `<username>` and `<password>` with a username and password for the new user.
6891

69-
1. Grant the user the necessary replication permissions:
92+
1. <a id="grant-the-user-the-necessary-replication-permissions"></a>
93+
Grant the user the necessary replication permissions:
7094

7195
```sql
7296
GRANT rds_replication TO <username>;
7397
```
7498

7599
Replace `<username>` with the username of the Debezium user.
76100

77-
1. Connect to your database as the `postgres` user and grant the new user access to one or more schemas in the database:
101+
1. <a id="grant-the-user-access-to-the-database"></a>
102+
Connect to your database as the `postgres` user and grant the new user access to one or more schemas in the database:
78103

79104
```sql
80105
GRANT SELECT ON ALL TABLES IN SCHEMA <schema> TO <username>;

content/integrate/redis-data-integration/data-pipelines/prepare-dbs/aws-aurora-rds/aws-rds-sqlserver.md

Lines changed: 26 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -22,13 +22,24 @@ Follow the steps in the sections below to prepare a [Microsoft SQL Server on AWS
2222
Change Data Capture (CDC) is not supported on SQL Server Express Edition. Only the Standard, Enterprise, and Developer editions support CDC and are supported by RDI.
2323
{{< /note >}}
2424

25+
```checklist {id="rds-sqlserver-list" nointeractive="true" }
26+
- [ ] [Create the Debezium user](#create-the-debezium-user)
27+
- [ ] [Enable CDC on the database](#enable-cdc-on-the-database)
28+
```
29+
2530
## Create the Debezium user
2631

2732
The Debezium connector needs a user account to connect to SQL Server. This
2833
user must have appropriate permissions on all databases where you want Debezium
2934
to capture changes.
3035

31-
1. Connect to your database as an admin user and create a new user for the connector:
36+
```checklist {id="rds-sqlserver-create-debezium-user" nointeractive="true" }
37+
- [ ] [Connect to SQL Server as an admin user](#connect-to-sql-server-as-an-admin-user)
38+
- [ ] [Grant the user the necessary permissions](#grant-the-user-the-necessary-permissions)
39+
```
40+
41+
1. <a id="connect-to-sql-server-as-an-admin-user"></a>
42+
Connect to your database as an admin user and create a new user for the connector:
3243

3344
```sql
3445
USE master
@@ -43,7 +54,8 @@ to capture changes.
4354

4455
Replace `<username>` and `<password>` with a username and password for the new user and replace `<database>` with the name of your database.
4556

46-
1. Grant the user the necessary permissions:
57+
1. <a id="grant-the-user-the-necessary-permissions"></a>
58+
Grant the user the necessary permissions:
4759

4860
```sql
4961
USE master
@@ -62,7 +74,14 @@ to capture changes.
6274

6375
Change Data Capture (CDC) must be enabled for the database and for each table you want to capture.
6476

65-
1. Enable CDC for the database by running the following command:
77+
```checklist {id="rds-sqlserver-enable-cdc" nointeractive="true" }
78+
- [ ] [Enable CDC for the database](#enable-cdc-for-the-database)
79+
- [ ] [Enable CDC for each table you want to capture](#enable-cdc-for-each-table-you-want-to-capture)
80+
- [ ] [Add the Debezium user to the CDC role](#add-the-debezium-user-to-the-cdc-role)
81+
```
82+
83+
1. <a id="enable-cdc-for-the-database"></a>
84+
Enable CDC for the database by running the following command:
6685

6786
```sql
6887
EXEC msdb.dbo.rds_cdc_enable_db '<database>'
@@ -71,7 +90,8 @@ Change Data Capture (CDC) must be enabled for the database and for each table yo
7190

7291
Replace `<database>` with the name of your database.
7392

74-
1. Enable CDC for each table you want to capture by running the following commands:
93+
1. <a id="enable-cdc-for-each-table-you-want-to-capture"></a>
94+
Enable CDC for each table you want to capture by running the following commands:
7595

7696
```sql
7797
USE <database>
@@ -92,7 +112,8 @@ Specifying a new name will create a corresponding database role that has full ac
92112
captured change data.
93113
{{< /note >}}
94114

95-
1. Add the Debezium user to the CDC role:
115+
1. <a id="add-the-debezium-user-to-the-cdc-role"></a>
116+
Add the Debezium user to the CDC role:
96117

97118
```sql
98119
USE <database>

content/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb.md

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ This guide describes the steps required to prepare a MongoDB database as a sourc
2525
{{< note >}}The MongoDB connector is not capable of monitoring the changes of a standalone MongoDB server, since standalone servers do not have an oplog. The connector will work if the standalone server is converted to a replica set with one member.{{< /note >}}
2626
## Summary
2727

28-
The following table summarizes the steps to prepare a MongoDB database for RDI. The sections below explain the steps in more detail.
28+
The following table summarizes the considerations to prepare a MongoDB database for RDI.
2929

3030
| Requirement | Description |
3131
|---------------------|-----------------------------------------------------------------------------|
@@ -37,6 +37,20 @@ The following table summarizes the steps to prepare a MongoDB database for RDI.
3737
| MongoDB Atlas | **[SSL required](https://debezium.io/documentation/reference/stable/connectors/mongodb.html#mongodb-property-mongodb-ssl-enabled)**, provide root CA as `SOURCE_DB_CACERT` secret in RDI |
3838
| Network | RDI Collector must reach all MongoDB nodes on required ports |
3939

40+
The following checklist shows the steps to prepare a MongoDB database for RDI,
41+
with links to the sections that explain the steps in full detail.
42+
You may find it helpful to track your progress with the checklist as you
43+
complete each step.
44+
45+
```checklist {id="mongodblist"}
46+
- [ ] [Configure oplog size](#1-configure-oplog-size)
47+
- [ ] [Create a MongoDB user for RDI](#2-create-a-mongodb-user-for-rdi)
48+
- [ ] [Connection string format](#3-connection-string-format)
49+
- [ ] [Enable change streams and pre/post images (only if using a custom key)](#4-enable-change-streams-and-prepost-images-only-if-using-a-custom-key)
50+
- [ ] [MongoDB Atlas specific requirements](#5-mongodb-atlas-specific-requirements)
51+
- [ ] [Network and security](#6-network-and-security)
52+
```
53+
4054
## 1. Configure oplog size
4155

4256
The Debezium MongoDB connector relies on the [oplog](https://www.mongodb.com/docs/manual/core/replica-set-oplog/) to capture changes from a replica set. The oplog is a fixed-size, capped collection. When it reaches its maximum size, it overwrites the oldest entries. If the connector is stopped and restarted, it attempts to resume from its last recorded position in the oplog. If that position has been overwritten, the connector may fail to start and report an invalid resume token error.

0 commit comments

Comments
 (0)