Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,17 @@ Fully managed by CrateDB Cloud, integrations eliminate the need for manual
setup or maintenance of separate ETL pipelines.


```{figure} ../_assets/img/integrations-example.png
```{figure} ../../_assets/img/integrations-example.png
:width: 600px
:align: center
:alt: Integration in CrateDB Cloud
```

:::{toctree}
:maxdepth: 1
:titlesonly:
:hidden:

MongoDB CDC (Preview) <integrations/mongo-cdc>
MongoDB CDC (Preview) <mongo-cdc>
:::

---
Expand Down
142 changes: 71 additions & 71 deletions docs/cluster/integrations/mongo-cdc.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
(integrations-mongo-cdc)=
# MongoDB CDC (Preview)

CrateDB Cloud enables continuous data ingestion from MongoDB using Change Data
CrateDB Cloud enables continuous data ingestion from MongoDB using Change Data
Capture (CDC), providing seamless, real-time synchronization of your data.

:::{caution}
Expand All @@ -11,31 +11,31 @@ For more information, please [contact us](https://cratedb.com/contact).

## Key Concepts

The MongoDB CDC integration in CrateDB Cloud allows you to keep your data
synchronized between your MongoDB Atlas cluster and your CrateDB Cloud cluster
in real-time.
The MongoDB CDC integration in CrateDB Cloud allows you to keep your data
synchronized between your MongoDB Atlas cluster and your CrateDB Cloud cluster
in real-time.

### How It Works

The integration functions in two main stages:

1. **Initial Sync:**
The integration performs a complete scan of your MongoDB collections,
1. **Initial Sync:**
The integration performs a complete scan of your MongoDB collections,
importing all existing data into your CrateDB Cloud cluster.

2. **Continuous Sync:**
The integration uses MongoDB Change Streams to monitor changes in your
MongoDB collections and syncs these updates to your CrateDB Cloud cluster
2. **Continuous Sync:**
The integration uses MongoDB Change Streams to monitor changes in your
MongoDB collections and syncs these updates to your CrateDB Cloud cluster
in real-time, ensuring that your data remains current.

### Data Consistency and Mode

For continuous sync, CrateDB Cloud uses MongoDB's **full document mode** to
ensure data consistency. This mode guarantees that MongoDB returns the latest
majority-committed version of the updated document.
For continuous sync, CrateDB Cloud uses MongoDB's **full document mode** to
ensure data consistency. This mode guarantees that MongoDB returns the latest
majority-committed version of the updated document.

While receiving partial deltas is more efficient, full document mode provides
robust functionality. Support for partial deltas may be added in the future to
While receiving partial deltas is more efficient, full document mode provides
robust functionality. Support for partial deltas may be added in the future to
enhance performance and flexibility.

---
Expand All @@ -55,81 +55,81 @@ Supported authentication methods:
The following steps should be performed in the MongoDB Atlas UI.

#### Step 1: Create a Custom Role
1. **Navigate to Database Access**
Go to **Database Access** in the MongoDB Atlas UI for the cluster you want to
1. **Navigate to Database Access**
Go to **Database Access** in the MongoDB Atlas UI for the cluster you want to
connect to CrateDB Cloud.

2. **Add a Custom Role**
2. **Add a Custom Role**
Under **Custom Roles**, click **Add New Custom Role**.

3. **Set Up Read-Only Access**
3. **Set Up Read-Only Access**
Assign the following actions or roles to the custom role:
- `find`
- `changeStream`
- `collStats`

Specify the databases and collections for these actions. You can update
Specify the databases and collections for these actions. You can update
access permissions in the MongoDB Atlas UI later if needed.


#### Step 2: Create a User

Depending on whether you plan to use SCRAM or X.509 authentication, create a
Depending on whether you plan to use SCRAM or X.509 authentication, create a
database user with one of the following methods:

:::{tab} SCRAM Auhentication

1. **Navigate to Database Access**
In the MongoDB Atlas UI, go to **Database Access** and click **Add New
1. **Navigate to Database Access**
In the MongoDB Atlas UI, go to **Database Access** and click **Add New
Database User**.

2. **Set Authentication Method**
Choose **Password** as the authentication method and enter a username and
2. **Set Authentication Method**
Choose **Password** as the authentication method and enter a username and
password for the database user.

3. **Assign the Role**
3. **Assign the Role**
Under **Database User Privileges**, select the custom role created in Step 1.

4. **Copy User Credentials**
Click **Add User**, and make sure to record the username and password. These
4. **Copy User Credentials**
Click **Add User**, and make sure to record the username and password. These
credentials will be used later in the CrateDB Cloud Console.
:::

:::{tab} x.509 Authentication

1. **Navigate to Database Access**
In the MongoDB Atlas UI, go to **Database Access** and click **Add New
1. **Navigate to Database Access**
In the MongoDB Atlas UI, go to **Database Access** and click **Add New
Database User**.

2. **Set Authentication Method**
2. **Set Authentication Method**
Choose **Certificate** as the authentication method.

3. **Assign the Role**
3. **Assign the Role**
Under **Database User Privileges**, select the custom role created in Step 1.

4. **Save the Certificate**
Click **Add User**, and store the certificate securely. This will be required
4. **Save the Certificate**
Click **Add User**, and store the certificate securely. This will be required
later in the CrateDB Cloud Console.
:::


#### Step 3: Configure IP Access

To allow CrateDB Cloud to access your MongoDB Atlas cluster, you must add the
To allow CrateDB Cloud to access your MongoDB Atlas cluster, you must add the
CrateDB Cloud IP addresses to the IP Access List in MongoDB Atlas.

1. **Navigate to Network Access**
1. **Navigate to Network Access**
In the MongoDB Atlas UI, go to **Network Access** from the left navigation.

2. **Add IP Address**
Click **Add IP Address** and choose an IP address or range to allow access.
For testing purposes, you can select **Allow Access from Anywhere**, but for
2. **Add IP Address**
Click **Add IP Address** and choose an IP address or range to allow access.
For testing purposes, you can select **Allow Access from Anywhere**, but for
production, it is recommended to specify only the required IPs.

:::{note}
The specific IP addresses depend on the region of your CrateDB Cloud cluster.
These IP addresses can also be found in the **Connection Details** section of the
CrateDB Cloud Console, just before you click **Test Connection** during the
The specific IP addresses depend on the region of your CrateDB Cloud cluster.
These IP addresses can also be found in the **Connection Details** section of the
CrateDB Cloud Console, just before you click **Test Connection** during the
setup process.

**Outbound IP Addresses**:
Expand All @@ -156,17 +156,17 @@ out to our support team.
You'll need to provide the connection string for your MongoDB Atlas cluster so
that CrateDB Cloud can connect to it.

1. **Navigate to Your Cluster**
1. **Navigate to Your Cluster**
In the MongoDB Atlas UI, navigate to the cluster you want to connect to CrateDB Cloud.

2. **Click "Connect"**
2. **Click "Connect"**
From the cluster view, click on **Connect**.

3. **Select "Connect Your Application"**
3. **Select "Connect Your Application"**
Choose **Connect your application** as the connection method.

4. **Copy the Connection String**
Copy the connection string provided in the MongoDB Atlas UI. It will look like this:
4. **Copy the Connection String**
Copy the connection string provided in the MongoDB Atlas UI. It will look like this:

```
mongodb+srv://:@/?retryWrites=true&w=majority
Expand All @@ -193,55 +193,55 @@ in CrateDB Cloud.

Follow these steps in the CrateDB Cloud Console to set up the MongoDB CDC integration:

#### Step 1: Create an Integration
1. Navigate to the **Import** section in the CrateDB Cloud Console.
#### Step 1: Create an Integration
1. Navigate to the **Import** section in the CrateDB Cloud Console.
2. Click **Create Integration** and select **MongoDB** as the source type.

#### Step 2: Configure Connection
#### Step 2: Configure Connection
1. Choose **Create New Connection** or select an existing one.
2. Fill in the following details:
:::{tab} SCRAM Auhentication
- **Connection Name**: Provide a unique name for the connection.
- **Connection String**: Paste the connection string from MongoDB Atlas.
- **Username**: Enter the database username (required for SCRAM).
- **Connection Name**: Provide a unique name for the connection.
- **Connection String**: Paste the connection string from MongoDB Atlas.
- **Username**: Enter the database username (required for SCRAM).
- **Password**: Enter the database password (required for SCRAM).
- **Default Database**: Specify the default database to use for this connection.
:::
:::{tab} X.509 Auhentication
- **Connection Name**: Provide a unique name for the connection.
- **Connection String**: Paste the connection string from MongoDB Atlas.
- **Connection Name**: Provide a unique name for the connection.
- **Connection String**: Paste the connection string from MongoDB Atlas.
- **Certificate**: Upload the X.509 certificate file.
- **Default Database**: Specify the default database to use for this connection.
:::

#### Step 3: Test the Connection
#### Step 3: Test the Connection
Click **Test Connection** to verify CrateDB Cloud can connect to your MongoDB
Atlas cluster. Resolve any issues if the test fails.

#### Step 4: Select Collection
#### Step 4: Select Collection
Enter the database and collection name from your MongoDB Atlas cluster, that you
want to sync with CrateDB Cloud.

#### Step 5: Select Target Table
1. Specify the target table in your CrateDB Cloud cluster where the data will be synced.
2. MongoDB records will be inserted into an object column called `document`.
3. Select the object type for the column:
- **`dynamic`**: Allows indexing and columnar storage for faster querying.
- **`ignored`**: Prevents type conflicts in CrateDB if your source data lacks a strict schema.
#### Step 5: Select Target Table
1. Specify the target table in your CrateDB Cloud cluster where the data will be synced.
2. MongoDB records will be inserted into an object column called `document`.
3. Select the object type for the column:
- **`dynamic`**: Allows indexing and columnar storage for faster querying.
- **`ignored`**: Prevents type conflicts in CrateDB if your source data lacks a strict schema.

:::{note}
If your source data doesn't follow a strict schema, select `ignored` to avoid type conflicts.
However, selecting `dynamic` provides faster query performance by utilizing indexes and columnar storage.
If your source data doesn't follow a strict schema, select `ignored` to avoid type conflicts.
However, selecting `dynamic` provides faster query performance by utilizing indexes and columnar storage.
:::

#### Step 6: Configure Integration Settings
1. Enter a name for the integration.
2. Select the integration mode:
- **Full Load Only**: Imports the data once but doesn’t sync changes.
- **Full Load and CDC**: Imports the data and syncs changes in real-time.
#### Step 6: Configure Integration Settings
1. Enter a name for the integration.
2. Select the integration mode:
- **Full Load Only**: Imports the data once but doesn’t sync changes.
- **Full Load and CDC**: Imports the data and syncs changes in real-time.
- **CDC Only**: Syncs only new changes in real-time without importing existing data.

#### Step 7: Create the Integration
#### Step 7: Create the Integration
Click **Create Integration** to finalize the setup. CrateDB Cloud will now sync
your MongoDB data based on the selected settings.

Expand Down Expand Up @@ -275,5 +275,5 @@ CDC integration:
value.
- **Binary data types** other than UUIDs, which are converted to `TEXT` and
**vectors**, which are converted to `ARRAY`s of numbers.
- The `Decimal128` data type is not supported and is converted to a string, as
- The `Decimal128` data type is not supported and is converted to a string, as
CrateDB does not support a decimal data type.
6 changes: 3 additions & 3 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ A command-line tool to operate your managed clusters.
::::


Do you want to learn about key database drivers and client applications for
Do you want to learn about key database drivers and client applications for
CrateDB, such as CrateDB Admin UI, crash, psql, DataGrip, and DBeaver? Discover
how to configure these tools and explore CrateDB's compatibility with analytics,
ETL, BI, and monitoring solutions.
Expand Down Expand Up @@ -141,15 +141,15 @@ for improvements], and [source code contributions], are always welcome. {fab}`gi


:::{toctree}
:maxdepth: 1
:titlesonly:
:hidden:

Quick Start <tutorials/quick-start>
Services <reference/services>
Import <cluster/import>
Console <cluster/console>
Automation <cluster/automation>
Integrations <cluster/integrations>
Integrations <cluster/integrations/index>
Export <cluster/export>
Backups <cluster/backups>
Manage Cluster <cluster/manage>
Expand Down
4 changes: 3 additions & 1 deletion docs/reference/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,10 @@ clusters in the cloud – without needing to worry about database management.
CrateDB is a distributed, open-source database that combines the performance of
NoSQL with the power and simplicity of standard SQL.

:::{rubric} Table of contents
:::

:::{toctree}
:hidden:
:maxdepth: 1

User Roles <user-roles>
Expand Down