Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@

## Snapshot {#snapshot}

Once setup is complete, we enter the `Snapshot` state. `Snapshot`, `Initial Snapshot` and `Initial Load` (more common) are interchangeable terms. In this state, we take a snapshot of the source MongoDB collections and load them into ClickHouse. Retention setting for the oplog should account for initial load time. The pipe will also enter the `Snapshot` state when a resync is triggered or when new tables are added to an existing pipe.
Once setup is complete, we enter the `Snapshot` state (unless it's a CDC-only pipe, which would transition to `Running`). `Snapshot`, `Initial Snapshot` and `Initial Load` (more common) are interchangeable terms. In this state, we take a snapshot of the source MongoDB collections and load them into ClickHouse. Retention setting for the oplog should account for initial load time. The pipe will also enter the `Snapshot` state when a resync is triggered or when new tables are added to an existing pipe.

Check warning on line 22 in docs/integrations/data-ingestion/clickpipes/mongodb/lifecycle.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.FutureTense

Instead of future tense 'will also', use present tense.

## Running {#running}

Once the initial load is complete, the pipe enters the `Running` state. This is where the pipe begins `Change-Data Capture`. In this state, we start streaming changes from the source MongoDB cluster to ClickHouse. For information on controlling CDC, see [the doc on controlling CDC](./sync_control).
Once the initial load is complete, the pipe enters the `Running` state (unless it's a snapshot-only pipe, which would transition to `Completed`). This is where the pipe begins `Change-Data Capture`. In this state, we start streaming changes from the source MongoDB cluster to ClickHouse. For information on controlling CDC, see [the doc on controlling CDC](./sync_control).

## Paused {#paused}

Expand All @@ -48,6 +48,10 @@
:::
This state indicates the pipe is in the phase of resync where it is performing an atomic swap of the _resync tables with the original tables. More information on resync can be found in the [resync documentation](./resync).

## Completed {#completed}

This state applies to snapshot-only pipes and indicates that the snapshot has been completed and there's no more work to do.

## Failed {#failed}

If there is an irrecoverable error in the pipe, it will enter the `Failed` state. You can reach out to support or [resync](./resync) your pipe to recover from this state.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@

## Snapshot {#snapshot}

Once setup is complete, we enter the `Snapshot` state. `Snapshot`, `Initial Snapshot` and `Initial Load` (more common) are interchangeable terms. In this state, we take a snapshot of the source MySQL tables and load them into ClickHouse. Retention setting for binary logs should account for initial load time. For more information on initial load, see the [parallel initial load documentation](./parallel_initial_load). The pipe will also enter the `Snapshot` state when a resync is triggered or when new tables are added to an existing pipe.
Once setup is complete, we enter the `Snapshot` state (unless it's a CDC-only pipe, which would transition to `Running`). `Snapshot`, `Initial Snapshot` and `Initial Load` (more common) are interchangeable terms. In this state, we take a snapshot of the source MySQL tables and load them into ClickHouse. Retention setting for binary logs should account for initial load time. For more information on initial load, see the [parallel initial load documentation](./parallel_initial_load). The pipe will also enter the `Snapshot` state when a resync is triggered or when new tables are added to an existing pipe.

Check warning on line 22 in docs/integrations/data-ingestion/clickpipes/mysql/lifecycle.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.FutureTense

Instead of future tense 'will also', use present tense.

## Running {#running}

Once the initial load is complete, the pipe enters the `Running` state. This is where the pipe begins `Change-Data Capture`. In this state, we start reading binary logs from the source database and sync the data to ClickHouse in batches. For information on controlling CDC, see [the doc on controlling CDC](./sync_control).
Once the initial load is complete, the pipe enters the `Running` state (unless it's a snapshot-only pipe, which would transition to `Completed`). This is where the pipe begins `Change-Data Capture`. In this state, we start reading binary logs from the source database and sync the data to ClickHouse in batches. For information on controlling CDC, see [the doc on controlling CDC](./sync_control).

## Paused {#paused}

Expand All @@ -48,6 +48,10 @@
:::
This state indicates the pipe is in the phase of resync where it is performing an atomic swap of the _resync tables with the original tables. More information on resync can be found in the [resync documentation](./resync).

## Completed {#completed}

This state applies to snapshot-only pipes and indicates that the snapshot has been completed and there's no more work to do.

## Failed {#failed}

If there is an irrecoverable error in the pipe, it will enter the `Failed` state. You can reach out to support or [resync](./resync) your pipe to recover from this state.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@

## Snapshot {#snapshot}

Once setup is complete, we enter the `Snapshot` state. `Snapshot`, `Initial Snapshot` and `Initial Load` (more common) are interchangeable terms. In this state, we take a snapshot of the source database tables and load them into ClickHouse. This does not use logical replication, but the replication slot is created at this step, therefore your `max_slot_wal_keep_size` and storage parameters should account for slot growth during initial load. For more information on initial load, see the [parallel initial load documentation](./parallel_initial_load). The pipe will also enter the `Snapshot` state when a resync is triggered or when new tables are added to an existing pipe.
Once setup is complete, we enter the `Snapshot` state (unless it's a CDC-only pipe, which would transition to `Running`). `Snapshot`, `Initial Snapshot` and `Initial Load` (more common) are interchangeable terms. In this state, we take a snapshot of the source database tables and load them into ClickHouse. This does not use logical replication, but the replication slot is created at this step, therefore your `max_slot_wal_keep_size` and storage parameters should account for slot growth during initial load. For more information on initial load, see the [parallel initial load documentation](./parallel_initial_load). The pipe will also enter the `Snapshot` state when a resync is triggered or when new tables are added to an existing pipe.

Check warning on line 22 in docs/integrations/data-ingestion/clickpipes/postgres/lifecycle.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.FutureTense

Instead of future tense 'will also', use present tense.

## Running {#running}

Once the initial load is complete, the pipe enters the `Running` state. This is where the pipe begins `Change-Data Capture`. In this state, we start logical replication from the source database to ClickHouse. For information on controlling CDC, see [the doc on controlling CDC](./sync_control).
Once the initial load is complete, the pipe enters the `Running` state (unless it's a snapshot-only pipe, which would transition to `Completed`). This is where the pipe begins `Change-Data Capture`. In this state, we start logical replication from the source database to ClickHouse. For information on controlling CDC, see [the doc on controlling CDC](./sync_control).

## Paused {#paused}

Expand All @@ -48,6 +48,10 @@
:::
This state indicates the pipe is in the phase of resync where it is performing an atomic swap of the _resync tables with the original tables. More information on resync can be found in the [resync documentation](./resync).

## Completed {#completed}

This state applies to snapshot-only pipes and indicates that the snapshot has been completed and there's no more work to do.

## Failed {#failed}

If there is an irrecoverable error in the pipe, it will enter the `Failed` state. You can reach out to support or [resync](./resync) your pipe to recover from this state.
Expand Down