Skip to content

Commit

Permalink
Merge pull request #711 from Altinity/subkanthi-patch-1
Browse files Browse the repository at this point in the history
Update production_setup.md for incremental snapshot size
  • Loading branch information
subkanthi committed Jul 23, 2024
2 parents faa2ac9 + 2cb50cc commit 353a4b4
Showing 1 changed file with 7 additions and 1 deletion.
8 changes: 7 additions & 1 deletion doc/production_setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,14 +45,20 @@ in terms of number of elements the queue can hold and the maximum size of the qu

## Snapshots (Out of Memory)

1. The following parameters might be useful to reduce the memory usage of the connector during the snapshotting phase.
The following parameters might be useful to reduce the memory usage of the connector during the snapshotting phase.


**Reduce the number of tables**: Use the `snapshot.include.collection.list` to filter include only the necessary tables
as part of the snapshot which will reduce the data retrieved from source.

**snapshot.fetch.size**: This defines the maximum number of rows that are fetched as part of the batch from the database.
Use a lower number to reduce memory usage.
**Incremental**
```
incremental.snapshot.chunk.size
Default value: 1024
The maximum number of rows that the connector fetches and reads into memory when it retrieves an incremental snapshot chunk. Increasing the chunk size provides greater efficiency, because the snapshot runs fewer snapshot queries of a greater size. However, larger chunk sizes also require more memory to buffer the snapshot data. Adjust the chunk size to a value that provides the best performance in your environment.
```

**snapshot.max.threads**: Increase this number from 1 to a higher value to enable parallel snapshotting.

0 comments on commit 353a4b4

Please sign in to comment.