Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IGNITE-18515 CDC: add documentation about metadata replication #10481

Merged
merged 2 commits into from
Jan 16, 2023
Merged

IGNITE-18515 CDC: add documentation about metadata replication #10481

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2f6c9e3
IGNITE-18515 CDC: add documentation about metadata replication
shishkovilja Jan 16, 2023
d35b052
IGNITE-18515 WIP
shishkovilja Jan 16, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ link:https://github.com/apache/ignite-extensions/tree/master/modules/cdc-ext[Cha

NOTE: For each cache replicated between clusters link:https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/version/CacheVersionConflictResolver.java[CacheVersionConflictResolver] should be defined.

NOTE: All implementations of CDC replication support replication of link:https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/binary/BinaryType.html[BinaryTypes] and link:https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cdc/TypeMapping.html[TypeMappings]

== Ignite to Java Thin Client CDC streamer
This streamer starts link:thin-clients/java-thin-client[Java Thin Client] which connects to destination cluster.
After connection is established, all changes captured by CDC will be replicated to destination cluster.
Expand Down Expand Up @@ -92,6 +94,8 @@ This way to replicate changes between clusters requires setting up two applicati

NOTE: Instances of `ignite-cdc.sh` with configured streamer should be started on each server node of source cluster to capture all changes.

IMPORTANT: CDC trough Kafka requires _metadata topic with the only one partition_ for sequential ordering guarantees.

image:../../assets/images/integrations/CDC-ignite2kafka.svg[]

=== IgniteToKafkaCdcStreamer Configuration
Expand All @@ -101,8 +105,9 @@ image:../../assets/images/integrations/CDC-ignite2kafka.svg[]
|Name |Description | Default value
| `caches` | Set of cache names to replicate. | null
| `kafkaProperties` | Kafka producer properties. | null
| `topic` | Name of the Kafka topic. | null
| `kafkaParts` | Number of Kafka topic partitions. | null
| `topic` | Name of the Kafka topic for CDC events. | null
| `kafkaParts` | Number of Kafka partitions in CDC events topic. | null
| `metadataTopic` | Name of topic for replication of BinaryTypes and TypeMappings. | null
| `onlyPrimary` | Flag to handle changes only on primary node. | `false`
| `maxBatchSize` | Maximum size of concurrently produced Kafka records. When streamer reaches this number, it waits for Kafka acknowledgements, and then commits CDC offset. | `1024`
| `kafkaRequestTimeout` | Kafka request timeout in milliseconds. | `3000`
Expand Down Expand Up @@ -160,17 +165,19 @@ Kafka to Ignite configuration file should contain the following beans that will
|===
|Name |Description | Default value
| `caches` | Set of cache names to replicate. | null
| `topic` | Name of the Kafka topic. | null
| `kafkaPartsFrom` | Lower Kafka partitions number (inclusive). | -1
| `kafkaPartsTo` | Lower Kafka partitions number (exclusive). | -1
| `topic` | Name of the Kafka topic for CDC events. | null
| `kafkaPartsFrom` | Lower Kafka partitions number (inclusive) for CDC events topic. | -1
| `kafkaPartsTo` | Lower Kafka partitions number (exclusive) for CDC events topic. | -1
| `metadataTopic` | Name of topic for replication of BinaryTypes and TypeMappings. | null
| `metadataConsumerGroup` | Group for `KafkaConsumer`, which polls from metadata topic | ignite-metadata-update-<kafkaPartsFrom>-<kafkaPartsTo>
| `kafkaRequestTimeout` | Kafka request timeout in milliseconds. | `3000`
| `maxBatchSize` | Maximum number of events to be sent to destination cluster in a single batch. | 1024
| `threadCount` | Count of threads to proceed consumers. Each thread poll records from dedicated partitions in round-robin manner. | 16
|===

==== Logging

`kakfa-to-ignite.sh` uses the same logging configuration as the Ignite node does. The only difference is that the log is written in the "kafka-ignite-streamer.log" file.
`kafka-to-ignite.sh` uses the same logging configuration as the Ignite node does. The only difference is that the log is written in the "kafka-ignite-streamer.log" file.

== CacheVersionConflictResolver implementation

Expand Down Expand Up @@ -223,4 +230,4 @@ Configuration is done via Ignite node plugin:
</property>
</bean>
</property>
```
```