From 2f6c9e3b8e46d449f5fc30b0a826ee0d40a79086 Mon Sep 17 00:00:00 2001 From: Ilya Shishkov Date: Mon, 16 Jan 2023 14:42:16 +0300 Subject: [PATCH 1/2] IGNITE-18515 CDC: add documentation about metadata replication --- .../change-data-capture-extensions.adoc | 21 ++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc b/docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc index b9e248cf05343..a7af36df94670 100644 --- a/docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc +++ b/docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc @@ -25,6 +25,8 @@ link:https://github.com/apache/ignite-extensions/tree/master/modules/cdc-ext[Cha NOTE: For each cache replicated between clusters link:https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/version/CacheVersionConflictResolver.java[CacheVersionConflictResolver] should be defined. +NOTE: All implementations of CDC replication support replication of link:https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/binary/BinaryType.html[BinaryTypes] and link:https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cdc/TypeMapping.html[TypeMappings] + == Ignite to Java Thin Client CDC streamer This streamer starts link:thin-clients/java-thin-client[Java Thin Client] which connects to destination cluster. After connection is established, all changes captured by CDC will be replicated to destination cluster. @@ -92,6 +94,8 @@ This way to replicate changes between clusters requires setting up two applicati NOTE: Instances of `ignite-cdc.sh` with configured streamer should be started on each server node of source cluster to capture all changes. +NOTE: CDC trough Kafka requires _metadata topic with the only one partition_ for sequential ordering guaranties. + image:../../assets/images/integrations/CDC-ignite2kafka.svg[] === IgniteToKafkaCdcStreamer Configuration @@ -101,8 +105,9 @@ image:../../assets/images/integrations/CDC-ignite2kafka.svg[] |Name |Description | Default value | `caches` | Set of cache names to replicate. | null | `kafkaProperties` | Kafka producer properties. | null -| `topic` | Name of the Kafka topic. | null -| `kafkaParts` | Number of Kafka topic partitions. | null +| `topic` | Name of the Kafka topic for CDC events. | null +| `kafkaParts` | Number of Kafka partitions in CDC events topic. | null +| `metadataTopic` | Name of topic for replication of BinaryTypes and TypeMappings. | null | `onlyPrimary` | Flag to handle changes only on primary node. | `false` | `maxBatchSize` | Maximum size of concurrently produced Kafka records. When streamer reaches this number, it waits for Kafka acknowledgements, and then commits CDC offset. | `1024` | `kafkaRequestTimeout` | Kafka request timeout in milliseconds. | `3000` @@ -160,9 +165,11 @@ Kafka to Ignite configuration file should contain the following beans that will |=== |Name |Description | Default value | `caches` | Set of cache names to replicate. | null -| `topic` | Name of the Kafka topic. | null -| `kafkaPartsFrom` | Lower Kafka partitions number (inclusive). | -1 -| `kafkaPartsTo` | Lower Kafka partitions number (exclusive). | -1 +| `topic` | Name of the Kafka topic for CDC events. | null +| `kafkaPartsFrom` | Lower Kafka partitions number (inclusive) for CDC events topic. | -1 +| `kafkaPartsTo` | Lower Kafka partitions number (exclusive) for CDC events topic. | -1 +| `metadataTopic` | Name of topic for replication of BinaryTypes and TypeMappings. | null +| `metadataConsumerGroup` | Group for `KafkaConsumer`, which polls from metadata topic | ignite-metadata-update-- | `kafkaRequestTimeout` | Kafka request timeout in milliseconds. | `3000` | `maxBatchSize` | Maximum number of events to be sent to destination cluster in a single batch. | 1024 | `threadCount` | Count of threads to proceed consumers. Each thread poll records from dedicated partitions in round-robin manner. | 16 @@ -170,7 +177,7 @@ Kafka to Ignite configuration file should contain the following beans that will ==== Logging -`kakfa-to-ignite.sh` uses the same logging configuration as the Ignite node does. The only difference is that the log is written in the "kafka-ignite-streamer.log" file. +`kafka-to-ignite.sh` uses the same logging configuration as the Ignite node does. The only difference is that the log is written in the "kafka-ignite-streamer.log" file. == CacheVersionConflictResolver implementation @@ -223,4 +230,4 @@ Configuration is done via Ignite node plugin: -``` \ No newline at end of file +``` From d35b05224559a121b5cc46398d0efeb022c45a9c Mon Sep 17 00:00:00 2001 From: Ilya Shishkov Date: Mon, 16 Jan 2023 14:50:59 +0300 Subject: [PATCH 2/2] IGNITE-18515 WIP --- .../change-data-capture-extensions.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc b/docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc index a7af36df94670..736f754eb3f45 100644 --- a/docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc +++ b/docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc @@ -94,7 +94,7 @@ This way to replicate changes between clusters requires setting up two applicati NOTE: Instances of `ignite-cdc.sh` with configured streamer should be started on each server node of source cluster to capture all changes. -NOTE: CDC trough Kafka requires _metadata topic with the only one partition_ for sequential ordering guaranties. +IMPORTANT: CDC trough Kafka requires _metadata topic with the only one partition_ for sequential ordering guarantees. image:../../assets/images/integrations/CDC-ignite2kafka.svg[]