IGNITE-18574 CDC: add documentation about Kafka request timeouts #10488

shishkovilja · 2023-01-18T16:10:51Z

Thank you for submitting the pull request to the Apache Ignite.

In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:

The Contribution Checklist

There is a single JIRA ticket related to the pull request.
The web-link to the pull request is attached to the JIRA ticket.
The JIRA ticket has the Patch Available state.
The pull request body describes changes that have been made.
The description explains WHAT and WHY was made instead of HOW.
The pull request title is treated as the final commit message.
The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
A reviewer has been mentioned through the JIRA comments
(see the Maintainers list)
The pull request has been checked by the Teamcity Bot and
the green visa attached to the JIRA ticket (see TC.Bot: Check PR)

Notes

If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com #ignite channel.

timoninmaxim · 2023-04-10T09:35:28Z

docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc

+
+NOTE: _Too low value_ of this timeout can lead to frequent failures of `IgniteToKafkaCdcStreamer`
+because of possible delays in network or on Kafka broker.
+Also, proper `KafkaProducer` configuration should be provided, including link:https://kafka.apache.org/documentation/#producerconfigs_request.timeout.ms[request.timeout.ms].


What is a purpose of this new line? Should this sentence be part of the NOTE or not? If not, please add additional empty line between the NOTE paragraph and the sentence.

Good point!

timoninmaxim · 2023-04-10T09:40:48Z

docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc

@@ -113,6 +113,14 @@ image:../../assets/images/integrations/CDC-ignite2kafka.svg[]
 | `kafkaRequestTimeout` | Kafka request timeout in milliseconds.  | `3000`
 |===

+`kafkaRequestTimeout` property sets how much `IgniteToKafkaCdcStreamer` will wait for `KafkaProducer` to finish request.
+
+NOTE: _Too low value_ of this timeout can lead to frequent failures of `IgniteToKafkaCdcStreamer`


For me, using NOTE is enough for the emphasis. No need in using italic font here.

Good point!

timoninmaxim · 2023-04-10T09:51:53Z

docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc

@@ -175,6 +183,16 @@ Kafka to Ignite configuration file should contain the following beans that will
 | `threadCount` | Count of threads to proceed consumers. Each thread poll records from dedicated partitions in round-robin manner. | 16
 |===

+`kafkaRequestTimeout` property sets timeouts for a following `KafkaConsumer` operations:
+
+- Poll from event and metadata topics.


Do we actually need list of these operations? Is there any kafka request that doesn't use this timeout?

Timeouts for #poll are, of cause, is on the surface. But I think, that we can at least tell about request types, that they are affected by a timeout, because they can (unlike #poll) raise timeout exceptions.

timoninmaxim · 2023-04-10T09:52:32Z

docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc

+- Offsets request (during metadata updates).
+- Metadata topic partitions request (during metadata updater start).
+
+NOTE: _Too low value_ of `kafkaRequestTimeout` can lead to frequent failures of `kafka-to-ignite.sh` because of possible delays in network or on Kafka broker. Also, proper `KafkaConsumer` configuration should be provided, including link:https://kafka.apache.org/documentation/#consumerconfigs_request.timeout.ms[request.timeout.ms]. For more details you should refer to a link:https://kafka.apache.org/documentation/#configuration[configuration] section of the official Kafka documentation.
