Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IGNITE-18574 CDC: add documentation about Kafka request timeouts #10488

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

IGNITE-18574 CDC: add documentation about Kafka request timeouts #10488

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d621f71
IGNITE-18574 CDC: add documentation about Kafka request timeouts
shishkovilja Apr 10, 2023
560e2ac
IGNITE-18574 Code review fixes.
shishkovilja Apr 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/_docs/extensions-and-integrations/change-data-capture-extensions.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,14 @@ image:../../assets/images/integrations/CDC-ignite2kafka.svg[]
| `kafkaRequestTimeout` | Kafka request timeout in milliseconds. | `3000`
|===

`kafkaRequestTimeout` property sets how much `IgniteToKafkaCdcStreamer` will wait for `KafkaProducer` to finish request.

NOTE: _Too low value_ of this timeout can lead to frequent failures of `IgniteToKafkaCdcStreamer`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me, using NOTE is enough for the emphasis. No need in using italic font here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point!

because of possible delays in network or on Kafka broker.
Also, proper `KafkaProducer` configuration should be provided, including link:https://kafka.apache.org/documentation/#producerconfigs_request.timeout.ms[request.timeout.ms].
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is a purpose of this new line? Should this sentence be part of the NOTE or not? If not, please add additional empty line between the NOTE paragraph and the sentence.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point!

For more details you should refer to a link:https://kafka.apache.org/documentation/#configuration[configuration]
section of the official Kafka documentation.

=== IgniteToKafkaCdcStreamer Metrics

|===
Expand Down Expand Up @@ -175,6 +183,16 @@ Kafka to Ignite configuration file should contain the following beans that will
| `threadCount` | Count of threads to proceed consumers. Each thread poll records from dedicated partitions in round-robin manner. | 16
|===

`kafkaRequestTimeout` property sets timeouts for a following `KafkaConsumer` operations:

- Poll from event and metadata topics.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually need list of these operations? Is there any kafka request that doesn't use this timeout?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Timeouts for #poll are, of cause, is on the surface. But I think, that we can at least tell about request types, that they are affected by a timeout, because they can (unlike #poll) raise timeout exceptions.

- Offsets commit.
- Closing of a consumer.
- Offsets request (during metadata updates).
- Metadata topic partitions request (during metadata updater start).

NOTE: _Too low value_ of `kafkaRequestTimeout` can lead to frequent failures of `kafka-to-ignite.sh` because of possible delays in network or on Kafka broker. Also, proper `KafkaConsumer` configuration should be provided, including link:https://kafka.apache.org/documentation/#consumerconfigs_request.timeout.ms[request.timeout.ms]. For more details you should refer to a link:https://kafka.apache.org/documentation/#configuration[configuration] section of the official Kafka documentation.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment for font.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point!


==== Logging

`kafka-to-ignite.sh` uses the same logging configuration as the Ignite node does. The only difference is that the log is written in the "kafka-ignite-streamer.log" file.
Expand Down