Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-36420: fix a potential race condition getting historical data #255

Merged
merged 1 commit into from Oct 5, 2022
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
13 changes: 11 additions & 2 deletions python/lsst/ts/salobj/sal_info.py
Expand Up @@ -756,9 +756,19 @@ def _blocking_create_consumer(self) -> None:

self._consumer = Consumer(
{
"group.id": get_random_string(),
"bootstrap.servers": self.kafka_broker_addr,
# Make sure every consumer is in its own consumer group,
# since each consumer acts independently.
"group.id": get_random_string(),
# Require explicit topic creation, so we can control
# topic configuration, and to reduce startup latency.
"allow.auto.create.topics": False,
# Protect against a race condition in the on_assign callback:
# if the broker purges data while the on_assign callback
# is assigning the desired historical data offset,
# data might no longer exist at that offset; in that case
# case read from the earliest data.
"auto.offset.reset": "earliest",
}
)

Expand Down Expand Up @@ -921,7 +931,6 @@ def _blocking_on_assign_callback(
partition.offset = desired_offset
history_offsets[partition.topic] = max_offset - 1

# Sort of like seek, but simpler
self._consumer.assign(partitions)
# print(f"{self.index} assign:")
# for partition in partitions:
Expand Down