Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recover the offset from HDFS, even if topic name is not present in storage path #528

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

hariprasad-k
Copy link

Problem

Exactly-once semantics should also work without topic name included in the path.

Solution

The function that recover offsets from file names will use the correct path by taking into account a new configuration flag path_include_topicname introduced with confluentinc/kafka-connect-storage-common#126.

Further, the backward compatibility of appending the topic name is retained by default value true for path_include_topicname.

Does this solution apply anywhere else?
  • yes
  • no
If yes, where?

This change is necessary to retain EOS semantics to support flexible storage partitioning scheme as proposed here: confluentinc/kafka-connect-storage-common#126

Test Strategy

Testing done:
  • Unit tests
  • Integration tests
  • System tests
  • Manual tests

Release Plan

…ng or removing the topic name from the path.

Fix a test that failed after the patch
@hariprasad-k hariprasad-k requested a review from a team as a code owner November 3, 2020 17:23
@ghost
Copy link

ghost commented Nov 3, 2020

@confluentinc It looks like @hariprasad-k just signed our Contributor License Agreement. 👍

Always at your service,

clabot

@hariprasad-k
Copy link
Author

Fixes #515

@cla-assistant
Copy link

cla-assistant bot commented Aug 27, 2023

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant