Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KAFKA-6661: Ensure sink connectors don’t resume consumer when task is paused #4716

Closed
wants to merge 1 commit into from
Closed

KAFKA-6661: Ensure sink connectors don’t resume consumer when task is paused #4716

wants to merge 1 commit into from

Conversation

rhauch
Copy link
Contributor

@rhauch rhauch commented Mar 15, 2018 •
edited by ewencp
Loading

Changed WorkerSinkTaskContext to only resume the consumer topic partitions when the connector/task is not in the paused state.

The context tracks the set of topic partitions that are explicitly paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks it currently resumes all topic partitions except those that are still explicitly paused in the context. Therefore, the change above should result in the desired behavior.

Several debug statements were added to record when the context is called by the connector.

This can be backported to older releases, since this bug goes back to 0.10 or 0.9.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@rhauch
KAFKA-6661: Ensure sink connectors don’t resume consumer when task is…
d6b3fd2 
… paussed

Changed WorkerSinkTaskContext to only resume the consumer topic partitions when the connector/task is not in the paused state.

The context tracks the set of topic partitions that are explicitly paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks it currently resumes all topic partitions *except* those that are still explicitly paused in the context. Therefore, the change above should result in the desired behavior.

Several debug statements were added to record when the context is called by the connector.
@rhauch
Copy link
Contributor Author

rhauch commented Mar 15, 2018

@ewencp please review.

ewencp
ewencp approved these changes Mar 15, 2018
View reviewed changes
Copy link
Contributor

@ewencp ewencp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM. Going to cherry-pick all the way back to 0.9.0 assuming I don't encounter too much cherry-pick conflicts.

@ewencp ewencp closed this in e7ef719 Mar 15, 2018
ewencp pushed a commit that referenced this pull request Mar 15, 2018
@rhauch @ewencp
KAFKA-6661: Ensure sink connectors don’t resume consumer when task is…
ab0cbe7 
… paused

Changed WorkerSinkTaskContext to only resume the consumer topic partitions when the connector/task is not in the paused state.

The context tracks the set of topic partitions that are explicitly paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks it currently resumes all topic partitions *except* those that are still explicitly paused in the context. Therefore, the change above should result in the desired behavior.

Several debug statements were added to record when the context is called by the connector.

This can be backported to older releases, since this bug goes back to 0.10 or 0.9.

Author: Randall Hauch <rhauch@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #4716 from rhauch/kafka-6661

(cherry picked from commit e7ef719)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
ewencp pushed a commit that referenced this pull request Mar 15, 2018
@rhauch @ewencp
KAFKA-6661: Ensure sink connectors don’t resume consumer when task is…
c544a88 
… paused

Changed WorkerSinkTaskContext to only resume the consumer topic partitions when the connector/task is not in the paused state.

The context tracks the set of topic partitions that are explicitly paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks it currently resumes all topic partitions *except* those that are still explicitly paused in the context. Therefore, the change above should result in the desired behavior.

Several debug statements were added to record when the context is called by the connector.

This can be backported to older releases, since this bug goes back to 0.10 or 0.9.

Author: Randall Hauch <rhauch@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #4716 from rhauch/kafka-6661

(cherry picked from commit e7ef719)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
ewencp pushed a commit that referenced this pull request Mar 15, 2018
@rhauch @ewencp
KAFKA-6661: Ensure sink connectors don’t resume consumer when task is…
4d398e7 
… paused

Changed WorkerSinkTaskContext to only resume the consumer topic partitions when the connector/task is not in the paused state.

The context tracks the set of topic partitions that are explicitly paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks it currently resumes all topic partitions *except* those that are still explicitly paused in the context. Therefore, the change above should result in the desired behavior.

Several debug statements were added to record when the context is called by the connector.

This can be backported to older releases, since this bug goes back to 0.10 or 0.9.

Author: Randall Hauch <rhauch@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #4716 from rhauch/kafka-6661

(cherry picked from commit e7ef719)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
ewencp pushed a commit that referenced this pull request Mar 15, 2018
@rhauch @ewencp
KAFKA-6661: Ensure sink connectors don’t resume consumer when task is…
df5d142 
… paused

Changed WorkerSinkTaskContext to only resume the consumer topic partitions when the connector/task is not in the paused state.

The context tracks the set of topic partitions that are explicitly paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks it currently resumes all topic partitions *except* those that are still explicitly paused in the context. Therefore, the change above should result in the desired behavior.

Several debug statements were added to record when the context is called by the connector.

This can be backported to older releases, since this bug goes back to 0.10 or 0.9.

Author: Randall Hauch <rhauch@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #4716 from rhauch/kafka-6661

(cherry picked from commit e7ef719)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
ewencp pushed a commit that referenced this pull request Mar 15, 2018
@rhauch @ewencp
KAFKA-6661: Ensure sink connectors don’t resume consumer when task is…
1691a7a 
… paused

Changed WorkerSinkTaskContext to only resume the consumer topic partitions when the connector/task is not in the paused state.

The context tracks the set of topic partitions that are explicitly paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks it currently resumes all topic partitions *except* those that are still explicitly paused in the context. Therefore, the change above should result in the desired behavior.

Several debug statements were added to record when the context is called by the connector.

This can be backported to older releases, since this bug goes back to 0.10 or 0.9.

Author: Randall Hauch <rhauch@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #4716 from rhauch/kafka-6661

(cherry picked from commit e7ef719)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
ewencp pushed a commit that referenced this pull request Mar 15, 2018
@rhauch @ewencp
KAFKA-6661: Ensure sink connectors don’t resume consumer when task is…
7fd224d 
… paused

Changed WorkerSinkTaskContext to only resume the consumer topic partitions when the connector/task is not in the paused state.

The context tracks the set of topic partitions that are explicitly paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks it currently resumes all topic partitions *except* those that are still explicitly paused in the context. Therefore, the change above should result in the desired behavior.

Several debug statements were added to record when the context is called by the connector.

This can be backported to older releases, since this bug goes back to 0.10 or 0.9.

Author: Randall Hauch <rhauch@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes #4716 from rhauch/kafka-6661

(cherry picked from commit e7ef719)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
rhauch added a commit to confluentinc/kafka that referenced this pull request Mar 19, 2018
@rhauch
KAFKA-6661: Ensure sink connectors don’t resume consumer when task is…
2cf9580 
… paused

Changed WorkerSinkTaskContext to only resume the consumer topic partitions when the connector/task is not in the paused state.

The context tracks the set of topic partitions that are explicitly paused/resumed by the connector, and when the WorkerSinkTask resumes the tasks it currently resumes all topic partitions *except* those that are still explicitly paused in the context. Therefore, the change above should result in the desired behavior.

Several debug statements were added to record when the context is called by the connector.

This can be backported to older releases, since this bug goes back to 0.10 or 0.9.

Author: Randall Hauch <rhauch@gmail.com>

Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>

Closes apache#4716 from rhauch/kafka-6661

(cherry picked from commit e7ef719)
Signed-off-by: Ewen Cheslack-Postava <me@ewencp.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ewencp ewencp ewencp approved these changes

Assignees
No one assigned
Labels
connect
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@rhauch @ewencp @kkonstantine