Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added support for changing tx topic properties #16797

Merged
merged 2 commits into from
Mar 5, 2024
Merged

Added support for changing tx topic properties #16797

merged 2 commits into from
Mar 5, 2024

Conversation

mmaslankaprv
Copy link
Member

@mmaslankaprv mmaslankaprv commented Feb 29, 2024 •
edited

Added possibility to change the __kafka_internal/tx topic properties.
The topic properties are reconciled with cluster::tx_topic_manager.

Previously changing configuration properties related with tx topic
had no effect despite being a properties that did not require restart.

Fixes: #16795

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.3.x
  • v23.2.x
  • v23.1.x

Release Notes

Improvements

  • Added ability to change transactional manage topic properties

@mmaslankaprv
c/tx_gateway: extracted tx manager topic creation logic
b5ba68e 
Extracted topic manager topic creation logic to separate component. The
component is going to be responsible for managing the tx_manager topic
and its properties.

Signed-off-by: Michal Maslanka <michal@redpanda.com>
@mmaslankaprv mmaslankaprv marked this pull request as ready for review February 29, 2024 11:06
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Feb 29, 2024 •
edited

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/45491#018df4cc-0f80-4199-a9f0-eba8d8382020

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/45684#018e0e04-9075-4201-a4ad-73eb8d102443

bharathv
bharathv reviewed Feb 29, 2024
View reviewed changes
src/v/cluster/tx_topic_manager.cc Outdated Show resolved Hide resolved
src/v/cluster/tx_topic_manager.cc
*/
auto u = _reconciliation_mutex.get_units();

vlog(txlog.trace, "Reconciling tx manager topic properties");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this happen only on one node? (controller leader?), otherwise we will as many updates as # nodes in the cluster?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i was thinking about this, the topic update property is idempotent. Those redundant topic updates will not negatively impact the cluster. If we would want to do it only on controller leader we would need additional complexity to check for the leadership changes. Also this operation will be pretty rare.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, a hack is probably to add some jitter before checking for reconciliation (like voter priority) but thats too much for this code I guess :-)

@mmaslankaprv
c/tx_topic_manager: added support for changing tx topic properties
022b16f 
Added possibility to change the `__kafka_internal/tx` topic properties.
The topic properties are reconciled with `cluster::tx_topic_manager`.

Previously changing configuration properties related with `tx` topic
had no effect despite being a properties that did not require restart.

Signed-off-by: Michal Maslanka <michal@redpanda.com>
@vbotbuildovich
Copy link
Collaborator

new failures in https://buildkite.com/redpanda/redpanda/builds/45610#018e0956-025b-4287-b363-9241b0a2ae2c:

"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_volume"

bharathv
bharathv approved these changes Mar 4, 2024
View reviewed changes
src/v/cluster/tx_topic_manager.cc
*/
auto u = _reconciliation_mutex.get_units();

vlog(txlog.trace, "Reconciling tx manager topic properties");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, a hack is probably to add some jitter before checking for reconciliation (like voter priority) but thats too much for this code I guess :-)

@mmaslankaprv
Copy link
Member Author

/ci-repeat 1

@mmaslankaprv mmaslankaprv merged commit a6c345a into redpanda-data:dev Mar 5, 2024
16 checks passed
@vbotbuildovich
Copy link
Collaborator

/backport v23.3.x

@vbotbuildovich
Copy link
Collaborator

/backport v23.2.x

@vbotbuildovich
Copy link
Collaborator

Failed to create a backport PR to v23.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-16797-v23.2.x-762 remotes/upstream/v23.2.x
git cherry-pick -x b5ba68e1b92fe65b37a84ba13e1ae81a03f16141 022b16f9a31ae660baf994d6ea2b27ffb8adbf18

Workflow run logs.

@vbotbuildovich
Copy link
Collaborator

Failed to create a backport PR to v23.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-16797-v23.3.x-902 remotes/upstream/v23.3.x
git cherry-pick -x b5ba68e1b92fe65b37a84ba13e1ae81a03f16141 022b16f9a31ae660baf994d6ea2b27ffb8adbf18

Workflow run logs.

This was referenced Mar 5, 2024
Open
Closed
@mmaslankaprv mmaslankaprv mentioned this pull request Mar 8, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bharathv bharathv bharathv approved these changes

Assignees
No one assigned
Labels
area/redpanda
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Support TS and AlterConfig for kafka_internal topics
3 participants
@mmaslankaprv @vbotbuildovich @bharathv