transform: prevent double processor starts #15933
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rockwotj
force-pushed
the
ref-counting-sucks
branch
from
406a35f
to
664381e
Compare
There is an edge case when the wasm VM immediately halts that we can get double starts because the task is "failed", but that can cause something else to come along in `transform::manager::start_processor` and start it again, which messes up our ref counting in the shared wasm engine. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
Double starts can mess up our ref counting as seen in the previous commit. Guard against this in the future by mirroring stop and checking the abort source. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
rockwotj
force-pushed
the
ref-counting-sucks
branch
from
664381e
to
497785a
Compare
dotnwat
approved these changes
Jan 4, 2024
|
/backport v23.3.x |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It's possible to get a double start of the processor if the wasm VM immediately stops, as we check to see if the task is running in the manager to decide if we start a processor or not. Fix that by correcting the predicate to be "have I called start without calling stop". Since the consequence of messing up reference counting is a crash, I've also added another cautionary check to start to prevent double starts.
I will followup with some ducktape tests around failing modules to hopefully check this kind of thing in the future.
Backports Required
Release Notes
Bug Fixes