transforms: fix offset at deploy time #17590
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/47327#018ea56e-9cf7-425c-ad19-479d8516c26f ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/47327#018ea56e-9cef-431a-8745-40b70b6caa62 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/47341#018ea5f4-8e12-4889-9705-7b19261b2e3e ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/47341#018ea5f4-8e10-43ac-9a90-b311a7ea97f5 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/47390#018eac69-e9b2-4591-a81e-9138bba96f36 |
rockwotj
force-pushed
the
fix-offset-at-deploy-time
branch
2 times, most recently
from
626c86d
to
2df072a
Compare
rockwotj
force-pushed
the
fix-offset-at-deploy-time
branch
from
2df072a
to
01e1586
Compare
rockwotj
force-pushed
the
fix-offset-at-deploy-time
branch
from
c2879df
to
53fd207
Compare
|
Force push: rebase to fix conflicts |
oleiman
reviewed
Apr 4, 2024
oleiman
previously approved these changes
Apr 5, 2024
Comment on lines
+183
to
+189
| return _cond_var | ||
| .wait( | ||
| 1s, | ||
| [this, index, o] { | ||
| auto it = _committed.find(index); | ||
| return it != _committed.end() && it->second >= o; | ||
| }) |
There was a problem hiding this comment.
clang-format
really
went for
it
here!
I rather like it
There was a problem hiding this comment.
seriously. Not how I would have formatted it, but meh
rockwotj
force-pushed
the
fix-offset-at-deploy-time
branch
from
53fd207
to
3278bd0
Compare
|
Force push: drop the serde commit |
In order to support starting from specific points in time during deploys, we add a variant that allows starting from the end of the log or a specific point in time, at which we will resolve to an offset using a timequery. Adding a new field to the end of the struct is backwards compatible, so we are tree to add this new field - the default value will be latest_offset, which is consistent with existing behavior. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
Require that Wasm transform sources can perform time queries to resolve a timestamp to an offset. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
A new change in clang-tidy-17 complains when exceptions are ignored. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
This commits respects the initial position in the metadata, and means that when we start processing records we'll start from the deploy time for newly deployed transforms, while old transforms we will still start at the end of the log. This will remove the need for the song and dance around waiting for transforms to commit before producing records in tests and matches the expectations and papercuts customers have ran into when trying transforms. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
This makes debugging test failures a little easier, as you can narrow down which cond_var timed out. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
Add a ducktape test proving that we no longer need to wait after a deploy to produce records to a data transform. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
rockwotj
force-pushed
the
fix-offset-at-deploy-time
branch
from
3278bd0
to
b0d4393
Compare
|
Force push rebase against dev |
oleiman
approved these changes
Apr 10, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes testing transforms easier as the offset that we start from is relative to the deploy and not when the VM starts. This means you can do:
Without dropping records (there is a 3 second injected delay between deploy and processor start, so today everything in the first 3 seconds is dropped).
Ducktape tests will be written in a followup.
Fixes: CORE-1927, #17370
Backports Required
Release Notes
Improvements
This allows for easier testing of transforms, as one does not have to wait for the VM to boot before producing.