Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

transform-sdk/rust: borrow output record #16210

Merged
merged 3 commits into from
Jan 26, 2024
Merged

transform-sdk/rust: borrow output record #16210

merged 3 commits into from
Jan 26, 2024

Conversation

rockwotj
Copy link
Contributor

@rockwotj rockwotj commented Jan 21, 2024 •
edited

We only need to borrow the output record. To do that we make the writer
take Into<BorrowedRecord<'_>>, so that it can take any type of record
for transforms (we can now do no copy transforms!).

Since Into<_> is a trait, we can't have a monomorphic trait take a
monomorphic trait as a function argument (see
https://doc.rust-lang.org/reference/items/traits.html#object-safety). So
we wrap our current trait in a struct that handles resolving the trait
argument to the concrete type our write function will take.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.3.x
  • v23.2.x
  • v23.1.x

Release Notes

  • none

@rockwotj
transform-sdk/rust: borrow output record
6cd47a8 
We only need to borrow the output record. To do that we make the writer
take `Into<BorrowedRecord<'_>>`, so that it can take any type of record
for transforms (we can now do no copy transforms!).

Since `Into<_>` is a trait, we can't have a monomorphic trait take a
monomorphic trait as a function argument (see
https://doc.rust-lang.org/reference/items/traits.html#object-safety). So
we wrap our current trait in a struct that handles resolving the trait
argument to the concrete type our write function will take.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
@rockwotj
transform-sdk/rust: reuse buffers
b5c5806 
Reuse input and output buffers for better performance.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
@rockwotj rockwotj marked this pull request as ready for review January 21, 2024 05:18
@github-actions github-actions bot added area/rpk area/wasm WASM Data Transforms labels Jan 21, 2024
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Jan 21, 2024 •
edited

new failures in https://buildkite.com/redpanda/redpanda/builds/44051#018d2aa4-18c0-4c78-9bf4-f22d1fcd2882:

"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Delete==True.SpilloverManifestUploaded==True.TS_Spillover_ManifestDeleted==True"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Read==True.AdjacentSegmentMergerReupload==True.SpilloverManifestUploaded==True"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.SpilloverManifestUploaded==True"

new failures in https://buildkite.com/redpanda/redpanda/builds/44051#018d2aa4-18ba-4a6e-b0d0-a9b62da64ad7:

"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.AdjacentSegmentMergerReupload==True.SpilloverManifestUploaded==True"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Read==True.SpilloverManifestUploaded==True"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Delete==True.SpilloverManifestUploaded==True.TS_Spillover_ManifestDeleted==True"

new failures in https://buildkite.com/redpanda/redpanda/builds/44051#018d2acb-b10e-4f6e-95d1-651675e7fd32:

"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.SpilloverManifestUploaded==True"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Delete==True.SpilloverManifestUploaded==True.TS_Spillover_ManifestDeleted==True"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Read==True.AdjacentSegmentMergerReupload==True.SpilloverManifestUploaded==True"

new failures in https://buildkite.com/redpanda/redpanda/builds/44051#018d2acb-b116-402f-a369-1ef80de16745:

"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.TS_Timequery==True.SpilloverManifestUploaded==True"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Read==True.TS_TxRangeMaterialized==True.SpilloverManifestUploaded==True"

new failures in https://buildkite.com/redpanda/redpanda/builds/44051#018d2acb-b110-4729-9708-fd287cd98d26:

"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Read==True.TS_Timequery==True.SpilloverManifestUploaded==True"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.TS_TxRangeMaterialized==True.SpilloverManifestUploaded==True"

new failures in https://buildkite.com/redpanda/redpanda/builds/44051#018d2acb-b113-4d91-8f9d-33929be24ea6:

"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Delete==True.SpilloverManifestUploaded==True.TS_Spillover_ManifestDeleted==True"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.AdjacentSegmentMergerReupload==True.SpilloverManifestUploaded==True"
"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.S3.test_case=.TS_Read==True.SpilloverManifestUploaded==True"

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Jan 21, 2024 •
edited

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44051#018d2acb-b116-402f-a369-1ef80de16745

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44051#018d2acb-b110-4729-9708-fd287cd98d26

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44318#018d42de-2964-4ec3-9aac-04581d16cefa

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44318#018d42ef-b919-4f4e-b331-b959e8a41931

@rockwotj
Copy link
Contributor Author

/ci-repeat

@rockwotj rockwotj self-assigned this Jan 25, 2024
@rockwotj
Copy link
Contributor Author

/ci-repeat

@rockwotj
transform-sdk/rust: update examples
66b3aed 
Add the same docs we have for Golang, but in Rust.

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
michael-redpanda
michael-redpanda approved these changes Jan 26, 2024
View reviewed changes
Copy link
Contributor

@michael-redpanda michael-redpanda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice! I love learning new things about rust whenever I read rust code :)

@rockwotj
Copy link
Contributor Author

Our good friend CloudStorageTimingStressTest 😞 #15950

@rockwotj rockwotj merged commit 48f29ec into redpanda-data:dev Jan 26, 2024
22 of 25 checks passed
@rockwotj rockwotj deleted the rust-borrow branch January 26, 2024 01:26
@vbotbuildovich
Copy link
Collaborator

/backport v23.3.x

@vbotbuildovich vbotbuildovich mentioned this pull request Jan 26, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@michael-redpanda michael-redpanda michael-redpanda approved these changes

@twmb twmb Awaiting requested review from twmb twmb is a code owner

@r-vasquez r-vasquez Awaiting requested review from r-vasquez r-vasquez is a code owner

@gene-redpanda gene-redpanda Awaiting requested review from gene-redpanda gene-redpanda is a code owner

@oleiman oleiman Awaiting requested review from oleiman

Assignees

@rockwotj rockwotj

Labels
area/rpk area/wasm WASM Data Transforms
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@rockwotj @vbotbuildovich @michael-redpanda