rpk/transform: introduce logs command #16923

rockwotj · 2024-03-06T22:01:34Z

Fixes: https://github.com/redpanda-data/core-internal/issues/1009, https://github.com/redpanda-data/core-internal/issues/1079

Backports Required

Release Notes

Features

Introduce rpk transform logs NAME to view logs for a transform

rockwotj · 2024-03-07T16:59:31Z

Force push: fixes from manual testing and fixed broker output to be more user friendly

rockwotj · 2024-03-07T18:37:13Z

Force push: Another round of manual testing. Should be good to go now.

vbotbuildovich · 2024-03-07T20:31:42Z

new failures in https://buildkite.com/redpanda/redpanda/builds/45815#018e1a64-7c19-4dd5-a4e8-eb67b3e215ad:

"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_volume"

new failures in https://buildkite.com/redpanda/redpanda/builds/45815#018e1a64-7c1f-4e7d-a3c2-0cccc43d6567:

"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_cc_escaping"

new failures in https://buildkite.com/redpanda/redpanda/builds/45815#018e1a64-7c1c-4ffb-9d6b-90807df1fb35:

"rptest.tests.consumer_group_balancing_test.ConsumerGroupBalancingTest.test_coordinator_nodes_balance"

new failures in https://buildkite.com/redpanda/redpanda/builds/45833#018e1b23-5c87-4a2a-b94d-739b8ff197cb:

"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_volume"

new failures in https://buildkite.com/redpanda/redpanda/builds/45833#018e1b4b-56c1-4197-8d0c-448cce43e921:

"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_volume"

new failures in https://buildkite.com/redpanda/redpanda/builds/45880#018e1edd-8c9a-4c73-bdcf-8a28e4b531d2:

"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_volume"

new failures in https://buildkite.com/redpanda/redpanda/builds/45880#018e1eee-bcb1-4d46-92ee-e22b8e303806:

"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_volume"

new failures in https://buildkite.com/redpanda/redpanda/builds/45906#018e1ff9-078a-4ea7-aa9a-6f91333529d2:

"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_volume"

new failures in https://buildkite.com/redpanda/redpanda/builds/45906#018e2020-de1c-4276-9a5b-785fcb7cb20f:

"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_volume"

new failures in https://buildkite.com/redpanda/redpanda/builds/45906#018e2020-de19-4cec-8861-37248fcc06e5:

"rptest.tests.tx_coordinator_migration_test.TxCoordinatorMigrationTest.test_migrating_tx_manager_coordinator.with_failures=True.upgrade=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/45906#018e2328-4fb4-40e8-b028-4f4d09f0d4f5:

"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_volume"

new failures in https://buildkite.com/redpanda/redpanda/builds/46062#018e33b7-aac8-4489-af46-b2e2baddfa2f:

"rptest.tests.delete_records_test.DeleteRecordsTest.test_delete_records_concurrent_truncations.cloud_storage_enabled=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/46135#018e3ae9-3cb6-4547-994f-c7d028a1c73e:

"rptest.tests.data_transforms_test.DataTransformsLoggingTest.test_logs_volume"

rockwotj · 2024-03-07T20:59:49Z

Force push: fix tests

rockwotj · 2024-03-07T21:13:22Z

Force push: remove some files I didn't mean to add

r-vasquez

🔥

src/go/rpk/pkg/cli/transform/logs.go

rockwotj · 2024-03-07T22:04:03Z

@redpanda-data/documentation asking for a review of the help text for the new rpk transform logs NAME command.

oleiman

core bits lgtm. I'll defer to @r-vasquez for signoff.

(I miss the Gerrit +1)

oleiman · 2024-03-07T19:27:17Z

tests/rptest/tests/data_transforms_test.py

-        self.body = self._rep.get(self.BODY_FIELD, None)
+        body_object = self._rep.get(self.BODY_FIELD, None)
+        if isinstance(body_object, dict):
+            self.body = body_object.get("stringValue", None)


Fun. I knew I should have tried parsing this into the real protobuf. Oh well.

Meh yeah I missed this in the reviews too 🤷

oleiman · 2024-03-07T22:24:52Z

src/v/wasm/wasi.cc

-        for (const iovec_t& vec : iovecs) {
-            try {
-                ffi::array<char> data = mem->translate_array<char>(
+        auto level = fd == 1 ? ss::log_level::info : ss::log_level::warn;


@rockwotj - Thanks for sorting this out btw. I spent a few hours staring at it while you were out and couldn't quite visualize where things ought to go.

dotnwat

lgtm

src/go/rpk/pkg/cli/transform/logs.go

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

rockwotj · 2024-03-13T18:45:17Z

Force push: docs fixes

rockwotj · 2024-03-15T15:55:58Z

Force push: make logs test more robust now that we produce exactly the correct number of logs (see slack thread)

tests/rptest/util.py

oleiman

lgtm

Offsets are 0 indexed, so the last record in the transform should be n - 1. NOTE: It's still possible for this test to be flaky if all the input partitions end up on the same core, as all of the partitions could produce logs at once, overshooting the log buffer. Maybe that is unlikely enough it will never happen. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

This was always recording 0 bytes because there was an unrolled batch, which this method doesn't report. It seems cleaner to just count the bytes that are returned from the builder anyways. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

vbotbuildovich · 2024-03-16T07:04:29Z

/backport v23.3.x

vbotbuildovich · 2024-03-16T07:05:19Z

Failed to create a backport PR to v23.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-16923-v23.3.x-71 remotes/upstream/v23.3.x
git cherry-pick -x 4bb172bb759a7ecbe8ab63c2f9c03e7c6de54d73 f15257eb1eb0403d4413620256e76ba0d0d34a11 6458d3e51ca5f37268d11ea6664fa60c7e061b65 74d0af91d1c8fd24918cd2cb35228aa124ad95ba 2e648a04d969d5191afdc9cdc8dadad36d20a606 c9a0889ac563201e93b082df8a843f2149e12ce0 080964b4170752ab96687163c740dd54b1d735be 74de39c8adfa699103a6faba6d327e94a8afaa63 bd6ed7fa7c5ad908fa29d584bf6171f6d8e0dec7 d03086c7a94c79b31dbbc7b5707e92e7b9f6262e

Workflow run logs.

rockwotj requested review from twmb, r-vasquez and gene-redpanda as code owners March 6, 2024 22:01

github-actions bot added the area/rpk label Mar 6, 2024

rockwotj force-pushed the rpk-transform-logs branch 2 times, most recently from ec477f5 to 778bb82 Compare March 7, 2024 15:19

github-actions bot added area/redpanda area/wasm WASM Data Transforms labels Mar 7, 2024

rockwotj force-pushed the rpk-transform-logs branch 2 times, most recently from 632bd49 to 858fca7 Compare March 7, 2024 16:58

rockwotj requested a review from oleiman March 7, 2024 16:59

twmb removed their request for review March 7, 2024 16:59

rockwotj force-pushed the rpk-transform-logs branch from 858fca7 to 58ab00a Compare March 7, 2024 17:07

oleiman assigned rockwotj Mar 7, 2024

rockwotj force-pushed the rpk-transform-logs branch from 58ab00a to 5e0af4b Compare March 7, 2024 18:36

rockwotj force-pushed the rpk-transform-logs branch from 5e0af4b to 2a4e31e Compare March 7, 2024 20:58

rockwotj force-pushed the rpk-transform-logs branch from 2a4e31e to 51159aa Compare March 7, 2024 21:13

r-vasquez reviewed Mar 7, 2024

View reviewed changes

rockwotj requested a review from a team March 7, 2024 22:03

rockwotj requested a review from r-vasquez March 7, 2024 22:05

oleiman reviewed Mar 7, 2024

View reviewed changes

r-vasquez previously approved these changes Mar 7, 2024

View reviewed changes

dotnwat reviewed Mar 8, 2024

View reviewed changes

rockwotj requested a review from oleiman March 12, 2024 21:31

oleiman previously approved these changes Mar 12, 2024

View reviewed changes

kbatuigas previously approved these changes Mar 13, 2024

View reviewed changes

src/go/rpk/pkg/cli/transform/logs.go Outdated Show resolved Hide resolved

rpk/transform/logs: revamp help text

74de39c

Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

rockwotj dismissed stale reviews from kbatuigas and oleiman via e45f27d March 13, 2024 18:44

rockwotj force-pushed the rpk-transform-logs branch from 1390b3d to e45f27d Compare March 13, 2024 18:44

rockwotj requested review from kbatuigas and oleiman March 13, 2024 18:45

oleiman previously approved these changes Mar 13, 2024

View reviewed changes

rockwotj dismissed oleiman’s stale review via e8c7f66 March 15, 2024 15:55

rockwotj force-pushed the rpk-transform-logs branch from e45f27d to e8c7f66 Compare March 15, 2024 15:55

rockwotj requested a review from oleiman March 15, 2024 15:56

oleiman reviewed Mar 15, 2024

View reviewed changes

tests/rptest/util.py Outdated Show resolved Hide resolved

rockwotj requested a review from oleiman March 15, 2024 16:03

oleiman previously approved these changes Mar 15, 2024

View reviewed changes

rockwotj added 2 commits March 15, 2024 22:22

transform/logging: correct log statement

d03086c

This was always recording 0 bytes because there was an unrolled batch, which this method doesn't report. It seems cleaner to just count the bytes that are returned from the builder anyways. Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>

rockwotj dismissed oleiman’s stale review via d03086c March 16, 2024 03:22

rockwotj force-pushed the rpk-transform-logs branch from e8c7f66 to d03086c Compare March 16, 2024 03:22

rockwotj requested a review from oleiman March 16, 2024 03:23

oleiman approved these changes Mar 16, 2024

View reviewed changes

rockwotj merged commit 3772047 into redpanda-data:dev Mar 16, 2024
24 checks passed

vbotbuildovich mentioned this pull request Mar 16, 2024

[v23.3.x] rpk/transform: introduce logs command #17147

Closed

rockwotj mentioned this pull request Mar 16, 2024

[v23.3.x] rpk/transform: introduce logs command #17148

Merged

rockwotj deleted the rpk-transform-logs branch March 22, 2024 15:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpk/transform: introduce logs command #16923

rpk/transform: introduce logs command #16923

rockwotj commented Mar 6, 2024 •

edited

rockwotj commented Mar 7, 2024

rockwotj commented Mar 7, 2024

vbotbuildovich commented Mar 7, 2024 •

edited

rockwotj commented Mar 7, 2024

rockwotj commented Mar 7, 2024

r-vasquez left a comment

rockwotj commented Mar 7, 2024

oleiman left a comment

oleiman Mar 7, 2024 •

edited

rockwotj Mar 8, 2024

oleiman Mar 7, 2024

dotnwat left a comment

rockwotj commented Mar 13, 2024

rockwotj commented Mar 15, 2024

oleiman left a comment

vbotbuildovich commented Mar 16, 2024

vbotbuildovich commented Mar 16, 2024

rpk/transform: introduce logs command #16923

rpk/transform: introduce logs command #16923

Conversation

rockwotj commented Mar 6, 2024 • edited

Backports Required

Release Notes

Features

rockwotj commented Mar 7, 2024

rockwotj commented Mar 7, 2024

vbotbuildovich commented Mar 7, 2024 • edited

rockwotj commented Mar 7, 2024

rockwotj commented Mar 7, 2024

r-vasquez left a comment

Choose a reason for hiding this comment

rockwotj commented Mar 7, 2024

oleiman left a comment

Choose a reason for hiding this comment

oleiman Mar 7, 2024 • edited

Choose a reason for hiding this comment

rockwotj Mar 8, 2024

Choose a reason for hiding this comment

oleiman Mar 7, 2024

Choose a reason for hiding this comment

dotnwat left a comment

Choose a reason for hiding this comment

rockwotj commented Mar 13, 2024

rockwotj commented Mar 15, 2024

oleiman left a comment

Choose a reason for hiding this comment

vbotbuildovich commented Mar 16, 2024

vbotbuildovich commented Mar 16, 2024

rockwotj commented Mar 6, 2024 •

edited

vbotbuildovich commented Mar 7, 2024 •

edited

oleiman Mar 7, 2024 •

edited