Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v23.3.x] Wire transform::logging::manager into transform API, rpc client #16663

Merged
merged 14 commits into from
Feb 23, 2024
Merged

[v23.3.x] Wire transform::logging::manager into transform API, rpc client #16663

merged 14 commits into from
Feb 23, 2024

Conversation

oleiman
Copy link
Member

@oleiman oleiman commented Feb 21, 2024 •
edited
Loading

Backport of PR #16485

Closes #16570
Closes #16572

@oleiman
xform/rpc/client: Add methods to manage logging topic
dfcc5b1 
rpc::create_transform_logs_topic
  - create model::transform_log_internal_topic
  - subject to existing rpc::client retry policy, 5x max, exponential backoff
    from 100ms
  - partition count reflects config::default_topic_partitions
  - batch_max_bytes reflects config::kafka_batch_max_bytes
  - topic cannot be produced to or deleted via Kafka API

Also adds condition to cluster::health_manager to bump topic replication
if node count increases from 1 to 3.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
(cherry picked from commit 9e357d1)
@oleiman
xform/logging/io: Change json_batch::name to iobuf
9019cbd 
Previously model::transform_name. Building the iobuf here simplifies
downstream record batching logic slightly.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
(cherry picked from commit 6463799)
@oleiman
xform/log_manager: Wake up flusher in interval config watcher
0a6e045 
This way the new value gets picked up by the condition variable right away.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
(cherry picked from commit 91ce99e)
@oleiman
xform/logging: Introduce transform::logging::errc
7595843 
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
(cherry picked from commit c41eaa1)
@oleiman oleiman added this to the v23.3.x-next milestone Feb 21, 2024
@oleiman oleiman added the kind/backport PRs targeting a stable branch label Feb 21, 2024
@oleiman oleiman self-assigned this Feb 21, 2024
@github-actions github-actions bot added area/redpanda area/wasm WASM Data Transforms labels Feb 21, 2024
@oleiman
xform/logging/io: Add client::initialize and update log_manager
3c7102d 
To create the transform logs topic on demand. Also adds code in log_manager.cc
to perform this operation lazily at flush time.

Additionally, update logging::client to return transform::logging::errc
(or a result<> thereof). This becomes increasingly useful for error
reporting as we start hooking things up to the rest of the cluster.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
(cherry picked from commit d59c928)
@oleiman
model/record: Replace comment with static_assert
9e0db1f 
A comment on model::packed_record_batch_header_size reflected a
presumably outdated value for the constant. Was 57, should be 61.

Replaces that comment with a static_assert to avoid similar errors
in future.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
(cherry picked from commit 5280839)
@oleiman
xform/logging/log_manager: Use absl::CHexEscape for control chars
fea0c5f 
This is slightly easier to test and reason about, as well as being
nominally less costly in terms of buffer capacity.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
(cherry picked from commit f036951)
@oleiman
model/record: Add record constructor that computes sizes
a5b6ec7 
```
record(
      record_attributes attributes,
      int64_t timestamp_delta,
      int32_t offset_delta,
      iobuf key,
      iobuf value,
      std::vector<record_header> hdrs)
```

(cherry picked from commit a637233)
@oleiman oleiman force-pushed the vbotbuildovich/backport-16485-v23.3.x-310 branch from 06875e0 to d600c58 Compare February 21, 2024 19:58
@oleiman
Copy link
Member Author

oleiman commented Feb 21, 2024

/dt

@oleiman
xform/logging: Introduce record_batcher
5e9f532 
Encapsulates a storage::record_batch_builder and aims to build up batches
of log records. Batches are kept under a specified maximum size to avoid
data loss. If appending a given kvp would cause a batch to exceed the
specified max, we roll to a new builder, appending the previous batch to
a running collection of completed, size-limited record_batches.

Includes a simple unit test for constructed record_batch sizes.

(cherry picked from commit ddea1e2)
@oleiman
xform/logging: Introduce rpc_client
0696062 
Adapts transform::rpc::client for transform logging purposes.

Implements transform::logging::client.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
(cherry picked from commit 9562bbd)
@oleiman
xform/api: Wire up transform::logging::manager
ea441f1 
Also Add sharded<metadata_cache> field to transform::service
in service of transform::logging::rpc_client.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
(cherry picked from commit 38f99f9)
@oleiman
dt/utils: generate_string_with_control_characters to utf8 module
dd91e9b 
Previously a static method on a SCRAM test.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
(cherry picked from commit ad2c4e6)
@oleiman
dt: Tests for transform logging
2f87f0c 
Includes a slightly modified identity transform that also logs the
incoming record to stderr.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
(cherry picked from commit a7f6ba0)
@oleiman
xform/service: Remove exception handling from _log_manager->start
3b41a9d 
The previous handling was incorrect (awaiting on an invalid future),
this probably shouldn't fail silently, and everything in logging::manager::start
is noexcept anyway.

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
(cherry picked from commit 84dac04)
@oleiman oleiman force-pushed the vbotbuildovich/backport-16485-v23.3.x-310 branch from d600c58 to 3b41a9d Compare February 21, 2024 21:46
@oleiman
Copy link
Member Author

oleiman commented Feb 21, 2024

force push headers 😐

@oleiman
Copy link
Member Author

oleiman commented Feb 21, 2024

/dt

@oleiman
Copy link
Member Author

oleiman commented Feb 22, 2024 •
edited
Loading

CI Failures:

  • build TopicRecoveryTest.test_fast3
  • build AuditLogTestsAppLifecycle.test_drain_on_audit_disabled

@oleiman oleiman marked this pull request as ready for review February 22, 2024 04:55
@oleiman oleiman requested a review from a team as a code owner February 22, 2024 04:55
@oleiman oleiman requested review from savex and removed request for a team February 22, 2024 04:55
@oleiman
Copy link
Member Author

oleiman commented Feb 22, 2024

For anyone reviewing - cherry-picks two PRs (see description). Mostly clean, changes were headers and a couple of wonky textual diffs. Nothing functional (incl tests).

michael-redpanda
michael-redpanda approved these changes Feb 23, 2024
View reviewed changes
Copy link
Contributor

@michael-redpanda michael-redpanda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eyeballed the original PR and this one and appears to be good

@oleiman
Copy link
Member Author

oleiman commented Feb 23, 2024

Thanks @michael-redpanda!

@oleiman oleiman changed the title [v23.3.x] Wire into transform API, rpc client [v23.3.x] Wire transform::logging::manager into transform API, rpc client Feb 23, 2024
@oleiman oleiman merged commit 40600e4 into redpanda-data:v23.3.x Feb 23, 2024
20 checks passed
@piyushredpanda piyushredpanda modified the milestones: v23.3.x-next, v23.3.7 Feb 27, 2024
@oleiman oleiman mentioned this pull request Mar 6, 2024
Closed
This was referenced Mar 13, 2024
Open
Merged
Open
Open
@renovate renovate bot mentioned this pull request Mar 16, 2024
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@michael-redpanda michael-redpanda michael-redpanda approved these changes

@dotnwat dotnwat Awaiting requested review from dotnwat

@rockwotj rockwotj Awaiting requested review from rockwotj

@savex savex Awaiting requested review from savex savex is a code owner automatically assigned from redpanda-data/devprod

Assignees

@oleiman oleiman

Labels
area/redpanda area/wasm WASM Data Transforms kind/backport PRs targeting a stable branch
Projects
None yet
Milestone
v23.3.7
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@oleiman @michael-redpanda @piyushredpanda