Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][broker] Optimization protobuf code in the bucket delayed tracker #20158

Merged
merged 3 commits into from
Apr 21, 2023

Conversation

coderzc
Copy link
Member

@coderzc coderzc commented Apr 20, 2023

PIP: #16763

Motivation

Optimization protobuf code in the bucket delayed tracker

Modifications

  • Split DelayedMessageIndexBucketSnapshotFormat.proto to DelayedMessageIndexBucketMetadata.proto and DelayedMessageIndexBucketSegment.proto.
  • Use lightproto-maven-plugin to generate DelayedMessageIndexBucketSegment.proto code (Due to lightproto-maven-plugin don't support map type so keep using protobuf-maven-plugin to generate DelayedMessageIndexBucketMetadata.proto code)
  • Reduce data copy in the deserialize
  • Add popToObject and peekTimestamp method in the DelayedIndexQueue to reduce object create

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Apr 20, 2023
@coderzc coderzc self-assigned this Apr 20, 2023
@coderzc coderzc added area/broker type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages labels Apr 20, 2023
@coderzc coderzc force-pushed the improve_bucket_delayed_proto branch from aec64c3 to 4046480 Compare April 20, 2023 15:56
@coderzc coderzc force-pushed the improve_bucket_delayed_proto branch 4 times, most recently from 34d136b to d4633ad Compare April 20, 2023 17:27
@coderzc coderzc force-pushed the improve_bucket_delayed_proto branch from d4633ad to 52dfa5e Compare April 20, 2023 17:29
@codecov-commenter
Copy link

Codecov Report

Merging #20158 (999a8c6) into master (9b72302) will increase coverage by 39.75%.
The diff coverage is 85.38%.

Impacted file tree graph

@@              Coverage Diff              @@
##             master   #20158       +/-   ##
=============================================
+ Coverage     33.17%   72.93%   +39.75%     
- Complexity    12236    31943    +19707     
=============================================
  Files          1499     1868      +369     
  Lines        114413   138431    +24018     
  Branches      12431    15235     +2804     
=============================================
+ Hits          37962   100971    +63009     
+ Misses        71499    29432    -42067     
- Partials       4952     8028     +3076     
Flag Coverage Δ
inttests 24.18% <13.45%> (?)
systests 24.72% <11.69%> (?)
unittests 72.23% <85.38%> (+39.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...e/pulsar/broker/authentication/oidc/JwksCache.java 56.62% <0.00%> (ø)
...thentication/oidc/OpenIDProviderMetadataCache.java 78.48% <0.00%> (ø)
...rg/apache/pulsar/broker/delayed/bucket/Bucket.java 88.70% <ø> (+88.70%) ⬆️
...nt/internal/PulsarClientImplementationBinding.java 62.50% <ø> (+12.50%) ⬆️
...ava/org/apache/pulsar/client/api/MessageIdAdv.java 84.21% <ø> (ø)
...unctions/runtime/kubernetes/KubernetesRuntime.java 38.34% <0.00%> (+38.34%) ⬆️
...nt/impl/PulsarClientImplementationBindingImpl.java 80.00% <28.57%> (+24.82%) ⬆️
...broker/intercept/ManagedLedgerInterceptorImpl.java 75.00% <75.00%> (+75.00%) ⬆️
.../pulsar/broker/delayed/bucket/ImmutableBucket.java 85.71% <79.16%> (+85.71%) ⬆️
...ulsar/client/impl/transaction/TransactionImpl.java 87.20% <80.95%> (+48.00%) ⬆️
... and 15 more

... and 1516 files with indirect coverage changes

@coderzc coderzc merged commit a20d5e9 into apache:master Apr 21, 2023
@coderzc coderzc deleted the improve_bucket_delayed_proto branch April 21, 2023 01:52
coderzc added a commit that referenced this pull request Apr 21, 2023
@RobertIndie RobertIndie added this to the 3.0.0 milestone Apr 21, 2023
@coderzc coderzc modified the milestones: 3.0.0, 3.1.0 Apr 21, 2023
coderzc added a commit that referenced this pull request Apr 23, 2023
poorbarcode pushed a commit that referenced this pull request May 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/broker cherry-picked/branch-3.0 doc-not-needed Your PR changes do not impact docs ready-to-test release/3.0.1 type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants