Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve][broker] Optimize ConcurrentOpenLongPairRangeSet by RoaringBitmap #22908

Merged
merged 24 commits into from
Jun 20, 2024
Merged

[improve][broker] Optimize ConcurrentOpenLongPairRangeSet by RoaringBitmap #22908

merged 24 commits into from
Jun 20, 2024

Conversation

dao-jun
Copy link
Member

@dao-jun dao-jun commented Jun 14, 2024 •
edited
Loading

Fixes #22866

Motivation

Optimize ConcurrentOpenLongPairRangeSet to prevent long GC pause

Modifications

  1. Bump RoaringBitmap version to 1.1.0
  2. Optimize ConcurrentOpenLongPairRangeSet by org.roaringbitmap.RoaringBitSet
  3. Make ConcurrentRoaringBitSet extends org.roaringbitmap.RoaringBitSet to make it concurrency.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

@dao-jun dao-jun self-assigned this Jun 14, 2024
@dao-jun dao-jun added the category/performance Performance issues fix or improvements label Jun 14, 2024
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jun 14, 2024
@dao-jun dao-jun marked this pull request as ready for review June 15, 2024 15:27
@dao-jun dao-jun closed this Jun 15, 2024
@dao-jun dao-jun reopened this Jun 15, 2024
@lhotari
Copy link
Member

lhotari commented Jun 16, 2024

Please upgrade RoaringBitmap to latest version 1.0.6. it contains useful improvements such as RoaringBitmap/RoaringBitmap#713.

@dao-jun
Copy link
Member Author

dao-jun commented Jun 16, 2024

Please upgrade RoaringBitmap to latest version 1.0.6. it contains useful improvements such as RoaringBitmap/RoaringBitmap#713.

Thanks for reminding, I'll create another PR to do the upgrading

@dao-jun dao-jun marked this pull request as draft June 16, 2024 15:37
@dao-jun dao-jun marked this pull request as ready for review June 16, 2024 15:46
@dao-jun dao-jun closed this Jun 16, 2024
@dao-jun dao-jun reopened this Jun 16, 2024
@lhotari
Copy link
Member

lhotari commented Jun 16, 2024

Please upgrade RoaringBitmap to latest version 1.0.6. it contains useful improvements such as RoaringBitmap/RoaringBitmap#713.

Thanks for reminding, I'll create another PR to do the upgrading

The new features could be useful in this PR. That's why it could be useful to simply do in this PR.

@dao-jun
Copy link
Member Author

dao-jun commented Jun 17, 2024

@lhotari @mattisonchao @shibd PTAL

@dao-jun
Copy link
Member Author

dao-jun commented Jun 20, 2024

merging...

@dao-jun dao-jun merged commit 5b1f653 into apache:master Jun 20, 2024
49 of 57 checks passed
@dao-jun dao-jun deleted the dev/optimize_rangeset branch June 20, 2024 07:18
lhotari added a commit that referenced this pull request Jun 20, 2024
@lhotari
[improve][broker] Optimize ConcurrentOpenLongPairRangeSet by Roarin…
ffeb5c8 
…gBitmap (#22908)

(cherry picked from commit 5b1f653)

# Conflicts:
#	distribution/server/src/assemble/LICENSE.bin.txt
#	pom.xml
lhotari pushed a commit that referenced this pull request Jun 20, 2024
@dao-jun @lhotari
[improve][broker] Optimize ConcurrentOpenLongPairRangeSet by Roarin…
6b59c35 
…gBitmap (#22908)

(cherry picked from commit 5b1f653)
lhotari pushed a commit that referenced this pull request Jun 20, 2024
@dao-jun @lhotari
[improve][broker] Optimize ConcurrentOpenLongPairRangeSet by Roarin…
f99040d 
…gBitmap (#22908)

(cherry picked from commit 5b1f653)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 21, 2024
@dao-jun @nikhil-ctds
[improve][broker] Optimize ConcurrentOpenLongPairRangeSet by Roarin…
8792cec 
…gBitmap (apache#22908)

(cherry picked from commit 5b1f653)
(cherry picked from commit f99040d)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 24, 2024
@dao-jun @nikhil-ctds
[improve][broker] Optimize ConcurrentOpenLongPairRangeSet by Roarin…
47caa79 
…gBitmap (apache#22908)

(cherry picked from commit 5b1f653)
(cherry picked from commit f99040d)
lhotari added a commit that referenced this pull request Jun 24, 2024
@lhotari
Revert "[improve][broker] Optimize ConcurrentOpenLongPairRangeSet b…
e45071e 
…y RoaringBitmap (#22908)"

This reverts commit 6b59c35.
lhotari added a commit that referenced this pull request Jun 24, 2024
@lhotari
Revert "[improve][broker] Optimize ConcurrentOpenLongPairRangeSet b…
a7bd307 
…y RoaringBitmap (#22908)"

This reverts commit ffeb5c8.
lhotari added a commit that referenced this pull request Jun 24, 2024
@lhotari
Revert "[improve][broker] Optimize ConcurrentOpenLongPairRangeSet b…
266f98c 
…y RoaringBitmap (#22908)"

This reverts commit f99040d.
@lhotari
Copy link
Member

lhotari commented Jun 24, 2024 •
edited
Loading

I have reverted the changes in branch-3.0 branch-3.2 and branch-3.3 due to regressions referred in #22966

lhotari added a commit to lhotari/pulsar that referenced this pull request Jun 24, 2024
@lhotari
[fix] Revert "[improve][broker] Optimize `ConcurrentOpenLongPairRange…
310b234 
…Set` by RoaringBitmap (apache#22908)"

This reverts commit 5b1f653.
@lhotari lhotari mentioned this pull request Jun 24, 2024
Merged
4 tasks
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 25, 2024
@dao-jun @nikhil-ctds
[improve][broker] Optimize ConcurrentOpenLongPairRangeSet by Roarin…
a25c6cd 
…gBitmap (apache#22908)

(cherry picked from commit 5b1f653)
(cherry picked from commit f99040d)
@dao-jun dao-jun mentioned this pull request Jun 25, 2024
Merged
15 tasks
merlimat pushed a commit that referenced this pull request Jun 25, 2024
@lhotari
[revert] "[improve][broker] Optimize ConcurrentOpenLongPairRangeSet…
2da4ee8 
… by RoaringBitmap (#22908)" (#22968)
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 27, 2024
@lhotari @nikhil-ctds
Revert "[improve][broker] Optimize ConcurrentOpenLongPairRangeSet b…
f800bcb 
…y RoaringBitmap (apache#22908)"

This reverts commit f99040d.

(cherry picked from commit 266f98c)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 1, 2024
@dao-jun @srinath-ctds
[improve][broker] Optimize ConcurrentOpenLongPairRangeSet by Roarin…
f11db32 
…gBitmap (apache#22908)

(cherry picked from commit 5b1f653)
(cherry picked from commit f99040d)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 1, 2024
@lhotari @srinath-ctds
Revert "[improve][broker] Optimize ConcurrentOpenLongPairRangeSet b…
245b197 
…y RoaringBitmap (apache#22908)"

This reverts commit f99040d.

(cherry picked from commit 266f98c)
@lhotari lhotari mentioned this pull request Jul 5, 2024
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lhotari lhotari lhotari approved these changes

@mattisonchao mattisonchao Awaiting requested review from mattisonchao

@shibd shibd Awaiting requested review from shibd

@codelipenghui codelipenghui Awaiting requested review from codelipenghui

@merlimat merlimat Awaiting requested review from merlimat

@hangc0276 hangc0276 Awaiting requested review from hangc0276

@eolivelli eolivelli Awaiting requested review from eolivelli

@shoothzj shoothzj Awaiting requested review from shoothzj

Assignees

@dao-jun dao-jun

Labels
category/performance Performance issues fix or improvements doc-not-needed Your PR changes do not impact docs ready-to-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

High GC pause cases high publish latency
2 participants
@dao-jun @lhotari