Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow to config pulsar client allocator out of memory policy #12200

Merged
merged 4 commits into from
Oct 20, 2021
Merged

Allow to config pulsar client allocator out of memory policy #12200

merged 4 commits into from
Oct 20, 2021

Conversation

shoothzj
Copy link
Member

@shoothzj shoothzj commented Sep 26, 2021 •
edited

Motivation

When the client jvm direct memory is full, we don't want pulsar client to use heap memory which may lead to a Heap OutOfMemoryError. In this scene, we want the client allocate memory failure until the direct memory is free enough.

Modifications

Make the OutOfMemoryPolicy configurely.

Documentation

  • no-need-doc

    Currently, this part has no doc, I will start a new PR to add these below configs to our document.
    image

@shoothzj
Copy link
Member Author

@eolivelli, @codelipenghui, @BewareMyPower, @sijie, @hangc0276, @merlimat - PTAL, thanks.

@shoothzj
Copy link
Member Author

/pulsarbot run-failure-checks

@codelipenghui codelipenghui added this to the 2.9.0 milestone Sep 26, 2021
@codelipenghui codelipenghui added release/2.8.2 type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages labels Sep 26, 2021
@shoothzj
Copy link
Member Author

/pulsarbot run-failure-checks

eolivelli
eolivelli requested changes Sep 26, 2021
View reviewed changes
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add an unit test as usual

Thanks

@Anonymitaet Anonymitaet added the doc-required Your PR changes impact docs and you will update later. label Sep 30, 2021
@Anonymitaet
Copy link
Member

Thanks for your contribution. Please do not forget to update docs later. And you can ping me to review the docs, thanks.

@eolivelli eolivelli modified the milestones: 2.9.0, 2.10.0 Oct 6, 2021
@shoothzj
Copy link
Member Author

shoothzj commented Oct 8, 2021

@eolivelli PTAL

@shoothzj
Copy link
Member Author

shoothzj commented Oct 8, 2021

Thanks for your contribution. Please do not forget to update docs later. And you can ping me to review the docs, thanks.

Currently, this part has no doc, I think start a new PR will be better.

@shoothzj
Copy link
Member Author

shoothzj commented Oct 9, 2021

/pulsarbot run-failure-checks

3 similar comments
@shoothzj
Copy link
Member Author

shoothzj commented Oct 9, 2021

/pulsarbot run-failure-checks

@shoothzj
Copy link
Member Author

/pulsarbot run-failure-checks

@shoothzj
Copy link
Member Author

/pulsarbot run-failure-checks

315157973
315157973 approved these changes Oct 10, 2021
View reviewed changes
Copy link
Contributor

@315157973 315157973 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not convert the string to enum? In this way, even if new items are added to the enum, the code here does not need to be modified

@shoothzj
Copy link
Member Author

Why not convert the string to enum? In this way, even if new items are added to the enum, the code here does not need to be modified

You are right. Enum will be better.

@codelipenghui
Copy link
Contributor

@eolivelli Please help review PR again.

@Anonymitaet
Copy link
Member

@shoothzj double check: this PR is labeled with 2.8.2, should the doc be added to both the 2.8.2 doc set and master?

eolivelli
eolivelli approved these changes Oct 19, 2021
View reviewed changes
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eolivelli eolivelli merged commit 3641f29 into apache:master Oct 20, 2021
zeo1995 pushed a commit to zeo1995/pulsar that referenced this pull request Oct 20, 2021
Merge branch 'up/master' into website/branch-2.7.2-chapter-3
88ac2a0 
* up/master:
  [pulsar-java-client] Auto-recovery after exception like out of direct memory (apache#12170)
  Allow to config pulsar client allocator out of memory policy (apache#12200)
  [Transaction] Fix bugs, Exception thrower by TB::appendBufferToTxn must be ManagedLedgerException.  (apache#12376)
  Bumped version to 2.10.0-SNAPSHOT (apache#12285)
  [docs][Website] add docs of broker entry metadata (apache#12404)
  [C++] Use weak ref to ClientConnection for timeout task (apache#12409)
  fix windows test path probleam (apache#12398)
  [website][upgrade]feat: home page (apache#12383)
  [docs] [ISSUE 11526] Update the description of `topic` (apache#12375)
  [Docs] Add document label check robot (apache#12371)
  [Admin] Get schema validation enforce add applied. (apache#12349)
  [Doc] add scope explanations (apache#12380)
  Fix java doc for MultipleListenerValidator (apache#12389)
zeo1995 pushed a commit to zeo1995/pulsar that referenced this pull request Oct 20, 2021
Merge branch 'website/branch-2.7.2-chapter-3' into website/branch-2.7…
987d7e8 
….2-chapter-4

* website/branch-2.7.2-chapter-3:
  Update the nesting of forms in the document
  [pulsar-java-client] Auto-recovery after exception like out of direct memory (apache#12170)
  Allow to config pulsar client allocator out of memory policy (apache#12200)
  [Transaction] Fix bugs, Exception thrower by TB::appendBufferToTxn must be ManagedLedgerException.  (apache#12376)
  Bumped version to 2.10.0-SNAPSHOT (apache#12285)
  [docs][Website] add docs of broker entry metadata (apache#12404)
  [C++] Use weak ref to ClientConnection for timeout task (apache#12409)
  fix windows test path probleam (apache#12398)
  [website][upgrade]feat: home page (apache#12383)
  [docs] [ISSUE 11526] Update the description of `topic` (apache#12375)
  [Docs] Add document label check robot (apache#12371)
  [Admin] Get schema validation enforce add applied. (apache#12349)
  [Doc] add scope explanations (apache#12380)
  Fix java doc for MultipleListenerValidator (apache#12389)
codelipenghui pushed a commit that referenced this pull request Oct 21, 2021
@shoothzj @codelipenghui
Allow to config pulsar client allocator out of memory policy (#12200)
b018bd8 
(cherry picked from commit 3641f29)
@codelipenghui codelipenghui added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Oct 21, 2021
@shoothzj shoothzj deleted the pulsar-client-allocate-policy branch October 24, 2021 10:47
@Anonymitaet
Copy link
Member

Doc is added here: #12481

@Anonymitaet
Copy link
Member

Confirmed w/ @shoothzj, this feature is not merged in 2.8, so the doc is only added to master.

@Anonymitaet Anonymitaet removed the doc-required Your PR changes impact docs and you will update later. label Oct 27, 2021
eolivelli pushed a commit to eolivelli/pulsar that referenced this pull request Nov 29, 2021
@shoothzj @eolivelli
Allow to config pulsar client allocator out of memory policy (apache#…
a7a1d1e 
…12200)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hangc0276 hangc0276 hangc0276 approved these changes

@eolivelli eolivelli eolivelli approved these changes

@315157973 315157973 315157973 approved these changes

@codelipenghui codelipenghui codelipenghui approved these changes

@merlimat merlimat Awaiting requested review from merlimat

Assignees

@shoothzj shoothzj

Labels
cherry-picked/branch-2.8 Archived: 2.8 is end of life release/2.8.2 type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages
Projects
None yet
Milestone
2.10.0
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@shoothzj @Anonymitaet @codelipenghui @hangc0276 @eolivelli @315157973