issue#9345 -- bug fix for RuntimeUtils.getCmd not setting pending-async-requests #9349
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
codelipenghui
requested review from
srkukarni,
jerrypeng,
sijie,
wolfstudy and
zymap
|
@nlu90 Please also help review this PR |
|
/pulsarbot run-failure-checks |
|
Please merge the newest master and rerun tests |
csthomas1
force-pushed
the
bugfix_issue_9345_max_pending_async_requests
branch
from
5b2b4cf
to
e9d5078
Compare
sijie
approved these changes
Feb 2, 2021
jerrypeng
approved these changes
Feb 2, 2021
|
@csthomas1 thanks for fix! |
csthomas1
force-pushed
the
bugfix_issue_9345_max_pending_async_requests
branch
from
9f32c2c
to
bbcc5af
Compare
eolivelli
approved these changes
Feb 2, 2021
…pending-async-requests argument supported by JavaInstanceStarter
csthomas1
force-pushed
the
bugfix_issue_9345_max_pending_async_requests
branch
from
bbcc5af
to
52badee
Compare
|
/pulsarbot run-failure-checks |
…ax_pending_async_requests
|
/pulsarbot run-failure-checks |
1 similar comment
|
/pulsarbot run-failure-checks |
|
@csthomas1 Looks like the test failed by this change. Could you please take a look? move this to the next release. |
… to Java instances, and updated Runtime unit tests accordingly
|
/pulsarbot run-failure-checks |
|
@csthomas1 Awesome! Thank you for your contribution! |
fmiguelez
pushed a commit
to fmiguelez/pulsar
that referenced
this pull request
Mar 16, 2021
…nc-requests (apache#9349) Fixes apache#9345 ### Motivation When I enable either the Process or Kubernetes function Runtimes and set the parameter "maxPendingAsyncRequests" to a value other than 1000 in the function worker configuration (e.g. functions_worker.yml), any function instances launched by the worker continue to have maxPendingAsyncRequests = 1000. I'd like to be able to adjust this parameter to guard against my function running out of memory -- without increasing the function heap size -- due to too many publish operations in flight, as can occur when the output topic's backlog quota has been reached. ### Modifications Modified RuntimeUtils.getCmd to add --pending-sync-requests to the list of arguments passed to JavaInstanceStarter ### Verifying this change This change added tests and can be verified as follows: - Updated pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeTest.java to test for presence of --pending-async-requests argument in generated commandline, and increased expected arg counts by 2 to account for expected presence of --pending-async-requests <value> - Updated pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/process/ProcessRuntimeTest.java to test for presence of --pending-async-requests argument in generated commandline, and increased expected arg counts by 2 to account for expected presence of --pending-async-requests <value> - Updated pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/worker/WorkerApiV2ResourceConfigTest.java to verify that the configured value of "maxPendingAsyncRequests" was being properly loaded into the WorkerConfig
|
this patch does not apply to branch-7.2 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #9345
Motivation
When I enable either the Process or Kubernetes function Runtimes and set the parameter "maxPendingAsyncRequests" to a value other than 1000 in the function worker configuration (e.g. functions_worker.yml), any function instances launched by the worker continue to have maxPendingAsyncRequests = 1000. I'd like to be able to adjust this parameter to guard against my function running out of memory -- without increasing the function heap size -- due to too many publish operations in flight, as can occur when the output topic's backlog quota has been reached.
Modifications
Modified RuntimeUtils.getCmd to add --pending-sync-requests to the list of arguments passed to JavaInstanceStarter
Verifying this change
This change added tests and can be verified as follows:
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation