Allow pulsar_tool_env.sh PULSAR_MEM to be Overridden #15868
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
michaeljmarshall
approved these changes
May 31, 2022
There was a problem hiding this comment.
LGTM. Thanks for your contribution @jabbaugh! While we're making this change, it is probably worth adding the same override logic for the PULSAR_GC variable. Are you able to add that as well?
jabbaugh
force-pushed
the
pulsar_mem_tool_env
branch
from
69bd081
to
dcedcb1
Compare
|
@michaeljmarshall I have added the PULSAR_GC override as well. |
|
@codelipenghui @lhotari @eolivelli - given that this change affects configuration for pulsar connectors, I think it's worth cherry picking it to 2.8, 2.9, and 2.10. Do you agree? |
The pulsar_tool_env.sh sets the PULSAR_MEM and PULSAR_GC environment variables without allowing them to be overridden. This change keps the default values but allows PULSAR_MEM & PULSAR_GC to be overridden which aligns with the pulsar_env.sh file. This allows adjustments to be made to the memory settings when more memory is needed.
jabbaugh
force-pushed
the
pulsar_mem_tool_env
branch
from
dcedcb1
to
5ef2aee
Compare
michaeljmarshall
pushed a commit
that referenced
this pull request
Jun 9, 2022
…15868) The pulsar_tool_env.sh sets the PULSAR_MEM and PULSAR_GC environment variables without allowing them to be overridden. This change keps the default values but allows PULSAR_MEM & PULSAR_GC to be overridden which aligns with the pulsar_env.sh file. This allows adjustments to be made to the memory settings when more memory is needed. Co-authored-by: Jim Baugh <jim.baugh@oracle.com> ### Motivation The pulsar_tool_env.sh sets the PULSAR_MEM environment variable without allowing it to be overridden. When running an pulsar-admin function (e.g. running the kafka to pulsar connector) we can hit java memory issues without a way to change the memory settings. This PR resolves this issue. ### Modifications This change keeps the default value but allows PULSAR_MEM to be overridden which aligns with the pulsar_env.sh file. This allows adjustments to be made to the memory settings when more memory is needed. ### Verifying this change This change is a trivial rework / code cleanup without any test coverage. ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation Check the box below or label this PR directly. Need to update docs? - [X] `doc-not-needed` (Please explain why) There is not currently documentation around the pulsar_tools_env.sh PULSAR_MEM setting. This change doesn't change the default behavior. (cherry picked from commit fa6288e)
michaeljmarshall
pushed a commit
that referenced
this pull request
Jun 9, 2022
…15868) The pulsar_tool_env.sh sets the PULSAR_MEM and PULSAR_GC environment variables without allowing them to be overridden. This change keps the default values but allows PULSAR_MEM & PULSAR_GC to be overridden which aligns with the pulsar_env.sh file. This allows adjustments to be made to the memory settings when more memory is needed. Co-authored-by: Jim Baugh <jim.baugh@oracle.com> ### Motivation The pulsar_tool_env.sh sets the PULSAR_MEM environment variable without allowing it to be overridden. When running an pulsar-admin function (e.g. running the kafka to pulsar connector) we can hit java memory issues without a way to change the memory settings. This PR resolves this issue. ### Modifications This change keeps the default value but allows PULSAR_MEM to be overridden which aligns with the pulsar_env.sh file. This allows adjustments to be made to the memory settings when more memory is needed. ### Verifying this change This change is a trivial rework / code cleanup without any test coverage. ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation Check the box below or label this PR directly. Need to update docs? - [X] `doc-not-needed` (Please explain why) There is not currently documentation around the pulsar_tools_env.sh PULSAR_MEM setting. This change doesn't change the default behavior. (cherry picked from commit fa6288e)
michaeljmarshall
pushed a commit
that referenced
this pull request
Jun 9, 2022
…15868) The pulsar_tool_env.sh sets the PULSAR_MEM and PULSAR_GC environment variables without allowing them to be overridden. This change keps the default values but allows PULSAR_MEM & PULSAR_GC to be overridden which aligns with the pulsar_env.sh file. This allows adjustments to be made to the memory settings when more memory is needed. Co-authored-by: Jim Baugh <jim.baugh@oracle.com> ### Motivation The pulsar_tool_env.sh sets the PULSAR_MEM environment variable without allowing it to be overridden. When running an pulsar-admin function (e.g. running the kafka to pulsar connector) we can hit java memory issues without a way to change the memory settings. This PR resolves this issue. ### Modifications This change keeps the default value but allows PULSAR_MEM to be overridden which aligns with the pulsar_env.sh file. This allows adjustments to be made to the memory settings when more memory is needed. ### Verifying this change This change is a trivial rework / code cleanup without any test coverage. ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation Check the box below or label this PR directly. Need to update docs? - [X] `doc-not-needed` (Please explain why) There is not currently documentation around the pulsar_tools_env.sh PULSAR_MEM setting. This change doesn't change the default behavior. (cherry picked from commit fa6288e)
michaeljmarshall
added
cherry-picked/branch-2.8
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
Jun 13, 2022
…pache#15868) The pulsar_tool_env.sh sets the PULSAR_MEM and PULSAR_GC environment variables without allowing them to be overridden. This change keps the default values but allows PULSAR_MEM & PULSAR_GC to be overridden which aligns with the pulsar_env.sh file. This allows adjustments to be made to the memory settings when more memory is needed. Co-authored-by: Jim Baugh <jim.baugh@oracle.com> ### Motivation The pulsar_tool_env.sh sets the PULSAR_MEM environment variable without allowing it to be overridden. When running an pulsar-admin function (e.g. running the kafka to pulsar connector) we can hit java memory issues without a way to change the memory settings. This PR resolves this issue. ### Modifications This change keeps the default value but allows PULSAR_MEM to be overridden which aligns with the pulsar_env.sh file. This allows adjustments to be made to the memory settings when more memory is needed. ### Verifying this change This change is a trivial rework / code cleanup without any test coverage. ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation Check the box below or label this PR directly. Need to update docs? - [X] `doc-not-needed` (Please explain why) There is not currently documentation around the pulsar_tools_env.sh PULSAR_MEM setting. This change doesn't change the default behavior. (cherry picked from commit fa6288e) (cherry picked from commit f1f1d4b)
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
Jun 13, 2022
…pache#15868) The pulsar_tool_env.sh sets the PULSAR_MEM and PULSAR_GC environment variables without allowing them to be overridden. This change keps the default values but allows PULSAR_MEM & PULSAR_GC to be overridden which aligns with the pulsar_env.sh file. This allows adjustments to be made to the memory settings when more memory is needed. Co-authored-by: Jim Baugh <jim.baugh@oracle.com> ### Motivation The pulsar_tool_env.sh sets the PULSAR_MEM environment variable without allowing it to be overridden. When running an pulsar-admin function (e.g. running the kafka to pulsar connector) we can hit java memory issues without a way to change the memory settings. This PR resolves this issue. ### Modifications This change keeps the default value but allows PULSAR_MEM to be overridden which aligns with the pulsar_env.sh file. This allows adjustments to be made to the memory settings when more memory is needed. ### Verifying this change This change is a trivial rework / code cleanup without any test coverage. ### Does this pull request potentially affect one of the following parts: *If `yes` was chosen, please highlight the changes* - Dependencies (does it add or upgrade a dependency): (no) - The public API: (no) - The schema: (no) - The default values of configurations: (no) - The wire protocol: (no) - The rest endpoints: (no) - The admin cli options: (no) - Anything that affects deployment: (no) ### Documentation Check the box below or label this PR directly. Need to update docs? - [X] `doc-not-needed` (Please explain why) There is not currently documentation around the pulsar_tools_env.sh PULSAR_MEM setting. This change doesn't change the default behavior. (cherry picked from commit fa6288e) (cherry picked from commit 12e4e61)
4 tasks
poorbarcode
added a commit
that referenced
this pull request
Apr 11, 2023
…ls (#20031) ### Motivation After #15868, we allow `PULSAR_MEM` & `PULSAR_GC` to be overridden in `pulsar_tool_env.sh`. Many users set `-Xms` to `2G` or larger in `PULSAR_MEM`, this will make the tools(such as `pulsar-admin`) cost a lot of memory, and if users execute `pulsar-admin` or another tool on the machine where the Broker is deployed, the current device will not have enough memory to allocate, resulting in a broker crash. ### Modifications When `PULSAR_MEM` is overridden in `pulsar_tool_env.sh`, delete parameter `-Xms`
poorbarcode
added a commit
that referenced
this pull request
Apr 11, 2023
…ls (#20031) ### Motivation After #15868, we allow `PULSAR_MEM` & `PULSAR_GC` to be overridden in `pulsar_tool_env.sh`. Many users set `-Xms` to `2G` or larger in `PULSAR_MEM`, this will make the tools(such as `pulsar-admin`) cost a lot of memory, and if users execute `pulsar-admin` or another tool on the machine where the Broker is deployed, the current device will not have enough memory to allocate, resulting in a broker crash. ### Modifications When `PULSAR_MEM` is overridden in `pulsar_tool_env.sh`, delete parameter `-Xms` (cherry picked from commit 4f503fd)
poorbarcode
added a commit
that referenced
this pull request
Apr 11, 2023
…ls (#20031) ### Motivation After #15868, we allow `PULSAR_MEM` & `PULSAR_GC` to be overridden in `pulsar_tool_env.sh`. Many users set `-Xms` to `2G` or larger in `PULSAR_MEM`, this will make the tools(such as `pulsar-admin`) cost a lot of memory, and if users execute `pulsar-admin` or another tool on the machine where the Broker is deployed, the current device will not have enough memory to allocate, resulting in a broker crash. ### Modifications When `PULSAR_MEM` is overridden in `pulsar_tool_env.sh`, delete parameter `-Xms` (cherry picked from commit 4f503fd)
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
May 11, 2023
…ls (apache#20031) ### Motivation After apache#15868, we allow `PULSAR_MEM` & `PULSAR_GC` to be overridden in `pulsar_tool_env.sh`. Many users set `-Xms` to `2G` or larger in `PULSAR_MEM`, this will make the tools(such as `pulsar-admin`) cost a lot of memory, and if users execute `pulsar-admin` or another tool on the machine where the Broker is deployed, the current device will not have enough memory to allocate, resulting in a broker crash. ### Modifications When `PULSAR_MEM` is overridden in `pulsar_tool_env.sh`, delete parameter `-Xms` (cherry picked from commit 4f503fd) (cherry picked from commit 1fe05d5)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
The pulsar_tool_env.sh sets the PULSAR_MEM environment variable without allowing it to be overridden. When running an pulsar-admin function (e.g. running the kafka to pulsar connector) we can hit java memory issues without a way to change the memory settings. This PR resolves this issue.
Modifications
This change keeps the default value but allows PULSAR_MEM to be overridden which aligns with the
pulsar_env.sh file. This allows adjustments to be made to the memory settings when more memory is needed.
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation
Check the box below or label this PR directly.
Need to update docs?
doc-not-needed(Please explain why)
There is not currently documentation around the pulsar_tools_env.sh PULSAR_MEM setting. This change doesn't change the default behavior.