Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to naming conventions #1401

Merged
merged 2 commits into from
Sep 13, 2020
Merged

HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to naming conventions #1401

merged 2 commits into from
Sep 13, 2020

Conversation

smengcl
Copy link
Contributor

@smengcl smengcl commented Sep 5, 2020 •
edited
Loading

What changes were proposed in this pull request?

Rename the environment variables to be in accordance with the convention, and keep the compatibility by deprecating the old variable names.

  • HDFS_OM_OPTS -> OZONE_OM_OPTS
  • HDDS_DN_OPTS -> OZONE_DATANODE_OPTS
  • HDFS_STORAGECONTAINERMANAGER_OPTS -> OZONE_SCM_OPTS

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3927

How was this patch tested?

No new tests added. Tested manually:

  1. HADOOP_SHELL_SCRIPT_DEBUG=1 OZONE_OM_OPTS=-Dtest123 ./ozone om
DEBUG: Appending OZONE_OM_OPTS onto HADOOP_OPTS
...
DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dtest123 -Dlog4j.configurationFile=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/etc/hadoop/om-audit-log4j2.properties -Dhadoop.log.dir=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT -Dhadoop.id.str=smeng -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
  1. HADOOP_SHELL_SCRIPT_DEBUG=1 OZONE_DATANODE_OPTS=-Dtest123 ./ozone datanode
DEBUG: Appending OZONE_DATANODE_OPTS onto HADOOP_OPTS
...
DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dorg.apache.ratis.thirdparty.io.netty.allocator.useCacheForAllThreads=false -Dorg.apache.ratis.thirdparty.io.netty.leakDetection.level=disabled -Dlog4j.configurationFile=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/etc/hadoop/dn-audit-log4j2.properties -Dtest123 -Dhadoop.log.dir=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT -Dhadoop.id.str=smeng -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
  1. HADOOP_SHELL_SCRIPT_DEBUG=1 OZONE_SCM_OPTS=-Dtest123 ./ozone scm
DEBUG: Appending OZONE_SCM_OPTS onto HADOOP_OPTS
...
DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dtest123 -Dlog4j.configurationFile=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/etc/hadoop/scm-audit-log4j2.properties -Dhadoop.log.dir=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/Users/smeng/repo/ozone/hadoop-ozone/dist/target/ozone-1.1.0-SNAPSHOT -Dhadoop.id.str=smeng -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender
  1. Deprecation warning: HADOOP_SHELL_SCRIPT_DEBUG=1 HDFS_OM_OPTS=-Dtest123 ./ozone om
WARNING: HDFS_OM_OPTS has been replaced by OZONE_OM_OPTS. Using value of HDFS_OM_OPTS.

Caveat: when both deprecated and new variables names are present, the new variable will be ignored. So admins should refrain from using both at the same time. e.g. for HDFS_OM_OPTS=-Dtest123 OZONE_OM_OPTS=-Dtest456 ./ozone om, OZONE_OM_OPTS=-Dtest456 will be completely ignored, only HDFS_OM_OPTS=-Dtest123 will be respected.

Note

hadoop_deprecate_envvar takes care of the copying the values from the old variable name to the new one, and prints the deprecation warning.

@smengcl
HDDS-3927. Rename HDFS_OM_OPTS/HDDS_DN_OPTS to OZONE_MANAGER_OPTS/OZO…
8e5bfc2 
…NE_DATANODE_OPTS

Change-Id: I629f87486b62b6a3b7f108addf89ce1ec2bc9e18
@smengcl smengcl self-assigned this Sep 5, 2020
elek
elek requested changes Sep 7, 2020
View reviewed changes
Copy link
Member

@elek elek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks to @smengcl the patch. I like this change.

Recently I learned that some OPTS are added automatically by hadoop_subcommand_opts, I think we can use that functionality (and naming convention).

Also: I would consider renaming HDFS_STORAGECONTAINERMANAGER_OPTS to OZONE_SCM_OPTS, too. Just for consistency.

hadoop-hdds/common/src/main/conf/hadoop-env.sh Outdated Show resolved Hide resolved
hadoop-ozone/dist/src/shell/ozone/ozone Outdated Show resolved Hide resolved
hadoop-ozone/dist/src/shell/ozone/ozone Outdated Show resolved Hide resolved
@smengcl
Conform to hadoop_subcommand_opts's OPTS naming conventions.
350b98b 
Change-Id: I67259db97eabf04a55b9b5bbc582a6e5732cdb58
@smengcl smengcl changed the title HDDS-3927. Rename HDFS_OM_OPTS/HDDS_DN_OPTS to OZONE_MANAGER_OPTS/OZONE_DATANODE_OPTS HDDS-3927. Rename HDFS_OM_OPTS,HDDS_DN_OPTS,HDFS_STORAGECONTAINERMANAGER_OPTS to OZONE_OM_OPTS,OZONE_DATANODE_OPTS,OZONE_SCM_OPTS Sep 7, 2020
@smengcl smengcl changed the title HDDS-3927. Rename HDFS_OM_OPTS,HDDS_DN_OPTS,HDFS_STORAGECONTAINERMANAGER_OPTS to OZONE_OM_OPTS,OZONE_DATANODE_OPTS,OZONE_SCM_OPTS HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to naming conventions Sep 7, 2020
@smengcl
Copy link
Contributor Author

smengcl commented Sep 9, 2020

Thanks @elek for the review. I have made changes accordingly.

Do the changes look good to you?

elek
elek approved these changes Sep 13, 2020
View reviewed changes
Copy link
Member

@elek elek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the changes look good to you?

Sure, I am +1. Thank you very much for the update.

@elek elek merged commit 48e8e50 into apache:master Sep 13, 2020
llemec pushed a commit to llemec/hadoop-ozone that referenced this pull request Sep 15, 2020
@smengcl @llemec
HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to namin…
a530246 
…g conventions (apache#1401)
errose28 added a commit to errose28/ozone that referenced this pull request Sep 18, 2020
@errose28
Merge branch 'master' into HDDS-4122-quota-attempt2
90ab92d 
* master: (47 commits)
  HDDS-4104. Provide a way to get the default value and key of java-based-configuration easily (apache#1369)
  HDDS-4250. Fix wrong logger name (apache#1429)
  HDDS-4244. Container deleted wrong replica cause mis-replicated. (apache#1423)
  HDDS-4053. Volume space: add quotaUsageInBytes and update it when write and delete key. (apache#1296)
  HDDS-4210. ResolveBucket during checkAcls fails. (apache#1398)
  HDDS-4075. Retry request on different OM on AccessControlException (apache#1303)
  HDDS-4166. Documentation index page redirects to the wrong address (apache#1372)
  HDDS-4039. Reduce the number of fields in hdds.proto to improve performance (apache#1289)
  HDDS-4155. Directory and filename can end up with same name in a path. (apache#1361)
  HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to naming conventions (apache#1401)
  HDDS-4119. Improve performance of the BufferPool management of Ozone client (apache#1336)
  HDDS-4217.Remove test TestOzoneContainerRatis (apache#1408)
  HDDS-4218.Remove test TestRatisManager (apache#1409)
  HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE. (apache#1337)
  HDDS-4228: add field 'num' to ALLOCATE_BLOCK of scm audit log. (apache#1413)
  HDDS-4196. Add an endpoint in Recon to query Prometheus (apache#1390)
  HDDS-4211. [OFS] Better owner and group display for listing Ozone volumes and buckets (apache#1397)
  HDDS-4150. recon.api.TestEndpoints test is flaky (apache#1396)
  HDDS-4170 - Fix typo in method description. (apache#1406)
  HDDS-4064. Show container verbose info with verbose option (apache#1290)
  ...
errose28 added a commit to errose28/ozone that referenced this pull request Sep 18, 2020
@errose28
Incorporate volume size tracking into open key commit request and res…
5fccc60 
…ponse

* HDDS-4122-quota-attempt2: (51 commits)
  Remove redundant check status calls in children of AbstractOMOpenKeyDeleteRequest
  Remove unused inports and fix super constructor calls
  Move common volume byte usage update code to AbstractOMKeyDeleteResponse
  Add volume quota update to open key delete response, and group duplicate code
  HDDS-4104. Provide a way to get the default value and key of java-based-configuration easily (apache#1369)
  HDDS-4250. Fix wrong logger name (apache#1429)
  HDDS-4244. Container deleted wrong replica cause mis-replicated. (apache#1423)
  HDDS-4053. Volume space: add quotaUsageInBytes and update it when write and delete key. (apache#1296)
  HDDS-4210. ResolveBucket during checkAcls fails. (apache#1398)
  HDDS-4075. Retry request on different OM on AccessControlException (apache#1303)
  HDDS-4166. Documentation index page redirects to the wrong address (apache#1372)
  HDDS-4039. Reduce the number of fields in hdds.proto to improve performance (apache#1289)
  HDDS-4155. Directory and filename can end up with same name in a path. (apache#1361)
  HDDS-3927. Rename Ozone OM,DN,SCM runtime options to conform to naming conventions (apache#1401)
  HDDS-4119. Improve performance of the BufferPool management of Ozone client (apache#1336)
  HDDS-4217.Remove test TestOzoneContainerRatis (apache#1408)
  HDDS-4218.Remove test TestRatisManager (apache#1409)
  HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE. (apache#1337)
  HDDS-4228: add field 'num' to ALLOCATE_BLOCK of scm audit log. (apache#1413)
  HDDS-4196. Add an endpoint in Recon to query Prometheus (apache#1390)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@elek elek elek approved these changes

@bharatviswa504 bharatviswa504 Awaiting requested review from bharatviswa504

Assignees

@smengcl smengcl

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@smengcl @elek