Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDDS-4816. Add UsageInfoSubcommand to get Datanode usage information. #1919

Closed
wants to merge 10 commits into from
Closed

HDDS-4816. Add UsageInfoSubcommand to get Datanode usage information. #1919

wants to merge 10 commits into from

Conversation

siddhantsangwan
Copy link
Contributor

@siddhantsangwan siddhantsangwan commented Feb 15, 2021 •
edited
Loading

What changes were proposed in this pull request?

The UsageInfoSubcommand is a datanode admin sub command that can be used to query information such as Capacity, SCMUsed, and Remaining space in a datanode. IP address or UUID is used to get the datanode.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-4816

How was this patch tested?

Manually tested using docker-compose. Screenshots have been attached.

Initial, before adding keys:
initial

After adding keys:
keys added

Using UUID:
uuid

@siddhantsangwan
HDDS-4816. Add DatanodeInfo as a Datanode SubCommand.
adec667 
Add basic client side functionality.
@siddhantsangwan
HDDS-4816. Add client and server side serialization and deserialization.
8b2483b 
Edit the Request and Response functionality.
@siddhantsangwan
HDDDS-4816. Refactor and improve output formatting.
b10a907
@siddhantsangwan
HDDS-4816. Refactor command, protos, and corresponding methods names.
28829ed 
Change command name from DatanodeDiskInfo to DiskMetricsSubCommand. Change corresponding methods names from getDatanodeDiskInfo to getDatanodeDiskMetrics. Also change the corresponding hdds.proto and ScmAdminProtocol.proto messages.
@siddhantsangwan
HDDS-4816. Add Apache license.
72a6f39
@siddhantsangwan
HDDS-4816. Trigger new CI check
868f025
lokeshj1703
lokeshj1703 reviewed Feb 22, 2021
View reviewed changes
Copy link
Contributor

@lokeshj1703 lokeshj1703 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@siddhantsangwan Thanks for working on this! The changes look good to me. I have few minor comments inline. Let's also change the command name to usageinfo.

...pache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java Outdated Show resolved Hide resolved
hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto Outdated Show resolved Hide resolved
hadoop-hdds/interface-client/src/main/proto/hdds.proto Outdated Show resolved Hide resolved
.../apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocolServerSideTranslatorPB.java Outdated Show resolved Hide resolved
...hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java Outdated Show resolved Hide resolved
...hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java Outdated Show resolved Hide resolved
...hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java Outdated Show resolved Hide resolved
...hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java Outdated Show resolved Hide resolved
...-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/DiskMetricsSubCommand.java Outdated Show resolved Hide resolved
...ntegration-test/src/test/java/org/apache/hadoop/ozone/freon/TestOzoneClientKeyGenerator.java Outdated Show resolved Hide resolved
lokeshj1703
lokeshj1703 reviewed Feb 24, 2021
View reviewed changes
Copy link
Contributor

@lokeshj1703 lokeshj1703 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@siddhantsangwan Thanks for updating the PR! I have one minor comment.
Please change the name of the PR and jira to reflect the command name change.

...hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java Outdated
* @return Usage info such as capacity, SCMUsed, and remaining space.
* @throws IOException
*/
public HddsProtos.DatanodeUsageInfo getUsageInfoFromDatanodeDetails(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can make it private.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@siddhantsangwan siddhantsangwan changed the title HDDS-4816. Add DiskMetricsSubCommand to get Datanode usage information. HDDS-4816. Add UsageInfoSubcommand to get Datanode usage information. Feb 24, 2021
lokeshj1703
lokeshj1703 approved these changes Feb 25, 2021
View reviewed changes
Copy link
Contributor

@lokeshj1703 lokeshj1703 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@siddhantsangwan Thanks for updating the PR! The changes look good to me. +1.

lokeshj1703 pushed a commit that referenced this pull request Feb 25, 2021
@siddhantsangwan @lokeshj1703
HDDS-4816. Add UsageInfoSubcommand to get Datanode usage information. (…
bc501d4 
…#1919)
@lokeshj1703
Copy link
Contributor

@siddhantsangwan Thanks for the contribution! I have committed the PR to master branch.

errose28 added a commit to errose28/ozone that referenced this pull request Mar 2, 2021
@errose28
Merge remote-tracking branch 'upstream/master' into HDDS-3698-nonroll…
ce9148f 
…ing-upgrade

* upstream/master: (29 commits)
  HDDS-4741. Modularize upgrade test (apache#1928)
  HDDS-4864. Add acceptance tests to certify Ozone with boto3 python client. (apache#1976)
  HDDS-4791. StateContext.getReports may return list with size larger t… (apache#1892)
  HDDS-4867. Ozone admin datanode list should report dead and stale nodes (apache#1966)
  HDDS-4858. Useless Maven cache cleanup (apache#1956)
  HDDS-4769. Simplify insert operation of ContainerAttribute (apache#1865)
  HDDS-4847. Fix typo in name of IdentityService (apache#1941)
  HDDS-4869. Bump jackson version number (apache#1963)
  HDDS-4871. Fix intellij runConfigurations for datanode (apache#1968)
  HDDS-4870. Bump jetty version (apache#1964)
  HDDS-4722. Creating RDBStore fails due to RDBMetrics instance race (apache#1820)
  HDDS-4138. Improve crc efficiency by using Java.util.zip.CRC when available (apache#1950)
  HDDS-4816. Add UsageInfoSubcommand to get Datanode usage information. (apache#1919)
  HDDS-4754. Make scm heartbeat rpc retry interval configurable (apache#1942)
  HDDS-4832. Show Datanode OperationalState in Recon (apache#1937)
  HDDS-4653. Support TDE for MPU Keys on Encrypted Buckets (apache#1766)
  HDDS-4853. libexec/entrypoint.sh might copy from wrong path (apache#1951)
  HDDS-4857. Format ReplicationType.java which indentation are confusion (apache#1952)
  HDDS-4850. Intermittent failure in ozonesecure due to unable to allocate block (apache#1948)
  HDDS-4808. Add Genesis benchmark for various CRC implementations (apache#1910)
  ...

Conflicts:
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
	hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java
	hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto
	hadoop-hdds/interface-client/src/main/proto/hdds.proto
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocolServerSideTranslatorPB.java
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
	hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ContainerOperationClient.java
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
errose28 added a commit to errose28/ozone that referenced this pull request Mar 11, 2021
@errose28
Merge remote-tracking branch 'upstream/master' into HDDS-3698-nonroll…
7eefee7 
…ing-upgrade-merge-candidate

* upstream/master: (29 commits)
  HDDS-4741. Modularize upgrade test (apache#1928)
  HDDS-4864. Add acceptance tests to certify Ozone with boto3 python client. (apache#1976)
  HDDS-4791. StateContext.getReports may return list with size larger t… (apache#1892)
  HDDS-4867. Ozone admin datanode list should report dead and stale nodes (apache#1966)
  HDDS-4858. Useless Maven cache cleanup (apache#1956)
  HDDS-4769. Simplify insert operation of ContainerAttribute (apache#1865)
  HDDS-4847. Fix typo in name of IdentityService (apache#1941)
  HDDS-4869. Bump jackson version number (apache#1963)
  HDDS-4871. Fix intellij runConfigurations for datanode (apache#1968)
  HDDS-4870. Bump jetty version (apache#1964)
  HDDS-4722. Creating RDBStore fails due to RDBMetrics instance race (apache#1820)
  HDDS-4138. Improve crc efficiency by using Java.util.zip.CRC when available (apache#1950)
  HDDS-4816. Add UsageInfoSubcommand to get Datanode usage information. (apache#1919)
  HDDS-4754. Make scm heartbeat rpc retry interval configurable (apache#1942)
  HDDS-4832. Show Datanode OperationalState in Recon (apache#1937)
  HDDS-4653. Support TDE for MPU Keys on Encrypted Buckets (apache#1766)
  HDDS-4853. libexec/entrypoint.sh might copy from wrong path (apache#1951)
  HDDS-4857. Format ReplicationType.java which indentation are confusion (apache#1952)
  HDDS-4850. Intermittent failure in ozonesecure due to unable to allocate block (apache#1948)
  HDDS-4808. Add Genesis benchmark for various CRC implementations (apache#1910)
  ...

Conflicts:
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/client/ScmClient.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
	hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java
	hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto
	hadoop-hdds/interface-client/src/main/proto/hdds.proto
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocolServerSideTranslatorPB.java
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
	hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ContainerOperationClient.java
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
	hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconNodeManager.java
errose28 added a commit to errose28/ozone that referenced this pull request Mar 16, 2021
@errose28
Merge branch 'HDDS-3698-nonrolling-upgrade' into HDDS-4180
5c64f04 
* HDDS-3698-nonrolling-upgrade: (29 commits)
  HDDS-4741. Modularize upgrade test (apache#1928)
  HDDS-4864. Add acceptance tests to certify Ozone with boto3 python client. (apache#1976)
  HDDS-4791. StateContext.getReports may return list with size larger t… (apache#1892)
  HDDS-4867. Ozone admin datanode list should report dead and stale nodes (apache#1966)
  HDDS-4858. Useless Maven cache cleanup (apache#1956)
  HDDS-4769. Simplify insert operation of ContainerAttribute (apache#1865)
  HDDS-4847. Fix typo in name of IdentityService (apache#1941)
  HDDS-4869. Bump jackson version number (apache#1963)
  HDDS-4871. Fix intellij runConfigurations for datanode (apache#1968)
  HDDS-4870. Bump jetty version (apache#1964)
  HDDS-4722. Creating RDBStore fails due to RDBMetrics instance race (apache#1820)
  HDDS-4138. Improve crc efficiency by using Java.util.zip.CRC when available (apache#1950)
  HDDS-4816. Add UsageInfoSubcommand to get Datanode usage information. (apache#1919)
  HDDS-4754. Make scm heartbeat rpc retry interval configurable (apache#1942)
  HDDS-4832. Show Datanode OperationalState in Recon (apache#1937)
  HDDS-4653. Support TDE for MPU Keys on Encrypted Buckets (apache#1766)
  HDDS-4853. libexec/entrypoint.sh might copy from wrong path (apache#1951)
  HDDS-4857. Format ReplicationType.java which indentation are confusion (apache#1952)
  HDDS-4850. Intermittent failure in ozonesecure due to unable to allocate block (apache#1948)
  HDDS-4808. Add Genesis benchmark for various CRC implementations (apache#1910)
  ...
@siddhantsangwan siddhantsangwan deleted the HDDS-4816 branch May 25, 2021 06:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lokeshj1703 lokeshj1703 lokeshj1703 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@siddhantsangwan @lokeshj1703