HDDS-4504. Datanode deletion config should be based on number of blocks#1885
Closed
aryangupta1998 wants to merge 12 commits intoapache:masterfrom
Closed
HDDS-4504. Datanode deletion config should be based on number of blocks#1885aryangupta1998 wants to merge 12 commits intoapache:masterfrom
aryangupta1998 wants to merge 12 commits intoapache:masterfrom
Conversation
lokeshj1703
reviewed
Feb 15, 2021
Contributor
lokeshj1703
left a comment
lokeshj1703
left a comment
There was a problem hiding this comment.
@aryangupta1998 Thanks for working on this PR! Please find my comments inline.
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
Outdated
Show resolved
Hide resolved
...ava/org/apache/hadoop/ozone/container/common/impl/RandomContainerDeletionChoosingPolicy.java
Outdated
Show resolved
Hide resolved
...org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java
Outdated
Show resolved
Hide resolved
...org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java
Outdated
Show resolved
Hide resolved
...org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java
Outdated
Show resolved
Hide resolved
...org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java
Outdated
Show resolved
Hide resolved
...org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java
Outdated
Show resolved
Hide resolved
...org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java
Outdated
Show resolved
Hide resolved
...org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java
Outdated
Show resolved
Hide resolved
...org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java
Outdated
Show resolved
Hide resolved
lokeshj1703
reviewed
Feb 15, 2021
...service/src/test/java/org/apache/hadoop/ozone/container/common/TestBlockDeletingService.java
Show resolved
Hide resolved
lokeshj1703
reviewed
Feb 15, 2021
.../java/org/apache/hadoop/ozone/container/common/impl/TestContainerDeletionChoosingPolicy.java
Outdated
Show resolved
Hide resolved
lokeshj1703
reviewed
Mar 16, 2021
Contributor
lokeshj1703
left a comment
lokeshj1703
left a comment
There was a problem hiding this comment.
@aryangupta1998 Thanks for updating the PR! I have a few other comments inline, mostly minor.
...c/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java
Outdated
Show resolved
Hide resolved
...org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java
Outdated
Show resolved
Hide resolved
...ava/org/apache/hadoop/ozone/container/common/impl/RandomContainerDeletionChoosingPolicy.java
Outdated
Show resolved
Hide resolved
...ava/org/apache/hadoop/ozone/container/common/impl/RandomContainerDeletionChoosingPolicy.java
Show resolved
Hide resolved
...org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java
Outdated
Show resolved
Hide resolved
lokeshj1703
reviewed
Mar 17, 2021
...c/main/java/org/apache/hadoop/ozone/container/common/statemachine/DatanodeConfiguration.java
Outdated
Show resolved
Hide resolved
lokeshj1703
reviewed
Apr 1, 2021
Contributor
lokeshj1703
left a comment
lokeshj1703
left a comment
There was a problem hiding this comment.
@aryangupta1998 Thanks for updating the PR! The changes look good to me. I have few minor comments. +1 o.w.
...org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java
Outdated
Show resolved
Hide resolved
...org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingService.java
Outdated
Show resolved
Hide resolved
lokeshj1703
approved these changes
Apr 5, 2021
Contributor
lokeshj1703
left a comment
lokeshj1703
left a comment
There was a problem hiding this comment.
@aryangupta1998 Thanks for updating! The changes look good to me. +1.
lokeshj1703
pushed a commit
that referenced
this pull request
Apr 5, 2021
Contributor
|
@aryangupta1998 Thanks for the contribution! I have committed the PR to master branch. |
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Apr 6, 2021
* master: HDDS-5066. Use fixed vesion from pnpm to build recon (apache#2115) HDDS-5035. Use default config values to solve generated config file conflict (apache#2087) HDDS-5032. DN stopped to load containers on volume after a container load exception. (apache#2109) HDDS-4504. Datanode deletion config should be based on number of blocks (apache#1885)
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Apr 7, 2021
* HDDS-3698-nonrolling-upgrade: (144 commits) fix project name in NOTICE.txt (apache#2112) HDDS-5066. Use fixed vesion from pnpm to build recon (apache#2115) HDDS-5014. Add non-rolling upgrade design docs. HDDS-5035. Use default config values to solve generated config file conflict (apache#2087) HDDS-5032. DN stopped to load containers on volume after a container load exception. (apache#2109) HDDS-4504. Datanode deletion config should be based on number of blocks (apache#1885) Fix ozone-ha acceptance test. HDDS-5058. Make getScmInfo retry for a duration. HDDS-4506. Support query parameter based v4 auth in S3g (apache#1628) HDDS-4553. ChunkInputStream should release buffer as soon as last byte in the buffer is read (apache#2062) HDDS-5022. SCM get roles command should provide Ratis Leader/Follower… (apache#2098) HDDS-5033. SCM may not be able to know full port list of Datanode after Datanode is started. (apache#2090) HDDS-3752. Fix o3fs list bucket contents issue when without tailing "/" (apache#2088) HDDS-4901. Remove OmOzoneAclMap from OmVolumeArgs to avoid OzoneAcl conversions (apache#1992) HDDS-4987. Import container should not delete container contents if container already exists (apache#2077) Checkstyle fix. Intialize DN layout version before security init. HDDS-4915. [SCM HA Security] Integrate CertClient. (apache#2000) HDDS-5049. Add timeout support for ratis requests in SCM HA. (apache#2099) trigger new CI check ...
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Apr 7, 2021
* HDDS-3698-nonrolling-upgrade: fix project name in NOTICE.txt (apache#2112) HDDS-5066. Use fixed vesion from pnpm to build recon (apache#2115) HDDS-5014. Add non-rolling upgrade design docs. HDDS-5035. Use default config values to solve generated config file conflict (apache#2087) HDDS-5032. DN stopped to load containers on volume after a container load exception. (apache#2109) HDDS-4504. Datanode deletion config should be based on number of blocks (apache#1885)
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Apr 9, 2021
* HDDS-3698-nonrolling-upgrade: (150 commits) HDDS-5056. Avoid false positiver error messages during pipeline creations (apache#2105) HDDS-5027. [SCM HA Security] Handle leader changes during bootstrap. (apache#2113) HDDS-5032. Fix findbugs (apache#2120) HDDS-5062. Add a config to bypass clusterId validation for bootstrapping SCM. (apache#2114) HDDS-5011. Introduce Java based ReplicationConfig implementation (apache#2089) HDDS-4925. Introduce ContainerBalancer in SCM with start/stop capabilities. (apache#2097) fix project name in NOTICE.txt (apache#2112) HDDS-5066. Use fixed vesion from pnpm to build recon (apache#2115) HDDS-5014. Add non-rolling upgrade design docs. HDDS-5035. Use default config values to solve generated config file conflict (apache#2087) HDDS-5032. DN stopped to load containers on volume after a container load exception. (apache#2109) HDDS-4504. Datanode deletion config should be based on number of blocks (apache#1885) Fix ozone-ha acceptance test. HDDS-5058. Make getScmInfo retry for a duration. HDDS-4506. Support query parameter based v4 auth in S3g (apache#1628) HDDS-4553. ChunkInputStream should release buffer as soon as last byte in the buffer is read (apache#2062) HDDS-5022. SCM get roles command should provide Ratis Leader/Follower… (apache#2098) HDDS-5033. SCM may not be able to know full port list of Datanode after Datanode is started. (apache#2090) HDDS-3752. Fix o3fs list bucket contents issue when without tailing "/" (apache#2088) HDDS-4901. Remove OmOzoneAclMap from OmVolumeArgs to avoid OzoneAcl conversions (apache#1992) ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently datanode's deletion config is based on number of containers on which deletion should be performed(default value 10). The deletion is only performed on those containers.
It should rather be based on number of blocks to delete per interval rather than number of containers to process. If a datanode has 100000 containers and every container has 10 blocks to delete then deletion of all these blocks can take a lot of time with the current configuration. It would take 100000/10 = 10000 intervals for deletion to complete.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4504
How was this patch tested?
Tested Manually