Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable #4935

Merged
merged 4 commits into from
Jun 23, 2023
Merged

HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable #4935

merged 4 commits into from
Jun 23, 2023

Conversation

aswinshakil
Copy link
Member

What changes were proposed in this pull request?

This change allows the KeyDeletingService to reclaim those trapped keys from the snapshot's deletedTable. Every snapshot is deep cleaned once they are created, after that snapshots are only deep cleaned if the previous snapshot is deleted.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7968

How was this patch tested?

Existing and New UTs.

@aswinshakil aswinshakil added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Jun 19, 2023
@aswinshakil aswinshakil self-assigned this Jun 19, 2023
smengcl
smengcl approved these changes Jun 22, 2023
View reviewed changes
Copy link
Contributor

@smengcl smengcl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm pending CI

hemantk-12
hemantk-12 reviewed Jun 22, 2023
View reviewed changes
Copy link
Contributor

@hemantk-12 hemantk-12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch @aswinshakil

...anager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotPurgeRequest.java Show resolved Hide resolved
...anager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotPurgeRequest.java Outdated Show resolved Hide resolved
...anager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotPurgeRequest.java Outdated Show resolved Hide resolved
...ager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/OMSnapshotPurgeResponse.java Outdated Show resolved Hide resolved
...ager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/OMSnapshotPurgeResponse.java Outdated Show resolved Hide resolved
...ager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/OMSnapshotPurgeResponse.java Outdated Show resolved Hide resolved
...one-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java Outdated Show resolved Hide resolved
...one-manager/src/main/java/org/apache/hadoop/ozone/om/service/AbstractKeyDeletingService.java Outdated Show resolved Hide resolved
hemantk-12
hemantk-12 approved these changes Jun 22, 2023
View reviewed changes
Copy link
Contributor

@hemantk-12 hemantk-12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/SnapshotInfo.java
@@ -125,6 +125,7 @@ public static SnapshotStatus valueOf(SnapshotStatusProto status) {
* RocksDB's transaction sequence number at the time of checkpoint creation.
*/
private long dbTxSequenceNumber;
private boolean deepClean;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Should it be deepCleaned?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aswinshakil ?

prashantpogde
prashantpogde approved these changes Jun 23, 2023
View reviewed changes
Copy link
Contributor

@prashantpogde prashantpogde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the changes @aswinshakil

@prashantpogde prashantpogde merged commit 9b78205 into apache:master Jun 23, 2023
28 checks passed
errose28 added a commit to errose28/ozone that referenced this pull request Jun 25, 2023
@errose28
Merge branch 'master' into tmp-dir-refactor
e502a07 
* master: (96 commits)
  HDDS-8586 Recon. - API for Count of deletePending keys and amount of data mapped to such keys. (apache#4923)
  HDDS-8908. Intermittent failure in TestBlockDeletion#testBlockDeletion (apache#4958)
  HDDS-8910. Replace LockManager with striped lock in ContainerStateManager (apache#4962)
  HDDS-8917. Move protobuf conversion out of the lock in PipelineStateManagerImpl (apache#4965)
  HDDS-8825. Use apache/hadoop 3.3.5 docker image (apache#4963)
  HDDS-8906. Avoid stream when getting in-service healthy nodes (apache#4960)
  HDDS-8907. Store volume count when storage report is updated (apache#4957)
  HDDS-8905. PipelineManager metrics should not be synchronized (apache#4959)
  HDDS-8553. Improve scanner integration tests. (apache#4936)
  HDDS-8854. Avoid unnecessary DatanodeDetails creation for NodeStateManager lookup (apache#4925)
  HDDS-8315. [Snapshot] Added unit tests for SnapshotDiffManager (apache#4716)
  HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable (apache#4935)
  HDDS-8838. Update default datanode check empty containter on disk to false (apache#4937)
  HDDS-8763. Support RocksDB iterator with ByteBuffer. (apache#4942)
  HDDS-8543. FSO directory should reflect bucket/cluster default replication (apache#4947)
  HDDS-8898. Replication limit should not be less than reconstruction weight (apache#4954)
  HDDS-8739. Snapdiff should return complete absolute path in Diff Entry (apache#4823)
  HDDS-8908. Mark TestBlockDeletion#testBlockDeletion as flaky
  HDDS-8534. Support asynchronous service logging (apache#4663)
  HDDS-8879. Cleanup SecurityConfig and related class initialization (apache#4921)
  ...
errose28 added a commit to errose28/ozone that referenced this pull request Jun 26, 2023
@errose28
Merge branch 'tmp-dir-refactor' into improve-volume-scanner
fa4207c 
* tmp-dir-refactor: (99 commits)
  HDDS-8586 Recon. - API for Count of deletePending keys and amount of data mapped to such keys. (apache#4923)
  Fix SCM HA finalization compat test
  HDDS-8908. Intermittent failure in TestBlockDeletion#testBlockDeletion (apache#4958)
  HDDS-8910. Replace LockManager with striped lock in ContainerStateManager (apache#4962)
  HDDS-8917. Move protobuf conversion out of the lock in PipelineStateManagerImpl (apache#4965)
  HDDS-8825. Use apache/hadoop 3.3.5 docker image (apache#4963)
  HDDS-8906. Avoid stream when getting in-service healthy nodes (apache#4960)
  HDDS-8907. Store volume count when storage report is updated (apache#4957)
  HDDS-8905. PipelineManager metrics should not be synchronized (apache#4959)
  HDDS-8553. Improve scanner integration tests. (apache#4936)
  HDDS-8854. Avoid unnecessary DatanodeDetails creation for NodeStateManager lookup (apache#4925)
  HDDS-8315. [Snapshot] Added unit tests for SnapshotDiffManager (apache#4716)
  HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable (apache#4935)
  HDDS-8838. Update default datanode check empty containter on disk to false (apache#4937)
  HDDS-8763. Support RocksDB iterator with ByteBuffer. (apache#4942)
  HDDS-8543. FSO directory should reflect bucket/cluster default replication (apache#4947)
  HDDS-8898. Replication limit should not be less than reconstruction weight (apache#4954)
  HDDS-8739. Snapdiff should return complete absolute path in Diff Entry (apache#4823)
  HDDS-8908. Mark TestBlockDeletion#testBlockDeletion as flaky
  HDDS-8534. Support asynchronous service logging (apache#4663)
  ...
errose28 added a commit to errose28/ozone that referenced this pull request Jun 26, 2023
@errose28
Merge branch 'master' into HDDS-7100-scanner-shutdown
7726336 
* master: (79 commits)
  HDDS-8914. Datanode may fail to start due to duplicate VolumeInfoMetrics (apache#4966)
  HDDS-8921. Add support for EC in Freon SCM block generator (apache#4982)
  HDDS-8927. Metadata scanner should not scan unhealthy containers. (apache#4976)
  HDDS-8929. Avoid list allocation for pipeline search (apache#4980)
  HDDS-8778. Support recursive volume delete using Ozone sh command. (apache#4842)
  HDDS-8885. Quota repair count enable quota feature for old bucket/volume. (apache#4941)
  HDDS-8771. Refactor volume level tmp directory for generic usage. (apache#4838)
  HDDS-8922. Random EC read pipeline ID causes XceiverClient cache churn (apache#4971)
  HDDS-8586 Recon. - API for Count of deletePending keys and amount of data mapped to such keys. (apache#4923)
  HDDS-8908. Intermittent failure in TestBlockDeletion#testBlockDeletion (apache#4958)
  HDDS-8910. Replace LockManager with striped lock in ContainerStateManager (apache#4962)
  HDDS-8917. Move protobuf conversion out of the lock in PipelineStateManagerImpl (apache#4965)
  HDDS-8825. Use apache/hadoop 3.3.5 docker image (apache#4963)
  HDDS-8906. Avoid stream when getting in-service healthy nodes (apache#4960)
  HDDS-8907. Store volume count when storage report is updated (apache#4957)
  HDDS-8905. PipelineManager metrics should not be synchronized (apache#4959)
  HDDS-8553. Improve scanner integration tests. (apache#4936)
  HDDS-8854. Avoid unnecessary DatanodeDetails creation for NodeStateManager lookup (apache#4925)
  HDDS-8315. [Snapshot] Added unit tests for SnapshotDiffManager (apache#4716)
  HDDS-7968. [Snapshot] Improve KeyDeletingService to reclaim eligible key blocks in snapshot's deletedTable (apache#4935)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@umamaheswararao umamaheswararao umamaheswararao left review comments

@hemantk-12 hemantk-12 hemantk-12 approved these changes

@prashantpogde prashantpogde prashantpogde approved these changes

@smengcl smengcl smengcl approved these changes

Assignees

@aswinshakil aswinshakil

Labels
snapshot https://issues.apache.org/jira/browse/HDDS-6517
Projects
None yet
Milestone
No milestone
5 participants
@aswinshakil @umamaheswararao @hemantk-12 @prashantpogde @smengcl