Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-7916. [Snapshot] Support for Distcp with Ozone Snapshots. #4252

Closed
wants to merge 1 commit into from

Conversation

sadanand48
Copy link
Contributor

What changes were proposed in this pull request?

Main changes

  1. Make org.apache.hadoop.ozone.snapshot.SnapshotDiffReport extend org.apache.hadoop.hdfs.protocol.SnapshotDiffReport
  2. Add Ozone's DistcpSync implementation that provides flexibility to use Ozone specific checks/ functions.

Changes to pom.xml

  1. added hadoop-distcp dependency to ozonefs-common
  2. added hadoop-hdfs-client to ozone-common (required to access org.apache.hadoop.hdfs.protocol.SnapshotDiffReport)
  3. update hadoop version to 3.4.0-Snapshot that contains changes for distcp to work with ozone with snapshots.
  4. Had to update kerby.version as it was throwing Dependency convergence error for org.apache.kerby:kerb-core:jar:1.0.1 since latest version of hadoop uses this version.

this change requires latest version of hadoop that contains HDFS-16911.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7916

How was this patch tested?

Unit tests

@sadanand48 sadanand48 changed the title Distcp with snapshots Ozone change HDDS-7916. [Snapshot] Support for Distcp with Ozone Snapshots. Feb 7, 2023
@aswinshakil aswinshakil added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Feb 27, 2023
@prashantpogde
Copy link
Contributor

@sadanand48 Can you please resolve the conflicts ?

@Override
public void readUnlock(String opName) {
// not used anywhere, implemented as part of interface.
this.lock.writeLock().lock();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

writeLock ??

Copy link
Contributor

@prashantpogde prashantpogde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other changes look good to me.

@smengcl
Copy link
Contributor

smengcl commented Mar 17, 2023

Thanks @sadanand48 . Can we rebase this and mark this as ready for review?

@smengcl smengcl requested a review from jojochuang March 17, 2023 05:50
@smengcl smengcl marked this pull request as ready for review March 28, 2023 18:22
@sadanand48
Copy link
Contributor Author

Broke down this patch into a change which doesn't depend on hdfs version upgrade and created #4491

@smengcl
Copy link
Contributor

smengcl commented Mar 31, 2023

Broke down this patch into a change which doesn't depend on hdfs version upgrade and created #4491

Thanks @sadanand48 . Shall we close this PR then?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
snapshot https://issues.apache.org/jira/browse/HDDS-6517
Projects
None yet
4 participants