Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-4074. [OFS] Implement AbstractFileSystem for RootedOzoneFileSystem #1330

Merged
merged 4 commits into from Aug 25, 2020

Conversation

adoroszlai
Copy link
Contributor

What changes were proposed in this pull request?

Add RootedOzFs, implementation of AbstractFileSystem for RootedOzoneFileSystem. This will allow running MapReduce tests with OFS.

Extracted from #1088. Also applied the fix from HDDS-3482.

https://issues.apache.org/jira/browse/HDDS-4074

How was this patch tested?

Executed ozone-mr acceptance tests locally:

export OZONE_TEST_SELECTOR='ozone-mr\|hadoop..'
hadoop-ozone/dev-support/checks/acceptance.sh

after tweaking config and MR test to use ofs instead of o3fs:

diff --git hadoop-ozone/dist/src/main/compose/ozone-mr/common-config hadoop-ozone/dist/src/main/compose/ozone-mr/common-config
index 46e752731..91588aa77 100644
--- hadoop-ozone/dist/src/main/compose/ozone-mr/common-config
+++ hadoop-ozone/dist/src/main/compose/ozone-mr/common-config
@@ -30,7 +30,8 @@ OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
 HDFS-SITE.XML_dfs.datanode.address=0.0.0.0:1019
 HDFS-SITE.XML_dfs.datanode.http.address=0.0.0.0:1012

-CORE-SITE.xml_fs.defaultFS=o3fs://bucket1.volume1/
+CORE-SITE.xml_fs.defaultFS=ofs://om/volume1/bucket1/
+CORE-SITE.XML_fs.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem

 MAPRED-SITE.XML_mapreduce.framework.name=yarn
 MAPRED-SITE.XML_yarn.app.mapreduce.am.env=HADOOP_MAPRED_HOME=$HADOOP_HOME
diff --git hadoop-ozone/dist/src/main/smoketest/mapreduce.robot hadoop-ozone/dist/src/main/smoketest/mapreduce.robot
index 654dd49c6..e2dfd11a1 100644
--- hadoop-ozone/dist/src/main/smoketest/mapreduce.robot
+++ hadoop-ozone/dist/src/main/smoketest/mapreduce.robot
@@ -39,5 +39,5 @@ Execute PI calculation
 Execute WordCount
                     ${exampleJar}    Find example jar
                     ${random}        Generate Random String  2   [NUMBERS]
-                    ${output} =      Execute                 yarn jar ${exampleJar} wordcount o3fs://bucket1.volume1/key1 o3fs://bucket1.volume1/key1-${random}.count
+                    ${output} =      Execute                 yarn jar ${exampleJar} wordcount ofs://om/volume1/bucket1/key1 ofs://om/volume1/bucket1/key1-${random}.count
                     Should Contain   ${output}               completed successfully

Regular CI:
https://github.com/adoroszlai/hadoop-ozone/runs/975699201

@smengcl
Copy link
Contributor

smengcl commented Aug 24, 2020

@elek Would you like to take a look at this?

Copy link
Member

@elek elek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 Merging it

Thanks @adoroszlai and @smengcl

@elek elek merged commit 122eac5 into apache:master Aug 25, 2020
@adoroszlai
Copy link
Contributor Author

Thanks @elek for reviewing and merging it.

Unfortunately GitHub UI merge assigned me as author of the commit, sorry @smengcl about that.

errose28 added a commit to errose28/ozone that referenced this pull request Aug 25, 2020
* master:
  HDDS-4111. Keep the CSI.zh.md consistent with CSI.md (apache#1320)
  HDDS-3654. Let backgroundCreator create pipeline for the support replication factors alternately (apache#984)
  HDDS-4112. Improve SCM webui page performance (apache#1323)
  HDDS-4074. [OFS] Implement AbstractFileSystem for RootedOzoneFileSystem (apache#1330)
  HDDS-4144. Update version info in hadoop client dependency readme (apache#1348)
  HDDS-4139. Update version number in upgrade tests (apache#1347)
@smengcl
Copy link
Contributor

smengcl commented Aug 27, 2020

@adoroszlai No worries! Thanks you for creating this PR )

@adoroszlai adoroszlai deleted the HDDS-4074 branch August 28, 2020 06:30
rakeshadr pushed a commit to rakeshadr/hadoop-ozone that referenced this pull request Sep 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants