HDDS-1525. Mapreduce failure when using Hadoop 2.7.5 #1065
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
As it's a bigger patch here is a quick guidance what is included: This is the original fix by @xiaoyuyao It makes the RpcClient compatible with older hadoop versions ozone-mr dir is reorganized to have subdirectories for mapreduce tests (hadoop 2.7, 3.1, 3.2). ozone-fs dir is deleted as ozone-mr includes all the functionality and it's better structured. Improved to pass custom robot test arguments. you case see the usage in ozone-mr/hadoop27/test.sh Trival fix, the test was wrong until now. This is the second part of the fix (on top of the fixes from @xiaoyuyao. As the FileStatus (hadoop3) is not compatible with FileStatus (hadoop2), we need a class which can be shared between hadoop3 and hadoop2 (FileStatusAdapter, includes only simple java types). This is the third part of the fix. The token files should be available for hadoop2 classloader even if they are the cluster is not secure (to serialize/unserialize tokens). The required classes are copied directly to the jar file. |
|
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
reviewed
Jul 9, 2019
hadoop-yetus
reviewed
Jul 9, 2019
hadoop-yetus
reviewed
Jul 9, 2019
|
💔 -1 overall
This message was automatically generated. |
hadoop-yetus
reviewed
Jul 9, 2019
hadoop-yetus
reviewed
Jul 9, 2019
hadoop-yetus
reviewed
Jul 9, 2019
|
💔 -1 overall
This message was automatically generated. |
|
Thanks @elek for the patch. It looks pretty good to me. +1 |
|
Thanks the review @xiaoyuyao (and the initial fix for the problem). I rebased the version, will commit it soon. Tested again locally and all the mr tests are passed. |
shanthoosh
pushed a commit
to shanthoosh/hadoop
that referenced
this pull request
Oct 15, 2019
…onfigs Author: Prateek Maheshwari <pmaheshwari@apache.org> Reviewers: Ray Matharu <rmatharu@linkedin.com>, Cameron Lee <calee@linkedin.com> Closes apache#1065 from prateekm/allow-task-inputs-override
amahussein
pushed a commit
to amahussein/hadoop
that referenced
this pull request
Oct 29, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Integrate Ozone(0.4 branch) with Hadoop 2.7.5, "hdfs dfs -ls /" can pass, while teragen failed.
When add -verbose:class to java options, it shows that class KeyProvider is loaded twice by different classloader while it is only loaded once when execute "hdfs dfs -ls /"
All jars under share/ozone/lib are added into hadoop classpath except ozone file system current lib jar.
See: https://issues.apache.org/jira/browse/HDDS-1525