Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAPREDUCE-7371. DistributedCache alternative APIs should not use DistributedCache APIs internally #3855

Merged
merged 3 commits into from Jan 8, 2022

Conversation

virajjasani
Copy link
Contributor

Description of PR

DistributedCache has been deprecated long back and it's still being used as of today. However, all the alternative Job APIs (for deprecated DistributedCache APIs) still internally use DistributedCache APIs only, this is a deadlock and it leaves no room for removal of DistributedCache APIs in future. We should move core logic to Job or JobContext as required and let deprecated DistributedCache APIs point to the right alternatives internally.

How was this patch tested?

Unit Tests

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 55s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 27m 28s Maven dependency ordering for branch
+1 💚 mvninstall 24m 35s trunk passed
+1 💚 compile 24m 5s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 20m 39s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 3m 52s trunk passed
+1 💚 mvnsite 2m 53s trunk passed
+1 💚 javadoc 2m 21s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 2m 11s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 4m 19s trunk passed
+1 💚 shadedclient 23m 30s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for patch
+1 💚 mvninstall 1m 53s the patch passed
+1 💚 compile 23m 31s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 23m 31s root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 0 new + 1867 unchanged - 63 fixed = 1867 total (was 1930)
+1 💚 compile 20m 54s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 20m 54s root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu120.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu120.04-b10 generated 0 new + 1744 unchanged - 63 fixed = 1744 total (was 1807)
-1 ❌ blanks 0m 0s /blanks-tabs.txt The patch 1 line(s) with tabs.
-0 ⚠️ checkstyle 3m 53s /results-checkstyle-root.txt root: The patch generated 6 new + 364 unchanged - 12 fixed = 370 total (was 376)
+1 💚 mvnsite 2m 50s the patch passed
+1 💚 javadoc 2m 17s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 2m 11s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 5m 3s the patch passed
+1 💚 shadedclient 23m 35s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 5m 45s hadoop-mapreduce-client-core in the patch passed.
+1 💚 unit 1m 9s hadoop-mapreduce-client-common in the patch passed.
+1 💚 unit 129m 37s hadoop-mapreduce-client-jobclient in the patch passed.
+1 💚 unit 7m 34s hadoop-streaming in the patch passed.
+1 💚 asflicense 1m 3s The patch does not generate ASF License warnings.
372m 32s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3855/1/artifact/out/Dockerfile
GITHUB PR #3855
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux f64ad3b912c3 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / c27cb37
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3855/1/testReport/
Max. process+thread count 1214 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-streaming U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3855/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 5s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 20m 46s Maven dependency ordering for branch
+1 💚 mvninstall 28m 56s trunk passed
+1 💚 compile 31m 23s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 26m 37s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 4m 33s trunk passed
+1 💚 mvnsite 3m 25s trunk passed
+1 💚 javadoc 2m 40s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 2m 24s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 5m 36s trunk passed
+1 💚 shadedclient 26m 29s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 25s Maven dependency ordering for patch
+1 💚 mvninstall 1m 58s the patch passed
+1 💚 compile 23m 51s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 23m 51s root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 0 new + 1863 unchanged - 67 fixed = 1863 total (was 1930)
+1 💚 compile 20m 40s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 20m 40s root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu120.04-b10 with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu120.04-b10 generated 0 new + 1740 unchanged - 67 fixed = 1740 total (was 1807)
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 3m 50s root: The patch generated 0 new + 363 unchanged - 13 fixed = 363 total (was 376)
+1 💚 mvnsite 2m 51s the patch passed
+1 💚 javadoc 2m 17s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 2m 12s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 5m 4s the patch passed
+1 💚 shadedclient 23m 33s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 5m 45s hadoop-mapreduce-client-core in the patch passed.
+1 💚 unit 1m 10s hadoop-mapreduce-client-common in the patch passed.
+1 💚 unit 127m 24s hadoop-mapreduce-client-jobclient in the patch passed.
+1 💚 unit 6m 49s hadoop-streaming in the patch passed.
+1 💚 asflicense 0m 51s The patch does not generate ASF License warnings.
386m 29s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3855/2/artifact/out/Dockerfile
GITHUB PR #3855
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux ae16b3032651 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1fe00a0
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3855/2/testReport/
Max. process+thread count 1212 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-streaming U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3855/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 2s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 11m 44s Maven dependency ordering for branch
+1 💚 mvninstall 29m 1s trunk passed
+1 💚 compile 24m 57s trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 compile 20m 48s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 3m 54s trunk passed
+1 💚 mvnsite 2m 58s trunk passed
+1 💚 javadoc 2m 23s trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 2m 11s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 4m 22s trunk passed
+1 💚 shadedclient 23m 38s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 25s Maven dependency ordering for patch
+1 💚 mvninstall 1m 54s the patch passed
+1 💚 compile 23m 46s the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 javac 23m 46s root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 0 new + 1863 unchanged - 67 fixed = 1863 total (was 1930)
+1 💚 compile 26m 8s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 26m 8s root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu120.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu120.04-b07 generated 0 new + 1740 unchanged - 67 fixed = 1740 total (was 1807)
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 4m 3s root: The patch generated 0 new + 362 unchanged - 13 fixed = 362 total (was 375)
+1 💚 mvnsite 2m 51s the patch passed
+1 💚 javadoc 2m 16s the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 2m 10s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 6m 19s the patch passed
+1 💚 shadedclient 28m 35s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 6m 41s hadoop-mapreduce-client-core in the patch passed.
+1 💚 unit 1m 22s hadoop-mapreduce-client-common in the patch passed.
+1 💚 unit 141m 33s hadoop-mapreduce-client-jobclient in the patch passed.
+1 💚 unit 7m 55s hadoop-streaming in the patch passed.
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
387m 42s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3855/3/artifact/out/Dockerfile
GITHUB PR #3855
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 8871ebdcb38b 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1fe00a0
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3855/3/testReport/
Max. process+thread count 1206 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-streaming U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3855/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@virajjasani
Copy link
Contributor Author

@jojochuang @aajisaka @tasanuma Could you please take a look?

@tasanuma
Copy link
Member

tasanuma commented Jan 7, 2022

@virajjasani It makes sense to me. We might want to copy the javadocs from DistributedCache to Job and JobContextImpl as well.

@virajjasani
Copy link
Contributor Author

Oh yes, makes sense. Let me put all the Javadocs. Thanks @tasanuma

@tasanuma
Copy link
Member

tasanuma commented Jan 7, 2022

Thanks for updating it. +1, pending Jenkins.

@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 12m 26s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 5 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 13m 6s Maven dependency ordering for branch
+1 💚 mvninstall 22m 14s trunk passed
+1 💚 compile 23m 52s trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 compile 26m 41s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 4m 36s trunk passed
+1 💚 mvnsite 3m 18s trunk passed
+1 💚 javadoc 2m 43s trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 2m 32s trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 5m 6s trunk passed
+1 💚 shadedclient 27m 29s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 29s Maven dependency ordering for patch
+1 💚 mvninstall 2m 16s the patch passed
+1 💚 compile 28m 52s the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 javac 28m 52s root-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 generated 0 new + 1863 unchanged - 67 fixed = 1863 total (was 1930)
+1 💚 compile 25m 7s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 25m 7s root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu120.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu120.04-b07 generated 0 new + 1740 unchanged - 67 fixed = 1740 total (was 1807)
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 4m 28s root: The patch generated 0 new + 363 unchanged - 13 fixed = 363 total (was 376)
+1 💚 mvnsite 3m 22s the patch passed
+1 💚 javadoc 2m 41s the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 2m 32s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 spotbugs 5m 56s the patch passed
+1 💚 shadedclient 20m 46s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 5m 45s hadoop-mapreduce-client-core in the patch passed.
+1 💚 unit 1m 21s hadoop-mapreduce-client-common in the patch passed.
+1 💚 unit 122m 29s hadoop-mapreduce-client-jobclient in the patch passed.
+1 💚 unit 6m 55s hadoop-streaming in the patch passed.
+1 💚 asflicense 1m 5s The patch does not generate ASF License warnings.
382m 28s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3855/4/artifact/out/Dockerfile
GITHUB PR #3855
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 085b5c19889c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / a4615dc
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3855/4/testReport/
Max. process+thread count 1557 (vs. ulimit of 5500)
modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-tools/hadoop-streaming U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3855/4/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@tasanuma tasanuma merged commit 08c803e into apache:trunk Jan 8, 2022
@tasanuma
Copy link
Member

tasanuma commented Jan 8, 2022

Merged it. Thanks for your contribution, @virajjasani!

@steveloughran
Copy link
Contributor

pulling this in to branch-3.3

asfgit pushed a commit that referenced this pull request Jun 22, 2022
…ributedCache APIs internally (#3855)

Contributed by Viraj Jasani
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants