Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HADOOP-13500. Synchronizing iteration of Configuration properties object #3775

Merged
merged 2 commits into from Dec 17, 2021
Merged

HADOOP-13500. Synchronizing iteration of Configuration properties object #3775

merged 2 commits into from Dec 17, 2021

Conversation

dbadaya1
Copy link
Contributor

@dbadaya1 dbadaya1 commented Dec 9, 2021 •
edited by aajisaka

Description of PR

It is possible to encounter a ConcurrentModificationException while trying to iterate a Configuration object. The iterator method tries to walk the underlying Property object without proper synchronization, so another thread simultaneously calling the set method can trigger it. setProperty method on Property object is also synchronized on the same property object.

How was this patch tested?

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@dbadaya1 dbadaya1 changed the title HADOOP-13500. Synchronizing iteration of underlying properties object… HADOOP-13500. Synchronizing iteration of Configuration properties object Dec 9, 2021
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 42s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
-1 ❌ mvninstall 32m 55s /branch-mvninstall-root.txt root in trunk failed.
+1 💚 compile 23m 25s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 19m 57s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 1m 8s trunk passed
+1 💚 mvnsite 1m 41s trunk passed
+1 💚 javadoc 1m 14s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 45s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 31s trunk passed
+1 💚 shadedclient 22m 17s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 0s the patch passed
+1 💚 compile 22m 29s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 22m 29s the patch passed
+1 💚 compile 20m 21s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 20m 21s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 1m 3s /results-checkstyle-hadoop-common-project_hadoop-common.txt hadoop-common-project/hadoop-common: The patch generated 2 new + 217 unchanged - 4 fixed = 219 total (was 221)
+1 💚 mvnsite 1m 37s the patch passed
+1 💚 javadoc 1m 4s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 38s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 44s the patch passed
+1 💚 shadedclient 23m 54s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 17m 39s /patch-unit-hadoop-common-project_hadoop-common.txt hadoop-common in the patch passed.
+1 💚 asflicense 1m 0s The patch does not generate ASF License warnings.
201m 53s
Reason Tests
Failed junit tests hadoop.ipc.TestIPC
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3775/1/artifact/out/Dockerfile
GITHUB PR #3775
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux cc2b4759a499 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 04c1d9d2bba363436b0dc7d9cf7731953bddc257
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3775/1/testReport/
Max. process+thread count 1258 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3775/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@aajisaka aajisaka self-requested a review December 14, 2021 06:33
aajisaka
aajisaka requested changes Dec 14, 2021
View reviewed changes
Copy link
Member

@aajisaka aajisaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @dbadaya1, thank you for providing the patch. Would you fix the check style issue?

https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3775/1/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt

./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java:2981:    Properties properties = getProps();:16: 'properties' hides a field. [HiddenField]
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java:2695:    Configuration conf = new Configuration();:19: 'conf' hides a field. [HiddenField]

We need to rename the variables to fix the warnings.

@dbadaya1
HADOOP-13500. Synchronizing iteration of underlying properties object…
5901f86 
… in Configuration class

It is possible to encounter a ConcurrentModificationException while trying to iterate a Configuration object. The iterator method tries to walk the underlying Property object without proper synchronization, so another thread simultaneously calling the set method can trigger it. setProperty method on Property object is also synchronized on the same property object.
@dbadaya1
Fix checkstyle issues
275ce35
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 40s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
-1 ❌ mvninstall 32m 44s /branch-mvninstall-root.txt root in trunk failed.
+1 💚 compile 22m 13s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 19m 32s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 1m 8s trunk passed
+1 💚 mvnsite 1m 40s trunk passed
+1 💚 javadoc 1m 13s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 42s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 30s trunk passed
+1 💚 shadedclient 22m 11s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 58s the patch passed
+1 💚 compile 21m 23s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 21m 23s the patch passed
+1 💚 compile 19m 20s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 19m 20s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 1m 7s /results-checkstyle-hadoop-common-project_hadoop-common.txt hadoop-common-project/hadoop-common: The patch generated 2 new + 216 unchanged - 4 fixed = 218 total (was 220)
+1 💚 mvnsite 1m 39s the patch passed
+1 💚 javadoc 1m 11s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 43s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 36s the patch passed
+1 💚 shadedclient 22m 15s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 17m 46s /patch-unit-hadoop-common-project_hadoop-common.txt hadoop-common in the patch passed.
+1 💚 asflicense 0m 54s The patch does not generate ASF License warnings.
196m 59s
Reason Tests
Failed junit tests hadoop.ipc.TestIPC
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3775/2/artifact/out/Dockerfile
GITHUB PR #3775
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 264b73ed6a06 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 8d86051576d447412350689eb541d1926762429e
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3775/2/testReport/
Max. process+thread count 1275 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3775/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 12s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
-1 ❌ mvninstall 38m 13s /branch-mvninstall-root.txt root in trunk failed.
+1 💚 compile 24m 9s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 compile 21m 25s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 checkstyle 1m 2s trunk passed
+1 💚 mvnsite 1m 39s trunk passed
+1 💚 javadoc 1m 9s trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 35s trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 27s trunk passed
+1 💚 shadedclient 24m 53s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 0s the patch passed
+1 💚 compile 23m 22s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javac 23m 22s the patch passed
+1 💚 compile 20m 38s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 javac 20m 38s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 1s hadoop-common-project/hadoop-common: The patch generated 0 new + 216 unchanged - 4 fixed = 216 total (was 220)
+1 💚 mvnsite 1m 35s the patch passed
+1 💚 javadoc 1m 4s the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚 javadoc 1m 38s the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚 spotbugs 2m 39s the patch passed
+1 💚 shadedclient 25m 12s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 17m 33s /patch-unit-hadoop-common-project_hadoop-common.txt hadoop-common in the patch passed.
+1 💚 asflicense 0m 49s The patch does not generate ASF License warnings.
214m 7s
Reason Tests
Failed junit tests hadoop.ipc.TestIPC
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3775/3/artifact/out/Dockerfile
GITHUB PR #3775
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 9037bfbefa82 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 275ce35
Default Java Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3775/3/testReport/
Max. process+thread count 3143 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3775/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

aajisaka
aajisaka approved these changes Dec 17, 2021
View reviewed changes
Copy link
Member

@aajisaka aajisaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. The unit test failure is not related and tracked by https://issues.apache.org/jira/browse/HADOOP-18046

@aajisaka aajisaka merged commit 4483607 into apache:trunk Dec 17, 2021
aajisaka pushed a commit that referenced this pull request Dec 17, 2021
@dbadaya1 @aajisaka
HADOOP-13500. Synchronizing iteration of Configuration properties obj…
e7b1f87 
…ect (#3775)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4483607)
aajisaka pushed a commit that referenced this pull request Dec 17, 2021
@dbadaya1 @aajisaka
HADOOP-13500. Synchronizing iteration of Configuration properties obj…
253b217 
…ect (#3775)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4483607)
aajisaka pushed a commit that referenced this pull request Dec 17, 2021
@dbadaya1 @aajisaka
HADOOP-13500. Synchronizing iteration of Configuration properties obj…
f2d12b3 
…ect (#3775)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4483607)
(cherry picked from commit 253b217)
ashutoshcipher pushed a commit to ashutoshcipher/hadoop that referenced this pull request Dec 22, 2021
@dbadaya1 @ashutoshcipher
HADOOP-13500. Synchronizing iteration of Configuration properties obj…
cd37788 
…ect (apache#3775)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
sunchao pushed a commit that referenced this pull request Jan 4, 2022
@dbadaya1 @sunchao
HADOOP-13500. Synchronizing iteration of Configuration properties obj…
4201cdb 
…ect (#3775)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4483607)
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
@dbadaya1
HADOOP-13500. Synchronizing iteration of Configuration properties obj…
574294a 
…ect (apache#3775)

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aajisaka aajisaka aajisaka approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@dbadaya1 @hadoop-yetus @aajisaka