YARN-10536. Client in distributedShell swallows interrupt exceptions #2554

amahussein · 2020-12-17T03:30:54Z

While investigating YARN-10334 I found that in
applications.distributedshell.Client , the method monitorApplication loops waiting for the following conditions:

Application fails: reaches YarnApplicationState.KILLED, or YarnApplicationState.FAILED
Application succeeds: FinalApplicationStatus.SUCCEEDED or YarnApplicationState.FINISHED
the time spent waiting is longer than clientTimeout (if it exists in the parameters).

When the Client thread is interrupted, it ignores the exception:

      // Check app status every 1 second.
      try {
        Thread.sleep(1000);
      } catch (InterruptedException e) {
        LOG.debug("Thread sleep in monitoring loop interrupted");
      }

Also, the timeout will be measured against the object creation timestamp which is incorrect.

hadoop-yetus · 2020-12-17T05:13:10Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 38s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 31s		trunk passed
+1 💚	compile	0m 25s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 24s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 23s		trunk passed
+1 💚	mvnsite	0m 29s		trunk passed
+1 💚	shadedclient	16m 16s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 29s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 21s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	0m 46s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 44s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 27s		the patch passed
+1 💚	compile	0m 21s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 21s		the patch passed
+1 💚	compile	0m 20s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 20s		the patch passed
-0 ⚠️	checkstyle	0m 16s	/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell: The patch generated 1 new + 150 unchanged - 5 fixed = 151 total (was 155)
+1 💚	mvnsite	0m 23s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 22s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 20s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 18s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	0m 47s		the patch passed
			_ Other Tests _
-1 ❌	unit	24m 48s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt	hadoop-yarn-applications-distributedshell in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		101m 4s

Reason	Tests
Failed junit tests	hadoop.yarn.applications.distributedshell.TestDistributedShell

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2554/1/artifact/out/Dockerfile
GITHUB PR	#2554
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 1445b247b16b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4c033ba`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2554/1/testReport/
Max. process+thread count	779 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2554/1/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

amahussein · 2020-12-17T14:19:19Z

The changes should be fine to merge.
The two failing tests are failing for more than a year. there are opened jiras to address them.

hadoop-yetus · 2020-12-17T16:13:23Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	33m 51s		trunk passed
+1 💚	compile	0m 30s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 27s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 28s		trunk passed
+1 💚	mvnsite	0m 32s		trunk passed
+1 💚	shadedclient	17m 21s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 29s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 23s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	0m 47s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 45s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s		the patch passed
+1 💚	compile	0m 20s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 20s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 18s		the patch passed
+1 💚	checkstyle	0m 17s		hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell: The patch generated 0 new + 150 unchanged - 5 fixed = 150 total (was 155)
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	15m 44s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 21s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 22s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	0m 49s		the patch passed
			_ Other Tests _
-1 ❌	unit	24m 51s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt	hadoop-yarn-applications-distributedshell in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		102m 14s

Reason	Tests
Failed junit tests	hadoop.yarn.applications.distributedshell.TestDistributedShell

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2554/2/artifact/out/Dockerfile
GITHUB PR	#2554
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux e31b8b30d089 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `4c033ba`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2554/2/testReport/
Max. process+thread count	764 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2554/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

goiri · 2020-12-17T17:23:39Z

...tributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java

-        return false;
+        LOG.info("Application did not finish. YarnState={}, DSFinalStatus={}. "
+                + "Breaking monitoring loop",
+            state.toString(), dsStatus.toString());


No need for toString()

Thanks @goiri ! I added a commit to get rid of the "toString()".

hadoop-yetus · 2020-12-17T20:15:39Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚		0m 0s	test4tests	The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 25s		trunk passed
+1 💚	compile	0m 30s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	compile	0m 27s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	checkstyle	0m 28s		trunk passed
+1 💚	mvnsite	0m 31s		trunk passed
+1 💚	shadedclient	16m 19s		branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 29s		trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 25s		trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+0 🆗	spotbugs	0m 46s		Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 44s		trunk passed
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s		the patch passed
+1 💚	compile	0m 21s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javac	0m 21s		the patch passed
+1 💚	compile	0m 20s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	javac	0m 20s		the patch passed
+1 💚	checkstyle	0m 19s		hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell: The patch generated 0 new + 150 unchanged - 5 fixed = 150 total (was 155)
+1 💚	mvnsite	0m 22s		the patch passed
+1 💚	whitespace	0m 0s		The patch has no whitespace issues.
+1 💚	shadedclient	14m 45s		patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 23s		the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
+1 💚	javadoc	0m 21s		the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
+1 💚	findbugs	0m 45s		the patch passed
			_ Other Tests _
-1 ❌	unit	24m 45s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-distributedshell.txt	hadoop-yarn-applications-distributedshell in the patch passed.
+1 💚	asflicense	0m 32s		The patch does not generate ASF License warnings.
		98m 44s

Reason	Tests
Failed junit tests	hadoop.yarn.applications.distributedshell.TestDistributedShell

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2554/3/artifact/out/Dockerfile
GITHUB PR	#2554
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux bc6e2f150a4f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `c2672bb`
Default Java	Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2554/3/testReport/
Max. process+thread count	780 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2554/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

goiri · 2020-12-17T20:19:43Z

I don't think we broke the TestDistributedShell with this, did we?

amahussein · 2020-12-17T21:07:40Z

I don't think we broke the TestDistributedShell with this, did we?

No, we didn't. Those two test cases were broken for almost a year
See YARN-10040

…2554) (cherry picked from commit 7a88f45)

YARN-10536. distributedshell client handles interrupt

c0c7b39

amahussein force-pushed the yarn-10536 branch from 4704192 to c0c7b39 Compare December 17, 2020 14:29

goiri changed the title ~~YARN-10536. distributedshell client handles interrupt~~ YARN-10536. Client in distributedShell swallows interrupt exceptions Dec 17, 2020

goiri approved these changes Dec 17, 2020

View reviewed changes

YARN-10536. address code reviews.

c985342

goiri approved these changes Dec 17, 2020

View reviewed changes

goiri merged commit 7a88f45 into apache:trunk Dec 17, 2020

asfgit pushed a commit that referenced this pull request Jan 9, 2021

YARN-10536. Client in distributedShell swallows interrupt exceptions (#…

a40eabd

…2554) (cherry picked from commit 7a88f45)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-10536. Client in distributedShell swallows interrupt exceptions #2554

YARN-10536. Client in distributedShell swallows interrupt exceptions #2554

amahussein commented Dec 17, 2020

hadoop-yetus commented Dec 17, 2020

amahussein commented Dec 17, 2020

hadoop-yetus commented Dec 17, 2020

goiri Dec 17, 2020

amahussein Dec 17, 2020

hadoop-yetus commented Dec 17, 2020

goiri commented Dec 17, 2020

amahussein commented Dec 17, 2020

YARN-10536. Client in distributedShell swallows interrupt exceptions #2554

YARN-10536. Client in distributedShell swallows interrupt exceptions #2554

Conversation

amahussein commented Dec 17, 2020

hadoop-yetus commented Dec 17, 2020

amahussein commented Dec 17, 2020

hadoop-yetus commented Dec 17, 2020

goiri Dec 17, 2020

Choose a reason for hiding this comment

amahussein Dec 17, 2020

Choose a reason for hiding this comment

hadoop-yetus commented Dec 17, 2020

goiri commented Dec 17, 2020

amahussein commented Dec 17, 2020