TEZ-4311: Bump hadoop dependency version to 3.3.x #132

abstractdog · 2021-06-22T20:47:06Z

No description provided.

belugabehr · 2021-06-30T15:22:04Z

Hey, just FYI. I came across this issue when upgrading Hive 3.3.1:

https://issues.apache.org/jira/browse/HIVE-24484?focusedCommentId=17371600&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17371600

Probably needs to be handled by Tez unless HADOOP-17779 is accepted (and released in a subsequent version).

jteagles · 2021-06-30T22:48:21Z

@abstractdog, if we are to start hadoop 3.3 changes we should consider branching branch-0.10. Here's a thought. Instead of calling it 0.11 for hadoop 3.3 support, what if we change version on master to 3.3 to reflect hadoop compatibility.

abstractdog · 2021-07-01T09:35:19Z

@abstractdog, if we are to start hadoop 3.3 changes we should consider branching branch-0.10. Here's a thought. Instead of calling it 0.11 for hadoop 3.3 support, what if we change version on master to 3.3 to reflect hadoop compatibility.

I like the idea, especially because tez is battle-tested and stable for a very long time, deserves more than 0.x
however, this is a historical change, we're in the 0.x line since the very beginning, so I would ask the community if anyone has a strong opinion about this (interested in pros/cons, or an alternative versioning idea if any)

belugabehr · 2021-07-01T13:33:50Z

Hey Guys,

I would be careful hitching the wagon to any other projects. I think it's helpful to be on your own release cadence. However, I think 1.0 is a good first step and would lead the way to some more thoughtful approach to backwards compatibility and stableness (not that these things aren't true now, but it make it more formalized for new folks).

Thanks.

abstractdog · 2021-10-27T19:38:22Z

new failures are found in ShuffleHandler while running tests:

java.lang.NoSuchMethodError: io.netty.buffer.ByteBuf.forEachByte(Lio/netty/buffer/ByteBufProcessor;)I
	at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:791)
	at io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:840)
	at io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:208)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:834)

full log file is attached to JIRA: https://issues.apache.org/jira/secure/attachment/13035409/org.apache.tez.auxservices.TestShuffleHandler-output.txt

abstractdog · 2022-01-02T20:28:00Z

current unit test failure is reproduced as:

 mvn clean test -Dtest=TestATSHistoryWithACLs#testDAGACls -pl ./tez-plugins/tez-yarn-timeline-history-with-acls

related to YARN-9554, it's under investigation

tez-yetus · 2022-01-03T13:46:33Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	13m 15s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 3 new or modified test files.
		_ master Compile Tests _
+0 🆗	mvndep	5m 2s	Maven dependency ordering for branch
+1 💚	mvninstall	8m 22s	master passed
+1 💚	compile	4m 18s	master passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	4m 3s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	2m 47s	master passed
+1 💚	javadoc	4m 12s	master passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	3m 33s	master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗	spotbugs	5m 27s	Used deprecated FindBugs config; considering switching to SpotBugs.
+0 🆗	findbugs	5m 23s	root in master has 2 extant findbugs warnings.
		_ Patch Compile Tests _
+0 🆗	mvndep	1m 25s	Maven dependency ordering for patch
+1 💚	mvninstall	5m 9s	the patch passed
+1 💚	compile	4m 20s	the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	4m 20s	the patch passed
+1 💚	compile	4m 4s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	4m 4s	the patch passed
+1 💚	checkstyle	2m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	javadoc	4m 0s	the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	3m 35s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	findbugs	9m 28s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 18s	tez-api in the patch passed.
+1 💚	unit	5m 35s	tez-runtime-library in the patch passed.
+1 💚	unit	45m 53s	tez-tests in the patch passed.
+1 💚	unit	2m 1s	tez-yarn-timeline-history-with-acls in the patch passed.
+1 💚	unit	77m 57s	root in the patch passed.
+1 💚	asflicense	2m 14s	The patch does not generate ASF License warnings.
		227m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-132/9/artifact/out/Dockerfile
GITHUB PR	#132
JIRA Issue	TEZ-4311
Optional Tests	dupname asflicense javac javadoc unit xml compile spotbugs findbugs checkstyle
uname	Linux 506d36c30493 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `7e09dfa`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-132/9/testReport/
Max. process+thread count	2099 (vs. ulimit of 5500)
modules	C: tez-api tez-runtime-library tez-tests tez-plugins/tez-yarn-timeline-history-with-acls . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-132/9/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2022-01-03T15:17:53Z

@jteagles: could you please take a look at the commits?
finally, it looks like we're quite close to the hadoop 3.3.x upgrade, because several issues were solved in hadoop 3.3.1 (that were present in 3.3.0), now I got a clean precommit test run here

jteagles

+1. I feel we are ready for this change. Probably we should plan a release once we can certify

abstractdog · 2022-01-06T19:47:08Z

thanks @jteagles, I'm about to commit this soon
just double-checked, the new code can still be successfully built with the older hadoop:

mvn clean install -DskipTests -Dhadoop.version=3.2.2