TEZ-4363: Bump protobuf dependency to 3.x #192

mark-bathori · 2022-02-28T14:16:06Z

No description provided.

tez-yetus · 2022-02-28T15:40:44Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/2/console in case of problems.

tez-yetus · 2022-02-28T21:12:23Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/3/console in case of problems.

abstractdog · 2022-03-18T13:11:21Z

tez-dag/src/main/java/org/apache/tez/dag/history/events/AMLaunchedEvent.java

@@ -90,7 +91,7 @@ public void toProtoStream(CodedOutputStream outputStream) throws IOException {

  @Override
  public void fromProtoStream(CodedInputStream inputStream) throws IOException {
-    AMLaunchedProto proto = inputStream.readMessage(AMLaunchedProto.PARSER, null);
+    AMLaunchedProto proto = inputStream.readMessage(AMLaunchedProto.parser(), ExtensionRegistry.newInstance());


is this PARSER -> parser() change backward compatible?

what is ExtensionRegistry.newInstance() about exactly?

in general, I want to avoid every direct change in the first place which makes tez unable to compile on 2.5

The PARSER field is deprecated in version 3 but replacing it with parser() is breaking backward compatibility so this needs to be reverted.

Protobuf3 introduced a nullcheck on the extensionRegistry parameter so if it gets null value then it will throw a NullpointerException, that’s why I replaced it with ExtensionRegistry.newInstance() which is creating an empty instance (after checking this getEmptyRegistry() would be better since it is creating empty maps for the new instance)
I checked the generated code both on Protobuf2 and Protobuf3 but it seems that extensionRegistry param is not used in our inputStream.readMessage use case.

thanks @mbathori-cloudera, I think now we can move on with PARSER (even if it's deprecated), as we haven't decided about a major tez release yet, so backward compatibility now looks more important to me

guptanikhil007 · 2022-05-01T06:39:36Z

@mbathori-cloudera @abstractdog
Can we prioritize this PR? There is known security issue with protobuf 2.5.0.

abstractdog · 2022-05-02T06:48:25Z

by default, I wanted to merge the benchmark ticket first (TEZ-4362), but considering that the world is moving towards protobuf 3, I believe it's fine to merge this patch first and then do the perf benchmark later
@mbathori-cloudera, could you please rebase the patch on master and let us now if you need any assistance with this PR?

tez-yetus · 2022-05-02T09:45:51Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/5/console in case of problems.

mark-bathori · 2022-05-02T12:23:49Z

@abstractdog I would need some help with the Yetus build. I’ve changed the PROTOBUF_VERSION variable in the install-protobuf.sh but in the build it is still getting “2.5.0” as value. In local build it's working correctly.
Could you help with this?

abstractdog · 2022-05-09T06:39:05Z

@mbathori-cloudera, I was able to reproduce, maybe this is related to TEZ-4300, not sure at the moment
try to build from a sub dir, similarly to precommit, I did below:

rm -rf build-tools/protobuf #this is needed
cd tez-dag
mvn clean test-compile -DskipTests=true

I believe the issue might be related to the automatically downloaded protobuf (under build-tools/protobuf), I'm suspecting that if you build from a folder of a submodule, it's not downloaded automatically before building and still has only 2.5.0 present, hence the failure:

protoc version is 'libprotoc 2.5.0', expected version is '3.19.4'

mark-bathori · 2022-05-09T12:57:45Z

@abstractdog thanks I could reproduce the issue with trying to compile only the submodule. It seems like the protobuf download script is not triggered from the submodules and the compile-protoc will fail. If I add the install-protobuf.sh plugin execution to the submodule it will execute successfully but it doesn't seems to be the proper solution to add it to every submodule's pom file.

amanraj2520 · 2022-05-24T10:14:00Z

@mbathori-cloudera Any updates on this PR. Can we prioritize this fix.

abstractdog · 2022-05-25T05:46:45Z

hey all! I forgot to review @mbathori-cloudera 's last comment so regarding:

it doesn't seem to be the proper solution to add it to every submodule's pom file.

I don't mind if we have to add this to every submodule if it works only this way, as we want to be able to build submodules alone in their subfolders too
it doesn't have to touch all submodules, only the ones with proto files:

find . -name "*.proto" | awk -F'/' '{ print $2}' | uniq

tez-plugins
tez-runtime-library
tez-api
tez-ext-service-tests
tez-runtime-internals
tez-mapreduce
tez-dag

also, I'm wondering if it's enough to define the plugin only once in root pom.xml's pluginManagement and re-declare only "some parts" of it in submodules (so exploit maven's ability to inherit plugins)

tez-yetus · 2022-05-25T08:10:26Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/6/console in case of problems.

mark-bathori · 2022-05-25T12:56:08Z

@abstractdog I’ve added the protobuf install plugin to the required submodules and now they can be built independently on localhost however the PR build is still getting PROTOBUF_VERSION=2.5.0 somehow. The Dockerfile is installing a fix version of protobuf defined there so I'm not really sure if we need to run install-protobuf.sh in this case.

Another issue seems to be that wget is missing from the environment wget: command not found.

abstractdog · 2022-05-25T14:08:02Z

okay, so we have different problems, let me think about the stuck 2.5.0 version
in the meantime, do you have the bandwidth to create new jira for adding wget to the image used in precommit?

mark-bathori · 2022-05-25T14:17:32Z

Sure I'm creating a ticket for the wget issue.

abstractdog · 2022-05-29T18:40:50Z

I think I found the problem
precommit compiles master branch without the patch first, but the yetus image is already built with protobuf 3.x image according to the patch
we cannot make this work easily, the only thing we can do is:

confirm all tests and checks passed for this patch
1a) unit tests look good: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/6/testReport/
1b) there are checkstyle issues, please fix them: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/6/artifact/out/diff-checkstyle-root.txt
merge this change yo master
create a followup PR to ensure correctness, and handle issues if any

tez-yetus · 2022-05-30T08:14:00Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/7/console in case of problems.

mark-bathori · 2022-05-30T12:28:57Z

thanks @abstractdog for checking it, I've updated the checkstyle issues and as I can see the wget patch was merged so now this issue seems to be fixed based on the logs however another problem occured:
/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-192/src/tez-api/../build-tools/install-protobuf.sh: line 42: sudo: command not found

Should I make a ticket and PR for this missing sudo command?

slachiewicz · 2022-05-31T21:21:49Z

would it be possible to use latest 3.21.1 version? it contains artefact for Mac M1 aarm64

we are doing similar upgrade in HIVE apache/hive#3309

and to compile .proto with maven we use https://github.com/os72/protoc-jar-maven-plugin (ie no need to have .sh to download protoc)

<plugin>
        <groupId>com.github.os72</groupId>
        <artifactId>protoc-jar-maven-plugin</artifactId>
        <version>3.11.4</version>
        <executions>
          <execution>
            <phase>generate-sources</phase>
            <goals>
              <goal>run</goal>
            </goals>
            <configuration>
              <protocArtifact>com.google.protobuf:protoc:${protobuf.version}</protocArtifact>

abstractdog · 2022-06-01T06:24:37Z

thanks @abstractdog for checking it, I've updated the checkstyle issues and as I can see the wget patch was merged so now this issue seems to be fixed based on the logs however another problem occured: /home/jenkins/jenkins-home/workspace/tez-multibranch_PR-192/src/tez-api/../build-tools/install-protobuf.sh: line 42: sudo: command not found

Should I make a ticket and PR for this missing sudo command?

yes, please
can you reproduce this locally for sudo to check further problems?
I believe when you build the docker image, you can test it locally by something like:

docker run -v $HOME:$HOME -w `pwd` -e "HOME=$HOME" "asdfouwergwerg_new_image_hash_here_23045828345" /bin/bash -c "mvn clean install -s $HOME/.m2/settings.xml -DskipTests"

here, you might try to attach your tez folder to docker's home I guess

abstractdog · 2022-06-01T06:28:30Z

would it be possible to use latest 3.21.1 version? it contains artefact for Mac M1 aarm64

we are doing similar upgrade in HIVE apache/hive#3309

and to compile .proto with maven we use https://github.com/os72/protoc-jar-maven-plugin (ie no need to have .sh to download protoc)
<plugin>
        <groupId>com.github.os72</groupId>
        <artifactId>protoc-jar-maven-plugin</artifactId>
        <version>3.11.4</version>
        <executions>
          <execution>
            <phase>generate-sources</phase>
            <goals>
              <goal>run</goal>
            </goals>
            <configuration>
              <protocArtifact>com.google.protobuf:protoc:${protobuf.version}</protocArtifact>

yes, thanks @slachiewicz, 3.21.1 makes sense, let's do it in this PR (cc: @mbathori-cloudera)

regarding protoc-jar-maven-plugin, currently we use this for building protos:

tez/tez-api/pom.xml

Lines 170 to 198 in 24a77c9

    
           <plugin> 
        
             <groupId>org.apache.hadoop</groupId> 
        
             <artifactId>hadoop-maven-plugins</artifactId> 
        
             <executions> 
        
               <execution> 
        
                 <id>compile-protoc</id> 
        
                 <phase>generate-sources</phase> 
        
                 <goals> 
        
                   <goal>protoc</goal> 
        
                 </goals> 
        
                 <configuration> 
        
                   <protocVersion>${protobuf.version}</protocVersion> 
        
                   <protocCommand>${protoc.path}</protocCommand> 
        
                   <imports> 
        
                     <param>${basedir}/src/main/proto</param> 
        
                   </imports> 
        
                   <source> 
        
                     <directory>${basedir}/src/main/proto</directory> 
        
                     <includes> 
        
                       <include>DAGApiRecords.proto</include> 
        
                       <include>DAGClientAMProtocol.proto</include> 
        
                       <include>Events.proto</include> 
        
                     </includes> 
        
                   </source> 
        
                   <output>${project.build.directory}/generated-sources/java</output> 
        
                 </configuration> 
        
               </execution> 
        
             </executions> 
        
           </plugin>

I guess we can move to protoc-jar-maven-plugin

the .sh for downloading protoc is different than generating the sources I think

mark-bathori · 2022-06-01T13:01:03Z

@abstractdog I've built and tested locally the docker image and I've managed to run the protobuf install script after adding sudo, so it seems like it was the last missing command. I've made a ticket and a PR for this change: TEZ-4421

I also bumped the protobuf version to 3.21.1 which needed some further fixing in the install script since they removed the major version number from the file names e.g:

https://github.com/protocolbuffers/protobuf/releases/download/v21.1/protoc-21.1-linux-x86_64.zip
https://github.com/protocolbuffers/protobuf/releases/download/v3.19.4/protoc-3.19.4-linux-x86_64.zip

I'll update this PR after TEZ-4421 gets merged.

tez-yetus · 2022-06-01T18:42:52Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/8/console in case of problems.

tez-yetus · 2022-06-02T15:44:30Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/9/console in case of problems.

mark-bathori · 2022-06-02T20:58:34Z

@abstractdog I replaced the protobuf install and compile plugin with protoc-jar-maven-plugin as slachiewicz recommended it. It handles the plugin download and compilation so I think we can get rid of the install-protobuf.sh script. It also supports version 2.5.0 so we can keep the backward compatibility. I’ve run a couple of test build locally for different versions and they worked properly.
Unfortunately the PR build won't succeed on this either, since it still use the master branch for compilation without the PR changes.
What do you think about this change?

abstractdog · 2022-06-06T07:53:53Z

protoc-jar-maven-plugin absolutely makes sense to me, however I would like to see it in a different PR, as it's not related to protobuf 3.x upgrade, could you please take care of it @mbathori-cloudera?

mark-bathori · 2022-06-08T14:24:37Z

@abstractdog I've created a ticket for the protobuf plugin change (TEZ-4428) and a pull request for it: #218

abstractdog · 2022-06-21T07:46:04Z

@mbathori-cloudera: could you update this PR with the latest patch? please squash commits together for easier reference, thanks!

tez-yetus · 2022-06-24T13:01:27Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/console in case of problems.

tez-yetus · 2022-06-24T15:56:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	36m 45s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+0 🆗	shelldocs	0m 1s	Shelldocs was not available.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	7m 2s	Maven dependency ordering for branch
-1 ❌	mvninstall	5m 58s	root in master failed.
-1 ❌	compile	0m 47s	tez-dag in master failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
-1 ❌	compile	0m 37s	tez-protobuf-history-plugin in master failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
-1 ❌	compile	0m 41s	root in master failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
-1 ❌	compile	0m 39s	tez-dag in master failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.
-1 ❌	compile	0m 33s	tez-protobuf-history-plugin in master failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.
-1 ❌	compile	0m 36s	root in master failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.
-0 ⚠️	checkstyle	0m 57s	The patch fails to run checkstyle in root
-1 ❌	javadoc	0m 45s	tez-dag in master failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
-1 ❌	javadoc	0m 32s	tez-protobuf-history-plugin in master failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
-1 ❌	javadoc	0m 37s	root in master failed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.
-1 ❌	javadoc	1m 34s	root in master failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.
+0 🆗	spotbugs	7m 5s	Used deprecated FindBugs config; considering switching to SpotBugs.
-1 ❌	findbugs	0m 41s	tez-dag in master failed.
-1 ❌	findbugs	0m 35s	tez-protobuf-history-plugin in master failed.
-1 ❌	findbugs	1m 6s	root in master failed.
-0 ⚠️	patch	7m 30s	Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 19s	Maven dependency ordering for patch
+1 💚	mvninstall	6m 32s	the patch passed
+1 💚	compile	4m 4s	the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚	javac	0m 52s	tez-dag-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 0 unchanged - 200 fixed = 0 total (was 200)
+1 💚	javac	0m 39s	tez-plugins_tez-protobuf-history-plugin-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 0 unchanged - 200 fixed = 0 total (was 200)
-1 ❌	javac	2m 33s	root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 4 new + 0 unchanged - 200 fixed = 4 total (was 200)
+1 💚	compile	3m 39s	the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 47s	tez-dag-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 0 new + 0 unchanged - 200 fixed = 0 total (was 200)
+1 💚	javac	0m 37s	tez-plugins_tez-protobuf-history-plugin-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 0 new + 0 unchanged - 200 fixed = 0 total (was 200)
-1 ❌	javac	2m 15s	root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 4 new + 0 unchanged - 200 fixed = 4 total (was 200)
+1 💚	checkstyle	2m 9s	the patch passed
+1 💚	hadolint	0m 3s	There were no new hadolint issues.
+1 💚	shellcheck	0m 1s	There were no new shellcheck issues.
+1 💚	whitespace	0m 1s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
-1 ❌	javadoc	0m 46s	tez-dag-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 2 new + 0 unchanged - 99 fixed = 2 total (was 99)
+1 💚	javadoc	0m 34s	tez-plugins_tez-protobuf-history-plugin-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 0 unchanged - 31 fixed = 0 total (was 31)
-1 ❌	javadoc	2m 17s	root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 81 new + 0 unchanged - 99 fixed = 81 total (was 99)
+1 💚	javadoc	0m 44s	tez-dag-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 0 new + 0 unchanged - 100 fixed = 0 total (was 100)
+1 💚	javadoc	0m 33s	tez-plugins_tez-protobuf-history-plugin-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 0 new + 0 unchanged - 32 fixed = 0 total (was 32)
+1 💚	javadoc	1m 55s	root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~~20.04-b07 generated 0 new + 0 unchanged - 484 fixed = 0 total (was 484)
+1 💚	findbugs	8m 59s	the patch passed
		_ Other Tests _
+1 💚	unit	5m 32s	tez-dag in the patch passed.
+1 💚	unit	0m 45s	tez-protobuf-history-plugin in the patch passed.
+1 💚	unit	69m 35s	root in the patch passed.
+1 💚	asflicense	1m 52s	The patch does not generate ASF License warnings.
		175m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/Dockerfile
GITHUB PR	#192
JIRA Issue	TEZ-4363
Optional Tests	dupname asflicense hadolint shellcheck shelldocs javac javadoc unit xml compile spotbugs findbugs checkstyle
uname	Linux a1e51856f9cb 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `adcc313`
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
mvninstall	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-mvninstall-root.txt
compile	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-compile-tez-dag-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt
compile	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-compile-tez-plugins_tez-protobuf-history-plugin-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt
compile	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt
compile	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-compile-tez-dag-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt
compile	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-compile-tez-plugins_tez-protobuf-history-plugin-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt
compile	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt
checkstyle	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/buildtool-branch-checkstyle-root.txt
javadoc	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-javadoc-tez-dag-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt
javadoc	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-javadoc-tez-plugins_tez-protobuf-history-plugin-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt
javadoc	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-javadoc-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt
javadoc	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-javadoc-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt
findbugs	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-findbugs-tez-dag.txt
findbugs	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-findbugs-tez-plugins_tez-protobuf-history-plugin.txt
findbugs	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/branch-findbugs-root.txt
javac	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/diff-compile-javac-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt
javac	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt
javadoc	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/diff-javadoc-javadoc-tez-dag-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt
javadoc	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/artifact/out/diff-javadoc-javadoc-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/testReport/
Max. process+thread count	2089 (vs. ulimit of 5500)
modules	C: tez-dag tez-plugins/tez-protobuf-history-plugin . U: .
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-192/10/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.1 hadolint=1.18.0-0-g76eee5c
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

amanraj2520 · 2022-07-13T07:21:31Z

@abstractdog @mark-bathori Has this change completed or are there further patches that need to be done. Please let me know if I can also help in getting this done.

amanraj2520 · 2022-07-20T06:04:36Z

@abstractdog @mark-bathori Are there any blockers for this. This has been idle for over couple of months

abstractdog · 2022-07-20T13:10:23Z

checked locally, test failures are only because of the nature of precommit: we can change protobuf version on-the-fly locally by -Dprotobuf.version, also can change the protobuf version in yetus docker image, but we don't guarantee they're in sync
for example: we build yetus image (protobuf 3.x), but run mvn install without the patch also before testing (protobuf 2.x)

so failures should not happen when this patch is committed to master

what I checked locally, all passed:

#applied this patch
mvn clean install -DskipTests #protobuf version is the new 3.x
mvn clean install -pl tez-dag #protobuf version is the new 3.x, run some tests
mvn clean install -DskipTests-Dprotobuf.version=2.5.0
mvn clean install -pl tez-dag -Dprotobuf.version=2.5.0, run some tests

abstractdog

LGTM +1

amanraj2520 · 2022-07-22T07:38:22Z

@abstractdog LGTM, +1. Can we merge these changes ?

TEZ-4363: Bump protobuf dependency to 3.x

e6e05e2