Unresolved compilation problem: org.apache.hadoop.shaded.com.google.protobuf.GeneratedMessage$Builder cannot be resolved. It is indirectly referenced from required .class files

[mjwall]

Building an assembly from master and running with a local Hadoop 3.1.2, I ran into issues with the Master and Tserver. The err log for both was the same

Exception in thread "tserver" java.lang.Error: Unresolved compilation problem: 
	The type org.apache.hadoop.shaded.com.google.protobuf.GeneratedMessage$Builder cannot be resolved. It is indirectly referenced from required .class files

	at org.apache.accumulo.server.security.delegation.AuthenticationTokenSecretManager.<init>(AuthenticationTokenSecretManager.java:1)
	at org.apache.accumulo.tserver.TabletServer.<init>(TabletServer.java:426)
	at org.apache.accumulo.tserver.TabletServer.main(TabletServer.java:3260)
	at org.apache.accumulo.tserver.TServerExecutable.execute(TServerExecutable.java:43)
	at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:129)
	at java.lang.Thread.run(Thread.java:748)

So what is using that?

cd server/base
mvn dependency:copy-dependencies
cd target/dependency
jar tf hadoop-client-runtime-3.1.1.jar  | grep GeneratedMesage$Builder\.class
org/apache/hadoop/shaded/com/google/protobuf/GeneratedMessage$Builder.class

So this too looks like a problem with the runtime jar similar to https://bugs.eclipse.org/bugs/show_bug.cgi?id=541772

That jar and class are in the Hadoop install

cd $HADOOP_HOME
jar tf share/hadoop/client/hadoop-client-runtime-3.1.2.jar | grep GeneratedMessage$Builder\.class
org/apache/hadoop/shaded/com/google/protobuf/GeneratedMessage.class

The tserver process is not running, but the master is still running even though the monitor can not connect to it. So I can check that jar is on the classpath.

jcmd | grep master
9597 org.apache.accumulo.start.Main master
jcmd 9597 VM.system_properties | grep ^java.class.path | tr ':' '\n' | grep runtime
HADOOP_HOME/share/hadoop/client/hadoop-client-runtime-3.1.2.jar\
HADOOP_HOME/share/hadoop/client/hadoop-client-runtime-3.1.2.jar\

It is there, but why twice?

I do not have protoc installed but I don't think that is the issue.

[ctubbsii]

Does all this work with 3.1.1?

[keith-turner]

HADOOP_HOME/share/hadoop/client/hadoop-client-runtime-3.1.2.jar

Is some config missing a a $? Should HADOOP_HOME be $HADOOP_HOME?

[mjwall]

Does all this work with 3.1.1?

I can test that. Do you know of a change between 3.1.1 and 3.1.2?

[mjwall]

Is some config missing a a $? Should HADOOP_HOME be $HADOOP_HOME?

No, HADOOP_HOME is my edit. I was a full path on my filesystem

[ctubbsii]

Does all this work with 3.1.1?

I can test that. Do you know of a change between 3.1.1 and 3.1.2?

No. The default build uses 3.1.1. Since I haven't seen this issue, I'm trying to determine if the problem is your setup, or a change in Hadoop. I've recently tried 3.1.1, and 3.2.0 and have not encountered this problem.

[mjwall]

No. The default build uses 3.1.1. Since I haven't seen this issue, I'm trying to determine if the problem is your setup, or a change in Hadoop. I've recently tried 3.1.1, and 3.2.0 and have not encountered this problem.

K, good to know it worked for you with 3.1.1. I'll try that

[mjwall]

No. The default build uses 3.1.1. Since I haven't seen this issue, I'm trying to determine if the problem is your setup, or a change in Hadoop. I've recently tried 3.1.1, and 3.2.0 and have not encountered this problem.

K, good to know it worked for you with 3.1.1. I'll try that

Hadoop 3.1.1 is having the same issue. I will keep digging

[ctubbsii]

Is this only in Eclipse, or is this when compiling from the command-line also? What OS/platform/JDK? Do you have steps to reproduce?
Have you tried running with https://github.com/apache/fluo-uno/ or https://github.com/apache/fluo-muchos/ ?

[mjwall]

No IDE, just mvn clean package on master and running from the build tar.gz.

I do not have protobuf installed, but I can't see how that would be the issue. I see the runtime jar being loaded, something weird is happening.

Going to try some different things, like building 1.9.3 and running against that. Mostly just wanted to document.

Thanks @ctubbsii

[ctubbsii]

Try cleaning out your local maven repository, like:

rm -rf ~/.m2/repository/org/apache/{hadoop,accumulo}

[mjwall]

1.9.2 ran fine on Hadoop 3.1.1 after I tweaked the bin/accumulo script to load jars from HADOOP_HOME/share/common/lib on the classpath. Did run into one issue with bootstrap_config.sh that I will document, but not this issue.

~/.m2 was empty when I built, so I don't think that is it.

[ctubbsii]

FWIW, the workaround for https://bugs.eclipse.org/bugs/show_bug.cgi?id=541772 is to ensure hadoop-client-runtime is on your compile-time class path. This indicates that perhaps hadoop-client-runtime isn't on your runtime class path for running the tserver.

You will probably need to edit CLASSPATH in your conf/accumulo-env.sh script. By default, that script puts ${HADOOP_HOME}/share/hadoop/client/* on the CLASSPATH, but that may not be applicable to your installation.

[mjwall]

I am unable to replicate now, not sure changed but I am guessing it is on my system. Closing.

[seemasinghh]

These items were not used in the code at all, and should not have been required for compilation error marker occurred in the editor on the first line of the file. Again, this class had no compile-time references to this class is only used within the runtime library. Read more: BigData Hadoop Training in Delhi