HBASE-24963 [HBOSS] UTs failing when running against hadoop 3.3.

[wchevreuil]

No description provided.

[joshelser]

[INFO] Running org.apache.hadoop.hbase.oss.contract.TestHBOSSContractRenameS3A
[ERROR] Tests run: 10, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 9.049 s <<< FAILURE! - in org.apache.hadoop.hbase.oss.contract.TestHBOSSContractRenameS3A
[ERROR] testRenamePopulatesFileAncestors(org.apache.hadoop.hbase.oss.contract.TestHBOSSContractRenameS3A)  Time elapsed: 0.144 s  <<< ERROR!
java.io.FileNotFoundException: Expected to find a directory: not found s3a://embedded/hboss-junit-fork-0001/contract-tests/testRenamePopulatesFileAncestorsNew in s3a://embedded/hboss-junit-fork-0001/contract-tests
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://embedded/hboss-junit-fork-0001/contract-tests/testRenamePopulatesFileAncestorsNew

[ERROR] testRenameNewFileSameDir(org.apache.hadoop.hbase.oss.contract.TestHBOSSContractRenameS3A)  Time elapsed: 0.095 s  <<< FAILURE!
java.lang.AssertionError: rename(s3a://embedded/hboss-junit-fork-0001/contract-tests/rename_src, s3a://embedded/hboss-junit-fork-0001/contract-tests/rename_dest) returned false

I see this when running locally mvn clean verify -Dhadoop.profile=3 -Dhadoop3.version=3.3.0. Am I doing something wrong?

[apurtell]

I tried mvn clean install -Dhadoop.profile=3 -Dhadoop3.version=3.3.0 -Dhbase.profile=2 -Dhbase2.version=2.3.1

[INFO] Running org.apache.hadoop.hbase.oss.contract.TestHBOSSContractSetTimes
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.363 s <<< FAILURE! - in org.apache.hadoop.hbase.oss.contract.TestHBOSSContractSetTimes
[ERROR] testSetTimesNonexistentFile(org.apache.hadoop.hbase.oss.contract.TestHBOSSContractSetTimes)  Time elapsed: 0.214 s  <<< ERROR!
java.io.IOException: Failed to get FS
Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.lang.NoClassDefFoundError: org/apache/zookeeper/common/X509Exception$SSLContextException
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.common.X509Exception$SSLContextException

Not related to the changes proposed by this PR, but every unit test fails for me.

[apurtell]

When I compile a custom stack based on Hadoop 3.3.0, HBase 2.3.1, and ZooKeeper 3.5.8, such that all of Hadoop, HBase, and ZK versions are harmonized across those components (meaning: Hadoop is built with ZK 3.5.8, HBase is built with Hadoop 3.3.0 hadoop.profile=3.0 and ZK 3.5.8), I still see the ZK related runtime link errors. (NoClassDefFoundError: org/apache/zookeeper/common/X509Exception$SSLContextException)

[wchevreuil]

"Strangerthings here":
1)

I see this when running locally mvn clean verify -Dhadoop.profile=3 -Dhadoop3.version=3.3.0

Tried it locally. First run got same error, but subsequent runs succeeded.

When I compile a custom stack based on Hadoop 3.3.0, HBase 2.3.1, and ZooKeeper 3.5.8, such that all of Hadoop, HBase, and ZK versions are harmonized across those components (meaning: Hadoop is built with ZK 3.5.8, HBase is built with Hadoop 3.3.0 hadoop.profile=3.0 and ZK 3.5.8), I still see the ZK related runtime link errors. (NoClassDefFoundError: org/apache/zookeeper/common/X509Exception$SSLContextException)

Got same result. Among the test errors, saw this The value of property fs.hboss.sync.zk.connectionString must not be null message, which makes me suspect the hbase profile specified is missing some properties definition. Looking further into this, will reply later.

[wchevreuil]

When I compile a custom stack based on Hadoop 3.3.0, HBase 2.3.1, and ZooKeeper 3.5.8, such that all of Hadoop, HBase, and ZK versions are harmonized across those components (meaning: Hadoop is built with ZK 3.5.8, HBase is built with Hadoop 3.3.0 hadoop.profile=3.0 and ZK 3.5.8), I still see the ZK related runtime link errors. (NoClassDefFoundError: org/apache/zookeeper/common/X509Exception$SSLContextException)

Looked further into this, so the real issue is the NCDFE mentioned by @apurtell earlier. Yeah, hboss is defining a ZK dependency on version 3.4.10, whilst HBase 2.3.1 is on 3.5.7.

[wchevreuil]

Right, so here some findings:

1. Setting hbase version to 2.3.1 requires updating ZK version in hboss to 3.5.7, otherwise, we get the NCDFE already mentioned above.
2. Additional adjustment is required in EmbeddedZK class to use loopback address, instead of localhost, per the changes in MiniZooKeeperCluster by HBASE-23993.

[wchevreuil]

Had pushed a new commit, with a change to EmbeddedZK to make sure it works with hbase versions including HBASE-23993 (like 2.3.1), for example. As mentioned on my previous comments, testing against hbase 2.3.1 works as long as ZK version specified is 3.5.7:

mvn clean install -Dhadoop.profile=3 -Dhadoop3.version=3.3.0 -Dhbase.profile=2 -Dhbase2.version=2.3.1 -Dzookeeper.version=3.5.7

@apurtell @joshelser , mind give it another try? I was thinking maybe worth raise these dependency versions on another jira?

[joshelser]

I was thinking maybe worth raise these dependency versions on another jira?

Yeah, don't need to hold up this one fix for a full harmonization of all branches. We can do them in pieces (hopefully with testing to catch each one and keep them stable as we're moving forward).

[joshelser]

mvn clean install -Dhadoop.profile=3 -Dhadoop3.version=3.3.0 -Dhbase.profile=2 -Dhbase2.version=2.3.1 -Dzookeeper.version=3.5.7 worked just fine for me.