…uster".
The reason the SBT test framework failed to put the HBase test artifacts in its classpath is because of an IVY bug: sbt/sbt#861.
SBT/IVY can fail to resolve transitive dependencies defined in a 'pom-only' module, and 'hbase-testing-util' is a pom-only module. (Maven resolves them without any problem.)
The workaround is to replace the hbase-testing-util dependency definition in spark/sql/hbase/pom.xml with the dependencies listed in that pom-only module, at the cost of developers having to edit this pom file to make it work for unprofiled combinations of HBase/Hadoop releases.
=================================
This works only for hadoop2-compat hbase distributions. The spark/sql/hbase/pom.xml must be edited to make it work for hadoop1-compat hbase distributions (more on that later).
The fix has been tested against HBase 0.98.5-hadoop2 and 0.98.7-hadoop2, with hadoop 2.2.0, 2.3.0, and 2.4.0.
=================
SBT test commands
=================
sbt/sbt -Phive,hbase,yarn,hadoop-2.2 -Dhadoop.version=2.2.0 "hbase/test-only org.apache.spark.sql.hbase.BasicQueriesSuite"
sbt/sbt -Phive,hbase,yarn,hadoop-2.3 -Dhadoop.version=2.3.0 "hbase/test-only org.apache.spark.sql.hbase.BasicQueriesSuite"
sbt/sbt -Phive,hbase,yarn,hadoop-2.4 -Dhadoop.version=2.4.0 "hbase/test-only org.apache.spark.sql.hbase.BasicQueriesSuite"
=================
MVN test commands
=================
mvn -e -Pyarn,hbase,hadoop-2.2 -Dhadoop.version=2.2.0 -pl sql/hbase clean test -DwildcardSuites=org.apache.spark.sql.hbase.BasicQueriesSuite
mvn -e -Pyarn,hbase,hadoop-2.3 -Dhadoop.version=2.3.0 -pl sql/hbase clean test -DwildcardSuites=org.apache.spark.sql.hbase.BasicQueriesSuite
mvn -e -Pyarn,hbase,hadoop-2.4 -Dhadoop.version=2.4.0 -pl sql/hbase clean test -DwildcardSuites=org.apache.spark.sql.hbase.BasicQueriesSuite
===========================
To build against a hadoop1-compatible HBase release (for example: Hadoop 1.2.1), you will need to edit the pom to your needs.
Add the appropriate hadoop profile to import the hadoop 1.2.1 libraries, and redefine the hbase dependencies to match the module structure of you chosen hbase release.
The SBT libraryDependencies declaration below describes the mvn dependencies you would need to define in spark/sql/hbase/pom.xml to build
Spark against HBase v0.98.5-hadoop1 over Hadoop v1.2.1.
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "2.2.1" % "test",
"com.novocode" % "junit-interface" % "0.9" % "test",
"org.apache.hbase" % "hbase-common" % "0.98.5-hadoop1",
"org.apache.hbase" % "hbase-common" % "0.98.5-hadoop1" % "test" classifier "tests",
// no hbase-annotation module before v0.98.7
//"org.apache.hbase" % "hbase-annotations" % "0.98.5-hadoop1",
//"org.apache.hbase" % "hbase-annotations" % "0.98.5-hadoop1" % "test" classifier "tests",
"org.apache.hbase" % "hbase-protocol" % "0.98.5-hadoop1",
"org.apache.hbase" % "hbase-client" % "0.98.5-hadoop1",
"org.apache.hbase" % "hbase-server" % "0.98.5-hadoop1",
"org.apache.hbase" % "hbase-server" % "0.98.5-hadoop1" % "test" classifier "tests",
"org.apache.hbase" % "hbase-hadoop-compat" % "0.98.5-hadoop1",
"org.apache.hbase" % "hbase-hadoop-compat" % "0.98.5-hadoop1" % "test" classifier "tests",
"org.apache.hbase" % "hbase-hadoop1-compat" % "0.98.5-hadoop1",
"org.apache.hbase" % "hbase-hadoop1-compat" % "0.98.5-hadoop1" % "test" classifier "tests",
"org.slf4j" % "slf4j-log4j12" % "1.6.4",
"org.apache.hadoop" % "hadoop-core" % "1.2.1",
"org.apache.hadoop" % "hadoop-client" % "1.2.1",
"org.apache.hadoop" % "hadoop-minicluster" % "1.2.1",
"org.apache.hadoop" % "hadoop-test" % "1.2.1",
"org.apache.hadoop" % "hadoop-tools" % "1.2.1"
)
Note that there are no 'hbase-annotations' modules in HBase releases prior to v0.98.7.
