[SPARK-3880] HBase as data source to SparkSQL #4084
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…uster". The reason the SBT test framework failed to put the HBase test artifacts in its classpath is because of an IVY bug: sbt/sbt#861. SBT/IVY can fail to resolve transitive dependencies defined in a 'pom-only' module, and 'hbase-testing-util' is a pom-only module. (Maven resolves them without any problem.) The workaround is to replace the hbase-testing-util dependency definition in spark/sql/hbase/pom.xml with the dependencies listed in that pom-only module, at the cost of developers having to edit this pom file to make it work for unprofiled combinations of HBase/Hadoop releases. ================================= This works only for hadoop2-compat hbase distributions. The spark/sql/hbase/pom.xml must be edited to make it work for hadoop1-compat hbase distributions (more on that later). The fix has been tested against HBase 0.98.5-hadoop2 and 0.98.7-hadoop2, with hadoop 2.2.0, 2.3.0, and 2.4.0. ================= SBT test commands ================= sbt/sbt -Phive,hbase,yarn,hadoop-2.2 -Dhadoop.version=2.2.0 "hbase/test-only org.apache.spark.sql.hbase.BasicQueriesSuite" sbt/sbt -Phive,hbase,yarn,hadoop-2.3 -Dhadoop.version=2.3.0 "hbase/test-only org.apache.spark.sql.hbase.BasicQueriesSuite" sbt/sbt -Phive,hbase,yarn,hadoop-2.4 -Dhadoop.version=2.4.0 "hbase/test-only org.apache.spark.sql.hbase.BasicQueriesSuite" ================= MVN test commands ================= mvn -e -Pyarn,hbase,hadoop-2.2 -Dhadoop.version=2.2.0 -pl sql/hbase clean test -DwildcardSuites=org.apache.spark.sql.hbase.BasicQueriesSuite mvn -e -Pyarn,hbase,hadoop-2.3 -Dhadoop.version=2.3.0 -pl sql/hbase clean test -DwildcardSuites=org.apache.spark.sql.hbase.BasicQueriesSuite mvn -e -Pyarn,hbase,hadoop-2.4 -Dhadoop.version=2.4.0 -pl sql/hbase clean test -DwildcardSuites=org.apache.spark.sql.hbase.BasicQueriesSuite =========================== To build against a hadoop1-compatible HBase release (for example: Hadoop 1.2.1), you will need to edit the pom to your needs. Add the appropriate hadoop profile to import the hadoop 1.2.1 libraries, and redefine the hbase dependencies to match the module structure of you chosen hbase release. The SBT libraryDependencies declaration below describes the mvn dependencies you would need to define in spark/sql/hbase/pom.xml to build Spark against HBase v0.98.5-hadoop1 over Hadoop v1.2.1. libraryDependencies ++= Seq( "org.scalatest" %% "scalatest" % "2.2.1" % "test", "com.novocode" % "junit-interface" % "0.9" % "test", "org.apache.hbase" % "hbase-common" % "0.98.5-hadoop1", "org.apache.hbase" % "hbase-common" % "0.98.5-hadoop1" % "test" classifier "tests", // no hbase-annotation module before v0.98.7 //"org.apache.hbase" % "hbase-annotations" % "0.98.5-hadoop1", //"org.apache.hbase" % "hbase-annotations" % "0.98.5-hadoop1" % "test" classifier "tests", "org.apache.hbase" % "hbase-protocol" % "0.98.5-hadoop1", "org.apache.hbase" % "hbase-client" % "0.98.5-hadoop1", "org.apache.hbase" % "hbase-server" % "0.98.5-hadoop1", "org.apache.hbase" % "hbase-server" % "0.98.5-hadoop1" % "test" classifier "tests", "org.apache.hbase" % "hbase-hadoop-compat" % "0.98.5-hadoop1", "org.apache.hbase" % "hbase-hadoop-compat" % "0.98.5-hadoop1" % "test" classifier "tests", "org.apache.hbase" % "hbase-hadoop1-compat" % "0.98.5-hadoop1", "org.apache.hbase" % "hbase-hadoop1-compat" % "0.98.5-hadoop1" % "test" classifier "tests", "org.slf4j" % "slf4j-log4j12" % "1.6.4", "org.apache.hadoop" % "hadoop-core" % "1.2.1", "org.apache.hadoop" % "hadoop-client" % "1.2.1", "org.apache.hadoop" % "hadoop-minicluster" % "1.2.1", "org.apache.hadoop" % "hadoop-test" % "1.2.1", "org.apache.hadoop" % "hadoop-tools" % "1.2.1" ) Note that there are no 'hbase-annotations' modules in HBase releases prior to v0.98.7.
…w null col-value load/select test
… same mapreduce hfile output dir during the bulkloader's text->hfile transform, resulting in broken HFile imports after the 1st test. The workaround is to make sure the physical HTable names are unique across test cases in the suite.
…classloading problem and fix typo: ExludedDependencies -> ExcludedDependencies
…' to fix 3 test cases expecting parser to return ParallelizedBulkLoadIntoTableCommand.
…le on an nonexistent HBase table created a HBase table with only one presplit region
|
Can one of the admins verify this patch? |
|
@yzhou2001, we need rebase our branch |
|
and we need check the coding style. |
|
Hi - thanks for working on this... it looks interesting. I'd like to close this issue (i.e. the PR) and discuss more on the JIRA/dev list rather than having a big pull request like this. For very large features this is the way we do it. If you look on your wiki it says "If you are proposing a larger change, attach a design document to your JIRA first (example) and email the dev mailing list to discuss it." https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark |
|
Also one thing that would help is if you could create a standalone project for this on github (see spark-avro). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.