Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Embed HBaseTestingUtility to isolate tests #14

Merged
merged 3 commits into from

2 participants

@jasonbray

This change spins up an embedded HBase instance to run the core tests against.

I removed the 0.90 profile (per our IRC conversation) and added a CDH profile for cloudera distributions.

HBaseTestingUtility is pretty noisy, so we might consider suppressing info level messages in the future if it becomes a problem. Exceptions are also thrown when the instance is shut down, but those exceptions are innocuous.

Note that with this change you can also load clojure-hbase.core-test in a repl and run (once-start) to write code against a running hbase instance.

Jason Bray added some commits
Jason Bray Embedding an HBaseTestingUtility to isolate the core tests. Removing …
…the old hbase90 profile and adding a CDH4 profile to support the Cloudera distribution.
4c36a14
Jason Bray Appropriately assigning scope to hadoop-common and hadoop-hdfs in the…
… cdh4 profile.
8835149
@davidsantiago

I've merged this into the develop branch for now. It seems to work, but the increased output is indeed annoying, I had to search back through the output to see that the tests had indeed passed. A patch to cut back the verbosity would be appreciated.

Thanks,
David

@jasonbray

Okay David, this log4j config nearly eliminates the noise. The root logger is set to WARN, but if you would rather see it at INFO I can add more fine-grain filtering (with a long list of packages).

@davidsantiago

OK, this is great. Pushed this up to develop branch as well. I'll wait to hear from you about the timestamped Puts thing before pushing this into a release, since this is only affect unit tests (although it's very helpful).

@davidsantiago davidsantiago merged commit f38e5fd into davidsantiago:master
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 10, 2012
  1. Embedding an HBaseTestingUtility to isolate the core tests. Removing …

    Jason Bray authored
    …the old hbase90 profile and adding a CDH4 profile to support the Cloudera distribution.
Commits on Dec 11, 2012
  1. Appropriately assigning scope to hadoop-common and hadoop-hdfs in the…

    Jason Bray authored
    … cdh4 profile.
Commits on Jan 9, 2013
This page is out of date. Refresh to see the latest.
View
12 dev-resources/log4j.properties
@@ -0,0 +1,12 @@
+log4j.rootLogger = WARN, standard
+
+; Turn off HBaseTestingUtility noise
+log4j.logger.org.apache.zookeeper = FATAL
+log4j.logger.org.apache.hadoop.metrics2 = FATAL
+log4j.logger.org.apache.hadoop.hbase.zookeeper = FATAL
+log4j.logger.org.apache.hadoop.hdfs.server.datanode = FATAL
+log4j.logger.org.apache.hadoop.hdfs.server.namenode = FATAL
+
+log4j.appender.standard = org.apache.log4j.ConsoleAppender
+log4j.appender.standard.layout = org.apache.log4j.PatternLayout
+log4j.appender.standard.layout.ConversionPattern = %d{ABSOLUTE} %p %c: %m%n
View
22 project.clj
@@ -1,15 +1,27 @@
(defproject clojure-hbase "0.92.1"
:description "A convenient Clojure interface to HBase."
:dependencies [[org.clojure/clojure "1.4.0"]
- [org.apache.hadoop/hadoop-core "1.0.3"]
[org.apache.hbase/hbase "0.92.2"]
[log4j/log4j "1.2.16" :exclusions [javax.mail/mail
javax.jms/jms
com.sun.jdmk/jmxtools
com.sun.jmx/jmxri]]
- [org.clojure/tools.logging "0.2.3"]]
+ [org.clojure/tools.logging "0.2.3"]
+ [org.apache.hbase/hbase "0.92.2" :classifier "tests" :scope "test"]
+ [org.apache.hadoop/hadoop-test "1.1.0" :scope "test"]]
:profiles {:clojure1.2 {:dependencies [[org.clojure/clojure "1.2.0"]]}
:clojure1.3 {:dependencies [[org.clojure/clojure "1.3.0"]]}
- :hbase90 {:dependencies [[org.apache.hbase/hbase "0.90.5"]]}
- :hbase92 {:dependencies [[org.apache.hbase/hbase "0.92.2"]]}
- :hbase94 {:dependencies [[org.apache.hbase/hbase "0.94.1"]]}})
+ :hbase92 {:dependencies [[org.apache.hbase/hbase "0.92.2"]
+ [org.apache.hbase/hbase "0.92.2" :classifier "tests" :scope "test"]
+ [org.apache.hadoop/hadoop-test "1.1.0" :scope "test"]]}
+ :hbase94 {:dependencies [[org.apache.hbase/hbase "0.94.1"]
+ [org.apache.hbase/hbase "0.94.1" :classifier "tests" :scope "test"]
+ [org.apache.hadoop/hadoop-test "1.1.0" :scope "test"]]}
+ :cdh4 {:dependencies [[org.apache.hbase/hbase "0.92.1-cdh4.1.2"]
+ [org.apache.hadoop/hadoop-common "2.0.0-cdh4.1.2" :scope "test"]
+ [org.apache.hadoop/hadoop-hdfs "2.0.0-cdh4.1.2" :scope "test"]
+ [org.apache.hbase/hbase "0.92.1-cdh4.1.2" :classifier "tests" :scope "test"]
+ [org.apache.hadoop/hadoop-common "2.0.0-cdh4.1.2" :classifier "tests" :scope "test"]
+ [org.apache.hadoop/hadoop-hdfs "2.0.0-cdh4.1.2" :classifier "tests" :scope "test"]
+ [org.apache.hadoop/hadoop-test "2.0.0-mr1-cdh4.1.2" :scope "test"]]
+ :repositories [["cloudera" "https://repository.cloudera.com/content/groups/public/"]]}})
View
36 test/clojure_hbase/core_test.clj
@@ -4,9 +4,14 @@
clojure.stacktrace
[clojure-hbase.core]
[clojure-hbase.admin :exclude [flush]])
- (:import [org.apache.hadoop.hbase.util Bytes]
+ (:import [org.apache.hadoop.hbase HBaseTestingUtility]
+ [org.apache.hadoop.hbase.util Bytes]
[java.util UUID]))
+;; HBaseTestingUtility instance
+
+(def ^:dynamic *test-util* (atom nil))
+
;; Testing Utilities
(defn keywordize [x] (keyword (Bytes/toString x)))
@@ -306,16 +311,19 @@
:map-value #(Bytes/toString %)))
"latest-as-map works.")))))
-(deftest test-set-config
- (try
- (as-test
- (is (thrown? Exception
- (do
- (set-config (make-config {"hbase.zookeeper.quorum"
- "asdsa"})) ;; not valid
- (table test-tbl-name)))) ;; This should throw an exception.
- (is (do
- (set-config (make-config {"hbase.zookeeper.quorum" "127.0.0.1"}))
- (table test-tbl-name))))
- (finally
- (set-config (make-config (default-config))))))
+(defn once-start []
+ (.startMiniCluster
+ (reset! *test-util* (HBaseTestingUtility.)) 1)
+ (let [config (.getConfiguration @*test-util*)]
+ (set-config config)
+ (set-admin-config config)))
+
+(defn once-stop []
+ (.shutdownMiniCluster @*test-util*))
+
+(defn once-fixture [f]
+ (once-start)
+ (f)
+ (once-stop))
+
+(use-fixtures :once once-fixture)
Something went wrong with that request. Please try again.