Skip to content

Start MiniDFSCLuster in Travis Environment #1240

Closed
mobiusinversion opened this Issue Jul 12, 2013 · 5 comments

2 participants

@mobiusinversion

Hi there,

I am building a library to wrap hbase operations in Clojure. As part of the unit tests, the code attempts to spin up a MiniDFSCluster. Here is what I get locally:

$ lein test

lein test hbase.config-test

lein test hbase.table-test
2013-07-12 10:11:48.935 java[26113:1203] Unable to load realm info from SCDynamicStore
Starting DataNode 0 with dfs.data.dir: /Users/dwilliams/Desktop/Repos/mobiusinversion/hbase/target/test-data/3081c625-2f41-427b-b1d0-badcd6b9bcaa/dfscluster_644e4c63-8810-4316-85e3-bfede95579e2/dfs/data/data1,/Users/dwilliams/Desktop/Repos/mobiusinversion/hbase/target/test-data/3081c625-2f41-427b-b1d0-badcd6b9bcaa/dfscluster_644e4c63-8810-4316-85e3-bfede95579e2/dfs/data/data2
Cluster is active

Ran 11 tests containing 14 assertions.
0 failures, 0 errors.

However, this is what I get in travis, which appears to be a NullPointerException generated in MiniDFSCluster.java, part of the HBase core. Is this a permissions issue? Is this kind of testing supported?

Testing hbase.config-test
Testing hbase.table-test
Starting DataNode 0 with dfs.data.dir: /home/travis/build/mobiusinversion/hbase/target/test-data/9c7a156d-a71c-4d60-a876-513c782446c4/dfscluster_1e561cb6-5791-404f-928a-46dcc8ca5b96/dfs/data/data1,/home/travis/build/mobiusinversion/hbase/target/test-data/9c7a156d-a71c-4d60-a876-513c782446c4/dfscluster_1e561cb6-5791-404f-928a-46dcc8ca5b96/dfs/data/data2
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
  actual: java.lang.NullPointerException: null
 at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes (MiniDFSCluster.java:426)
    org.apache.hadoop.hdfs.MiniDFSCluster.<init> (MiniDFSCluster.java:284)
    org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster (HBaseTestingUtility.java:444)
    org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:612)
    org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:568)
    org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:555)
    hbase.table_test$test_config.doInvoke (table_test.clj:10)
    clojure.lang.RestFn.invoke (RestFn.java:397)
    hbase.table_test/fn (table_test.clj:19)
@henrikhodne
Travis CI member

@mobiusinversion The test command still uses lein instead of lein2. Could you try switching that and see if that helps?

@mobiusinversion

Hi Henrik,

I got the same error with

language: clojure
script: lein test
lein: lein2

and

language: clojure
script: lein2 test
lein: lein2

in my .travis.yml. Am I missing something?

There are no other calls to lein that I am aware, and for reference here is my project.clj

(defproject hbase "0.1.6"
    :description "HBase Access in Clojure"
    :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"}
    :url "https://github.com/mobiusinversion/hbase"
    :dependencies [
        [org.clojure/clojure "1.5.1"]
        [org.apache.hadoop/hadoop-core "1.2.0"]
        [org.apache.hbase/hbase "0.94.6.1"]
        [org.apache.hadoop/hadoop-test "1.2.0"]
        [org.apache.hbase/hbase "0.94.6.1" :classifier "tests"]]
    :plugins [[lein-marginalia "0.7.1"]])
@henrikhodne
Travis CI member

You actually don't need the script: lein2 test line, since that's the command we run by default, but that won't change the behaviour, so that's just a side note.

As for the error you're seeing, I'm afraid I don't know enough about Clojure to be of much help. I'd suggest asking on Stack Overflow or some Clojure-specific forum.

I just tried running the tests locally, and they seem to pass for me, so this looks like it's an issue with the environment being different, maybe because the VMs are Linux instead of OS X (I don't know what you develop on locally). If it would be of any help, we can spin up a VM so you can debug things in the environment the tests run on, just shoot us an email at support@travis-ci.org.

@mobiusinversion

Hi Henrik, Thanks, it also failed for me on Linux, CentOS x86_64 with both the Sun and Open JDKs. I did as you suggested and opened a Stack Overflow question:

http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments

And also emailed the HBase mailing list.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.