Start MiniDFSCLuster in Travis Environment #1240

Closed
mobiusinversion opened this Issue Jul 12, 2013 · 5 comments

Comments

Projects
None yet
2 participants
@mobiusinversion

Hi there,

I am building a library to wrap hbase operations in Clojure. As part of the unit tests, the code attempts to spin up a MiniDFSCluster. Here is what I get locally:

$ lein test

lein test hbase.config-test

lein test hbase.table-test
2013-07-12 10:11:48.935 java[26113:1203] Unable to load realm info from SCDynamicStore
Starting DataNode 0 with dfs.data.dir: /Users/dwilliams/Desktop/Repos/mobiusinversion/hbase/target/test-data/3081c625-2f41-427b-b1d0-badcd6b9bcaa/dfscluster_644e4c63-8810-4316-85e3-bfede95579e2/dfs/data/data1,/Users/dwilliams/Desktop/Repos/mobiusinversion/hbase/target/test-data/3081c625-2f41-427b-b1d0-badcd6b9bcaa/dfscluster_644e4c63-8810-4316-85e3-bfede95579e2/dfs/data/data2
Cluster is active

Ran 11 tests containing 14 assertions.
0 failures, 0 errors.

However, this is what I get in travis, which appears to be a NullPointerException generated in MiniDFSCluster.java, part of the HBase core. Is this a permissions issue? Is this kind of testing supported?

Testing hbase.config-test
Testing hbase.table-test
Starting DataNode 0 with dfs.data.dir: /home/travis/build/mobiusinversion/hbase/target/test-data/9c7a156d-a71c-4d60-a876-513c782446c4/dfscluster_1e561cb6-5791-404f-928a-46dcc8ca5b96/dfs/data/data1,/home/travis/build/mobiusinversion/hbase/target/test-data/9c7a156d-a71c-4d60-a876-513c782446c4/dfscluster_1e561cb6-5791-404f-928a-46dcc8ca5b96/dfs/data/data2
ERROR in (create-table) (MiniDFSCluster.java:426)
Uncaught exception, not in assertion.
expected: nil
  actual: java.lang.NullPointerException: null
 at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes (MiniDFSCluster.java:426)
    org.apache.hadoop.hdfs.MiniDFSCluster.<init> (MiniDFSCluster.java:284)
    org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster (HBaseTestingUtility.java:444)
    org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:612)
    org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:568)
    org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster (HBaseTestingUtility.java:555)
    hbase.table_test$test_config.doInvoke (table_test.clj:10)
    clojure.lang.RestFn.invoke (RestFn.java:397)
    hbase.table_test/fn (table_test.clj:19)
@sarahhodne

This comment has been minimized.

Show comment
Hide comment
@sarahhodne

sarahhodne Jul 12, 2013

Contributor

@mobiusinversion The test command still uses lein instead of lein2. Could you try switching that and see if that helps?

Contributor

sarahhodne commented Jul 12, 2013

@mobiusinversion The test command still uses lein instead of lein2. Could you try switching that and see if that helps?

@mobiusinversion

This comment has been minimized.

Show comment
Hide comment
@mobiusinversion

mobiusinversion Jul 12, 2013

Hi Henrik,

I got the same error with

language: clojure
script: lein test
lein: lein2

and

language: clojure
script: lein2 test
lein: lein2

in my .travis.yml. Am I missing something?

There are no other calls to lein that I am aware, and for reference here is my project.clj

(defproject hbase "0.1.6"
    :description "HBase Access in Clojure"
    :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"}
    :url "https://github.com/mobiusinversion/hbase"
    :dependencies [
        [org.clojure/clojure "1.5.1"]
        [org.apache.hadoop/hadoop-core "1.2.0"]
        [org.apache.hbase/hbase "0.94.6.1"]
        [org.apache.hadoop/hadoop-test "1.2.0"]
        [org.apache.hbase/hbase "0.94.6.1" :classifier "tests"]]
    :plugins [[lein-marginalia "0.7.1"]])

Hi Henrik,

I got the same error with

language: clojure
script: lein test
lein: lein2

and

language: clojure
script: lein2 test
lein: lein2

in my .travis.yml. Am I missing something?

There are no other calls to lein that I am aware, and for reference here is my project.clj

(defproject hbase "0.1.6"
    :description "HBase Access in Clojure"
    :license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"}
    :url "https://github.com/mobiusinversion/hbase"
    :dependencies [
        [org.clojure/clojure "1.5.1"]
        [org.apache.hadoop/hadoop-core "1.2.0"]
        [org.apache.hbase/hbase "0.94.6.1"]
        [org.apache.hadoop/hadoop-test "1.2.0"]
        [org.apache.hbase/hbase "0.94.6.1" :classifier "tests"]]
    :plugins [[lein-marginalia "0.7.1"]])
@sarahhodne

This comment has been minimized.

Show comment
Hide comment
@sarahhodne

sarahhodne Jul 12, 2013

Contributor

You actually don't need the script: lein2 test line, since that's the command we run by default, but that won't change the behaviour, so that's just a side note.

As for the error you're seeing, I'm afraid I don't know enough about Clojure to be of much help. I'd suggest asking on Stack Overflow or some Clojure-specific forum.

I just tried running the tests locally, and they seem to pass for me, so this looks like it's an issue with the environment being different, maybe because the VMs are Linux instead of OS X (I don't know what you develop on locally). If it would be of any help, we can spin up a VM so you can debug things in the environment the tests run on, just shoot us an email at support@travis-ci.org.

Contributor

sarahhodne commented Jul 12, 2013

You actually don't need the script: lein2 test line, since that's the command we run by default, but that won't change the behaviour, so that's just a side note.

As for the error you're seeing, I'm afraid I don't know enough about Clojure to be of much help. I'd suggest asking on Stack Overflow or some Clojure-specific forum.

I just tried running the tests locally, and they seem to pass for me, so this looks like it's an issue with the environment being different, maybe because the VMs are Linux instead of OS X (I don't know what you develop on locally). If it would be of any help, we can spin up a VM so you can debug things in the environment the tests run on, just shoot us an email at support@travis-ci.org.

@mobiusinversion

This comment has been minimized.

Show comment
Hide comment
@mobiusinversion

mobiusinversion Jul 13, 2013

Hi Henrik, Thanks, it also failed for me on Linux, CentOS x86_64 with both the Sun and Open JDKs. I did as you suggested and opened a Stack Overflow question:

http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments

And also emailed the HBase mailing list.

Hi Henrik, Thanks, it also failed for me on Linux, CentOS x86_64 with both the Sun and Open JDKs. I did as you suggested and opened a Stack Overflow question:

http://stackoverflow.com/questions/17625938/hbase-minidfscluster-java-fails-in-certain-environments

And also emailed the HBase mailing list.

@mobiusinversion

This comment has been minimized.

Show comment
Hide comment
@mobiusinversion

mobiusinversion Jul 13, 2013

Filed this HBase jira ticket:

https://issues.apache.org/jira/browse/HBASE-8944

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment