ERROR in (create-test) (FutureTask.java:122)

[pilvitaneli]

FYI. I've been running elasticsearch tests in a loop, recently saw this failure (the test run before and after this finished just fine). Just dumping full log of jepsen printout here as I don't understand its internals:

lein test jepsen.system.elasticsearch-test
INFO  jepsen.os.debian - :n5 setting up debian
INFO  jepsen.os.debian - :n4 setting up debian
INFO  jepsen.os.debian - :n3 setting up debian
INFO  jepsen.os.debian - :n1 setting up debian
INFO  jepsen.os.debian - :n2 setting up debian
INFO  jepsen.os.debian - :n3 debian set up
INFO  jepsen.os.debian - :n1 debian set up
INFO  jepsen.os.debian - :n5 debian set up
INFO  jepsen.os.debian - :n4 debian set up
INFO  jepsen.os.debian - :n2 debian set up
INFO  jepsen.system.elasticsearch - :n3 elasticsearch nuked
INFO  jepsen.system.elasticsearch - :n2 elasticsearch nuked
INFO  jepsen.system.elasticsearch - :n5 elasticsearch nuked
INFO  jepsen.system.elasticsearch - :n4 elasticsearch nuked
INFO  jepsen.system.elasticsearch - :n1 elasticsearch nuked
INFO  jepsen.system.elasticsearch - :n4 configuring elasticsearch
INFO  jepsen.system.elasticsearch - :n3 configuring elasticsearch
INFO  jepsen.system.elasticsearch - :n2 configuring elasticsearch
INFO  jepsen.system.elasticsearch - :n1 configuring elasticsearch
INFO  jepsen.system.elasticsearch - :n5 configuring elasticsearch
INFO  jepsen.system.elasticsearch - :n4 starting elasticsearch
INFO  jepsen.system.elasticsearch - :n2 starting elasticsearch
INFO  jepsen.system.elasticsearch - :n1 starting elasticsearch
INFO  jepsen.system.elasticsearch - :n5 starting elasticsearch
INFO  jepsen.system.elasticsearch - :n4 elasticsearch ready
INFO  jepsen.system.elasticsearch - :n2 elasticsearch ready
INFO  jepsen.system.elasticsearch - :n5 elasticsearch ready
INFO  jepsen.system.elasticsearch - :n1 elasticsearch ready
INFO  jepsen.system.elasticsearch - :n3 elasticsearch nuked
INFO  jepsen.system.elasticsearch - :n4 elasticsearch nuked
INFO  jepsen.system.elasticsearch - :n5 elasticsearch nuked
INFO  jepsen.system.elasticsearch - :n2 elasticsearch nuked
INFO  jepsen.system.elasticsearch - :n1 elasticsearch nuked

lein test :only jepsen.system.elasticsearch-test/create-test

ERROR in (create-test) (FutureTask.java:122)
Uncaught exception, not in assertion.
expected: nil
  actual: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: com.jcraft.jsch.JSchException: verify: false
 at java.util.concurrent.FutureTask.report (FutureTask.java:122)
    java.util.concurrent.FutureTask.get (FutureTask.java:188)
    clojure.core$deref_future.invoke (core.clj:2180)
    clojure.core$future_call$reify__6320.deref (core.clj:6417)
    clojure.core$deref.invoke (core.clj:2200)
    clojure.core$map$fn__4245.invoke (core.clj:2559)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.Cons.next (Cons.java:39)
    clojure.lang.RT.next (RT.java:599)
    clojure.core$next.invoke (core.clj:64)
    clojure.core$dorun.invoke (core.clj:2856)
    jepsen.core$on_nodes.invoke (core.clj:72)
    jepsen.core$run_BANG_.invoke (core.clj:315)
    jepsen.system.elasticsearch_test/fn (elasticsearch_test.clj:87)
    clojure.test$test_var$fn__7187.invoke (test.clj:704)
    clojure.test$test_var.invoke (test.clj:704)
    clojure.test$test_vars$fn__7209$fn__7214.invoke (test.clj:721)
    clojure.test$default_fixture.invoke (test.clj:674)
    clojure.test$test_vars$fn__7209.invoke (test.clj:721)
    clojure.test$default_fixture.invoke (test.clj:674)
    clojure.test$test_vars.invoke (test.clj:718)
    clojure.test$test_all_vars.invoke (test.clj:727)
    clojure.test$test_ns.invoke (test.clj:746)
    clojure.core$map$fn__4245.invoke (core.clj:2559)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.Cons.next (Cons.java:39)
    clojure.lang.RT.boundedLength (RT.java:1655)
    clojure.lang.RestFn.applyTo (RestFn.java:130)
    clojure.core$apply.invoke (core.clj:626)
    clojure.test$run_tests.doInvoke (test.clj:761)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invoke (core.clj:624)
    user$eval85$fn__140$fn__171.invoke (form-init968023791120871162.clj:1)
    user$eval85$fn__140$fn__141.invoke (form-init968023791120871162.clj:1)
    user$eval85$fn__140.invoke (form-init968023791120871162.clj:1)
    user$eval85.invoke (form-init968023791120871162.clj:1)
    clojure.lang.Compiler.eval (Compiler.java:6676)
    clojure.lang.Compiler.eval (Compiler.java:6666)
    clojure.lang.Compiler.load (Compiler.java:7103)
    clojure.lang.Compiler.loadFile (Compiler.java:7059)
    clojure.main$load_script.invoke (main.clj:274)
    clojure.main$init_opt.invoke (main.clj:279)
    clojure.main$initialize.invoke (main.clj:307)
    clojure.main$null_opt.invoke (main.clj:342)
    clojure.main$main.doInvoke (main.clj:420)
    clojure.lang.RestFn.invoke (RestFn.java:421)
    clojure.lang.Var.invoke (Var.java:383)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.main.main (main.java:37)
Caused by: java.util.concurrent.ExecutionException: com.jcraft.jsch.JSchException: verify: false
 at java.util.concurrent.FutureTask.report (FutureTask.java:122)
    java.util.concurrent.FutureTask.get (FutureTask.java:188)
    clojure.core$deref_future.invoke (core.clj:2180)
    clojure.core$future_call$reify__6320.deref (core.clj:6417)
    clojure.core$deref.invoke (core.clj:2200)
    clojure.core$map$fn__4245.invoke (core.clj:2557)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.RT.seq (RT.java:484)
    clojure.core$seq.invoke (core.clj:133)
    clojure.core$map$fn__4249.invoke (core.clj:2562)
    clojure.lang.LazySeq.sval (LazySeq.java:40)
    clojure.lang.LazySeq.seq (LazySeq.java:49)
    clojure.lang.RT.seq (RT.java:484)
    clojure.core$seq.invoke (core.clj:133)
    clojure.core.protocols$seq_reduce.invoke (protocols.clj:30)
    clojure.core.protocols/fn (protocols.clj:54)
    clojure.core.protocols$fn__6031$G__6026__6044.invoke (protocols.clj:13)
    clojure.core$reduce.invoke (core.clj:6286)
    clojure.core$into.invoke (core.clj:6338)
    jepsen.system.elasticsearch$reify__5932$fn__5933.invoke (elasticsearch.clj:110)
    jepsen.system.elasticsearch$reify__5932.setup_BANG_ (elasticsearch.clj:74)
    jepsen.db$cycle_BANG_.invoke (db.clj:22)
    clojure.lang.AFn.applyToHelper (AFn.java:160)
    clojure.lang.AFn.applyTo (AFn.java:144)
    clojure.core$apply.invoke (core.clj:626)
    clojure.core$partial$fn__4228.doInvoke (core.clj:2468)
    clojure.lang.RestFn.invoke (RestFn.java:421)
    jepsen.core$on_nodes$fn__7382.invoke (core.clj:71)
    clojure.core$pmap$fn__6328$fn__6329.invoke (core.clj:6463)
    clojure.core$binding_conveyor_fn$fn__4145.invoke (core.clj:1910)
    clojure.lang.AFn.call (AFn.java:18)
    java.util.concurrent.FutureTask.run (FutureTask.java:262)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
    java.lang.Thread.run (Thread.java:744)
Caused by: com.jcraft.jsch.JSchException: verify: false
 at com.jcraft.jsch.Session.connect (Session.java:330)
    com.jcraft.jsch.Session.connect (Session.java:183)
    clj_ssh.ssh$connect.invoke (ssh.clj:327)
    jepsen.control$session.invoke (control.clj:177)
    jepsen.system.elasticsearch$reify__5932$fn__5933$fn__5943$fn__5944.invoke (elasticsearch.clj:110)
    clojure.core$binding_conveyor_fn$fn__4145.invoke (core.clj:1910)
    clojure.lang.AFn.call (AFn.java:18)
    java.util.concurrent.FutureTask.run (FutureTask.java:262)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
    java.lang.Thread.run (Thread.java:744)

Ran 1 tests containing 1 assertions.
0 failures, 1 errors.
Error encountered performing task 'test' with profile(s): 'base,system,user,provided,dev,elasticsearch'
Tests failed.

[aphyr]

Looks like there was some sort of SSH error.

[abailly]

This is typically an error occurring because the server's key cannot be verified by the client. How are the hosts n1...n5 setup? Can you try setting {:strict-host-key-checking :no} in the configuration of the ssh session that is run by jepsen?

[aphyr]

If it's intermittent, that suggests a network problem or race condition. I've seen intermittent weirdness with jsch.

[pilvitaneli]

{:strict-host-key-checking :no} was already in place. I've seen this happen only this once, in more than a hundred test runs. Rare network problem is most plausible explanation, closing.