Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docker] rabbitmq test fails with "rabbitmq-server: unrecognized service" #160

Open
mbsimonovic opened this issue Dec 23, 2016 · 8 comments
Open
Labels

Comments

@mbsimonovic
Copy link

@mbsimonovic mbsimonovic commented Dec 23, 2016

from jepsen-control docker container:

$ cd rabbitmq && lein test :only jepsen.rabbitmq-test


lein test jepsen.rabbitmq-test
INFO  jepsen.os.debian - :n5 setting up debian
INFO  jepsen.os.debian - :n2 setting up debian
INFO  jepsen.os.debian - :n1 setting up debian
INFO  jepsen.os.debian - :n4 setting up debian
INFO  jepsen.os.debian - :n3 setting up debian
INFO  jepsen.os.debian - Installing #{man}
INFO  jepsen.os.debian - Installing #{man}
INFO  jepsen.os.debian - Installing #{man}
INFO  jepsen.os.debian - Installing #{man}
INFO  jepsen.os.debian - Installing #{man}
INFO  jepsen.rabbitmq - :n2 Nuking rabbit
INFO  jepsen.rabbitmq - :n4 Nuking rabbit
INFO  jepsen.rabbitmq - :n5 Nuking rabbit
INFO  jepsen.rabbitmq - :n3 Nuking rabbit
INFO  jepsen.rabbitmq - :n1 Nuking rabbit
INFO  jepsen.rabbitmq - Fetching deb package
INFO  jepsen.rabbitmq - Fetching deb package
INFO  jepsen.rabbitmq - Fetching deb package
INFO  jepsen.rabbitmq - Installing rabbitmq
INFO  jepsen.rabbitmq - Installing rabbitmq
INFO  jepsen.rabbitmq - Installing rabbitmq
INFO  jepsen.rabbitmq - :n2 Nuking rabbit
INFO  jepsen.rabbitmq - :n1 Nuking rabbit
INFO  jepsen.rabbitmq - :n3 Nuking rabbit
INFO  jepsen.rabbitmq - :n4 Nuking rabbit
INFO  jepsen.rabbitmq - :n5 Nuking rabbit
INFO  jepsen.rabbitmq - :n3 Rabbit dead
INFO  jepsen.rabbitmq - :n1 Rabbit dead

lein test :only jepsen.rabbitmq-test/rabbit-test

ERROR in (rabbit-test) (FutureTask.java:122)
Uncaught exception, not in assertion.
expected: nil
  actual: java.util.concurrent.ExecutionException: java.lang.RuntimeException: rabbitmq-server: unrecognized service


...

Caused by: java.lang.RuntimeException: rabbitmq-server: unrecognized service
 at jepsen.control$throw_on_nonzero_exit.invoke (control.clj:105)
    jepsen.control$exec_STAR_.doInvoke (control.clj:121)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invoke (core.clj:624)
    jepsen.control$exec.doInvoke (control.clj:135)
    clojure.lang.RestFn.invoke (RestFn.java:436)
    jepsen.rabbitmq$reify__8631.teardown_BANG_ (rabbitmq.clj:97)
    jepsen.db$eval5744$fn__5745$G__5734__5749.invoke (db.clj:4)
    jepsen.db$eval5744$fn__5745$G__5733__5754.invoke (db.clj:4)
    clojure.lang.AFn.applyToHelper (AFn.java:160)
    clojure.lang.AFn.applyTo (AFn.java:144)
    clojure.core$apply.invoke (core.clj:626)
    clojure.core$partial$fn__4228.doInvoke (core.clj:2468)
    clojure.lang.RestFn.invoke (RestFn.java:421)
    jepsen.core$on_nodes$fn__7547.invoke (core.clj:85)
    clojure.core$pmap$fn__6328$fn__6329.invoke (core.clj:6466)
    clojure.core$binding_conveyor_fn$fn__4145.invoke (core.clj:1910)
    clojure.lang.AFn.call (AFn.java:18)
    java.util.concurrent.FutureTask.run (FutureTask.java:266)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
    java.lang.Thread.run (Thread.java:745)

hm, rabbitmq is installed only on n1,n2 and n3, but isn't it supposed to be install on all 5 nodes?

@mbsimonovic
Copy link
Author

@mbsimonovic mbsimonovic commented Dec 24, 2016

adding :nodes [:n1 :n2 :n3 :n4 :n5] to rabbitmq-test makes it try to start rabbitmq on 4 nodes?! btw how can I enable DEBUG logging level?

@vutruongqa
Copy link

@vutruongqa vutruongqa commented Nov 30, 2018

Same issue above, anyone help please?

@tvhphuong
Copy link

@tvhphuong tvhphuong commented Nov 30, 2018

I have the same issue, but when run again. It stuck at another step
root@control:/jepsen/rabbitmq# lein test jepsen.rabbitmq-test

lein test jepsen.rabbitmq-test

INFO jepsen.os.debian - :n5 setting up debian INFO jepsen.os.debian - :n4 setting up debian INFO jepsen.os.debian - :n3 setting up debian INFO jepsen.os.debian - :n2 setting up debian INFO jepsen.os.debian - :n1 setting up debian INFO jepsen.os.debian - Installing #{man} INFO jepsen.os.debian - Installing #{man} INFO jepsen.os.debian - Installing #{man} INFO jepsen.os.debian - Installing #{man} INFO jepsen.os.debian - Installing #{man} INFO jepsen.rabbitmq - :n4 Nuking rabbit INFO jepsen.rabbitmq - :n1 Nuking rabbit INFO jepsen.rabbitmq - :n3 Nuking rabbit INFO jepsen.rabbitmq - :n2 Nuking rabbit INFO jepsen.rabbitmq - :n5 Nuking rabbit INFO jepsen.rabbitmq - Fetching deb package INFO jepsen.rabbitmq - Installing rabbitmq INFO jepsen.rabbitmq - :n4 Rabbit dead INFO jepsen.rabbitmq - :n2 Rabbit dead INFO jepsen.rabbitmq - :n1 Rabbit dead INFO jepsen.rabbitmq - Setting cookie INFO jepsen.rabbitmq - Setting cookie INFO jepsen.rabbitmq - uploading config INFO jepsen.rabbitmq - uploading config INFO jepsen.rabbitmq - Starting rabbitmq INFO jepsen.rabbitmq - Starting rabbitmq

Do you know why it has different issues while starting the test ?

@aphyr
Copy link
Collaborator

@aphyr aphyr commented Dec 1, 2018

I haven't worked on the rabbit tests in 3 years, but my guess is that whatever version of Debian (and possibly Jepsen) you're using no longer runs the rabbitmq service in the same way that it did circa 2015.

@vutruongqa
Copy link

@vutruongqa vutruongqa commented Dec 3, 2018

@aphyr,
Can you suggest any tool to test rabbitmq that is easier to read than Jepsen(clojure)? I have spent a week to read on that but still can't understand how it works.
If you have a document that guide step by step, please help to share.
Thanks much for your answer

@vjuranek
Copy link
Contributor

@vjuranek vjuranek commented Dec 3, 2018

If you have a document that guide step by step, please help to share.

have you checked tutorial?

@aphyr
Copy link
Collaborator

@aphyr aphyr commented Dec 3, 2018

Sorry @vutruongqa, I don't really know any other tool that does what Jepsen does. At least, not in this degree of depth. There are a few Jepsen clones out there but they're usually oriented towards testing one particular system, or just inducing faults without correctness testing.

@aphyr
Copy link
Collaborator

@aphyr aphyr commented Dec 3, 2018

Oh, and as for support, well, there's a lot of documentation in Jepsen. The README links to the API docs, architecture overview, there are extensive code comments and docstrings, a comprehensive tutorial (as @vjuranek noted), and there are several free resources for learning Clojure; Clojure for the Brave and True is quite nice.

As far as bitrot goes, I'd love to spend some time bringing the RabbitMQ tests back up to speed with modern Jepsen, but my backlog right now is measured in years, not days. :-(

If you'd like commercial support, there are also engineers who work on Jepsen on a contract basis, and they might have some availability to help in your testing. I understand if you're budget constrained, though.

@mkcp mkcp added the question label Feb 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.