New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various cleanups for Elasticsearch test #51

Merged
merged 5 commits into from Apr 17, 2015

Conversation

Projects
None yet
2 participants
@dakrone
Contributor

dakrone commented Apr 7, 2015

This change makes four changes:

  • Adding a logging configuration

In order to actually determine the root cause of issues, more verbose logging is needed. This defaults to more verbose logging for Elasticsearch and adds the ability to change it from Jepsen in the future (instead of manually by hand).

  • Moving nuke! to before tests

Without this, Jepsen deletes all traces of itself after running, which makes debugging much more difficult (no logs and no data left).

  • Use more reasonable settings for scroll

An optional change, but I figured I would make this change anyway.

  • Wait for index to become green after creation

This is a key part of testing Elasticsearch, and clients should always do this when creating indices.


Note that I was able to reproduce the failures in elastic/elasticsearch#10426 (about half of the time) without these changes, however after the change which waits for green after index creation, I am no longer able to reproduce data loss with the create-pause test (still evaluating the other tests).

dakrone added some commits Apr 7, 2015

Add logging configuration
This increases the default logging to DEBUG, and sets TRACE logging for
gateway and discovery packages.
Move `nuke!` to before the test starts
This allows someone to collect the ES logs and data *after* a test run.
Otherwise the logs and data is removed and ES is stopped, making further
debugging impossible.
Wait for index to become green after creation
After an index is created, clients should always wait for the index to
be fully created (the request returns immediately) before starting the
test.
Use more reasonable settings for scroll
No need for the `query_then_fetch` setting, use ten seconds instead of
one minute, and a more reasonable size of 20 rather than 2.
@dakrone

This comment has been minimized.

Show comment
Hide comment
@dakrone

dakrone Apr 8, 2015

Contributor

@aphyr also, how would you feel about me replacing Elastisch with vanilla clj-http? Elastisch uses clj-http internally anyway and it would reduce the number of moving parts in this test. I'm happy to submit another PR if you are interested.

Contributor

dakrone commented Apr 8, 2015

@aphyr also, how would you feel about me replacing Elastisch with vanilla clj-http? Elastisch uses clj-http internally anyway and it would reduce the number of moving parts in this test. I'm happy to submit another PR if you are interested.

:scroll "1m"
:size 2)
:scroll "10s"
:size 20)

This comment has been minimized.

@aphyr

aphyr Apr 15, 2015

Collaborator

Yes, that's much more sensible, thank you. :)

@aphyr

aphyr Apr 15, 2015

Collaborator

Yes, that's much more sensible, thank you. :)

Catch only the `IndexAlreadyExistsException`, use dynamic node count
Instead of catching any throwable during index creation, catch a
specific exception and ensure it was only because the index already
existed.

Additionally, this changes the hardcoded node count of 5 to
`(count (:nodes test))` so a dynamic number of nodes can be used.
@dakrone

This comment has been minimized.

Show comment
Hide comment
@dakrone

dakrone Apr 17, 2015

Contributor

Pushed another commit addressing your feedback, thanks for taking a look!

Contributor

dakrone commented Apr 17, 2015

Pushed another commit addressing your feedback, thanks for taking a look!

@aphyr

This comment has been minimized.

Show comment
Hide comment
@aphyr

aphyr Apr 17, 2015

Collaborator

Excellent, thanks @dakrone :)

Collaborator

aphyr commented Apr 17, 2015

Excellent, thanks @dakrone :)

aphyr added a commit that referenced this pull request Apr 17, 2015

Merge pull request #51 from dakrone/es-cleanups
Various cleanups for Elasticsearch test

@aphyr aphyr merged commit 8bbd973 into jepsen-io:master Apr 17, 2015

@aphyr

This comment has been minimized.

Show comment
Hide comment
@aphyr

aphyr Apr 28, 2015

Collaborator

Do these tests run for you? Elasticsearch doesn't even start on my nodes any more; times out waiting for cluster recovery.

Collaborator

aphyr commented Apr 28, 2015

Do these tests run for you? Elasticsearch doesn't even start on my nodes any more; times out waiting for cluster recovery.

@dakrone

This comment has been minimized.

Show comment
Hide comment
@dakrone

dakrone Apr 28, 2015

Contributor

@aphyr I just double-checked this and the tests are still running for me

Contributor

dakrone commented Apr 28, 2015

@aphyr I just double-checked this and the tests are still running for me

aphyr added a commit that referenced this pull request Aug 23, 2016

Merge pull request #51 from dakrone/es-cleanups
Various cleanups for Elasticsearch test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment