Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add option elasticsearch-start-es-after-block to es plugin #1458

Merged
merged 2 commits into from Dec 21, 2018

Conversation

Projects
4 participants
@oxarbitrage
Copy link
Member

commented Nov 28, 2018

#1455

Will start inserting after specified block has passed, example usage:

./programs/witness_node/witness_node --plugins "elasticsearch" --elasticsearch-start-es-after-block 100000

In the first commit(5909b13) i add the check for the option right after getting a block.

In the second one(ee4cc4b) i modify that and place it much more inside the plugin logic. This is because i realized we were going to lose ID numeration, undos, total_ops, etc.

So at the end this basically don't add anything to the bulk buffer and by that will not send any data to elasticsearch.
I hope this alone can speed up the process when the node already have elasticsearch data up to a block but need testing. Any help with that is appreciated.

@pmconrad

This comment has been minimized.

Copy link
Contributor

commented Dec 14, 2018

Code looks good, but I'm unsure if this is a desirable feature. It's probably a good way to shoot yourself in the foot, by supplying the wrong block number.
How about an automatism? Write database version and current LIB into a special index entry, read that on startup and use the info to decide whether to do a full ES replay or start at the previous LIB.

@clockworkgr

This comment has been minimized.

Copy link
Member

commented Dec 21, 2018

Most ES node operators are savvy enough to do this properly imho.

If unsure what you're doing you should always sync from scratch.

The setup doesn't allow duplicates anyway so starting further back is not an issue.

I'm gonna give this patch a go to restart my ES node when the proposal patch is done.

@clockworkgr

This comment has been minimized.

Copy link
Member

commented Dec 21, 2018

Appears to be working fine for me btw

@pmconrad
Copy link
Contributor

left a comment

Ok, I'll not be standing in the way of a happy user. :-)

@oxarbitrage

This comment has been minimized.

Copy link
Member Author

commented Dec 21, 2018

In conversations with @clockworkgr there are several use cases that you want to start adding data to ES after a specific block number such as:

  • accidentally deleted some data.
  • applied extra index or data and want to roll back.
  • remove old data, save space.

Plus the main purpose of bringing a crashed ES node back confirmed by clockwork that was able to bring his es node back in a few mins after mainnet patch was released(#1479)

LIB is a good idea but the specific option should be also available so i am merging this as it is.

@oxarbitrage oxarbitrage merged commit 3c76228 into bitshares:develop Dec 21, 2018

2 checks passed

ci/dockercloud Your tests passed in Docker Cloud
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@abitmore abitmore added this to the 201901 - Feature Release milestone Jan 1, 2019

@abitmore abitmore added this to In progress in Feature Release (201902) via automation Jan 1, 2019

@abitmore abitmore moved this from In progress to Done in Feature Release (201902) Jan 1, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.