Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add start-es-inserts-after-block option to elasticsearch plugin #1455

Closed
oxarbitrage opened this issue Nov 27, 2018 · 16 comments

Comments

Projects
7 participants
@oxarbitrage
Copy link
Member

commented Nov 27, 2018

Due to the blockchain halt we recently had all ES nodes stopped working and replay is required.

This takes a huge amount of time. Most of the data is already on ES so i am thinking on an option start-es-inserts-after-block or similar that will start the heavy work after a block number passed, resulting in a faster synchronization.

As some important business rely on the ES plugin for their business, i propose that feature or something to speed up the process to be added.

Looking for your opinions.

Tasks:

  • start-es-after-block
  • [ ] dump-es-at-block
@crypto-1

This comment has been minimized.

Copy link

commented Nov 27, 2018

This feature would be very useful for CryptoBridge.

@clockworkgr

This comment has been minimized.

Copy link
Member

commented Nov 27, 2018

yaaaaaaaaaaaaaaaaaaay!

I pretty much suggested the same thing earlier today in TG Node Admins...

@cogutvalera

This comment has been minimized.

Copy link
Member

commented Nov 28, 2018

Very nice idea !

@sschiessl-bcp

This comment has been minimized.

Copy link

commented Nov 28, 2018

Could an ES dump be provided as well that matches es-inserts-after-block=32600000?

That would be asweome!

@oxarbitrage

This comment has been minimized.

Copy link
Member Author

commented Nov 28, 2018

Could an ES dump be provided as well that matches es-inserts-after-block=32600000?

This is also a good idea. I added 2 tasks to this issue to do start-es-at-block and dump-es-at-block

@sschiessl-bcp

This comment has been minimized.

Copy link

commented Nov 28, 2018

I suppose if the backend creates the dump, it could also read it, i.e. 'import-es-on-block'

Just to say it, intent was to use the native dump and import es functional

@sschiessl-bcp

This comment has been minimized.

Copy link

commented Nov 28, 2018

Maybe replace 'at' with 'before' or 'after', so the timeline is clear with respect to the block

@clockworkgr

This comment has been minimized.

Copy link
Member

commented Nov 28, 2018

i see no need to add this to the plugin...just documentation of how to use native ES dump/import

And maybe we can provide weekly ES snapshots (those of us running a node)

@sschiessl-bcp

This comment has been minimized.

Copy link

commented Nov 28, 2018

No need maybe, but it's certainly nice. Install ES in default, rest is done by plugin.

@clockworkgr

This comment has been minimized.

Copy link
Member

commented Nov 28, 2018

where would the plugin get the dump from?

@sschiessl-bcp

This comment has been minimized.

Copy link

commented Nov 28, 2018

That command would have a file, or github reference to e.g. BitShares/bitshares-core-dumps as parameter

@oxarbitrage

This comment has been minimized.

Copy link
Member Author

commented Nov 28, 2018

  • i removed the dump option from the task list, lets keep it simple. i agree with @clockworkgr; it is probably better to use ES native tools or third parties(https://github.com/taskrabbit/elasticsearch-dump) to do this because also if we export we will want to import and that can start to became more complicated without need.
  • changed the option to start-es-after-block
@Agorise

This comment has been minimized.

Copy link

commented Nov 28, 2018

As we are bringing entire cities onto point-of-sale now, any down time could cost merchants dearly, so if this will help get nodes up faster, I'm all for it.

@oxarbitrage

This comment has been minimized.

Copy link
Member Author

commented Dec 21, 2018

Closed by #1455

Now ES nodes can recover from crashes in minutes by start saving data only after a specific block pass.

@clockworkgr

This comment has been minimized.

Copy link
Member

commented Dec 21, 2018

Closed by #1455

Now ES nodes can recover from crashes in minutes by start saving data only after a specific block pass.

closed by #1458... 1455 is this issue

@oxarbitrage

This comment has been minimized.

Copy link
Member Author

commented Dec 21, 2018

oops, sorry and thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.