Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split chain.db into multiple files #1003

Open
jerkyrs opened this issue Apr 11, 2019 · 0 comments

Comments

Projects
None yet
2 participants
@jerkyrs
Copy link
Contributor

commented Apr 11, 2019

What would you like to be added:
Chain.db should be split into multiple files maybe of ~100MB each so you can more easily synchronize the tip the chain to another node. If chain.db is 100GB it will be difficult to sync / copy with something like rsync as you will need to freeze the node while you do it.

In addition to splitting the chain.db ideally database filesystem is split into logical directories based on some timestamp , for example

/data/archive/2019-04-09/chain-000.db
/data/archive/2019-04-10/chain-001.db
/data/archive/2019-04-11/chain-002.db
/data/active/chain-003.db

This would avoid having large files or lots of files in a single directory, it also means that older data can be archived by administrators to less expensive storage then SSD where active data would exist. When 100MB is exceeded a new file is created and previous file archived into folder of whatever day it might be.

Why is this needed:
Ability to backup and restore faster without needing to stop node for long period of time. Ability to have HOT data in /data/active with fast SSD and archived data on Spinning drive (1/4 cost generally per GB)

@jerkyrs jerkyrs added the enhancement label Apr 11, 2019

@zjshen14 zjshen14 self-assigned this Apr 30, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.