Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: The client should have a separate option for the storage location of the database data #10736

Closed
fresheneesz opened this Issue Jul 3, 2017 · 11 comments

Comments

Projects
None yet
7 participants
@fresheneesz
Copy link

fresheneesz commented Jul 3, 2017

After wondering why my bitcoin client was taking literally weeks to download the blockchain, I found out that it was going so ultra slow because I was storing the data directory on an external hard drive. After using a symlink to put the database data on my SSD and keep the blockchain stored on my external HD (via the symlink), my blockchain validation sped up by at least 6 fold. Details here: #10647 .

Anyways, because this can be so important for a non-frustrating experience for users who don't have enough SSD space, I'm requesting that bitcoin core store the database data on the same drive as the program executables, and give the user the option of only where to put the blockchain data (what is currently the blocks directory within the data directory). Alternatively, we could give the user separate options for where to store the blockchain and where to store everything else, with some info about why the storage location is important.

If I'd wanted to buy something with bitcoins in the last month, I simply wouldn't have been able to because my client was taking forfuckingever to sync up. Its really important to make these kinds of performance improvements idiot-proof so users with less technical savvy can use bitcoin core without rage quitting.

@bytting

This comment has been minimized.

Copy link
Contributor

bytting commented Jul 10, 2017

store the database data on the same drive as the program executables

This shouldn't be necessary. I have everything on a secondary drive and have no issues with speed.
It's probably the USB that causes the problem

@sipa

This comment has been minimized.

Copy link
Member

sipa commented Jul 10, 2017

The client should store the database data on the same drive as the program

That's not generally possible. On UNIX systems, the filesystem with binaries may not be writable by the user.

I do however think it's useful to allow the chainstate to be stored in another directory from the blockchain. The former mostly needs high IOPS, the latter only needs serial read/writes.

@fresheneesz fresheneesz changed the title Feature Request: The client should store the database data on the same drive as the program Feature Request: The client should have a separate option for the storage location of the database data Jul 10, 2017

@fresheneesz

This comment has been minimized.

Copy link
Author

fresheneesz commented Jul 10, 2017

Alright, I changed the name of the request to be a separate option for storage location.

@corebob Sipa also mentioned that USB is probably the issue. Do you have any information you could point me to that substantiates this? I don't see any logical reason why USB would cause a bottleneck here nor does searching the googles turn up anything obvious.

@bytting

This comment has been minimized.

Copy link
Contributor

bytting commented Jul 11, 2017

@fresheneesz
Not at the moment, it was just an assumption I made based on your description.

@jimhashhq

This comment has been minimized.

Copy link

jimhashhq commented Jul 24, 2017

Hi, I just opened: pull request #10922 which adds a file-partition.md doc file which describes what I believe to be a simple interim solution to putting the large blockchain files on an external drive (like my inexpensive USB 3.0 in the example). Longer term, it might be easier if there was a configuration setting or flag for storing all of the leveldb stuff on an internal drive and/or the blockchain blocks folder on an external drive. As I mentioned in the pull request, this may not be the best way for these instructions to reach their intended audience so please feel free to redirect me. Thanks!

@opacey

This comment has been minimized.

Copy link

opacey commented Jan 17, 2018

+1

This is an excellent suggestion. The blockchain itself, by nature is massive and doesn't need to be on local or high speed and throughput storage medium.

The data directory contains reasonable sized files and folders which could go anywhere, and an elephant of a directory called blocks (172GB today), we really need flexibility to optionally relocate this one.

I have in the past tried multiple way to store either the whole datadir or just blocks on...

  • Network attached drives via SMB or AFP
  • Network attached drives via link
  • Locally attached USB 3.0 drive

I haven't tried iSCSI NAS which I gather works but my two NAS attempts failed with corrupted data and the USB 3.0 drive had the same issues @fresheneesz encountered (thanks also to @sipa for the analysis!). Would be delighted if this could be worked into the pipeline.

@jimhashhq

This comment has been minimized.

Copy link

jimhashhq commented Jan 19, 2018

@opacey It's not clear from your comment if you read file-partition.md which describes setting up the necessary symlinks, allowing you to run on low-end commodity hardware without the large internal disk space requirements.

@fresheneesz

This comment has been minimized.

Copy link
Author

fresheneesz commented Jan 19, 2018

@jimhashhq These things are far too complicated for the average user (even granting that the average bitcoin user is more tech savvy than the general population). What we need is a gui wizard that asks "what's your main harddrive? (SSD works best) [10GB required]" and "Which drive do you want to store the bulk of the blockchain? (150GB+ required)". These should be simple dropdowns for the user. And this would be a perfect place to ask the user "Do you want to prune the blockchain? To what number of GBs?"

Its great that its possible to split this up, but its a huge, error prone PITA doing this through command line right now. And many people might not even be aware its possible to do this even if they are already savvy enough to find these guides and correctly follow them.

@opacey

This comment has been minimized.

Copy link

opacey commented Jan 22, 2018

@jimhashhq Indeed I neglected to address your formalised sym link solution. I like it a lot as an interim solution, as you suggest, and my vote is to have in included. I then also second @fresheneesz urging for a GUI level version. Thanks again.

@Sjors

This comment has been minimized.

Copy link
Member

Sjors commented Mar 16, 2018

See #12653

@Sjors

This comment has been minimized.

Copy link
Member

Sjors commented Mar 29, 2018

This seems fixed by #12653.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.