New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Very slow syncing on Hard Drive #14895

Closed
SleepingShell opened this Issue Aug 4, 2017 · 26 comments

Comments

Projects
None yet
@SleepingShell

SleepingShell commented Aug 4, 2017

System information

Geth version: Geth/v1.6.7-stable-ab5646c5/linux-amd64/go1.8.1
OS & Version: Linux (Debian)

Expected behaviour

Sync the most recent 30,000 blocks at a rate of at least 1 block a second.

Actual behaviour

I used to sync the ethereum blockchain on an SSD. The syncing would always be fine and I would be able to catch up after being offline for a few days within 20-30 minutes. However, my SSD no longer has room. So I set up a symlink from the ~/.ethereum/chaindata folder to /blockchain/ethereum/chaindata, which is a separate HDD mounted at /blockchain. I copied the chaindata folder to that location, and geth/Mist recognize the blocks there. However, now when I sync, it syncs VERY slowly. Due to security reasons, I do not want to move the whole datadir to the HDD.

I have also tried running the geth client with --cache=1024, 2048, 4096. None of these options help. I recognize that ethereum can be intense with its read/writes, however I don't see why it should be this slow to sync blocks. I am connected to plenty of peers so I am unsure why this is happening.

I have done the same process with Bitcoin and it works completely fine. I do NOT want to use a different mode of syncing, as I would like to host the whole history of the chain, I don't believe that should have to be a workaround.

Steps to reproduce the behaviour

Setup a symlink from the ~/.ethereum/chaindata folder to a separate HDD.

Backtrace

[backtrace]

image

@AlisaHevic

This comment has been minimized.

AlisaHevic commented Aug 4, 2017

Windows 7 pro, 100% disk usage, but only 10 Mb/s read-write (my hdd can 180 Mb/s). System is very, very slow. It is impossible to use browser or play games when geth is syncing ... geth1 6 6 hdd
geth cache=4096. Geth use 4.3 Gb of 8 Gb system RAM. Swap disabled.

@nathanph

This comment has been minimized.

nathanph commented Aug 6, 2017

Having this exact same issue. And when I eventually get close to finishing syncing it won't sync the last 200-600 blocks.

@w0nnapl4y

This comment has been minimized.

w0nnapl4y commented Aug 11, 2017

Commenting here, because I came to search a solution for this exact phenomenon.

@jlippuner

This comment has been minimized.

jlippuner commented Aug 23, 2017

I'm experiencing the same problem. I used to have the blockchain on my SSD but it became too big. So I moved it to a HDD (moved the geth dir inside ~/.ethereum to HDD and made a symlink) and now the block sync is extremely slow. I'm using geth version 1.6.5-stable-cf87713d.

@bblboy54

This comment has been minimized.

bblboy54 commented Aug 23, 2017

Very similar scenario for me except that I did the symlinking to an HDD long before having serious issues. What really did me in was I went on vacation the first full week of August and my laptop sat in it's bag... when I returned I was expecting a lot of syncing being needed but it NEVER catches up... This weekend was a long weekend for me so I figured it would be the solution because I was able to start the sync on Friday afternoon and let it run clear until this morning.... last night I was still extremely far behind and blocks are being generated faster than my node is syncing them. What is really perplexing to me is that geth is only using around 10% CPU on average but my HDD is pegged..... I did some experimenting with cache sizes last night and I've found that if I use a higher cache size like 2048 the process never wants to quit even if I tell it to shutdown.....

I'm using the ethereum wallet package for 64bit linux (installed from deb) with the latest version and I'm using the geth that is packaged with it which is 1.6.6-stable-10a45cb5 and the system is running Ubuntu 16.04.3 LTS with an i7-4760HQ CPU and 8GB of RAM.

Again, I moved everything over to my HDD and added the symlink probably a month before I went on vacation and I didn't have any issues if I left ethereum closed for a few days and opened it up -- it always seemed to catch up within an hour or so but after I returned from vacation on August 13th it's been a nightmare.... This weekend it was running for more than 3 days straight without interruption and it came nowhere close to catching up.

This HDD access thing is of the biggest concern.... why is it pegging my HDD while not really using much CPU and even less bandwidth.... And a valid solution isn't going to be move it to an SSD because it was working on the HDD acceptably before and after seeing how much drive activity is going on I wouldn't want that much wear on a SSD.

@AlisaHevic

This comment has been minimized.

AlisaHevic commented Aug 26, 2017

So, SSD is response for this problem?

@lyricalpolymath

This comment has been minimized.

lyricalpolymath commented Aug 27, 2017

what is an average time for each group of blocks that you see in the logs?
Is it taking minutes or just seconds?

@DimaLg

This comment has been minimized.

DimaLg commented Aug 28, 2017

On my system (windows 7, intel core i5, 8 Gb ram, WD green 1 Tb) ~ 20 seconds per chain segment. In chain segment usually 1-2 blocks.

@bblboy54

This comment has been minimized.

bblboy54 commented Aug 28, 2017

It varies quite a bit.... I'd say the average is about 30-40 seconds per block.... occasionally I see some come in around 10 seconds but I also see some come in at 1-2mins.... I rarely see more than 1 block come in per segment but it does happen every so often where I'll see 2 or 3 come in a single segment but then it goes right back to a single block.

@maxgillett

This comment has been minimized.

maxgillett commented Sep 12, 2017

I have the same issue. Ubuntu 16.04 with WD 2TB drive. Copied my synced chaindata to an external drive, and now it's impossible to catch up to the latest block.

@bassmaster187

This comment has been minimized.

bassmaster187 commented Sep 13, 2017

same problem here. HDD has 100% of load on windows 10

@EchebKeso

This comment has been minimized.

EchebKeso commented Nov 10, 2017

Another +1, I moved from an SSD to a decent speed HDD (a write rate of 120MB/s as opposed to 400MB/s on the SSD) - and now it can't ever catch up after missing just one day! It takes longer than 10 seconds per block right now on average (per "Imported new chain segment x 1").

@fridaynext

This comment has been minimized.

fridaynext commented Nov 19, 2017

Same here - flawless performance on my 2014 MacBook Pro 512GB SSD, but only downloads about 1% per day on my WD 5TB MyBook external USB3 drive. Blockchain is too big for my SSD, but I can't reliably use the wallet when the blockchain won't ever sync.

@thedrbits

This comment has been minimized.

thedrbits commented Dec 12, 2017

Same problem here -- 1TB USB 3.0 External Toshiba drive -- Prior using internal SSD

@thedrbits

This comment has been minimized.

thedrbits commented Dec 12, 2017

Things that affected the performance:
-External HD partition format
-Geth Cache Size
It appears this performance issue is caused by Geth's interaction with the Leveldb chaindata files.

@ddeputy1

This comment has been minimized.

ddeputy1 commented Dec 13, 2017

Is having the DAG and the chain data on different drives, as I do, causing this issue? Anybody have other ideas or resolution?

@DavidCWebs

This comment has been minimized.

DavidCWebs commented Dec 13, 2017

Another + 1 here. Ubuntu 16.04 running geth 1.7.3-stable. I switched from an SSD to a (new) HDD when chaindata started getting too large. Symlinked to chaindata on the HDD from the default ~/.ethereum/geth directory on the SSD. Looks like it isn't going to catch up to the latest block.

@gasan050

This comment has been minimized.

gasan050 commented Dec 15, 2017

  • same here, Ubuntu, symlinks to HDD, and the sync is extremely slow!
@thedrbits

This comment has been minimized.

thedrbits commented Dec 18, 2017

I just setup an external USB SSD to compare, and this solved the issue. Syncing is roughly 30 times faster after switching to an SSD hard drive.

@jgostylo

This comment has been minimized.

jgostylo commented Jan 3, 2018

Yes, any type of answer at all would be nice. Even if it is to say that we are kind of screwed at the moment but here are some solutions... One might be the ability to span the blockchain across drives. This seems doable since the chaindata files are already separated. Then you could operate the latest blocks on a small portion of your SSD and relegate the majority of the chain to a large HDD.

Blockchain size is up to 420GB. That is not very palatable getting a single SSD that will last to the end of the year.

@jtktam

This comment has been minimized.

jtktam commented Jan 28, 2018

I went from a 256gb drive to a 512gb in a last month and it's filling up quickly.. i wonder if the older files can be moved to "cold" hdd and/or symbolic link back to the main chaindata?

@dslip

This comment has been minimized.

dslip commented Feb 7, 2018

+1 here

@holiman

This comment has been minimized.

Contributor

holiman commented Feb 8, 2018

As of earlier this week, we merged #15857. This should considerably improve the syncing experience. Some notes:

  • It fixes some bugs in the p2p/networking, which previously caused peer churn and caused sync to stop.
  • It does in-memory pruning of state, and considerably lessens the state-growth after fast-sync has completed.

We're hoping that this will resolve most issues with sync, but users with HDD instead of SSD may still experience problems. I'll close this ticket, since it's becoming rather old -- originally reported on Geth/v1.6.7-stable-ab5646c5/linux-amd64/go1.8.1. For sync problems with latest master(not yet released), please open a new ticket.

@holiman holiman closed this Feb 8, 2018

@lostmsu

This comment has been minimized.

lostmsu commented Jul 3, 2018

As of today, this issue does not seem to be solved yet. Block download rate is in 20-40 second interval.

@wupeaking

This comment has been minimized.

wupeaking commented Aug 16, 2018

+1 here

@stelloxx

This comment has been minimized.

stelloxx commented Aug 21, 2018

What is the "knownStates" value for those experiencing sync issues on this thread? I am on 197,323,466 after about 7 days of fastsync with geth (rig: quad 3.4 Xeon, 16GB RAM, HDD on Ubuntu latest)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment