Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Rolling forward" at startup can take a long time, and is not interruptible #11600

Open
rebroad opened this issue Nov 3, 2017 · 23 comments
Open

"Rolling forward" at startup can take a long time, and is not interruptible #11600

rebroad opened this issue Nov 3, 2017 · 23 comments

Comments

@rebroad
Copy link
Contributor

@rebroad rebroad commented Nov 3, 2017

bitcond is spending hours "Rolling forward" upon startup, seemingly reprocessing blocks that were previously processed. There have been reports of this happening since July on stackexchange and reddit, so I thought it time an issue was raised, given it's still happening in V0.15.1.

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor

@TheBlueMatt TheBlueMatt commented Nov 3, 2017

This happens when bitcoind isnt allowed to shut down cleanly after connecting many blocks with a large dbcache. Ideally we'd have a mechanism for background flushing the cache to disk so that we dont end up in this state, but if you do a full, normal, clean shutdown, this shouldn't happen.

@rebroad

This comment has been minimized.

Copy link
Contributor Author

@rebroad rebroad commented Nov 3, 2017

it looks like bitcoind does not respond to a kill request during these hours - so it's not possible to shut down cleanly...

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor

@TheBlueMatt TheBlueMatt commented Nov 3, 2017

Ah, well I meant shutdown cleanly prior to the "roll forward", but, indeed, it looks like there is an issue here in that the "roll forward" doesn't allow shutdown in the middle of it.

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Nov 8, 2017

This is a very long-running issue, I'm surprised there isn't another open for it but couldn't find it.
(though #8117 #10005 are similar, the GUI problem has been solved, but the underlying potentially long wait at startup hasn't)

@laanwj laanwj changed the title hours spent "Rolling forward" "Rolling forward" at startup can take a long time, and is not interruptible Nov 8, 2017
@snobu

This comment has been minimized.

Copy link

@snobu snobu commented Nov 27, 2017

Outside of making sure the system always allows bitcoind to cleanly exit, is there maybe a workaround for those times the infrastructure just slips from underneath you?

@mecampbellsoup

This comment has been minimized.

Copy link

@mecampbellsoup mecampbellsoup commented May 4, 2018

In my case, I had a very large dbcache=6144 and left my machine unattended at which point it ran out of battery and had a very un-clean exit 😆

So for the past day or so bitcoind has been 'rolling forward' all the blocks it was processing, most of which resided in the potentially-tainted DB cache:
image

Since I can't bitcoin-cli stop until this resync completes, is it safe to kill bitcoind, i.e. will I lose all the work that's been done so far to roll these blocks forward? Or is each block being processed and written to disk (is that the definition of 'rolling forward')? Any links to source code would be appreciated! 😄

@rbndg

This comment has been minimized.

Copy link

@rbndg rbndg commented May 6, 2018

I had the same issue and I did exit and it was fine after I reopened the bitcoind. However bitcoind will continue to roll forward after reopening

@Sjors

This comment has been minimized.

Copy link
Member

@Sjors Sjors commented Jun 9, 2018

I just ran into this rolling forward issue after running out of memory, although when that same machine ran out of memory earlier it just started syncing the regular way from where it left off. This rolling forward also seems to be 5x slower per block than the sync was (update: actually, it speeds up after a few minutes).

@ccdle12

This comment has been minimized.

Copy link
Contributor

@ccdle12 ccdle12 commented Jun 16, 2018

same issue, testing a fault tolerant node in the cloud and spinning it back up after a forced crash using a persisted datadir. The resync has to complete before calling bitcoind or bitcoin-cli to stop. Just wondering if there could be a mechanism to improve the resync?

@StayCoolDK

This comment has been minimized.

Copy link

@StayCoolDK StayCoolDK commented Jul 11, 2018

I've also had this issue. Recently i've experienced that it's getting harder to shutdown Bitcoin Core. The debug.log stops when the mempool is dumped, forcing me to do a "unclean shutdown", forcing the rolling forwards of blocks. Why is it so hard to shutdown Bitcoin Core cleanly, and why is there no appropriate feedback to what is happening? I'm running it on the newest MacOS, version 0.16.1.

@Sjors

This comment has been minimized.

Copy link
Member

@Sjors Sjors commented Jul 11, 2018

How long did you wait? How big is your dbcache? The database cache is flushed to disk after that mempool log message, and it can take a while if you synced many new blocks.

Maybe there should be a log message indicating that it's ongoing.

@StayCoolDK

This comment has been minimized.

Copy link

@StayCoolDK StayCoolDK commented Jul 11, 2018

I waited about an hour. The dbcache is 8000MB. Alright, that's good to know for future reference, thanks! And i do agree. A log message should indicate that the cache is being flushed to disk.

@reqshark

This comment has been minimized.

Copy link

@reqshark reqshark commented Sep 6, 2018

this bitcoin.conf file has:

dbcache=16384

so boring, there's nothing informative, just hella zero padded hashes all damn day...

so what's the resolution on this one? HODL your roll forward i guess

@romainPellerin

This comment has been minimized.

Copy link

@romainPellerin romainPellerin commented Sep 21, 2018

I concur, we are syncing a node with txindex=1, and it takes forever already. So when the node starts rolling forward it adds up... 1 day to roll forward... This must be fixed since it slows down adoption.

@romainPellerin

This comment has been minimized.

Copy link

@romainPellerin romainPellerin commented Sep 21, 2018

For future users who could face the same issue, we solved it by setting:
dbcache=16384 in bitcoin.conf
and running our node on an AWS r4.large instance.
Solution not accessible to everyone though.

@rraallvv

This comment has been minimized.

Copy link

@rraallvv rraallvv commented Oct 31, 2018

Same problem here, is there some alternative to the official bitcoin daemon?
I'm just started to run a full node and had only about 4G of synced data, when the problem showed up, so I'd prefer using something else if possible.

@mecampbellsoup

This comment has been minimized.

Copy link

@mecampbellsoup mecampbellsoup commented Oct 31, 2018

@rraallvv not to be a jerk but did you Google that?

In addition to btcd, you can use bcoin from purse.io as well: https://bcoin.io/

@rraallvv

This comment has been minimized.

Copy link

@rraallvv rraallvv commented Oct 31, 2018

@mecampbellsoup Thanks, I'll give it a try.

I should have pointed out that googling for Bitcoin alternatives isn't that help full, all I get is spammy posts about alt-coins and ICO offerings. Something similar happened to me when trying to search for different alternatives to the official Ethereum client.

@vizeet

This comment has been minimized.

Copy link

@vizeet vizeet commented Jun 12, 2019

I am struck by the same problem. My dbcache is 10G. My battery ran out and it was forward rolling till the point it got stuck. debug.log is not updating anymore. I see chainstate is still rising. Probably I'll leave the system for couple of hours till chainstate is fully loaded.

@setpill

This comment has been minimized.

Copy link
Contributor

@setpill setpill commented Aug 8, 2019

How does one shut down bitcoind cleanly to prevent this? I ran into this problem after calling systemctl stop bitcoind.service. I am using this bitcoind.service (with a different blocksdir). As far as I understand the changes I made (fixing the initial setup of the config dir permissions) should not affect the way bitcoind is shut down via systemctl, leading me to believe the provided bitcoind.service file from the bitcoin core repo also has this problem.

@setpill

This comment has been minimized.

Copy link
Contributor

@setpill setpill commented Aug 8, 2019

bitcoind.service specific issue: #13736

@glendon144

This comment has been minimized.

Copy link

@glendon144 glendon144 commented Oct 29, 2019

I find this behavior extremely frustrating. I run several Bitcoin Core pruning nodes, and one of them
has been rolling forward for about 24 hours with no status report. I would at least like to see a percent indicator based on what percentage of blocks have been processed. This wouldn't have to update constantly, updating once every 10 minutes or so would help a lot. I realize that the status of this source code has almost reached religious significance for those of us who are bitcoin maximalists, but I think this would be a great feature to add, if anyone wants to do it. (If it's not done already by the time you read this.) Thanks!

@somewhatnmc

This comment has been minimized.

Copy link

@somewhatnmc somewhatnmc commented Nov 30, 2019

*somewhat waves!

Naive question, is there an upper bound on the time this rolling forward takes? Could it be more than IBD? Somewhat asks as they may just delete everything and start afresh?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.