Reindex fails

[pbies]

Is there an existing issue for this?

• I have searched the existing issues

Current behaviour

Currently with over 800k blocks, when dbcache is set to 24 GB, reindexing the blockchain with txindex=1 at 16 GB of cache used Bitcoin Core hangs.

Expected behaviour

Bitcoin Core should be able to use more than 16 GB of dbcache and shouldn't hang forever when reindexing.

Steps to reproduce

Download whole blockchain, set txindex=1, restart Core with -reindex option.

Relevant log output

Information about dbcache being used fully.

How did you obtain Bitcoin Core

Pre-built binaries

What version of Bitcoin Core are you using?

25.0.0

Operating system and version

Microsoft Windows [Version 10.0.19045.3393]

Machine specifications

CPU 9900K @ max 5 GHz
16 GB RAM 4000 MHz
SSD drives
GPU 1080 Ti

[maflcko]

dbcache is set to 24 GB

16 GB RAM 4000 MHz

You can't set a dbcache max memory usage higher than the available memory on your system.

Maybe there should be an Init error on such cases?

[pbies]

dbcache is set to 24 GB

16 GB RAM 4000 MHz

You can't set a dbcache max memory usage higher than the available memory on your system.

Maybe there should be an Init error on such cases?

I have 1 TB of swapfile available.

And as I can see in source code there is limit to 16 GB, imprinted.

[sipa]

The dbcache value cannot be set higher than 16384. See also #28249

[pbies]

The dbcache value cannot be set higher than 16384. See also #28249

Yes, that's true and now for reindexing the value should be higher as dbcache raises to 16 GB and hangs Bitcoin Core.

[sipa]

How long does it hang? Writing out 16 GB of database operations can take several minutes. Alternatively, it may appear to stall just because it's starting to thrash your swap space once you exceed the available actual RAM.

[pbies]

How long does it hang? Writing out 16 GB of database operations can take several minutes. Alternatively, it may appear to stall just because it's starting to thrash your swap space once you exceed the available actual RAM.

I waited few minutes and then killed Core. Swap space as I have seen wasn't used.

Anyway it does not matter, Core just needs to make maximum dbcache larger. I suggest 32 GB to make it work for few next years.

[sipa]

Yes, there is an issue about that already, #28249. Let's move that part of the discussion there.

But Bitcoin Core hanging is an independent issue. If it's actually hanging, that's a bug. If it's just slow, which is certainly possible with such huge caches, then it's possibly expected.

[sipa]

Did anything appear in debug.log while Bitcoin Core appeared to hang?

[pbies]

Did anything appear in debug.log while Bitcoin Core appeared to hang?

Yes: last message was something like dbcache crossed past allowed value of maximum memory and was two values given in bytes about 16 GiB.

[sipa]

That means it started flushing the whole 16 GiB worth of database changes to disk. I'd recommend waiting longer for it to finish. How long will depend on your setup, but probably not more than a few minutes. If it doesn't complete in say an hour, report back and we can investigate further.

[pbies]

That means it started flushing the whole 16 GiB worth of database changes to disk. I'd recommend waiting longer for it to finish. How long will depend on your setup, but probably not more than a few minutes. If it doesn't complete in say an hour, report back and we can investigate further.

Ok, will do that next time. This does not cancel idea to make dbcache max size larger to be possibly set.

[maflcko]

Closing for now, but please let us know once and if you have more details