geth takes over an hour to start with a lot of keys in the keystore

[Crypto2]

Hi,

This has been going on for a long time but geth takes longer and longer to start up. Currently using the pre-compiled 1.8.10 on Linux 64-bit. Pretty beefy machine with 128GB of RAM, NVMe storage, 3 GHz Xeon w/ 8 cores / 16 threads.

My best guess is it's because we have a lot of keys since it's for a wallet provider.

Quick example from today showing timestamps between these 2 lines of geth output:

INFO [06-04|04:26:23] Maximum peer count ETH=25 LES=0 total=25
INFO [06-04|05:38:10] Starting peer-to-peer node instance=Geth/v1.8.10-stable-eae63c51/linux-amd64/go1.10.

[jsvisa]

The time before node starting is the sync time, you can check out #16875 (comment) to see more

[Crypto2]

This is before syncing ever starts. These are the 1st two lines of geth output.

[timnugent]

Same issue here with 1.8.10 - also a powerful machine and previously synced up to block ~5770000. Only a few keys on this one.

[Crypto2]

Any update on this? It persists in 1.8.12.

[Crypto2]

At least in my case it does seem to be related to having a lot of keys, I renamed my keystore folder and it started up right away as would be expected.

[Crypto2]

@jsvisa Yep, just tried this on another machine with the same result.
with keystore = over an hour to start, without keystore = immediate start

[jsvisa]

@Crypto2 How much keys under keystore do you have? I'm try to start geth with 100 keys and no more much time to start in my local macOS.

[Crypto2]

Several million, they should be loaded/scanned in another thread or something?

[Crypto2]

Any updates on this?

[Crypto2]

Any updates? Still takes hours to start.

If I start it with keystore renamed then move the files back in will it at least scan them in the background then even if it takes a long time? So we can process the blockchain and do things besides sending at least?

[Crypto2]

@jsvisa Any updates? I can do the rename as indicated in my last post but it takes nearly a day for the keys to scan in the background. There has to be some way to speed this up?

[ligi]

How many keys do you have in your keystore? And what are your cache settings?

[holiman]

@Crypto2 could you provide your startup flags ?

[Crypto2]

~4.5 million keys

Using: geth --ipcdisable --rpc --rpcapi "eth,net,web3,personal,admin,debug" --cache 16384 --syncmode fast --gcmode archive

[ligi]

@Crypto2 thanks for the info - this is most likely the cause of the delay then. Can you elaborate a bit on your use-case? Why do you need 4.5 million keys in your keystore?

[holiman]

So, fun fact, although the filename contains the address, geth doesn't rely on that. Instead, geth opens every one of those 4.5 million keys and checks the address within, and places them in a cache, to provide mappings from address to file.

But I'm also curious why 4.5 million keys would be useful. A crypto exchange might want to consider using custom software to manage accounts in bulk -- geth is mainly trying to be a general usecase node. If there were some simple way of optimizing away that check, that'd be one thing, but that's not really feasible.

[ligi]

one thing that could be done for sure:

• print a warning when the keystore contains too many files so users know this is the culprit

I could imagine optimizations that could be done there - e.g. cache the address->file mapping - but I am not sure this should be done as it really looks like the wrong tool is used for the job in this case.

[Crypto2]

Yes it's a hosted wallet. If we knew it was going to be an issue beforehand we would have tried to do something different; it wasn't until years in that it became a problem. I would say there for sure should be some kind of address <-> file mapping since addresses have to be unlocked before use that's the perfect time for it to actually try to load and decrypt the key.

[Crypto2]

Any update on this to load the keys faster and/or in the background?

[adamschmideg]

This is a feature that could be implemented in Clef but it's far from obvious.

[Crypto2]

Any updates on this? Such as:

1. Saving the mapping to a file periodically that can be reloaded on the next start or into an LMDB table.
2. Scanning the filenames into a list 1st before trying to load them, that way if a key is needed that hasn't been checked yet it can try based on the filename on demand.