Indexing Accounts
If you're handling a large number of files (~10k -> ~100k+, depending on your system capacities) it becomes advantageous for ngind to implement persistent indexing of the key files.
🚀 This feature relatively young -- having only been included in 3.5.0 -- and we'll be very happy to hear your feedback if you've taken advantage of it. 👓
Ngind allows for a large amount of leniency in managing and interacting with key files. For example, you can have a keyfile named "foobar.md", and that's just fine. The downside of this is that it presents some inefficiencies in how ngind is able to match accounts (by address) with their respective files. Ngind also keeps a live sync of your /keystore directory, enabling you to add/change/remove files, with those changes reflected in realtime in ngind. These conveniences and complications tonginder -- combined with the innate awkwardness of handling a large number of small files (try running ls -l for a directory with 500k files) -- present a hurdle when the number of key files grows to a high number.
The solutions which persistent indexing uses to address these problems are
- disable FS watching and live reload, and
- establish ngind's use of
accounts.dbas its persistent cache instead of holding file indexes in memory.
Instead of asking ngind to re-index and store all key files each time a ngind instance is started, it'll instead connect with a lightweight key-value store in /keystore/accounts.db.
Before running ngind with the indexed accounts option, you'll first need to build the index. Run:
$ ngind --verbosity 5 --index-accounts account indexto create an index store (accounts.db). The --verbosity flag in this example is optional, but may be desirable for monitoring progress.
You may additionally use --data-dir, --chain, and --keystore flags to specify custom base data directory, chain subdirectory, and keystore directory as well.
Once you've built an index, running:
$ ngind --index-accountstoggles the persistent index option.
All calls managing key files via ngind API/RPC/CLI will be automatically updated and persisted in the index file. However, if you make outside changes to your key files, those changes will not be synced to the index. In this case, rebuilding your index -- with the exact same command as setup) -- will suffice to to update the index with those changes.
All measurements in nanoseconds/operation with watching disabled on *2.8 GHz Intel Core 2 Duo @ 8GB RAM.
This tests a bundled general suite of four interactions without querying. Here, in-memory is king, because the test essentially measures manipulations from memory vs. disk i/o. While persistent storage takes significantly more time (as high as 0.24 seconds for the suite), it's also worth noting that both rates behave approximately constantly across every tested number of key files.
| Number of Accounts | In-memory caching | Persistent caching |
|---|---|---|
| 100 | 29147464 | 88912525 |
| 500 | 58596449 | 144582028 |
| 1000 | 32701806 | 203180542 |
| 5000 | 33743063 | 74667460 |
| 10000 | 24605203 | 133318754 |
| 20000 | 26082395 | 92159681 |
| 100000 | 21709988 | 168614805 |
| 200000 | 22346704 | 249697976 |
| 500000 | 39045149 | 161666411 |
This tests a commonly used key transaction for a pre-existing key file/account. Here's where the payoff for persistent indexing comes; persisted key indexes must never be sorted or parsed entirely.
| Number of Accounts | In-memory caching | Persistent caching |
|---|---|---|
| 100 | 2963462 | 2923034 |
| 500 | 2892381 | 2871620 |
| 1000 | 5049611 | 3070000 |
| 5000 | 5862756 | 3035130 |
| 10000 | 5824062 | 3328642 |
| 20000 | 724523342 | 2990753 |
| 100000 | 356307574244 | 3314435 |
| 200000 | DNF | 5204148 |
| 500000 | DNF | 3114311 |