multi: Add UtxoCache. #2591

rstaudt2 · 2021-02-11T12:18:37Z

NOTE: If you plan to test this, it is recommended to make a copy of your data directory first. Though this does not have a database migration, this changes the way that the utxo set is synced to the database and in the case of an unclean shutdown on this branch, the old code will not be able to recover.

UtxoCache is an unspent transaction output cache that sits on top of the utxo set database and provides significant runtime performance benefits at the cost of some additional memory usage. It drastically reduces the amount of reading and writing to disk, especially during initial block download when a very large number of blocks are being processed in quick succession.

Initial block download performance results are as follows. This testing was done syncing from a local peer (to avoid peer speed/latency variance) on a 2.4 GHz 8-Core w/ SSD machine (I don't have a HDD to test with currently but the performance improvement may be even more significant in that case since the speedup is primarily due to reducing reads/writes to disk):

Master
- Average IBD Time: ~48 mins
- Peak Real Memory Usage: ~850MB
With UtxoCache (w/ default utxocachemaxsize of 150 MiB)
- Average IBD Time: ~32 mins (~33% faster)
- Peak Real Memory Usage: ~1250MB
- Cache Hit Ratio: ~99%

The UtxoCache is a read-through cache. All utxo reads go through the cache. When there is a cache miss, the cache loads the missing data from the database, caches it, and returns it to the caller.

The UtxoCache is a write-back cache. Writes to the cache are acknowledged by the cache immediately but are only periodically flushed to the database. This allows intermediate steps to effectively be skipped. For example, a utxo that is created and then spent in between flushes never needs to be written to the utxo set in the database.

Due to the write-back nature of the cache, at any given time the database may not be in sync with the cache, and therefore all utxo reads and writes now go through the cache, and never read or write to the database directly.

This has been broken down into separate commits where possible to make the review process a bit easier since it is a fairly large changeset. An overview of the changes is as follows:

Update UtxoEntry for the cache
- Separate utxo state from tx flags
- Add utxoStateFresh to UtxoEntry, which indicates that an entry exists in the utxo cache but does not exist in the
  underlying database
- Add size method to UtxoEntry to be used as part of tracking the total size of the utxo cache
Add utxocachemaxsize
- Add a utxocachemaxsize configuration option which represents the maximum size in MiB of the utxo cache. The default value is 150 MiB and the minimum value is 25 MiB
Deep copy view entry script from tx
- Update utxo viewpoints to deep copy the script when getting it from a msg tx. This is required since the tx out script is a subslice of the overall contiguous buffer that the msg tx houses for all scripts within the tx. It is deep copied here since this entry may be added to the utxo cache, and we don't want the utxo cache holding the entry to prevent all of the other tx scripts from getting garbage collected
Add utxoSetState to the database
- Add a utxoSetState type that is tracked in the database. The utxo set state contains information regarding the current state of the utxo set. In particular, it tracks the block height and block hash of the last completed flush
- The utxo set state is tracked in the database since at any given time, the utxo cache may not be consistent with the utxo set in the database. This is due to the fact that the utxo cache only flushes changes to the database periodically. Therefore, during initialization, the utxo set state is used to identify the last flushed state of the utxo set and it can be caught up to the current best state of the main chain.
- Add full test coverage for the new serialization and deserialization functions
Add UtxoCache and UtxoCacheConfig struct types and NewUtxoCache method
- Update server to create the utxo cache with the configured max size and pass to the block chain instance that is created
- Update all test block chains to create a utxo cache
Add FetchEntry to UtxoCache
- FetchEntry returns the specified transaction output from the utxo set
- If the output exists in the cache, it is returned immediately. Otherwise, it uses an existing database transaction to fetch the output from the database, caches it, and returns it to the caller.
Add AddEntry to UtxoCache
- AddEntry adds the specified output to the cache
Add SpendEntry to UtxoCache
- SpendEntry marks the specified output as spent
- Remove entries that are marked as fresh and then subsequently spent. This is an optimization to skip writing to the database for outputs that are added and spent in between flushes to the database.
Update UtxoViewpoint to hold the UtxoCache
- Update fetching entries from the database to fetch entries from the cache instead
Add Commit to UtxoCache
- Commit updates all entries in the cache based on the state of each entry in the provided view
- All entries in the provided view that are marked as modified and spent are removed from the view
- Additionally, all entries that are added to the cache are removed from the provided view
Add MaybeFlush to UtxoCache
- MaybeFlush conditionally flushes the cache to the database
- If the maximum size of the cache has been reached, or if the periodic flush duration has been reached, then a flush is required
- A flush can be forced by setting the force flush parameter
- Flushing commits all modified entries to the database and conditionally evicts entries
- Entries that are nil or spent are always evicted since they are unlikely to be accessed again. Additionally, if the cache has reached its maximum size, entries are evicted based on the height of the block that they are contained in
Update connect block and disconnect block to commit to the cache and conditionally flush to the database
- Rather than writing to the utxo set in the database every time that a block is connected or disconnected, commit the updated view to the cache and call MaybeFlush on the cache to conditionally flush it to the database
Add InitUtxoCache to UtxoCache
- InitUtxoCache initializes the utxo cache by ensuring that the utxo set is caught up to the tip of the best chain
- Since the cache is only flushed to the database periodically, the utxo set may not be caught up to the tip of the best chain
- InitUtxoCache catches the utxo set up by replaying all blocks from the block after the block that was last flushed to the tip block through the cache
Add ShutdownUtxoCache to BlockChain
- ShutdownUtxoCache flushes the utxo cache to the database on shutdown. Since the cache is flushed periodically during initial block download and flushed after every block is connected after initial block download is complete, this flush that occurs during shutdown should finish relatively quickly
- Note that if an unclean shutdown occurs, the cache will still be initialized properly when restarted as during initialization it will replay blocks to catch up to the tip block if it was not fully flushed before shutting down. However, it is still preferred to flush when shutting down versus always recovering on startup since it is faster
Track the hit ratio of UtxoCache
- Track the number of hits and misses when accessing the cache in order to calculate the overall hit ratio of the cache to gauge its performance
Add UtxoCache test coverage
- Full test coverage has been added to the UtxoCache type and its methods

Other implementation notes:
- There is definitely still plenty of room for further optimizations.
- I did experiment a bit with chunked allocation/free lists/sync pool for the entries and scripts that the cache holds but they were ultimately resulting in worse performance. There may still be an optimization there but what ended up working best was just the UtxoCache taking ownership of the entries from the view and avoiding additional allocations when adding the entries altogether.
- One optimization that will be followed up on (suggested by @davecgh) is to separate out the utxo database, which should bring both performance and memory usage improvements.
  - One reason why the cache results in more than a 100 MiB increase in overall memory usage when setting the max to 100 MiB is due to the database caching layer copying entries and holding them before flushing them to the db. If we separate out the utxo database we can likely just remove the db caching since we have an optimized higher-level cache in place now.

Part of #1145.

davecgh · 2021-02-11T18:19:13Z

davecgh · 2021-02-11T18:27:05Z

We'll need to update the minimum recommended specs in the main README.md to 2GB.

blockchain/utxoentry.go

blockchain/utxoviewpoint.go

config.go

davecgh · 2021-02-11T22:09:42Z

I marked this as a database upgrade even though it technically isn't 100% one because you can't really go back without causing potential issues when you upgrade again.

rstaudt2 · 2021-02-11T23:02:41Z

We'll need to update the minimum recommended specs in the main README.md to 2GB.

Updated the README.md to now recommended a minimum of 2GB of memory.

blockchain/chainio.go

blockchain/chainio_test.go

rstaudt2 · 2021-02-12T12:08:33Z

In the last push, I also updated the cache flush log message to only print the eviction height part of the message when eviction is required (previously it would print eviction height: 0 in this case, which was a bit misleading).

dnldd

Nicely done, looks good.

blockchain/utxocache.go

davecgh

As of this review, I've now finished reviewing all of the code. Very nice job overall and the test coverage is solid. I know this was a lot of work, but it will have a major impact not only at the current time, but, perhaps even more importantly, it significantly improves scalability.

I have just a few inline comments regarding the tests and I'll begin testing the scenarios outlined in an earlier comment and provide feedback with the results.

blockchain/utxocache_test.go

blockchain/utxocache.go

davecgh

I've tested the following scenarios and, aside from the aforementioned issue, everything is working as expected:

Sync from new database (HDD and SSD)
Sync from existing older upgraded database
Reconciliation after hard killing the process during sync
Reconciliation after physically removing hardware from the system during sync (disconnect USB drive)
Double spend attempts on flushed entries
Double spend attempts on unflushed entries
Intentional corruption of the underlying database to ensure appropriate errors happen

blockchain/chainio.go

davecgh

I've gone over this several times now and tested it quite thoroughly. Looks good to go!

jholdstock

I ran this PR through a whole bunch of IBDs and also for a decent amount of normal operation. All looking stable and improvements are very noticeable.

Some benchmarking data from a system running NVMe SSD + Ryzen 2700x, syncing from a local peer.

Without indexes

Code	Network	Time	Notes
master	mainnet	61m 1s
master	testnet	84m 24s
PR - 100MiB cache	mainnet	44m 26s (-27%)	Min cache hit ratio: 98.65%
PR - 100MiB cache	testnet	67m 44s (-20%)	Min cache hit ratio: 99.85%
PR - 2000MiB cache	mainnet	36m 32s (-40%)	Min cache hit ratio: 100.00%. Max cache size 261.05 MiB (3,139,811 entries)
PR - 2000MiB cache	testnet	58m 39s (-31%)	Min cache hit ratio: 100.00%. Max cache size 243.06 MiB (2,025,762 entries)

With `--addrindex` and `--txindex`

Code	Network	Time	Notes
master	mainnet	98m 53s
master	testnet	128m 38s
PR - 100MiB cache	mainnet	84m 6s (-15%)	Min cache hit ratio: 98.99%. Noticed during this run that the cache size can exceed the max specified - 100.01MiB vs 100MiB max.
PR - 100MiB cache	testnet	114m 39s (-11%)	Min cache hit ratio: 99.86%

rstaudt2 · 2021-02-15T13:54:15Z

Thanks for testing, @jholdstock!

Noticed during this run that the cache size can exceed the max specified - 100.01MiB vs 100MiB max.

This is expected, it can go over the max size by the total size of the outputs of 1 block (at most), which is pretty minimal, since it only checks the size and potentially flushes after connecting each block.

davecgh · 2021-02-17T15:56:16Z

Mind rebasing this? It was already behind master a bit and there is a minor merge conflict now.

blockchain/utxocache.go

This splits the utxo packed flags into two separate types, utxoState and utxoFlags. The reasoning is that: - This cleanly separates the purpose of the flags. utxoState defines the in-memory state of a utxo entry, whereas utxoFlags defines additional information for the containing transaction of a utxo entry. - This makes room for an additional state that is required for the utxo cache, namely whether or not a utxo entry is fresh (does not exist as an unspent transaction output in the database).

This adds utxoStateFresh to UtxoEntry to indicate that a txout is fresh, which means that it exists in the utxo cache but does not exist in the underlying database. The utxo cache will use the fresh flag as an optimization to skip writing to the database for outputs that are added and spent in between flushes to the database.

This adds a size method to UtxoEntry, which returns the number of bytes that the entry uses on a 64-bit platform. This will be used as part of tracking the total size of the utxo cache.

This adds a utxocachemaxsize configuration option which represents the maximum size in MiB of the utxo cache. The default value is 150 MiB, the minimum value is 25 MiB, and the maximum value is 32768 MiB (32 GiB).

This updates utxo viewpoints to deep copy the script when getting it from a msg tx. This is required since the tx out script is a subslice of the overall contiguous buffer that the msg tx houses for all scripts within the tx. It is deep copied here since this entry may be added to the utxo cache, and we don't want the utxo cache holding the entry to prevent all of the other tx scripts from getting garbage collected.

This adds a utxoSetState type that is tracked in the database. The utxo set state contains information regarding the current state of the utxo set. In particular, it tracks the block height and block hash of the last completed flush. The utxo set state is tracked in the database since at any given time, the utxo cache may not be consistent with the utxo set in the database. This is due to the fact that the utxo cache only flushes changes to the database periodically. Therefore, during initialization, the utxo set state is used to identify the last flushed state of the utxo set and it can be caught up to the current best state of the main chain. This additionally adds full test coverage to the new serialization and deserialization functions.

UtxoCache is an unspent transaction output cache that sits on top of the utxo set database and provides significant runtime performance benefits at the cost of some additional memory usage. It drastically reduces the amount of reading and writing to disk, especially during initial block download when a very large number of blocks are being processed in quick succession. The UtxoCache is a read-through cache. All utxo reads go through the cache. When there is a cache miss, the cache loads the missing data from the database, caches it, and returns it to the caller. The UtxoCache is a write-back cache. Writes to the cache are acknowledged by the cache immediately but are only periodically flushed to the database. This allows intermediate steps to effectively be skipped. For example, a utxo that is created and then spent in between flushes never needs to be written to the utxo set in the database. Due to the write-back nature of the cache, at any given time the database may not be in sync with the cache, and therefore all utxo reads and writes MUST go through the cache, and never read or write to the database directly. An overview of the changes is as follows: - Add UtxoCache and UtxoCacheConfig struct types and NewUtxoCache method - Update server to create the utxo cache with the configured max size and pass to the block chain instance that is created - Update all test block chains to create a utxo cache - Add FetchEntry to UtxoCache - FetchEntry returns the specified transaction output from the utxo set - If the output exists in the cache, it is returned immediately. Otherwise, it uses an existing database transaction to fetch the output from the database, caches it, and returns it to the caller. - Add AddEntry to UtxoCache - AddEntry adds the specified output to the cache - Add SpendEntry to UtxoCache - SpendEntry marks the specified output as spent - Remove entries that are marked as fresh and then subsequently spent. This is an optimization to skip writing to the database for outputs that are added and spent in between flushes to the database. - Update UtxoViewpoint to hold the UtxoCache - Update fetching entries from the database to fetch entries from the cache instead - Add Commit to UtxoCache - Commit updates all entries in the cache based on the state of each entry in the provided view - All entries in the provided view that are marked as modified and spent are removed from the view - Additionally, all entries that are added to the cache are removed from the provided view - Add MaybeFlush to UtxoCache - MaybeFlush conditionally flushes the cache to the database - If the maximum size of the cache has been reached, or if the periodic flush duration has been reached, then a flush is required - A flush can be forced by setting the force flush parameter - Flushing commits all modified entries to the database and conditionally evicts entries - Entries that are nil or spent are always evicted since they are unlikely to be accessed again. Additionally, if the cache has reached its maximum size, entries are evicted based on the height of the block that they are contained in. - Update connect block and disconnect block to commit to the cache and conditionally flush to the database - Rather than writing to the utxo set in the database every time that a block is connected or disconnected, commit the updated view to the cache and call MaybeFlush on the cache to conditionally flush it to the database - Add InitUtxoCache to UtxoCache - InitUtxoCache initializes the utxo cache by ensuring that the utxo set is caught up to the tip of the best chain - Since the cache is only flushed to the database periodically, the utxo set may not be caught up to the tip of the best chain - InitUtxoCache catches the utxo set up by replaying all blocks from the block after the block that was last flushed to the tip block through the cache - Add ShutdownUtxoCache to BlockChain - ShutdownUtxoCache flushes the utxo cache to the database on shutdown. Since the cache is flushed periodically during initial block download and flushed after every block is connected after initial block download is complete, this flush that occurs during shutdown should finish relatively quickly - Note that if an unclean shutdown occurs, the cache will still be initialized properly when restarted as during initialization it will replay blocks to catch up to the tip block if it was not fully flushed before shutting down. However, it is still preferred to flush when shutting down versus always recovering on startup since it is faster - Track the hit ratio of UtxoCache - Track the number of hits and misses when accessing the cache in order to calculate the overall hit ratio of the cache to gauge its performance

This adds full test coverage to the UtxoCache type and its methods. Additionally, since this uses the testing Cleanup function that was introduced in Go 1.14, this bumps the required go version for the blockchain package from 1.13 to 1.14.

This updates the minimum recommended memory (RAM) from 1GB to 2GB in the main README.md. The minimum recommended memory is being increased due to the introduction of the utxo cache.

davecgh added the optimization label Feb 11, 2021

davecgh added this to the 1.7.0 milestone Feb 11, 2021

davecgh reviewed Feb 11, 2021

View reviewed changes

blockchain/utxoentry.go Show resolved Hide resolved

blockchain/utxoviewpoint.go Outdated Show resolved Hide resolved

davecgh requested changes Feb 11, 2021

View reviewed changes

config.go Show resolved Hide resolved

davecgh added the database upgrade Issues and/or pull requests that involve a new database version. label Feb 11, 2021

rstaudt2 force-pushed the utxo-cache branch from 426de84 to 7f90c82 Compare February 11, 2021 22:58

davecgh mentioned this pull request Feb 12, 2021

Multi-peer Checklist #1145

Open

28 tasks

davecgh requested changes Feb 12, 2021

View reviewed changes

blockchain/chainio.go Outdated Show resolved Hide resolved

blockchain/chainio_test.go Show resolved Hide resolved

rstaudt2 force-pushed the utxo-cache branch from 7f90c82 to 80a14d1 Compare February 12, 2021 12:02

dnldd approved these changes Feb 12, 2021

View reviewed changes

davecgh requested changes Feb 13, 2021

View reviewed changes

blockchain/utxocache.go Outdated Show resolved Hide resolved

davecgh requested changes Feb 13, 2021

View reviewed changes

rstaudt2 force-pushed the utxo-cache branch from 80a14d1 to d56710a Compare February 13, 2021 20:11

davecgh requested changes Feb 13, 2021

View reviewed changes

blockchain/utxocache_test.go Show resolved Hide resolved

blockchain/utxocache_test.go Outdated Show resolved Hide resolved

blockchain/utxocache_test.go Outdated Show resolved Hide resolved

rstaudt2 force-pushed the utxo-cache branch from d56710a to 82bc4a7 Compare February 13, 2021 22:55

davecgh requested changes Feb 14, 2021

View reviewed changes

blockchain/utxocache.go Outdated Show resolved Hide resolved

davecgh reviewed Feb 14, 2021

View reviewed changes

rstaudt2 force-pushed the utxo-cache branch 3 times, most recently from 65894d6 to 5cddcbc Compare February 14, 2021 20:43

davecgh requested changes Feb 15, 2021

View reviewed changes

blockchain/chainio.go Outdated Show resolved Hide resolved

rstaudt2 force-pushed the utxo-cache branch from 5cddcbc to 564e2f2 Compare February 15, 2021 00:57

davecgh approved these changes Feb 15, 2021

View reviewed changes

jholdstock approved these changes Feb 15, 2021

View reviewed changes

rstaudt2 mentioned this pull request Feb 16, 2021

blockchain: Use new style for chainio test errors. #2595

Merged

rstaudt2 force-pushed the utxo-cache branch from 564e2f2 to c16a57e Compare February 17, 2021 16:14

JoeGruffins reviewed Feb 19, 2021

View reviewed changes

rstaudt2 added 10 commits February 19, 2021 05:06

blockchain: Add test name to TestUtxoEntry errors.

896c159

blockchain: Add size method to UtxoEntry.

c5d7003

This adds a size method to UtxoEntry, which returns the number of bytes that the entry uses on a 64-bit platform. This will be used as part of tracking the total size of the utxo cache.

config: Add utxocachemaxsize.

4b42840

This adds a utxocachemaxsize configuration option which represents the maximum size in MiB of the utxo cache. The default value is 150 MiB, the minimum value is 25 MiB, and the maximum value is 32768 MiB (32 GiB).

blockchain: Add UtxoCache test coverage.

3afb445

This adds full test coverage to the UtxoCache type and its methods. Additionally, since this uses the testing Cleanup function that was introduced in Go 1.14, this bumps the required go version for the blockchain package from 1.13 to 1.14.

docs: Update min recommended specs in README.md.

e4d6623

This updates the minimum recommended memory (RAM) from 1GB to 2GB in the main README.md. The minimum recommended memory is being increased due to the introduction of the utxo cache.

rstaudt2 force-pushed the utxo-cache branch from c16a57e to e4d6623 Compare February 19, 2021 11:11

rstaudt2 mentioned this pull request Feb 19, 2021

blockchain: Add UtxoCache Initialize tests. #2599

Merged

JoeGruffins approved these changes Feb 19, 2021

View reviewed changes

davecgh merged commit 15be1ff into decred:master Feb 22, 2021

rstaudt2 deleted the utxo-cache branch March 13, 2021 11:27

rstaudt2 mentioned this pull request Dec 10, 2021

[WIP] Implement a UTXO cache btcsuite/btcd#1373

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multi: Add UtxoCache. #2591

multi: Add UtxoCache. #2591

rstaudt2 commented Feb 11, 2021 •

edited

davecgh commented Feb 11, 2021 •

edited

davecgh commented Feb 11, 2021

davecgh commented Feb 11, 2021

rstaudt2 commented Feb 11, 2021

rstaudt2 commented Feb 12, 2021

dnldd left a comment

davecgh left a comment

davecgh left a comment

davecgh left a comment

jholdstock left a comment

rstaudt2 commented Feb 15, 2021

davecgh commented Feb 17, 2021

multi: Add UtxoCache. #2591

multi: Add UtxoCache. #2591

Conversation

rstaudt2 commented Feb 11, 2021 • edited

davecgh commented Feb 11, 2021 • edited

davecgh commented Feb 11, 2021

davecgh commented Feb 11, 2021

rstaudt2 commented Feb 11, 2021

rstaudt2 commented Feb 12, 2021

dnldd left a comment

Choose a reason for hiding this comment

davecgh left a comment

Choose a reason for hiding this comment

davecgh left a comment

Choose a reason for hiding this comment

davecgh left a comment

Choose a reason for hiding this comment

jholdstock left a comment

Choose a reason for hiding this comment

Without indexes

With --addrindex and --txindex

rstaudt2 commented Feb 15, 2021

davecgh commented Feb 17, 2021

rstaudt2 commented Feb 11, 2021 •

edited

davecgh commented Feb 11, 2021 •

edited

With `--addrindex` and `--txindex`