Skip to content

Heavy hash persistent cache#21

Merged
hidaloop merged 6 commits intoobtc-nightlyfrom
barrystyle/hash-cache
Jul 13, 2022
Merged

Heavy hash persistent cache#21
hidaloop merged 6 commits intoobtc-nightlyfrom
barrystyle/hash-cache

Conversation

@hidaloop
Copy link
Copy Markdown
Collaborator

Reopening PR #17, as the old branch was deleted.
(by @barrystyle)

Rational:
Obtc node is extremely inefficient on the start-up due. As the blocks retrieved from the disk, they are put in the runtime structures, which require calling heavy-hash many times. Heavy-hash is clock cycle intensive, thus at the current chain height starting a node takes roughly 8 mins on average, which is really annoying.
@barrystyle suggests storing previously encountered hashes to a db, indexing each with a lightweight SHA1 hash, thereby bringing the start-up time back to almost instant.

barrystyle and others added 2 commits February 21, 2022 10:31
Cache powhashes in a persistent database. Test purposes.
@hidaloop hidaloop marked this pull request as draft June 21, 2022 15:33
@hidaloop
Copy link
Copy Markdown
Collaborator Author

hidaloop commented Jul 10, 2022

@barrystyle have you obtained a successful build of this version on your machine?
I am encountering a lot of problems with linking. Even the obvious one with logging, where none of the logging related files were linked.

hidaloop added 4 commits July 13, 2022 11:16
Rationel:
Convience libraries built by bitcoin contain logically separated code, which allows for modularity. Introducing powcache -> dbwrapper dependecy breaks this modularity by making consensus dependant on server/wallet functionality.
Apart from that it breaks builds for libconsesnsus shared library, and bruteforce fixing by satisfying every symbol is incorrect. It is designed to be stand-alone and lightweight for a reason.
Thus this commit relinks powcache.* files to server library, and implements cache lookup outside of primitives.
1. Adds a wrapper around PoW cache database, which allows for easy substitution of any heavy block.GetHash() encounter.
2. Removes redundant calls.

Consideration:
Performing a database lookup with 160 bit long buffer instead of 256 bit blob.
Due to the fact that SHA1 output is 160 bits long 256 - 160 = 96 bits are zeros simply taking up space. It is not a big issue, but may result in extra 1 MB of data every week.
@hidaloop hidaloop marked this pull request as ready for review July 13, 2022 12:38
@hidaloop hidaloop changed the base branch from master to obtc-nightly July 13, 2022 12:39
@hidaloop
Copy link
Copy Markdown
Collaborator Author

This should conclude a round of development. Currently merging to obtc-nightly branch for pre-production testing on mainnet.
Further commits will be squashed.

@barrystyle thank you for the initiative!

@hidaloop hidaloop merged commit 9a6ba2f into obtc-nightly Jul 13, 2022
@hidaloop hidaloop deleted the barrystyle/hash-cache branch July 13, 2022 12:43
@barrystyle
Copy link
Copy Markdown

@barrystyle have you obtained a successful build of this version on your machine? I am encountering a lot of problems with linking. Even the obvious one with logging, where none of the logging related files were linked.

Ensure you're building with latest g++, not 5 series which dates back to 2015.

@hidaloop
Copy link
Copy Markdown
Collaborator Author

Thank you for the update.
However, the way it was originally implemented would still create a couple of problems:

  1. bitcoin-tx would break because it would not be able to acquire the lock, as the powcache leveldb would be open by the core process.
  2. libbitcoinconsensus builds would not be satisfied by only adding BITCOIN_INCLUDES to preprocessor flags. A lot of symbols would be left out.
  3. Also making consensus dependent on leveldb goes against the introduced modularity.

@barrystyle
Copy link
Copy Markdown

barrystyle commented Jul 23, 2022

Issues shown and resolved in #17

  1. Was a chicken-and-egg issue, where the db was not initialized when chainparams is being declared.

2 and 3) Builds fine, and from memory had issues unless they were added. As for leveldb, that is neither here nor there - Bitcoin is not at the point where someone can use whichever key/value store database they please. The point remains that the hashing functions are defined and used from within the consensus library (which requires the crypto primitives library too by the way), and whilst they are used in there - they require linking against leveldb.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants