Create Non-Indexing Module

[joshmg]

Create Non-Indexing Module

GitHub Issue #7

Problem Description

Currently Bitcoin Verde indexes large parts of the blockchain.

Indexed components include (but are not limited to):

• all transactions
• all outputs and inputs (both spent and unspent)
• P2PK/P2SH addresses
• SLP tokens
• SLP validation
• contentious/orphaned blocks.

Keeping indexes on these components greatly increases the disk footprint of the node, as well as increases the initial block download (and validation) time.
Indexing these fields with modern hardware, including an M.2 SSD, can take over a week.
These indexes also increase the CPU and RAM resources required to optimally run the node.

Despite being very useful for explorers and wallet services, these indexes are mostly unneeded for blockchain validation and provide no value for mining nodes.

Value Proposition

This solution will provide the following benefits:

• reduce the time for the initial block download
  • allow mining pools to start running a Bitcoin Verde node in a more reasonable amount of time
• reduce minimum required resources
  • reduces barriers of entry for both miners and non-indexing node operators
  • allows pool operators to run redundant/backup nodes for their pool at a lower cost
• reduce overall infrastructure cost to run a Bitcoin Verde node

Solution Overview

Running the non-indexing module inherently runs the node with the cacheBlocks option enabled.
With cacheBlocks, blocks are stored serialized on-disk.
Many of the transaction-, address-, and slp-related SQL tables will be removed.
A new transactions table will be created with columns that can be used to point to the location on disk where the transaction's block is stored, and an offset of where the transaction is within the block flat file.
This migrates mined transactions from the database (which is indexed and normalized, which is space inefficient and less performant) to a more compact format in a flat file.

In order to keep mempool/unconfirmed-transaction logic consistent between modules, the former transaction_* tables will be renamed and reused for unconfirmed transactions.
Having unconfirmed transactions indexed and normalized allows Bitcoin Verde to keep its infinite transaction-chaining limit, without any additional development.
As transactions are mined in a block they will be removed from the unconfirmed_transactions_* table, and will be only stored in block flat files.
Since the transaction SQL tables will not be large, indexing only unconfirmed transactions nominally increases the disk footprint of the node, while still allowing extensibility for future features and complex decisions to be made regarding next-block inclusion of unconfirmed transactions.

Additionally, total transaction size and fee amount will be added to the transaction table to improve block template generation.

Changes will affect mostly the data-layer, which is encapsulated by the TransactionDatabaseManager and related classes. Validation logic and network logic may be minimally affected.

Many of the existing tests depend on direct manipulation of the database's data for their setup.
With a different schema, these tests will be broken when run with the schema changes for this module.
The above schema changes won't cause the original test suite to break since the test suite loads the indexing schema by default.
However, with this configuration, many of the tests will not be run against the new schema, causing a fairly large gap in test coverage for critical functionality.
This proposal also includes extension to existing tests to include coverage of the non-indexing schema as well as the indexing schema.

Solution Milestones

1. SQL schema refactoring and data-layer migrations.

   The first milestone will consist of the SQL schema and logic changes discussed in the solution overview. This milestone is considered completed once the tables are removed and the node successfully completes its initial block download on main-net.

2. Updating tests for new data-layer.

   The second milestone consists of updating all broken tests to ensure existing regression tests pass.
   Additionally, the second milestone includes expanding the existing test suite to run against both indexing and non-indexing schemas.

3. Month-long main-net tests ran via the block template aggregator.

   The third milestone will conclude after the node is synced to main-net and its template block is deemed compatible with both Bitcoin ABC and Bitcoin Unlimited nodes. Completion of this milestone requires the node be updated for the new sigops ruleset included in the 2020-05 upgrade, and requires the template block aggregator be completed so that the template block generated by Bitcoin Verde may be automatically validated by other node implementations against current main-net nodes. After a month of creating main-net template blocks without incompatibility, the milestone will be deemed completed.

Estimated Relative Complexity

• Milestone 1 - 80 / 180 (45%)
• Milestone 2 - 60 / 180 (33%)
• Milestone 3 - 40 / 180 (22%)

Budget

This proposal does not have a minimum starting budget.

Completing this proposal will require approximately 180 hours.
At a rate of 0.5 BCH/hr, the total requested budget for this proposal is 90 BCH.

Funding Address

Funding this proposal may be sponsored by sending Bitcoin Cash to the following address:

1716CkGxHLVj2q9b4hKxRb3KgY11xReiv8
(bitcoincash:qpqa2x8cmrd2c6h6acnf7euqgkkl4prvwu7quzc4cc)

Authorization Signature:

The signature is signed with our primary donation address, 1VerdeMuXH1ApSZsQDkuHHZrwJaAtXTVn, which can be found on bitcoinverde.org.

The signature message consists of a Bitcoin Signed Message with the following format:

Issue-Number | Issue Title | Funding Address | Estimate Hours | Budget BCH

Notes:

1. The pre-image includes the concatenation symbol.

Pre-image:

7|Create Non-Indexing Module|1716CkGxHLVj2q9b4hKxRb3KgY11xReiv8|180|90

Signature:
HO1xu3BDJopi/PsAK2sqql0pHD62KtnmmhqoVyzJWkuAA9N6vhla4Pz4ABkCWU+3Qw/fpY+9wsLwM1zR1XO1oDQ=

[joshmg]

This issue was funded in our flipstarter campaign and work on it has begun: https://verde.flipstarter.cash

[joshmg]

Completed with release of v2.0.0.