Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Twisted Based Wallet Implementation #1155

Merged
merged 250 commits into from Aug 24, 2018

Conversation

@eukreign
Copy link
Member

commented Mar 26, 2018

For Reviewers

This PR is split across three repositories,

  1. a brand new project called torba which is a generic implementation of bitcoin based wallets (integration tests run against bitcoinsegwit and eventually bitcoincash and others);
  2. the custom LBRY extensions ontop of torba are part of this PR and found in lbrynet/wallet;
  3. orchstr8 integration testing framework that orchestrates starting the blockchain binary (-regtest), electrumx (lbryumx) and the torba (lbrynet.wallet) and facilitating their interaction in a controlled environment.

This is a very major change to the daemon and should be evaluated carefully for security, performance and reliability before making it into a release. Furthermore, this PR also ports lbrynet to Python 3 and is no longer compatible with Python 2. **(see explanation at end)

Here are the main points of interest that should be reviewed:

Coin (UTXO) Selection Algorithm

Torba implements the same coin selection algorithm as latest C++ Bitcoin. Please review the algorithm and tests for completeness and correctness:

Torba (impl): https://github.com/lbryio/torba/blob/master/torba/coinselection.py
Bitcoin (impl): https://github.com/bitcoin/bitcoin/blob/master/src/wallet/coinselection.cpp

Torba (tests): https://github.com/lbryio/torba/blob/master/tests/unit/test_coinselection.py
Bitcoin (tests): https://github.com/bitcoin/bitcoin/blob/master/src/wallet/test/coinselector_tests.cpp

Whitepaper: https://murch.one/wp-content/uploads/2016/11/erhardt2016coinselection.pdf

SQLite Storage

Old wallet stored things either in JSON in the wallet file or in memory in Python dictoinaries and lists when running. This has been completely replaced by an SQLite storage solution. Please review the base torba DDL and then the LBRY extension adding extra claim related attributes to the txo table:

Torba: https://github.com/lbryio/torba/blob/master/torba/basedatabase.py
LBRY: https://github.com/lbryio/lbry/blob/lbryum-refactor/lbrynet/wallet/database.py

Getting the DDL right from the beginning is a good idea, because updating code is cheap but updating DDL is time consuming for us (writing migrations) and the users (waiting for migrations to finish running).

Initial Sync & Subsequent Starts

Faster wallet operation is the main reason for this PR; so it's important that the header and TX synchronization code paths fast and correct. Please review the following areas:

https://github.com/lbryio/torba/blob/d4107600a5882fd9d8ccd13a0d3694c96d2d7c05/torba/baseledger.py#L226-L249
update_headers is called on startup (or if you were offline for too long and have more than one header to catchup) while process_header is called on regular sequential updates from electrumx (but switched to update_headers if it determines we're behind more than one).

https://github.com/lbryio/torba/blob/d4107600a5882fd9d8ccd13a0d3694c96d2d7c05/torba/baseledger.py#L251-L331
update_account is called on startup to catchup with blockchain or recreate wallet from scratch, this is done by generating addresses, sync'ing them, then generating more addresses, sync'ing them, etc, this repeats until all addresses have been generated (stopping when address gap is reached).
after the initial startup most of the normal process is between process_status which receives events from electrumx and calls update_history which then proceeds to process the history for an address.
one complication to keep in mind with update_history is that electrumx will send duplicate updates for the same TX if it involves two of your addresses (common case when you pay and have change, or when you pay yourself, etc), it's important to avoid race conditions in those cases (done with self._transaction_processing_locks).

Simple Payment Verification (SPV)

Torba is called an SPV wallet because it verifies that TXs are in the blockchain without having a copy of the blockchain. This verification is predicated on having correct headers (or at least the headers and the TX should come from different parties that were unlikely to have conspired with each other).

SPV works by organizing TXs in a block into merkle trees, then headers contain the root hash of this merkle tree and you can separately, for each TX, request all of the hashes between your TX and the root hash. Then you just traverse up the tree from your TX hashing every two complementary nodes until you reach the root hash, if the TX is valid the root hash in the header should mach the root hash you calculated. Done.

Here is the implementation, please verify for correctness:
https://github.com/lbryio/torba/blob/d4107600a5882fd9d8ccd13a0d3694c96d2d7c05/torba/baseledger.py#L189-L198

This feature is currently covered by integration tests and does not have dedicated unit tests.

Bitcoin Script

Torba implements a general purpose extensible bitcoin script parser. LBRY extends this to add our 3 custom op codes.

Torba (impl): https://github.com/lbryio/torba/blob/master/torba/basescript.py
LBRY (impl): https://github.com/lbryio/lbry/blob/lbryum-refactor/lbrynet/wallet/script.py

Torba (tests): https://github.com/lbryio/torba/blob/master/tests/unit/test_script.py
LBRY (tests): https://github.com/lbryio/lbry/blob/lbryum-refactor/tests/unit/wallet/test_script.py

Creating Transactions

Can't forget this one, the whole point we're all here :-D

There is a special create constructor on Transaction class that will accept any inputs/outputs and then try to balance the two sides of the equation until it is correct (adding necessary inputs/outputs and adding change for excessive inputs). On lbrynet side there are convenient wrappers around it to make it even easier to create LBRY specific claim transactions.

Torba: https://github.com/lbryio/torba/blob/d4107600a5882fd9d8ccd13a0d3694c96d2d7c05/torba/basetransaction.py#L380-L428
LBRY: https://github.com/lbryio/lbry/blob/lbryum-refactor/lbrynet/wallet/transaction.py

All The Things

Above is a list of some of the critical aspects that should get some review before we go to releases. There is a lot more code than outlined above, if you feel motivated please review all of it and provide feedback. I would greatly appreciate it if people find issues that we can fix now before the code goes live! Thanks!

Essentially this means all of torba: https://github.com/lbryio/torba
Everything in lbrynet.wallet directory: https://github.com/lbryio/lbry/tree/lbryum-refactor/lbrynet/wallet

Try Me

You can downoad automated builds of the new daemon here:
http://build.lbry.io/daemon/

** Python 3 port was done because 1) https://pythonclock.org 2) torba's testing framework orchstr8 loads electrumx (lbryumx) and the wallet into a single Python process using the same asyncio reactor which makes debugging very nice (when you hit breakpoint, both server and client freeze), electrumx is Python 3 only, therefore torba had to straddle Python2/Python3 support to work in both orchstr8 and in lbrynet, keeping this compatibility was a real time sink 3) Python 3 is significantly faster (thus new daemon should have perceptible speed increase) and stricter / failing faster (therefore we should hit bugs earlier in testing).


Progress

This is a pure twisted wallet implementation for lbry providing increased stability and performance; completely replaces lbryum. Still a work in progress, do not merge.

Depends on this pull request in lbryum-server: lbryio/lbryum-server#64

Each leaf task below is 5pt (about a day of work).

Core Functionality I1

  • groking existing thread/plain socket based architecture, considering how to refactor it
    • lbry portion
    • lbryum portion
  • header sync
    • basic implementation / initial pass
    • match current implementation + integration and/or unit tests (whichever ones don't get done as part of TDD)
  • subscribe to get address notifications
  • creating transactions
    • refactor Transaction class
    • round trip-able script parser / generator
      • input scripts (pub key, script hash / multi-sig)
      • output scripts (pub key, scripts hash) x (claim name, claim support, claim update)
    • refactor Transaction class II
      • use new script parser / generator
      • coin chooser
      • signing (single-sig and multi-sig)
  • move any code still being used from lbryum into lbrynet/wallet

Core Functionality I2

Core Functionality I3

  • fix orchstr8 to run with lbryumx
  • release generic parts of wallet as https://github.com/lbryio/torba
  • add regtest support to lbry-app (filed issue and provided branch with problematic code, lbryio/lbry-desktop#1526)
  • works with lbry-app (ALPHA release, pending lbrycrd and lbry-app regtest support)

Separate Issues Created After I3

  • store almost everything in SQLite (#1242)
  • SPV (#1221)
  • save and use channel certificates (#1223)
  • locking of addresses being used in transaction until the tx is written to block chain (#1222)
  • migrating wallets (#1224)
  • verify lbryumx responses (#1272)
  • plain API calls (such as resolve) (#1220)
  • performance test cases (#1219)
  • regression tests:
    • Allow update of claims even if they don't exist in file_list (#1165)
    • UTXOs remain stuck unconfirmed (#1192)

@jackrobison jackrobison force-pushed the dht-inlinecallbacks-refactor branch 3 times, most recently from f220124 to 9b82667 Mar 26, 2018

@lbryio lbryio deleted a comment from codacy-bot Mar 27, 2018

@lyoshenka lyoshenka added this to the Apr 9 (protocol) milestone Mar 27, 2018

@eukreign eukreign force-pushed the lbryum-refactor branch from f698141 to 827e2b0 Mar 27, 2018

@lbryio lbryio deleted a comment from codacy-bot Mar 27, 2018

@jackrobison jackrobison force-pushed the dht-inlinecallbacks-refactor branch 4 times, most recently from b8511b5 to 4928585 Mar 27, 2018

@eukreign eukreign force-pushed the lbryum-refactor branch from 827e2b0 to cd3324b Apr 8, 2018

@eukreign eukreign changed the base branch from dht-inlinecallbacks-refactor to master Apr 8, 2018

@lbryio lbryio deleted a comment from codacy-bot Apr 8, 2018

@lbryio lbryio deleted a comment from codacy-bot Apr 8, 2018

@eukreign eukreign force-pushed the lbryum-refactor branch from 5a4b05d to 9d82dee Apr 19, 2018

@lbryio lbryio deleted a comment from codacy-bot Apr 19, 2018

@lbryio lbryio deleted a comment from codacy-bot Apr 19, 2018

@lbryio lbryio deleted a comment from codacy-bot Apr 23, 2018

eukreign and others added some commits Aug 16, 2018

back up original lbryum wallet before migration
-use same directory and file name for new wallet as the old wallet
-catch error when trying to migrate an abandoned certificate
Revert "bytes/string"
This reverts commit 594dd61
DHT fixes from review and an attempt at removing hashing and equals (#…
…1370)

* use int to_bytes/from_bytes instead of struct
* fix ping queue bug and dht functional tests
* run functional tests on travis
* re-add contact comparison unit test
* dont need __ne__ if its just inverting __eq__ result
wallet_send command working (#1382)
* wallet_send command

@jackrobison jackrobison force-pushed the lbryum-refactor branch from 0623e61 to 5a1f551 Aug 24, 2018

@jackrobison jackrobison merged commit d2aaf7b into master Aug 24, 2018

2 of 4 checks passed

Linux (LBRY Daemon) TeamCity build failed
Details
OSX (LBRY Daemon) TeamCity build failed
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@eukreign eukreign deleted the lbryum-refactor branch Aug 24, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.