Features/headersv integration (#858)

* HeaderSV API Integration

- Replaced ElectrumX header APIs with new ElectrumSV-Reference-Server APIs.
- Reorgs are handled.
- Uses binary header APIs wherever practical to do so.
- Removed most dead code related to ElectrumX which helped reduce cognitive load.
- The proxy tab of network_dialogue.py is deleted but the BlockchainTab and NodesListWidget code has been commented out to defer the work for later (to represent the HeaderSV chain tips instead of ElectrumX sessions).
- Moved GeneralAPIError exceptions to their own file to resolve circular import issues.
- UI correctly shows server status as either being behind x number of blocks, Connected or main server pending.

PR review changes

- Also note: aiohttp's testing APIs have breaking changes:
'test_client' was renamed to 'aiohttp_client' as per their documentation:
https://docs.aiohttp.org/en/stable/testing.html
And the types and input arguments have also changed.

Review changes. Including:

- Support multiple electrumsv reference server connections & an initial reorg check on startup
- Move async db functions out of the AsynchronousFunctions class to the top level
- Fixing bugs and testing

Run `startup_reorg_check_async` as a background task awaiting initial synchronization of headers on the main server
Move the concept of "main server" from Network daemon to each Wallet instance
- BlockchainScanner uses the main server & master token for this main server
- Each wallet's chain state will be relative to its main server even if it is not on the longest chain of all forks.

Database migration - add TransactionProofs table

- The pathways for insertion to this table are Wallet._obtain_merkle_proofs_worker_async, Wallet._obtain_transactions_worker_async and wait_for_merkle_proofs_and_double_spends (not yet in use) via mAPI callbacks
- All proofs from all chains should be inserted here (i.e. including orphaned proofs). They can be pruned when the proof on the main server chain is buried by sufficient proof of work.

Reorg handling updates taking into account pending mAPI callbacks

Latest Header management integration work & Fixes

* Remaining PR review changes

* Minor updates after review (HeaderSV related changes)

electrumsv/app_state.py

@@ -211,6 +211,7 @@ def get_amount_and_units(self, amount: int) -> Tuple[str, str]:
             fiat_text = ''
         return bitcoin_text, fiat_text
 
+    # TODO(1.4.0) Networking. Replace with non-electrumx equivalent
     def electrumx_message_size_limit(self) -> int:
         return max(0,
             self.config.get_explicit_type(int, 'electrumx_message_size_limit',

electrumsv/blockchain_scanner.py

@@ -42,18 +42,16 @@
 import concurrent.futures
 from dataclasses import dataclass, field
 from enum import IntEnum
-import random
 from typing import Callable, cast, Dict, List, NamedTuple, Optional, Sequence, TYPE_CHECKING
 
 from bitcoinx import bip32_key_from_string, BIP32PublicKey, PublicKey
 
 from .app_state import app_state
 from .constants import (ACCOUNT_SCRIPT_TYPES, AccountType, CHANGE_SUBPATH, DerivationType,
-    DerivationPath, NetworkServerType, RECEIVING_SUBPATH, ScriptType, ServerCapability)
+    DerivationPath, RECEIVING_SUBPATH, ScriptType)
 from .exceptions import UnsupportedAccountTypeError
 from .i18n import _
 from .logs import logs
-from .network_support.api_server import select_servers
 from .network_support.general_api import post_restoration_filter_request_binary, \
     RestorationFilterRequest, RestorationFilterResult, unpack_binary_restoration_entry
 from .wallet import AbstractAccount
@@ -189,17 +187,29 @@ async def search_entries(self, entries: List[SearchEntry]) -> None:
             means that the connection was closed mid-transmission.
         Raises `ServerConnectionError` if the remote computer cannot be connected to.
         """
-        all_candidates = self._account._wallet.get_servers_for_account(
-            self._account, NetworkServerType.GENERAL)
-        restoration_candidates = select_servers(ServerCapability.RESTORATION, all_candidates)
-        if not len(restoration_candidates):
-            raise PushDataSearchError(_("No servers available."))
 
-        # TODO(1.4.0) Networking. Standardised server selection / endpoint url resolution.
-        candidate = random.choice(restoration_candidates)
-        assert candidate.api_server is not None
+        # TODO(1.4.0) Networking. Discuss this with Roger - the fact that we want to pin to the
+        #    main server for the wallet for consistent chain state.
+
+        # all_candidates = self._account._wallet.get_servers_for_account(
+        #     self._account, NetworkServerType.GENERAL)
+        # restoration_candidates = select_servers(ServerCapability.RESTORATION, all_candidates)
+        # if not len(restoration_candidates):
+        #     raise PushDataSearchError(_("No servers available."))
+
+        # # TODO(1.4.0) Networking. Standardised server selection / endpoint url resolution.
+        # candidate = random.choice(restoration_candidates)
+        # assert candidate.api_server is not None
+
+        # url = f"{candidate.api_server.url}api/v1/restoration/search"
+
+        main_server = self._account.get_wallet().main_server
+        if not main_server:
+            raise PushDataSearchError(_("No servers available."))
 
-        url = f"{candidate.api_server.url}api/v1/restoration/search"
+        url = main_server._state.server.url
+        url = url if url.endswith("/") else url +"/"
+        url = f"{url}api/v1/restoration/search"
 
         # These are the pushdata hashes that have been passed along.
         entry_mapping: Dict[bytes, SearchEntry] = { entry.item_hash: entry for entry in entries }
@@ -208,7 +218,11 @@ async def search_entries(self, entries: List[SearchEntry]) -> None:
                 entry.item_hash.hex() for entry in entries
             ]
         }
-        async for payload_bytes in post_restoration_filter_request_binary(url, request_data):
+        credential_id = main_server._state.credential_id
+        assert credential_id is not None
+        master_token = app_state.credentials.get_indefinite_credential(credential_id)
+        async for payload_bytes in post_restoration_filter_request_binary(url, request_data,
+                master_token):
             filter_result = unpack_binary_restoration_entry(payload_bytes)
             search_entry = entry_mapping[filter_result.push_data_hash]
             self.record_match_for_entry(filter_result, search_entry)

electrumsv/constants.py

@@ -412,12 +412,12 @@ class NetworkEventNames(Enum):
     BANNER = "banner"
     SESSIONS = "sessions"
     MAIN_CHAIN = "main_chain"
+    NEW_TIP = "new_tip"
 
 
 class NetworkServerType(IntEnum):
-    ELECTRUMX = 1
-    MERCHANT_API = 2
-    GENERAL = 3
+    MERCHANT_API = 1
+    GENERAL = 2
 
 
 API_SERVER_TYPES = { NetworkServerType.MERCHANT_API, NetworkServerType.GENERAL }
@@ -528,7 +528,7 @@ class PendingHeaderWorkKind(IntEnum):
     # We have a merkle proof for a transaction but no synchronised chain including it.
     MERKLE_PROOF = 1
     # A new header arrived.
-    NEW_HEADER = 2
+    NEW_TIP = 2
 
 
 NO_BLOCK_HASH = bytes(32)

electrumsv/data/api_servers_regtest.json

@@ -24,5 +24,59 @@
         "url": "http://127.0.0.1:5050/mapi/",
         "api_key_required": false,
         "static_data_date": "2021-05-13T04:45:05+0000"
+    },
+    {
+        "id": 21003,
+        "url": "http://127.0.0.1:47123/",
+        "type": "GENERAL",
+        "capabilities": [
+            "HEADERS",
+            "PEER_CHANNELS",
+            "RESTORATION",
+            "MERKLE_PROOF_REQUEST",
+            "TRANSACTION_REQUEST",
+            "OUTPUT_SPENDS"
+        ],
+        "enabled_for_all_wallets": true,
+        "api_key_supported": false,
+        "last_try": 0.0,
+        "last_good": 0.0,
+        "static_data_date": "2021-12-15T04:45:05+0000"
+    },
+    {
+        "id": 21004,
+        "url": "http://127.0.0.1:47122/",
+        "type": "GENERAL",
+        "capabilities": [
+            "HEADERS",
+            "PEER_CHANNELS",
+            "RESTORATION",
+            "MERKLE_PROOF_REQUEST",
+            "TRANSACTION_REQUEST",
+            "OUTPUT_SPENDS"
+        ],
+        "enabled_for_all_wallets": true,
+        "api_key_supported": false,
+        "last_try": 0.0,
+        "last_good": 0.0,
+        "static_data_date": "2021-12-15T04:45:05+0000"
+    },
+    {
+        "id": 21005,
+        "url": "http://127.0.0.1:47121/",
+        "type": "GENERAL",
+        "capabilities": [
+            "HEADERS",
+            "PEER_CHANNELS",
+            "RESTORATION",
+            "MERKLE_PROOF_REQUEST",
+            "TRANSACTION_REQUEST",
+            "OUTPUT_SPENDS"
+        ],
+        "enabled_for_all_wallets": true,
+        "api_key_supported": false,
+        "last_try": 0.0,
+        "last_good": 0.0,
+        "static_data_date": "2021-12-15T04:45:05+0000"
     }
 ]

electrumsv/gui/qt/blockchain_scan_dialog.py

@@ -59,7 +59,7 @@
     PushDataHashHandler, PushDataSearchError, SearchKeyEnumerator
 from ...exceptions import ServerConnectionError
 from ...i18n import _
-from ...network_support.general_api import FilterResponseIncompleteError, FilterResponseInvalidError
+from ...network_support.exceptions import FilterResponseIncompleteError, FilterResponseInvalidError
 from ...logs import logs
 from ...wallet import Wallet
 from ...wallet_database.types import TransactionLinkState, TransactionRow

electrumsv/gui/qt/main_window.py

@@ -59,14 +59,12 @@
 import electrumsv
 from ... import bitcoin, commands, paymentrequest, qrscanner, util
 from ...app_state import app_state
-from ...bitcoin import (COIN, is_address_valid, address_from_string,
-    script_template_to_string, TSCMerkleProof)
+from ...bitcoin import (address_from_string, COIN, script_template_to_string, TSCMerkleProof)
 from ...constants import (AccountType, CredentialPolicyFlag, DATABASE_EXT, NetworkEventNames,
     ScriptType, TransactionImportFlag, TransactionOutputFlag, TxFlags, WalletEvent)
 from ...exceptions import UserCancelled
 from ...i18n import _
 from ...logs import logs
-from ...network import broadcast_failure_reason
 from ...network_support.api_server import broadcast_transaction
 from ...network_support.mapi import BroadcastResponse
 from ...networks import Net
@@ -230,9 +228,7 @@ def __init__(self, wallet: Wallet):
             # partials, lambdas or methods of subobjects.  Hence...
             self.network.register_callback(self.on_network, [ NetworkEventNames.GENERIC_UPDATE,
                 NetworkEventNames.GENERIC_STATUS, NetworkEventNames.BANNER ])
-            # set initial message
-            if self.network.main_server:
-                self.console.showMessage(self.network.main_server.state.banner)
+
             self.network.register_callback(self._on_exchange_rate_quotes,
                 [ NetworkEventNames.EXCHANGE_RATE_QUOTES ])
             self.network.register_callback(self._on_historical_exchange_rates,
@@ -544,10 +540,6 @@ def on_network_qt(self, event: NetworkEventNames, args: Any=None) -> None:
         # Handle a network message in the GUI thread
         if event == NetworkEventNames.GENERIC_STATUS:
             self.update_status_bar()
-        elif event == NetworkEventNames.BANNER:
-            assert self.network is not None
-            assert self.network.main_server is not None
-            self.console.showMessage(self.network.main_server.state.banner)
         else:
             self._logger.debug("unexpected network_qt signal event='%s' args='%s'", event, args)
 
@@ -775,7 +767,6 @@ def add_toggle_action(view_menu: QMenu, tab: QWidget) -> None:
             QKeySequence.HelpContents)
         help_menu.addAction(_("&Report Bug"), self.show_report_bug)
         help_menu.addSeparator()
-        help_menu.addAction(_("&Donate to server"), self.donate_to_server)
 
         self.setMenuBar(menubar)
 
@@ -982,17 +973,6 @@ def new_payment(self) -> None:
     def has_connected_main_server(self) -> bool:
         return self.network is not None and self.network.is_connected()
 
-    def donate_to_server(self) -> None:
-        assert self.network is not None
-        server = self.network.main_server
-        assert server is not None
-        addr = server.state.donation_address
-        if is_address_valid(addr):
-            self.pay_to_URI(web.create_URI(addr, 0, _('Donation for {}').format(server.host)))
-        else:
-            self.show_error(_('The server {} has not provided a valid donation address')
-                            .format(server))
-
     def show_about(self) -> None:
         QMessageBox.about(self, "ElectrumSV",
             _("Version")+" %s" % PACKAGE_VERSION + "\n\n" +
@@ -1200,19 +1180,23 @@ def _update_network_status(self) -> None:
                 text = _("Synchronizing...")
                 text += f' {response_count:,d}/{request_count:,d}'
             else:
-                server_height = self.network.get_server_height()
-                if server_height == 0:
+                if self._wallet.main_server is not None:
+                    server_chain_tip = self._wallet.main_server.tip
+                    server_height = server_chain_tip.height if server_chain_tip else 0
+                    server_lag = self.network.get_local_height() - server_height
+                    if server_height == 0:
+                        text = _("Main server pending")
+                    elif server_lag > 1:
+                        text = _("Server {} blocks behind").format(server_lag)
+                    else:
+                        text = _("Connected")
+                else:
                     # This is shown when for instance, there is a forced main server setting and
                     # the main server is offline. It might also be used on first start up before
                     # the headers are synced (?).
                     text = _("Main server pending")
                     tooltip_text = _("You are not currently connected to a valid main server.")
-                else:
-                    server_lag = self.network.get_local_height() - server_height
-                    if server_lag > 1:
-                        text = _("Server {} blocks behind").format(server_lag)
-                    else:
-                        text = _("Connected")
+
         self._status_bar.set_network_status(text, tooltip_text)
 
     def update_tabs(self, *args: Any) -> None:
@@ -1495,11 +1479,10 @@ def on_done(future: concurrent.futures.Future[Optional[str]]) -> None:
                     "Transactions tab and can be rebroadcast from there."), )
             except Exception as exception:
                 self._logger.exception('unhandled exception broadcasting transaction')
-                reason = broadcast_failure_reason(exception)
+                reason = str(exception)
                 d = UntrustedMessageDialog(
                     window, _("Transaction Broadcast Error"),
-                    _("Your transaction was not sent: ") + reason +".",
-                    exception)
+                    _("Your transaction was not sent: ") + reason +".", exception)
                 d.exec()
             else:
                 if account and tx_id: