Merge #550: ledger: Support Bitcoin App 2.0

daeec1d ledger: Try legacy protocol when unable to sign multisigs (Andrew Chow)
711e6ba ledger: P2WSH display is also not supported (Andrew Chow)
6df9c1b ledger: Detect when a ledger is not in a Bitcoin app (Andrew Chow)
edbb7c0 ledger: document multisig limitation (Andrew Chow)
6b8381e tests: Use external keys for some multisigs (Andrew Chow)
4f223a5 test: Include xpubs in psbt (Andrew Chow)
f3f20e0 tests, ledger: Test multisigs (Andrew Chow)
ac66cbc tests: additional speculos automation things (Andrew Chow)
dffd014 ledger: Support multisigs (Andrew Chow)
ae2adb5 key: Add from_bytes to ExtendedKey (Andrew Chow)
ef1cf9d docs: Document Taproot signing (Andrew Chow)
3953000 tests: Don't rely on coin selection for signtx tests (Andrew Chow)
cb6b352 tests: Use --chain test more consistently (Andrew Chow)
30b42e4 ci: Test ledger legacy api (Andrew Chow)
3b0680d tests: Test Taproot on Ledger (Andrew Chow)
8536786 ledger: Remove btchip (Andrew Chow)
c154cf7 ledger: Use ledger_bitcoin rather than btchip (Andrew Chow)
b645164 ledger_bitcoin: Change LegacyClient.sign_psbt to return pubkeys too (Andrew Chow)
23baff5 ledger_bitcoin: Modifications to get_wallet_address for our legacy usage (Andrew Chow)
50e2d01 hwwclient: Add Chain as a parameter to constructor (Andrew Chow)
5aad268 key: Add helpers for determing addrtype and standard paths (Andrew Chow)
26a1ccc ledger_bitcoin: Optionally allow HID path in TransportClient (Andrew Chow)
57d5c2f ledgercomm: Add the option to pass in HID path (Andrew Chow)
15e5427 ledgercomm: Use relative imports of itself (Andrew Chow)
88235d6 Include ledgercomm library (Andrew Chow)
b709028 ledger_bitcoin: Fix sign_psbt for our implementation of PSBTv2 (Andrew Chow)
e34f6fd Add ledger_bitcoin library (Andrew Chow)

Pull request description:

  The 2.0 version of the Ledger Bitcoin app uses a completely new protocol which utilizes PSBTv2 and descriptors(ish). Additionally, this new version supports signing Taproot inputs.

  This PR adds support for using the new protocol when the app major version number is detected to be greater than or equal to 2. Ledger has provided a python library in https://github.com/LedgerHQ/app-bitcoin-new/tree/master/bitcoin_client. Given that many components of this library are copied from HWI itself, it doesn't make sense to include the entire thing. Additionally, as we already have access to the device via btchip-python, there is no need to use any of the transport things. So instead, parts of the library are copied to `hwilib/devices/ledgerbitcoin2` and modifications are made to use relative imports to common HWI components. Our copy of btchip-python is slightly modified to work with the new version as well, particularly with returning the status code rather than consuming it when receiving APDU responses.

  As the new protocol uses PSBTv2, this PR also requires #549 for PSBTv2 itself.

  As v2.0.0 supports Taproot, this branch is based on #544 so that Taproot can be supported.

ACKs for top commit:
  Sjors:
    re-ACK daeec1d

Tree-SHA512: 8f956622b439afe15be0c169e02d852d2ec221af9c75a70819956f78a9a8c4c3a7b555d0de4f3fd4df61bb862abda68505c5c2297a13f5c310e200d123cd609b

.cirrus.yml

@@ -79,6 +79,20 @@ device_matrix_template: &DEVICE_MATRIX_TEMPLATE
     sim_install_script:
       - poetry run pip install construct flask-restful jsonschema mnemonic pyelftools pillow requests
       - pip install construct flask-restful jsonschema mnemonic pyelftools pillow requests
+  - env:
+      DEVICE: --ledger-legacy
+    depends_on:
+      - Ledger Sim Builder
+      - dist_builder
+      - bitcoind_builder
+    fetch_sim_script:
+      - wget -nv "https://api.cirrus-ci.com/v1/artifact/build/$CIRRUS_BUILD_ID/Ledger Sim Builder/sim/speculos.tar.gz"
+      - tar -xvf "speculos.tar.gz"
+      - wget -nv "https://api.cirrus-ci.com/v1/artifact/build/$CIRRUS_BUILD_ID/bitcoind_builder/bitcoin/bitcoin.tar.gz"
+      - tar -xvf "bitcoin.tar.gz"
+    sim_install_script:
+      - poetry run pip install construct flask-restful jsonschema mnemonic pyelftools pillow requests
+      - pip install construct flask-restful jsonschema mnemonic pyelftools pillow requests
   - env:
       DEVICE: --keepkey
     depends_on:

.flake8

@@ -1,4 +1,4 @@
 [flake8]
-exclude = *.pyc,__pycache__,hwilib/devices/btchip/,hwilib/devices/ckcc/,hwilib/devices/jadepy/,hwilib/devices/trezorlib/,test/work/,hwilib/ui
+exclude = *.pyc,__pycache__,hwilib/devices/ledger_bitcoin/,hwilib/devices/btchip,hwilib/devices/ckcc/,hwilib/devices/jadepy/,hwilib/devices/trezorlib/,test/work/,hwilib/ui
 ignore = E261,E302,E305,E501,E722,W5
 per-file-ignores = setup.py:E122

docs/devices/index.rst

@@ -43,6 +43,8 @@ The table below lists what devices and features are supported for each device.
 +------------------------------------+---------------+---------------+------------+----------------+----------+----------+---------+----------+------------------+
 | Bare Multisig Inputs               | ✓             | ✓             | ―          | ―              | ✓        | ―        | ―       | ―        | ✓                |
 +------------------------------------+---------------+---------------+------------+----------------+----------+----------+---------+----------+------------------+
+| Taproot Inputs                     | ✓*            | ✓*            | ✓*         | ✓*             |  ―       | ―        | ―       | ―        | ―                |
++------------------------------------+---------------+---------------+------------+----------------+----------+----------+---------+----------+------------------+
 | Arbitrary scriptPubKey Inputs      | ✓             | ✓             | ―          | ―              | ✓        | ―        | ―       | ―        | ✓                |
 +------------------------------------+---------------+---------------+------------+----------------+----------+----------+---------+----------+------------------+
 | Arbitrary redeemScript Inputs      | ✓             | ✓             | ―          | ―              | ✓        | ―        | ―       | ―        | ✓                |
@@ -56,6 +58,8 @@ The table below lists what devices and features are supported for each device.
 | Display on device screen           | ✓             | ✓             | ✓          | ✓              | ―        | ✓        | ✓       | ✓        | ✓                |
 +------------------------------------+---------------+---------------+------------+----------------+----------+----------+---------+----------+------------------+
 
+\* There are some caveats. See the `sign_tx` for these devices.
+
 Support Policy
 ================
 

hwilib/_cli.py

@@ -277,12 +277,12 @@ def process_commands(cli_args: List[str]) -> Any:
 
     # Auto detect if we are using fingerprint or type to identify device
     if args.fingerprint or (args.device_type and not args.device_path):
-        client = find_device(args.password, args.device_type, args.fingerprint, args.expert)
+        client = find_device(args.password, args.device_type, args.fingerprint, args.expert, args.chain)
         if not client:
             return {'error': 'Could not find device with specified fingerprint', 'code': DEVICE_CONN_ERROR}
     elif args.device_type and args.device_path:
         with handle_errors(result=result, code=DEVICE_CONN_ERROR):
-            client = get_client(device_type, device_path, password, args.expert)
+            client = get_client(device_type, device_path, password, args.expert, args.chain)
         if 'error' in result:
             return result
     else:
@@ -291,8 +291,6 @@ def process_commands(cli_args: List[str]) -> Any:
     if client is None:
         return {"error": "Unable to communicated with device", "code": UNKNOWN_ERROR}
 
-    client.chain = args.chain
-
     # Do the commands
     with handle_errors(result=result, debug=args.debug):
         result = args.func(args, client)

hwilib/_gui.py

@@ -356,8 +356,7 @@ def get_client_and_device_info(self, index):
 
         # Get the client
         self.device_info = self.devices[index - 1]
-        self.client = commands.get_client(self.device_info['model'], self.device_info['path'], self.passphrase)
-        self.client.chain = self.chain
+        self.client = commands.get_client(self.device_info['model'], self.device_info['path'], self.passphrase, self.chain)
 
         if self.device_info['type'] == 'bitbox02':
             self.client.set_noise_config(BitBox02NoiseConfig())

hwilib/commands.py

@@ -52,6 +52,7 @@
 from .devices import __all__ as all_devs
 from .common import (
     AddressType,
+    Chain,
 )
 from .hwwclient import HardwareWalletClient
 from .psbt import PSBT
@@ -70,14 +71,15 @@
 
 
 # Get the client for the device
-def get_client(device_type: str, device_path: str, password: str = "", expert: bool = False) -> Optional[HardwareWalletClient]:
+def get_client(device_type: str, device_path: str, password: str = "", expert: bool = False, chain: Chain = Chain.MAIN) -> Optional[HardwareWalletClient]:
     """
     Returns a HardwareWalletClient for the given device type at the device path
 
     :param device_type: The type of device
     :param device_path: The path specifying where the device can be accessed as returned by :func:`~enumerate`
     :param password: The password to use for this device
     :param expert: Whether the device should be opened in expert mode (prints more information for some commands)
+    :param chain: The Chain this client will be using
     :return: A :class:`~hwilib.hwwclient.HardwareWalletClient` to interact with the device
     :raises: UnknownDeviceError: if the device type is not known by HWI
     """
@@ -90,7 +92,7 @@ def get_client(device_type: str, device_path: str, password: str = "", expert: b
     try:
         imported_dev = importlib.import_module('.devices.' + module, __package__)
         client_constructor = getattr(imported_dev, class_name + 'Client')
-        client = client_constructor(device_path, password, expert)
+        client = client_constructor(device_path, password, expert, chain)
     except ImportError:
         if client:
             client.close()
@@ -126,6 +128,7 @@ def find_device(
     device_type: Optional[str] = None,
     fingerprint: Optional[str] = None,
     expert: bool = False,
+    chain: Chain = Chain.MAIN,
 ) -> Optional[HardwareWalletClient]:
     """
     Find a device from the device type or fingerprint and get a client to access it.
@@ -138,6 +141,7 @@ def find_device(
         The client returned will have a master public key fingerprint matching this.
         If not provided, device_type must be provided.
     :param expert: Whether the device should be opened in expert mode (enables additional output for some actions)
+    :param chain: The Chain this client will be using
     :return: A client to interact with the found device
     """
 
@@ -149,7 +153,7 @@ def find_device(
         try:
             assert isinstance(d["type"], str)
             assert isinstance(d["path"], str)
-            client = get_client(d['type'], d['path'], password, expert)
+            client = get_client(d['type'], d['path'], password, expert, chain)
             if client is None:
                 raise Exception()
 

hwilib/devices/bitbox02.py

@@ -258,11 +258,11 @@ def func(*args, **kwargs):  # type: ignore
 
 # This class extends the HardwareWalletClient for BitBox02 specific things
 class Bitbox02Client(HardwareWalletClient):
-    def __init__(self, path: str, password: str = "", expert: bool = False) -> None:
+    def __init__(self, path: str, password: str = "", expert: bool = False, chain: Chain = Chain.MAIN) -> None:
         """
         Initializes a new BitBox02 client instance.
         """
-        super().__init__(path, password=password, expert=expert)
+        super().__init__(path, password=password, expert=expert, chain=chain)
         if password:
             raise BadArgumentError(
                 "The BitBox02 does not accept a passphrase from the host. Please enable the passphrase option and enter the passphrase on the device during unlock."

hwilib/devices/coldcard.py

@@ -89,8 +89,8 @@ def func(*args: Any, **kwargs: Any) -> Any:
 # This class extends the HardwareWalletClient for ColdCard specific things
 class ColdcardClient(HardwareWalletClient):
 
-    def __init__(self, path: str, password: str = "", expert: bool = False) -> None:
-        super(ColdcardClient, self).__init__(path, password, expert)
+    def __init__(self, path: str, password: str = "", expert: bool = False, chain: Chain = Chain.MAIN) -> None:
+        super(ColdcardClient, self).__init__(path, password, expert, chain)
         # Simulator hard coded pipe socket
         if path == CC_SIMULATOR_SOCK:
             self.device = ColdcardDevice(sn=path)

hwilib/devices/digitalbitbox.py

@@ -345,15 +345,15 @@ def format_backup_filename(name: str) -> str:
 # This class extends the HardwareWalletClient for Digital Bitbox specific things
 class DigitalbitboxClient(HardwareWalletClient):
 
-    def __init__(self, path: str, password: str, expert: bool = False) -> None:
+    def __init__(self, path: str, password: str, expert: bool = False, chain: Chain = Chain.MAIN) -> None:
         """
         The `DigitalbitboxClient` is a `HardwareWalletClient` for interacting with BitBox01 devices (previously known as the Digital BitBox).
 
         :param path: Path to the device as given by `enumerate`
         :param password: The password required to communicate with the device. Must be provided.
         :param expert: Whether to be in expert mode and return additional information.
         """
-        super(DigitalbitboxClient, self).__init__(path, password, expert)
+        super(DigitalbitboxClient, self).__init__(path, password, expert, chain)
         if not password:
             raise NoPasswordError('Password must be supplied for digital BitBox')
         if path.startswith('udp:'):

hwilib/devices/jade.py

@@ -125,8 +125,8 @@ def _get_multisig_name(type: str, threshold: int, signers: List[Tuple[bytes, Seq
         hash_summary = sha256(summary.encode()).hex()
         return 'hwi' + hash_summary[:12]
 
-    def __init__(self, path: str, password: str = '', expert: bool = False, timeout: Optional[int] = None) -> None:
-        super(JadeClient, self).__init__(path, password, expert)
+    def __init__(self, path: str, password: str = '', expert: bool = False, chain: Chain = Chain.MAIN, timeout: Optional[int] = None) -> None:
+        super(JadeClient, self).__init__(path, password, expert, chain)
         self.jade = JadeAPI.create_serial(path, timeout=timeout)
         self.jade.connect()
 

hwilib/devices/keepkey.py

@@ -3,6 +3,7 @@
 *******
 """
 
+from ..common import Chain
 from ..errors import (
     DEVICE_NOT_INITIALIZED,
     DeviceNotReadyError,
@@ -140,7 +141,7 @@ def __init__(
 
 
 class KeepkeyClient(TrezorClient):
-    def __init__(self, path: str, password: str = "", expert: bool = False) -> None:
+    def __init__(self, path: str, password: str = "", expert: bool = False, chain: Chain = Chain.MAIN) -> None:
         """
         The `KeepkeyClient` is a `HardwareWalletClient` for interacting with the Keepkey.
 
@@ -158,7 +159,7 @@ def __init__(self, path: str, password: str = "", expert: bool = False) -> None:
         if path.startswith("udp"):
             model.default_mapping.register(KeepkeyDebugLinkState)
 
-        super(KeepkeyClient, self).__init__(path, password, expert, KEEPKEY_HID_IDS, KEEPKEY_WEBUSB_IDS, KEEPKEY_SIMULATOR_PATH, model)
+        super(KeepkeyClient, self).__init__(path, password, expert, chain, KEEPKEY_HID_IDS, KEEPKEY_WEBUSB_IDS, KEEPKEY_SIMULATOR_PATH, model)
         self.type = 'Keepkey'
 
     def can_sign_taproot(self) -> bool: