Support Lightning Network

[ysangkok]

Today Thomas and I discussed how to integrate Lightning into Electrum.

One options would be to have the following setup:

• Normal ElectrumX
• Electrum Lightning Hub, based on an existing Lightning implementation, but supporting a new protocol for talking with the Electrum Client
• Transaction watcher would gain the capability to watch for send breach remedy transactions
• A btcd instance would be needed in addition if lnd is used

The Electrum Client would not need to do Lightning routing, since it would always talk directly to the Lightning Hub over a custom protocol. Maybe the hub could be implemented as talking to something like lncli, so that it wouldn't be necessary to fork.

The Electrum Client would be an independent Lightning node, so the server would need to hold money but users should fund their own channels with the server. So a client would only be able to receive after opening a channel itself.

Please, everyone, does this sound realistic? Many you can help out fleshing out the details.

[ysangkok]

@Roasbeef says on #lightning-network on bitcoincore slack: "you just need to watch for a particualr utxo being spent, once the utxo is spent, one needs to decode the state hint taht's encoded in the sequence+locktime, if this number is less that the current height of the remote party's commitment chain, then it's a contract breach and you should sweep all le funds"

[ecdsa]

thanks @ysangkok
the electrum server does not let you watch a utxo, but an output script (it usually is an address, but it does not have to). the client gets a notification everytime the address history has changed.

[ecdsa]

note: to learn how electrum watches an address, see the file scripts/watch_address
it is a simple example, that uses blockchain.address.subscribe
it returns a hash of the address history, as returned by blockchain.address.get_history
the server also has blockchain.script_hash.subscribe RPC, to watch a script that is not an address

[ysangkok]

Turns out that LND does not currently implement the standard Lightning protocol at all. Considering that, and the fact that Roasbeef mentioned it would be a lot of work to re-implement Lightning in Python, even if it would not include routing, noise and onion parts, I am considering if there is a simpler way.

Roasbeef mentioned using go-py and call into lnd from Electrum. But this would create a runtime dependency on Go, which would make it hard to use on e.g. Android (it uses different ABI's than normal Linux). Even if we used c-lightning, we'd have this problem.

So I am considering whether it would be realistic to implement LND's WalletController interface to make it call into Electrum, over an RPC port that LND would have open and that Electrum could connect to. This would probably initially require one LND instance per Electrum instance, but that could hopefully be worked around later. That interface was designed with the goal of separating signing from other operations. As I see it, it fits with the Electrum ideology since it allows the user to keep his private keys, and doesn't require us to reimplement Lightning. It would require making this new custom protocol and Electrum support e.g. the SignDescriptor interface used in WalletController: https://github.com/lightningnetwork/lnd/blob/2e6800e1ed65c4c2d5b681612976834aed3d822c/lnwallet/btcwallet/signer.go#L93

What do you think? Roasbeef mentioned to me on #ln-testers on Bitcoin Core Slack that he would explain his vision the next few days.

[ecdsa]

I too believe we should avoid reimplemening the lightning protocol.
go-py is not a solution, the client must stay pure python.

We need a custom protocol between the electrum client an a lightning hub running LND.
I do not know about LND's WalletController interface. From what you write it seems to be a good solution. If that interface is too heavy, we can also define our custom protocol between the electrum client and a python daemon that runs on the hub and forwards commands to LND.

[ecdsa]

Note: It is probably better to have a python daemon on the hub, that forwards the requests from electrum clients to LND. If the developers of LND decide to change their RPCs, we do not want to have to upgrade all the electrum clients.

[ysangkok]

I have an initial test environment up at https://github.com/ysangkok/electrum-lightning-test

It contains patches for btcd, lnd, electrumx, electrum, bitcoin core and btcwallet.

For most projects, it adds simnet support and disables TLS so that I can snoop on the test traffic and I don't need to fiddle with certificates.

For lnd, it adds a websocket server at websocket port 19283 that sends the master seed and all signing requests, awaiting the signed transactions.

The electrum-lightning-hub connects to this port, saves the key and any outstanding signing request, and listens on websocket port 8765.

This allows for Electrum being offline since the signing request queue is buffered in the hub.

Next, I was planning on making Electrum derive the right private keys, decode the signing requests (they contain the pubkey to be used), and sign and return the transactions. Then it should theoretically work. After that, the hub needs to manage the lifecycle of lnd instances and support authentication.

The interface implemented by Electrum would be the Signer interface. The Signer is passed SignDescriptors. Electrum would also have RPC access to lnd.

[ysangkok]

I got SignDescriptors decoded successfully.

First, I am working on ComputeInputScript, and now I am trying to figure out how to create input scripts with given hashPrevouts/hashSequence/hashOutputs (BIP 143) and the Electrum code base.

Working through https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki#native-p2wsh

[fresheneesz]

Are you guys aware of https://github.com/mit-dci/lit ? Its also in Go, but it doesn't require running a full node (I believe it uses SPV). Might be useful?

[ysangkok]

Yeah, I am aware if it. LND has Neutrino, doesn't require a full node either.

Currently I am working on porting DeriveRevocationPrivKey and friends to Python.

[neocogent]

Will this be implemented as a plugin or built in? Would there be a tab added that lists open channels and state/balance/history, and/or would txs be integrated into the History tab?

I'd be happy to help with some part of integrating this. It would be very nice to be able to use LN natively in Electrum as long as it doesn't alter Electrum still being a "light" client.

[dabura667]

It would be nice to add new functionality to the server side to assist Lightning Client peer discovery. Also if there was a simple way for a server admin to deposit bitcoins, set a preferred fee rate to charge people, etc. Electrum would be in a unique position to help bootstrap the Lightning Network...

Since ElectrumX and Electrum-server are based on bitcoin core the LN backend would also need to be Bitcoin Core based imo...

I would love to hear @kyuupichan and his thoughts on this.

[neocogent]

I thought I read that btcd was RPC compatible with bitcoind (or to some extent if not perfectly). Shouldn't it be possible to run lnd over bitcoind? Or does it hook in directly in ways that bypass the RPC interface?

I know btcd and lnd are both Go based so maybe there is closer interfacing that would prevent this. It's just with all electrum servers already running on bitcoind it seems like a hefty change to expect operators to move over to btcd. Personally I'd rather see a Python version of lnd since that can be leveraged in many native Python projects but I know that's a big deal. No other LN implementation in Python?

c-lightning has an RPC interface - couldn't a fairly simple python module wrap that? It appears to run on bitcoind.

[ysangkok]

@neocogent Regarding using the RPC interface for c-lightning: We want a pure Python solution. A non-Python solution would be a heftier requirement than having server operators install extra software. And LND can run in Neutrino mode, so btcd is not needed. Also, if you wrap the c-lightning RPC interface, who does the signing? Most Lightning implementations have their own wallets, but we want Electrum to stay in control of the keys.

Regarding the GUI, I don't know how it will look, and I don't plan to start on it soon, there is still much work to be done, the GUI can be made when this is ready for testing.

@dabura667 What makes you think peer discovery will be a problem? Let's focus on actual issues encountered so far.

[dabura667]

It's not a problem.

It doesn't exist currently and needs to be implemented somehow otherwise no one can open any channels with anyone.

Unless you were under the assumption that Electrum lightning users would only open channels directly with Electrum servers... in which case I disagree.

A sub-optimal solution would be for Electrum servers to cache lists of currently connected peers and return them to the client when queried.

[ysangkok]

The major Lightning implementations have this same problem and will solve it before Electrum deploys Lightning. Electrum users will be able to Lightning will all Lightning users on their coin.

[dabura667]

Electrum users will be able to Lightning will all Lightning users on their coin.

Assuming the "other Lightning implementations" include one in pure python.

Just so we're on the same page here:

You: Let's join ElectrumX and Electrum Client with something like lnd/lncli and make Electrum support Lightning.

Thomas: To keep everything pure python, "We need a custom protocol between the electrum client an a lightning hub running LND."

You: I made an Electrum-Lightning-Hub that queues a lot of stuff for the clients so they can be offline.

Me: Rather than relying on the hub for everything, Electrum-Lightning-Hub (Electrum Server) should leave the peer selection to the client and default to clients connecting directly to create channels.

You: Peer discovery is not a problem.

Me: Never said it was a problem. It doesn't exist in your current design. It should.

You: lnd et al will take care of that for us.

Me (right now): On the client side? Didn't Thomas just get done saying "pure python"? Is there a pure python lnd/lncli alternative that can run based on SPV (Currently clients only hold blockheaders)?

[ysangkok]

Pure Python only goes for the client. LND and Electrum won't be speaking Lightning, but rather some custom interface probably resembling lnwallet's interface to the rest of LND. The hub is just for managing LND instances and currently buffering signature requests.

If I understand you correctly, you suggest that Electrum implements all the P2P stuff of Lightning. That is a lot of work. I also think it would be nicest to have a full Lightning implementation in Electrum. But if you look at how much effort it was to develop LND, I don't think we can afford to do that, with the manpower we have.

Having the server assist in peer discovery makes sense to me only if Electrum were an actual Lightning peer.

There won't be a separate network though. The LND instances running for Electrum users can be equal to other LND nodes, but anything like autopilot of course wouldn't work. So it probably wouldn't make sense for most of them to route payments.

[ecdsa]

@dabura667 implementing the full lightning node is out of scope of the moment.
I asked @ysangkok to implement channel signing in the client, for a server running a lnd node.
We'll see later if we add more stuff client side.

[neocogent]

@ysangkok
I did a little looking at lnd and c-lightning details after my last post. I see now it's pretty complicated to figure out how to use either because they both act as wallets and want to generate and sign txs according to the LN standards (BOLTs). I thought they would have written it as a layer over a wallet but not the case from what I saw. It pretty much needs to be broken into 2 parts to work - server and client wallet.

So it sounds like you're working out a way to let electrum client do the signing and address management, which is what would be desired rather than electrum server doing that. It ends up being complicated.

I was looking at c-lightning from a different angle and I am not sure if it is feasible. I thought that c-lightning could be wrapped as a python module (much like other aes or other c-libraries get wrapped to be native python modules), and then an Electrum client would actually use that as it's interface to LN directly (also giving the module a root key derived from xprv to base on). I thought this way would be preferable as more private - the server could only take on the task of monitoring timeouts and not missing finalizing tx broadcasts, something the server needs to handle since the client may not be online.

The whole aspect of an Electrum server taking on responsibility for channel timeouts seems like a dangerous area fraught with liability - something probably not many "free" servers would like to take on. It's almost prudent to build in redundancy across several servers with some kind of tx-pool.

[ysangkok]

@neocogent What is the difference of the "split" approach to running plain LND with Watchtower (unlinkable channel monitoring)? Watchtower works just as well for a "split" implementation as for normal implementations. But it IS extra effort to run LND for users, and is it of course scaling much worse than plain LND. That is another issue, but I think it is the real problem with a split approach long term.

c-lightning and LND (havn't looked a lot at the others) define isolated wallet interfaces. The signer interface in LND is wrapped in the wallet interface. The key derivation (there are MANY keys in lightning) is in the wallet interface. I have the signer interface almost completely implemented, but currently it relies on LND generating the private keys and sending them, which of course makes no sense, but facilitated testing. Roasbeef suggested to me that I implement the whole lnwallet interface, which I also think is the best approach. But it is thousands of lines of code, and it will take time. The workflow I have is working though, and by starting with the innermost signing features and extracting testing vectors from LND, I convinced myself that it is a viable strategy.

You're right that it is a bit complicated, but I don't really see a way around it. Lightning is pretty complex compared to plain Bitcoin. Just like many application layer protocols are more complicated than UDP.

[neocogent]

I'm sure you're waist deep in it and I'm just dabbling a toe at this point. When you need some testing be sure to post as I'd like to help if I can. Meantime I'll have to go read about Watchtower a bit as I don't know about that.

[SomberNight]

Work is being done on the lightning branch of this repo.

Some broad notes regarding the current ideas:

• Electrum will have its own Lightning implementation in python
• the client itself will function as a Lightning node
• no forwarding of payments for now (Electrum cannot be intermediate node in a payment route)
• all channels created by Electrum are "private" (no channel_announcement, won't show up in channel gossip)
  • in order to receive payments, invoices are populated with r fields (routing hints)
• the existing client <-> electrum server <-> bitcoind infrastructure remains the same
  • the server does not know about Lightning, it's an interface into the blockchain
• the server can lie by omission to the client regarding channel closures
  • bitcoin soft fork of UTXO commitments or equivalent would be nice
    • never going to happen
  • could use bip157/158
    • maybe in the future. sync would get slower then. maybe only do it for lightning related stuff and NOT onchain payments?
  • for now, we ignore this issue as it can be considered a subcase of a more generic problem: what if the client goes offline?
• the client cannot be expected to run 24/7, electrum is a light client afterall
  • custom watchtower implementation
    • open-source, anyone can run it
    • Electrum Technologies GmbH will also run one, and that will be the default option for users
      • the one run by the company will be a paid service (most likely flat monthly fee)
    • users upload encrypted blobs of presigned transactions to react to breaches
      • blobs are encrypted with txid of commitment txn
    • the blobs CAN be linked to a channel (funding outpoint) and to a user/subscription
      • but server does not know the channel balances
    • the server can delete blobs of deeply closed channels to free up space
    • the server can delete blobs of expired subscriptions

[shivaenigma]

no forwarding of payments for now (Electrum cannot be intermediate node in a payment route)
the client cannot be expected to run 24/7, electrum is a light client afterall

Totally agree with this philosophy 🛩️

Just wanted to make sure we have a backup mechanism in place for LN. It is very important for user to be able to recover the funds easily if his computer/hard disk crashes. It am not sure about the technicals of this , will watchtowers do channel backup ?

Feel point to point me to the correct thread if this is already covered.
Thanks a lot

[ecdsa]

Closing in favor of lightning-specific issues. The lightning branch has been merged.