Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pylightning: Implement the lightning handshake and wire protocol #2803

Merged
merged 2 commits into from Sep 30, 2019

Conversation

@cdecker
Copy link
Member

commented Jul 9, 2019

Simple transcript of the specification in python :-)

Sadly I had to drag in two more dependencies (cryptography and coincurve), I will consolidate on top of a simple secp256k1 FFI wrapper at a later point, but these were the minimal ones I could use atm.

@cdecker cdecker added the pylightning label Jul 9, 2019
@cdecker cdecker requested a review from rustyrussell Jul 9, 2019
@cdecker

This comment has been minimized.

Copy link
Member Author

commented Jul 9, 2019

----------- coverage: platform linux, python 3.6.7-final-0 -----------
Name                     Stmts   Miss  Cover
--------------------------------------------
lightning/__init__.py        2      0   100%
lightning/lightning.py     316    200    37%
lightning/plugin.py        283    131    54%
lightning/wire.py          259     24    91%
--------------------------------------------

test_listen_connect contains a minimal example on how to use the library:

from wire import LightningServerSocket, PrivateKey
from binascii import unhexlify
ls_privkey = PrivateKey(unhexlify('1111111111111111111111111111111111111111111111111111111111111111'))
l = LightningServerSocket(ls_privkey)
l.bind(('0.0.0.0', 19735))
l.listen()
c, a = l.accept()
print(c.read_message())
c.send_message(b'\x00\x10\x00\x00\x00\x01\xaa')
from wire import connect, PrivateKey, PublicKey
from binascii import unhexlify

ls_privkey = PrivateKey(unhexlify(b'1111111111111111111111111111111111111111111111111111111111111111'))
remote_pubkey = PublicKey(unhexlify(b'03b31e5bbf2cdbe115b485a2b480e70a1ef3951a0dc6df4b1232e0e56f3dce18d6'))

lc = connect(ls_privkey, remote_pubkey, '127.0.0.1', 19375)
lc.send_message(b'\x00\x10\x00\x00\x00\x01\xaa')
print(lc.read_message())

(these were running against my own c-lightning daemon, hence the port and node ID mismatch in the scripts).

@rustyrussell

This comment has been minimized.

Copy link
Contributor

commented Jul 9, 2019

TBH I think this belongs in a separate "protocol" library, rather than pylightning.

For example, if we were to extend it to generating commitment txs, etc. It's all outside the scope of talking to c-lightning. Of course, I'm tempted to steal this as-is for my lightning-rfc test branch.

@cdecker

This comment has been minimized.

Copy link
Member Author

commented Jul 9, 2019

TBH I think this belongs in a separate "protocol" library, rather than pylightning.

Good point, we might be better off just creating a separate library and add that to pypi (pylightning-wire / pylightning-proto maybe?)

@darosior

This comment has been minimized.

Copy link
Collaborator

commented Jul 13, 2019

This is great ! Actually I had planned to do something like this myself to dive deeper into the protocol but you did all the work : tweaking with it will for sure be helpful for a better understanding of the specs :).

TBH I think this belongs in a separate "protocol" library, rather than pylightning.

Good point, we might be better off just creating a separate library and add that to pypi (pylightning-wire / pylightning-proto maybe?)

Why not extend pylightning to be the reference Lightning Network Python library ? We could use just part of it for C-lightning (from pylightning import Plugin or LightningRpc) and some other Lightning Network applications (or some plugins!) could use other parts (from pylightning import wire).

@cdecker

This comment has been minimized.

Copy link
Member Author

commented Jul 13, 2019

Why not extend pylightning to be the reference Lightning Network Python library ? We could use just part of it for C-lightning (from pylightning import Plugin or LightningRpc) and some other Lightning Network applications (or some plugins!) could use other parts (from pylightning import wire).

I looked into how we can have multiple packages that use the same namespace (lightning in our case), but apparently having namespace-top-level code prevents us from having a second module that then also uses the same prefix. So either we break the pylightning into lightning.rpc and lightning.plugin, which would then allow us to have lightning.network for the wire implementation, or we create a new namespace and map things in there, or we create a completely new prefix.

@darosior

This comment has been minimized.

Copy link
Collaborator

commented Jul 13, 2019

or we create a new namespace and map things in there

Maybe lightning and clightning ? Otherwise the first solution is good if you prefer to not put C-lightning aside.

@cdecker cdecker force-pushed the cdecker:pywire branch from aa4787f to e4c0f30 Jul 23, 2019
@@ -0,0 +1,101 @@
# pylightning: A python client library for lightningd

This comment has been minimized.

Copy link
@darosior

darosior Jul 24, 2019

Collaborator

s/pylightning/pyln-client/ ?

b'03b31e5bbf2cdbe115b485a2b480e70a1ef3951a0dc6df4b1232e0e56f3dce18d6'
))

lc = connect(ls_privkey, remote_pubkey, '127.0.0.1', 19375)

This comment has been minimized.

Copy link
@darosior

darosior Jul 24, 2019

Collaborator

9735 ? Or is it intended ?

This comment has been minimized.

Copy link
@cdecker

cdecker Jul 24, 2019

Author Member

I was using a non-standard port for testing. Will change.

@cdecker cdecker force-pushed the cdecker:pywire branch from af51be0 to f0730c9 Jul 25, 2019
@cdecker

This comment has been minimized.

Copy link
Member Author

commented Jul 25, 2019

  • Added an alias for pylightning in pyln-client with the correct module prefix (pyln.client). pylightning remains fully functional for now, in order to maintain backward compatibility
  • Added the wire package in pyln-proto (import path pyln.proto.wire) as proposed by @rustyrussell

I haven't published the modules on pypi yet (pip install will work once we do), but better be sure that this is what we want. Also I didn't migrate the testing-related things into pyln-testing yet, since I need to figure out where to make the cut between library and c-lightning testing specific things.

Ping @rustyrussell @niftynei

@@ -0,0 +1,189 @@
from binascii import hexlify, unhexlify
from lightning.wire import PrivateKey, PublicKey, LightningConnection

This comment has been minimized.

Copy link
@ysangkok

ysangkok Sep 6, 2019

shouldn't this be pyln.proto.wire?

from binascii import hexlify, unhexlify
from lightning.wire import PrivateKey, PublicKey, LightningConnection
import socket
from lightning import wire

This comment has been minimized.

Copy link
@ysangkok
@@ -6,6 +6,9 @@
with io.open('README.md', encoding='utf-8') as f:
long_description = f.read()

with io.open('requirements.txt', encoding='utf-8') as f:
requirements = [r for r in f.read().split('\n') if len(r)]

setup(name='pylightning',
version=lightning.__version__,

This comment has been minimized.

Copy link
@ysangkok

ysangkok Sep 6, 2019

should this be pylightning or pyln?

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 17, 2019

Author Member

Seems correct to me, this is still the pylightning package, which contains the lightning module, which in turn is where the version is stored.

@cdecker cdecker self-assigned this Sep 6, 2019
@cdecker cdecker force-pushed the cdecker:pywire branch 3 times, most recently from 4caf76b to 6a75342 Sep 7, 2019
self.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

def accept(self):
print("Accepting a new socket")

This comment has been minimized.

Copy link
@ysangkok

ysangkok Sep 7, 2019

shouldn't this use a logging framework?

for i in range(0, 498):
lc.send_message(msg)
m = c2.recv(18 + 21)
# Check the last message against the test vector

This comment has been minimized.

Copy link
@ysangkok

ysangkok Sep 10, 2019

this is not actually checking a message, but the sk key.


def _maybe_rotate_keys(self):
if self.sn == 1000:
self.sck, self.sk = hkdf_two_keys(salt=self.sck, ikm=self.sk)

This comment has been minimized.

Copy link
@ysangkok

ysangkok Sep 11, 2019

only the test rotation and self.shake set self.sck. if i expand test_handshake to send more, i get an exception as self.sck is not set.

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 17, 2019

Author Member

I didn't intend for test_handshake to leave lc1 and lc2 in a usable state, since they just step through the test vector to pin-point an eventual regression. I added the initialization of sck and rck anyway :-)

@cdecker cdecker force-pushed the cdecker:pywire branch 3 times, most recently from 1c11b33 to 5ec50f8 Sep 17, 2019
@cdecker cdecker requested a review from darosior Sep 17, 2019
@cdecker

This comment has been minimized.

Copy link
Member Author

commented Sep 17, 2019

I think I addressed all the things that @ysangkok's thorough review unearthed. Thanks for the time you invested @ysangkok.

Ping @rustyrussell and @darosior for final compliance tests 😉

Copy link
Collaborator

left a comment

ACK 5ec50f8 (Code review only, could not test for now)

I think this is a great ressource not only for testing but also for learning more about the protocol. Also, if that's something we want (and you didn't already implement it locally :-) ) I'd like to go ahead and implement the bolt02-03 part. What do you think ?

))

lc = connect(ls_privkey, remote_pubkey, '127.0.0.1', 9375)
lc.send_message(b'\x00\x10\x00\x00\x00\x01\xaa')

This comment has been minimized.

Copy link
@darosior

darosior Sep 17, 2019

Collaborator

nit: Since this is an example, maybe add a comment indicating the message type ? And maybe even a link to bolt01 :-)

Simple transcript of the specification in python :-)

Signed-off-by: Christian Decker <decker.christian@gmail.com>
@cdecker cdecker force-pushed the cdecker:pywire branch from 5ec50f8 to 209ff34 Sep 19, 2019
This is the first step to transition to a better organized python module
structure. Sadly we can't reuse the `pylightning` module as a namespace module
since having importable things in the top level of the namespace is not
allowed in any of the namespace variants [1], hence we just switch over to the
`pyln` namespace. The code the was under `lightning` will now be reachable
under `pyln.client` and we add the `pyln.proto` module for all the things that
are independent of talking to lightningd and can be used for protocol testing.

[1] https://packaging.python.org/guides/packaging-namespace-packages/

Signed-off-by: Christian Decker <decker.christian@gmail.com>
@cdecker cdecker force-pushed the cdecker:pywire branch from 209ff34 to 067a7cd Sep 19, 2019
@cdecker

This comment has been minimized.

Copy link
Member Author

commented Sep 19, 2019

Thanks @darosior, I have no immediate plan to re-implement bolts 02 and 03, so go for it :-)

I added a comment in the example and fixed a spurious empty line, making make check-python happy again.

@cdecker cdecker added this to the 0.7.3 milestone Sep 23, 2019
@cdecker cdecker merged commit 3418e59 into ElementsProject:master Sep 30, 2019
2 checks passed
2 checks passed
bitcoin-bot/fixups PR does not contain unsquashed fixups
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.