Browse files

Merge branch 'dilate-5'

This adds (but does not enable/expose) the low-level code for the new
Dilation protocol (refs #312). The spec and docs are done, the unit tests
pass (with full branch coverage).

The next step is to write some higher-level integration tests, which use a
fake/short-circuited mailbox connection (Manager.send) but real localhost TCP

Then we need to figure out backwards compatibility with non-dilation-capable
versions. I've got a table in my notes, I'll add it to the ticket.
  • Loading branch information...
warner committed Dec 25, 2018
2 parents e6b4ccb + 803aa07 commit ddba0fc8408fb68c67b980859408ba109dac7804
Showing with 8,436 additions and 353 deletions.
  1. +5 −3 .travis.yml
  2. +115 −25 docs/
  3. +540 −0 docs/
  4. +2,000 −0 docs/new-protocol.svg
  5. +5 −1 docs/state-machines/
  6. +3 −0 setup.cfg
  7. +1 −0
  8. +6 −6 src/wormhole/
  9. +38 −1 src/wormhole/
  10. 0 src/wormhole/_dilation/
  11. +11 −0 src/wormhole/_dilation/
  12. +520 −0 src/wormhole/_dilation/
  13. +387 −0 src/wormhole/_dilation/
  14. +19 −0 src/wormhole/_dilation/
  15. +134 −0 src/wormhole/_dilation/
  16. +580 −0 src/wormhole/_dilation/
  17. +389 −0 src/wormhole/_dilation/
  18. +1 −0 src/wormhole/_dilation/
  19. +300 −0 src/wormhole/_dilation/
  20. +138 −0 src/wormhole/
  21. +24 −0 src/wormhole/
  22. +1 −6 src/wormhole/
  23. +1 −0 src/wormhole/
  24. +21 −0 src/wormhole/
  25. 0 src/wormhole/test/dilate/
  26. +21 −0 src/wormhole/test/dilate/
  27. +217 −0 src/wormhole/test/dilate/
  28. +463 −0 src/wormhole/test/dilate/
  29. +26 −0 src/wormhole/test/dilate/
  30. +98 −0 src/wormhole/test/dilate/
  31. +112 −0 src/wormhole/test/dilate/
  32. +174 −0 src/wormhole/test/dilate/
  33. +650 −0 src/wormhole/test/dilate/
  34. +655 −0 src/wormhole/test/dilate/
  35. +44 −0 src/wormhole/test/dilate/
  36. +269 −0 src/wormhole/test/dilate/
  37. +144 −0 src/wormhole/test/dilate/
  38. +196 −0 src/wormhole/test/
  39. +9 −3 src/wormhole/test/
  40. +26 −1 src/wormhole/test/
  41. +52 −182 src/wormhole/test/
  42. +12 −120 src/wormhole/
  43. +8 −0 src/wormhole/
  44. +18 −4 src/wormhole/
  45. +3 −1 tox.ini
@@ -17,13 +17,16 @@ before_script:
flake8 *.py src --count --select=E901,E999,F821,F822,F823 --statistics ;
- tox -e coverage
- if [[ $TRAVIS_PYTHON_VERSION == 2.7 || $TRAVIS_PYTHON_VERSION == 3.4 ]]; then
tox -e no-dilate ;
tox -e coverage ;
- codecov
- python: 2.7
- python: 3.3
- python: 3.4
- python: 3.5
- python: 3.6
@@ -34,5 +37,4 @@ matrix:
dist: xenial
- python: nightly
- python: 3.3
- python: nightly
@@ -524,25 +524,33 @@ object twice.

## Dilation

(NOTE: this section is speculative: this code has not yet been written)
(NOTE: this API is still in development)

In the longer term, the Wormhole object will incorporate the "Transit"
functionality (see directly, removing the need to instantiate a
second object. A Wormhole can be "dilated" into a form that is suitable for
bulk data transfer.
To send bulk data, or anything more than a handful of messages, a Wormhole
can be "dilated" into a form that uses a direct TCP connection between the
two endpoints.

All wormholes start out "undilated". In this state, all messages are queued
on the Rendezvous Server for the lifetime of the wormhole, and server-imposed
number/size/rate limits apply. Calling `w.dilate()` initiates the dilation
process, and success is signalled via either `d=w.when_dilated()` firing, or
`dg.wormhole_dilated()` being called. Once dilated, the Wormhole can be used
as an IConsumer/IProducer, and messages will be sent on a direct connection
(if possible) or through the transit relay (if not).
process, and eventually yields a set of Endpoints. Once dilated, the usual
`.send_message()`/`.get_message()` APIs are disabled (TODO: really?), and
these endpoints can be used to establish multiple (encrypted) "subchannel"
connections to the other side.

Each subchannel behaves like a regular Twisted `ITransport`, so they can be
glued to the Protocol instance of your choice. They also implement the
IConsumer/IProducer interfaces.

These subchannels are *durable*: as long as the processes on both sides keep
running, the subchannel will survive the network connection being dropped.
For example, a file transfer can be started from a laptop, then while it is
running, the laptop can be closed, moved to a new wifi network, opened back
up, and the transfer will resume from the new IP address.

What's good about a non-dilated wormhole?:

* setup is faster: no delay while it tries to make a direct connection
* survives temporary network outages, since messages are queued
* works with "journaled mode", allowing progress to be made even when both
sides are never online at the same time, by serializing the wormhole

@@ -556,21 +564,103 @@ Use non-dilated wormholes when your application only needs to exchange a
couple of messages, for example to set up public keys or provision access
tokens. Use a dilated wormhole to move files.

Dilated wormholes can provide multiple "channels": these are multiplexed
through the single (encrypted) TCP connection. Each channel is a separate
stream (offering IProducer/IConsumer)

To create a channel, call `c = w.create_channel()` on a dilated wormhole. The
"channel ID" can be obtained with `c.get_id()`. This ID will be a short
(unicode) string, which can be sent to the other side via a normal
`w.send()`, or any other means. On the other side, use `c =
w.open_channel(channel_id)` to get a matching channel object.

Then use `c.send(data)` and `d=c.when_received()` to exchange data, or wire
them up with `c.registerProducer()`. Note that channels do not close until
the wormhole connection is closed, so they do not have separate `close()`
methods or events. Therefore if you plan to send files through them, you'll
need to inform the recipient ahead of time about how many bytes to expect.
Dilated wormholes can provide multiple "subchannels": these are multiplexed
through the single (encrypted) TCP connection. Each subchannel is a separate
stream (offering IProducer/IConsumer for flow control), and is opened and
closed independently. A special "control channel" is available to both sides
so they can coordinate how they use the subchannels.

The `d = w.dilate()` Deferred fires with a triple of Endpoints:

d = w.dilate()
def _dilated(res):
(control_channel_ep, subchannel_client_ep, subchannel_server_ep) = res

The `control_channel_ep` endpoint is a client-style endpoint, so both sides
will connect to it with `ep.connect(factory)`. This endpoint is single-use:
calling `.connect()` a second time will fail. The control channel is
symmetric: it doesn't matter which side is the application-level
client/server or initiator/responder, if the application even has such
concepts. The two applications can use the control channel to negotiate who
goes first, if necessary.

The subchannel endpoints are *not* symmetric: for each subchannel, one side
must listen as a server, and the other must connect as a client. Subchannels
can be established by either side at any time. This supports e.g.
bidirectional file transfer, where either user of a GUI app can drop files
into the "wormhole" whenever they like.

The `subchannel_client_ep` on one side is used to connect to the other side's
`subchannel_server_ep`, and vice versa. The client endpoint is reusable. The
server endpoint is single-use: `.listen(factory)` may only be called once.

Applications are under no obligation to use subchannels: for many use cases,
the control channel is enough.

To use subchannels, once the wormhole is dilated and the endpoints are
available, the listening-side application should attach a listener to the
`subchannel_server_ep` endpoint:

def _dilated(res):
(control_channel_ep, subchannel_client_ep, subchannel_server_ep) = res
f = Factory(MyListeningProtocol)

When the connecting-side application wants to connect to that listening
protocol, it should use `.connect()` with a suitable connecting protocol

def _connect():
f = Factory(MyConnectingProtocol)

For a bidirectional file-transfer application, both sides will establish a
listening protocol. Later, if/when the user drops a file on the application
window, that side will initiate a connection, use the resulting subchannel to
transfer the single file, and then close the subchannel.

def FileSendingProtocol(internet.Protocol):
def __init__(self, metadata, filename):
self.file_metadata = metadata
self.file_name = filename
def connectionMade(self):
sender = protocols.basic.FileSender()
f = open(self.file_name,"rb")
d = sender.beginFileTransfer(f, self.transport)
d.addBoth(self._done, f)
def _done(res, f):
def _send(metadata, filename):
f = protocol.ClientCreator(reactor,
FileSendingProtocol, metadata, filename)
def FileReceivingProtocol(internet.Protocol):
state = INITIAL
def dataReceived(self, data):
if state == INITIAL:
self.state = DATA
metadata = parse(data)
self.f = open(metadata.filename, "wb")
# local file writes are blocking, so don't bother with IConsumer
def connectionLost(self, reason):
def _dilated(res):
(control_channel_ep, subchannel_client_ep, subchannel_server_ep) = res
f = Factory(FileReceivingProtocol)

## Bytes, Strings, Unicode, and Python 3

Oops, something went wrong.

0 comments on commit ddba0fc

Please sign in to comment.