Add STREAM Transport Protocol #417

emschwartz · 2018-04-02T21:22:03Z

Still need to add each of the frame specs but I figured I'd get the discussion started here.

What do you think? What needs more explanation?

sharafian

Looks really good overall, but I had a few questions

sharafian · 2018-04-02T22:26:03Z

0000-stream/0000-stream.md

+- **Data Stream** - A bidirectional data channel used for structuring data sent over a Connection. Inspired by [QUIC's Bidirectional Streams](https://tools.ietf.org/html/draft-ietf-quic-transport-10#page-66).
+- **Money Stream** - A bidirectional money "channel" used for sending money over a Connection. Users of STREAM MAY use a single Money Stream for each Connection or multiple to account for different intended purposes for the value sent.
+- **Receiver** - The entity that gets a STREAM packet attached to an ILP Prepare and responds with either an ILP Fulfill or Reject packet. The Receiver may refer to either the Client or Server.
+- **Sender** - The entity that sends a STREAM packet attached to an ILP Prepare. The Sender may refer to either the Client or Server.


You say entity here and party in other places. I think we should try to be consistent with one or the other, unless there's a difference in the two terms that I'm missing

Updated to remove "party"

sharafian · 2018-04-02T22:27:07Z

0000-stream/0000-stream.md

+
+Before using STREAM, the Client and Server use an authenticated, encrypted communication channel (generally provided as part of an [Application Layer Protocol](../0001-interledger-architecture/0001-interledger-architecture.md#application-layer)) to exchange:
+- A 32-byte random or pseudorandom shared secret generated by the Server
+- The server's ILP address


nitpick: you capitalize Server in the previous bullet but not here

sharafian · 2018-04-02T22:28:46Z

0000-stream/0000-stream.md

+
+STREAM accounts for value sent over a Connection using the Money Stream abstraction. Money Streams are bidirectional, meaning they can be used to send value from the Client to the Server or from the Server to the Client. A Connection may include any number of Money Streams.
+
+Multiple Money Streams MAY be used to separately account for value sent and received for different purposes. A Money Stream can be long-lived and the send and receive limits MAY be adjusted over time (for example, in response to an application-level action) or a Money Stream can be used to account for a single logical payment, in which case it would be closed when the payment is finished.


You reference send and receive limits here, but I don't think it's obvious to a new reader what those mean

sharafian · 2018-04-02T22:31:14Z

0000-stream/0000-stream.md

+
+### Multiplexed Data Streams
+
+STREAM also offers the ability to send data through the Connection over ILP (rather than over the Internet). Data Streams are the abstraction used for grouping bytes sent over the connection. Data Streams are heavily inspired by [QUIC Bidirectional Streams](https://tools.ietf.org/html/draft-ietf-quic-transport-10#section-10).


could you add a couple sentences here on what the data streams provide in terms of ordering, chunking, etc.?

sharafian · 2018-04-02T22:32:26Z

0000-stream/0000-stream.md

+
+### Stream and Connection Flow Control
+
+STREAM implements stream- and connection-level flow control. Clients and Servers communicate the maximum amount of money or data they are willing to receive on a given stream or for the connection as a whole. Limits may be changed and retransmitted throughout the life of the Connection to accept more money or data as desired by the application.


maybe add something saying that implementations SHOULD start the limits at 0 until the party says otherwise to prevent accidental transferring? Also what would be the purpose of a data limit? Just to prevent flooding?

Added e486b50#diff-58f381369b35d5dd3708619716745006R58

sharafian · 2018-04-02T22:39:05Z

0000-stream/0000-stream.md

+- Does not include a cryptographic handshake, because STREAM assumes a symmetric secret is communicated out of band.
+- Does not support unidirectional frames. The QUIC community had significant debate over whether to include unidirectional streams, bidirectional streams, or both. They settled on both primarily to support the HTTP request/response pattern as well as HTTP/2 Server Push. Unidirectional streams were left out of STREAM because they add complexity and are a premature optimization for this protocol now.
+- Does not have ACK frames, because ILP Prepare packets must be acknowledged with either a Fulfill or Reject packet. If the response includes an (authenticated) STREAM packet, the sender can treat that as an acknowledgement of the control and data frames from the Prepare packet they sent.
+- Does not have Ping, Path Challenge, or Path Challenge Response frames, because a STREAM packet with no frames can be used instead. As long as the Client and Server increment the packet sequence for each packet they send, a valid Fulfill or Reject packet from the other party that includes the correct sequence in the encrypted data serves as the path challenge and response.


What does the packet sequence do exactly, and are there any ways it could get out of sync?

sharafian · 2018-04-02T22:39:48Z

0000-stream/0000-stream.md

+
+All STREAM packets are intended to be sent in the `data` of an [ILP Prepare, Fulfill, or Reject](../0027-interledger-protocol-4/0027-interledger-protocol-4.md) packet.
+
+### Encryption


There's an Authenticated Encryption section above; this seems like a duplicate

Removed that duplicate description line

sharafian · 2018-04-02T22:43:01Z

0000-stream/0000-stream.md

+| Prepare Amount | VarUInt | If the STREAM packet is sent on an ILP Prepare, this represents the minimum the receiver should accept. If the packet is sent on an ILP Fulfill or Reject, this represents the amount that the receiver got in the Prepare. |
+| Frames | SEQUENCE OF Frame | The rest of the packet is comprised of type- and length-prefixed Frames, as specified below. Note that the array of frames is NOT length-prefixed. |
+
+**TODO**: Should we length-prefix the frames array and make the padding a field in the packet format to better comply with OER? Note that we cannot length-prefix the padding because in certain edge cases it could make it impossible to pad to the right length when the length prefix is included.


I think it's better to length prefix the frames array than use 0x00 as a delimiter. Then if we want to extend the protocol later we wouldn't have a permanent 0 byte delimiter as support for previous versions

Actually I tried this an ran into the problem that length-prefixing the padding potentially changes the amount of padding you need. I think it's possible to have a situation where it is impossible to fill the packet exactly while putting the correct length prefix. I think this is why QUIC implemented the padding this way as well.

Then if we want to extend the protocol later we wouldn't have a permanent 0 byte delimiter as support for previous versions

What do you mean by this? It's easy to add new frames to the protocol because they'll just be ignored by old versions. I think that's a better way to extend the protocol than modifying the packet header.

sharafian · 2018-04-02T22:44:57Z

asn1/Stream.asn

+
+FrameSet FRAME ::= {
+  -- Connection-related frames (hex 0x0?)
+  {1 ConnectionError} |


These numbers don't seem to match what's written in the file above

sharafian · 2018-04-02T22:47:36Z

asn1/Stream.asn

+  -- Identifier of the MoneyStream
+  streamId VarUInt,
+
+  -- The proportion of the Prepare amount that should go to this stream


maybe it's fine, but it seems slightly strange to have this use the sum of all streamMoney frames. You need to keep a running total on the side and then execute the streamMoney logic after all other frames are parsed, rather than being able to apply each frame's effect as you go through them

If we used something like percents or put the sum of all shares into each of the streammoney frames, then you could apply each frame without having to execute money frames at the end.

If we used something like percents

What if the sum of all streamMoney frame percentages adds up to more than or less than 100%? If you have a total (the prepare amount) and want to partition it among multiple shares (streamMoney frames) but atomically apply this (i.e. reject the packet if sum of shares != total) then you can't avoid delayed execution of the money frames.

There's no real need to apply it atomically; if the sender makes an error and gives out more than 100% of shares then the last frames just wouldn't be applied to their streams.

What if it's < 100% where does the remaining money go?
How does the sender get notified which frames were applied and which weren't?

michielbdejong

Can you add a frame type to query the balance of the virtual accounting relationship between sender and receiver?

Overall the protocol looks very ingenious, impressive! I do think a downside of adding this to the RFCs repo is that, although you already implemented the reference implementation, maybe the Quilt project doesn't currently have the bandwidth to implement all of this, so that would require some planning maybe. Will discuss with @adrianhopebailie and @sappenin next week!

michielbdejong · 2018-04-03T07:31:22Z

0000-stream/0000-stream.md

+
+## Overview
+
+### Relation to Other Protocols


Please explain the relation to LT as well.

|-----------------| | LT | STREAM | |-----------------| | ILPv4 | |-----------------| | ledger layer | |-----------------|

michielbdejong · 2018-04-03T07:34:26Z

0000-stream/0000-stream.md

+
+### Setup
+
+Before using STREAM, the Client and Server use an authenticated, encrypted communication channel (generally provided as part of an [Application Layer Protocol](../0001-interledger-architecture/0001-interledger-architecture.md#application-layer)) to exchange:


If you start from a situation where ILPv4 is ubiquitous, then ILPv4 accounting relationships will be everywhere, and it's useful to explain why you would still want to set up an end-to-end piggy-backed protocol on top of a semi-circle ILP path. I discussed this with @BobWay yesterday, and one great application of the semi-circle family-of-protocols (psk, psk2, psk2-draft2, PayStream, and STREAM) is when for instance you scan a barcode (with for instance an ILP address and a public key) on a vending machine: yes, there is an accounting relationship between you and that vending machine, but:

the relationship is short-lived

there is an air-gapped line-of-sight communication channel in the physical world, and converting that to a data connection (like in Add Bluetooth Payment Protocol Draft #186) can be avoided by using a semi-circle payment as a sort of "vpn over ilp"

So then this statement is incorrect; client and server do not need to use a data connection during the setup! :)

As we discussed, you might still use a semi-circle protocol like STREAM to reduce latency, but by saying in the first step that STREAM requires a communication channel, aren't you removing one of its greatest unique selling points?

So then this statement is incorrect; client and server do not need to use a data connection during the setup! :)

Sure, they need a private communication channel where they are sure that no one is eavesdropping. We could add a note that those properties don't need to come from authenticated encryption, but I think in most cases that would just be something like HTTPS (whether the data is sent in the body or headers).

by saying in the first step that STREAM requires a communication channel, aren't you removing one of its greatest unique selling points?

No, I don't see how it would be. Even if you need to communicate a single message on startup, it makes a big difference to avoid an extra delay for every packet. In many cases that message may not even require an extra roundtrip because it can be sent alongside other application data that you'd be sending anyway. Also, I think we may end up implementing Diffie-Hellman over ILP/STREAM so that you only need to start with a public key and ILP address (that you could get from some kind of public lookup), but I don't think we need to do that right now.

Your said elsewhere that "it seems crazy to require a connection over the internet", so then why are you against highlighting that as an advantage here?

As I see it the main advantage is that the connection over the internet isn't a part of the protocol flow. Usually you have access to the internet, but because STREAM doesn't need a special purpose channel you can just piggyback the initial setup on top of some communication you were already doing (i.e. in the response headers for an API call, or communicated through a laser)

emschwartz · 2018-04-03T12:20:21Z

Can you add a frame type to query the balance of the virtual accounting relationship between sender and receiver?

I don't think that's a good idea for the following reasons:

As @sharafian pointed out when we were discussing removing getBalance from the Ledger Plugin Interface, having a static view of the balance does not help if it's changing constantly over time (for example when streaming some content) according to some more complex function
I think it would be unused by many if not most applications you could build on this. For example, if I'm sending you money and set my limit to 1000 because that's the maximum I'm willing to pay, knowing what you think my balance is will not change what I'm willing to pay
Since STREAM includes a data channel, any applications or protocols that do want to be able to query the static balance or rate function can easily add that to their protocol

dappelt · 2018-04-03T14:53:24Z

0000-stream/0000-stream.md

+
+STREAM accounts for value sent over a Connection using the Money Stream abstraction. Money Streams are bidirectional, meaning they can be used to send value from the Client to the Server or from the Server to the Client. A Connection may include any number of Money Streams.
+
+Money Streams use absolute amounts to track the total amounts sent and received, as well as the limits for sending and receiving. For example, the Sender might set their send maximum to 1000 to send 1000 units and the Connection would keep sending ILP Prepare packets until the total sent is greater than or equal to 1000.


The term "send maximum" is confusing given that it sends "greater than or equal to 1000". Should be rather called send minimum.

That's a mistake. It should be the maximum. I'll update it to get rid of the greater than part

dappelt · 2018-04-03T15:10:19Z

0000-stream/0000-stream.md

+
+All STREAM packets are encrypted using [AES-256-GCM](https://en.wikipedia.org/wiki/Galois/Counter_Mode) with a 12-byte Initialization Vector (IV) and a 16-Byte Authentication Tag.
+
+If subsequent versions support additional encryption algorithms, those details should be exchanged between the sender and receiver when they establish the shared secret. If a receiver attempts to decrypt an incoming packet but is unable to (perhaps because the sender is using an unsupported cipher), they SHOULD reject the incoming transfer with an `F06: Unexpected Payment` error.


... they SHOULD ... -> ... the receiver SHOULD ...

sentientwaffle · 2018-04-03T15:16:36Z

0000-stream/0000-stream.md

+- **Client** - One of the two entities using STREAM. The Client initiates a Connection to the Server using STREAM packets sent over ILP. Because STREAM uses bidirectional streams, the Client may be either the Sender or the Receiver of a given packet.
+- **Connection** - The session established between a Client and a Server. A Connection may have zero or more Money Streams and zero or more Data Streams.
+- **Data Stream** - A bidirectional data channel used for structuring data sent over a Connection. Inspired by [QUIC's Bidirectional Streams](https://tools.ietf.org/html/draft-ietf-quic-transport-10#page-66).
+- **Money Stream** - A bidirectional money "channel" used for sending money over a Connection. Users of STREAM MAY use a single Money Stream for each Connection or multiple to account for different intended purposes for the value sent.


Why is channel in quotes here but not in the Data Stream definition? What's the difference?

adrianhopebailie

Awesome stuff. Great idea to leverage QUIC.

adrianhopebailie · 2018-04-03T16:04:36Z

0000-stream/0000-stream.md

+| Prepare Amount | VarUInt | If the STREAM packet is sent on an ILP Prepare, this represents the minimum the receiver should accept. If the packet is sent on an ILP Fulfill or Reject, this represents the amount that the receiver got in the Prepare. |
+| Frames | SEQUENCE OF Frame | The rest of the packet is comprised of type- and length-prefixed Frames, as specified below. Note that the array of frames is NOT length-prefixed. |
+
+**TODO**: Should we length-prefix the frames array and make the padding a field in the packet format to better comply with OER? Note that we cannot length-prefix the padding because in certain edge cases it could make it impossible to pad to the right length when the length prefix is included.


If we use ASN.1's SEQUENCE OF (which you have) then I believe the length (i.e. count of frames) is part of the OER encoding. Why do you need to length prefix that?

It is technically part of the OER format but I'm not currently length-prefixing that array (I know, it's not the right way to do it). The reason I'm not is related to the padding issue described in #417 (comment).

adrianhopebailie · 2018-04-03T16:07:19Z

asn1/Stream.asn

+  -- If attached to a Fulfill or Reject packet, amount that arrived at the receiver
+  prepareAmount VarUInt,
+
+  frames SEQUENCE OF Frame


Should be frames SEQUENCE OF StreamFrame?

adrianhopebailie · 2018-04-03T16:14:27Z

0000-stream/0000-stream.md

+
+#### Padding
+
+Zero (hex `0x00`) bytes MAY be appended after the other frames, for example to ensure that all packets are the same size to minimize the information gained from passive packet analysis. A parser SHOULD stop reading once it reaches a zero byte instead of a Frame Type Identifier.


Why not simply define a padding frame that is always ignored?

There are really two questions here:

Is the padding frame just a single zero byte (0x00, like Padding in QUIC) or a proper frame that has a length prefix for the contents?

Can padding come between other frames (see Padding between frames quicwg/base-drafts#158) or does it have to come at the end?

The problem I ran into with length-prefixing the padding is that OER length prefixes are themselves variable length. You need to know how much padding there is to know how long the length prefix is, but the length of the length prefix affects how much padding you need. I'm not completely sure but I think there is a possible edge case where it's impossible to get the right amount of padding because of that.

If you don't length-prefix the padding, you definitely want to mandate that padding should come at the end, because checking individual bytes is very slow. I ran into this when adding padding to the implementation and at first the padded packets made the tests run thousands of times slower than the tests for unpadded packets. I thought it might be encryption related at first, but it turned out that it was just the speed of checking the value of each byte to see if it was zero (there were >30k of them).

adrianhopebailie · 2018-04-03T16:21:31Z

asn1/Stream.asn

+  -- Identifier of the MoneyStream
+  streamId VarUInt,
+
+  -- The proportion of the Prepare amount that should go to this stream


If we used something like percents

What if the sum of all streamMoney frame percentages adds up to more than or less than 100%? If you have a total (the prepare amount) and want to partition it among multiple shares (streamMoney frames) but atomically apply this (i.e. reject the packet if sum of shares != total) then you can't avoid delayed execution of the money frames.

dappelt · 2018-04-04T09:53:54Z

0000-stream/0000-stream.md

+Clients and Servers MAY change the address the other entity will send packets to at any time by sending a `ConnectionNewAddress` frame with their new ILP address.
+
+If one entity changes their address, the other SHOULD NOT assume that the asset is the same or that the exchange rate will be similar.
+


Given that STREAM Connections can be long-lived, did you consider the implications for the encryption key? A given key should only be used to encrypt a limited number of messages.

That number is large, but it might become relevant for long-lived connections. Quote from the NIST publication on AES-GCM:

The total number of invocations of the authenticated encryption function shall not exceed 2^32, including all IV lengths and all instances of the authenticated encryption function with the given key.

Good point. Do you think it's better to drop the connection or add a frame that allows you to change the shared secret? I originally thought about adding a shared secret to the ConnectionNewAddress frame but opted against it.

Let's break down the pros and cons of both options.

The options are: After the total number of encrypted messages exceeds some threshold, 1) the connection should either be dropped or 2) the encryption key needs to be renewed.

1) Dropping the connection
+ simple to implement
- higher-level protocols have to assume that the connection might be dropped and have to reestablish

2) Renewing the key
+ no connection interruption due to exhausting the key/nonces
- More complex. Requires a new frame type or an existing frame needs to be extended to communicate the new key
- Duplicates Logic. Key establishment is explicitly not part of PSK but assumed to be done over a separate, secure communication channel. If we add a way to exchange keys within the protocol, we are essentially duplicating the key establishing logic.
- Potentially Insecure. Depending on how it is implemented, it might be insecure. For example, a man-in-the-middle could just count how many messages went over a connection and when it is about time to renew the key, he would drop all packets, thus preventing client and server to exchange a new key. Once the max. number of invocations for AES-GCM is reached, client/server have no other (secure) option as to drop the connection. Otherwise, the man-in-the-middle can decrypt packets and learn the new encryption key.

The last point makes me realize that as a last resort the connection NEEDS to be dropped. Renewing the key within the protocol is just a convenience feature. So I definitely recommend to go with 1) for now.

rhuairahrighairidh

Nice work. Slightly hard to see how all the pieces fit together without a good understanding of QUIC, but I assume that can be taken as required reading.

dora-gt · 2018-04-08T14:45:18Z

0000-stream/0000-stream.md

+
+# STREAM: A Multiplexed Money and Data Transport Protocol
+
+ILP/STREAM, or the STREAMing Transport for the Real-time Exchange of Assets and Messages, is an Interledger transport protocol that provides for multiplexed streams of money and data over a virtual ILP connection. STREAM is a successor to [PSK2](../0025-pre-shared-key-2/0025-pre-shared-key-2.md) and takes significant inspiration from the [QUIC](https://tools.ietf.org/html/draft-ietf-quic-transport-10) Internet transport protocol.


I could not understand what multiplexed means at a glance. Giving some examples may help.

What cannot be achieved WITHOUT multiplexed transport?

What can be achieved WITH multiplexed transport?

dora-gt · 2018-04-08T15:25:56Z

0000-stream/0000-stream.md

+
+## Definitions
+
+- **Client** - One of the two entities using STREAM. The Client initiates a Connection to the Server using STREAM packets sent over ILP. Because STREAM uses bidirectional streams, the Client may be either the Sender or the Receiver of a given packet.


Do the two entities include Server?
I could not make it clear that...

Two entities mean sender/receiver AND server

Two entities mean sender AND receiver, and do not contain server

Also, I could not understand what initiates means here because Setup section explains the initiation, doesn't it?

In the section, it seems that we don't use ILP packet for the initiation.

It may be hard to explain the structure and the procedure without some diagrams.

dora-gt · 2018-04-08T15:54:13Z

0000-stream/0000-stream.md

+- **Server** - One of the two entities using STREAM. The Server listens through an ILP account for incoming Connections from Clients. Note that STREAM Servers DO NOT listen for Connections over the Internet, but rather over Interledger. Because STREAM uses bidirectional streams, the Server may be either the Sender or the Receiver of a given packet.
+
+## Overview
+


Before the Relation, I strongly recommend to explain what multiplexed or STREAM means essentially. What we can do with STREAM. Up to here, I could not imagine what it is exactly.

Sending money to multiple addresses using a single connection?

Sending not only money transfer data but also data that is not concerned with the money transfer, and these are bidirectional?

Basically, readers want a big and essential picture first of all.

emschwartz · 2018-05-10T03:46:53Z

I simplified the protocol further, fleshed out the descriptions more, and incorporated feedback from the first round of review. Let me know if you have any comments or questions!

adrianhopebailie

This is fantastic! Some editorial suggestions and a few questions but nothing to block publishing

adrianhopebailie · 2018-05-10T10:29:34Z

0000-stream/0000-stream.md

+
+In an application that uses an ongoing flow of money, an endpoint can open a single stream and continue sending money on it for the duration of some paid interaction.
+
+In another another application, an endpoint can open a stream, send a specific amount of money through it, and then close the stream to indicate the payment is complete. In this case, the stream abstraction provides a mechanism to "frame" a payment or message that may be split across multiple packets. New streams can be opened and closed on the same connection to send multiple messages.


s/another another/another/

adrianhopebailie · 2018-05-10T10:30:32Z

0000-stream/0000-stream.md

+
+STREAM uses bidirectional streams, which can be used for request/response flows or one-way messages.
+
+The choice streams as the key abstraction is inspired by [QUIC](https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#rfc.section.9) and the earlier [Structured Streams Transport (SST)](http://www.brynosaurus.com/pub/net/sst.pdf).


s/choice streams/choice of streams/

adrianhopebailie · 2018-05-10T10:39:49Z

0000-stream/0000-stream.md

+
+#### 4.4.5. Closing Streams
+
+Either endpoint can close a stream using a `StreamClose` frame. Implementations MAY allow half-open streams.


Define half-open streams

adrianhopebailie · 2018-05-10T10:41:56Z

0000-stream/0000-stream.md

+
+### 4.6. Closing Connections
+
+Either endpoint can close the connection using a `ConnectionClose` frame. Implementations MAY allow half-open connections.


Define half-open connections

adrianhopebailie · 2018-05-10T10:44:48Z

0000-stream/0000-stream.md

+- STREAM Version (optional -- assumed to be version 1 unless specified)
+- Server ILP Address
+- Cryptographically secure random or pseudorandom shared secret (it is RECOMMNEDED to use 32 bytes)
+


The encryption algorithm that will be used to encrypt the ILP packet data (optional -- assumed to be AES-256-GCM)

I don't think it's necessary to put that. Right now version 1 doesn't let you use other ciphers. A version 1.1 or 2 could, in which case you might need to communicate that. But this spec really just defines v1

Sure, I was just being consistent with the first bullet (STREAM version)

adrianhopebailie · 2018-05-10T10:49:01Z

0000-stream/0000-stream.md

+
+#### 5.1.2. Encryption Pseudocode
+
+The encryption key used for every packet sent for a given connection is derived from the shared secret and the string `"ilp_stream_encryption"` using an HMAC.


specify encoding of the string (UTF-8 I think if you're using Nodejs defaults)

Since it's all plain ascii characters, the string is the same in ASCII and UTF8

True. Although, I think specifying the encoding is still worth doing because crypto algorithms are generally defined with either numbers or octet streams as input not strings. Specifying the encoding just explicitly tells the reader how to go from the string to the desired octets.

0000-stream/0000-stream.md

+
+The amount of money that should go to each stream is calculated by dividing the number of shares for the given stream by the total number of shares in all of the `StreamMoney` frames in the packet.
+
+For example, if an ILP Prepare packet has an amount of 100 and three `StreamMoney` frames with 5, 15, and 30 shares for streams 2, 4, and 6, respectively, that would indicate that stream 2 should get 10 units, stream 4 gets 30 units, and stream 6 gets 60 units.


adrianhopebailie · 2018-05-10T11:13:24Z

asn1/Stream.asn

+  ilpPacketType UInt8,
+
+  -- Packet ID
+  sequence VarUInt,


Since we recommend closing connections after 2^32 packets have been sent couldn't this be fixed length?

True. I don't feel that strongly about this but VarUInts are shorter until you're in the tens of millions, which I doubt we're going to get to very frequently.

Yeh, I think if you made this UInt32 it would still be less efficient than VarUInt for the majority of packets.

adrianhopebailie · 2018-05-10T11:13:41Z

asn1/Stream.asn

+
+  -- If attached to a Prepare packet, minimum amount that the receiver should accept
+  -- If attached to a Fulfill or Reject packet, amount that arrived at the receiver
+  prepareAmount VarUInt,


Is this not UInt64 to save space?

Yeah, and to be more consistent with the rest of the fields in this protocol. I could be convinced if someone feels strongly that it should be a UInt64

My only thought was that this will always correspond to an amount in an ILP packet but I think you are right that saving bytes should be the priority.

sharafian

Some questions regarding the explanations, but overall LGTM

sharafian · 2018-05-11T20:54:11Z

0000-stream/0000-stream.md

+
+- **Client** - The endpoint initiating a STREAM connection
+- **Server** - The endpoint accepting incoming STREAM connections
+- **Endpoint** - The client or server end of a connection


Endpoint makes me think of an HTTP endpoint, so I keep having to do a double take. The fact that we're using this for paid APIs which one may describe as a "paid endpoint" could make it more confusing too.

I used that term in this rewrite just because that was what QUIC used. Before I was using "entity" or "party" which seem too vague. I hear you on the potential confusion though.

I think Endpoint is a really good word because it makes me imagine something except for connectors that lie in the middle, the very end of the stream which means sender or receiver even if it sometimes recalls us HTTP endpoint.

To make it better, I propose modifying

Endpoint - The client or server end of a connection
to

Endpoint - The client or server end of a connection, which can be terminal sender or receiver of money and data.

sharafian · 2018-05-11T20:58:46Z

0000-stream/0000-stream.md

+
+Streams are given numerical IDs. Client-initiated streams are given odd numbers starting with 1 and server-initiated streams are given even numbers starting with 2 (this is used to avoid collisions if both endpoints open streams at the same time).
+
+Endpoints can limit the number of concurrently active incoming streams by adjusting the maximum stream ID and communicating that limit to the other endpoint.


What's the default maximum stream ID?

In the implementation it's 20 (so it'll let the other side open 10 streams).

One thing QUIC has that I didn't include here is an initial parameter negotiation. I thought it seemed unnecessary to have another data format for communicating these, but we could suggest (or mandate) that the first packet include frames to indicate the max stream ID and the max connection data.

sharafian · 2018-05-11T20:59:55Z

0000-stream/0000-stream.md

+
+### 3.4. Exchange Rates
+
+A critical function of Interledger Transport Protocols is to determine the path exchange rate and handle any changes in the rate. STREAM packets include a minimum acceptable amount in ILP Prepare packets and the amount that arrived at the other endpoint in the Fulfill or Reject packet.


Given that the receiver's units can never be trusted nor relied upon, we should explain why the exchange rate is important to keep track of.

sharafian · 2018-05-11T21:07:32Z

0000-stream/0000-stream.md

+
+When a client connects to a server, they MUST communicate their ILP address to the server using a `ConnectionNewAddress` frame.
+
+Either endpoint MAY change their ILP address at any point during a connection by sending a `ConnectionNewAddress` frame.


what happens if the ConnectionNewAddress frame is dropped or is otherwise rejected? Also, can ConnectionNewAddress be sent in a response packet or only a prepare?

If you are actually going to do a switch-over you probably want to keep listening on both for a couple seconds or so (so that you get packets that might have been in flight before you switched). If you want to know for sure that the frame was received, you need to put it in a Prepare.

sharafian · 2018-05-11T21:12:46Z

0000-stream/0000-stream.md

+
+Either endpoint MAY change their ILP address at any point during a connection by sending a `ConnectionNewAddress` frame.
+
+An attacker could cause an unsuspecting endpoint to participate in a Denial of Service (DoS) attack on a third-party endpoint. To mitigate this, endpoints SHOULD refrain from sending large numbers of packets or large amounts of data to a new ILP address before validating that the ILP address is controlled by the same party that knows the shared secret. STREAM uses the authenticated request/response packets in lieu of [QUIC's explicit Path Validation](https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#rfc.section.6.7).


Doesn't the authenticated encryption make it impossible for an attacker to pull this attack off? This paragraph makes it sound like that attack is possible under the current protocol.

Authenticated encryption means you know if you're not talking to the right party. However, the point of this attack is to create a legitimate connection to you and then tell you I've changed my address to that of the victim so that you bombard the new address with data. The mitigation for this is that if you get an address change you probably shouldn't send more than one packet before getting a valid (authenticated) message back from the new address.

emschwartz · 2018-05-12T01:05:21Z

0000-stream/0000-stream.md

+
+### 5.1. Encryption
+
+All STREAM packets are encrypted using [AES-256-GCM](https://en.wikipedia.org/wiki/Galois/Counter_Mode) with a 12-byte Initialization Vector (IV) and a 16-Byte Authentication Tag.


That isn't in the ILP address, those values are in the encryption envelope in the ILP packet data

sharafian · 2018-05-11T21:18:01Z

0000-stream/0000-stream.md

+
+#### 5.1.3. Maximum Number of Packets Per Connection
+
+Implementations MUST close the connection once either endpoint has sent 2^31 packets. According to [NIST](https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38d.pdf), it is unsafe to use AES-GCM for more than 2^32 packets using the same encryption key.


is it 2^31 in order to give a bigger safety margin, or did you mean to write 2^32-1?

Both sides are encrypting messages with the same key, so unless they keep track of how many packets they've gotten from the other side they should only send 2^31 messages each

We discussed the loopback protocol and concluded not to pursue it for now

@dappelt

based on feedback from @dappelt, @sentientwaffle, @adrianhopebailie

@dora-gt

- combines data and money streams into one type of stream - details frame types - incorporates feedback from @dora-gt, @rhuairahrighairidh, @dappelt

@dora-gt

incorporates feedback from @dora-gt

traviscrist · 2018-05-14T20:23:53Z

0029-stream/0029-stream.md

+|---|---|---|
+| Version | UInt8 | `1` for this version |
+| ILP Packet Type | UInt8 | ILPv4 packet type this STREAM packet MUST be sent in (`12` for Prepare, `13` for Fulfill, and `14` for Reject). Endpoints MUST discard STREAM packets that comes in on the wrong ILP Packet Type. (This is done to prevent malicious intermediaries from swapping the `data` fields from different valid ILP packets.) |
+| Sequence | VarUInt | Identifier for an ILP request / resopnse. Clients and Servers track their own outgoing packet sequence numbers and increment the `Sequence` for each ILP Prepare they send. The Receiver MUST respond with a STREAM packet that includes the same `Sequence` as the Sender's Prepare packet. A sender MUST discard a STREAM packet in which the `Sequence` does not match the STREAM packet sent with their ILP Prepare. |


Minor typo resopnse

adrianhopebailie · 2018-05-14T20:51:33Z

Ship it!

emschwartz requested review from adrianhopebailie, justmoon, michielbdejong, sentientwaffle, dora-gt, sharafian, rhuairahrighairidh and karzak April 2, 2018 21:22

sharafian requested changes Apr 2, 2018

View reviewed changes

michielbdejong previously requested changes Apr 3, 2018

View reviewed changes

emschwartz mentioned this pull request Apr 3, 2018

Proposal: Duplex PSK #388

Closed

dappelt reviewed Apr 3, 2018

View reviewed changes

sentientwaffle reviewed Apr 3, 2018

View reviewed changes

adrianhopebailie approved these changes Apr 3, 2018

View reviewed changes

sharafian approved these changes Apr 3, 2018

View reviewed changes

dappelt reviewed Apr 4, 2018

View reviewed changes

rhuairahrighairidh approved these changes Apr 5, 2018

View reviewed changes

dora-gt reviewed Apr 8, 2018

View reviewed changes

emschwartz mentioned this pull request Apr 10, 2018

Idea: Non-repudiable payment receipts for chunked payments #421

Closed

emschwartz mentioned this pull request May 8, 2018

Drop connection when it has reached a certain number of packets interledgerjs/ilp-protocol-stream#21

Closed

emschwartz force-pushed the es-stream branch from f225bee to 9d81cde Compare May 10, 2018 03:37

emschwartz requested a review from traviscrist May 10, 2018 03:38

emschwartz force-pushed the es-stream branch from 9d81cde to ff52967 Compare May 10, 2018 03:40

emschwartz changed the title ~~[WIP] Add ILP/STREAM protocol~~ Add STREAM Transport Protocol May 10, 2018

emschwartz mentioned this pull request May 10, 2018

Update RFC interledgerjs/ilp-protocol-stream#13

Closed

adrianhopebailie approved these changes May 10, 2018

View reviewed changes

sharafian approved these changes May 11, 2018

View reviewed changes

emschwartz added 12 commits May 14, 2018 13:52

docs(asn1): add initial ASN.1 for ILP/STREAM

eb8496c

docs(stream): initial write-up of ILP/STREAM

2e5c761

docs(stream): add padding

109f1f3

docs(stream): incorporate feedback from @sharafian

5719409

docs(stream): minor text edits

337929e

based on feedback from @dappelt, @sentientwaffle, @adrianhopebailie

docs(stream): write-up of stream v1

6a6d410

- combines data and money streams into one type of stream - details frame types - incorporates feedback from @dora-gt, @rhuairahrighairidh, @dappelt

docs(stream): incorporate feedback from @adrianhopebailie

da17480

docs(stream): make title fit on one line

ef13686

docs(stream): specify string encoding for key derivation strings

d09cac5

docs(stream): explain sender/receiver, fix error codes

74f106a

incorporates feedback from @dora-gt

docs(stream): address @sharafian's comments

a802b08

docs(stream): assign RFC number 29

55c8dc4

emschwartz force-pushed the es-stream branch from 4cf9a11 to 55c8dc4 Compare May 14, 2018 17:55

traviscrist reviewed May 14, 2018

View reviewed changes

traviscrist and others added 2 commits May 14, 2018 14:57

fix: minor document corrections

3aeb84f

docs(stream): add abstract and table of contents

87fbd53

emschwartz merged commit 1f86c78 into master May 15, 2018

emschwartz deleted the es-stream branch May 15, 2018 16:28


		STREAM accounts for value sent over a Connection using the Money Stream abstraction. Money Streams are bidirectional, meaning they can be used to send value from the Client to the Server or from the Server to the Client. A Connection may include any number of Money Streams.

		Multiple Money Streams MAY be used to separately account for value sent and received for different purposes. A Money Stream can be long-lived and the send and receive limits MAY be adjusted over time (for example, in response to an application-level action) or a Money Stream can be used to account for a single logical payment, in which case it would be closed when the payment is finished.


		### Multiplexed Data Streams

		STREAM also offers the ability to send data through the Connection over ILP (rather than over the Internet). Data Streams are the abstraction used for grouping bytes sent over the connection. Data Streams are heavily inspired by [QUIC Bidirectional Streams](https://tools.ietf.org/html/draft-ietf-quic-transport-10#section-10).


		### Stream and Connection Flow Control

		STREAM implements stream- and connection-level flow control. Clients and Servers communicate the maximum amount of money or data they are willing to receive on a given stream or for the connection as a whole. Limits may be changed and retransmitted throughout the life of the Connection to accept more money or data as desired by the application.


		All STREAM packets are intended to be sent in the `data` of an [ILP Prepare, Fulfill, or Reject](../0027-interledger-protocol-4/0027-interledger-protocol-4.md) packet.

		### Encryption


		### Setup

		Before using STREAM, the Client and Server use an authenticated, encrypted communication channel (generally provided as part of an [Application Layer Protocol](../0001-interledger-architecture/0001-interledger-architecture.md#application-layer)) to exchange:


		STREAM accounts for value sent over a Connection using the Money Stream abstraction. Money Streams are bidirectional, meaning they can be used to send value from the Client to the Server or from the Server to the Client. A Connection may include any number of Money Streams.

		Money Streams use absolute amounts to track the total amounts sent and received, as well as the limits for sending and receiving. For example, the Sender might set their send maximum to 1000 to send 1000 units and the Connection would keep sending ILP Prepare packets until the total sent is greater than or equal to 1000.


		All STREAM packets are encrypted using [AES-256-GCM](https://en.wikipedia.org/wiki/Galois/Counter_Mode) with a 12-byte Initialization Vector (IV) and a 16-Byte Authentication Tag.

		If subsequent versions support additional encryption algorithms, those details should be exchanged between the sender and receiver when they establish the shared secret. If a receiver attempts to decrypt an incoming packet but is unable to (perhaps because the sender is using an unsupported cipher), they SHOULD reject the incoming transfer with an `F06: Unexpected Payment` error.


		#### Padding

		Zero (hex `0x00`) bytes MAY be appended after the other frames, for example to ensure that all packets are the same size to minimize the information gained from passive packet analysis. A parser SHOULD stop reading once it reaches a zero byte instead of a Frame Type Identifier.

		Clients and Servers MAY change the address the other entity will send packets to at any time by sending a `ConnectionNewAddress` frame with their new ILP address.

		If one entity changes their address, the other SHOULD NOT assume that the asset is the same or that the exchange rate will be similar.


		# STREAM: A Multiplexed Money and Data Transport Protocol

		ILP/STREAM, or the STREAMing Transport for the Real-time Exchange of Assets and Messages, is an Interledger transport protocol that provides for multiplexed streams of money and data over a virtual ILP connection. STREAM is a successor to [PSK2](../0025-pre-shared-key-2/0025-pre-shared-key-2.md) and takes significant inspiration from the [QUIC](https://tools.ietf.org/html/draft-ietf-quic-transport-10) Internet transport protocol.


		## Definitions

		- Client - One of the two entities using STREAM. The Client initiates a Connection to the Server using STREAM packets sent over ILP. Because STREAM uses bidirectional streams, the Client may be either the Sender or the Receiver of a given packet.

		- Server - One of the two entities using STREAM. The Server listens through an ILP account for incoming Connections from Clients. Note that STREAM Servers DO NOT listen for Connections over the Internet, but rather over Interledger. Because STREAM uses bidirectional streams, the Server may be either the Sender or the Receiver of a given packet.

		## Overview


		In an application that uses an ongoing flow of money, an endpoint can open a single stream and continue sending money on it for the duration of some paid interaction.

		In another another application, an endpoint can open a stream, send a specific amount of money through it, and then close the stream to indicate the payment is complete. In this case, the stream abstraction provides a mechanism to "frame" a payment or message that may be split across multiple packets. New streams can be opened and closed on the same connection to send multiple messages.


		STREAM uses bidirectional streams, which can be used for request/response flows or one-way messages.

		The choice streams as the key abstraction is inspired by [QUIC](https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#rfc.section.9) and the earlier [Structured Streams Transport (SST)](http://www.brynosaurus.com/pub/net/sst.pdf).


		#### 4.4.5. Closing Streams

		Either endpoint can close a stream using a `StreamClose` frame. Implementations MAY allow half-open streams.


		### 4.6. Closing Connections

		Either endpoint can close the connection using a `ConnectionClose` frame. Implementations MAY allow half-open connections.


		#### 5.1.2. Encryption Pseudocode

		The encryption key used for every packet sent for a given connection is derived from the shared secret and the string `"ilp_stream_encryption"` using an HMAC.


		The amount of money that should go to each stream is calculated by dividing the number of shares for the given stream by the total number of shares in all of the `StreamMoney` frames in the packet.

		For example, if an ILP Prepare packet has an amount of 100 and three `StreamMoney` frames with 5, 15, and 30 shares for streams 2, 4, and 6, respectively, that would indicate that stream 2 should get 10 units, stream 4 gets 30 units, and stream 6 gets 60 units.


		Streams are given numerical IDs. Client-initiated streams are given odd numbers starting with 1 and server-initiated streams are given even numbers starting with 2 (this is used to avoid collisions if both endpoints open streams at the same time).

		Endpoints can limit the number of concurrently active incoming streams by adjusting the maximum stream ID and communicating that limit to the other endpoint.


		### 3.4. Exchange Rates

		A critical function of Interledger Transport Protocols is to determine the path exchange rate and handle any changes in the rate. STREAM packets include a minimum acceptable amount in ILP Prepare packets and the amount that arrived at the other endpoint in the Fulfill or Reject packet.

Add STREAM Transport Protocol #417

Add STREAM Transport Protocol #417

Conversation

emschwartz commented Apr 2, 2018

sharafian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michielbdejong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emschwartz commented Apr 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrianhopebailie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhuairahrighairidh left a comment

Choose a reason for hiding this comment

dora-gt Apr 8, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dora-gt Apr 9, 2018 • edited

Choose a reason for hiding this comment

dora-gt Apr 8, 2018 • edited

Choose a reason for hiding this comment

emschwartz commented May 10, 2018

adrianhopebailie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

Choose a reason for hiding this comment

emschwartz May 10, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sharafian left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dora-gt Apr 8, 2018 •

edited

dora-gt Apr 9, 2018 •

edited

dora-gt Apr 8, 2018 •

edited

emschwartz May 10, 2018 •

edited

dora-gt May 14, 2018 •

edited


		When a client connects to a server, they MUST communicate their ILP address to the server using a `ConnectionNewAddress` frame.

		Either endpoint MAY change their ILP address at any point during a connection by sending a `ConnectionNewAddress` frame.


		Either endpoint MAY change their ILP address at any point during a connection by sending a `ConnectionNewAddress` frame.

		An attacker could cause an unsuspecting endpoint to participate in a Denial of Service (DoS) attack on a third-party endpoint. To mitigate this, endpoints SHOULD refrain from sending large numbers of packets or large amounts of data to a new ILP address before validating that the ILP address is controlled by the same party that knows the shared secret. STREAM uses the authenticated request/response packets in lieu of [QUIC's explicit Path Validation](https://quicwg.github.io/base-drafts/draft-ietf-quic-transport.html#rfc.section.6.7).


		### 5.1. Encryption

		All STREAM packets are encrypted using [AES-256-GCM](https://en.wikipedia.org/wiki/Galois/Counter_Mode) with a 12-byte Initialization Vector (IV) and a 16-Byte Authentication Tag.


		#### 5.1.3. Maximum Number of Packets Per Connection

		Implementations MUST close the connection once either endpoint has sent 2^31 packets. According to [NIST](https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38d.pdf), it is unsafe to use AES-GCM for more than 2^32 packets using the same encryption key.