From a15275602b75b12b7589c9df862c97386c39c6a7 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sun, 4 Feb 2018 21:03:04 -0800 Subject: [PATCH 01/23] skeleton of wire protocol DEP --- proposals/0000-wire-protocol.md | 360 ++++++++++++++++++++++++++++++++ 1 file changed, 360 insertions(+) create mode 100644 proposals/0000-wire-protocol.md diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md new file mode 100644 index 0000000..fa8ac14 --- /dev/null +++ b/proposals/0000-wire-protocol.md @@ -0,0 +1,360 @@ + +Title: **DEP-0000: Wire Protocol** + +Short Name: `0000-wire-protocol` + +Type: Standard + +Status: Undefined (as of 2018-02-04) + +Github PR: (add HTTPS link here after PR is opened) + +Authors: [Paul Frazee](https://github.com/pfrazee), +[Bryan Newbold](https://github.com/bnewbold) + + +# Summary +[summary]: #summary + +This DEP describes the Dat wire protocol: a transport-agnostic message stream +spoken between nodes in a swarm of hypercore network peers (including Dat +clients). The wire protocol includes mechanisms for framing, stream encryption, +and feed key authentication. + + +# Motivation +[motivation]: #motivation + +The protocol described here is already in use as of 2017 (by hypercore, Dat, +and Beaker Browser users), and was partially described in an earlier +[whitepaper][whitepaper]. This document fills in some additional details. + +[whitepaper]: https://TODO + + +# Stream Connections +[stream-details]: #stream-details + +The Dat wire protocol depends on a lower binary transport channel which +provides the following semantics: + +- reliable delivery (no dropped messages, or partial messages) +- in-order delivery of messages + +Peers wishing to connect need to discover each other using some mechanism or +another (see forthcoming DEPs on some options; this process is modular and +swappable), and both need to have the public key for the primary hypercore they +wish to exchange. + +Messages are framed by the Dat protocol itself (see Messages section for +details). + + +## Channels +[channels]: #channels + +Multiple hypercore registers can be synchronized over the same protocol +connection. Messages pertaining to the separate registers (aka, "Feeds", +"channels") are tagged with an id for disambiguation. + +Note that at least one feed is necessary for each connection (for handshaking +to succeed), and that the first feed is the one used for discovery and +as an encryption key. + +To initiate a new channel (after the primary is established), + +## Handshake Procedure +[handshake]: #handshake + +A handshake procedure needs to occur for each feed on a channel; the first part +of the first handshake happens in cleartext and both validates discovery keys +and establishes encyption paramters used for the rest of the connection. The +first (primary) channel has `id=0`. + +The first (cleartext) message is a Feed message, and includes two fields: a +nonce and a discovery key. + +The **nonce** is generated by each peer as a random 32-byte sequence. + +The **discovery key** is generated from the public encryption key for a +hypercore register (in this case the first, or "primary" register) by using the +public key to sign the 9-byte ASCII string "hypercore" (with no trailing NULL +byte) using the BLAKE2b keyed hashing algorithm (provided by most BLAKE2b hash +implementations). The discovery key is 32 bytes long. + +The discovery key is used in cleartext instead of the public key to avoid +leaking the public key to the network; read access to hypercore registers +(including Dat archives) is controlled by limiting access to public keys. + +When the connection is first opened, the connecting peer sends their Feed +message. The receiving peer checks that the discovery key was what they were +expecting (eg, that they know of a public key matching that discovery key and +are willing to synchronize the register associated with that key). If so, they +reply with their own Feed. If not, they drop the connection. + +Once Feed messages are exchanged, both peers have all information they need to +encrypt all further content on the channel, and do so (see below for details). +The second part of the handshake is to exchange Handshake messages, which set +some parameters for the channel. Handshakes also include the self-identified +peer id, which can be used to detect accidental self-connections or redundant +connections to the same peer (eg, over different transports). + + +## Encryption Scheme +[encryption]: #encryption + +After the first Feed messages are exchanged (one message in each direction, in +cleartext), all further bytes exchanged over the channel are encrypted. + +Framing metadata (aka, message length and type) is encrypted, but a third party +could likely infer message lengths (and thus potentially message types) by +observing packet sizes and timing; no padding is applied at the protocol layer. + +The encryption scheme used is libsodium's stream primative, specifically the +XSalsa20 cipher. The cipher is fed a shared key (the primary hypercore register +public key), a nonce (selected by the sender and exchanged during handshake), +and a block offset (representing all encrypted bytes sent on the connection in +this direction so far). + +*TODO: the following paragraph gets in to implementation details... some mention +of the 64 byte chunks is needed, but maybe not this much detail?* + +The specific libsodium function used is usually +`crypto_stream_xsalsa20_xor_ic()`. Some interfacing code is necessary to +process messages that don't align with the cipher's 64-byte chunk size; unused +bytes in any particular chunk can be ignored. For example, if 1000 encrypted +bytes had been sent on a connection already, and then a new 50 byte message +needed to be encrypted and sent, then one would offset the message by `1000 % +64 = 40` bytes and XOR the first 24 bytes against block 15, then XOR the +remaining 26 bytes against block 16. The bytes would be shifted back and +recombined before sending, so only 50 bytes would go down the connection; the +same process would be followed by the receiver. + + +# Message Details +[message-details]: #message-details + +TODO: description of framing + +Wire format is `(
)`. `header` is a varint, of form +`channel << 4 | <4-bit-type>`. + +Messages are encoded (serialized) using Google's [profobuf][protobuf] encoding. + +[protobuf]: https://TODO + + +
`type` code Name +
N/A [Keep-Alive][msg-keepalive] +
0 [Feed][msg-feed] +
1 [Handshake][msg-handshake] +
2 [Info][msg-info] +
3 [Have][msg-have] +
4 [Unhave][msg-unhave] +
5 [Want][msg-want] +
6 [Unwant][msg-unwant] +
7 [Request][msg-request] +
8 [Cancel][msg-cancel] +
9 [Data][msg-data] +
15 [Extension][msg-extension] +
+ +#### Keep-Alive +[msg-keepalive]: #msg-keepalive + +A message of body length 0 (giving a total message size of 1 byte for the `len` +varint) is a keep-alive. Depending on transport and application needs, peers +may optionally send keep-alive messages to help detect and prevent channel +loss. Peers must always handle keep-alive messages correctly (aka, ignore +them), regardless of transport. + +TODO: what is a good default interval? + +#### Feed +[msg-feed]: #msg-feed + + // type=0, should be the first message sent on a channel + message Feed { + required bytes discoveryKey = 1; + optional bytes nonce = 2; + } + +#### Handshake +[msg-handshake]: #msg-handshake + + // type=1, overall connection handshake. should be send just after the feed message on the first channel only + message Handshake { + optional bytes id = 1; + optional bool live = 2; // keep the connection open forever? both ends have to agree + optional bytes userData = 3; + repeated string extensions = 4; + } + +TODO: What are semantics of 'live' bit? what if there is disagreement? + +#### Info +[msg-info]: #msg-info + + // type=2, message indicating state changes etc. + // initial state for uploading/downloading is true + // if both ends are not downloading and not live it is safe to consider the stream ended + message Info { + optional bool uploading = 1; + optional bool downloading = 2; + } + +#### Have +[msg-have]: #msg-have + + // type=3, what do we have? + message Have { + required uint64 start = 1; + optional uint64 length = 2 [default = 1]; // defaults to 1 + optional bytes bitfield = 3; + } + +#### Unhave +[msg-unhave]: #msg-unhave + + // type=4, what did we lose? + message Unhave { + required uint64 start = 1; + optional uint64 length = 2 [default = 1]; // defaults to 1 + } + +#### Want +[msg-want]: #msg-want + + // type=5, what do we want? remote should start sending have messages in this range + message Want { + required uint64 start = 1; + optional uint64 length = 2; // defaults to Infinity or feed.length (if not live) + } + +#### Unwant +[msg-unwant]: #msg-unwant + + // type=6, what don't we want anymore? + message Unwant { + required uint64 start = 1; + optional uint64 length = 2; // defaults to Infinity or feed.length (if not live) + } + +#### Request +[msg-request]: #msg-request + + // type=7, ask for data + message Request { + required uint64 index = 1; + optional uint64 bytes = 2; + optional bool hash = 3; + optional uint64 nodes = 4; + } + +#### Cancel +[msg-cancel]: #msg-cancel + + // type=8, cancel a request + message Cancel { + required uint64 index = 1; + optional uint64 bytes = 2; + optional bool hash = 3; + } + +#### Data +[msg-data]: #msg-data + + // type=9, get some data + message Data { + message Node { + required uint64 index = 1; + required bytes hash = 2; + required uint64 size = 3; + } + required uint64 index = 1; + optional bytes value = 2; + repeated Node nodes = 3; + optional bytes signature = 4; + } + +#### Extension +[msg-extension]: #msg-extension + +`type=15` is an extension message that is encoded like: + + + +# Examples + +## Simple Download + +Alyssa P Hacker and Ben Bitdiddle want to share a book... B connects to A. + +- full public key and discovery key for the connection +- example nonces (in full) +- messages in "struct" syntax and raw hex: + - B: sends Feed + - A: replies Feed + - B: Handshake (downloading, not live) + - A: Handshake (uploading, not live) + - B: Info: downloading only + - A: Info: uploading only + - B: Have: nothing + - A: Have: everything + - B: Want: first register entry + - A: Data: first entry + - (repeat for all other chunks) +- connection closes + +## Multiple Feeds + +Describe in detail how to "add" a new channel (feed/register) to an existing +connection, using Feed (and Handshake?) messages. + +## Swarm Synchronization + +TODO: should this more involved example actually live here? or in hypercore +DEP? It feels pretty message-level, but does involve more hypercore semantics. + +This example wouldn't include actual messages, but would describe an N-way (3+) +node swarm, with a single (complete) seeder, two peers that both download from +the seeder and exchange messages, and a fourth peer that downloads from one of +the non-seeder peers only. + +- Peer A: seeder, writer. Starts with full history and appends to log +- Peer B: swarm, reader. Starts with full history. Live connection. Connected + to A, C, D. +- Peer C: swarm, reader. Starts with sparse (old) history. Only wants "latest" + data. Connected to A and, B. +- Peer D: like Peer C, but only connected to B. + +# Rationale and alternatives +[alternatives]: #alternatives + +- Why is this design the best in the space of possible designs? +- What other designs have been considered and what is the rationale for not choosing them? +- What is the impact of not doing this? + + +# Unresolved questions +[unresolved]: #unresolved-questions + +What are extension strings? What can 'userData' bytes be used for? + +Encryption might not make sense in some contexts (eg, IPC, or if the transport +layer is already providing encryption). Should this DEP recognize this +explicitly? + +- What parts of the design do you expect to resolve through the DEP consensus process before this gets merged? +- What parts of the design do you expect to resolve through implementation and code review, or are left to independent library or application developers? +- What related issues do you consider out of scope for this DEP that could be addressed in the future independently of the solution that comes out of this DEP? + + +# Changelog +[changelog]: #changelog + +A brief statemnt about current status can go here, follow by a list of dates +when the status line of this DEP changed (in most-recent-last order). + +- YYYY-MM-DD: First complete draft submitted for review + From 79604398ad9bcfba1109e6a3aed0d2d15cb48a4b Mon Sep 17 00:00:00 2001 From: Paul Frazee Date: Mon, 5 Feb 2018 21:00:31 -0600 Subject: [PATCH 02/23] Add "Block Tree Digest" to 000-wire-protocol --- proposals/0000-wire-protocol.md | 47 +++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index fa8ac14..1dbbc95 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -131,6 +131,53 @@ recombined before sending, so only 50 bytes would go down the connection; the same process would be followed by the receiver. +## Block Tree Digest +[block-tree-digest]: #block-tree-digest + +When asking for a block of data we want to reduce the amount of duplicate hashes that are sent back. To communicate which hashes we have, we just have to communicate two things: which uncles we have and whether or not we have any parent node that can verify the tree. + +Consider the following tree: + +``` +0 + 1 +2 + 3 +4 + 5 +6 +``` + +If we want to fetch block 0, we need to communicate whether of not we already have the uncles (2, 5) and the parent (3). This information can be compressed into very small bit vector using the following scheme: + + - Let the trailing bit denote whether or not the leading bit is a parent and not a uncle. + - Let the previous trailing bits denote whether or not we have the next uncle. + +Let's consider an example. Suppose we want to fetch block 0, and we have 2 and 3 but not 5. We need to therefore communicate: + +``` +the leading bit is a parent, not an uncle +we already have the first uncle, 2 so don't send us that +we don't have the next uncle, 5 +we have the next parent, 3 +``` + +We would encode this into the bit vector `1011`. Decoded: + +``` +101(1) <-- the leading bit is a parent, not an uncle +10(1)1 <-- we already have the first uncle, 2 so don't send us that +1(0)11 <-- we don't have the next uncle, 5 +(1)000 <-- we have the next parent, 3 +``` + +So using this digest the recipient can easily figure out that they only need to send us one hash, 5, for us to verify block 0. + +The bit vector 1 (only contains a single one) means that we already have all the hashes we need so just send us the block. + +These digests are very compact in size, only `(log2(number-of-blocks) + 2) / 8` bytes needed in the worst case. For example if you are sharing one trillion blocks of data the digest would be `(log2(1000000000000) + 2) / 8 ~= 6` bytes long. + + # Message Details [message-details]: #message-details From 22db0da3bbcaf8dc07d41e1b812f905808d41eb9 Mon Sep 17 00:00:00 2001 From: Paul Frazee Date: Sun, 4 Mar 2018 13:32:08 -0600 Subject: [PATCH 03/23] Remove manual line-wrapping (sorry bryan) --- proposals/0000-wire-protocol.md | 124 ++++++++------------------------ 1 file changed, 29 insertions(+), 95 deletions(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 1dbbc95..113145c 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -9,25 +9,19 @@ Status: Undefined (as of 2018-02-04) Github PR: (add HTTPS link here after PR is opened) -Authors: [Paul Frazee](https://github.com/pfrazee), -[Bryan Newbold](https://github.com/bnewbold) +Authors: [Paul Frazee](https://github.com/pfrazee), [Bryan Newbold](https://github.com/bnewbold) # Summary [summary]: #summary -This DEP describes the Dat wire protocol: a transport-agnostic message stream -spoken between nodes in a swarm of hypercore network peers (including Dat -clients). The wire protocol includes mechanisms for framing, stream encryption, -and feed key authentication. +This DEP describes the Dat wire protocol: a transport-agnostic message stream spoken between nodes in a swarm of hypercore network peers (including Dat clients). The wire protocol includes mechanisms for framing, stream encryption, and feed key authentication. # Motivation [motivation]: #motivation -The protocol described here is already in use as of 2017 (by hypercore, Dat, -and Beaker Browser users), and was partially described in an earlier -[whitepaper][whitepaper]. This document fills in some additional details. +The protocol described here is already in use as of 2017 (by Hypercore, Dat, and Beaker Browser users), and was partially described in an earlier [whitepaper][whitepaper]. This document fills in some additional details. [whitepaper]: https://TODO @@ -35,100 +29,55 @@ and Beaker Browser users), and was partially described in an earlier # Stream Connections [stream-details]: #stream-details -The Dat wire protocol depends on a lower binary transport channel which -provides the following semantics: +The Dat wire protocol depends on the use of a binary transport channel which provides the following semantics: -- reliable delivery (no dropped messages, or partial messages) +- reliable delivery (no dropped or partial messages) - in-order delivery of messages -Peers wishing to connect need to discover each other using some mechanism or -another (see forthcoming DEPs on some options; this process is modular and -swappable), and both need to have the public key for the primary hypercore they -wish to exchange. +Peers wishing to connect need to discover each other using some mechanism or another (see forthcoming DEPs on some options; this process is modular and swappable), and both need to have the public key for the primary hypercore they wish to exchange. -Messages are framed by the Dat protocol itself (see Messages section for -details). +Messages are framed by the Dat protocol itself (see Messages section for details). ## Channels [channels]: #channels -Multiple hypercore registers can be synchronized over the same protocol -connection. Messages pertaining to the separate registers (aka, "Feeds", -"channels") are tagged with an id for disambiguation. +Multiple hypercore registers can be synchronized over the same protocol connection. Messages pertaining to the separate channels are tagged with an id for disambiguation. -Note that at least one feed is necessary for each connection (for handshaking -to succeed), and that the first feed is the one used for discovery and -as an encryption key. +Note that at least one feed is necessary for each connection (for handshaking to succeed), and that the first feed is the one used for discovery and as an encryption key. To initiate a new channel (after the primary is established), ## Handshake Procedure [handshake]: #handshake -A handshake procedure needs to occur for each feed on a channel; the first part -of the first handshake happens in cleartext and both validates discovery keys -and establishes encyption paramters used for the rest of the connection. The -first (primary) channel has `id=0`. +A handshake procedure needs to occur for each feed on a channel; the first part of the first handshake happens in cleartext and both validates discovery keys and establishes encyption paramters used for the rest of the connection. The first (primary) channel has `id=0`. -The first (cleartext) message is a Feed message, and includes two fields: a -nonce and a discovery key. +The first (cleartext) message is a Feed message, and includes two fields: a nonce and a discovery key. The **nonce** is generated by each peer as a random 32-byte sequence. -The **discovery key** is generated from the public encryption key for a -hypercore register (in this case the first, or "primary" register) by using the -public key to sign the 9-byte ASCII string "hypercore" (with no trailing NULL -byte) using the BLAKE2b keyed hashing algorithm (provided by most BLAKE2b hash -implementations). The discovery key is 32 bytes long. +The **discovery key** is generated from the public encryption key for a hypercore register (in this case the first, or "primary" register) by using the public key to sign the 9-byte ASCII string "hypercore" (with no trailing NULL byte) using the BLAKE2b keyed hashing algorithm (provided by most BLAKE2b hash implementations). The discovery key is 32 bytes long. -The discovery key is used in cleartext instead of the public key to avoid -leaking the public key to the network; read access to hypercore registers -(including Dat archives) is controlled by limiting access to public keys. +The discovery key is used in cleartext instead of the public key to avoid leaking the public key to the network; read access to hypercore registers (including Dat archives) is controlled by limiting access to public keys. -When the connection is first opened, the connecting peer sends their Feed -message. The receiving peer checks that the discovery key was what they were -expecting (eg, that they know of a public key matching that discovery key and -are willing to synchronize the register associated with that key). If so, they -reply with their own Feed. If not, they drop the connection. +When the connection is first opened, the connecting peer sends their Feed message. The receiving peer checks that the discovery key was what they were expecting (eg, that they know of a public key matching that discovery key and are willing to synchronize the register associated with that key). If so, they reply with their own Feed. If not, they drop the connection. -Once Feed messages are exchanged, both peers have all information they need to -encrypt all further content on the channel, and do so (see below for details). -The second part of the handshake is to exchange Handshake messages, which set -some parameters for the channel. Handshakes also include the self-identified -peer id, which can be used to detect accidental self-connections or redundant -connections to the same peer (eg, over different transports). +Once Feed messages are exchanged, both peers have all information they need to encrypt all further content on the channel, and do so (see below for details). The second part of the handshake is to exchange Handshake messages, which set some parameters for the channel. Handshakes also include the self-identified peer id, which can be used to detect accidental self-connections or redundant connections to the same peer (eg, over different transports). ## Encryption Scheme [encryption]: #encryption -After the first Feed messages are exchanged (one message in each direction, in -cleartext), all further bytes exchanged over the channel are encrypted. +After the first Feed messages are exchanged (one message in each direction, in cleartext), all further bytes exchanged over the channel are encrypted. -Framing metadata (aka, message length and type) is encrypted, but a third party -could likely infer message lengths (and thus potentially message types) by -observing packet sizes and timing; no padding is applied at the protocol layer. +Framing metadata (aka, message length and type) is encrypted, but a third party could likely infer message lengths (and thus potentially message types) by observing packet sizes and timing; no padding is applied at the protocol layer. -The encryption scheme used is libsodium's stream primative, specifically the -XSalsa20 cipher. The cipher is fed a shared key (the primary hypercore register -public key), a nonce (selected by the sender and exchanged during handshake), -and a block offset (representing all encrypted bytes sent on the connection in -this direction so far). +The encryption scheme used is libsodium's stream primative, specifically the XSalsa20 cipher. The cipher is fed a shared key (the primary hypercore register public key), a nonce (selected by the sender and exchanged during handshake), and a block offset (representing all encrypted bytes sent on the connection in this direction so far). -*TODO: the following paragraph gets in to implementation details... some mention -of the 64 byte chunks is needed, but maybe not this much detail?* +*TODO: the following paragraph gets in to implementation details... some mention of the 64 byte chunks is needed, but maybe not this much detail?* -The specific libsodium function used is usually -`crypto_stream_xsalsa20_xor_ic()`. Some interfacing code is necessary to -process messages that don't align with the cipher's 64-byte chunk size; unused -bytes in any particular chunk can be ignored. For example, if 1000 encrypted -bytes had been sent on a connection already, and then a new 50 byte message -needed to be encrypted and sent, then one would offset the message by `1000 % -64 = 40` bytes and XOR the first 24 bytes against block 15, then XOR the -remaining 26 bytes against block 16. The bytes would be shifted back and -recombined before sending, so only 50 bytes would go down the connection; the -same process would be followed by the receiver. +The specific libsodium function used is usually `crypto_stream_xsalsa20_xor_ic()`. Some interfacing code is necessary to process messages that don't align with the cipher's 64-byte chunk size; unused bytes in any particular chunk can be ignored. For example, if 1000 encrypted bytes had been sent on a connection already, and then a new 50 byte message needed to be encrypted and sent, then one would offset the message by `1000 % 64 = 40` bytes and XOR the first 24 bytes against block 15, then XOR the remaining 26 bytes against block 16. The bytes would be shifted back and recombined before sending, so only 50 bytes would go down the connection; the same process would be followed by the receiver. ## Block Tree Digest @@ -183,8 +132,7 @@ These digests are very compact in size, only `(log2(number-of-blocks) + 2) / 8` TODO: description of framing -Wire format is `(
)`. `header` is a varint, of form -`channel << 4 | <4-bit-type>`. +Wire format is `(
)`. `header` is a varint, of form `channel << 4 | <4-bit-type>`. Messages are encoded (serialized) using Google's [profobuf][protobuf] encoding. @@ -209,11 +157,7 @@ Messages are encoded (serialized) using Google's [profobuf][protobuf] encoding. #### Keep-Alive [msg-keepalive]: #msg-keepalive -A message of body length 0 (giving a total message size of 1 byte for the `len` -varint) is a keep-alive. Depending on transport and application needs, peers -may optionally send keep-alive messages to help detect and prevent channel -loss. Peers must always handle keep-alive messages correctly (aka, ignore -them), regardless of transport. +A message of body length 0 (giving a total message size of 1 byte for the `len` varint) is a keep-alive. Depending on transport and application needs, peers may optionally send keep-alive messages to help detect and prevent channel loss. Peers must always handle keep-alive messages correctly (aka, ignore them), regardless of transport. TODO: what is a good default interval? @@ -355,24 +299,17 @@ Alyssa P Hacker and Ben Bitdiddle want to share a book... B connects to A. ## Multiple Feeds -Describe in detail how to "add" a new channel (feed/register) to an existing -connection, using Feed (and Handshake?) messages. +Describe in detail how to "add" a new channel (feed/register) to an existing connection, using Feed (and Handshake?) messages. ## Swarm Synchronization -TODO: should this more involved example actually live here? or in hypercore -DEP? It feels pretty message-level, but does involve more hypercore semantics. +TODO: should this more involved example actually live here? or in hypercore DEP? It feels pretty message-level, but does involve more hypercore semantics. -This example wouldn't include actual messages, but would describe an N-way (3+) -node swarm, with a single (complete) seeder, two peers that both download from -the seeder and exchange messages, and a fourth peer that downloads from one of -the non-seeder peers only. +This example wouldn't include actual messages, but would describe an N-way (3+) node swarm, with a single (complete) seeder, two peers that both download from the seeder and exchange messages, and a fourth peer that downloads from one of the non-seeder peers only. - Peer A: seeder, writer. Starts with full history and appends to log -- Peer B: swarm, reader. Starts with full history. Live connection. Connected - to A, C, D. -- Peer C: swarm, reader. Starts with sparse (old) history. Only wants "latest" - data. Connected to A and, B. +- Peer B: swarm, reader. Starts with full history. Live connection. Connected to A, C, D. +- Peer C: swarm, reader. Starts with sparse (old) history. Only wants "latest" data. Connected to A and, B. - Peer D: like Peer C, but only connected to B. # Rationale and alternatives @@ -388,9 +325,7 @@ the non-seeder peers only. What are extension strings? What can 'userData' bytes be used for? -Encryption might not make sense in some contexts (eg, IPC, or if the transport -layer is already providing encryption). Should this DEP recognize this -explicitly? +Encryption might not make sense in some contexts (eg, IPC, or if the transport layer is already providing encryption). Should this DEP recognize this explicitly? - What parts of the design do you expect to resolve through the DEP consensus process before this gets merged? - What parts of the design do you expect to resolve through implementation and code review, or are left to independent library or application developers? @@ -400,8 +335,7 @@ explicitly? # Changelog [changelog]: #changelog -A brief statemnt about current status can go here, follow by a list of dates -when the status line of this DEP changed (in most-recent-last order). +A brief statemnt about current status can go here, follow by a list of dates when the status line of this DEP changed (in most-recent-last order). - YYYY-MM-DD: First complete draft submitted for review From c820c76f97dce18eb7b32bbc33b46846ca48dd29 Mon Sep 17 00:00:00 2001 From: Paul Frazee Date: Sun, 4 Mar 2018 16:37:35 -0600 Subject: [PATCH 04/23] More work on 000-wire-protocol.md --- proposals/0000-wire-protocol.md | 402 ++++++++++++++++++++------------ 1 file changed, 248 insertions(+), 154 deletions(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 113145c..402fb55 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -1,5 +1,5 @@ -Title: **DEP-0000: Wire Protocol** +Title: **DEP-0000: Hypercore Wire Protocol** Short Name: `0000-wire-protocol` @@ -15,7 +15,7 @@ Authors: [Paul Frazee](https://github.com/pfrazee), [Bryan Newbold](https://gith # Summary [summary]: #summary -This DEP describes the Dat wire protocol: a transport-agnostic message stream spoken between nodes in a swarm of hypercore network peers (including Dat clients). The wire protocol includes mechanisms for framing, stream encryption, and feed key authentication. +This DEP describes the Hypercore wire protocol: a transport-agnostic message stream spoken between nodes in a swarm of hypercore network peers (including Dat clients). The wire protocol includes mechanisms for framing, stream encryption, and feed key authentication. # Motivation @@ -23,13 +23,13 @@ This DEP describes the Dat wire protocol: a transport-agnostic message stream sp The protocol described here is already in use as of 2017 (by Hypercore, Dat, and Beaker Browser users), and was partially described in an earlier [whitepaper][whitepaper]. This document fills in some additional details. -[whitepaper]: https://TODO +[whitepaper]: https://github.com/datprotocol/whitepaper # Stream Connections [stream-details]: #stream-details -The Dat wire protocol depends on the use of a binary transport channel which provides the following semantics: +The Dat wire protocol depends on the use of a binary transport protocol which provides the following semantics: - reliable delivery (no dropped or partial messages) - in-order delivery of messages @@ -42,28 +42,38 @@ Messages are framed by the Dat protocol itself (see Messages section for details ## Channels [channels]: #channels -Multiple hypercore registers can be synchronized over the same protocol connection. Messages pertaining to the separate channels are tagged with an id for disambiguation. +Multiple hypercore feeds can be synchronized over the same protocol connection. Messages pertaining to the separate channels are tagged with an id for disambiguation. Note that at least one feed is necessary for each connection (for handshaking to succeed), and that the first feed is the one used for discovery and as an encryption key. -To initiate a new channel (after the primary is established), +To initiate a new channel (after the primary is established), TODO + ## Handshake Procedure [handshake]: #handshake -A handshake procedure needs to occur for each feed on a channel; the first part of the first handshake happens in cleartext and both validates discovery keys and establishes encyption paramters used for the rest of the connection. The first (primary) channel has `id=0`. +A handshake procedure needs to occur for each feed on a channel; the first part of the first handshake happens in cleartext and both validates discovery keys and establishes encyption parameters used for the rest of the connection. The first (primary) channel has `id=0`. The first (cleartext) message is a Feed message, and includes two fields: a nonce and a discovery key. The **nonce** is generated by each peer as a random 32-byte sequence. -The **discovery key** is generated from the public encryption key for a hypercore register (in this case the first, or "primary" register) by using the public key to sign the 9-byte ASCII string "hypercore" (with no trailing NULL byte) using the BLAKE2b keyed hashing algorithm (provided by most BLAKE2b hash implementations). The discovery key is 32 bytes long. +The **discovery key** is generated from the public encryption key for a hypercore feed (in this case the first, or "primary" feed) by using the public key to sign the 9-byte ASCII string "HYPERCORE" (with no trailing NULL byte) using the BLAKE2b keyed hashing algorithm (provided by most BLAKE2b hash implementations). The discovery key is 32 bytes long. + +```js +var discoveryKey = new Buffer(32) +sodium.crypto_generichash( + discoveryKey, // out + 'HYPERCORE', // in (message) + publicKey // key +) +``` -The discovery key is used in cleartext instead of the public key to avoid leaking the public key to the network; read access to hypercore registers (including Dat archives) is controlled by limiting access to public keys. +The discovery key is used in cleartext instead of the public key to avoid leaking the public key to the network; read access to hypercore feeds (including Dat archives) is controlled by limiting access to public keys. -When the connection is first opened, the connecting peer sends their Feed message. The receiving peer checks that the discovery key was what they were expecting (eg, that they know of a public key matching that discovery key and are willing to synchronize the register associated with that key). If so, they reply with their own Feed. If not, they drop the connection. +When the connection is first opened, the connecting peer sends their Feed message. The receiving peer checks that the discovery key was expected (eg, that they know of a public key matching that discovery key and are willing to synchronize the feed associated with that key). If so, they reply with their own Feed message. If not, they drop the connection. -Once Feed messages are exchanged, both peers have all information they need to encrypt all further content on the channel, and do so (see below for details). The second part of the handshake is to exchange Handshake messages, which set some parameters for the channel. Handshakes also include the self-identified peer id, which can be used to detect accidental self-connections or redundant connections to the same peer (eg, over different transports). +Once Feed messages are exchanged, both peers have all information they need to encrypt all further content on the channel, and do so (see below for details). The second part of the handshake is to exchange Handshake messages, which set some parameters for the channel. Handshakes also include the self-identified peer id, which can be used to detect accidental self-connections or redundant connections to the same peer (eg, over different transports). The peer id is typically random and is not authenticated. ## Encryption Scheme @@ -73,58 +83,35 @@ After the first Feed messages are exchanged (one message in each direction, in c Framing metadata (aka, message length and type) is encrypted, but a third party could likely infer message lengths (and thus potentially message types) by observing packet sizes and timing; no padding is applied at the protocol layer. -The encryption scheme used is libsodium's stream primative, specifically the XSalsa20 cipher. The cipher is fed a shared key (the primary hypercore register public key), a nonce (selected by the sender and exchanged during handshake), and a block offset (representing all encrypted bytes sent on the connection in this direction so far). +The encryption scheme used is libsodium's stream primitive, specifically the XSalsa20 cipher. The cipher is fed a shared key (the primary hypercore feed public key), a nonce (selected by the sender and exchanged during handshake), and a block offset (representing all encrypted bytes sent on the connection in this direction so far). -*TODO: the following paragraph gets in to implementation details... some mention of the 64 byte chunks is needed, but maybe not this much detail?* +The specific libsodium function used is `crypto_stream_xsalsa20_xor_ic()`. Some interfacing code is necessary to process messages that don't align with the cipher's 64-byte chunk size; unused bytes in any particular chunk can be ignored. For example, if 1000 encrypted bytes had been sent on a connection already, and then a new 50 byte message needed to be encrypted and sent, then one would offset the message by `1000 % 64 = 40` bytes and XOR the first 24 bytes against block 15, then XOR the remaining 26 bytes against block 16. The bytes would be shifted back and recombined before sending, so only 50 bytes would go down the connection; the same process would be followed by the receiver. -The specific libsodium function used is usually `crypto_stream_xsalsa20_xor_ic()`. Some interfacing code is necessary to process messages that don't align with the cipher's 64-byte chunk size; unused bytes in any particular chunk can be ignored. For example, if 1000 encrypted bytes had been sent on a connection already, and then a new 50 byte message needed to be encrypted and sent, then one would offset the message by `1000 % 64 = 40` bytes and XOR the first 24 bytes against block 15, then XOR the remaining 26 bytes against block 16. The bytes would be shifted back and recombined before sending, so only 50 bytes would go down the connection; the same process would be followed by the receiver. +## Want/Have Procedure +[want-have]: #want-have -## Block Tree Digest -[block-tree-digest]: #block-tree-digest +The wire protocol is designed to be efficient when syncing extremely large Hypercore feeds (ie millions of blocks in length). Therefore a procedure exists for each peer to indicate which subset of the feed they would like to synchronize, and for the remote to then announce which blocks within those subsets they possess. This is the Want/Have Procedure. -When asking for a block of data we want to reduce the amount of duplicate hashes that are sent back. To communicate which hashes we have, we just have to communicate two things: which uncles we have and whether or not we have any parent node that can verify the tree. +By default, a peer wants no blocks. It can add or remove wanted block using the Want and Unwant messages. -Consider the following tree: +When new wanted blocks are made available, the peer should react by sending a Have message. Likewise, when wanted blocks are no longer available, the peer should react by sending an Unhave message. Finally, any time a Want message is received, the peer should react by sending a Have message. -``` -0 - 1 -2 - 3 -4 - 5 -6 -``` -If we want to fetch block 0, we need to communicate whether of not we already have the uncles (2, 5) and the parent (3). This information can be compressed into very small bit vector using the following scheme: +## Request Procedure +[requests]: #requests - - Let the trailing bit denote whether or not the leading bit is a parent and not a uncle. - - Let the previous trailing bits denote whether or not we have the next uncle. +After the Want/Have Procedure, a peer will know what Hypercore feed blocks are available on the remote, and can then send Request messages for the individual blocks. Each Request specifies which block it needs, and also specifies some information about Merkle tree proof nodes which should be included. In reaction, the remote sends a Data message with the requested content. -Let's consider an example. Suppose we want to fetch block 0, and we have 2 and 3 but not 5. We need to therefore communicate: +At present, Request messages can only specify one block at a time. This is to encourage an equal distribution of requests between multiple connected peers. However, multiple requests can be sent in parallel. -``` -the leading bit is a parent, not an uncle -we already have the first uncle, 2 so don't send us that -we don't have the next uncle, 5 -we have the next parent, 3 -``` -We would encode this into the bit vector `1011`. Decoded: +## Extension Procedure +[extensions]: #extensions -``` -101(1) <-- the leading bit is a parent, not an uncle -10(1)1 <-- we already have the first uncle, 2 so don't send us that -1(0)11 <-- we don't have the next uncle, 5 -(1)000 <-- we have the next parent, 3 -``` - -So using this digest the recipient can easily figure out that they only need to send us one hash, 5, for us to verify block 0. +To allow for experimentation within userland, the wire protocol supports an extension process. All extensions are identified by strings, and sent in an array in the Handshake message. Both peers must declare an extension in the handshake to consider it 'supported'. -The bit vector 1 (only contains a single one) means that we already have all the hashes we need so just send us the block. - -These digests are very compact in size, only `(log2(number-of-blocks) + 2) / 8` bytes needed in the worst case. For example if you are sharing one trillion blocks of data the digest would be `(log2(1000000000000) + 2) / 8 ~= 6` bytes long. +Each extension is capable of sending custom payloads through the Extension message type. # Message Details @@ -134,9 +121,9 @@ TODO: description of framing Wire format is `(
)`. `header` is a varint, of form `channel << 4 | <4-bit-type>`. -Messages are encoded (serialized) using Google's [profobuf][protobuf] encoding. +Messages are encoded (serialized) using Google's [protobuf][protobuf] encoding. -[protobuf]: https://TODO +[protobuf]: https://developers.google.com/protocol-buffers/
`type` code Name @@ -157,179 +144,286 @@ Messages are encoded (serialized) using Google's [profobuf][protobuf] encoding. #### Keep-Alive [msg-keepalive]: #msg-keepalive -A message of body length 0 (giving a total message size of 1 byte for the `len` varint) is a keep-alive. Depending on transport and application needs, peers may optionally send keep-alive messages to help detect and prevent channel loss. Peers must always handle keep-alive messages correctly (aka, ignore them), regardless of transport. +A message of body length 0 (giving a total message size of 1 byte for the `len` varint) is a keep-alive. Depending on transport and application needs, peers may optionally send keep-alive messages to help detect and prevent connection loss. Peers must always handle keep-alive messages correctly (aka, ignore them), regardless of transport. TODO: what is a good default interval? #### Feed [msg-feed]: #msg-feed - // type=0, should be the first message sent on a channel - message Feed { - required bytes discoveryKey = 1; - optional bytes nonce = 2; - } +`type=0` Should be the first message sent on a channel. Establishes the content which will be exchanged on the channel. + +``` +message Feed { + required bytes discoveryKey = 1; + optional bytes nonce = 2; +} +``` #### Handshake [msg-handshake]: #msg-handshake - // type=1, overall connection handshake. should be send just after the feed message on the first channel only - message Handshake { - optional bytes id = 1; - optional bool live = 2; // keep the connection open forever? both ends have to agree - optional bytes userData = 3; - repeated string extensions = 4; - } +`type=1` Overall connection handshake. Should be sent just after the Feed message on the first channel only. + +Some notes on the fields: -TODO: What are semantics of 'live' bit? what if there is disagreement? + - **id** Typically a 32-byte identifier which is allocated randomly by a peer at the start of its swarming session. Not authenticated. Used to help detect multiple connections to a single peer. + - **live** If both peers set to true, the connection will be kept open indefinitely. + - **userData** An open field for sending data which you can retrieve on handshake. No value is prescribed by the protocol. + - **extentions** A list of strings identifying additional message-types which are supported via the Extension message. + - **ack** Should all blocks be explicitly acknowledged? TODO how should this work? + +``` +message Handshake { + optional bytes id = 1; + optional bool live = 2; + optional bytes userData = 3; + repeated string extensions = 4; + optional bool ack = 5; +} +``` #### Info [msg-info]: #msg-info - // type=2, message indicating state changes etc. - // initial state for uploading/downloading is true - // if both ends are not downloading and not live it is safe to consider the stream ended - message Info { - optional bool uploading = 1; - optional bool downloading = 2; - } +`type=2` Indicates state changes in a channel. The initial state for uploading & downloading is `true`. If both ends are not downloading and not live, it is safe to consider the channel ended. + +``` +message Info { + optional bool uploading = 1; + optional bool downloading = 2; +} +``` #### Have [msg-have]: #msg-have - // type=3, what do we have? - message Have { - required uint64 start = 1; - optional uint64 length = 2 [default = 1]; // defaults to 1 - optional bytes bitfield = 3; - } +`type=3` Announces the availability of Hypercore feed blocks for download. If no `bitfield` is present, the `start` and `length` represent a continuous set of blocks. If `bitfield` is present, it will convey the availability of individual blocks, offset from the `start`. For information about the encoding of `bitfield`, see "Run-Length Encoded Bitfield" below. + +The Have message is sent in the following contexts: + + - In response to a Want message. + - To announce new blocks which have been added within a remote-wanted range. + +``` +message Have { + required uint64 start = 1; + optional uint64 length = 2 [default = 1]; // defaults to 1 + optional bytes bitfield = 3; +} +``` #### Unhave [msg-unhave]: #msg-unhave - // type=4, what did we lose? - message Unhave { - required uint64 start = 1; - optional uint64 length = 2 [default = 1]; // defaults to 1 - } +`type=4` Announces the loss of availability of Hypercore feed blocks for download. The `start` and `length` represent a continuous set of blocks. + +This message is sent in the following contexts: + + - To inform the peer that data which was sent was rejected, for instance because it was not requested and pushes are not being allowed. + - To announce blocks have been from local storage within a remote-wanted range. + +``` +message Unhave { + required uint64 start = 1; + optional uint64 length = 2 [default = 1]; // defaults to 1 +} +``` #### Want [msg-want]: #msg-want - // type=5, what do we want? remote should start sending have messages in this range - message Want { - required uint64 start = 1; - optional uint64 length = 2; // defaults to Infinity or feed.length (if not live) - } +`type=5` Announces a range of Hypercore feed blocks which the peer would like to receive Have/Unhave messages about. Remote should respond with a Have message which describes which of the wanted blocks are available, and should send additional Have/Unhave messages if availability within the wanted range changes. + +``` +message Want { + required uint64 start = 1; + optional uint64 length = 2; // defaults to Infinity or feed.length (if not live) +} +``` #### Unwant [msg-unwant]: #msg-unwant - // type=6, what don't we want anymore? - message Unwant { - required uint64 start = 1; - optional uint64 length = 2; // defaults to Infinity or feed.length (if not live) - } +`type=6` Announces a range of Hypercore feed blocks which the peer would no longer like to receive Have/Unhave messages about. + +``` +message Unwant { + required uint64 start = 1; + optional uint64 length = 2; // defaults to Infinity or feed.length (if not live) +} +``` #### Request [msg-request]: #msg-request - // type=7, ask for data - message Request { - required uint64 index = 1; - optional uint64 bytes = 2; - optional bool hash = 3; - optional uint64 nodes = 4; - } +`type=7` Request data from the remote. The remote should be react by sending a Data message. + +The fields are as follows: + + - **index** The Hypercore feed block being requested. + - **bytes** An alternative to `index`, a byte offset into the Hypercore feed. Instructs the remote to send the block at the given `bytes` offset. + - **hash** If true, only send the nodes, not the block data. + - **nodes** Communicates which parent and uncle nodes should be included in the Data response. See "Block Tree Digest" (below) for an explanation of the encoding. + +``` +message Request { + required uint64 index = 1; + optional uint64 bytes = 2; + optional bool hash = 3; + optional uint64 nodes = 4; +} +``` #### Cancel [msg-cancel]: #msg-cancel - // type=8, cancel a request - message Cancel { - required uint64 index = 1; - optional uint64 bytes = 2; - optional bool hash = 3; - } +`type=8` Cancel a Request message sent earlier. The remote should react by aborting any active or queued Request message which matches the `index`, `bytes`, and `hash` parameters. + +``` +message Cancel { + required uint64 index = 1; + optional uint64 bytes = 2; + optional bool hash = 3; +} +``` #### Data [msg-data]: #msg-data - // type=9, get some data - message Data { - message Node { - required uint64 index = 1; - required bytes hash = 2; - required uint64 size = 3; - } +`type=9` Send a Hypercore feed block. May contain the data of the block, and may contain the hashes of the ancestor nodes in the merkle tree. Should only be sent in reaction to a Request message. + +When a Data message is received, its node hashes and signature should be verified against any local tree information. If the nodes can be verified, they and the block data should be stored locally. + +If a Data message is received for a block which was not requested, the peer can react by ignoring the data and sending an Unhave message in response. This will inform the remote that the data was rejected, and is not stored. + +``` +message Data { + message Node { required uint64 index = 1; - optional bytes value = 2; - repeated Node nodes = 3; - optional bytes signature = 4; + required bytes hash = 2; + required uint64 size = 3; } + required uint64 index = 1; + optional bytes value = 2; + repeated Node nodes = 3; + optional bytes signature = 4; +} +``` #### Extension [msg-extension]: #msg-extension -`type=15` is an extension message that is encoded like: +`type=15` Send a custom message. - +The `user-type` is an index into the `extensions` array sent in the Handshake. For instance, if two extensions were declared `['foo', 'bar']` then the `user-type` of `'foo'` would be `0` and of `'bar'` would be `1`. -# Examples +The `payload` may be any message content, as needed by the extension. -## Simple Download +```` + +``` -Alyssa P Hacker and Ben Bitdiddle want to share a book... B connects to A. -- full public key and discovery key for the connection -- example nonces (in full) -- messages in "struct" syntax and raw hex: - - B: sends Feed - - A: replies Feed - - B: Handshake (downloading, not live) - - A: Handshake (uploading, not live) - - B: Info: downloading only - - A: Info: uploading only - - B: Have: nothing - - A: Have: everything - - B: Want: first register entry - - A: Data: first entry - - (repeat for all other chunks) -- connection closes +## Run-Length Encoded Bitfield +[run-length-encoded-bitfield]: #run-length-encoded-bitfield -## Multiple Feeds +The RLE bitfield is a series of compressed and uncompressed bit sequences.All sequences start with a header that is a varint. If the last bit is set in the varint (it is an odd number) then a header represents a compressed bit sequence. If the last bit is not set then a header represents an non compressed sequence. -Describe in detail how to "add" a new channel (feed/register) to an existing connection, using Feed (and Handshake?) messages. +``` +compressed-sequence = varint(byte-length-of-sequence << 2 | bit << 1 | 1) +uncompressed-sequence = varint(byte-length-of-bitfield << 1 | 0) + bitfield +``` -## Swarm Synchronization -TODO: should this more involved example actually live here? or in hypercore DEP? It feels pretty message-level, but does involve more hypercore semantics. +## Block Tree Digest +[block-tree-digest]: #block-tree-digest -This example wouldn't include actual messages, but would describe an N-way (3+) node swarm, with a single (complete) seeder, two peers that both download from the seeder and exchange messages, and a fourth peer that downloads from one of the non-seeder peers only. +When asking for a block of data we want to reduce the amount of duplicate hashes that are sent back. To communicate which hashes we have, we just have to communicate two things: which uncles we have and whether or not we have any parent node that can verify the tree. -- Peer A: seeder, writer. Starts with full history and appends to log -- Peer B: swarm, reader. Starts with full history. Live connection. Connected to A, C, D. -- Peer C: swarm, reader. Starts with sparse (old) history. Only wants "latest" data. Connected to A and, B. -- Peer D: like Peer C, but only connected to B. +Consider the following tree: -# Rationale and alternatives -[alternatives]: #alternatives +``` +0 + 1 +2 + 3 +4 + 5 +6 +``` -- Why is this design the best in the space of possible designs? -- What other designs have been considered and what is the rationale for not choosing them? -- What is the impact of not doing this? +If we want to fetch block 0, we need to communicate whether of not we already have the uncles (2, 5) and the parent (3). This information can be compressed into very small bit vector using the following scheme: + - Let the trailing bit denote whether or not the leading bit is a parent and not a uncle. + - Let the previous trailing bits denote whether or not we have the next uncle. -# Unresolved questions -[unresolved]: #unresolved-questions +Let's consider an example. Suppose we want to fetch block 0, and we have 2 and 3 but not 5. We need to therefore communicate: + +``` +the leading bit is a parent, not an uncle +we already have the first uncle, 2 so don't send us that +we don't have the next uncle, 5 +we have the next parent, 3 +``` + +We would encode this into the bit vector `1011`. Decoded: + +``` +101(1) <-- the leading bit is a parent, not an uncle +10(1)1 <-- we already have the first uncle, 2 so don't send us that +1(0)11 <-- we don't have the next uncle, 5 +(1)000 <-- we have the next parent, 3 +``` + +So using this digest the recipient can easily figure out that they only need to send us one hash, 5, for us to verify block 0. + +The bit vector 1 (only contains a single one) means that we already have all the hashes we need so just send us the block. + +These digests are very compact in size, only `(log2(number-of-blocks) + 2) / 8` bytes needed in the worst case. For example if you are sharing one trillion blocks of data the digest would be `(log2(1000000000000) + 2) / 8 ~= 6` bytes long. + + +# Examples -What are extension strings? What can 'userData' bytes be used for? +## Simple Download + +Alice has an e-book identified by a public-key (PK) which Bob would like to download. The e-book also has a discovery-key (DK). Bob connects to Alice, and the following exchange occurs: + +``` + BOB: sends Feed {discoveryKey: DK, nonce: BobNonce} + BOB: sends Handshake {id: BobID} +ALICE: sends Feed {discoveryKey: DK, nonce: AliceNonce} +ALICE: sends Handshake {id: AliceID} + BOB: waits for Feed + BOB: waits for Handshake +ALICE: waits for Feed +ALICE: waits for Handshake + BOB: sends Want {start: 0} +ALICE: receives Want {start: 0} +ALICE: sends Have {start: 0, bitfield: ...} + BOB: receives Have {start: 0, bitfield: ...} + BOB: sends Request (for block 0) +ALICE: receives Request (for block 0) +ALICE: sends Data (for block 0) + BOB: receives Data (for block 0) + BOB: sends Request (for block 1) +ALICE: receives Request (for block 1) +ALICE: sends Data (for block 1) + ... (repeat for all other blocks) ... + BOB: sends Info {downloading: false} +ALICE: closes connection +``` -Encryption might not make sense in some contexts (eg, IPC, or if the transport layer is already providing encryption). Should this DEP recognize this explicitly? + +# Unresolved questions +[unresolved]: #unresolved-questions + +- Encryption might not make sense in some contexts (eg, IPC, or if the transport layer is already providing encryption). Should this DEP recognize this explicitly? - What parts of the design do you expect to resolve through the DEP consensus process before this gets merged? - What parts of the design do you expect to resolve through implementation and code review, or are left to independent library or application developers? - What related issues do you consider out of scope for this DEP that could be addressed in the future independently of the solution that comes out of this DEP? +- How do "ack"s work? # Changelog From 4fc1641963a89cb07a21896a83e881a8464a4301 Mon Sep 17 00:00:00 2001 From: Paul Frazee Date: Sun, 4 Mar 2018 16:40:08 -0600 Subject: [PATCH 05/23] Fix typo --- proposals/0000-wire-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 402fb55..acaf810 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -320,7 +320,7 @@ The `user-type` is an index into the `extensions` array sent in the Handshake. F The `payload` may be any message content, as needed by the extension. -```` +``` ``` From a3e0f181901eff22fe0ab3ca61487b2d54346744 Mon Sep 17 00:00:00 2001 From: Paul Frazee Date: Fri, 1 Jun 2018 11:44:34 -0500 Subject: [PATCH 06/23] Typo fix in proposals/0000-wire-protocol.md --- proposals/0000-wire-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index acaf810..36783ea 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -221,7 +221,7 @@ message Have { This message is sent in the following contexts: - To inform the peer that data which was sent was rejected, for instance because it was not requested and pushes are not being allowed. - - To announce blocks have been from local storage within a remote-wanted range. + - To announce blocks have been removed from local storage within a remote-wanted range. ``` message Unhave { From d1118ab3030950f807a1f6f8f8e2767fa16fb701 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sun, 18 Nov 2018 13:16:40 -0800 Subject: [PATCH 07/23] address some review feedback --- proposals/0000-wire-protocol.md | 46 +++++++++++++++++---------------- 1 file changed, 24 insertions(+), 22 deletions(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 36783ea..1000636 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -7,7 +7,7 @@ Type: Standard Status: Undefined (as of 2018-02-04) -Github PR: (add HTTPS link here after PR is opened) +Github PR: [Draft](https://github.com/datprotocol/DEPs/pull/8) Authors: [Paul Frazee](https://github.com/pfrazee), [Bryan Newbold](https://github.com/bnewbold) @@ -34,6 +34,8 @@ The Dat wire protocol depends on the use of a binary transport protocol which pr - reliable delivery (no dropped or partial messages) - in-order delivery of messages +Two notable transport protocols that satisfy these requirements are TCP and µTP (the "Micro Transport Protocol"). + Peers wishing to connect need to discover each other using some mechanism or another (see forthcoming DEPs on some options; this process is modular and swappable), and both need to have the public key for the primary hypercore they wish to exchange. Messages are framed by the Dat protocol itself (see Messages section for details). @@ -46,7 +48,7 @@ Multiple hypercore feeds can be synchronized over the same protocol connection. Note that at least one feed is necessary for each connection (for handshaking to succeed), and that the first feed is the one used for discovery and as an encryption key. -To initiate a new channel (after the primary is established), TODO +To initiate a new channel (after the primary is established), a peer can send a Feed message, followed by an Info message. Unlike the first message sent on an overall connection, later Feed messages are encrypted. Either party may initiate a new channel with a Feed message at any time. ## Handshake Procedure @@ -119,7 +121,7 @@ Each extension is capable of sending custom payloads through the Extension messa TODO: description of framing -Wire format is `(
)`. `header` is a varint, of form `channel << 4 | <4-bit-type>`. +Wire format is `(
)`. `header` is a varint, of form `channel << 4 | <4-bit-type>`. `len` is a varint with the number of bytes in the following message (the sum of the `header` and `message`). Messages are encoded (serialized) using Google's [protobuf][protobuf] encoding. @@ -151,7 +153,7 @@ TODO: what is a good default interval? #### Feed [msg-feed]: #msg-feed -`type=0` Should be the first message sent on a channel. Establishes the content which will be exchanged on the channel. +`type=0` Should be the first message sent on a channel. Establishes the content which will be exchanged on the channel. See the [Channels][channels] section for details. ``` message Feed { @@ -293,7 +295,7 @@ message Cancel { `type=9` Send a Hypercore feed block. May contain the data of the block, and may contain the hashes of the ancestor nodes in the merkle tree. Should only be sent in reaction to a Request message. -When a Data message is received, its node hashes and signature should be verified against any local tree information. If the nodes can be verified, they and the block data should be stored locally. +When a Data message is received, its node hashes and signature should be verified against any local tree information. If the nodes can be verified, they and the block data may be stored locally. If a Data message is received for a block which was not requested, the peer can react by ignoring the data and sending an Unhave message in response. This will inform the remote that the data was rejected, and is not stored. @@ -390,32 +392,31 @@ These digests are very compact in size, only `(log2(number-of-blocks) + 2) / 8` Alice has an e-book identified by a public-key (PK) which Bob would like to download. The e-book also has a discovery-key (DK). Bob connects to Alice, and the following exchange occurs: ``` - BOB: sends Feed {discoveryKey: DK, nonce: BobNonce} - BOB: sends Handshake {id: BobID} -ALICE: sends Feed {discoveryKey: DK, nonce: AliceNonce} -ALICE: sends Handshake {id: AliceID} + BOB: sends Feed (unencrypted) {discoveryKey: DK, nonce: BobNonce} + BOB: sends Handshake {id: BobID} +ALICE: sends Feed (unencrypted) {discoveryKey: DK, nonce: AliceNonce} +ALICE: sends Handshake {id: AliceID} BOB: waits for Feed BOB: waits for Handshake ALICE: waits for Feed ALICE: waits for Handshake - BOB: sends Want {start: 0} -ALICE: receives Want {start: 0} -ALICE: sends Have {start: 0, bitfield: ...} - BOB: receives Have {start: 0, bitfield: ...} - BOB: sends Request (for block 0) -ALICE: receives Request (for block 0) -ALICE: sends Data (for block 0) - BOB: receives Data (for block 0) - BOB: sends Request (for block 1) -ALICE: receives Request (for block 1) -ALICE: sends Data (for block 1) + BOB: sends Want {start: 0} +ALICE: receives Want {start: 0} +ALICE: sends Have {start: 0, bitfield: ...} + BOB: receives Have {start: 0, bitfield: ...} + BOB: sends Request (for block 0) +ALICE: receives Request (for block 0) +ALICE: sends Data (for block 0) + BOB: receives Data (for block 0) + BOB: sends Request (for block 1) +ALICE: receives Request (for block 1) +ALICE: sends Data (for block 1) ... (repeat for all other blocks) ... - BOB: sends Info {downloading: false} + BOB: sends Info {downloading: false} ALICE: closes connection ``` - # Unresolved questions [unresolved]: #unresolved-questions @@ -424,6 +425,7 @@ ALICE: closes connection - What parts of the design do you expect to resolve through implementation and code review, or are left to independent library or application developers? - What related issues do you consider out of scope for this DEP that could be addressed in the future independently of the solution that comes out of this DEP? - How do "ack"s work? +- There is a potential race condition with channel index numbers. If each peer sends a new Feed message on a new channel at the same time (aka, before the remote message is received), what should peers do? Probably ignore the channel and try again. Possibly channel indices should go even/odd depending on the peer proposing to prevent conflicts. # Changelog From 086735635b8203f70d5ea06732a2b670526b5579 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sun, 18 Nov 2018 14:07:56 -0800 Subject: [PATCH 08/23] more wire protocol review feedback --- proposals/0000-wire-protocol.md | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 1000636..eeb88e1 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -119,9 +119,7 @@ Each extension is capable of sending custom payloads through the Extension messa # Message Details [message-details]: #message-details -TODO: description of framing - -Wire format is `(
)`. `header` is a varint, of form `channel << 4 | <4-bit-type>`. `len` is a varint with the number of bytes in the following message (the sum of the `header` and `message`). +The connection between peers is an endless stream of bytes, so each message must be "framed" so the recipient knows when it starts and ends. The wire framing format is `(
)`. `len` is a varint with the number of bytes in the following message (the sum of the `header` and `message`). `header` is a varint, of form `channel << 4 | <4-bit-type>`. Note that in most cases the `header` varint will be a single byte, but clients should treat it as a varint to accomodate large channel counts. Messages are encoded (serialized) using Google's [protobuf][protobuf] encoding. @@ -146,9 +144,10 @@ Messages are encoded (serialized) using Google's [protobuf][protobuf] encoding. #### Keep-Alive [msg-keepalive]: #msg-keepalive -A message of body length 0 (giving a total message size of 1 byte for the `len` varint) is a keep-alive. Depending on transport and application needs, peers may optionally send keep-alive messages to help detect and prevent connection loss. Peers must always handle keep-alive messages correctly (aka, ignore them), regardless of transport. +A message of body length 0 (giving a total message size of 1 byte for the `len` varint) is a keep-alive. Peers must always handle keep-alive messages correctly (aka, ignore them), regardless of transport. + +Depending on transport and application needs, peers may *optionally* send keep-alive messages to help detect and prevent connection loss. Implementors are free to chose their own period or strategy for sending keep-alives. A reasonable default period is 300 seconds (5 minutes). -TODO: what is a good default interval? #### Feed [msg-feed]: #msg-feed @@ -330,7 +329,7 @@ The `payload` may be any message content, as needed by the extension. ## Run-Length Encoded Bitfield [run-length-encoded-bitfield]: #run-length-encoded-bitfield -The RLE bitfield is a series of compressed and uncompressed bit sequences.All sequences start with a header that is a varint. If the last bit is set in the varint (it is an odd number) then a header represents a compressed bit sequence. If the last bit is not set then a header represents an non compressed sequence. +The Run-Length Encoded (RLE) bitfield is a series of compressed and uncompressed bit sequences.All sequences start with a header that is a varint. If the last bit is set in the varint (it is an odd number) then a header represents a compressed bit sequence. If the last bit is not set then a header represents an non compressed sequence. ``` compressed-sequence = varint(byte-length-of-sequence << 2 | bit << 1 | 1) @@ -377,7 +376,7 @@ We would encode this into the bit vector `1011`. Decoded: 1(0)11 <-- we don't have the next uncle, 5 (1)000 <-- we have the next parent, 3 ``` - + So using this digest the recipient can easily figure out that they only need to send us one hash, 5, for us to verify block 0. The bit vector 1 (only contains a single one) means that we already have all the hashes we need so just send us the block. @@ -396,7 +395,7 @@ Alice has an e-book identified by a public-key (PK) which Bob would like to down BOB: sends Handshake {id: BobID} ALICE: sends Feed (unencrypted) {discoveryKey: DK, nonce: AliceNonce} ALICE: sends Handshake {id: AliceID} - BOB: waits for Feed + BOB: waits for Feed BOB: waits for Handshake ALICE: waits for Feed ALICE: waits for Handshake @@ -420,12 +419,9 @@ ALICE: closes connection # Unresolved questions [unresolved]: #unresolved-questions -- Encryption might not make sense in some contexts (eg, IPC, or if the transport layer is already providing encryption). Should this DEP recognize this explicitly? -- What parts of the design do you expect to resolve through the DEP consensus process before this gets merged? -- What parts of the design do you expect to resolve through implementation and code review, or are left to independent library or application developers? -- What related issues do you consider out of scope for this DEP that could be addressed in the future independently of the solution that comes out of this DEP? -- How do "ack"s work? -- There is a potential race condition with channel index numbers. If each peer sends a new Feed message on a new channel at the same time (aka, before the remote message is received), what should peers do? Probably ignore the channel and try again. Possibly channel indices should go even/odd depending on the peer proposing to prevent conflicts. +- Encryption might not make sense in some contexts (eg, IPC, or if the transport layer is already providing encryption). Should this DEP recognize this explicitly? Does not need to be addressed before Draft status. +- How do "ack"s work? Should be resolved before Draft status. +- There is a potential race condition with channel index numbers. If each peer sends a new Feed message on a new channel at the same time (aka, before the remote message is received), what should peers do? Probably ignore the channel and try again. Possibly channel indices should go even/odd depending on the peer proposing to prevent conflicts. Does not need to be resolved before Draft status. # Changelog From 6f9d0266a5fbdee9e13bb012679756695ebbd965 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sun, 18 Nov 2018 14:08:15 -0800 Subject: [PATCH 09/23] wire protocol references section --- proposals/0000-wire-protocol.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index eeb88e1..6498e7a 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -415,6 +415,31 @@ ALICE: sends Data (for block 1) ALICE: closes connection ``` +# Privacy and Security Considerations + +TODO: + + +# References + +"Dat - Distributed Dataset Synchronization And Versioning" by Maxwell Ogden, Karissa McKelvey, and Mathias Buus Madsen. Whitepaper, May 2017 ([pdf](https://datproject.org/paper)). + +"The BitTorrent Protocol Specification (BEP #3)", by Bram Cohen. January 2008 ([html](http://www.bittorrent.org/beps/bep_0003.html)) + +"uTorrent transport protocol (BEP #29)", by Arvid Norberg. June 2009 ([html](http://www.bittorrent.org/beps/bep_0029.html)) + +"Merkle hash torrent extension (BEP #30)", by Arno Bakker. May 2009. ([html](http://www.bittorrent.org/beps/bep_0030.html)) + +"Updating Torrents Via DHT Mutable Items (BEP #46)", by Luca Matteis. July 2016. ([html](http://www.bittorrent.org/beps/bep_0046.html)) + +"Rarest First and Choke Algorithms Are Enough", by Arnaud Legout, G. Urvoy-Keller, and P. Michiardi. October 2006. ([pdf](http://conferences.sigcomm.org/imc/2006/papers/p20-legout.pdf)) + +"Extending the Salsa20 nonce", by Daniel J. Bernstein. February 22011. ([pdf](https://cr.yp.to/snuffle/xsalsa-20110204.pdf)) + +libsodium is a fork of NaCl (the "Networking and Cryptography library", developed by Daniel J. Bernstein). More information available from the [NaCl website](https://nacl.cr.yp.to/) and [libsodium website](https://libsodium.org). Specific information about the BLAKE2b hash function available from the [BLAKE2 website](https://blake2.net/). + +Google Protocol Buffers Documentation ([website](https://developers.google.com/protocol-buffers/)) + # Unresolved questions [unresolved]: #unresolved-questions From e9e383d9b1411be58fd4233f63f3daca37761a87 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sun, 18 Nov 2018 14:19:55 -0800 Subject: [PATCH 10/23] Buffer.alloc in example code --- proposals/0000-wire-protocol.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 6498e7a..d5d3336 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -62,8 +62,10 @@ The **nonce** is generated by each peer as a random 32-byte sequence. The **discovery key** is generated from the public encryption key for a hypercore feed (in this case the first, or "primary" feed) by using the public key to sign the 9-byte ASCII string "HYPERCORE" (with no trailing NULL byte) using the BLAKE2b keyed hashing algorithm (provided by most BLAKE2b hash implementations). The discovery key is 32 bytes long. +Example of generating a discovery key in Javascript: + ```js -var discoveryKey = new Buffer(32) +var discoveryKey = Buffer.alloc(32) sodium.crypto_generichash( discoveryKey, // out 'HYPERCORE', // in (message) From 3a303cb9625d45ba3a80146bf682a9cb407599e1 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sun, 18 Nov 2018 14:20:24 -0800 Subject: [PATCH 11/23] early wire protocol history/changelog --- proposals/0000-wire-protocol.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index d5d3336..1107d52 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -454,7 +454,8 @@ Google Protocol Buffers Documentation ([website](https://developers.google.com/p # Changelog [changelog]: #changelog -A brief statemnt about current status can go here, follow by a list of dates when the status line of this DEP changed (in most-recent-last order). +The Dat wire protocol was initially described in the 2016 white paper referenced above. +- 2018-03-04: Early "WIP" draft circulated on github - YYYY-MM-DD: First complete draft submitted for review From e761143a051dc9735364d8897baf691bd2ed9d68 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sun, 18 Nov 2018 15:07:31 -0800 Subject: [PATCH 12/23] privacy/security section for wire protocol --- proposals/0000-wire-protocol.md | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 1107d52..07c2820 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -419,7 +419,31 @@ ALICE: closes connection # Privacy and Security Considerations -TODO: +All security and privacy guarantees of this protocol implicitly depend on the soundness of the underlying cryptographic primitives, algorithms, and implementations, which include BLAKE2b and XSalsa20. + +A connecting peer, or any observer able to decrypt traffic, may be able to infer the following from protocol traffic: + +- what subset of content the local peer already has access to (via Have message) +- what subset of content the local peer is interested in (via Want and Request messages) + +A "persistent" (all-seeing) observer can infer the following with reasonable confidence, using only protocol traffic: + +- which peers (identified by IP address, timing, topology, or other network metadata) have connected which other peers +- the relative flow of content (which peer provided data to the other, and how much) +- which primary hypercore feeds which peers have knowledge of (identified by discovery key, not public key) +- which peers have the public key for a given discovery key (based on whether connections succeed) + +Such an observer can not determine the specific content of hypercore feeds, or (with confidence) which peer (or peers) have write access to a feed. + +The wire protocol does not pad message sizes. This means that a persistent observer could potentially identify traffic content by infering message sizes. + +The hypercore protocol does not intentionally identify peers across connections, but it has been shown that even the smallest amount of identifying metadata can be used, statistically, to track and surveil users. These techniques are sometimes called "device fingerprints", "browser fingerprints", or "permacookies". Hypercore protocol users should not consider themselves immune to such tracking without specific additional effort to identify and mitigate such fingerprints. + +Any network observer with the public key can fully decrypt the network traffic of *any* and *all* peers establishing a connection using that key. This includes channel content other than the first (discovery) channel. Peers should not consider extension messages, "Have"/"Want" metadata, or any other messages or metadata private from other peers (or observers) with the public key. + +The wire protocol encryption provides message *secrecy* (from parties who do not have the public key), but does not guarantee any form of *authenticity*. In the case of Dat and hypercore, the application layer itself verifies authenticity of transfered content using hashes and signatures. However, implementors should note that network agents who can manipulate traffic can modify data in flight without detection, regardless of whether they have the feed public key. However, peers are already not trusted parties, and thus implementors must already take care to treat protocol messages as potentially hostile input. + +The issues of observer decryption and message authenticity may be addressed in a future revision of the wire protocol. # References From db2bd3908dd45b641ccf586b9dc7bddc235d3058 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sat, 19 Jan 2019 22:06:04 -0800 Subject: [PATCH 13/23] wire protocol typos --- proposals/0000-wire-protocol.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 07c2820..850ab7d 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -54,7 +54,7 @@ To initiate a new channel (after the primary is established), a peer can send a ## Handshake Procedure [handshake]: #handshake -A handshake procedure needs to occur for each feed on a channel; the first part of the first handshake happens in cleartext and both validates discovery keys and establishes encyption parameters used for the rest of the connection. The first (primary) channel has `id=0`. +A handshake procedure needs to occur for each feed on a channel; the first part of the first handshake happens in cleartext and both validates discovery keys and establishes encryption parameters used for the rest of the connection. The first (primary) channel has `id=0`. The first (cleartext) message is a Feed message, and includes two fields: a nonce and a discovery key. @@ -121,7 +121,7 @@ Each extension is capable of sending custom payloads through the Extension messa # Message Details [message-details]: #message-details -The connection between peers is an endless stream of bytes, so each message must be "framed" so the recipient knows when it starts and ends. The wire framing format is `(
)`. `len` is a varint with the number of bytes in the following message (the sum of the `header` and `message`). `header` is a varint, of form `channel << 4 | <4-bit-type>`. Note that in most cases the `header` varint will be a single byte, but clients should treat it as a varint to accomodate large channel counts. +The connection between peers is an endless stream of bytes, so each message must be "framed" so the recipient knows when it starts and ends. The wire framing format is `(
)`. `len` is a varint with the number of bytes in the following message (the sum of the `header` and `message`). `header` is a varint, of form `channel << 4 | <4-bit-type>`. Note that in most cases the `header` varint will be a single byte, but clients should treat it as a varint to accommodate large channel counts. Messages are encoded (serialized) using Google's [protobuf][protobuf] encoding. @@ -435,13 +435,13 @@ A "persistent" (all-seeing) observer can infer the following with reasonable con Such an observer can not determine the specific content of hypercore feeds, or (with confidence) which peer (or peers) have write access to a feed. -The wire protocol does not pad message sizes. This means that a persistent observer could potentially identify traffic content by infering message sizes. +The wire protocol does not pad message sizes. This means that a persistent observer could potentially identify traffic content by inferring message sizes. The hypercore protocol does not intentionally identify peers across connections, but it has been shown that even the smallest amount of identifying metadata can be used, statistically, to track and surveil users. These techniques are sometimes called "device fingerprints", "browser fingerprints", or "permacookies". Hypercore protocol users should not consider themselves immune to such tracking without specific additional effort to identify and mitigate such fingerprints. Any network observer with the public key can fully decrypt the network traffic of *any* and *all* peers establishing a connection using that key. This includes channel content other than the first (discovery) channel. Peers should not consider extension messages, "Have"/"Want" metadata, or any other messages or metadata private from other peers (or observers) with the public key. -The wire protocol encryption provides message *secrecy* (from parties who do not have the public key), but does not guarantee any form of *authenticity*. In the case of Dat and hypercore, the application layer itself verifies authenticity of transfered content using hashes and signatures. However, implementors should note that network agents who can manipulate traffic can modify data in flight without detection, regardless of whether they have the feed public key. However, peers are already not trusted parties, and thus implementors must already take care to treat protocol messages as potentially hostile input. +The wire protocol encryption provides message *secrecy* (from parties who do not have the public key), but does not guarantee any form of *authenticity*. In the case of Dat and hypercore, the application layer itself verifies authenticity of transferred content using hashes and signatures. However, implementors should note that network agents who can manipulate traffic can modify data in flight without detection, regardless of whether they have the feed public key. However, peers are already not trusted parties, and thus implementors must already take care to treat protocol messages as potentially hostile input. The issues of observer decryption and message authenticity may be addressed in a future revision of the wire protocol. From 740b893872d355d08b6a502b10f7b80be5a05e7a Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sat, 19 Jan 2019 22:08:28 -0800 Subject: [PATCH 14/23] clarify discovery key generation (keyed hash) Thanks @emilbayes --- proposals/0000-wire-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 850ab7d..ee96067 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -60,7 +60,7 @@ The first (cleartext) message is a Feed message, and includes two fields: a nonc The **nonce** is generated by each peer as a random 32-byte sequence. -The **discovery key** is generated from the public encryption key for a hypercore feed (in this case the first, or "primary" feed) by using the public key to sign the 9-byte ASCII string "HYPERCORE" (with no trailing NULL byte) using the BLAKE2b keyed hashing algorithm (provided by most BLAKE2b hash implementations). The discovery key is 32 bytes long. +The **discovery key** is generated from the public encryption key for a hypercore feed (in this case the first, or "primary" feed) by using the public key to perform a keyed hash of the 9-byte ASCII string "HYPERCORE" (with no trailing NULL byte) using the BLAKE2b keyed hashing algorithm (provided by most BLAKE2b hash implementations). The discovery key is 32 bytes long. Example of generating a discovery key in Javascript: From c4e394a2e46731c0ab4a97ce475d36bcd43579ce Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sat, 19 Jan 2019 22:11:35 -0800 Subject: [PATCH 15/23] clarify stream encoding a bit (from review) --- proposals/0000-wire-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index ee96067..7606419 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -89,7 +89,7 @@ Framing metadata (aka, message length and type) is encrypted, but a third party The encryption scheme used is libsodium's stream primitive, specifically the XSalsa20 cipher. The cipher is fed a shared key (the primary hypercore feed public key), a nonce (selected by the sender and exchanged during handshake), and a block offset (representing all encrypted bytes sent on the connection in this direction so far). -The specific libsodium function used is `crypto_stream_xsalsa20_xor_ic()`. Some interfacing code is necessary to process messages that don't align with the cipher's 64-byte chunk size; unused bytes in any particular chunk can be ignored. For example, if 1000 encrypted bytes had been sent on a connection already, and then a new 50 byte message needed to be encrypted and sent, then one would offset the message by `1000 % 64 = 40` bytes and XOR the first 24 bytes against block 15, then XOR the remaining 26 bytes against block 16. The bytes would be shifted back and recombined before sending, so only 50 bytes would go down the connection; the same process would be followed by the receiver. +The specific libsodium function used is `crypto_stream_xsalsa20_xor_ic()`. Some interfacing code is necessary to process messages that don't align with the cipher's 64-byte chunk size; unused bytes in any particular chunk can be ignored (or, as an optimization, cached for use with the next message). For example, if 1000 encrypted bytes had been sent on a connection already, and then a new 50 byte message needed to be encrypted and sent, then one would offset the message by `1000 % 64 = 40` bytes and XOR the first 24 bytes against block 15, then XOR the remaining 26 bytes against block 16. The bytes would be shifted back and recombined before sending, so only 50 bytes would go down the connection; the same process would be followed by the receiver. ## Want/Have Procedure From e872f916c04fa9ea580f2e3bf7d8e8873cde75bc Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sat, 19 Jan 2019 22:14:43 -0800 Subject: [PATCH 16/23] clarify that tracking is shown for HTTP, not Dat --- proposals/0000-wire-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 7606419..e32b4c0 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -437,7 +437,7 @@ Such an observer can not determine the specific content of hypercore feeds, or ( The wire protocol does not pad message sizes. This means that a persistent observer could potentially identify traffic content by inferring message sizes. -The hypercore protocol does not intentionally identify peers across connections, but it has been shown that even the smallest amount of identifying metadata can be used, statistically, to track and surveil users. These techniques are sometimes called "device fingerprints", "browser fingerprints", or "permacookies". Hypercore protocol users should not consider themselves immune to such tracking without specific additional effort to identify and mitigate such fingerprints. +The hypercore protocol does not intentionally identify peers across connections, but it has been shown in other network protocols (like HTTP) that even the smallest amount of identifying metadata can be used, statistically, to track and surveil users. These techniques are sometimes called "device fingerprints", "browser fingerprints", or "permacookies". Hypercore protocol users should not consider themselves immune to such tracking without specific additional effort to identify and mitigate such fingerprints. Any network observer with the public key can fully decrypt the network traffic of *any* and *all* peers establishing a connection using that key. This includes channel content other than the first (discovery) channel. Peers should not consider extension messages, "Have"/"Want" metadata, or any other messages or metadata private from other peers (or observers) with the public key. From f17d15ea63529221a2c7c76f23bf3d9fbaddff0c Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sat, 19 Jan 2019 22:23:14 -0800 Subject: [PATCH 17/23] clarify ack flag Thanks @emilbayes. See also: https://github.com/mafintosh/hypercore/pull/130 --- proposals/0000-wire-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index e32b4c0..21c8737 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -174,7 +174,7 @@ Some notes on the fields: - **live** If both peers set to true, the connection will be kept open indefinitely. - **userData** An open field for sending data which you can retrieve on handshake. No value is prescribed by the protocol. - **extentions** A list of strings identifying additional message-types which are supported via the Extension message. - - **ack** Should all blocks be explicitly acknowledged? TODO how should this work? + - **ack** A peer (the "sender") can set this flag to request that every time another peer (a "receiver") gets a `Data` message from the sender, that they reply with an explicit `Have` message once they have successfully persisted the data in storage. This lets the "sender" track how many persisted copies of a chunk there are. ``` message Handshake { From 2df9b333a4ab77a0eee0a561936fca707dc68870 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sat, 19 Jan 2019 23:13:06 -0800 Subject: [PATCH 18/23] re-write block digest section --- proposals/0000-wire-protocol.md | 40 ++++++++++++++++----------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 21c8737..0e25300 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -342,7 +342,9 @@ uncompressed-sequence = varint(byte-length-of-bitfield << 1 | 0) + bitfield ## Block Tree Digest [block-tree-digest]: #block-tree-digest -When asking for a block of data we want to reduce the amount of duplicate hashes that are sent back. To communicate which hashes we have, we just have to communicate two things: which uncles we have and whether or not we have any parent node that can verify the tree. +As described in DEP-0002 (Hypercore), a peer should be able to verify both the integrity of received data (aka, was there corruption somewhere along the way, detected via hash) and the authenticity (aka, is this the data from the original writer, detected via signature). Hypercore transmits the hash for every data block, but only signatures for the root hashes of Merkel trees, not individual block hashes, which means a peer may need additional hashes (for data blocks they do not have a copy of) if they want to verify the signatures of individual blocks. + +Redundantly transmitting all such hashes on every request would be inefficient if the receiver already had some hashes, so requesting peers can specify which hashes they need in the `nodes` field of the `Request` message. Instead of sending node indexes, the peer can send a compact bitfield, indicating for each "uncle" and "parent" whether the hash should be transmitted. Consider the following tree: @@ -356,35 +358,33 @@ Consider the following tree: 6 ``` -If we want to fetch block 0, we need to communicate whether of not we already have the uncles (2, 5) and the parent (3). This information can be compressed into very small bit vector using the following scheme: +If the receiving peers wants to fetch, and verify, block 2, it needs to communicate whether it already have the uncle hashes (0, 5) and the parent hashes (3). This information can be compressed into a small bit vector with the following scheme: - - Let the trailing bit denote whether or not the leading bit is a parent and not a uncle. - - Let the previous trailing bits denote whether or not we have the next uncle. + - the least-significant bit indicates whether the most-significant bit is a "parent" (if '1') or an "uncle" (if '0') + - all other bits, in order from least- to most-significant, indicate whether the corresponding "uncle" hash *does* need to be transmitted (if bit '0') or *does not* (if bit '1') -Let's consider an example. Suppose we want to fetch block 0, and we have 2 and 3 but not 5. We need to therefore communicate: +An an example, suppose we want to fetch block 2 from a remote peer, and we already have the node metadata (hashes) for blocks 0 and 3, but not 5. In other words: -``` -the leading bit is a parent, not an uncle -we already have the first uncle, 2 so don't send us that -we don't have the next uncle, 5 -we have the next parent, 3 -``` + - 0, an uncle, we already have the hash + - 5, next uncle, we don't have hash + - 3, a parent, we do have hash -We would encode this into the bit vector `1011`. Decoded: +Our `Request` should include the digest bits: ``` -101(1) <-- the leading bit is a parent, not an uncle -10(1)1 <-- we already have the first uncle, 2 so don't send us that -1(0)11 <-- we don't have the next uncle, 5 -(1)000 <-- we have the next parent, 3 +101(1) <-- indicates that the most-significant bit is a parent, not an uncle +10(1)1 <-- do not send hash for the first uncle, 0 +1(0)11 <-- do send hash for the next uncle, 5 +(1)000 <-- do not send hash for next parent, 3 ``` -So using this digest the recipient can easily figure out that they only need to send us one hash, 5, for us to verify block 0. +This digest (bit vector `1011`) is transmitted in a `uint64`, in the least-significant bits. The receiving peer can calculate (from the index number) exactly how many bits are expected and extract the bitfield. The "most-significant bit" referenced above is of just the fixed-size bitfield, not the `uint64` as a whole. -The bit vector 1 (only contains a single one) means that we already have all the hashes we need so just send us the block. +From this, the remote peer will know to only send one hash (for block 5) for us to verify block 2. Note that we (the receiver) can calculate the hash for block 2 ourselves when we receive it. -These digests are very compact in size, only `(log2(number-of-blocks) + 2) / 8` bytes needed in the worst case. For example if you are sharing one trillion blocks of data the digest would be `(log2(1000000000000) + 2) / 8 ~= 6` bytes long. +As a special case, the bit vector `1` (only contains a single one) means that the sender should not send any hashes. +These digests are very compact in size. Only `(log2(number-of-blocks) + 2) / 8` bytes are needed in the worst case. For example if you are sharing one trillion blocks of data the digest would be `(log2(1000000000000) + 2) / 8 ~= 6` bytes long (which fits in a single `uint64`). # Examples @@ -470,8 +470,8 @@ Google Protocol Buffers Documentation ([website](https://developers.google.com/p # Unresolved questions [unresolved]: #unresolved-questions +- Why are `Request` message block digests (`nodes`) a `uint64`, not a `varint`? - Encryption might not make sense in some contexts (eg, IPC, or if the transport layer is already providing encryption). Should this DEP recognize this explicitly? Does not need to be addressed before Draft status. -- How do "ack"s work? Should be resolved before Draft status. - There is a potential race condition with channel index numbers. If each peer sends a new Feed message on a new channel at the same time (aka, before the remote message is received), what should peers do? Probably ignore the channel and try again. Possibly channel indices should go even/odd depending on the peer proposing to prevent conflicts. Does not need to be resolved before Draft status. From 1cff4fedffd33bc3cb1f52e398b6f442dd0fc04b Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sat, 19 Jan 2019 23:39:24 -0800 Subject: [PATCH 19/23] switch block digest example again (to 6) --- proposals/0000-wire-protocol.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 0e25300..bfa6e81 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -342,7 +342,7 @@ uncompressed-sequence = varint(byte-length-of-bitfield << 1 | 0) + bitfield ## Block Tree Digest [block-tree-digest]: #block-tree-digest -As described in DEP-0002 (Hypercore), a peer should be able to verify both the integrity of received data (aka, was there corruption somewhere along the way, detected via hash) and the authenticity (aka, is this the data from the original writer, detected via signature). Hypercore transmits the hash for every data block, but only signatures for the root hashes of Merkel trees, not individual block hashes, which means a peer may need additional hashes (for data blocks they do not have a copy of) if they want to verify the signatures of individual blocks. +As described in DEP-0002 (Hypercore), a peer should be able to verify both the integrity of received data (aka, was there corruption somewhere along the way, detected via hash) and the authenticity (aka, is this the data from the original writer, detected via signature). Hypercore transmits the hash for every data block, but only signatures for the root hashes of Merkle trees, not individual block hashes, which means a peer may need additional hashes (for data blocks they do not have a copy of) if they want to verify the signatures of individual blocks. Redundantly transmitting all such hashes on every request would be inefficient if the receiver already had some hashes, so requesting peers can specify which hashes they need in the `nodes` field of the `Request` message. Instead of sending node indexes, the peer can send a compact bitfield, indicating for each "uncle" and "parent" whether the hash should be transmitted. @@ -358,31 +358,32 @@ Consider the following tree: 6 ``` -If the receiving peers wants to fetch, and verify, block 2, it needs to communicate whether it already have the uncle hashes (0, 5) and the parent hashes (3). This information can be compressed into a small bit vector with the following scheme: +If the receiving peers wants to fetch, and verify, block 6, it needs to communicate whether it already has the uncle hashes (4, 1) and the parent hashes (3). This information can be compressed into a small bit vector with the following scheme: - the least-significant bit indicates whether the most-significant bit is a "parent" (if '1') or an "uncle" (if '0') - all other bits, in order from least- to most-significant, indicate whether the corresponding "uncle" hash *does* need to be transmitted (if bit '0') or *does not* (if bit '1') -An an example, suppose we want to fetch block 2 from a remote peer, and we already have the node metadata (hashes) for blocks 0 and 3, but not 5. In other words: +An an example, suppose we want to fetch block 6 from a remote peer, and we already have the sparse node metadata (hashes) for blocks 3 and 4, but not 1. In other words: - - 0, an uncle, we already have the hash - - 5, next uncle, we don't have hash - - 3, a parent, we do have hash + - 4, an uncle, we already have the hash + - 1, next uncle, we don't have hash + - 3, a parent (and root hash), we do have hash + - root signature (of hash 3) will be sent with request Our `Request` should include the digest bits: ``` 101(1) <-- indicates that the most-significant bit is a parent, not an uncle -10(1)1 <-- do not send hash for the first uncle, 0 -1(0)11 <-- do send hash for the next uncle, 5 +10(1)1 <-- do not send hash for the first uncle, 4 +1(0)11 <-- do send hash for the next uncle, 1 (1)000 <-- do not send hash for next parent, 3 ``` This digest (bit vector `1011`) is transmitted in a `uint64`, in the least-significant bits. The receiving peer can calculate (from the index number) exactly how many bits are expected and extract the bitfield. The "most-significant bit" referenced above is of just the fixed-size bitfield, not the `uint64` as a whole. -From this, the remote peer will know to only send one hash (for block 5) for us to verify block 2. Note that we (the receiver) can calculate the hash for block 2 ourselves when we receive it. +From this, the remote peer will know to only send one hash (for block 1) for us to verify block 6. Note that we (the receiver) can calculate the hash for block 6 ourselves when we receive it. -As a special case, the bit vector `1` (only contains a single one) means that the sender should not send any hashes. +As a special case, the bit vector `1` (only contains a single one) means that the sender should not send any hashes at all. These digests are very compact in size. Only `(log2(number-of-blocks) + 2) / 8` bytes are needed in the worst case. For example if you are sharing one trillion blocks of data the digest would be `(log2(1000000000000) + 2) / 8 ~= 6` bytes long (which fits in a single `uint64`). From bcbc9026f4d3485313a0e48bd7b49221f8276c99 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sat, 19 Jan 2019 23:50:38 -0800 Subject: [PATCH 20/23] switch to pseudo-code (not javascript) --- proposals/0000-wire-protocol.md | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index bfa6e81..d5b42bf 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -62,15 +62,10 @@ The **nonce** is generated by each peer as a random 32-byte sequence. The **discovery key** is generated from the public encryption key for a hypercore feed (in this case the first, or "primary" feed) by using the public key to perform a keyed hash of the 9-byte ASCII string "HYPERCORE" (with no trailing NULL byte) using the BLAKE2b keyed hashing algorithm (provided by most BLAKE2b hash implementations). The discovery key is 32 bytes long. -Example of generating a discovery key in Javascript: - -```js -var discoveryKey = Buffer.alloc(32) -sodium.crypto_generichash( - discoveryKey, // out - 'HYPERCORE', // in (message) - publicKey // key -) +Example of generating a discovery key in pseudo-code: + +``` +discoveryKey = BLAKE2b(message = 'HYPERCORE', key = publicKey, outputLengthBytes = 32) ``` The discovery key is used in cleartext instead of the public key to avoid leaking the public key to the network; read access to hypercore feeds (including Dat archives) is controlled by limiting access to public keys. From 9e753b2133665cc3b9234cd3c8afbec750c12057 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Sat, 19 Jan 2019 23:53:02 -0800 Subject: [PATCH 21/23] all protobuf ints are varint --- proposals/0000-wire-protocol.md | 1 - 1 file changed, 1 deletion(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index d5b42bf..61bd8bf 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -466,7 +466,6 @@ Google Protocol Buffers Documentation ([website](https://developers.google.com/p # Unresolved questions [unresolved]: #unresolved-questions -- Why are `Request` message block digests (`nodes`) a `uint64`, not a `varint`? - Encryption might not make sense in some contexts (eg, IPC, or if the transport layer is already providing encryption). Should this DEP recognize this explicitly? Does not need to be addressed before Draft status. - There is a potential race condition with channel index numbers. If each peer sends a new Feed message on a new channel at the same time (aka, before the remote message is received), what should peers do? Probably ignore the channel and try again. Possibly channel indices should go even/odd depending on the peer proposing to prevent conflicts. Does not need to be resolved before Draft status. From cb3c3a804a79513f1d14782d6fc58d5d1e3b71a9 Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Wed, 13 Feb 2019 09:12:25 -0800 Subject: [PATCH 22/23] first submitted 2019-01-19 --- proposals/0000-wire-protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 61bd8bf..6b95230 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -476,5 +476,5 @@ Google Protocol Buffers Documentation ([website](https://developers.google.com/p The Dat wire protocol was initially described in the 2016 white paper referenced above. - 2018-03-04: Early "WIP" draft circulated on github -- YYYY-MM-DD: First complete draft submitted for review +- 2019-01-19: First complete draft submitted for review From 03a1f1d35c86c0e4560948ee21d238ec856a93ce Mon Sep 17 00:00:00 2001 From: Bryan Newbold Date: Wed, 13 Feb 2019 09:30:34 -0800 Subject: [PATCH 23/23] updates from review Thanks @martinheidegger! --- proposals/0000-wire-protocol.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/proposals/0000-wire-protocol.md b/proposals/0000-wire-protocol.md index 6b95230..b7a7d01 100644 --- a/proposals/0000-wire-protocol.md +++ b/proposals/0000-wire-protocol.md @@ -339,7 +339,7 @@ uncompressed-sequence = varint(byte-length-of-bitfield << 1 | 0) + bitfield As described in DEP-0002 (Hypercore), a peer should be able to verify both the integrity of received data (aka, was there corruption somewhere along the way, detected via hash) and the authenticity (aka, is this the data from the original writer, detected via signature). Hypercore transmits the hash for every data block, but only signatures for the root hashes of Merkle trees, not individual block hashes, which means a peer may need additional hashes (for data blocks they do not have a copy of) if they want to verify the signatures of individual blocks. -Redundantly transmitting all such hashes on every request would be inefficient if the receiver already had some hashes, so requesting peers can specify which hashes they need in the `nodes` field of the `Request` message. Instead of sending node indexes, the peer can send a compact bitfield, indicating for each "uncle" and "parent" whether the hash should be transmitted. +Redundantly transmitting all such hashes on every request would be inefficient if the receiver already had some hashes, so requesting peers can specify which hashes they need in the `nodes` field of the `Request` message. Instead of sending an array of node indexes, the `nodes` field is a compact bitfield (serialized as a `uint64`), indicating for each "uncle" and "parent" whether the hash should be transmitted. Consider the following tree: @@ -365,22 +365,23 @@ An an example, suppose we want to fetch block 6 from a remote peer, and we alrea - 3, a parent (and root hash), we do have hash - root signature (of hash 3) will be sent with request -Our `Request` should include the digest bits: +Our `Request` should have the following `nodes` bitfield (`0b1011` encoded in the least-significant bits of a `uint64`): ``` -101(1) <-- indicates that the most-significant bit is a parent, not an uncle -10(1)1 <-- do not send hash for the first uncle, 4 -1(0)11 <-- do send hash for the next uncle, 1 -(1)000 <-- do not send hash for next parent, 3 +1011 +│││└── indicates that the most-significant bit is a parent, not an uncle +││└─── do not send hash for the first uncle, 4 +│└──── do send hash for the next uncle, 1 +└───── do not send hash for next parent, 3 ``` -This digest (bit vector `1011`) is transmitted in a `uint64`, in the least-significant bits. The receiving peer can calculate (from the index number) exactly how many bits are expected and extract the bitfield. The "most-significant bit" referenced above is of just the fixed-size bitfield, not the `uint64` as a whole. +Using the `index` field of the `Request` message, the receiving peer can calculate the number of packed bits and extract the bitfield. The "most-significant bit" referenced above is of just the fixed-size bitfield, not the `uint64` as a whole. From this, the remote peer will know to only send one hash (for block 1) for us to verify block 6. Note that we (the receiver) can calculate the hash for block 6 ourselves when we receive it. As a special case, the bit vector `1` (only contains a single one) means that the sender should not send any hashes at all. -These digests are very compact in size. Only `(log2(number-of-blocks) + 2) / 8` bytes are needed in the worst case. For example if you are sharing one trillion blocks of data the digest would be `(log2(1000000000000) + 2) / 8 ~= 6` bytes long (which fits in a single `uint64`). +These digests are very compact in size. Only `(log2(number-of-blocks) + 2) / 8` bytes are needed in the worst case. For example if you are sharing one trillion blocks of data the digest would be `(log2(1000000000000) + 2) / 8 ~= 6` bytes long (which fits in a single `uint64`). This scheme works for Hypercores with up to `2^62 = 4,611,686,018,427,387,904` blocks. # Examples