New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hypercore DEP - Ready for draft #4

Merged
merged 8 commits into from Feb 21, 2018

Conversation

Projects
None yet
3 participants
@pfrazee
Member

pfrazee commented Feb 5, 2018

Starting this tracking PR now.

@pfrazee pfrazee changed the title from Add Hypercore DEP (WIP) to Hypercore DEP (WIP) Feb 5, 2018

@pfrazee

This comment has been minimized.

Member

pfrazee commented Feb 5, 2018

Current draft has content extracted from the whitepaper and a little bit of commentary from me (drawbacks, alternatives, unresolved questions). The explanation needs more detail.

In the Hypercore feed, we only want one active root. Therefore, when there are multiple roots we hash all the roots together again. We also include a big endian uint64 binary representation of the corresponding node index (TODO: why?). At most there will be `log2(number of data blocks)`.
```
root = h(uint64be(#9) + 9 + uint64be(#3) + 3)

This comment has been minimized.

@mafintosh

mafintosh Feb 19, 2018

We encode the index also as that describes the structure of the tree not just the data. This might not be nessesary but my gut feeling is that you could do a tree preimage attack on it https://en.m.wikipedia.org/wiki/Merkle_tree#Second_preimage_attack. We already defend against the preimage using the hash 1 byte prefixes so this might not be nessesary here but it doesn't hurt

This comment has been minimized.

@mafintosh

mafintosh Feb 19, 2018

It basically very strongly encodes the length of the feed in the hash

# Specifications and parameters
[specifications-and-parameters]: #specifications-and-parameters
TODO: list what kind of hashes and asymmetric keys are used

This comment has been minimized.

@mafintosh

mafintosh Feb 19, 2018

Blake2b and curve 25519

This comment has been minimized.

@emilbayes

emilbayes Feb 19, 2018

More specifically hashes are Blake2b-256 and signatures are Ed25519 with SHA-512 :)

# Rationale and alternatives
[alternatives]: #alternatives
The Hypercore log is conceptually similar to Secure Scuttlebutt's log structure; both are designed to provide a single append-only history and to verify the integrity using only a public key identifier. However, Secure Scuttlebutt uses a Linked List structure with content-hash pointers to construct its log while Hypercore uses a Merkle Tree. This decision increases the amount of hashes computed and stored in Hypercore, but it enables more efficient partial replication of the dataset over the network as trees enable faster comparisons of dataset availability and verification of integrity. (Citation needed?)

This comment has been minimized.

@mafintosh

mafintosh Feb 19, 2018

Note the amount it is actually only one hash per block of data in hypercore (even numbers are data, odd ones are hashes)

This comment has been minimized.

@pfrazee

pfrazee Feb 20, 2018

Member

Don't you also have to deal with parent & root hashes?

This comment has been minimized.

@mafintosh

mafintosh Feb 20, 2018

still only one hash stored per data entry. if you include the hash of the data itself is 2x data hashes

The Hypercore log is conceptually similar to Secure Scuttlebutt's log structure; both are designed to provide a single append-only history and to verify the integrity using only a public key identifier. However, Secure Scuttlebutt uses a Linked List structure with content-hash pointers to construct its log while Hypercore uses a Merkle Tree. This decision increases the amount of hashes computed and stored in Hypercore, but it enables more efficient partial replication of the dataset over the network as trees enable faster comparisons of dataset availability and verification of integrity. (Citation needed?)
IPFS is designed for immutable hash-addressed content, but it provides a mechanism for mutable content using public key addresses (IPNS). IPNS is still under development but some concepts are established. Its premise is much simpler than Hypercore's; rather than encoding the history in a standard form, IPNS simply signs and publishes a content-hash identifier under the public key, therefore creating a `pubkey -> hash` lookup. The referenced content may choose to encode a history, but it is not required and no constraints on branching is enforced. Compared to Hypercore, IPNS may be more user-friendly as it does not suffer from a catastrophic error if the history splits, therefore enabling users to share private keys freely between devices. This comes at the cost that history may be freely rewritten by the dataset author.

This comment has been minimized.

@mafintosh

mafintosh Feb 19, 2018

Also means it's not a realtime change log

This comment has been minimized.

@mafintosh

mafintosh Feb 19, 2018

Ie in a hypercore feed you can subscribe to updates and they propagate fast and are ordered

# Unresolved questions
[unresolved]: #unresolved-questions
- Is there a potential "branch resolution" protocol which could remove the [linear history requirement](#linear-history-requirement) and therefore enable users to share private keys freely between their devices? Explaining the risks of branches to users is difficult.

This comment has been minimized.

@mafintosh

mafintosh Feb 19, 2018

An extension can be made in the future that allows an author so pick a correct branch

@pfrazee pfrazee changed the title from Hypercore DEP (WIP) to Hypercore DEP - Ready for draft Feb 21, 2018

@pfrazee pfrazee merged commit 642b40d into datprotocol:master Feb 21, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment