Permalink
Switch branches/tags
Nothing to show
Find file
Fetching contributors…
Cannot retrieve contributors at this time
194 lines (136 sloc) 6.52 KB

lolstorage: laughably good social networking and content sharing

Lolstorage is now at https://github.com/manuel/lolstorage. The text below is of an older design.


Description

lolstorage’s approach to P2P /social networking/ and content sharing is to layer a Git-style model with commits and trees-of-hashes (for representing histories of filesystem trees) on top of a cryptographic capability system inspired by /Tahoe-LAFS/ (for integrity and privacy of content even when stored on untrusted providers). It runs in the browser in JavaScript and uses local storage APIs. Application data is stored as filesystem trees. An example Twitter-like app may store a user’s tweets in a YYYY/MM/DD directory structure.

The user’s wallet contains write-caps for the user’s refs (which are mutable pointers to commits), and is encrypted with a passphrase. The user may generate a read-cap for a ref and share it with other peers. This read-cap allows the peers to locally clone the state of the shared ref, which is similar to following in Twitter. The local clone can be updated efficiently, using similar methods as those used in Git fetching and pushing. There is no shared state; only a single user ever writes to a ref. Presented to users is a locally computed overlay of content from local and cloned remote refs (e.g. a Twitter-like timeline) that may index the content to provide services like search and sorted and aggregated views.

The resulting system is simple, secure, and disruption-tolerant vaporware.

Cryptography

These constructions are based on /Tahoe-LAFS/. I more or less directly copied them, hoping to keep all of their security features intact. They are simpler because files are not split into Reed-Solomon encoded shares.

Objects

An immutable object is either a blob, a tree, or a commit.

Objects are encrypted using a fresh encryption key and stored under the hash of their ciphertext.

The verify cap for an object contains the ciphertext hash, allowing verification of stored content.

The read cap for an object contains the verify cap and the encryption key, allowing decryption of stored content.

[lolstorage/diagrams/immutable.png

To write an immutable object

  • Generate encryption key
  • Encrypt data using encryption key
  • Create verify cap by hashing ciphertext
  • Write ciphertext to store using verify cap as storage index
  • Return immutable read cap containing verify cap and encryption key

To read an immutable object given a read cap

  • Read ciphertext from store using verify cap as storage index
  • Check that ciphertext hashes to verify cap
  • Decrypt ciphertext using encryption key

Refs

A mutable ref “points to” the current commit: it contains an immutable read cap for the commit.

For every mutable ref, a private-key pair is created. The owner of the ref has the private key, and thus can sign new versions of the ref.

[lolstorage/diagrams/mutable.png

To write a mutable ref

  • Create read key by hashing private key
  • Generate salt
  • Create encryption key by hashing salt and read key
  • Encrypt ref value using encryption key
  • Sign ciphertext and salt using private key
  • Create content from ciphertext, salt, signature, and public key
  • Create verify cap by hashing public key
  • Write content to store using verify cap as storage index
  • Return mutable read cap containing verify cap and read key

To read a mutable ref given a mutable read cap

  • Read ciphertext, salt, signature, and public key from store using verify cap as storage index
  • Check that public key hashes to verify cap
  • Verify signature of salt and ciphertext using public key
  • Create encryption key from salt and read key
  • Decrypt ciphertext using encryption key

Storage

All binary data is encoded using “modified Base64 for URL” encoding, with “-” and “_” as additional characters beyond [A-Z][a-z][0-9], and requiring no trailing padding characters.

Some data may contain netstrings for additional structure. A netlist is a netstring containing further netstrings as components.

Capabilities

Verify Caps

Base64-encoded binary data.

Read Caps

Read capabilities are two-component netlists, containing the verify cap and the Base64-encoded encryption key or read key, depending on type of read cap (immutable or mutable, respectively).

Private Keys

Base64-encoded binary data.

Objects

Blob

Base64-encoded blob data.

Tree

A dentry is a netlist (name read-cap).

A tree is a netlist of dentries.

Commit

A name/value pair is a netlist (name value).

A commit is a netlist containing name/value pairs:

  • name: “parents”, value: a netlist of read caps (typically containing just one but defined as a list for conceptual compatibility with Git)
  • name: “tree”, value: read cap

Refs

A mutable ref stored on the server has media type “text/html”.

It contains the mutable ref content (a netlist with the Base64-encoded components ciphertext, salt, signature, public key) as a CDATA section in a HTML element with ID “x-lolstorage-storage-object”.

In addition it may include JavaScript code that launches a user interface for viewing the ref.

Server Protocol

Capability URLs

Immutable objects and refs are stored on plain HTTPS servers.

The capability URL for an immutable read cap is:

https://server/path/to/verify-cap.html#encryption-key

The capability URL for a mutable read cap is:

https://server/path/to/verify-cap.html#read-key

It’s important that the verify cap is the last element of the path, so it can be mechanically extracted from the URL.

The encryption and read keys are present in the URLs after the sharp sign fragment identifier, and thus are not sent to servers over HTTPS.

Alternate Servers

It is possible to list alternate storage servers in an URL.

This example URL includes the alternate server https://otherserver/otherpath with the URL parameter “alternate1”:

https://server/path/to/verify-cap.html?alternate1=https%3A%2F%2Fotherserver%2Fotherpath#read-key

The verify cap will be appended to alternate server, giving the alternate capability:

https://otherserver/otherpath/verify-cap.html#read-key

Additional servers can be listed with alternate2, alternate3, …

GET /path/verify-cap.html

POST /path/verify-cap.html