Dat ecosystem planning documents
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
README.md add planning doc to readme (#2) Nov 2, 2018

README.md

Dat Planning

This repository will help us organize and communicate planned features for the Dat ecosystem. Through the Dat working groups, decisions around the protocol and project are coordinated.

Dat is used by various organizations which loosely coordinate through IRC and the working groups. However, because the ecosystem is still young we still need to work together on adding new features and making sure we move forward together. This repository is a place to help communicate plans for projects using Dat and planned features. Because of the nature of the projects, these documents are not meant to be firm roadmaps but more so a starting place for individual projects, managed by separate teams, to create and align those roadmaps (e.g. the Dat CLI, Beaker Browser, Dat Desktop, etc.).


Protocol performance and features

In April of 2017, we shipped a new version of Dat and hyperdrive. Since then, we've worked with users to identify performance issues and other needed features. Through this feedback, the community identified three priorities for improvement:

  • Performance on many small files
  • Ability for multiple people to collaborate on a single Dat
  • Networking improvements for improved peer-to-peer connectivity and security.

These features are detailed below. Please note that the terminology and specifics are not settled, and we more than welcome any feedback or questions.

Performance Improvements

We have multiple improvements underway for the core Dat protocol. They stem from a new internal data structure which integrates a "Trie" to improve lookup performance and a CRDT to support multiple writers. The improvements will be deployed incrementally in order to simplify the process and give us time to test changes.

The first improvement will be performance. We will deploy the new trie structure and add code for handling the multiple data-structure versions on the Dat network. Updates to the applications should be largely invisible to the users – though the apps will go faster!

Multiple Writers

In adding support for multiple writings, we face both technical and design hurdles. As git has shown, the design needs for managing conflicts and branching of data can be as complex as the technical requirements. To simplify this process, we've identified a few steps we can take to incrementally work on this

The second improvement will be a new feature called "one-way mounts." Mounts merge dats together by causing one dat to act as a folder within another dat. A "one-way" mount sets the folder as a read-only subfolder of a parent. Mounts may also have their versions pinned, and so are a good way to handle upstream dependencies (as in the case of code dependencies). Finally, mounts give the Dat networking stack a way to multiplex data transfer over existing connections, reducing the number of DHT lookups necessary. (For instance, a dat with 5 mounts will only require one DHT lookup for all 6 dats.)

The third and final improvement will be "union mounts." Like one-way mounts, the union mount sets a dat as a subfolder of another dat. However, union mounts are able to overlap each other. This is accomplished using a CRDT. As a result, union mounts can be used to create multiwriter dats (dats with multiple authors).

  1. Performance improvements
    1. New trie data-structure
      1. Integrate the Hypertrie updates into HyperDB
      2. Update the Hyperdrive module to leverage HyperDB
      3. Add header-detection and incompatibility handling
      4. Backwards compatibility (correctly use the loaded version's code)
      5. Update the protocol specs
    2. Implement in applications and downstream modules
      1. Dat CLI
      2. Dat-node module
      3. Beaker browser
  2. "One-way" mounts
    1. Data structure updates
      1. Implement one-way mounts in Hyperdrive
      2. Update the protocol specs
    2. Implement in applications and downstream modules
      1. Dat CLI
      2. Dat-node module
      3. Beaker browser
  3. "Union" mounts (Multiwriter)
    1. Data structure updates
      1. Implement union mounts in Hyperdrive
      2. Update the protocol specs
    2. Implement in applications and downstream modules
      1. Dat CLI
      2. Dat-node module
      3. Beaker browser

Networking improvements

The networking layer of Dat is getting two sets of upgrades.

The first upgrade is Hyperswarm, a new DHT designed to arrange connections between users' devices. Hyperswarm will improve connection reliability and power new APIs in the Beaker browser such as PeerSockets.

Hyperswarm will be deployed in a backwards-compatible fashion so that old peers can still reach the new peers.

The second upgrade is a connection security system using the NOISE protocol. Noise will provide forward-secret connections with the option of authenticating connections. In the Dat network, this will improve overall privacy and open the possibility to implement network access-control.