Proposed convention for publicly searchable dats for expanding sneakernets
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
README.md

README.md

HYPERDAT

Publicly searchable dats for expanding sneakernets.

DISCLAIMER -> This is left here as an archive.. or snapshot of my thoughts. I'm not sure about some of this in retrospect. Also "DATLIST" would be a better or more descriptive name than "HYPERDAT", hyperdat implies the dat might take you somewhere which is not always the case.

Author: m-onz from TX11

current status: draft.

Dedications:

This paper is dedicated to louise and our legally binding napkin contract.

File under

decentralized, decent, p2p, dat, sneakernet, bit torrent.

HYPERDAT

The Dat project is a distributed file storage technology that uses cryptography to allow peers to download data from untrusted sources. It is similar to bit torrent but allows for updates to the data with historic versioning by design (bit torrents are used for static data). The Beaker Browser is a browser that lets you view dat folders, if you include an index.html within your dat folder it will be viewable like a conventional website from within Beaker.

Dats have an extremely simple security model which is basically a capability system. The addresses are unguessable public keys that sign the content, this means that its easy to verify the contents where sent by the sender. It also means that its impossible to traverse, brute force or otherwise guess the address of the dat folder unless you have been shown it. This makes dat extremely promising and interesting, allowing for bit torrent-esque inverse scaling of popular dats as well as private data sharing.

This paper proposes a simple convention for when you DO want to make your dats available to a wider audience. Simply creating a JSON file called hyperdat.json listing dats you own and some metadata, adding this JSON file to a normal dat makes it a hyperdat! A hyperdat can be submitted to indexing services which will allow others to search dats that they might be interested in without having to download each one first.

[todo... no mention of what a sneakernet is... expand on that]

Searchable dats for expanding sneakernets

The Beaker Browser and Dat project allow anyone to turn folders on their machines into dats. Dats are a bit like bit torrents but can update their data. Bit torrents allow many peers to share data and trust that the data is not intercepted or modified in transit. This security is achieved through cryptography. A big file is split into chunks which in turn are hashed. Any peer can verify the integritity of a downloading chunk by checkings its hash. A list of hashes is called a magnet link or tracker. Dat uses the the address of the dat as its public key, the contents of the dat are signed using this public key which allows many peers to validate the authenticity of the data.

One issue in Beaker Browser is that when you have a list of dats that you wish were more publicly available to others there is no way to share them easily. Conversely there is no one place to search dats and metadata, you must know a dat address and know what it is in advance. This paper aims to solve this problem by adding a simple convention ontop of existing dat file storage. It could be considered a "userland" convention. It does not add or modify any of the existing security or architectural principles related to dat.

Simply creating a dat with a list of other dats you seed (and meta data) makes it into a special "hyperdat". You can then send your hyperdat to a friend using a side channel. Your friend can then see what dats may of interest to them by reading each dats metadata. Once a friend opens a dat they are also seeding it to others making it highly available. Consumers of hyperdats are free to seed individual dats or all of them. Its also possible to create "private" hyperdats. In fact hyperdats are just normal dats and are private by default, if you do not share your hyperdat publicly it is private. Private hyperdats can be sent to trusted peers using a side channel to create sneakernet networks. Publicly known hyperdats can also be used to enhance and expand sneakernets by expanding the available resources of the network.

Proposed hyperdat

dat://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

hyperdat.json

{
  public_key: 'xxxxxxxx',
  meta: { * },
  dats: [
    {
      dat: 'xxxx',
      meta: { * }
    },
    ...
  ]
}

The meta data can be anything from keywords, geolocation co-ordinates to data that may not have any meaning yet but may be acted upon or useful later.

One proposed limitation of a hyperdat is that all dats included must be signed by the same owner of the hyperdat. This allows all the included dats to be verified. There is no reason why the dats couldn't be owned by others but I need to think about the security implecations of this.

Dat search engine and hyper-index (DATNET).

Once there are peers with hyperdats, there could be a hyperdat-index where trusted hyperdats are submitted. The hyperdats can be traversed and turned into DHT (distributed hash table) or simply a list of available dats. Duplicates could be filtered out and other transformations applied. Read-only views based on the metadata would allow a user to search available dats based on meta-data and decide whether they want to download the dat before opening it. Malicous peers could be excluded from the hyperdat-index based on reputation (*not sure how this would work yet).

If malicious peers create evil hyperdats the hyperdat-index is free to omit them from its index and protect consumers of the index as a whole.

This would allow someone to type in "turnips" into a search bar and see a list of turnip related dats. Currently this functionality does not exist (and cannot exist) due to the inherently private nature of dats and dat discovery. Users would have to opt-in to making their hyperdats public. No one will find your dats unless you make the explicitly public using a hyperdat index.

Implementation

I am working on a working implementation of hyperdat, hyperdat-index and the search engine or DATNET. If you would like to collaborate or sponsor this contact m-onz via email: stphnmnslw@gmail.com

Comments, critics or suggestions

All critic, comments or suggestions are welcome, this is a draft proposal.

Useful links and resources

Dat project

Beaker Browser

Bit Torrent

Sneakernet