IRC: #dfi
on Freenode
A distributed file sharing and indexing network.
- a queryable database of torrents and metadata
- a p2p network for the distribution of torrent info hashes and metadata
Peer discovery is done via an implementation of Kademlia, a DHT similar to the one Bittorrent uses. Peers are assigned addresses similar to Bitcoin addresses, except using Ed25519 and SHA3.
DFI allows users to discover file indexes. Searches can then be performed on the remote peer for files that they are indexing, and file data is sent as a response. This includes a BitTorrent info-hash, which can then be used to download the file.
DFI also allows users to mirror the index of a peer. This massively enhances search speed, and allows anyone to take a complete backup of an index.
DFI can also be routed through any SOCKS proxy, and can create a Tor onion address automatically - this aids privacy and traverses the NAT, at the cost of performance.
Now! It will likely have some bugs, but is mostly working.
You can download DFI here
Make sure that you have make
installed. You will also need git
, and, of course, go
.
git clone https://github.com/dfi/dfi
cd dfi
make
The resulting binary will be in dfi/bin
.
The Makefile doesn't seem to work so well with Windows, so you'll need to have Go properly setup and installed, this may require setting $GOPATH
go get github.com/dfi/dfi
The resulting "dfid" binary should be automatically installed into your $GOPATH.
DFI presents a HTTP interface for usage, so you can interact with it using curl
. There is a command program available here, which makes interaction easier. siv is also downloadable from the release page.
Presently I am running several DFI nodes, you can bootstrap from this network as follows (assuming you have dfid running and listening on port 8080).
First you need Tor running: to do this, cd into the tor
directory, and run tor -f torrc
. You will also need to edit dfid.toml
and change tor.enabled
and socks.enabled
to be true. The permissions on the DFI directory may well need to be 700
for Tor to be happy.
To get started, simply run dfid. The output will contain your DFI address, which will look something like this: ZncGWimPZHWxjTMj51QNKg25PTCXphtLbh
In order to connect to the rest of the network, you will need to bootstrap. This can either be done using the below API, or using siv.
curl localhost:8080/self/bootstrap/x4yknq5x7iijrmgy.onion/
curl localhost:8080/self/explore/
By default, DFI listens on localhost:8080
. This is configurable in dfid.toml
.
These routes affect the local peer, ie the client running on your machine. They're generally used to interact with your own database, or change settings, etc.
This is used to add a post to your database, a post is essentially a torrent infohash and some metadata. The POST body requires a parameter of data
and index
.
The former is JSON, and is specified as such:
InfoHash string - the torrent infohash
Title string - a name for the post
Size int - the size of the torrent in bytes
FileCount int - number of files the torrent has
Seeders int - number of seeders the torrent has
Leechers int - number of leechers the torrent has
UploadDate int - Unix timestamp in seconds
Tags string - a comma-separated list of alphanumeric tags
Meta string - a JSON-encoded object
The other parameter, index
, should be either "true" or "false". This indicates whether or not DFI should add the post to the full text search index. If this is true, then the Title
field will be indexed and the post will show up in search results.
This performs a full text search index on all posts that have an id greater than {since}
.
This resolves a DFI address into a JSON entry. Entries are specified as such:
address Address
name string
desc string
publicAddress string
publicKey []byte
postCount int
updated uint64
signature []byte
collectionHash []byte
port int
seeds [][]byte
seeding [][]byte
seen int
Bootstraps the DFI node from the given address. This address must be a non-dfi address - for instance, a domain name, IP address, onion address, or anything else. Note that dfi can be configured to use a SOCKS proxy, see dfid.toml.
Perform a full text search on the local database.
This takes the parameters of query
and page
, where query is the search term and page is the page of results we want - this starts at 0.
Gets the most recent posts. The page is given as the {page}
parameter.
Gets the most popular posts. The page is given as the {page}
parameter.
Returns a list of peers.
Begin network exploration. This should happen automatically at start if you have peers in your routing table, otherwise it needs to be ran manually.
This is used to set various settings for the node. Here are possible values for {name}
:
- name: This sets the name field of the entry and can be used to identify your node
- desc: A short description of your entry
- public: set the public address for your entry, this is what it's DFI address will resolve to
Gets values, much like set - supports the same values. Also supports:
- dfi: gets the DFI address
- postcount: the number of posts this node has
- entry: the full DFI DHT entry
These routes allow you to query a remote peer. The {address}
parameter refers to the encoded DFI address of the peer, like the example above.
Pings the peer.
Announces your own entry to the peer.
Performs a remote search on the peer.
Download a local copy of the peer's post database, which can then be indexed and searched.
Search the local copy of the peer's database, this only works after a successful mirror
.
Get the {page}
of most recent posts for the given peer.
Performs a remote search on the peer.
Add all posts with an id larger than {since}
to the FTS index.