[Gaia] Request for Storage Drivers

[pstan26]

Request for Drivers

We're looking for talented engineers to port their favorite storage systems to Gaia, the decentralized Blockstack storage system.

Systems of interest include:

• Sia
• Storj
• IPFS

We are offering a bounty of $5000 to the first person who submits a PR with a working driver implementation (among the list above) and test suite that meets our performance and code quality standards. If you have suggestions for other drivers, let us know in this issue! :-)

The specification is described here: https://github.com/blockstack/blockstack-core/blob/rc-0.14.2/blockstack_client/backend/drivers/_skel.py

If you have any code-related questions, please contact @jcnelson on this Github issue

NOTES:

• Please develop off of the rc-0.14.2 branch of blockstack-core and virtualchain.
• Please see the test.py storage driver in blockstack_client/backend/drivers/test.py. In particular, the storage driver must be compatible with the indexing system. This means that the actual loading and storing of data should conform to the requirements of driver_config() (called from storage_init()). The test_get_chunk(), test_put_chunk(), and test_delete_chunk() methods follow the required interface.
• The dropbox.py storage driver is also a good working example.

[muneeb-ali]

I'm assuming the bounty will be paid in bitcoins ;-)

[pstan26]

@muneeb-ali Seems most reasonable actually :-)

[willmadison]

@jcnelson Is it fairly safe to assume for a Sia implementation of the driver that it'd be talking to a Sia Daemon instance that is already unlocked, funded, and has contracts? These are all prerequisites for storing anything on the Sia network, or should the driver not make such assumptions?

[jcnelson]

Yes, the Sia driver may assume that it has access to an already-running and already-deployed Sia daemon. Same goes for a Storj bridge node and an IPFS daemon. However, the user needs the ability to set the relevant configuration parameters in the config file (such as the host/port and access credentials).

[willmadison]

Awesome thanks @jcnelson

[willmadison]

One additional question @jcnelson it appears pip install is having issues finding a suitable version of virtualchain such that it meets the minimum version requested in setup.py (i.e. 0.14.2 on the rc-0.14.2 branch). It appears that it's finding 0.14.1.2 but nothing higher than that. As a corollary to this I believe my unit tests are failing because I have the wrong version of virtual chain which doesn't contain a virtualchain.lib.ecdsalib module.

[willmadison]

Answering my own question here. Just used pip install like so:

pip install git+https://github.com/blockstack/virtualchain.git@rc-0.14.2

To install the rc-0.14.2 version of virtual chain.

[jcnelson]

Yeah, you'll want rc-0.14.2 for virtualchain and blockstack-core. Will update the top post.

[shea256]

The next step here is integration with the browser.

First, we'll add un-clickable buttons for connecting IPFS, Sia and Storj.

Then, we'll have to have custom views with instructions for each one.

I'll create issues on the browser repo and link them here.

[jcnelson]

Hey everyone, great job on the pull requests so far! Just a quick update on what we expect from get_mutable_handler() and get_immutable_handler():

The driver must be written such that once a user writes data, any other user can read it given its data_id (from get_mutable_handler()) or its key (from get_immutable_handler()). The other users will not have the writer's API tokens, access credentials, or system-specific public keys. In all three of these systems, this basically means you have to use the indexing API to build up an index that resolves data_id and key values to the URIs that these systems generate.

The indexing system works by creating a top-level "manifest" URI that the user can attach to their profile. That is, given a blockchain ID (passed as fqu in the drivers' keyword arguments), the indexing system can look up a user's index manifest URI. Using this index manifest URI, the indexing system will fetch and store segments of the index to read, write, and delete (data_id, URI) mappings. The top-level methods of interest are:

• driver_config(): sets up callbacks to be used by the indexer code for loading and storing both data and pages of the index. Returns a configuration structure to be passed into the below methods.

• get_indexed_data(): loads data from the storage by translating an app-given name into a service-specific URI.

• put_indexed_data(): stores data with a given name into the storage system, and inserts an entry for it in an index alongside the data.

• delete_indexed_data(): removes data with a given name from the storage system, and updates the co-located index to remove its name-to-URI link.

• index_setup(): instantiates an index (callable from the driver's storage_init() method).

You can see examples in the test.py and dropbox.py drivers on how to use this subsystem.

I have written a driver test harness to help confirm that your driver uses them correctly, located here. You have to test the drivers using two hosts: a "reader" host and a "writer" host.

To test the "writer", I'd do something like the following if I were testing the Dropbox driver:

    $ ssh jude@writer
    writer:$ cd ~/blockstack-core/integration_tests/bin
    writer:$ cp ~/.blockstack/client.ini /tmp/writer-config.ini
    writer:$ cat >> /tmp/writer-config.ini <<EOF
    [dropbox]
    token = $MY_DROPBOX_TOKEN
    EOF
    writer:$
    writer:$ cat ../datasets/test-datasets-dropbox.json.in | \
    sed -r 's~@MANIFEST_URI@~~g' > /tmp/test-datasets.json
    writer:$
    writer:$ ./blockstack-storage-driver-test \
    --debug \
    --config /tmp/writer-config.ini \
    --index \
    --datasets /tmp/test-datasets.json \
    blockstack_client.backend.drivers.dropbox put
    # ...
    
        index manifest URI: https://www.dropbox.com/s/ie8ab7qc8e5i0o3/index.manifest?dl=1

    writer:$ logout

Using the given URI above, I can now test the dropbox reader as follows:

    $ ssh jude@reader
    reader:$ cd blockstack-core/integration_tests/bin
    reader:$ cat ../datasets/test-datasets-dropbox.json.in | \
    sed -r 's~@MANIFEST_URI@~https://www.dropbox.com/s/ie8ab7qc8e5i0o3/index.manifest?dl=1~g' \ 
    > /tmp/test-datasets.json
    reader:$
    reader:$ ./blockstack-storage-driver-test \
    --debug \
    --index \
    --datasets /tmp/test-datasets.json \
    blockstack_client.backend.drivers.dropbox get

No Dropbox API token should be necessary for the last command to succeed. This is because the write paths in the Dropbox driver instantiated an index in my Dropbox account, and printed out the URI to the index manifest after writing some data.

If the user was connecting Dropbox to their account, what would normally happen is that this URI would be automatically added to the user's profile in a canonical location. This way, readers can proceed to access the writer's Dropbox-hosted data, despite the fact that Dropbox does not let readers simply construct URIs to data they did not write (like how they might in S3).

What we do to simulate this in the test harness is we use a "template" list of datasets to replicate and manually pass the manifest URI generated by the writer to the reader by means of including it as a driver hint. By doing so, the reader will succeed at resolving the data saved by the writer to the appropriate URI in Dropbox, which can then be fetched via HTTP.

Anyway, please experiment with this test harness with Dropbox, and please confirm that your drivers behave under similar circumstances. Thanks!