Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Gaia] Request for Storage Drivers #430

Closed
pstan26 opened this issue May 24, 2017 · 21 comments
Closed

[Gaia] Request for Storage Drivers #430

pstan26 opened this issue May 24, 2017 · 21 comments

Comments

@pstan26
Copy link
Contributor

@pstan26 pstan26 commented May 24, 2017

Request for Drivers

We're looking for talented engineers to port their favorite storage systems to Gaia, the decentralized Blockstack storage system.

Systems of interest include:

  • Sia
  • Storj
  • IPFS

We are offering a bounty of $5000 to the first person who submits a PR with a working driver implementation (among the list above) and test suite that meets our performance and code quality standards. If you have suggestions for other drivers, let us know in this issue! :-)

The specification is described here: https://github.com/blockstack/blockstack-core/blob/rc-0.14.2/blockstack_client/backend/drivers/_skel.py

If you have any code-related questions, please contact @jcnelson on this Github issue

NOTES:

  • Please develop off of the rc-0.14.2 branch of blockstack-core and virtualchain.
  • Please see the test.py storage driver in blockstack_client/backend/drivers/test.py. In particular, the storage driver must be compatible with the indexing system. This means that the actual loading and storing of data should conform to the requirements of driver_config() (called from storage_init()). The test_get_chunk(), test_put_chunk(), and test_delete_chunk() methods follow the required interface.
  • The dropbox.py storage driver is also a good working example.
@muneeb-ali
Copy link
Member

@muneeb-ali muneeb-ali commented May 24, 2017

I'm assuming the bounty will be paid in bitcoins ;-)

@pstan26
Copy link
Contributor Author

@pstan26 pstan26 commented May 24, 2017

@muneeb-ali Seems most reasonable actually :-)

@willmadison
Copy link

@willmadison willmadison commented May 24, 2017

@jcnelson Is it fairly safe to assume for a Sia implementation of the driver that it'd be talking to a Sia Daemon instance that is already unlocked, funded, and has contracts? These are all prerequisites for storing anything on the Sia network, or should the driver not make such assumptions?

@jcnelson
Copy link
Member

@jcnelson jcnelson commented May 24, 2017

Yes, the Sia driver may assume that it has access to an already-running and already-deployed Sia daemon. Same goes for a Storj bridge node and an IPFS daemon. However, the user needs the ability to set the relevant configuration parameters in the config file (such as the host/port and access credentials).

@willmadison
Copy link

@willmadison willmadison commented May 24, 2017

Awesome thanks @jcnelson

@willmadison
Copy link

@willmadison willmadison commented May 25, 2017

One additional question @jcnelson it appears pip install is having issues finding a suitable version of virtualchain such that it meets the minimum version requested in setup.py (i.e. 0.14.2 on the rc-0.14.2 branch). It appears that it's finding 0.14.1.2 but nothing higher than that. As a corollary to this I believe my unit tests are failing because I have the wrong version of virtual chain which doesn't contain a virtualchain.lib.ecdsalib module.

@willmadison
Copy link

@willmadison willmadison commented May 25, 2017

Answering my own question here. Just used pip install like so:

pip install git+https://github.com/blockstack/virtualchain.git@rc-0.14.2

To install the rc-0.14.2 version of virtual chain.

@jcnelson
Copy link
Member

@jcnelson jcnelson commented May 25, 2017

Yeah, you'll want rc-0.14.2 for virtualchain and blockstack-core. Will update the top post.

@shea256
Copy link
Member

@shea256 shea256 commented May 30, 2017

The next step here is integration with the browser.

First, we'll add un-clickable buttons for connecting IPFS, Sia and Storj.

Then, we'll have to have custom views with instructions for each one.

I'll create issues on the browser repo and link them here.

@jcnelson
Copy link
Member

@jcnelson jcnelson commented May 31, 2017

Hey everyone, great job on the pull requests so far! Just a quick update on what we expect from get_mutable_handler() and get_immutable_handler():

The driver must be written such that once a user writes data, any other user can read it given its data_id (from get_mutable_handler()) or its key (from get_immutable_handler()). The other users will not have the writer's API tokens, access credentials, or system-specific public keys. In all three of these systems, this basically means you have to use the indexing API to build up an index that resolves data_id and key values to the URIs that these systems generate.

The indexing system works by creating a top-level "manifest" URI that the user can attach to their profile. That is, given a blockchain ID (passed as fqu in the drivers' keyword arguments), the indexing system can look up a user's index manifest URI. Using this index manifest URI, the indexing system will fetch and store segments of the index to read, write, and delete (data_id, URI) mappings. The top-level methods of interest are:

  • driver_config(): sets up callbacks to be used by the indexer code for loading and storing both data and pages of the index. Returns a configuration structure to be passed into the below methods.

  • get_indexed_data(): loads data from the storage by translating an app-given name into a service-specific URI.

  • put_indexed_data(): stores data with a given name into the storage system, and inserts an entry for it in an index alongside the data.

  • delete_indexed_data(): removes data with a given name from the storage system, and updates the co-located index to remove its name-to-URI link.

  • index_setup(): instantiates an index (callable from the driver's storage_init() method).

You can see examples in the test.py and dropbox.py drivers on how to use this subsystem.

I have written a driver test harness to help confirm that your driver uses them correctly, located here. You have to test the drivers using two hosts: a "reader" host and a "writer" host.

To test the "writer", I'd do something like the following if I were testing the Dropbox driver:

    $ ssh jude@writer
    writer:$ cd ~/blockstack-core/integration_tests/bin
    writer:$ cp ~/.blockstack/client.ini /tmp/writer-config.ini
    writer:$ cat >> /tmp/writer-config.ini <<EOF
    [dropbox]
    token = $MY_DROPBOX_TOKEN
    EOF
    writer:$
    writer:$ cat ../datasets/test-datasets-dropbox.json.in | \
    sed -r 's~@MANIFEST_URI@~~g' > /tmp/test-datasets.json
    writer:$
    writer:$ ./blockstack-storage-driver-test \
    --debug \
    --config /tmp/writer-config.ini \
    --index \
    --datasets /tmp/test-datasets.json \
    blockstack_client.backend.drivers.dropbox put
    # ...
    
        index manifest URI: https://www.dropbox.com/s/ie8ab7qc8e5i0o3/index.manifest?dl=1

    writer:$ logout

Using the given URI above, I can now test the dropbox reader as follows:

    $ ssh jude@reader
    reader:$ cd blockstack-core/integration_tests/bin
    reader:$ cat ../datasets/test-datasets-dropbox.json.in | \
    sed -r 's~@MANIFEST_URI@~https://www.dropbox.com/s/ie8ab7qc8e5i0o3/index.manifest?dl=1~g' \ 
    > /tmp/test-datasets.json
    reader:$
    reader:$ ./blockstack-storage-driver-test \
    --debug \
    --index \
    --datasets /tmp/test-datasets.json \
    blockstack_client.backend.drivers.dropbox get

No Dropbox API token should be necessary for the last command to succeed. This is because the write paths in the Dropbox driver instantiated an index in my Dropbox account, and printed out the URI to the index manifest after writing some data.

If the user was connecting Dropbox to their account, what would normally happen is that this URI would be automatically added to the user's profile in a canonical location. This way, readers can proceed to access the writer's Dropbox-hosted data, despite the fact that Dropbox does not let readers simply construct URIs to data they did not write (like how they might in S3).

What we do to simulate this in the test harness is we use a "template" list of datasets to replicate and manually pass the manifest URI generated by the writer to the reader by means of including it as a driver hint. By doing so, the reader will succeed at resolving the data saved by the writer to the appropriate URI in Dropbox, which can then be fetched via HTTP.

Anyway, please experiment with this test harness with Dropbox, and please confirm that your drivers behave under similar circumstances. Thanks!

@willmadison
Copy link

@willmadison willmadison commented May 31, 2017

@jcnelson unfortunately the Sia driver can't support this yet until file sharing is enabled/possible on the Sia network. @DavidVorick noted this morning as we discussed this that file sharing is on the roadmap but hasn't been completed yet. The community advice was to not rush this feature as there are much more pressing matters in the interim but we could revisit this as soon as file-sharing is possible on the Sia network. Thoughts? (We all hate that this in some ways takes the Sia driver out of the running for the bounty but we had a great time learning and implementing the driver thus far).

@pstan26
Copy link
Contributor Author

@pstan26 pstan26 commented May 31, 2017

Yea thats a shame @willmadison. Well if adding drivers was interesting, we would be open to paying bounties for other drivers as well. We don't have a formal $ amount assigned to each just yet, but if you wanted to work on one, let us know which and we could then price it along with the other outstanding drivers for bounties.

@willmadison
Copy link

@willmadison willmadison commented May 31, 2017

Sure thing it was a ton of fun. Just let me know and I'd be happy to participate again for another backing service.

@pstan26 pstan26 changed the title Request for Drivers Request for Storage Drivers May 31, 2017
@pstan26 pstan26 changed the title Request for Storage Drivers [Gaia] Request for Storage Drivers Jun 6, 2017
@sayyedraza
Copy link

@sayyedraza sayyedraza commented Jul 22, 2017

is there any form to fill up for bounty ?

@pstan26
Copy link
Contributor Author

@pstan26 pstan26 commented Aug 28, 2017

@sayyedraza no form, just the first person who submits a PR with a working driver implementation and test suite that meets our performance and code quality standards gets the bounty.

@x5engine
Copy link
Contributor

@x5engine x5engine commented Jan 4, 2018

Am I too late for the bounty? I can submit a PR :)

@pstan26
Copy link
Contributor Author

@pstan26 pstan26 commented Jan 5, 2018

@meteorplus for which storage driver?

@jackzampolin
Copy link
Contributor

@jackzampolin jackzampolin commented Jan 11, 2018

We have moved the drivers to the Gaia hub and they are now written in JS. Closing this issue.

@pstan26
Copy link
Contributor Author

@pstan26 pstan26 commented Jan 12, 2018

@FIREFOXCYBER
Copy link

@FIREFOXCYBER FIREFOXCYBER commented Apr 14, 2020

So everything gone?

@x5engine
Copy link
Contributor

@x5engine x5engine commented Apr 17, 2020

woot woot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
You can’t perform that action at this time.