New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Gaia] Request for Storage Drivers #430

Closed
pstan26 opened this Issue May 24, 2017 · 20 comments

Comments

Projects
None yet
9 participants
@pstan26
Member

pstan26 commented May 24, 2017

Request for Drivers

We're looking for talented engineers to port their favorite storage systems to Gaia, the decentralized Blockstack storage system.

Systems of interest include:

  • Sia
  • Storj
  • IPFS

We are offering a bounty of $5000 to the first person who submits a PR with a working driver implementation (among the list above) and test suite that meets our performance and code quality standards. If you have suggestions for other drivers, let us know in this issue! :-)

The specification is described here: https://github.com/blockstack/blockstack-core/blob/rc-0.14.2/blockstack_client/backend/drivers/_skel.py

If you have any code-related questions, please contact @jcnelson on this Github issue

NOTES:

  • Please develop off of the rc-0.14.2 branch of blockstack-core and virtualchain.
  • Please see the test.py storage driver in blockstack_client/backend/drivers/test.py. In particular, the storage driver must be compatible with the indexing system. This means that the actual loading and storing of data should conform to the requirements of driver_config() (called from storage_init()). The test_get_chunk(), test_put_chunk(), and test_delete_chunk() methods follow the required interface.
  • The dropbox.py storage driver is also a good working example.
@muneeb-ali

This comment has been minimized.

Show comment
Hide comment
@muneeb-ali

muneeb-ali May 24, 2017

Member

I'm assuming the bounty will be paid in bitcoins ;-)

Member

muneeb-ali commented May 24, 2017

I'm assuming the bounty will be paid in bitcoins ;-)

@pstan26

This comment has been minimized.

Show comment
Hide comment
@pstan26

pstan26 May 24, 2017

Member

@muneeb-ali Seems most reasonable actually :-)

Member

pstan26 commented May 24, 2017

@muneeb-ali Seems most reasonable actually :-)

@willmadison

This comment has been minimized.

Show comment
Hide comment
@willmadison

willmadison May 24, 2017

@jcnelson Is it fairly safe to assume for a Sia implementation of the driver that it'd be talking to a Sia Daemon instance that is already unlocked, funded, and has contracts? These are all prerequisites for storing anything on the Sia network, or should the driver not make such assumptions?

willmadison commented May 24, 2017

@jcnelson Is it fairly safe to assume for a Sia implementation of the driver that it'd be talking to a Sia Daemon instance that is already unlocked, funded, and has contracts? These are all prerequisites for storing anything on the Sia network, or should the driver not make such assumptions?

@jcnelson

This comment has been minimized.

Show comment
Hide comment
@jcnelson

jcnelson May 24, 2017

Member

Yes, the Sia driver may assume that it has access to an already-running and already-deployed Sia daemon. Same goes for a Storj bridge node and an IPFS daemon. However, the user needs the ability to set the relevant configuration parameters in the config file (such as the host/port and access credentials).

Member

jcnelson commented May 24, 2017

Yes, the Sia driver may assume that it has access to an already-running and already-deployed Sia daemon. Same goes for a Storj bridge node and an IPFS daemon. However, the user needs the ability to set the relevant configuration parameters in the config file (such as the host/port and access credentials).

@willmadison

This comment has been minimized.

Show comment
Hide comment
@willmadison

willmadison commented May 24, 2017

Awesome thanks @jcnelson

@willmadison

This comment has been minimized.

Show comment
Hide comment
@willmadison

willmadison May 25, 2017

One additional question @jcnelson it appears pip install is having issues finding a suitable version of virtualchain such that it meets the minimum version requested in setup.py (i.e. 0.14.2 on the rc-0.14.2 branch). It appears that it's finding 0.14.1.2 but nothing higher than that. As a corollary to this I believe my unit tests are failing because I have the wrong version of virtual chain which doesn't contain a virtualchain.lib.ecdsalib module.

willmadison commented May 25, 2017

One additional question @jcnelson it appears pip install is having issues finding a suitable version of virtualchain such that it meets the minimum version requested in setup.py (i.e. 0.14.2 on the rc-0.14.2 branch). It appears that it's finding 0.14.1.2 but nothing higher than that. As a corollary to this I believe my unit tests are failing because I have the wrong version of virtual chain which doesn't contain a virtualchain.lib.ecdsalib module.

@willmadison

This comment has been minimized.

Show comment
Hide comment
@willmadison

willmadison May 25, 2017

Answering my own question here. Just used pip install like so:

pip install git+https://github.com/blockstack/virtualchain.git@rc-0.14.2

To install the rc-0.14.2 version of virtual chain.

willmadison commented May 25, 2017

Answering my own question here. Just used pip install like so:

pip install git+https://github.com/blockstack/virtualchain.git@rc-0.14.2

To install the rc-0.14.2 version of virtual chain.

@jcnelson

This comment has been minimized.

Show comment
Hide comment
@jcnelson

jcnelson May 25, 2017

Member

Yeah, you'll want rc-0.14.2 for virtualchain and blockstack-core. Will update the top post.

Member

jcnelson commented May 25, 2017

Yeah, you'll want rc-0.14.2 for virtualchain and blockstack-core. Will update the top post.

@shea256

This comment has been minimized.

Show comment
Hide comment
@shea256

shea256 May 30, 2017

Member

The next step here is integration with the browser.

First, we'll add un-clickable buttons for connecting IPFS, Sia and Storj.

Then, we'll have to have custom views with instructions for each one.

I'll create issues on the browser repo and link them here.

Member

shea256 commented May 30, 2017

The next step here is integration with the browser.

First, we'll add un-clickable buttons for connecting IPFS, Sia and Storj.

Then, we'll have to have custom views with instructions for each one.

I'll create issues on the browser repo and link them here.

@jcnelson

This comment has been minimized.

Show comment
Hide comment
@jcnelson

jcnelson May 31, 2017

Member

Hey everyone, great job on the pull requests so far! Just a quick update on what we expect from get_mutable_handler() and get_immutable_handler():

The driver must be written such that once a user writes data, any other user can read it given its data_id (from get_mutable_handler()) or its key (from get_immutable_handler()). The other users will not have the writer's API tokens, access credentials, or system-specific public keys. In all three of these systems, this basically means you have to use the indexing API to build up an index that resolves data_id and key values to the URIs that these systems generate.

The indexing system works by creating a top-level "manifest" URI that the user can attach to their profile. That is, given a blockchain ID (passed as fqu in the drivers' keyword arguments), the indexing system can look up a user's index manifest URI. Using this index manifest URI, the indexing system will fetch and store segments of the index to read, write, and delete (data_id, URI) mappings. The top-level methods of interest are:

  • driver_config(): sets up callbacks to be used by the indexer code for loading and storing both data and pages of the index. Returns a configuration structure to be passed into the below methods.

  • get_indexed_data(): loads data from the storage by translating an app-given name into a service-specific URI.

  • put_indexed_data(): stores data with a given name into the storage system, and inserts an entry for it in an index alongside the data.

  • delete_indexed_data(): removes data with a given name from the storage system, and updates the co-located index to remove its name-to-URI link.

  • index_setup(): instantiates an index (callable from the driver's storage_init() method).

You can see examples in the test.py and dropbox.py drivers on how to use this subsystem.

I have written a driver test harness to help confirm that your driver uses them correctly, located here. You have to test the drivers using two hosts: a "reader" host and a "writer" host.

To test the "writer", I'd do something like the following if I were testing the Dropbox driver:

    $ ssh jude@writer
    writer:$ cd ~/blockstack-core/integration_tests/bin
    writer:$ cp ~/.blockstack/client.ini /tmp/writer-config.ini
    writer:$ cat >> /tmp/writer-config.ini <<EOF
    [dropbox]
    token = $MY_DROPBOX_TOKEN
    EOF
    writer:$
    writer:$ cat ../datasets/test-datasets-dropbox.json.in | \
    sed -r 's~@MANIFEST_URI@~~g' > /tmp/test-datasets.json
    writer:$
    writer:$ ./blockstack-storage-driver-test \
    --debug \
    --config /tmp/writer-config.ini \
    --index \
    --datasets /tmp/test-datasets.json \
    blockstack_client.backend.drivers.dropbox put
    # ...
    
        index manifest URI: https://www.dropbox.com/s/ie8ab7qc8e5i0o3/index.manifest?dl=1

    writer:$ logout

Using the given URI above, I can now test the dropbox reader as follows:

    $ ssh jude@reader
    reader:$ cd blockstack-core/integration_tests/bin
    reader:$ cat ../datasets/test-datasets-dropbox.json.in | \
    sed -r 's~@MANIFEST_URI@~https://www.dropbox.com/s/ie8ab7qc8e5i0o3/index.manifest?dl=1~g' \ 
    > /tmp/test-datasets.json
    reader:$
    reader:$ ./blockstack-storage-driver-test \
    --debug \
    --index \
    --datasets /tmp/test-datasets.json \
    blockstack_client.backend.drivers.dropbox get

No Dropbox API token should be necessary for the last command to succeed. This is because the write paths in the Dropbox driver instantiated an index in my Dropbox account, and printed out the URI to the index manifest after writing some data.

If the user was connecting Dropbox to their account, what would normally happen is that this URI would be automatically added to the user's profile in a canonical location. This way, readers can proceed to access the writer's Dropbox-hosted data, despite the fact that Dropbox does not let readers simply construct URIs to data they did not write (like how they might in S3).

What we do to simulate this in the test harness is we use a "template" list of datasets to replicate and manually pass the manifest URI generated by the writer to the reader by means of including it as a driver hint. By doing so, the reader will succeed at resolving the data saved by the writer to the appropriate URI in Dropbox, which can then be fetched via HTTP.

Anyway, please experiment with this test harness with Dropbox, and please confirm that your drivers behave under similar circumstances. Thanks!

Member

jcnelson commented May 31, 2017

Hey everyone, great job on the pull requests so far! Just a quick update on what we expect from get_mutable_handler() and get_immutable_handler():

The driver must be written such that once a user writes data, any other user can read it given its data_id (from get_mutable_handler()) or its key (from get_immutable_handler()). The other users will not have the writer's API tokens, access credentials, or system-specific public keys. In all three of these systems, this basically means you have to use the indexing API to build up an index that resolves data_id and key values to the URIs that these systems generate.

The indexing system works by creating a top-level "manifest" URI that the user can attach to their profile. That is, given a blockchain ID (passed as fqu in the drivers' keyword arguments), the indexing system can look up a user's index manifest URI. Using this index manifest URI, the indexing system will fetch and store segments of the index to read, write, and delete (data_id, URI) mappings. The top-level methods of interest are:

  • driver_config(): sets up callbacks to be used by the indexer code for loading and storing both data and pages of the index. Returns a configuration structure to be passed into the below methods.

  • get_indexed_data(): loads data from the storage by translating an app-given name into a service-specific URI.

  • put_indexed_data(): stores data with a given name into the storage system, and inserts an entry for it in an index alongside the data.

  • delete_indexed_data(): removes data with a given name from the storage system, and updates the co-located index to remove its name-to-URI link.

  • index_setup(): instantiates an index (callable from the driver's storage_init() method).

You can see examples in the test.py and dropbox.py drivers on how to use this subsystem.

I have written a driver test harness to help confirm that your driver uses them correctly, located here. You have to test the drivers using two hosts: a "reader" host and a "writer" host.

To test the "writer", I'd do something like the following if I were testing the Dropbox driver:

    $ ssh jude@writer
    writer:$ cd ~/blockstack-core/integration_tests/bin
    writer:$ cp ~/.blockstack/client.ini /tmp/writer-config.ini
    writer:$ cat >> /tmp/writer-config.ini <<EOF
    [dropbox]
    token = $MY_DROPBOX_TOKEN
    EOF
    writer:$
    writer:$ cat ../datasets/test-datasets-dropbox.json.in | \
    sed -r 's~@MANIFEST_URI@~~g' > /tmp/test-datasets.json
    writer:$
    writer:$ ./blockstack-storage-driver-test \
    --debug \
    --config /tmp/writer-config.ini \
    --index \
    --datasets /tmp/test-datasets.json \
    blockstack_client.backend.drivers.dropbox put
    # ...
    
        index manifest URI: https://www.dropbox.com/s/ie8ab7qc8e5i0o3/index.manifest?dl=1

    writer:$ logout

Using the given URI above, I can now test the dropbox reader as follows:

    $ ssh jude@reader
    reader:$ cd blockstack-core/integration_tests/bin
    reader:$ cat ../datasets/test-datasets-dropbox.json.in | \
    sed -r 's~@MANIFEST_URI@~https://www.dropbox.com/s/ie8ab7qc8e5i0o3/index.manifest?dl=1~g' \ 
    > /tmp/test-datasets.json
    reader:$
    reader:$ ./blockstack-storage-driver-test \
    --debug \
    --index \
    --datasets /tmp/test-datasets.json \
    blockstack_client.backend.drivers.dropbox get

No Dropbox API token should be necessary for the last command to succeed. This is because the write paths in the Dropbox driver instantiated an index in my Dropbox account, and printed out the URI to the index manifest after writing some data.

If the user was connecting Dropbox to their account, what would normally happen is that this URI would be automatically added to the user's profile in a canonical location. This way, readers can proceed to access the writer's Dropbox-hosted data, despite the fact that Dropbox does not let readers simply construct URIs to data they did not write (like how they might in S3).

What we do to simulate this in the test harness is we use a "template" list of datasets to replicate and manually pass the manifest URI generated by the writer to the reader by means of including it as a driver hint. By doing so, the reader will succeed at resolving the data saved by the writer to the appropriate URI in Dropbox, which can then be fetched via HTTP.

Anyway, please experiment with this test harness with Dropbox, and please confirm that your drivers behave under similar circumstances. Thanks!

@willmadison

This comment has been minimized.

Show comment
Hide comment
@willmadison

willmadison May 31, 2017

@jcnelson unfortunately the Sia driver can't support this yet until file sharing is enabled/possible on the Sia network. @DavidVorick noted this morning as we discussed this that file sharing is on the roadmap but hasn't been completed yet. The community advice was to not rush this feature as there are much more pressing matters in the interim but we could revisit this as soon as file-sharing is possible on the Sia network. Thoughts? (We all hate that this in some ways takes the Sia driver out of the running for the bounty but we had a great time learning and implementing the driver thus far).

willmadison commented May 31, 2017

@jcnelson unfortunately the Sia driver can't support this yet until file sharing is enabled/possible on the Sia network. @DavidVorick noted this morning as we discussed this that file sharing is on the roadmap but hasn't been completed yet. The community advice was to not rush this feature as there are much more pressing matters in the interim but we could revisit this as soon as file-sharing is possible on the Sia network. Thoughts? (We all hate that this in some ways takes the Sia driver out of the running for the bounty but we had a great time learning and implementing the driver thus far).

@pstan26

This comment has been minimized.

Show comment
Hide comment
@pstan26

pstan26 May 31, 2017

Member

Yea thats a shame @willmadison. Well if adding drivers was interesting, we would be open to paying bounties for other drivers as well. We don't have a formal $ amount assigned to each just yet, but if you wanted to work on one, let us know which and we could then price it along with the other outstanding drivers for bounties.

Member

pstan26 commented May 31, 2017

Yea thats a shame @willmadison. Well if adding drivers was interesting, we would be open to paying bounties for other drivers as well. We don't have a formal $ amount assigned to each just yet, but if you wanted to work on one, let us know which and we could then price it along with the other outstanding drivers for bounties.

@willmadison

This comment has been minimized.

Show comment
Hide comment
@willmadison

willmadison May 31, 2017

Sure thing it was a ton of fun. Just let me know and I'd be happy to participate again for another backing service.

willmadison commented May 31, 2017

Sure thing it was a ton of fun. Just let me know and I'd be happy to participate again for another backing service.

@pstan26 pstan26 changed the title from Request for Drivers to Request for Storage Drivers May 31, 2017

@pstan26 pstan26 changed the title from Request for Storage Drivers to [Gaia] Request for Storage Drivers Jun 6, 2017

@sayyedraza

This comment has been minimized.

Show comment
Hide comment
@sayyedraza

sayyedraza Jul 22, 2017

is there any form to fill up for bounty ?

sayyedraza commented Jul 22, 2017

is there any form to fill up for bounty ?

@pstan26

This comment has been minimized.

Show comment
Hide comment
@pstan26

pstan26 Aug 28, 2017

Member

@sayyedraza no form, just the first person who submits a PR with a working driver implementation and test suite that meets our performance and code quality standards gets the bounty.

Member

pstan26 commented Aug 28, 2017

@sayyedraza no form, just the first person who submits a PR with a working driver implementation and test suite that meets our performance and code quality standards gets the bounty.

@meteorplus

This comment has been minimized.

Show comment
Hide comment
@meteorplus

meteorplus Jan 4, 2018

Am I too late for the bounty? I can submit a PR :)

meteorplus commented Jan 4, 2018

Am I too late for the bounty? I can submit a PR :)

@pstan26

This comment has been minimized.

Show comment
Hide comment
@pstan26

pstan26 Jan 5, 2018

Member

@meteorplus for which storage driver?

Member

pstan26 commented Jan 5, 2018

@meteorplus for which storage driver?

@jackzampolin

This comment has been minimized.

Show comment
Hide comment
@jackzampolin

jackzampolin Jan 11, 2018

Contributor

We have moved the drivers to the Gaia hub and they are now written in JS. Closing this issue.

Contributor

jackzampolin commented Jan 11, 2018

We have moved the drivers to the Gaia hub and they are now written in JS. Closing this issue.

@pstan26

This comment has been minimized.

Show comment
Hide comment
@pstan26

pstan26 Jan 12, 2018

Member
Member

pstan26 commented Jan 12, 2018

@Cryptolinux73

This comment has been minimized.

Show comment
Hide comment
@Cryptolinux73

Cryptolinux73 Jan 27, 2018

Alors j'ai installer Sia , j'ai synchronisé ,mais suis pas encore réussi a faire le Wallet sous version V1.6 sauf erreur !!

Je epnse que ça dois être moi , Peu pas copier et des mot en ENGLISH , donc un de ces jour je vais re essayer de config le Wallet pour essayer le mining sous linux , ubuntu17.10 :)

:) Killers of the's SCAMS ***-> Team RiriCoinCoin http:... & Partageons les Richesse mondial-> Team RiriCoinCoin ......(.Ne sais Jamais si c'est interdit !!P) :) http://t.me/.....

Cryptolinux73 commented Jan 27, 2018

Alors j'ai installer Sia , j'ai synchronisé ,mais suis pas encore réussi a faire le Wallet sous version V1.6 sauf erreur !!

Je epnse que ça dois être moi , Peu pas copier et des mot en ENGLISH , donc un de ces jour je vais re essayer de config le Wallet pour essayer le mining sous linux , ubuntu17.10 :)

:) Killers of the's SCAMS ***-> Team RiriCoinCoin http:... & Partageons les Richesse mondial-> Team RiriCoinCoin ......(.Ne sais Jamais si c'est interdit !!P) :) http://t.me/.....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment