New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

馃専 Parity to IPFS #763

Open
diasdavid opened this Issue Feb 10, 2017 · 28 comments

Comments

Projects
None yet
6 participants
@diasdavid
Member

diasdavid commented Feb 10, 2017

Update: I rewrote this first post to make it more useful for tracking the developments of this.

Goal: Add a 'Storage Backend' for js-ipfs that backed by Parity, the Rust implementation of Ethereum. This will enable js-ipfs nodes to fetch content from the Ethereum Network as if it was just a big bucket with all the blocks from the chain.

TASKS

  • rust-cid
  • Implement an http endpoint in Parity that lets us retrieve the blocks
  • Implement the datastore backend that mounts on top of parity
  • Try it out, a lot :)
@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis Feb 11, 2017

Contributor

from parity mh-ipfs branch

cargo build --release
./target/release/parity

while thats running, you can make requests by cid

curl 'http://localhost:5001/api/v0/block/get?arg=z45oqTRzb9a5xyvx5RbfSXH1K5jibyZ4AxnXyYReuLw7KU5veYw'
Contributor

kumavis commented Feb 11, 2017

from parity mh-ipfs branch

cargo build --release
./target/release/parity

while thats running, you can make requests by cid

curl 'http://localhost:5001/api/v0/block/get?arg=z45oqTRzb9a5xyvx5RbfSXH1K5jibyZ4AxnXyYReuLw7KU5veYw'
@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis Feb 11, 2017

Contributor

i made this util to help with this https://github.com/kumavis/eth-ipld

Contributor

kumavis commented Feb 11, 2017

i made this util to help with this https://github.com/kumavis/eth-ipld

@diasdavid diasdavid added the ready label Feb 11, 2017

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Mar 21, 2017

Member

@kumavis let's pick up on this again, now that we have datastore bubbling up all the way :)

Member

diasdavid commented Mar 21, 2017

@kumavis let's pick up on this again, now that we have datastore bubbling up all the way :)

@diasdavid diasdavid changed the title from Parity to IPFS to 馃専 Parity to IPFS Mar 21, 2017

@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis Mar 22, 2017

Contributor

whee

Contributor

kumavis commented Mar 22, 2017

whee

@diasdavid diasdavid referenced this issue Mar 22, 2017

Closed

馃殌 0.23 Release 馃専 #795

18 of 22 tasks complete
@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis Mar 29, 2017

Contributor

seems simple enough

https://github.com/ipfs/js-datastore-level
https://github.com/ipfs/interface-datastore

how do you mount multiple stores? and can you mount one as read-only?

Contributor

kumavis commented Mar 29, 2017

seems simple enough

https://github.com/ipfs/js-datastore-level
https://github.com/ipfs/interface-datastore

how do you mount multiple stores? and can you mount one as read-only?

@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis Mar 29, 2017

Contributor

also - do you have a preferred isomorphic xhr module? otherwise ill use what i usually do
EDIT: I guess the http module works fine for browserify, not sure about webpack

Contributor

kumavis commented Mar 29, 2017

also - do you have a preferred isomorphic xhr module? otherwise ill use what i usually do
EDIT: I guess the http module works fine for browserify, not sure about webpack

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Mar 29, 2017

Member

We shim stream-http in js-ipfs-api https://github.com/ipfs/js-ipfs-api/blob/master/package.json#L10

The code for reaching to the block store, given that it exposes the same API as go-ipfs does for blocks, is exactly what js-ipfs-api does https://github.com/ipfs/js-ipfs-api/blob/master/src/api/block.js

On read-only, it means that essentially when it calls write, you say 'All good folks', but in reality does nothing.

Right now, ipfs-repo is mounting a single datastore for the blockstore (blocks folder) https://github.com/ipfs/js-ipfs-repo/blob/master/src/index.js#L136-L139, which is them used throughout blockstore code: https://github.com/ipfs/js-ipfs-repo/blob/master/src/blockstore.js#L34-L137

Tasks:

  • Ability to mount more than one datastore for blockstore (make it an array essentially)
  • Make ./blockstore check if there is an array of this.store and issue the get/put to all of them.

Note: blockstore will pass a base32 encoded CID to your parity-datastore, which you then have to decode to send through the js-ipfs-api block.get.

Member

diasdavid commented Mar 29, 2017

We shim stream-http in js-ipfs-api https://github.com/ipfs/js-ipfs-api/blob/master/package.json#L10

The code for reaching to the block store, given that it exposes the same API as go-ipfs does for blocks, is exactly what js-ipfs-api does https://github.com/ipfs/js-ipfs-api/blob/master/src/api/block.js

On read-only, it means that essentially when it calls write, you say 'All good folks', but in reality does nothing.

Right now, ipfs-repo is mounting a single datastore for the blockstore (blocks folder) https://github.com/ipfs/js-ipfs-repo/blob/master/src/index.js#L136-L139, which is them used throughout blockstore code: https://github.com/ipfs/js-ipfs-repo/blob/master/src/blockstore.js#L34-L137

Tasks:

  • Ability to mount more than one datastore for blockstore (make it an array essentially)
  • Make ./blockstore check if there is an array of this.store and issue the get/put to all of them.

Note: blockstore will pass a base32 encoded CID to your parity-datastore, which you then have to decode to send through the js-ipfs-api block.get.

@kumavis kumavis referenced this issue May 6, 2017

Closed

libp2p mesh testing #719

15 of 36 tasks complete
@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis May 6, 2017

Contributor

would be nice if js-ipfs-repo's data store could be specified externally somehow
seems hardcoded in with a lot of other config https://github.com/ipfs/js-ipfs-repo/blob/master/src/index.js#L136-L139

Contributor

kumavis commented May 6, 2017

would be nice if js-ipfs-repo's data store could be specified externally somehow
seems hardcoded in with a lot of other config https://github.com/ipfs/js-ipfs-repo/blob/master/src/index.js#L136-L139

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid May 6, 2017

Member

Agreed, right now the decision is being made inside the ipfs-repo module by signalling through the browser property in the package.json, here:

https://github.com/ipfs/js-ipfs-repo/blob/master/package.json#L8

I would rather have those options passed in, so that we can hot swap it per application basis, just like libp2p bundles.

Member

diasdavid commented May 6, 2017

Agreed, right now the decision is being made inside the ipfs-repo module by signalling through the browser property in the package.json, here:

https://github.com/ipfs/js-ipfs-repo/blob/master/package.json#L8

I would rather have those options passed in, so that we can hot swap it per application basis, just like libp2p bundles.

@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis May 6, 2017

Contributor

would be nice if we could boot the http api by passing in an instance of an ipfs node, instead of it needing to boot its own

Contributor

kumavis commented May 6, 2017

would be nice if we could boot the http api by passing in an instance of an ipfs node, instead of it needing to boot its own

@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis May 6, 2017

Contributor

for now, i think this will work for ipfs-repo's datastore but NotAGoodIdea鈩

node.on('ready', () => {
  console.log('node ready!')
  // hacky late add to MountStore
  console.log('Adding mount to Parity')
  repo.store.mounts.push({
    prefix: new Key('/'),
    datastore: new RemoteIpfsDataStore({host: 'localhost', port: '5002', protocol: 'http'})
  })
  setupHttpApi()
})
Contributor

kumavis commented May 6, 2017

for now, i think this will work for ipfs-repo's datastore but NotAGoodIdea鈩

node.on('ready', () => {
  console.log('node ready!')
  // hacky late add to MountStore
  console.log('Adding mount to Parity')
  repo.store.mounts.push({
    prefix: new Key('/'),
    datastore: new RemoteIpfsDataStore({host: 'localhost', port: '5002', protocol: 'http'})
  })
  setupHttpApi()
})
@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis May 6, 2017

Contributor

hmmm seems bitswap is somehow not using my late-mounted datastore

Contributor

kumavis commented May 6, 2017

hmmm seems bitswap is somehow not using my late-mounted datastore

@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis May 6, 2017

Contributor

ipfs/js-ipfs-api#550 ( not blocking, using a string as a workaround )

Contributor

kumavis commented May 6, 2017

ipfs/js-ipfs-api#550 ( not blocking, using a string as a workaround )

@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis May 6, 2017

Contributor

ipfs-api uses POST for all requests, parity only implements GET for block.get

API spec seems to indicate it should be a GET
https://github.com/ipfs/http-api-spec/blob/master/apiary.apib

Contributor

kumavis commented May 6, 2017

ipfs-api uses POST for all requests, parity only implements GET for block.get

API spec seems to indicate it should be a GET
https://github.com/ipfs/http-api-spec/blob/master/apiary.apib

@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis May 6, 2017

Contributor

PR for the http method fix here ipfs/js-ipfs-api#551

Contributor

kumavis commented May 6, 2017

PR for the http method fix here ipfs/js-ipfs-api#551

@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis May 6, 2017

Contributor

Remaining IPFS todo's

optional:

  • fix js-ipfs-api cid serialization ipfs/js-ipfs-api#550
  • change the http-api server api to accept an ipfs instance instead of a ipfs repo location
  • change the ipfs-repo datastore config to be easier to configure at the app level

attn @diasdavid

Contributor

kumavis commented May 6, 2017

Remaining IPFS todo's

optional:

  • fix js-ipfs-api cid serialization ipfs/js-ipfs-api#550
  • change the http-api server api to accept an ipfs instance instead of a ipfs repo location
  • change the ipfs-repo datastore config to be easier to configure at the app level

attn @diasdavid

@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis May 8, 2017

Contributor

i have it working and deployed (via patched deps) https://github.com/kumavis/ipfs-eth-bridge

Contributor

kumavis commented May 8, 2017

i have it working and deployed (via patched deps) https://github.com/kumavis/ipfs-eth-bridge

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid May 21, 2017

Member

Checked the boxes that are already done

publish datastore-ipfs-http-api (is there a less horrible name to use?)

Yeah, just call itjs-datastore-ipfs, it is an IPFS adapter for the datastore. Kind of a mobius strip datastore, but that is what it is :) Just make it clear in the Readme


@kumavis did you manage to figure out the bug of the other day?

Member

diasdavid commented May 21, 2017

Checked the boxes that are already done

publish datastore-ipfs-http-api (is there a less horrible name to use?)

Yeah, just call itjs-datastore-ipfs, it is an IPFS adapter for the datastore. Kind of a mobius strip datastore, but that is what it is :) Just make it clear in the Readme


@kumavis did you manage to figure out the bug of the other day?

@kumavis

This comment has been minimized.

Show comment
Hide comment
@kumavis

kumavis May 22, 2017

Contributor

because of the GET/POST mismatch, ended up making a "read-only-hooked datastore" module, used here: https://github.com/kumavis/ipfs-eth-bridge/blob/master/index.js#L24-L39

Contributor

kumavis commented May 22, 2017

because of the GET/POST mismatch, ended up making a "read-only-hooked datastore" module, used here: https://github.com/kumavis/ipfs-eth-bridge/blob/master/index.js#L24-L39

@diasdavid diasdavid added the P1 - High label Oct 18, 2017

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Oct 18, 2017

Member

@kumavis What are the next steps?

Member

diasdavid commented Oct 18, 2017

@kumavis What are the next steps?

@b1u3h4t

This comment has been minimized.

Show comment
Hide comment
@b1u3h4t

b1u3h4t Dec 4, 2017

how can I find the block data in IPFS format when i use parity with --ipfs-api --ipfs-api-hosts all?
It also return the error block not found, anyone can help me?

b1u3h4t commented Dec 4, 2017

how can I find the block data in IPFS format when i use parity with --ipfs-api --ipfs-api-hosts all?
It also return the error block not found, anyone can help me?

@5chdn

This comment has been minimized.

Show comment
Hide comment
@5chdn

5chdn Dec 4, 2017

Currently, you can only get block headers, block lists, uncles, transactions, and the state-trie node.

See https://github.com/paritytech/parity/wiki/IPFS

5chdn commented Dec 4, 2017

Currently, you can only get block headers, block lists, uncles, transactions, and the state-trie node.

See https://github.com/paritytech/parity/wiki/IPFS

@b1u3h4t

This comment has been minimized.

Show comment
Hide comment
@b1u3h4t

b1u3h4t Dec 4, 2017

Do i need an ipfs instance running? if not, how the eth data is encode to ipfs data format?

b1u3h4t commented Dec 4, 2017

Do i need an ipfs instance running? if not, how the eth data is encode to ipfs data format?

@5chdn

This comment has been minimized.

Show comment
Hide comment
@5chdn

5chdn Dec 4, 2017

You need a Parity node running with IPFS enabled. There is no need for an additional IPFS node unless you want to access other content/APIs via IPFS.

5chdn commented Dec 4, 2017

You need a Parity node running with IPFS enabled. There is no need for an additional IPFS node unless you want to access other content/APIs via IPFS.

@b1u3h4t

This comment has been minimized.

Show comment
Hide comment
@b1u3h4t

b1u3h4t Dec 4, 2017

Is the eth data not saved in ipfs format? Is there no ipfs store indeed? I can't get any data directory about ipfs info, but when i start an additional ipfs node, i can find data in ~/.ipfs/ directory, so where is partiy's ipfs data store?

b1u3h4t commented Dec 4, 2017

Is the eth data not saved in ipfs format? Is there no ipfs store indeed? I can't get any data directory about ipfs info, but when i start an additional ipfs node, i can find data in ~/.ipfs/ directory, so where is partiy's ipfs data store?

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Feb 21, 2018

Member

It would be amazing to have a package on npm that downloads the Parity client and exposes the datastore with it to access the Eth network so that we can create a quick example on how to set it up. Does this sound like something you can handle in the near future, @kumavis @hermanjunge?

Member

diasdavid commented Feb 21, 2018

It would be amazing to have a package on npm that downloads the Parity client and exposes the datastore with it to access the Eth network so that we can create a quick example on how to set it up. Does this sound like something you can handle in the near future, @kumavis @hermanjunge?

@hermanjunge

This comment has been minimized.

Show comment
Hide comment
@hermanjunge

hermanjunge Feb 21, 2018

That's right @diasdavid. This is one of our main pain points for the ethereum browser light client currently. I am working on extracting the functions I need from go-ethereum to store the K-V data into a Redis DB cluster.

It would be nice to have a slim parity client version that synchronizes to the network, to store the data into a database we can easily mount as an IPFS client datastore.

For now, we have this docker compose package to do the trick, is a js-ipfs client that mounts the parity HTTP IPFS, that way we can access eth dag nodes one by one. The ideal, of course, is unleash more capabilities.

https://github.com/kumavis/ipfs-eth-bridge/blob/master/mustekala/docker-compose.yml

hermanjunge commented Feb 21, 2018

That's right @diasdavid. This is one of our main pain points for the ethereum browser light client currently. I am working on extracting the functions I need from go-ethereum to store the K-V data into a Redis DB cluster.

It would be nice to have a slim parity client version that synchronizes to the network, to store the data into a database we can easily mount as an IPFS client datastore.

For now, we have this docker compose package to do the trick, is a js-ipfs client that mounts the parity HTTP IPFS, that way we can access eth dag nodes one by one. The ideal, of course, is unleash more capabilities.

https://github.com/kumavis/ipfs-eth-bridge/blob/master/mustekala/docker-compose.yml

@jualy007

This comment has been minimized.

Show comment
Hide comment
@jualy007

jualy007 Jun 22, 2018

Great work

jualy007 commented Jun 22, 2018

Great work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment