Awesome IPLD endeavour #60

daviddias · 2016-10-03T10:16:34Z

WIP This PR updates the IPLD Resolver to become aware of multiple IPLD formats. It is also the entry point to track the migration that needed to happen across IPFS and IPLD module in order to support both.

Following up the technical discussions pre and post DEVCON2, this PR (and all the listed PRs) update the interfaces and add support to the panoply of IPLD Formats to live and be resolved through IPFS, without breaking links nor requiring transformations (this is huge).

Introducing interface-ipld-format

The interface-ipld-format is a specification for a data type that enables the IPLD Resolver to manipulate and resolve through different types of Merkle Data Structures. The interface is very simple:

UPDATE to the interface: See: #60 (comment)

Class DAGNode
- .serialize() - returns a serialized version of this instance
- .deserialize(<data>)- deserializes a binary blob into the instance
- .cid - returns the CID of this instance
resolver - the block level resolver (knows how to resolve paths at the block scope)
- .resolve(block, path) - resolves a path in block, returns the value and or a link and the partial missing path. This way the IPLD Resolver can fetch the link and continue to resolve.
- .tree(block, options) - returns all the paths available in this block

A full specification of the interface will be defined at https://github.com/ipld/interface-ipld-format. Meanwhile, you can use https://github.com/ipld/js-ipld-dag-pb/tree/awesome-ipld as an example.

UPDATE to the interface: See: #60 (comment)

Tasks

Moving to IPLD and starting to use IPLD formats, required changes to be applied from IPFS-Repo and up, like full support of CID, multicodec spec update and impl, etc. Here are the list of tasks that need to be performed to make this transition complete, so that is easier to track state.

daviddias · 2016-10-03T10:20:58Z

@wanderer o/ here is an update :) This is required me to make more changes than I hoped for, but things are getting ready and I'm feeling pretty happy about this "interface-ipld-format".

If you want to start hacking the js-ipld-eth-block, check ipld/js-ipld-dag-pb#1, it is pretty much there. I added you as a contributor to https://github.com/ipld/js-ipld-eth-block.

I'm going to continue and churn through the other things :)

dignifiedquire · 2016-10-03T10:22:02Z

@nicola I don't know how much you have seen of this, but this is important re the discussions we had in tr past two week. It seems this changes everything we planned for, so we have to start over again to a degree.

nicola · 2016-10-03T13:18:16Z

I don't think it's a drastic change, @diasdavid can you be in the call today?

dignifiedquire · 2016-10-03T13:23:18Z

I don't think it's a drastic change, @diasdavid can you be in the call today?
Add your review

Not in general, but in regards to the abstractions for the implementation I think it changes some things.

daviddias · 2016-10-17T01:50:04Z

Update to the interface-ipld-format

By implementing dag-cbor, I realized that we only (and benefit from) having an interface for serialize, deserialize and cid detached from the format class. This enables raw json objects to be passed around like dag-cbor, without having to do an instance of a specific DAGNode cbor class. That being said, the new interface for an IPLD format resumes to:

format.util
- .serialize(dagNode) - returns a serialized version of this instance
- .deserialize(<data>)- deserializes a binary blob into the instance
- .cid(dagNode) - returns the CID of this instance
format.resolver - the block level resolver (knows how to resolve paths at the block scope)
- .resolve(block, path) - resolves a path in block, returns the value and or a link and the partial missing path. This way the IPLD Resolver can fetch the link and continue to resolve.
- .tree(block, options) - returns all the paths available in this block

Another way to view it is that the main resolver doesn't really care what type does the 'deserialize' call return.

nicola · 2016-10-17T03:45:59Z

src/index.js

  }

  // Adds support for an IPLD format
  // default ones are dag-pb and dag-cbor
-  support (multicodec, type, resolver, util) {


how do you know now if something is cbor or pb?

Through the CID

wanderer · 2016-10-17T12:13:38Z

@diasdavid one other consideration (maybe its already been considered, im just still trying to grok) is that merkle links and the DAG topology / traversal should be independent of each other.

So for example, i would want to do something like

rootVertex.get(`/<ethereu, account hash>/`, account => {
   account.get('/balance/', (balanceVertex) => {
      console.log(balanceVertex.value) // how much ether this account has
  })
})

but underneath account doesn't have a true merkle link to balance. Its all part of the account blob. So account vertex has the edges balance, nonce, code, state. balance and nonce are just values. And code and state are merkle links.

daviddias · 2016-10-17T12:20:00Z

@wanderer you can achieve that in two ways:

// resolve the account and get its object
resolver.resolve('/<ethereum account hash>/', '/', (err, result) => {
  const account = result.value
  // then manipulate the account as you already do
})

// or resolve the account balance, using the path notation

resolver.resolve('/<ethereum account hash>/', 'balance', (err, result) => {
  const balance = result.value
  console.log(balance.value) // how much ether this account has
})

we can go through this during today's js-ipfs chat :)

daviddias · 2016-10-21T13:47:04Z

Getting close to that IPLD Week 2 goal :) An excellent way to achieve that is by CR the following PR:

PR	CR	RFM	M
#60	✔️	✔️	✔️
ipld/js-ipld-dag-pb#1	✔️	✔️	✔️
ipld/js-ipld-dag-cbor#25	✔️	✔️	✔️
multiformats/js-cid#4	✔️	✔️	✔️
ipfs-inactive/js-ipfs-unixfs-engine#74	✔️	✔️
multiformats/multicodec#16
ipld/js-ipld-block#5	✔️	✔️	✔️
ipfs/js-ipfs-block-service#27	✔️	✔️	✔️
ipfs/js-ipfs-repo#89	✔️	✔️	✔️
ipfs-inactive/interface-js-ipfs-core#78	✔️	✔️
ipfs/js-ipfs#532	✔️	✔️
ipfs-inactive/js-ipfs-http-client#398	✔️	✔️

Please, if you review it, remove the 'Needs review' label, so that work doesn't get duplicated.

//cc @victorbjelkholm @dignifiedquire @RichardLitt

MOVED TABLE TO: ipfs/js-ipfs#532 (comment)

nicola · 2016-10-21T14:16:01Z

src/index.js

+    }
+  }
+
+  resolve (cid, path, callback) {


why are cid and path two different inputs?
Can we not just pass cid+path?
the "deconcatenation" is something that I will happen to do often

@nicola note that CID is a instance of class CID that has several utility methods https://github.com/ipfs/js-cid/tree/feat/complete

What you are proposing is to use cids always as strings, which would assume that we have to be constantly parsing and mounting them again to get their props.

Oh, I see, perfect then.

There must be up in the levels of abstraction a sort of resolver that takes a path and resolve that (by using this, I guess)

nicola · 2016-10-21T14:18:15Z

src/index.js

+
+  // TODO: Consider if we still want this (fact: no one is using it
+  // right now and this will be just an IPLD selector anyway
+  getRecursive (cid, callback) {


During the Toronto's IPLD call, we mentioned that this should be a BFS and should have an optional parameter x where if set to false or -1, we explore recursively the entire graph, otherwise, we just move with depth x

What we talked was .tree method inside an IPLD format.

This getRecursive is an old method that comes from the previous "DAGService", which was never used in practice. This method returns the nodes directly, not their values.

What you make me think is that we should have a .tree at the main IPLD Resolver and make that option part of the interface-ipld-format for local resolvers too.

dignifiedquire · 2016-10-24T10:56:23Z

README.md


 [![](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](http://ipn.io)
 [![](https://img.shields.io/badge/project-IPFS-blue.svg?style=flat-square)](http://ipfs.io/)
 [![](https://img.shields.io/badge/freenode-%23ipfs-blue.svg?style=flat-square)](http://webchat.freenode.net/?channels=%23ipfs)
+[![Coverage Status](https://coveralls.io/repos/github/ipfs/js-ipld-resolver/badge.svg?branch=master)](https://coveralls.io/github/ipfs/js-ipld-resolver?branch=master)
+[![Travis CI](https://travis-ci.org/ipfs/js-ipld-resolver.svg?branch=master)](https://travis-ci.org/ipfs/js-ipld-resolver)


let's move this to the ipld organization as part of this effort

Oh, it's already on the IPLD org, forgot to change the badges. Good catch :)

dignifiedquire · 2016-10-24T10:56:44Z

package.json

-  "description": "IPLD implementation in JavaScript",
-  "main": "lib/index.js",
+  "description": "IPLD Resolver Implementation in JavaScript",
+  "main": "src/index.js",


TODO: remove before merge

dignifiedquire · 2016-10-24T10:58:16Z

src/index.js

+const CID = require('cids')
+const until = require('async/until')
+const IPFSRepo = require('ipfs-repo')
+const MemoryStore = require('../node_modules/interface-pull-blob-store/lib/reference.js')


you can do require('interface-pull-blob-store/lib/reference.js') instead, no need for the full path

right! (I do remember fighting with this one and this was how I got it to reference to the right one, not sure what was the error without node_modules though).

Could you also check ipfs-inactive/interface-pull-blob-store#6, this way we don't even have to point to a path inside.

dignifiedquire · 2016-10-24T10:58:56Z

src/index.js

+
+    // Support by default dag-pb and dag-cbor
+    this.support(dagPB.resolver.multicodec, dagPB.resolver, dagPB.util)
+    this.support(dagCBOR.resolver.multicodec, dagCBOR.resolver, dagCBOR.util)


should there be a way to disable these defaults?

What about having functions to add and remove, so that:

this.support.add

this.support.rm

dignifiedquire · 2016-10-24T10:59:30Z

src/index.js

+  // Adds support for an IPLD format
+  // default ones are dag-pb and dag-cbor
+  support (multicodec, resolver, util) {
+    this.resolvers[multicodec] = {


maybe warn on overriding?

dignifiedquire · 2016-10-24T11:00:38Z

src/index.js

+          if (err) {
+            return cb(err)
+          }
+          const result = this.resolvers[cid.codec].resolver.resolve(block, path)


there should be a check with a nice error message for this.resolvers[cid.codec] as this can easily be undefined

dignifiedquire · 2016-10-24T11:01:47Z

src/index.js

+    )
+  }
+
+  // TODO: Consider if we still want this (fact: no one is using it


lets use this opportunity to drop the recursive methods

dignifiedquire · 2016-10-24T11:02:35Z

test/test-ipld-all-together-now.js

+/*
+ * Test different types of data structures living together
+ * &
+ * Test data made of mixed data structures!


are you planning to fill this before merge?

Yes :) cbor + pb

dignifiedquire · 2016-10-24T11:04:22Z

test/test-ipld-dag-cbor.js

@@ -0,0 +1,205 @@
+/* eslint-env mocha */
+'use strict'


the tests in cbor and proto are super similar, maybe you could extract the test methods, and just pass in the type agains it should test?

Similar, but not the same, cbor is way more flexible and needs even more throughout tests. Creating a base tests would make every other format be tested very much like dag-pb, which is the less interesting data format because of its rigidity.

daviddias · 2016-10-26T12:13:17Z

@kumavis, @wanderer now you can git clone and npm install this repo without going through all the npm link hurdles, perfect time to test that new eth-block IPLD format :D

nicola · 2016-10-26T23:43:37Z

🎉

daviddias added 3 commits October 3, 2016 07:58

feat: Update API, use new block-service, block and ipld-dag-pb

edee5fa

test: let the testing begin

90ff073

feat: support for ipld-dag-pb

436f44c

daviddias added the status/in-progress In progress label Oct 3, 2016

daviddias mentioned this pull request Oct 3, 2016

Captain.log - IPLD v1 spec ipld/specs#13

Closed

2 tasks

daviddias mentioned this pull request Oct 3, 2016

Making eth an application rather than a format ipld/specs#29

Closed

daviddias added 2 commits October 16, 2016 05:22

feat: create a resolver in memory when no block-service is passed

00d5d46

feat: main resolver (understands dag-pb)

0818945

daviddias force-pushed the awesome-ipld branch from 03755d2 to 0818945 Compare October 16, 2016 07:16

feat: follow new dag-pb interface (from new interface-ipld-format

91ecfa4

daviddias mentioned this pull request Oct 17, 2016

Making resolve and IPLD service into a new package #58

Closed

2 tasks

nicola reviewed Oct 17, 2016

View reviewed changes

feat: add ipld-dag-cbor to the mix

a186055

daviddias force-pushed the awesome-ipld branch from de3a8ee to a186055 Compare October 17, 2016 10:25

daviddias added the needs review label Oct 21, 2016

nicola reviewed Oct 21, 2016

View reviewed changes

for testing

b7c2887

dignifiedquire reviewed Oct 24, 2016

View reviewed changes

daviddias removed the needs review label Oct 24, 2016

daviddias added 2 commits October 25, 2016 16:27

feat: move resolver to async IPLD format API, update dag-pb tests

39db563

feat: drop getRecursive functions

05d4e74

daviddias added 5 commits October 25, 2016 17:15

feat: update dag-cbor tests to async API, solve dag-cbor bug

733876b

docs: update badges

1bc5a87

fix: remove dead code

4aa591c

docs: update readme documentation

cd6e9b1

feat: resolve through different formats

7978fd6

daviddias merged commit fc98c00 into master Oct 26, 2016

daviddias deleted the awesome-ipld branch October 26, 2016 12:10

daviddias removed the status/in-progress In progress label Oct 26, 2016

daviddias mentioned this pull request Oct 26, 2016

Awesome IPLD endeavour ipfs/js-ipfs#532

Merged

daviddias mentioned this pull request Nov 23, 2016

feat: update DAGLink and DAGNode to have an immutable API ipld/js-ipld-dag-pb#6

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Awesome IPLD endeavour #60

Awesome IPLD endeavour #60

daviddias commented Oct 3, 2016 •

edited

daviddias commented Oct 3, 2016

dignifiedquire commented Oct 3, 2016

nicola commented Oct 3, 2016

dignifiedquire commented Oct 3, 2016

daviddias commented Oct 17, 2016 •

edited

nicola Oct 17, 2016

daviddias Oct 17, 2016

wanderer commented Oct 17, 2016

daviddias commented Oct 17, 2016

daviddias commented Oct 21, 2016 •

edited

nicola Oct 21, 2016

daviddias Oct 21, 2016

nicola Oct 21, 2016

nicola Oct 21, 2016

daviddias Oct 21, 2016 •

edited

nicola Oct 21, 2016

dignifiedquire Oct 24, 2016

daviddias Oct 24, 2016

dignifiedquire Oct 24, 2016

dignifiedquire Oct 24, 2016

daviddias Oct 24, 2016

dignifiedquire Oct 24, 2016

daviddias Oct 24, 2016

dignifiedquire Oct 24, 2016

daviddias Oct 24, 2016

dignifiedquire Oct 24, 2016

dignifiedquire Oct 24, 2016

daviddias Oct 24, 2016

dignifiedquire Oct 24, 2016

daviddias Oct 24, 2016 •

edited

dignifiedquire Oct 24, 2016

daviddias Oct 24, 2016 •

edited

daviddias commented Oct 26, 2016

nicola commented Oct 26, 2016

Awesome IPLD endeavour #60

Awesome IPLD endeavour #60

Conversation

daviddias commented Oct 3, 2016 • edited

Introducing interface-ipld-format

Tasks

daviddias commented Oct 3, 2016

dignifiedquire commented Oct 3, 2016

nicola commented Oct 3, 2016

dignifiedquire commented Oct 3, 2016

daviddias commented Oct 17, 2016 • edited

Update to the interface-ipld-format

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wanderer commented Oct 17, 2016

daviddias commented Oct 17, 2016

daviddias commented Oct 21, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias Oct 21, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias Oct 24, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias Oct 24, 2016 • edited

Choose a reason for hiding this comment

daviddias commented Oct 26, 2016

nicola commented Oct 26, 2016

daviddias commented Oct 3, 2016 •

edited

daviddias commented Oct 17, 2016 •

edited

daviddias commented Oct 21, 2016 •

edited

daviddias Oct 21, 2016 •

edited

daviddias Oct 24, 2016 •

edited

daviddias Oct 24, 2016 •

edited