New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: IPLD spec #37

Merged
merged 38 commits into from Feb 12, 2016

Conversation

Projects
None yet
@jbenet
Member

jbenet commented Nov 8, 2015

This PR adds a new IPLD spec.

Some things TODO:

  • paths: link to path issue in go-ipfs or go-ipld
  • paths: list path resolving restrictions
  • paths: show examples of path resolving
  • examples/bitcoin: make this a real txn
  • more examples

@mildred @diasdavid could you review?

@jbenet jbenet added the in progress label Nov 8, 2015

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Nov 8, 2015

Member

Another TODO:

  • multicodec: attribute to signal the canonical multicodec for a datastructure. I.e. if a datastructure ought to use a canonical serializing format (e.g. different than cbor), we can use an attribute like @multicodec to signal that to implementations.

It may need to be stored in the serialized format, since decoding it will yield a multicodec. maybe just in the in memory logical representation, so serialization happens correctly. (With the exception for old style proto/mdagv1) (thoughts @mildred?)

Member

jbenet commented Nov 8, 2015

Another TODO:

  • multicodec: attribute to signal the canonical multicodec for a datastructure. I.e. if a datastructure ought to use a canonical serializing format (e.g. different than cbor), we can use an attribute like @multicodec to signal that to implementations.

It may need to be stored in the serialized format, since decoding it will yield a multicodec. maybe just in the in memory logical representation, so serialization happens correctly. (With the exception for old style proto/mdagv1) (thoughts @mildred?)

TODO:
- [ ] list path resolving restrictions
- [ ] show examples

This comment has been minimized.

@jbenet

jbenet Nov 8, 2015

Member

@willglynn could you help me fill this out?

the more concise here the better, but i suspect this section may be a bit large.

see rendered doc here: https://github.com/ipfs/specs/blob/ipld-spec/merkledag/ipld.md

@jbenet

jbenet Nov 8, 2015

Member

@willglynn could you help me fill this out?

the more concise here the better, but i suspect this section may be a bit large.

see rendered doc here: https://github.com/ipfs/specs/blob/ipld-spec/merkledag/ipld.md

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Nov 8, 2015

Member

More TODOs:

  • @attributes: describe @ escaping for the potential future @attributes. (cc @mildred)
  • Linked Data: add a section describing relationship to proper linked data formats, JSON-LD, etc.
  • Linked Data: add a JSON-LD example
  • Linked Data: add an RDF example
Member

jbenet commented Nov 8, 2015

More TODOs:

  • @attributes: describe @ escaping for the potential future @attributes. (cc @mildred)
  • Linked Data: add a section describing relationship to proper linked data formats, JSON-LD, etc.
  • Linked Data: add a JSON-LD example
  • Linked Data: add an RDF example
Show outdated Hide outdated merkledag/ipld.md
}
}
> ipld cat --json QmBBB...BBB/author

This comment has been minimized.

@noffle

noffle Nov 9, 2015

Contributor

shouldn't this resolve to

"mlink": "QmAAA...AAA" // links to the node above.

? Same for the YAML example below.

@noffle

noffle Nov 9, 2015

Contributor

shouldn't this resolve to

"mlink": "QmAAA...AAA" // links to the node above.

? Same for the YAML example below.

This comment has been minimized.

@eminence

eminence Nov 16, 2015

I think you're right and this was a copy/paste typo

@eminence

eminence Nov 16, 2015

I think you're right and this was a copy/paste typo

This comment has been minimized.

@jbenet

jbenet Nov 22, 2015

Member

ah thanks. it should actually be:

{
  "name": "Vannevar Bush"
}

addressed in 37c662a

@jbenet

jbenet Nov 22, 2015

Member

ah thanks. it should actually be:

{
  "name": "Vannevar Bush"
}

addressed in 37c662a

Show outdated Hide outdated merkledag/ipld.md
|
v
+-------------------+
O_5 = | "hello": "world" | whose hash value is QmR8Bzg59Y4FGWHeu9iTYhwhiP8PHCNFiaGhP1UjywA43j

This comment has been minimized.

@noffle

noffle Nov 9, 2015

Contributor

Might it improve reading comprehensibility if you mention beforehand that "hello": "world" is stored at the path /ipfs/QmUmg7BZC1YP1ca66rRtWKxpXp77WgVHrnv263JtDuvs2k/a/b/c/d? I had a moment of "wait, what? why is that key-value at the end?", which may/may not happen to others.

@noffle

noffle Nov 9, 2015

Contributor

Might it improve reading comprehensibility if you mention beforehand that "hello": "world" is stored at the path /ipfs/QmUmg7BZC1YP1ca66rRtWKxpXp77WgVHrnv263JtDuvs2k/a/b/c/d? I had a moment of "wait, what? why is that key-value at the end?", which may/may not happen to others.

This comment has been minimized.

@jbenet

jbenet Nov 22, 2015

Member

Yep, thanks. addressed in 37c662a

@jbenet

jbenet Nov 22, 2015

Member

Yep, thanks. addressed in 37c662a

Show outdated Hide outdated merkledag/ipld.md
## Serialized Data Formats
IPLD supports a variety of serialized data formats trough [multicodec](https://github.com/jbenet/multicodec). These can be used however is idiomatic to the format, for example in `CBOR`, we can use `CBOR` type tags to represent the merkle-link, and avoid writing out the full string key `mlink`. Users are encouraged to use the formats to their fullest, and to store and transmit IPLD data in whatever format makes the most sense. The only requirement **is that there MUST be a well-defined one-to-one mapping with the IPLD Canonical format.** This is so that data can be transformed from one format to another, and back, without changing its meaning nor its cryptographic hashes.

This comment has been minimized.

@noffle

noffle Nov 9, 2015

Contributor

s/trough/through

@noffle

noffle Nov 9, 2015

Contributor

s/trough/through

This comment has been minimized.

@jbenet

jbenet Nov 22, 2015

Member

thanks addressed in 37c662a

@jbenet

jbenet Nov 22, 2015

Member

thanks addressed in 37c662a

Show outdated Hide outdated merkledag/ipld.md
### Canonical Format
In order to preserve merkle-linking's power, we muste ensure that there is a single **_canonical_** serialized representation of an IPLD document. This ensures that applications arrive at the same cryptographic hashes. It should be noted --though-- that this is a system-wide parameter. Future systems might change it to evolve representations. However we estimate this would need to be done no more than once per decade.

This comment has been minimized.

@noffle

noffle Nov 9, 2015

Contributor

s/muste/must

@noffle

noffle Nov 9, 2015

Contributor

s/muste/must

This comment has been minimized.

@mildred

mildred Nov 9, 2015

Contributor

At least, when IPFS attains a stable release, this should not be changed.

@mildred

mildred Nov 9, 2015

Contributor

At least, when IPFS attains a stable release, this should not be changed.

This comment has been minimized.

@jbenet

jbenet Nov 22, 2015

Member

@mildred i wonder if we should be specifying the canonical encoding in the expanded data itself (or at least allowing it to be overridden). storing the data will include the multicodec header, but when we parse it into logical representations (or even other wire formats), we might want to specify the canonical format.

Aside, for future thought, this is something that a @type could do -- similar to @context it could define some defaults that hold for all instances. So we could have something like:

  • some chosen canonical multicodec default (in our case, cbor)
  • can override canonical multicodec in the object's type or context.
  • can override canonical multicodec in the object itself
@jbenet

jbenet Nov 22, 2015

Member

@mildred i wonder if we should be specifying the canonical encoding in the expanded data itself (or at least allowing it to be overridden). storing the data will include the multicodec header, but when we parse it into logical representations (or even other wire formats), we might want to specify the canonical format.

Aside, for future thought, this is something that a @type could do -- similar to @context it could define some defaults that hold for all instances. So we could have something like:

  • some chosen canonical multicodec default (in our case, cbor)
  • can override canonical multicodec in the object's type or context.
  • can override canonical multicodec in the object itself

This comment has been minimized.

@mildred

mildred Nov 22, 2015

Contributor

I don't quite understand what you propose here. In my opinion, there isn't so much value to allow arbitrary wire representation.

@mildred

mildred Nov 22, 2015

Contributor

I don't quite understand what you propose here. In my opinion, there isn't so much value to allow arbitrary wire representation.

@noffle

This comment has been minimized.

Show comment
Hide comment
@noffle

noffle Nov 9, 2015

Contributor

I left a few nitpicks. On the whole I found this doc hugely helpful in understanding IPLD and its intent.

Contributor

noffle commented Nov 9, 2015

I left a few nitpicks. On the whole I found this doc hugely helpful in understanding IPLD and its intent.

Show outdated Hide outdated merkledag/ipld.md
- `ipfs` is a protocol namespace (to allow the computer to discern what to do)
- `QmUmg7BZC1YP1ca66rRtWKxpXp77WgVHrnv263JtDuvs2k` is a cryptographic hash.
- `a/b/c/d` is a path _traversal_, as in unix.
- this link traverses five objects.

This comment has been minimized.

@mildred

mildred Nov 9, 2015

Contributor

Not necessarily.

We could have the object QmUmg7BZC1YP1ca66rRtWKxpXp77WgVHrnv263JtDuvs2k having a link named a/b/c/d directly pointing to the final object (or any combination in between).

@mildred

mildred Nov 9, 2015

Contributor

Not necessarily.

We could have the object QmUmg7BZC1YP1ca66rRtWKxpXp77WgVHrnv263JtDuvs2k having a link named a/b/c/d directly pointing to the final object (or any combination in between).

This comment has been minimized.

@jbenet

jbenet Nov 9, 2015

Member

@mildred ah that's right we did say we would allow sparse. one question remains re ordering of links there-- do we want to take them by lexicographic order? or in order in the serialized fmt?

ordering based on the serialized fmt will be needed if links have same name/no name (someone WILL do it so ipld implementations should be written to handle the case even if we say people should not do it)

but ordering lexicographically when links do have names is useful for getting users to expect the same behavior.

how do we handle this?

> ipld cat --fmt yml $h1
---
foo: {mlink: $h2}
foo/bar: {mlink: $h3}

> ipld cat --fmt yml $h2
---
bar:
 hello: h2bar1

> ipld cat --fmt yml $h3
---
hello: h3bar2

> ipld cat --fmt yml $h1/foo/bar
# ??? should it be
---
hello: h2bar1

# or should it be
---
hello: h3bar2
@jbenet

jbenet Nov 9, 2015

Member

@mildred ah that's right we did say we would allow sparse. one question remains re ordering of links there-- do we want to take them by lexicographic order? or in order in the serialized fmt?

ordering based on the serialized fmt will be needed if links have same name/no name (someone WILL do it so ipld implementations should be written to handle the case even if we say people should not do it)

but ordering lexicographically when links do have names is useful for getting users to expect the same behavior.

how do we handle this?

> ipld cat --fmt yml $h1
---
foo: {mlink: $h2}
foo/bar: {mlink: $h3}

> ipld cat --fmt yml $h2
---
bar:
 hello: h2bar1

> ipld cat --fmt yml $h3
---
hello: h3bar2

> ipld cat --fmt yml $h1/foo/bar
# ??? should it be
---
hello: h2bar1

# or should it be
---
hello: h3bar2

This comment has been minimized.

@mildred

mildred Nov 9, 2015

Contributor

I would not expect this to be an issue because what you describe is the compact form of the following object:

> ipld cat --fmt yml $h1
---
foo: 
  mlink: $h2
  bar:
    mlink: $h3

And in this object, only the "foo" link will be considered valid, not "foo/bar" because :

  • keys should not be allowed to have / character (so we will never have "foo/bar" verbatim in a key) for the same reasons / is not allowed in a unix filename.

  • in the previous object, "foo/bar" will not be considered as a link as per the "Duplicate property keys" section :

    Note that having two properties with the same name IS NOT ALLOWED, but actually impossible to prevent (someone will do it and feed it to parsers), so to be safe, we define the value of the path traversal to be the first entry in the serialized representation. For example, suppose we have the object:

@mildred

mildred Nov 9, 2015

Contributor

I would not expect this to be an issue because what you describe is the compact form of the following object:

> ipld cat --fmt yml $h1
---
foo: 
  mlink: $h2
  bar:
    mlink: $h3

And in this object, only the "foo" link will be considered valid, not "foo/bar" because :

  • keys should not be allowed to have / character (so we will never have "foo/bar" verbatim in a key) for the same reasons / is not allowed in a unix filename.

  • in the previous object, "foo/bar" will not be considered as a link as per the "Duplicate property keys" section :

    Note that having two properties with the same name IS NOT ALLOWED, but actually impossible to prevent (someone will do it and feed it to parsers), so to be safe, we define the value of the path traversal to be the first entry in the serialized representation. For example, suppose we have the object:

This comment has been minimized.

@mildred

mildred Nov 9, 2015

Contributor
ipld cat --fmt yml $h1/foo/bar

should be h2bar1

@mildred

mildred Nov 9, 2015

Contributor
ipld cat --fmt yml $h1/foo/bar

should be h2bar1

This comment has been minimized.

@jbenet

jbenet Nov 22, 2015

Member
  • keys should not be allowed to have / character

i agree, but unlike unix pathnames, there already are datastructs out there that we should be able to store, even if the resolution through them is not perfect. I.e. if we define how the resolution would work even in this case, we avoid the problem of forcing users to change their data.

@jbenet

jbenet Nov 22, 2015

Member
  • keys should not be allowed to have / character

i agree, but unlike unix pathnames, there already are datastructs out there that we should be able to store, even if the resolution through them is not perfect. I.e. if we define how the resolution would work even in this case, we avoid the problem of forcing users to change their data.

This comment has been minimized.

@mildred

mildred Nov 25, 2015

Contributor

I'd say, don't resolve links for keys that have / in them. We can still store data structure which have those keys, we just can't resolve them through paths. I don't see a problem in that. We would have a separate API to parse the local data structure without following links.

@mildred

mildred Nov 25, 2015

Contributor

I'd say, don't resolve links for keys that have / in them. We can still store data structure which have those keys, we just can't resolve them through paths. I don't see a problem in that. We would have a separate API to parse the local data structure without following links.

This comment has been minimized.

@davidar

davidar Dec 5, 2015

Member

Why not escape slashes, so:

> ipld cat --fmt yml $h1/foo/bar
---
hello: h2bar1

> ipld cat --fmt yml $h1/foo\/bar
---
hello: h3bar2
@davidar

davidar Dec 5, 2015

Member

Why not escape slashes, so:

> ipld cat --fmt yml $h1/foo/bar
---
hello: h2bar1

> ipld cat --fmt yml $h1/foo\/bar
---
hello: h3bar2

This comment has been minimized.

@mildred

mildred Dec 10, 2015

Contributor

I suppose that you meant to put the path in simple quotes, else your bourne compatible shell will replace \/ by / and both commands are identical.

> ipld cat --fmt yml '$h1/foo\/bar'
---
hello: h3bar2

The kernel doesn't know escaping so what the linux kernel will understand when presented with the path foo\/bar is the entry bar enclosed in a directory called foo\. Backslashes in file names are valid and this notation would prevent using them.

@mildred

mildred Dec 10, 2015

Contributor

I suppose that you meant to put the path in simple quotes, else your bourne compatible shell will replace \/ by / and both commands are identical.

> ipld cat --fmt yml '$h1/foo\/bar'
---
hello: h3bar2

The kernel doesn't know escaping so what the linux kernel will understand when presented with the path foo\/bar is the entry bar enclosed in a directory called foo\. Backslashes in file names are valid and this notation would prevent using them.

This comment has been minimized.

@davidar

davidar Dec 10, 2015

Member

Sure, you wouldn't be able to access keys like that with the fuse mount, but I don't see why that doesn't mean the ipld command couldn't support escaping? If you wanted a literal backslash, you'd write \\

@davidar

davidar Dec 10, 2015

Member

Sure, you wouldn't be able to access keys like that with the fuse mount, but I don't see why that doesn't mean the ipld command couldn't support escaping? If you wanted a literal backslash, you'd write \\

Some Constraints:
- IPLD paths MUST be unambiguous. A given path string MUST always deterministically traverse to the same object. (e.g. avoid duplicating link names)
- IPLD paths MUST be universal and avoid opressing non-english societies (e.g. use UTF-8, not ASCII).
- IPLD paths MUST layer cleanly over UNIX and The Web (use `/`, have deterministic transforms for ASCII systems).

This comment has been minimized.

@mildred

mildred Nov 9, 2015

Contributor

Just nit picking but ASCII is defined on 7 bits only, and UTF-8 (in which we encode to) is 8 bits. What we have generally in unix systems as filepath encoding is arbitrary 8bit encoding that cannot be qualified as ASCII. It could be qualified as a binary string or locale dependent encoded string.

@mildred

mildred Nov 9, 2015

Contributor

Just nit picking but ASCII is defined on 7 bits only, and UTF-8 (in which we encode to) is 8 bits. What we have generally in unix systems as filepath encoding is arbitrary 8bit encoding that cannot be qualified as ASCII. It could be qualified as a binary string or locale dependent encoded string.

This comment has been minimized.

@jbenet

jbenet Nov 22, 2015

Member

ok, want to adjust the language directly? what should it say?

@jbenet

jbenet Nov 22, 2015

Member

ok, want to adjust the language directly? what should it say?

This comment has been minimized.

@mildred

mildred Nov 25, 2015

Contributor
  • IPLD paths MUST layer cleanly over UNIX and The Web (use /, have deterministic transforms for paths not having specified character encoding).
@mildred

mildred Nov 25, 2015

Contributor
  • IPLD paths MUST layer cleanly over UNIX and The Web (use /, have deterministic transforms for paths not having specified character encoding).
Show outdated Hide outdated merkledag/ipld.md
- Given the wide success of JSON, a huge number of systems present JSON interfaces. IPLD MUST be able to import and export to JSON trivially.
- The JSON data model is also very simple and easy to use. IPLD MUST be just as easy to use.
- Definining new datastructures MUST be trivially easy. It should not be cumbersome -- or require much knowledge -- to experiment with new definitions on top of IPLD.
- IPLD MUST be compatible with RDF and the proper W3C Semantic Web / Linked Data standards. We can achieve this easily through JSON-LD.

This comment has been minimized.

@mildred

mildred Nov 9, 2015

Contributor

compatibility is a very wide term. If this is a MUST, then we should probably define it more clearly. Or else move it elsewhere by saying for example:

As a result of this design and IPLD being based on the JSON data model, it is fully compatible with RDF and Linked Data standards.

@mildred

mildred Nov 9, 2015

Contributor

compatibility is a very wide term. If this is a MUST, then we should probably define it more clearly. Or else move it elsewhere by saying for example:

As a result of this design and IPLD being based on the JSON data model, it is fully compatible with RDF and Linked Data standards.

This comment has been minimized.

@jbenet

jbenet Nov 22, 2015

Member

sounds good. addressed in 37c662a

@jbenet

jbenet Nov 22, 2015

Member

sounds good. addressed in 37c662a

}
```
Suppose this hashes to the multihash value `QmBBB...BBB`. This node links the _subpath `author` to `QmAAA...AAA`, the node in the section above. So we can now do:

This comment has been minimized.

@mildred

mildred Nov 9, 2015

Contributor

Perhaps we should define this more precisely. What are links ?

In go-ipld, this is defined by a node having a mlink key that points to a string. it is assumed that the string content is a valid multihash but that is only checked when trying to access the link.

@mildred

mildred Nov 9, 2015

Contributor

Perhaps we should define this more precisely. What are links ?

In go-ipld, this is defined by a node having a mlink key that points to a string. it is assumed that the string content is a valid multihash but that is only checked when trying to access the link.

This comment has been minimized.

@davidar

davidar Dec 5, 2015

Member

This is why I like the data-constructor notation better

title: As We May Think
author: !mlink QmAAA...AAA

It makes it clear that we're constructing a link with the given hash as the argument to the constructor. The magic "mlink" key pointing to a random string is not so clear.

I know that JSON doesn't have that luxury, so it'll have to go with magic keys, but I think it's worth making the distinction at the protocol level.

@davidar

davidar Dec 5, 2015

Member

This is why I like the data-constructor notation better

title: As We May Think
author: !mlink QmAAA...AAA

It makes it clear that we're constructing a link with the given hash as the argument to the constructor. The magic "mlink" key pointing to a random string is not so clear.

I know that JSON doesn't have that luxury, so it'll have to go with magic keys, but I think it's worth making the distinction at the protocol level.

@mildred

This comment has been minimized.

Show comment
Hide comment
@mildred

mildred Nov 9, 2015

Contributor

Another TODO:

  • multicodec: attribute to signal the canonical multicodec for a datastructure. I.e. if a datastructure ought to use a canonical serializing format (e.g. different than cbor), we can use an attribute like @multicodec to signal that to implementations.

It may need to be stored in the serialized format, since decoding it will yield a multicodec. maybe just in the in memory logical representation, so serialization happens correctly. (With the exception for old style proto/mdagv1) (thoughts @mildred?)

In that case, we should either provision key escaping to avoid clashing with someone wanting to have "@multicodec" as an arbitrary key in his JSON data structure (think a @multicodec in a directory object). Or we should transmit this information outside of the IPLD object itself (not encoded as part of the datastructure).

Contributor

mildred commented Nov 9, 2015

Another TODO:

  • multicodec: attribute to signal the canonical multicodec for a datastructure. I.e. if a datastructure ought to use a canonical serializing format (e.g. different than cbor), we can use an attribute like @multicodec to signal that to implementations.

It may need to be stored in the serialized format, since decoding it will yield a multicodec. maybe just in the in memory logical representation, so serialization happens correctly. (With the exception for old style proto/mdagv1) (thoughts @mildred?)

In that case, we should either provision key escaping to avoid clashing with someone wanting to have "@multicodec" as an arbitrary key in his JSON data structure (think a @multicodec in a directory object). Or we should transmit this information outside of the IPLD object itself (not encoded as part of the datastructure).

@mildred

This comment has been minimized.

Show comment
Hide comment
@mildred

mildred Nov 9, 2015

Contributor

👍 on this whole thing. Still a major TODO for me is the description of the @ character escaping for introducing IPLD specific directives.

By the way, why choose the @ character to introduce directives? This might be confusing in the context of JSON-LD where the same character is used (but it will be escaped in the IPLD object). We are free to choose anything ($%!#&?*+...)

Contributor

mildred commented Nov 9, 2015

👍 on this whole thing. Still a major TODO for me is the description of the @ character escaping for introducing IPLD specific directives.

By the way, why choose the @ character to introduce directives? This might be confusing in the context of JSON-LD where the same character is used (but it will be escaped in the IPLD object). We are free to choose anything ($%!#&?*+...)

@jbenet jbenet referenced this pull request Nov 9, 2015

Closed

Sprint Nov 2 #46

7 of 11 tasks complete
- **IPLD Serialized Formats**: a set of formats in which IPLD objects can be represented, for example JSON, CBOR, CSON, YAML, Protobuf, XML, RDF, etc.
- **IPLD Canonical Format**: a deterministic description on a serialized format that ensures the same _logical_ object is always serialized to _the exact same sequence of bits_. This is critical for merkle-linking, and all cryptographic applications.
In short: JSON documents with named merkle-links that can be traversed.

This comment has been minimized.

@diasdavid

diasdavid Nov 13, 2015

Member

traversed or resolved?

@diasdavid

diasdavid Nov 13, 2015

Member

traversed or resolved?

This comment has been minimized.

@jbenet

jbenet Nov 22, 2015

Member

both -- i think one (traversed) implies the other (resolved)

@jbenet

jbenet Nov 22, 2015

Member

both -- i think one (traversed) implies the other (resolved)

This comment has been minimized.

@diasdavid

diasdavid Jan 2, 2016

Member

got it, my initial thought was that merkle links can be resolved and something like a smart bitswap can traverse

@diasdavid

diasdavid Jan 2, 2016

Member

got it, my initial thought was that merkle links can be resolved and something like a smart bitswap can traverse

Show outdated Hide outdated merkledag/ipld.md
{
"title": "As We May Think",
"author": {
"mlink": "QmAAA...AAA" // links to the node above.

This comment has been minimized.

@diasdavid

diasdavid Nov 13, 2015

Member

Better start calling them merkle-link so that we can have a clear distinction with mazieres-link

@diasdavid

diasdavid Nov 13, 2015

Member

Better start calling them merkle-link so that we can have a clear distinction with mazieres-link

This comment has been minimized.

@jbenet

jbenet Nov 22, 2015

Member

that will blow up verbosity and storage size. I was thinking, maybe we should drop mlink in favor of just link or @link (or the special char we go for). the link itself can denote what type of link it is:

  • if it has no prefix (not a path, just a hash), it is a merkle-link. ("QmAAA...AAA")
  • it may still have a prefix and be a merkle link ("/ipfs/QmAAA...AAA/foo/bar")
  • it may be another type of link ("/ipns/foo.com/a/b/c"`) but is not counted as immutable
@jbenet

jbenet Nov 22, 2015

Member

that will blow up verbosity and storage size. I was thinking, maybe we should drop mlink in favor of just link or @link (or the special char we go for). the link itself can denote what type of link it is:

  • if it has no prefix (not a path, just a hash), it is a merkle-link. ("QmAAA...AAA")
  • it may still have a prefix and be a merkle link ("/ipfs/QmAAA...AAA/foo/bar")
  • it may be another type of link ("/ipns/foo.com/a/b/c"`) but is not counted as immutable

This comment has been minimized.

@jbenet

jbenet Nov 22, 2015

Member

We could use href instead of link and be retro + HTML reminiscent.

@jbenet

jbenet Nov 22, 2015

Member

We could use href instead of link and be retro + HTML reminiscent.

This comment has been minimized.

@davidar

davidar Dec 5, 2015

Member

that will blow up verbosity and storage size.

If we're storing data as CBOR, and encoding mlink as a tag, that shouldn't be an issue? It would increase verbosity when displaying in a human-readable format though, yes.

@davidar

davidar Dec 5, 2015

Member

that will blow up verbosity and storage size.

If we're storing data as CBOR, and encoding mlink as a tag, that shouldn't be an issue? It would increase verbosity when displaying in a human-readable format though, yes.

Show outdated Hide outdated merkledag/ipld.md
|
v
+-------------------+
O_2 = | "b": "QmV76pU..." | whose hash value is QmV76pUdAAukxEHt9Wp2xwyTpiCmzJCvjnMxyQBreaUeKT

This comment has been minimized.

@diasdavid

diasdavid Nov 13, 2015

Member

The hashes on the pointers are repeated

@diasdavid

diasdavid Nov 13, 2015

Member

The hashes on the pointers are repeated

This comment has been minimized.

@jbenet

jbenet Nov 22, 2015

Member

good catch! addressed in: 37c662a

@jbenet

jbenet Nov 22, 2015

Member

good catch! addressed in: 37c662a

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Nov 13, 2015

Member

I don't see any specific mention to the data types the links point to, have we decided that isn't IPLD turf (and just json-ld)?

Member

diasdavid commented Nov 13, 2015

I don't see any specific mention to the data types the links point to, have we decided that isn't IPLD turf (and just json-ld)?

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Nov 13, 2015

Member

Also, instead of adding more TODO here, can we before go issue by issue here https://github.com/ipfs/go-ipld/issues and ipld/js-ipld-dag-cbor#2 and check what is already included in the spec, add what is missing, explicitly add what was accepted and then see what's left, bringing all of that conversation to this PR? So that we all talk over the same page :)

Member

diasdavid commented Nov 13, 2015

Also, instead of adding more TODO here, can we before go issue by issue here https://github.com/ipfs/go-ipld/issues and ipld/js-ipld-dag-cbor#2 and check what is already included in the spec, add what is missing, explicitly add what was accepted and then see what's left, bringing all of that conversation to this PR? So that we all talk over the same page :)

@mildred

This comment has been minimized.

Show comment
Hide comment
@mildred

mildred Nov 13, 2015

Contributor

I don't see any specific mention to the data types the links point to, have we decided that isn't IPLD turf (and just json-ld)?

It is possible to describe the merkle link in a JSON-LD context easily, but the context must also describe other parts of the JSON document. So it can't be a single context for all IPLD documents.

So, you'll have different contexts for small files, chunked files, directory, git blob, tree, commit, ..., and each of these contexts will be able to describe the merkle links the same way.

Also, we could imagine allowing any arbitrary key name instead of "mlink" and use the provided JSON-LD context to know which key is a merkle link, but this would mean parsing the JSON-LD context for every IPLD operation, which I believe would be way too slow.

Contributor

mildred commented Nov 13, 2015

I don't see any specific mention to the data types the links point to, have we decided that isn't IPLD turf (and just json-ld)?

It is possible to describe the merkle link in a JSON-LD context easily, but the context must also describe other parts of the JSON document. So it can't be a single context for all IPLD documents.

So, you'll have different contexts for small files, chunked files, directory, git blob, tree, commit, ..., and each of these contexts will be able to describe the merkle links the same way.

Also, we could imagine allowing any arbitrary key name instead of "mlink" and use the provided JSON-LD context to know which key is a merkle link, but this would mean parsing the JSON-LD context for every IPLD operation, which I believe would be way too slow.

@candeira

This comment has been minimized.

Show comment
Hide comment
@candeira

candeira Nov 16, 2015

From IRC:

< kandinski> jbenet: the IPLD spec doesn't define where the "IPLD" initials come from
< kandinski> Linked DAG?
@jbenet kandinski: good point. Linked Data. but it's not "legit linked data"
@jbenet linked dag could work :)
< kandinski> It's InterPlanetary Linked Data
@jbenet but will confuse people
< dignifiedquire> but InterPlanetaryLegitData sounds way cooler

In any case, origin of initials should be referenced at the top.

Since it's going to be pronounced "Eye Pee Ell Dee" anyway, we could say something like:

"The IPLD (short for InterPlanetary Linked DirectedAcyclicGraph) is...". We do call it a "thin-waist merkle dag" anyway, which is a killer description, by the way.

candeira commented Nov 16, 2015

From IRC:

< kandinski> jbenet: the IPLD spec doesn't define where the "IPLD" initials come from
< kandinski> Linked DAG?
@jbenet kandinski: good point. Linked Data. but it's not "legit linked data"
@jbenet linked dag could work :)
< kandinski> It's InterPlanetary Linked Data
@jbenet but will confuse people
< dignifiedquire> but InterPlanetaryLegitData sounds way cooler

In any case, origin of initials should be referenced at the top.

Since it's going to be pronounced "Eye Pee Ell Dee" anyway, we could say something like:

"The IPLD (short for InterPlanetary Linked DirectedAcyclicGraph) is...". We do call it a "thin-waist merkle dag" anyway, which is a killer description, by the way.

@candeira

This comment has been minimized.

Show comment
Hide comment
@candeira

candeira Nov 16, 2015

Same as above with the first mention of CDRTs. "Conflict-free replicated data type" is easier to understand for people coming in anew.

candeira commented Nov 16, 2015

Same as above with the first mention of CDRTs. "Conflict-free replicated data type" is easier to understand for people coming in anew.

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet
Member

jbenet commented Nov 21, 2015

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Nov 22, 2015

Member

@mildred

In that case, we should either provision key escaping to avoid clashing with someone wanting to have "@multicodec" as an arbitrary key in his JSON data structure

Yep, agreed. what's a good place to put that?

Member

jbenet commented Nov 22, 2015

@mildred

In that case, we should either provision key escaping to avoid clashing with someone wanting to have "@multicodec" as an arbitrary key in his JSON data structure

Yep, agreed. what's a good place to put that?

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Nov 22, 2015

Member

By the way, why choose the @ character to introduce directives? This might be confusing in the context of JSON-LD where the same character is used (but it will be escaped in the IPLD object). We are free to choose anything ($%!#&?*+...)

It was chosen because of JSON-LD. We could deviate,. we could use $, _, ! (it should be able to appear in a path / URL.

remaining question: should the text in paths be the raw case, or unescaped? eg, should this be possible or inaccessible?

> .../foo/bar/@multicodec
Member

jbenet commented Nov 22, 2015

By the way, why choose the @ character to introduce directives? This might be confusing in the context of JSON-LD where the same character is used (but it will be escaped in the IPLD object). We are free to choose anything ($%!#&?*+...)

It was chosen because of JSON-LD. We could deviate,. we could use $, _, ! (it should be able to appear in a path / URL.

remaining question: should the text in paths be the raw case, or unescaped? eg, should this be possible or inaccessible?

> .../foo/bar/@multicodec
@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Nov 22, 2015

Member

@diasdavid

I don't see any specific mention to the data types the links point to, have we decided that isn't IPLD turf (and just json-ld)?

I think there is definitely a strong case for @type to be defined here-- as something different from JSON-LD's. I don't know fully what it would do just yet, beyond knowing that i would want to have some behavior like JSON-LD/@context's, where identifiers can be aliased + shortened, and where defaults for values could be defined. Also definitely useful to have as an identifier for applications to act upon (eg, unixfs/file, unixfs/dir, commit, etc).

Member

jbenet commented Nov 22, 2015

@diasdavid

I don't see any specific mention to the data types the links point to, have we decided that isn't IPLD turf (and just json-ld)?

I think there is definitely a strong case for @type to be defined here-- as something different from JSON-LD's. I don't know fully what it would do just yet, beyond knowing that i would want to have some behavior like JSON-LD/@context's, where identifiers can be aliased + shortened, and where defaults for values could be defined. Also definitely useful to have as an identifier for applications to act upon (eg, unixfs/file, unixfs/dir, commit, etc).

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Nov 22, 2015

Member

@diasdavid

Also, instead of adding more TODO here, can we before go issue by issue here https://github.com/ipfs/go-ipld/issues and ipld/js-ipld-dag-cbor#2 and check what is already included in the spec, add what is missing, explicitly add what was accepted and then see what's left, bringing all of that conversation to this PR? So that we all talk over the same page :)

Agreed, let's do that.

Member

jbenet commented Nov 22, 2015

@diasdavid

Also, instead of adding more TODO here, can we before go issue by issue here https://github.com/ipfs/go-ipld/issues and ipld/js-ipld-dag-cbor#2 and check what is already included in the spec, add what is missing, explicitly add what was accepted and then see what's left, bringing all of that conversation to this PR? So that we all talk over the same page :)

Agreed, let's do that.

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Nov 22, 2015

Member

Accessing link properties problem (important issue to resolve)

Path resolution for data within links is currently not possible. specifically, i mean:

Suppose we have object which hashes to QmCCC...000:

# 
---
stuff:
  foo:
    mlink: QmCCC...111
    mode: 0755
    owner: jbenet

and we have object which hashes to QmCCC...111 (the foo link):

---
other:
  cat.jpg:
    mlink: QmCCC...222
    mode: 0644
    owner: jbenet

We can traverse so:

> ipld cat --yml QmCCC...000/stuff
foo:
  mlink: QmCCC...111
  mode: 0755
  owner: jbenet

> ipld cat --yml QmCCC...000/stuff/foo/other
---
cat.jpg:
  mlink: QmCCC...222
  mode: 0644
  owner: jbenet

But we cannot access the data in the links themselves, we cannot get the values of mode or owner properties directly.

Possible Solutions

keep as is, no direct access to link properties

Ok, but possibly a cop-out.

introduce another separator to traversal within single objects

suppose we use "." to separate levels within a single object, as in js dot notation. or some other character (|).

> ipld cat --yml QmCCC...000/stuff.foo.mode
0755
> ipld cat --yml QmCCC...000/stuff/foo
error: no link named "stuff" in object QmCCC...000

introduce another separator, to use ALSO within single objects

suppose we use "." to separate levels within a single object, as in js dot notation. or some other character (|). this separator would always be to disambiguate, and not replace the use of / within objects. It would also need to be escaped if found normally.

> ipld cat --yml QmCCC...000/stuff/foo.mode
0755
> ipld cat --yml QmCCC...000/stuff.foo.mode
0755
> ipld cat --yml QmCCC...000/stuff/foo/mode
error: no key "mode" in object QmCCC...000/stuff/foo/

Use a special key to access link properties specifically (not traverse all object parts)

suppose we use .link

> ipld cat --yml QmCCC...000/stuff/foo.link/mode
0755
Member

jbenet commented Nov 22, 2015

Accessing link properties problem (important issue to resolve)

Path resolution for data within links is currently not possible. specifically, i mean:

Suppose we have object which hashes to QmCCC...000:

# 
---
stuff:
  foo:
    mlink: QmCCC...111
    mode: 0755
    owner: jbenet

and we have object which hashes to QmCCC...111 (the foo link):

---
other:
  cat.jpg:
    mlink: QmCCC...222
    mode: 0644
    owner: jbenet

We can traverse so:

> ipld cat --yml QmCCC...000/stuff
foo:
  mlink: QmCCC...111
  mode: 0755
  owner: jbenet

> ipld cat --yml QmCCC...000/stuff/foo/other
---
cat.jpg:
  mlink: QmCCC...222
  mode: 0644
  owner: jbenet

But we cannot access the data in the links themselves, we cannot get the values of mode or owner properties directly.

Possible Solutions

keep as is, no direct access to link properties

Ok, but possibly a cop-out.

introduce another separator to traversal within single objects

suppose we use "." to separate levels within a single object, as in js dot notation. or some other character (|).

> ipld cat --yml QmCCC...000/stuff.foo.mode
0755
> ipld cat --yml QmCCC...000/stuff/foo
error: no link named "stuff" in object QmCCC...000

introduce another separator, to use ALSO within single objects

suppose we use "." to separate levels within a single object, as in js dot notation. or some other character (|). this separator would always be to disambiguate, and not replace the use of / within objects. It would also need to be escaped if found normally.

> ipld cat --yml QmCCC...000/stuff/foo.mode
0755
> ipld cat --yml QmCCC...000/stuff.foo.mode
0755
> ipld cat --yml QmCCC...000/stuff/foo/mode
error: no key "mode" in object QmCCC...000/stuff/foo/

Use a special key to access link properties specifically (not traverse all object parts)

suppose we use .link

> ipld cat --yml QmCCC...000/stuff/foo.link/mode
0755
@mildred

This comment has been minimized.

Show comment
Hide comment
@mildred

mildred Nov 25, 2015

Contributor

In that case, we should either provision key escaping to avoid clashing with someone wanting to have "@multicodec" as an arbitrary key in his JSON data structure

Yep, agreed. what's a good place to put that?

I would say, here in this spec. Something like:

Some keys are reserved in the maps of the datastructure for internal IPLD use. Every key starting with @ is considered an IPLD directive. Unknown directives should be ignored. Legitimate keys containing the @ character should be escaped using \@ instead of @ and \\ instead of \.

When converting from JSON to an IPLD document, the keys must be escaped appropriately. When decoding an IPLD document back to JSON, the keys must be decoded back. When manipulating an IPLD document, the IPLD directives must be evaluated. The system may also allow to represent an IPLD document with its directives, but it should be clear that the document must follow specific IPLD notation (special directives and escaping of keys)

IPLD directives:

  • @codec: at the root of the IPLD document, specifies a codec in which the IPLD document was encoded.
  • ...
Contributor

mildred commented Nov 25, 2015

In that case, we should either provision key escaping to avoid clashing with someone wanting to have "@multicodec" as an arbitrary key in his JSON data structure

Yep, agreed. what's a good place to put that?

I would say, here in this spec. Something like:

Some keys are reserved in the maps of the datastructure for internal IPLD use. Every key starting with @ is considered an IPLD directive. Unknown directives should be ignored. Legitimate keys containing the @ character should be escaped using \@ instead of @ and \\ instead of \.

When converting from JSON to an IPLD document, the keys must be escaped appropriately. When decoding an IPLD document back to JSON, the keys must be decoded back. When manipulating an IPLD document, the IPLD directives must be evaluated. The system may also allow to represent an IPLD document with its directives, but it should be clear that the document must follow specific IPLD notation (special directives and escaping of keys)

IPLD directives:

  • @codec: at the root of the IPLD document, specifies a codec in which the IPLD document was encoded.
  • ...
@mildred

This comment has been minimized.

Show comment
Hide comment
@mildred

mildred Nov 25, 2015

Contributor

remaining question: should the text in paths be the raw case, or unescaped? eg, should this be possible or inaccessible?

.../foo/bar/@multicodec

I say it shouldn't be possible. I see no use for that, and it will prevent paths like /ipfs/HASH/dir/@multicodec/filename. The whole point of escaping is to allow that.

Contributor

mildred commented Nov 25, 2015

remaining question: should the text in paths be the raw case, or unescaped? eg, should this be possible or inaccessible?

.../foo/bar/@multicodec

I say it shouldn't be possible. I see no use for that, and it will prevent paths like /ipfs/HASH/dir/@multicodec/filename. The whole point of escaping is to allow that.

@mildred

This comment has been minimized.

Show comment
Hide comment
@mildred

mildred Nov 25, 2015

Contributor

Accessing link properties problem (important issue to resolve)

I say, take the same approach as unix symbolic links:

  • Have the ability to specify when resolving the link if we want to follow the link or not. POSIX has this with stat() vs lstat() for instance.

    > ipld cat --no-resolve --yml QmCCC...000/stuff/foo/
    ---
    mlink: QmCCC...111
    mode: 0755
    owner: jbenet
    
    > ipld cat --resolve --yml QmCCC...000/stuff/foo/
    ------
    other:
      cat.jpg:
        mlink: QmCCC...222
        mode: 0644
        owner: jbenet
    
  • Or, change the behaviour depending on if the trailing / is present.

    > ipld cat --yml QmCCC...000/stuff/foo
    ---
    mlink: QmCCC...111
    mode: 0755
    owner: jbenet
    
    > ipld cat --yml QmCCC...000/stuff/foo/
    ------
    other:
      cat.jpg:
        mlink: QmCCC...222
        mode: 0644
        owner: jbenet
    

    You have the same on unix as well:

    > ln -s / foo
    > stat ./foo
      File: ‘foo’ -> ‘/’
      Size: 1           Blocks: 0          IO Block: 4096   symbolic link
    > stat ./foo/
      Size: 4096        Blocks: 8          IO Block: 4096   directory
    Device: fe02h/65026d    Inode: 2           Links: 25
    

    The problem with this solution is that it makes easier to access the link properties than the link content in itself. Generally we want the opposite (I would prefer to cat .../toto.txt instead of cat .../toto.txt/). It can be solved by:

  • Access link properties when the separator is // instead of /.

    > ipld cat --yml QmCCC...000/stuff/foo//
    ---
    mlink: QmCCC...111
    mode: 0755
    owner: jbenet
    
    > ipld cat --yml QmCCC...000/stuff/foo
    ------
    other:
      cat.jpg:
        mlink: QmCCC...222
        mode: 0644
        owner: jbenet
    

    This is close to your introduce another separator to traversal within single objects solution but does not introduces a new separator. This makes it incompatible with unix paths in a slight way because on unix foo/bar is strictly equivalent to foo//bar or foo///bar. This won't be the case here. This also means that fuse interfaces and unix applications might simplify // to / and this feature may not be available there.

As for the other solutions you proposed:

  • keep as is, no direct access to link properties: this is nice because it makes it compatible with everything we have elsewhere (unix paths, URLs, ...) for free

  • introduce another separator to traversal within single objects: Imagine the problem if I touch dir/foo.jpeg && rm -f dir/foo.jpeg.mode. What do I get ?

    We can also forbid ever using the . or | character (or the special selected character) in paths. We loose compatibility with unix paths.

  • introduce another separator, to use ALSO within single objects: same compatibility/escaping problem here.

  • Use a special key to access link properties specifically (not traverse all object parts): I prefer this one. It is conceptually closer to what we understand how a filesystem is. This has the same problem of clashing with legitimate file names: touch toto && rm -f toto.link/mode or rm -f test && mkdir test.link && cat test.

Contributor

mildred commented Nov 25, 2015

Accessing link properties problem (important issue to resolve)

I say, take the same approach as unix symbolic links:

  • Have the ability to specify when resolving the link if we want to follow the link or not. POSIX has this with stat() vs lstat() for instance.

    > ipld cat --no-resolve --yml QmCCC...000/stuff/foo/
    ---
    mlink: QmCCC...111
    mode: 0755
    owner: jbenet
    
    > ipld cat --resolve --yml QmCCC...000/stuff/foo/
    ------
    other:
      cat.jpg:
        mlink: QmCCC...222
        mode: 0644
        owner: jbenet
    
  • Or, change the behaviour depending on if the trailing / is present.

    > ipld cat --yml QmCCC...000/stuff/foo
    ---
    mlink: QmCCC...111
    mode: 0755
    owner: jbenet
    
    > ipld cat --yml QmCCC...000/stuff/foo/
    ------
    other:
      cat.jpg:
        mlink: QmCCC...222
        mode: 0644
        owner: jbenet
    

    You have the same on unix as well:

    > ln -s / foo
    > stat ./foo
      File: ‘foo’ -> ‘/’
      Size: 1           Blocks: 0          IO Block: 4096   symbolic link
    > stat ./foo/
      Size: 4096        Blocks: 8          IO Block: 4096   directory
    Device: fe02h/65026d    Inode: 2           Links: 25
    

    The problem with this solution is that it makes easier to access the link properties than the link content in itself. Generally we want the opposite (I would prefer to cat .../toto.txt instead of cat .../toto.txt/). It can be solved by:

  • Access link properties when the separator is // instead of /.

    > ipld cat --yml QmCCC...000/stuff/foo//
    ---
    mlink: QmCCC...111
    mode: 0755
    owner: jbenet
    
    > ipld cat --yml QmCCC...000/stuff/foo
    ------
    other:
      cat.jpg:
        mlink: QmCCC...222
        mode: 0644
        owner: jbenet
    

    This is close to your introduce another separator to traversal within single objects solution but does not introduces a new separator. This makes it incompatible with unix paths in a slight way because on unix foo/bar is strictly equivalent to foo//bar or foo///bar. This won't be the case here. This also means that fuse interfaces and unix applications might simplify // to / and this feature may not be available there.

As for the other solutions you proposed:

  • keep as is, no direct access to link properties: this is nice because it makes it compatible with everything we have elsewhere (unix paths, URLs, ...) for free

  • introduce another separator to traversal within single objects: Imagine the problem if I touch dir/foo.jpeg && rm -f dir/foo.jpeg.mode. What do I get ?

    We can also forbid ever using the . or | character (or the special selected character) in paths. We loose compatibility with unix paths.

  • introduce another separator, to use ALSO within single objects: same compatibility/escaping problem here.

  • Use a special key to access link properties specifically (not traverse all object parts): I prefer this one. It is conceptually closer to what we understand how a filesystem is. This has the same problem of clashing with legitimate file names: touch toto && rm -f toto.link/mode or rm -f test && mkdir test.link && cat test.

@diasdavid diasdavid added the libp2p label Nov 26, 2015

@jbenet jbenet referenced this pull request Nov 30, 2015

Closed

Sprint Nov 23 #56

24 of 42 tasks complete

mildred added some commits Feb 9, 2016

IPLD Protocol Buffer compatibility: fix errors
Fix the paragraph about the first byte that is able to determine if the
data in prefixed by a multicodec or is a protocol buffer object.
@mildred

This comment has been minimized.

Show comment
Hide comment
@mildred

mildred Feb 9, 2016

Contributor

Perhaps we can work in steps:

  • try to finish the issues where there is not much debate: #59 (legacy protocol buffer serialisation) and #64 (rename mlink to @link) and merge them here
  • try to finish up CBOR tagging quickly #61
  • try to find a subset of IPLD that can be merged in master now
  • leave the discussion about the path #62 in a separate issue to be resolved before we ship IPLD for real. Core IPLD can work without any path mechanism (I didn't implement one in go-ipld and go-ipfs/merkledag don't have a path mechanism) and implementation can benefit from having a solid spec for core features.
  • don't shy away from opening other issues if some are raised during IPLD implementation.

This way, we can perhaps avoid having a gigantic issue where everything in IPLD is discussed at once.

Contributor

mildred commented Feb 9, 2016

Perhaps we can work in steps:

  • try to finish the issues where there is not much debate: #59 (legacy protocol buffer serialisation) and #64 (rename mlink to @link) and merge them here
  • try to finish up CBOR tagging quickly #61
  • try to find a subset of IPLD that can be merged in master now
  • leave the discussion about the path #62 in a separate issue to be resolved before we ship IPLD for real. Core IPLD can work without any path mechanism (I didn't implement one in go-ipld and go-ipfs/merkledag don't have a path mechanism) and implementation can benefit from having a solid spec for core features.
  • don't shy away from opening other issues if some are raised during IPLD implementation.

This way, we can perhaps avoid having a gigantic issue where everything in IPLD is discussed at once.

@mildred

This comment has been minimized.

Show comment
Hide comment
@mildred

mildred Feb 9, 2016

Contributor

#59 #61 and #64 are LGTM. Waiting for other's opinions. #62 still needs more thoughts, but I don't really mind about the path format if we agree that it won't be used for unixfs if it is incompatible with most unix filenames.

Contributor

mildred commented Feb 9, 2016

#59 #61 and #64 are LGTM. Waiting for other's opinions. #62 still needs more thoughts, but I don't really mind about the path format if we agree that it won't be used for unixfs if it is incompatible with most unix filenames.

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Feb 11, 2016

Member

@mildred thanks for pushing on this. agreed to proceeding on the other fronts will review shortly.

Member

jbenet commented Feb 11, 2016

@mildred thanks for pushing on this. agreed to proceeding on the other fronts will review shortly.

Merge pull request #64 from mildred/ipld-spec-link-insteadof-mlink
Rename Merkle links key from mlink to link
@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Feb 11, 2016

Member

I see an issue with solutions (7) and (8). What if we want to address a data structure with a key named "link"? It is possible, but confusing.

well, the key would be taken, the benefit of @link is that we dont have to escape it (i think we've discussed this version before). i.e. if the user needs @link exactly for something else, user can escape it when constructing / importing to IPLD. (@link is very unlikely to clash with anything else, and importers libs could well escape for the user, but the core implementation doesnt have to)


@mildred

  • i think i like solution (8) the most, with / and // and /@link/ -- the reason i want (8) over (4) is that i do want ipld paths to be expressible in a unix path, web URI/URL, and having /@link/ make it possible to do. I would prefer // the rest of the time.
  • i like solution (4) with . and / second.
  • I like solution (1) as the last option
Member

jbenet commented Feb 11, 2016

I see an issue with solutions (7) and (8). What if we want to address a data structure with a key named "link"? It is possible, but confusing.

well, the key would be taken, the benefit of @link is that we dont have to escape it (i think we've discussed this version before). i.e. if the user needs @link exactly for something else, user can escape it when constructing / importing to IPLD. (@link is very unlikely to clash with anything else, and importers libs could well escape for the user, but the core implementation doesnt have to)


@mildred

  • i think i like solution (8) the most, with / and // and /@link/ -- the reason i want (8) over (4) is that i do want ipld paths to be expressible in a unix path, web URI/URL, and having /@link/ make it possible to do. I would prefer // the rest of the time.
  • i like solution (4) with . and / second.
  • I like solution (1) as the last option
@mildred

This comment has been minimized.

Show comment
Hide comment
@mildred

mildred Feb 11, 2016

Contributor

i think i like solution (8) the most, with / and // and /@link/ -- the reason i want (8) over (4) is that i do want ipld paths to be expressible in a unix path, web URI/URL, and having /@link/ make it possible to do. I would prefer // the rest of the time.

I like these options as well. If the filesystem layer contracts // to / it is always possible to use /@link/, and this can be nicely represented on a filesystem.

Contributor

mildred commented Feb 11, 2016

i think i like solution (8) the most, with / and // and /@link/ -- the reason i want (8) over (4) is that i do want ipld paths to be expressible in a unix path, web URI/URL, and having /@link/ make it possible to do. I would prefer // the rest of the time.

I like these options as well. If the filesystem layer contracts // to / it is always possible to use /@link/, and this can be nicely represented on a filesystem.

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Feb 12, 2016

Member

OK! #59 #61 #62 #64 are all merged! 👍 MASSIVE thanks to @mildred for pushing it through. We all are very thankful :)

What issues remain here? I think I will merge this (FINALLY!) and let's continue to iron it out with future PRs against master. I think we have a solid spec, and now https://github.com/ipfs/go-ipld/ and https://github.com/diasdavid/js-ipld/ can match it.

Member

jbenet commented Feb 12, 2016

OK! #59 #61 #62 #64 are all merged! 👍 MASSIVE thanks to @mildred for pushing it through. We all are very thankful :)

What issues remain here? I think I will merge this (FINALLY!) and let's continue to iron it out with future PRs against master. I think we have a solid spec, and now https://github.com/ipfs/go-ipld/ and https://github.com/diasdavid/js-ipld/ can match it.

jbenet added a commit that referenced this pull request Feb 12, 2016

@jbenet jbenet merged commit 5e5f3ba into master Feb 12, 2016

@jbenet

This comment has been minimized.

Show comment
Hide comment
@jbenet

jbenet Feb 12, 2016

Member

Note: let's keep the branch, as there's many links to that specifically.

Member

jbenet commented Feb 12, 2016

Note: let's keep the branch, as there's many links to that specifically.

@noffle

This comment has been minimized.

Show comment
Hide comment
@noffle

noffle Feb 12, 2016

Contributor

Great work, @mildred, @jbenet! 👏

Contributor

noffle commented Feb 12, 2016

Great work, @mildred, @jbenet! 👏

@parkan

This comment has been minimized.

Show comment
Hide comment
@parkan

parkan commented Feb 12, 2016

🎆

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment