Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Background on Address Scheme discussions (dweb, ipfs:// and more) #152

Closed
wants to merge 5 commits into from

Conversation

flyingzumwalt
Copy link

@flyingzumwalt flyingzumwalt commented Mar 3, 2017

This still needs work, but it gives a starting point for people to clarify the discussion. Please submit comments and/or changes.

ref ipfs/ipfs#227 ipfs/in-web-browsers#4 ipfs/in-web-browsers#28 #139

@flyingzumwalt
Copy link
Author

@flyingzumwalt flyingzumwalt commented Mar 3, 2017

@lgierth @jbenet @nicola @Gozala @lidel @jefft0 and anyone else, please help me make this document accurate and complete.

Copy link
Member

@jbenet jbenet left a comment

Thanks for pulling all this info together.

I think that what's also needed is:

  • the current spec, fleshed out.
  • a proposed different spec, fleshed out, that achieves the same things (re supporting ipfs, ipns, ipld).

that way, we can compare and see what is more complex, pros/cons, and so on.

it is ok to do "one thing for now" and "one thing for the future". This is often how upgrading happens. We don't have to argue about the second thing with anyone who is hostile to it. if the former thing satisfies them, that should be good enough for them.

### Why it's not good enough

#### Reason 1: We want IPFS, IPNS and IPLD to be handled by a single schema
Creating an `ipfs:` schema would not be enough because `ipfs:` only refers to mutable content. You would, at the very least, need an `ipns:` schema too.
Copy link
Member

@jbenet jbenet Mar 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ipld: would be needed too. as important as ipns:


also

> A minor reason is not having to force people to swallow N shemes (ipfs:// ipns:// ipld:// and counting), and instead use one that muxes.
Copy link
Member

@jbenet jbenet Mar 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now think this is a much bigger reason than that email suggested. Even discounting the "Unify the filesystem-database-web rift" rift, I would still press for this construction for this reason.

Copy link
Member

@jbenet jbenet Mar 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that it may be more than 3 in the future: /dns, /ens, /btc . NOT doing this now effectively shuts down future possibilities by making them "not worth doing the work of introducing a URL scheme".

Copy link
Member

@lidel lidel Mar 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel it should be explicitly stated in the document if this single scheme aims to be for IPFS ecosystem only, or something that can be adopted by other distributed systems.

If we want to share it with others, eg. Ethereum, Tor Onion Services etc, then I see potential problem with single scheme: who is responsible for muxing when multiple vendors are in play?
Let's say we have: dweb://ipfs/Qmbar.. and dweb://foo/buz. If IPFS browser add-on provides handler for dweb://, FOO is unable to handle its own URIs.
We should plan for this now, and have contingency written down..

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which I guess, means that dweb support has to be in its own package, so that tor wouldn't depend on ipfs...

There are a few goals tugging against each other:
1. The Noble Goal: Unify the filesystem-database-web rift
2. The Pragmatic Goal: Do the obvious, easy thing.
3. The Design Goal: Create Addresses that People will Love Using
Copy link
Member

@jbenet jbenet Mar 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting this document up this way -- calling one thing "the noble thing" and another "the pragmatic thing" -- strongly biases away from the former and towards the second. For many engineers, the framing "noble vs pragmatic" immediately discounts whatever is called noble. Therefore, I strongly disagree with this framing. This is not that hard, it's just confusing now because we haven't presented one doc with it.

@@ -0,0 +1,72 @@
(Unresolved) Address Scheme for IPFS
Copy link
Member

@jbenet jbenet Mar 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Address Scheme for IPFS" is not unresolved. We've had a clear path for a long time and going away from that is undoing decisions. This is the current address scheme, and is not an "unresolved thing waiting for a decision". There are suggestions to undo that decision and change to something else, for the sake of simplifying implementation of browser bridges, and satisfying people's URL orthodoxy.

Perhaps "(Unresolved) Changing the URL address scheme IPFS uses"

There are a few goals tugging against each other:
1. The Noble Goal: Unify the filesystem-database-web rift
2. The Pragmatic Goal: Do the obvious, easy thing.
3. The Design Goal: Create Addresses that People will Love Using
Copy link
Member

@jbenet jbenet Mar 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • "avoid polluting the scheme namespace with multiple schemes" is an important goal to add here, as its own thing.


# Backround for the Discussions about an Address Scheme for IPFS and `ipfs:` URIs

The discussions around `ipfs:` vs. `fs:` vs. `dweb:` is a confusing one that's been going on since @jbenet published the [IPFS whitepaper](https://ipfs.io/ipfs/QmR7GSQM93Cx5eAg6a6yRzNde1FQv7uL6X1o4k7zrJa3LX/ipfs.draft3.pdf).
Copy link
Member

@jbenet jbenet Mar 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

worth noting:

  • fs: is historical. can be gotten rid of (still early enough).
  • dweb: is a recent proposal.


## The Design Goal: Create Addresses that People will Love Using

From a design perspective, the challenge is to create a schema that makes intuitive sense, maximizes possibilities, and allows people to identify content with addresses that are reliable, powerful, and pleasant to use.
Copy link
Member

@jbenet jbenet Mar 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a great point, well phrased.

though this also runs counter to hashes in addresses :) -- hashes are not particularly pleasant to use.


@jbenet agreed to that pragmatism:
> **These goals are secondary in time to getting browser adoption. Meaning that we CAN do things like recommend ipfs:// ipns:// ipld://** IF browser vendors think that it's unlikely to get adoption this way now. We can work on unifying the fs-db-web rift later. **We're not dogmatic, we're pragmatic.** But we want to make sure we push in the right places and try to make as much as we can better.

Copy link
Member

@jbenet jbenet Mar 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also worth adding that some current browsers have trouble with the construction: origin = a:/b/c because they expect: `origin = a:/b, even though that's not in the URI spec.

Copy link
Member

@jbenet jbenet Mar 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is reason enough to provide a different mapping to the same underlying scheme.

meaning, make:

  • I'm ok with allowing ipfs://<hash> to work, for the sake of current expectations in firefox.
  • But this is not reason enough to abandon goals. we can just make those goals cause less friction by allowing both.

Copy link
Member

@jbenet jbenet Mar 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, worth noting:

  • browsers currently assume <origin> is a case insensitive thing.
  • so a construction like dweb://ipfs/<base58-hash> or even ipfs://<base58-hash> doesn't work without redirection or translation in the implementation.
  • today, given firefox constraints, what is actually fed into firefox as the proper URL is ipfs://<base16-or-32-hash> (this does not have to be the thing that shows up to the user or that we encourage people to use as links).

@timthelion
Copy link

@timthelion timthelion commented Mar 4, 2017

I felt kind of ignored, so I added a section on my concerns with /ipfs in a new PR here: https://github.com/ipfs/specs/pull/153/files

I know it is long, but I've put time into making real examples with real testable code and concrete examples.

@timthelion
Copy link

@timthelion timthelion commented Mar 4, 2017

@flyingzumwalt flyingzumwalt force-pushed the address-scheme branch 3 times, most recently from 78c416c to 7a77a02 Compare Mar 4, 2017
@flyingzumwalt
Copy link
Author

@flyingzumwalt flyingzumwalt commented Mar 4, 2017

I've added info from everyone's comments, except @timthelion -- provided suggestions on his PR. I also rearranged the document so that (hopefully) it will be easy for people to submit further corrections/additions, etc. Please keep your additions concise with lots of links out to the ongoing discussions in GH issues and other PRs.

Note: I'm going to be very busy traveling and presenting at a conference this week, so I will not have much bandwidth to guide this discussion or to keep it moving forward.

@timthelion
Copy link

@timthelion timthelion commented Mar 4, 2017

Just to let you konw, I will not be commenting further on this issue.

@flyingzumwalt
Copy link
Author

@flyingzumwalt flyingzumwalt commented Mar 4, 2017

Why @timthelion? You're making good points. I'm trying to gather the info and discussions so that your contributions move this discussion forward, rather than being yet another github issue that lies unresolved. Have I done something to discourage you?

@timthelion
Copy link

@timthelion timthelion commented Mar 4, 2017

@flyingzumwalt don't worry, you didn't do anything wrong or anything to discourage me. I put several hours into explaining my position, I explained it as well as I could and I don't beleive that I could do so better than I have already.

But the real reason why I do not want to continue, is that this discussion is making me feel unhappy, frustrated, and combative. For the sake of my own mental health, I would prefer not to continue with this discussion.

I just feel like this sometimes. Perhaps it is the coldness of text.

@flyingzumwalt
Copy link
Author

@flyingzumwalt flyingzumwalt commented Mar 5, 2017

I can certainly empathize with that @timthelion. 😓 If it's any consolation, the delay on getting a response to your final posix/unix questions was because the people who can give a good answer are busy with other stuff right now, but I'm going to make sure we get good answers on the record eventually. I wanted to make sure they answer your new questions rather than re-answering stuff that's already been discussed elsewhere or re-hashing debates that already happened. In order to do that, I needed to gather everything into a document that provides the context/history.

So please don't delete your fork with those comments and examples. If you're willing, it will help me cite your comments and avoid losing them over time if you put your comments into a separate file next to the readme.md -- ie. a file called "timthelion-comments.md" and submit a PR with that file in it. Alternatively, it's perfect material for a blog post. If you put it on your blog we can cite it there and save a copy on ipfs.

@flyingzumwalt
Copy link
Author

@flyingzumwalt flyingzumwalt commented Mar 5, 2017

And I completely agree about the coldness of text-based communication. It can be really rough on the human heart.

### Criticisms of this Approach

#### Criticism 1: We want IPFS, IPNS and IPLD to be handled by a single schema
Creating an `ipfs:` schema would not be enough because `ipfs:` only refers to mutable content. You would, at the very least, need an `ipns:` schema too.
Copy link

@jefft0 jefft0 Mar 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

`ipfs:` only refers to mutable content

This is a typo, right? It should be:

`ipfs:` only refers to immutable content

Copy link
Author

@flyingzumwalt flyingzumwalt Mar 6, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes that's a typo. you're right. I need to fix that.

@jefft0
Copy link

@jefft0 jefft0 commented Mar 5, 2017

@jbenet mentions the case insensitivity problem. I think the "Strengths" or "Criticisms" section of each approach should say whether the multihash is in the case insensitive part of the URI. (If an approach requires a browser to hack the URI to not conflate ipfs://QmAAA with ipfs://Qmaaa, then it's a pretty strong criticism of the approach.) Do you want me to add comments to this effect in the document?

@timthelion
Copy link

@timthelion timthelion commented Mar 5, 2017

@jefft0 "URLs should treat upper case letters as equivalent to lower case in scheme names" https://www.ietf.org/rfc/rfc1738.txt

So long as http://gateway.ipfs.io/ipfs/Qmaqx89FcFCQ3EocZZEbzGf7mJZpKBe8XAZt8ijoLXkojN works, then why would ipfs:/ be any different?


#### Strength: Fits with the way browsers identify origins

When enforcing the Single Origin Policy, some current browsers expect: `origin = a:/b`, even though that's not what the URI spec calls for. The `ipfs:` approach fits with that expectation, with URIs like `ipfs:/<root-hash>/foo/bar`, the origin would be `ipfs:/<root-hash>`
Copy link

@jefft0 jefft0 Mar 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused. The spec in the pull request looks like ipfs://ipfs/<root-hash>/foo/bar, not ipfs:/<root-hash>/foo/bar as you wrote here. So the origin would be ipfs://ipfs. Do you want to refer to a different draft of the ipfs: scheme?

Copy link
Author

@flyingzumwalt flyingzumwalt Mar 6, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting point.
A couple of things I will have to update the doc to reflect:

  1. @pawal's spec at #139 includes a brief reference to the fs: scheme which is fleshed out in @lgierth's https://github.com/ipfs/specs/tree/doc/fs-paths/fs-paths
  2. A couple of points are implicit in @jbenet's arguments, which hang off the fact that an ipfs: scheme should, due to its name, only return ipfs content:
    • if you create an ipfs: scheme, the /ipfs/ in the path is redundant.
    • if you create an ipfs: scheme you will end up needing more schemes: ipns:, ipld:, etc.

@jefft0
Copy link

@jefft0 jefft0 commented Mar 5, 2017

@timthelion: Yes, that's what the RFC says about the scheme part of scheme://host. The point is that browsers also treat the host part as case insensitive so that http://github.com and http://GitHub.COM are the same. If the multihash is in the "host" part of the URI, there could be problems.

@ghost
Copy link

@ghost ghost commented Mar 7, 2017

Thanks so much @flyingzumwalt, excellent write up ❤️ and thanks for covering for me. I failed at producing any useful writeup during the in-web-browsers sprint. I'll try my best to reframe this from a closer-to-implementation perspective that'll help us quickly wrap up the discussion.

I won't repeat the arguments for and against the various ideas proposed here and in older discussions, this PR already does a good job at that. Also @jbenet is right:

This is the current address scheme, and is not an "unresolved thing waiting for a decision". There are suggestions to undo that decision and change to something else, for the sake of simplifying implementation of browser bridges

We've had lengthy and fruitful discussions spanning years, and consensus about the long term goal of path addressing has been reached long ago (see ipfs/kubo#1678 (comment)). What is up for technical debate is what we usually call the upgrade path -- how to integrate with existing protocols and infrastructure in a way that:

  1. Makes it easy to simultaneously use the thing in its native environment, as well as the classic environments.
  2. Opens a long term perspective for transition.
  3. Yields an immediate benefit even within existing infrastructure.

Big protocol migrations happen over decades, and it's fundamentally important that new protocols interoperate well, otherwise the migration cost is simply too high. For example the IPv4/IPv6 migration came up with half a dozen IPv4-over-IPv6 and vice versa sub-protocols.

Note that for the upgrade path, we're not constrained to exactly what currently is. Many of the web's protocols (HTML, URLs, etc.) are so-called living standards, which aim to evolve relatively organically. We can propose new APIs and features, weigh in on discussions, and generally participate in the specification and development effort, as we did for example with the Suborigin working group.

Browser developers appear to not be generally opposed to the technicalities of dweb: URIs (and sometimes even inviting), if and only if we make an effort to make them play nice with the existing browser platform.

While browsers play a big role for ipfs (see ipfs/in-web-browsers) and are mentioned here a lot, we also want dweb: URIs to be supported in all kinds of tools/applications/libraries. One example is @Kubuxu's weechat plugin, which makes the URIs clickable.

The four stages of the upgrade path for path addressing.

Note that whenever we talk about these, it's important to be very exact in the notation of the address. The number of slashes matters a great deal.

  1. Current: HTTP-to-IPFS gateway
  • e.g. https://ipfs.io/ipfs/$hash
  • Has been working well for 2 years
  • Supported by many standard tools and applications out there because it's just HTTP
  • Issues:
    • Centralized in multiple ways: HTTP, Certificate Authorities, DNS
  1. Short term: URL
  • e.g. ipfs://$hash
  • Well understood, adhering to WHATWG URL standard
  • Straightforward to implement
  • $hash is host/origin
  • Issues:
    • Hash needs to be base32 encoded, because URL hosts are case-insensitive.
      • Solution 1/2: Needs a redirect to the base32 hash when pasting any non-base32 hashes.
      • Solution 2/2: Probably needs a special base32 CID that retains information about the original non-base32 CID, so that we can avoid confusing UX around changing hashes.
    • Doesn't retain the path, but instead needs conversion step to/from URI.
      • Solution: The URI path can be derived from URL scheme and host and path.
  1. Mid term: URI
  1. Long term: NURI
  • e.g. /ipfs/$hash
  • a.k.a. Nestable URI a.k.a. path addressing
  • Not just a noble goal, we do want to get there and lay the groundwork for it today.
  • You bring up good cases where path addressing / NURIs don't play well with current tooling and operating systems, @timthelion.
  • The FUSE integration of go-ipfs is only a very simple attempt at making NURIs work today.
  • Apart from FUSE, I can't think of any other significant effort we've made so far on this stage of the upgrade path.

These are not fixed serial steps, instead there's more of a wibbly-wobbly concurrence/simultaneity of all four stages. Networks are a wild mess of parts developing into differing directions, at varying speed (just like a rhizome). We definitely want to make the first three work in the foreseeable future, and also the fourth, although that one admittedly looks like a long shot at the moment :)

Regarding NURIs, I'd like to emphasize a comment @btrask wrote,
which I think is helpful to understand the ubiquity of NURIs and how they help fix the FS/databases/web rift.

  • canonical NURI is the path.
    • where NURI means Nestable URI. more universal than URIs. if you do not understand read the issue.
    • /ipfs/Qm.../foo/bar, /ipns/Qm.../foo/bar, and /dns/foo.com/bar.
    • use this everywhere you can!

An IPFS path works as a relative URL. That means it is a resource name without a canonical location. Browsers can interpret it without conversion as relative to the current server, which presumably supports the IPFS interface.

As a reference, the filesystem ideas of Plan 9 have been the inspiration for NURIs.

@jefft0
Copy link

@jefft0 jefft0 commented Mar 7, 2017

Is there a CID spec that shows how to unambiguously distinguish a base32 CID from a base58 CID? (Can't find the CID spec anywhere.)

@jefft0
Copy link

@jefft0 jefft0 commented Mar 7, 2017

distinguish a base32 CID from a base58 CID

Never mind. The multibase code handles it. With Google I found https://github.com/ipld/cid . (It wasn't linked to from https://github.com/ipfs/specs or https://github.com/ipld/specs/blob/master/README.md or https://github.com/ipld/specs/blob/master/ipld/README.md . Maybe it should be?)

@lidel
Copy link
Member

@lidel lidel commented Mar 7, 2017

I made a related comment in ipfs/in-web-browsers#6 (comment)
There should be tooling out-of-the-box for this, otherwise people will reinvent the wheel over and over. 🙃

@mib-kd743naq
Copy link

@mib-kd743naq mib-kd743naq commented Mar 7, 2017

Is there a CID spec that shows how to unambiguously distinguish a base32 CID from a base58 CID?

@jefft0 a couple months ago I walked someone through that in detail: https://botbot.me/freenode/ipfs/2017-01-04/?msg=78819903&page=9

I never found the time afterwards to put this in a document however :/

@lamarpavel
Copy link

@lamarpavel lamarpavel commented Nov 15, 2017

@dag If the browser interprets href="/ipfs/{hash}" as an absolute path to the local file system, then mounting ipfs to /ipfs should solve the problem without a need to add any support to browsers.

@dag
Copy link

@dag dag commented Nov 15, 2017

No, it would interpret it as an absolute path on the current host. Try this link for example. That was a link to /ipns/ipfs.io, but your browser will try to connect you with https://github.com/ipns/ipfs.io.

@RangerMauve
Copy link

@RangerMauve RangerMauve commented Nov 15, 2017

Yeah, the spec for href specifically expects it to be a URI, so if you want to use this new format you'll likely need a new attribute added to the HTML spec.

@ghost
Copy link

@ghost ghost commented Nov 15, 2017

It'll work just fine with href="ipfs://<hash>" and href="dweb:/ipfs/<hash>" -- I don't feel like this discussion is going anywhere at the moment, these are things that have been agreed on years ago.

I'll ping this thread once the DWeb Addressing spec is finally there (give me a week or two more). An outline and introduction have been merged in #167. We're looking for open questions and spots that need clarification.

@lidel
Copy link
Member

@lidel lidel commented Oct 17, 2019

Note to self: use this PR as a place to incorporate existing resources listed at https://github.com/ipfs/in-web-browsers/issues/147 (including ADDRESSING.md / https://docs.ipfs.io/guides/guides/addressing/)

@hsanjuan
Copy link
Contributor

@hsanjuan hsanjuan commented Mar 27, 2020

@lidel alive check on this...

@lidel
Copy link
Member

@lidel lidel commented Mar 31, 2020

Ack, specs consolidation was on the backburner due to other priorities but I hope to tackle it in 2020Q2 at some point (after subdomain gateways land everywhere).

I think we should close this outdated PR just to decrease noise and point people at https://github.com/ipfs/in-web-browsers/issues/147 which has a proper timeline of all decisions, making it a much better entry point than this PR.
Possibly should move mentioned issue to ipfs/specs too.

@hsanjuan can you close this and move mentioned issue to ipfs/specs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet