Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migration to CIDv1 (default base32) #337

Open
kyledrake opened this Issue Jul 12, 2018 · 13 comments

Comments

Projects
None yet
5 participants
@kyledrake
Copy link

kyledrake commented Jul 12, 2018

Lowercase identifiers are required in order to work with legacy URI, web browser security (sub)origins and the experimental Protocol Handler API. So we are moving to finish support for CIDv1 across the ecosystem, and use it as default (while still supporting CIDv0/Multihash).

There are some complications related to this switch, such as refactoring the blockstore to use multihashes instead of cid identifiers. A number of temporary solutions were proposed that would have made the change simpler with the current design, however it was decided that since this is going to be a significant update, we should focus on a full CIDv1 migration with one identifier change.

This is the parent issue for all issues related to migration to CIDv1 with default base32 (type 'b'). This issue closes when CIDv1b32 is enabled by default for new versions of go-ipfs, js-ipfs, and ipfs-companion.

@kevina @magik6k @lidel @whyrusleeping @diasdavid @alanshaw @olizilla @kyledrake @lgierth @mikeal

Berlin Jul 2018 developer meeting notes:
ipfs/developer-meetings#39

Initial proposal:
ipfs/go-ipfs#4143

🐦 Bird's-eye view (WIP by @lidel)

  • 🍎 = Not started
  • 🍊 = In progress
  • 🍏 = Complete
Scope PR/Issue Status Owner P
RFC
Handling of Alternative Multibases ipfs/go-ipfs#5349 🍏 @kevina
How to handle CIDv0 as we migrate to CIDV1 ipfs/go-ipfs#5291 🍎 ?
Do not return string CIDs from core ipfs/interface-js-ipfs-core#394 🍎 @alanshaw
Handling of CidV1/V0 at the bitswap level ipfs/go-ipfs#5378 🍎 ?
Updates to the blockstore interface ipfs/go-ipfs-blockstore#8 🍎 ?
Config option for specifying full default cidv1 parameters (AKA Reproducible CIDs) ipfs/go-ipfs#5230 🍎 ?
GO
META for go-ipfs ipfs/go-ipfs#5358 🍊 ?
Allow add to IPFS and get base32 CIDv1 back via an option to specify base encoding (defaults do not change)
ipfs/go-ipfs#5385 🍏 @kevina P0
ipfs/go-ipfs#5789 🍏 @kevina P0
Retrieve blocks even if given using a different CID version (compatibility)
ipfs/go-ipfs#5285 🍏 @kevina P0
Change defaults for adding content to CIDv1, with base32 string output (the end goal)
Origin isolation for HTTP websites
ipfs/go-ipfs#5982 🍎 ? P1
ipfs/go-ipfs#6096 🍊 @Stebalien
Refactor blockstore + DHT to work with bare multihashes instead of CIDs
META ipfs/go-ipfs#5231 🍎 ?
ipfs/go-ipfs#5378 🍎 ?
ipfs/go-ipfs#5510 🍊 ?
ipfs/go-ipfs-blockstore#13 🍊 ?
ipfs/go-blockservice#8 🍊 ?
ipfs/go-merkledag#17 🍊 ?
ipfs/go-ipfs-ds-help#4 🍊 ?
libp2p/go-libp2p-routing#33 🍊 ?
libp2p/go-libp2p-kad-dht#203 🍊 ?
libp2p/go-libp2p-pubsub-router#14 🍊 ?
libp2p/go-libp2p-routing-helpers#13 🍊 ?
ipfs/go-bitswap#18 🍊 ?
ipfs/go-ipfs-routing#15 🍊 ?
JS
META for js-ipfs ipfs/js-ipfs#1440 🍊 @alanshaw P0
Allow add to IPFS and get base32 CIDv1 back via an option to specify base encoding (defaults do not change)
ipfs/js-ipfs-mfs#4 🍏 @achingbrain P0
ipfs/js-ipfs#1560 🍏 @alanshaw P0
ipfs/js-ipfs#1552 🍏 @alanshaw P0
Retrieve blocks even if given using a different CID version (compatibility)
ipfs/js-ipfs-repo#184 🍏 @alanshaw P0
Origin isolation for HTTP websites
ipfs/js-ipfs#1877 🍎 ? P1
Change defaults for adding content to CIDv1, with base32 string output (the end goal)
multiformats/js-cid#73 🍊 @alanshaw
ipld/js-ipld#193 🍊 @alanshaw
ipld/js-ipld-bitcoin#43 🍊 @alanshaw
ipld/js-ipld-dag-pb#115 🍊 @alanshaw
ipld/js-ipld-dag-cbor#96 🍊 @alanshaw
ipld/js-ipld-ethereum#47 🍊 @alanshaw
ipld/js-ipld-git#42 🍊 @alanshaw
ipld/js-ipld-raw#28 🍊 @alanshaw
ipld/js-ipld-zcash#35 🍊 @alanshaw
ipfs/js-ipfs-unixfs-importer#21 🍊 @alanshaw
ipfs/js-ipfs-unixfs-engine#235 🍊 @alanshaw
ipfs/js-ipfs-mfs#38 🍊 @alanshaw
ipfs/js-ipfs#1868 🍊 @alanshaw
multiformats/js-cid-tool#7 🍊 @alanshaw
ipfs/js-ipfs-bitswap#188 🍊 @alanshaw
ipfs/js-ipfs-bitswap#188 🍊 @alanshaw
ipld/js-ipld-block#44 🍊 @alanshaw
ipfs/js-ipfs-http-response#17 🍊 @alanshaw
ipfs/js-ipfs-repo#192 🍊 @alanshaw
ipfs/js-ipfs-http-client#949 🍊 @alanshaw
ipfs/is-ipfs#29 🍊 @alanshaw
libp2p/js-libp2p-kad-dht#84 🍊 @alanshaw
INTEROP
META ipfs/interop#58 🍎 @lidel
ipfs/interface-js-ipfs-core#334 🍏 @achingbrain
ipfs/interface-js-ipfs-core#396 🍏 @alanshaw
ipfs/interface-js-ipfs-core#413 🍏 @alanshaw
ipfs/interface-js-ipfs-core#444 🍊 @alanshaw
Tests specific to special handling on go-ipfs side / datastore/dht 🍎 ?
ipfs-companion
META ipfs-shipyard/ipfs-companion#527 🍊 @lidel
Detect CID in subdomains ipfs-shipyard/ipfs-companion#537 🍏 @lidel
ipfs-cluster

🙌 Official Logo

base32

@Kubuxu

This comment has been minimized.

Copy link
Member

Kubuxu commented Jul 15, 2018

cc myself

@daviddias

This comment has been minimized.

Copy link
Member

daviddias commented Aug 12, 2018

@kyledrake can you give an update on the status of this migration in the next IPFS All Hands?

@kyledrake

This comment has been minimized.

Copy link
Author

kyledrake commented Aug 12, 2018

I will be unavailable for the call unfortunately, I'm moving tomorrow. @lidel do you think you could give a quick update? It doesn't need to be complicated, just a quick summary of progress based on the referenced issues on 337.

@kevina

This comment has been minimized.

Copy link

kevina commented Aug 12, 2018

@kyledrake @lidel @diasdavid I am the most qualified to give this update for go-ipfs. I will put something in the notes.

@lidel

This comment has been minimized.

Copy link
Contributor

lidel commented Nov 2, 2018

js-ipfs v0.33.0 and go-ipfs v0.4.18 shipped with some relevant changes:

This release makes some significant progress towards solving this issue by introducing two features:

(1) [..] ipfs cid base32 command for converting CID to a case intensive encoding required by domain names. This command converts a CID to version 1 and encodes it using base32.

(2) A hack to allow locally looking up blocks associated with a CIDv0 CID using the equivalent CIDv1 CID (or the reverse). This hack will eventually be replaced with a multihash indexed blockstore, which is agnostic to both the CID version and multicodec content type.

Release Notes for go-ipfs v0.4.18

This PR adds a jsipfs cid command with tools for formatting, converting and discovering properties of CIDs. All the docs are here: https://github.com/ipfs-shipyard/js-cid-tool
ipfs/js-ipfs#1560

The newly added ipfs cid subcommand (1) makes conversion super easy:

# cidv0(b58) → cidv1(b32)
$ ipfs cid base32 QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR
bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

# cidv1(b58) → cidv1(b32)
$ ipfs cid base32 zb2rhk6GMPQF3hfzwXTaNYFLKomMeC6UXdUt6jZKPpeVirLtV
bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy
@daviddias

This comment has been minimized.

Copy link
Member

daviddias commented Dec 3, 2018

@kyledrake can you compile a status update on this whole endeavour? How far are we from shipping it? Will there be a big launch?

@kyledrake

This comment has been minimized.

Copy link
Author

kyledrake commented Dec 6, 2018

@daviddias The referenced issues here show the current state of things pretty well. We're making good progress on better base32 support and are definitely at a point where we want to decide if we're comfortable with planning for a switch.

I would like an opinion from @alanshaw and @kevina on whether they think they're at a point where we would be comfortable migrating to base32, and if there's any blockers or issues on the technical side of things that would make them uncomfortable with switching until resolving. Then we can move to setting dates for transition, getting out announcements, etc.

@daviddias

This comment has been minimized.

Copy link
Member

daviddias commented Jan 13, 2019

Is there a timeline for when to expect this work to be finalized? What's the plan for the launch? How will the users be informed? Any caveats to the migration?

@kyledrake

This comment has been minimized.

Copy link
Author

kyledrake commented Jan 16, 2019

@daviddias We're getting closer to the finish line, so now is a good time to start discussing a rough timeline/plan.

1) Release go-ipfs, js-ipfs, ipfs-companion versions with optional cidv1-base32 support

js-ipfs is going to get there with the 0.34 release (ipfs/js-ipfs#1721), and @alanshaw has set 0.36 as the version with default cidv1-base32 as a rough plan (ipfs/js-ipfs#1721 (comment)), which I'm OK with. Absolutely incredible work by the js-ipfs team here!

go-ipfs has a pull request pending (ipfs/go-ipfs#5789) that will bring go-ipfs up to speed. I'm really excited to see this merged and finally put my hshca hack to rest. @kevina has done a ton of work on this (and had to re-do a lot of that work for various reasons) so kudos to him and everyone else at go-ipfs that has put a lot of good work into this.

I'm not sure what the timeline looks like for getting this in go-ipfs yet, @kevina @Stebalien is this something you foresee getting merged into master soon?

I'll let @lidel give us an update on ipfs-companion, it will be better than mine. Tracker issue for ipfs-companion is ipfs-shipyard/ipfs-companion#527

Dates: No hard dates for these releases, but I would like to see them released within the next month if it's not too inconvenient. Most of the work is done here.

Step 3 happens after we get releases of js-ipfs, go-ipfs, ipfs-companion which have the (optional, with arguments) support. Step 2 can be done in parallel:

2) Prepare the gateway for subdomains (*.dweb.link)

This will also be a good point to encourage users to start using the subdomains instead of the ipfs.io gateway, *.ipfs.dweb.link and *.ipns.dweb.link (cc @eefahy @mburns), getting content/apps on their own security origin. But we'll need to make sure whatever we choose works well with ipfs-companion (we good here @lidel?)

The main missing pieces right now:

  • The *.ipfs.dweb.link and *.ipns.dweb.link need SSL certificates (https://test.ipfs.dweb.link returns a bad cert)
  • A default redirect to HTTPS from HTTP
  • An HSTS preload header so we can get them added to the browser lists.

Dates: would be nice to see this done within the next month as well.

3) Blog post with new cidv1-base32 information

We will write a blog post indicating that we will be switching to cidv1-base32, why we are making this change, and how users can start using it now with optional arguments. This will get people more comfortable working with it, show them the new gateway URLs they can (really should!) use, and give them time to prepare.

We absolutely need to release both go-ipfs and js-ipfs (also ipfs-companion?) at the same time with the new default support, so a set date will eventually be needed.

Dates: I will publish this very soon after the optional support releases are made. 2-4 days to complete.

4) Update documentation to reflect use of cidv1-base32

When we release with the new defaults, we are going to have a lot of documentation across the organization that references the old output. We will need to prepare the documentation to reflect these changes when we do the release, particularly the docs portal.

Dates: Work will begin soon after optional support releases are made. Rough, hardly educated guess is 2 weeks of work.

5) Release go-ipfs, js-ipfs, ipfs-companion, documentation versions with cidv1-base32 support

We release everything at the same time, and add to a blog post announcing switch to cidv1-base32 and re-hashing a lot of what was stated in the first blog post.

Dates:
I'm not sure about a hard date for final release, it really depends on how comfortable we are with the current progress, I don't want to rush anybody. To throw out a few random dates: Spring Equinox is on March 20th and Summer Solstice is June 21st. I'm leaning towards a date closer to Solstice, as it would give us more room to fix any lingering issues and get some user testing in. But if we're feeling comfortable with progress, we can change to an earlier date. I think we'll have a better feel for things once Step 1 is finished.


Let's call this "proposed rough plan". Feel free to add any suggestions/missing info.

@kyledrake

This comment has been minimized.

Copy link
Author

kyledrake commented Jan 17, 2019

Adding a missing piece from my writeup:

go-ipfs 0.4.18 shipped with some ipfs cid changes for cidv1-base32. It gives us some clever ways to use it with the currently released version using pipes / additional API calls. For example:

$ echo "hello world" | ipfs add -q | ipfs cid base32
bafybeicg2rebjoofv4kbyovkw7af3rpiitvnl6i7ckcywaq6xjcxnc2mby
$ ipfs cat bafybeicg2rebjoofv4kbyovkw7af3rpiitvnl6i7ckcywaq6xjcxnc2mby
hello world
$ curl http://bafybeicg2rebjoofv4kbyovkw7af3rpiitvnl6i7ckcywaq6xjcxnc2mby.ipfs.dweb.link
hello world

So if used this way, you could technically start using cidv1-base32 with go-ipfs already (for a lot of common uses).

More info from the blog post: https://blog.ipfs.io/53-go-ipfs-0-4-18/#cidv1-base32-migration

@lidel

This comment has been minimized.

Copy link
Contributor

lidel commented Jan 23, 2019

ipfs-companion detects <cidv1b32>.ipfs.foo.tld and displays page actions:

2019-01-23--12-30-17

When js-ipfs/go-ipfs backends switch their defaults to cidv1b32, ipfs-companion will support it automatically.

Ad. preparing gateway for subdomains, there is an open question about how ipfs-companion should handle redirects of CID-in-Subdomain and DNSLink websites: ipfs-shipyard/ipfs-companion#667 ← would appreciate feedback on proposed solution.

@kyledrake

This comment has been minimized.

Copy link
Author

kyledrake commented Feb 18, 2019

Heads up that @lidel is now the coordinator for this work.

@lidel

This comment has been minimized.

Copy link
Contributor

lidel commented Mar 3, 2019

🏆 dweb.link is available over HTTPS!

Demo: https://bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy.ipfs.dweb.link

(thanks @scout!)

ℹ️ Plan to support CID-in-subdomain via HTTP Proxy

http://<cidv1b32>.ipfs.localhost

To provide Origin-based isolation for websites loaded from local gateway we are planning to use a special domain names in .localhost space. In the long term, we could move to a dedicated Special-Use Domain Name .ipfs.dweb.

We don't want to mess with people's DNS servers, so the cross-platform support for this relies on the idea of on-the-fly domain support via go-ipfs as HTTP proxy (proposed in ipfs/go-ipfs#5982, ipfs/js-ipfs#1877) + ipfs-companion setting go-ipfs a proxy for specific requests.

If proxied hostname is customizable, the same HTTP Proxy mode could be used on public gateways such as dweb.link: ipfs/infra#81 (comment)

ℹ️ WIP in js-ipfs: change defaults for adding content to cidv1b32

List of PRs in ipfs/js-ipfs#1440 (comment)

ℹ️ WIP in go-ipfs: switching DHT (& datastores) to raw multihashes

Some relevant go-ipfs updates were discussed at Golang Core Dev Team Weekly Sync 2019-02-11.
See Base32/CIDv1 section or TL;DR:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.