Migration towards base32 CIDv1

[kevina]

This issue is to form a migration plan to base32 CIDv1 specific to go-ipfs. It is separate from #4143 which was more of a discussion or https://github.com/ipfs/ipfs/issues/337 which is more of a meta issue for all of go-ipfs, js-ipfs, and ipfs-companion.

Phase 1

• Add ability to retrieve blocks even if given using a different CID version Add ability to retrieve blocks even if given using a different CID version #5285
• Add flag to ipfs add to return CIDs in alternative base Create ipfs.add flag for specifying base of produced CID #5233 p.r Add global option to specify the multibase encoding. #5464
• Provide utility to convert between multibase encoding and CID versions Tool for converting hashes #5229 p.r. Provide new "cid" sub-command. #5385

Phase 2 -- full migration.

• Refactor blockstore to work with bare multihashes instead of CIDs Refactor blockstore to work with bare multihashes instead of CIDs #5231
• Possible Add ability to retrieve using a different CID version from legacy nodes RFC: Handling of CidV1/V0 at the bitswap level #5378
• Figure out how to handle CID outputs in non-default multibase encoding RFC: Handling of Alternative Multibases #5349 (also ipfs resolve should respect CID base #5234 and WIP: Add global option to specify the multibase encoding. #5289). This includes if the multibase should be preserved in the CID and the addition of a global flag to specify the CID base.
• Create config option for specifying full default cidv1 parameters Create config option for specifying full default cidv1 parameters #5230
• Figure out how to handle existing CidV0 RFC: How to handle CIDv0 as we migrate to CIDV1 #5291

cc @whyrusleeping @Stebalien @lidel @magik6k @kyledrake.

[kyledrake]

This is great, thank you for setting this up. LMK if you need anything from me.

[kevina]

Issues that I currently need feedback on from the core developers:

• Handling of Alternative Multibases RFC: Handling of Alternative Multibases #5349
• How to handle CIDv0 as we migrate to CIDV1 RFC: How to handle CIDv0 as we migrate to CIDV1 #5291
• Create config option for specifying full default cidv1 parameters Create config option for specifying full default cidv1 parameters #5230

[hsanjuan]

(subscribing)

[daviddias]

(subscribing)

[alanshaw]

Kind request 🙏 - there's a button on the right you can press to subscribe - commenting "(subscribing)" sends me and everyone else mentioned or participating a notification.

[lidel]

Some relevant updates from Golang Core Dev Team Weekly Sync 2019-02-11:

Base32/CIDv1

[..]

@eingenito - Need to consider next steps and who/when this can be picked up.
@Stebalien indicates that the next work to be done is converting the blockstore to use raw multihashes (Issue, WIP). This is a complicated task that needs some context for whoever picks it up.

Q: What's the relative priority of this task compared to others on this list?
A: Most important next milestone is storing/fetching [raw] multihashes in the DHT in libp2p.

[DonaldTsang]

Idea: Use https://github.com/kstenerud/safe-encoding as a better alternative for safe encoding

[Stebalien]

On technical merits alone, those encoding schemes look superior (and the order-preserving property looks quite nice). Unfortunately,

1. We're far enough into this switch that changing the default will require a pretty strong motivation.
2. The motivation for Base32 is wide (often built-in) language support.

[lidel]

Discussion that lead us to picking Base32 as the new default can be found in #4143 (comment):

[..] chose RFC because it's a standard, it's been around a while, nginx-misc-module supports it, and it probably has the widest support across all programming languages.

The only reason I would chose the RFC one is because it is a standard and more likely to have an implementation available as part of the language.

@DonaldTsang encodings you linked look really interesting.
I believe picking RFC one is still a good choice when it comes to being the new default, but we surely can add order-preserving ones as alternative encodings. Thank you for opening multiformats/multibase#51, let's discuss them there.

[danimesq]

Important:

what is the workaround to get the base32 hash of an file?

via: #7852

[lidel]

@DaniellMesquita you can convert any CID to CIDv1 in Base32 via:

ipfs cid base32 <cidv0>

You can also make a request to a subdomain gateway, eg. https://dweb.link/ipfs/<cid>, and you will be redirected to a subdomain with DNS-safe base32 or base36 CID:

https://dweb.link/ipfs/QmbWqxBEKC3P8tqsKc98xmWNzrzDtRLMiMPL8wBuTGsMnR → https://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi.ipfs.dweb.link/

[danimesq]

@lidel, ok, but how to get the base32/CIDv1 of an local file like in ipfs add -q --only-hash? Using pipelines with both commands?

[danimesq]

Ok, the pipelines command to get the CIDv1 of an file is: ipfs add -q --only-hash <file.extension> | ipfs cid base32

[hsanjuan]

ipfs add --cid=version=1 should default to base32 output.

[danimesq]

@hsanjuan, this command isn't compatible with the purpose of only getting the CID of an given file, instead of adding it to IPFS.

lidel told me before that the flag to get CIDv1/base32 is --cid-version 1 probably instead of the --cid=version=1 you're mentioning.

Then, maybe ipfs add -q --cid-version 1 --only-hash is the command to just get the CIDv1 of an file.

@lidel, So, its ipfs add -q --cid-version 1 --only-hash?

Is it bad to produce an different merkletree/DAG? Then, how to keep compatibility?

via: #7852 (comment)