IPLD Prime In IPFS: Target Merge Branch

[hannahhoward]

This feature branch updates a number of dependencies to directly make use of go-ipld-prime under the hood. The major exception is the unixfs subsystem, which remains the same via the go-ipld-legacy shim.

Before this branch is feature complete, a known / remaining sub-conversation that is ongoing is "How should we directly expose the newer ipld-prime node interface to programs making use of IPFS?". This conversation is occurring in #8046 which will merge into this branch when ready.

• Node API #8046
• go-path integrated #8028
• IPLD Prime In IPFS: Target Merge Branch go-path#36
  • Address Remaining PR comments
  • @warpfork approval
  • @aschmahmann approval
• switch dag put cmd to directly use prime #7995
  • fix dag test fixtures for go-ipld-prime codecs #8257
    • Sort map entries marshalling dag-cbor ipld/go-ipld-prime#204 ?
    • Make go vet happy ipld/go-ipld-prime#205 ?
    • fix any remaining sharness tests
• Update to IPLD Prime go-ipfs-provider#32
  • Rebase on master
  • @warpfork review
  • @warpfork approval
  • @aschmahmann review
  • @aschmahmann approval
• IPLD In IPFS: Target Merge Branch interface-go-ipfs-core#67
  • Merge Add "Node" api for prime interactions interface-go-ipfs-core#68 -- since its already referenced by this branch
  • Update to tagged go-dagwriter
  • Rebase on master
  • Walk this one back a bit. Dep bumps only for this round -- no new API support promises.
  • @warpfork review
  • @warpfork approval
  • @aschmahmann approval? (already discussed changes)
• Use IPLD-prime: target merge branch go-merkledag#67
  • Rebase on master
  • @warpfork review
  • @warpfork approval
  • @aschmahmann re-review? (not clear if a full review conducted but there are comments)
  • @aschmahmann approval
• make tagged versions of sub-repos and propagate
  Do this all at once, on a mergeparty day.
  For each repo: look at the go mod file; update any dep not pointing to a clean tag; wait for CI green; create tag in that repo; move to next repo.
  Order of merge/tag should be:
  • https://github.com/ipld/go-ipld-prime (@master) tagged v0.11.0
  • https://github.com/multiformats/go-multicodec (@master) tagged v0.3.0
  • https://github.com/ipfs/go-ipld-legacy (@master/deps updated) tagged v0.1.0
  • https://github.com/ipld/go-codec-dagpb (@master/deps updated) tagged v1.3.0
  • https://github.com/ipfs/go-merkledag (#67 merged) tagged v0.4.0
  • https://github.com/ipfs/go-fetcher (@master/deps updated) tagged v1.4.0
  • https://github.com/ipfs/go-unixfsnode (@master/deps updated) tagged v1.1.2 (may have no changes, just explicit dep prop for clarity)
  • https://github.com/ipfs/go-path (#36 merged) tagged v0.1.0
  • https://github.com/ipfs/interface-go-ipfs-core (#67 merged) tagged v0.5.0
  • https://github.com/ipfs/go-ipfs-provider (#32 merged) tagged v0.6.1
  • https://github.com/ipfs/go-ipld-git (#46 merged) tagged v0.1.0
  • update the codec names for the dag put command: ipld-in-ipfs series of PRs go-ipfs-api#248 (no tag needed, master is pulled by CI)
  • ... now back to self, this PR!
• rebase on master
• @aschmahmann review of this branch for merge
• @aschmahmann approval of this branch
• merge

[BigLep]

2021-08-17 update on outstanding items:

@aschmahmann: create issues for the following to fix during the release phase:

1. Missing test for hang when doing dag get: 9339e82
2. Bugs related to codecs that return EOF on encode/decode (EOF error are ignored by CLI so we would need to either fix that or wrap the errors)
   • echo "test" | ipfs dag put --input-enc=git-raw doesn't error or return a CID
   • echo "" | ipfs dag put --input-enc=dag-json --format=dag-json
3. Bug where can only get raw bytes using the git codec but not round trip it. It should not do the first or also do the second.

❯ echo '{"foo" : 5}' | ipfs dag put --input-enc=raw
bafyreigg3nfoc7bnomgpdmziymlhhy3tgsistf2s36js2nwsfrumtjirba
🏠/workspace
❯ ipfs dag get --format=git-raw bafyreigg3nfoc7bnomgpdmziymlhhy3tgsistf2s36js2nwsfrumtjirba
{"foo" : 5}
🏠/workspace took 5s
❯ echo '{"foo" : 5}' | ipfs dag put --input-enc=git-raw
Error: unrecognized object type: {"foo"

[aschmahmann]

Merging as this seems good enough for go-ipfs v0.10.0-rc1. There are a few outstanding issues linked above that we'll need to deal with before the final release.

Thanks everyone who helped push this over the finish line 🙇‍♂️, this was a huge and much needed endeavor to start upgrading our IPLD library usage.