Support distributing OCI images on IPFS #6175

ktock · 2021-10-29T07:25:12Z

Moved to containerd/nerdctl#505

This commit enables containerd client to distribute OCI images on IPFS.

Push (ipfs daemon needs running):

# ctr image pull ghcr.io/stargz-containers/ubuntu:20.04-org
ghcr.io/stargz-containers/ubuntu:20.04-org:                                       resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:adf73ca014822ad8237623d388cedf4d5346aa72c270c5acc01431cc93e18e2d:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:86ac87f73641c920fb42cc9612d4fb57b5626b56ea2a19b894d0673fd5b4f2e9: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5e9250ddb7d0fa6d13302c7c3e6a0aa40390e42424caed1e5289077ee4054709:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:345e3491a907bb7c6f1bdddcf4a94284b8b6ddd77eb7d93f09432b17b20f2bbe:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:57671312ef6fdbecf340e5fed0fb0863350cd806c92b1fdd7978adbd02afc5c3:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:7e0aa2d69a153215c790488ed1fcec162015e973e49962d438e18249d16fa9bd:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 3.5 s                                                                    total:  27.2 M (7.8 MiB/s)                                       
unpacking linux/amd64 sha256:adf73ca014822ad8237623d388cedf4d5346aa72c270c5acc01431cc93e18e2d...
done: 886.582309ms	
# ctr ipfs push ghcr.io/stargz-containers/ubuntu:20.04-org
bafkreih7o4kebgbzdl5maeq2yxr6nn2abvdztb2lshxvszjzre4s4vsl5a

Pull and run (ipfs daemon needs running):

# ctr ipfs pull bafkreih7o4kebgbzdl5maeq2yxr6nn2abvdztb2lshxvszjzre4s4vsl5a
bafkreih7o4kebgbzdl5maeq2yxr6nn2abvdztb2lshxvszjzre4s4vsl5a:                      resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:c3d79ec819e7572dc0f374a88cc704ba167a867302a749c2d1ed0b0088770409:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:f5a09083ac89d020e91e19e685ad88148bc93810e101837a1854be8c8bc26b80: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:7e0aa2d69a153215c790488ed1fcec162015e973e49962d438e18249d16fa9bd:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:57671312ef6fdbecf340e5fed0fb0863350cd806c92b1fdd7978adbd02afc5c3:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:345e3491a907bb7c6f1bdddcf4a94284b8b6ddd77eb7d93f09432b17b20f2bbe:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5e9250ddb7d0fa6d13302c7c3e6a0aa40390e42424caed1e5289077ee4054709:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 1.2 s                                                                    total:  27.2 M (22.7 MiB/s)                                      
# ctr run --rm -t bafkreih7o4kebgbzdl5maeq2yxr6nn2abvdztb2lshxvszjzre4s4vsl5a foo echo hello
hello

This example uses overlayfs snapshotter but any snapshotters can be used in theory because IPFS is handled by the IPFS-aware remotes.Resolver not by snapshotters. For example, stargz snapshotter supports directly mounting eStargz from IPFS to containers's rootfs with experimental lazy pulling support. Please see https://github.com/containerd/stargz-snapshotter/blob/main/docs/ipfs.md for details.

IPFS-enabled OCI Image

Each descriptor in an IPFS-enabled OCI image must contain the following IPFS URL in urls field.

ipfs://<CID>

<CID> is the case-insensitive CIDv1 of the blob that the descriptor points to.
An image is represented as a CID pointing to the OCI descriptor of the top-level blob of the image (i.e. image index).

# ipfs cat bafkreih7o4kebgbzdl5maeq2yxr6nn2abvdztb2lshxvszjzre4s4vsl5a | jq
{
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "digest": "sha256:c3d79ec819e7572dc0f374a88cc704ba167a867302a749c2d1ed0b0088770409",
  "size": 313,
  "urls": [
    "ipfs://bafkreigd26pmqgphk4w4b43uvcgmobf2cz5im4ycu5e4fupnbmaiq5yebe"
  ]
}

Code changes to containerd client

This commit adds a package contrib/ipfs which contains containerd client codes for distributing OCI images on IPFS.

ipfs.Push pushes the image with converting to the IPFS-enabled format described above. Each blob is pushed as an UnixFS file for leveraging the chunking support.
The pushed image can be fetched using IPFS-aware ipfs.Resolver. When pulling the image, this resolver sees urls field in each descriptor and fetches the contents from IPFS.

Next steps

We will add the following changes to enable containerd to fully support IPFS.

Deciding the image reference naming convention of IPFS images and fixing references pkg of contianerd. (ipfs://<CID> or /ipfs/<CID>)
- Discussed in nerdctl (IPFS (nerdctl run /ipfs/QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV) nerdctl#465).
Fixing components/tools that use containerd (CRI, nerdctl, etc) to support the IPFS image ref.

Related discussions

IPFS (nerdctl run /ipfs/QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV) nerdctl#465
- discussion thread about the image reference for IPFS
Enable to run containers on IPFS stargz-snapshotter#496
- support of IPFS for containerd with experimental lazy pulling
ipcs: https://github.com/hinshun/ipcs
- enables containerd to pull IPFS images with an IPFS-aware content store.
- difference between ipcs and this PR is
  - the image format: ipcs fixes descriptor's digest field to the hash of the blob's root object but this PR leverages urls for keeping OCI compatibility.
  - the integration approach: ipcs integrates content store to IPFS but this PR (and stargz-snapshotter) change client-side (Converter and Resolver) and optionally snapshotter (if enabling lazy pulling).

Thanks @AkihiroSuda for the discussion.

theopenlab-ci · 2021-10-29T07:55:30Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 5m 08s (non-voting)
containerd-test-arm64 : SUCCESS in 6m 25s (non-voting)
containerd-integration-test-arm64 : SUCCESS in 25m 24s (non-voting)

AkihiroSuda · 2021-10-29T08:14:15Z

images/converter/default.go

+}
+
+// IndexConvertFuncWithHook is the convert func used by Convert with hook functions support.
+func IndexConvertFuncWithHook(layerConvertFunc ConvertFunc, docker2oci bool, platformMC platforms.MatchComparer, hooks ConvertHooks) ConvertFunc {


Can you split this to another PR for ease of reviewing and merging

Thank you for the review. Opened the converter hook PR on #6176.

AkihiroSuda · 2021-10-29T08:17:15Z

vendor/github.com/hashicorp/golang-lru/arc.go

+// additional tracking overhead to a standard LRU cache, computationally
+// it is roughly 2x the cost, and the extra memory overhead is linear
+// with the size of the cache. ARC has been patented by IBM, but is
+// similar to the TwoQueueCache (2Q) which requires setting parameters.


Are we actually using this patent-protected code?
Can we avoid importing this?

ref:
hashicorp/golang-lru#31
hashicorp/golang-lru#73

Maybe we should consider temporarily forking the repo to ensure that we aren't importing this

AkihiroSuda · 2021-10-29T08:30:14Z

vendor/github.com/ipfs/go-ipfs-blockstore/arc_cache.go

+// block Cids. This provides block access-time improvements, allowing
+// to short-cut many searches without query-ing the underlying datastore.
+type arccache struct {
+	arc        *lru.TwoQueueCache


It seems that this file has been patched to avoid hitting the ARC patent, but needs confirmation

ipfs/go-ipfs-blockstore@82da4c4

Opened https://github.com/ipfs/go-ipfs-blockstore/issues/87 to request clarification

This PR can compile without vendor/github.com/hashicorp/golang-lru/arc.go.

ktock@d4b8f4f

Should we import the forked version that doesn't contain vendor/github.com/hashicorp/golang-lru/arc.go?

Fixed to use the forked golang-lru that removes vendor/github.com/hashicorp/golang-lru/arc.go.

theopenlab-ci · 2021-10-29T09:48:17Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 5m 01s (non-voting)
containerd-test-arm64 : SUCCESS in 5m 48s (non-voting)
containerd-integration-test-arm64 : FAILURE in 30m 42s (non-voting)

AkihiroSuda · 2021-10-29T10:23:52Z

cmd/ctr/commands/ipfs/push.go

@@ -0,0 +1,105 @@
+/*


Can we have ctr run --ipfs in the same repo too?

How about ctr ipfs pull <CID>? Then we can run it using the normal ctr run.

Added ctr ipfs pull <CID>.

AkihiroSuda · 2021-10-29T10:24:56Z

images/converter/ipfs/ipfs.go

+   limitations under the License.
+*/
+
+package ipfs


Maybe this package should be under contrib/ipfs/converter or contrib/converter/ipfs

moved ipfs-related codes to contrib/ipfs.

theopenlab-ci · 2021-10-29T13:43:32Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 4m 43s (non-voting)
containerd-test-arm64 : SUCCESS in 6m 26s (non-voting)
containerd-integration-test-arm64 : FAILURE in 25m 11s (non-voting)

dmcgowan · 2021-10-29T19:36:22Z

Is this a feature that could start out in nerdctl? ctr doesn't seem like the right place to introduce it

ktock · 2021-10-30T01:38:26Z

Could we move #6176 forward at least? That is a generic patch and will be used by nerdctl and stargz-snapshotter.

Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>

theopenlab-ci · 2021-11-02T01:34:36Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 5m 17s (non-voting)
containerd-test-arm64 : SUCCESS in 6m 09s (non-voting)
containerd-integration-test-arm64 : SUCCESS in 24m 40s (non-voting)

kzys · 2021-11-12T00:22:50Z

#6176 has been merged.

Regarding IPFS, I agree with @dmcgowan. While IPFS is really cool, we don't have to have that in containerd itself.

ktock · 2021-11-12T00:28:07Z

IPFS will come to nerdctl soon containerd/nerdctl#505

ktock · 2021-11-12T00:40:44Z

@dmcgowan @kzys Thanks for the comments. I'm closing this PR in favor of the work in nerdctl (containerd/nerdctl#505). I'll reopen it when changes will be needed to containerd itself.

ktock · 2021-11-12T00:58:51Z

@AkihiroSuda I'll complete the PR in nerdctl first. Thank you for your time to review this PR and the one in nerdctl.

ktock force-pushed the ipfs-push branch from 49c20ed to 060b2d6 Compare October 29, 2021 07:28

AkihiroSuda reviewed Oct 29, 2021

View reviewed changes

ktock mentioned this pull request Oct 29, 2021

converter: Allow hooks during image conversion #6176

Merged

AkihiroSuda reviewed Oct 29, 2021

View reviewed changes

ktock force-pushed the ipfs-push branch 2 times, most recently from 8f96335 to dad1577 Compare October 29, 2021 09:15

AkihiroSuda reviewed Oct 29, 2021

View reviewed changes

ktock force-pushed the ipfs-push branch from dad1577 to c4c21d8 Compare October 29, 2021 13:03

ktock changed the title ~~Support pushing OCI image to IPFS~~ Support distributing OCI images on IPFS Oct 29, 2021

ktock force-pushed the ipfs-push branch from c4c21d8 to 420ace6 Compare October 29, 2021 13:16

AkihiroSuda approved these changes Oct 29, 2021

View reviewed changes

dmcgowan added this to New in Code Review via automation Oct 29, 2021

dmcgowan moved this from New to Needs Discussion in Code Review Oct 29, 2021

Support distributing OCI image on IPFS

5a549be

Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>

ktock force-pushed the ipfs-push branch from 420ace6 to 5a549be Compare November 2, 2021 01:08

ktock closed this Nov 12, 2021

Code Review automation moved this from Needs Discussion to Done Nov 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support distributing OCI images on IPFS #6175

Support distributing OCI images on IPFS #6175

ktock commented Oct 29, 2021 •

edited

Loading

theopenlab-ci bot commented Oct 29, 2021

AkihiroSuda Oct 29, 2021

ktock Oct 29, 2021

AkihiroSuda Oct 29, 2021

AkihiroSuda Oct 29, 2021

AkihiroSuda Oct 29, 2021

AkihiroSuda Oct 29, 2021

ktock Oct 29, 2021

ktock Oct 29, 2021

theopenlab-ci bot commented Oct 29, 2021

AkihiroSuda Oct 29, 2021

ktock Oct 29, 2021

AkihiroSuda Oct 29, 2021

ktock Oct 29, 2021

AkihiroSuda Oct 29, 2021 •

edited

Loading

ktock Oct 29, 2021

theopenlab-ci bot commented Oct 29, 2021

dmcgowan commented Oct 29, 2021

ktock commented Oct 30, 2021

theopenlab-ci bot commented Nov 2, 2021

kzys commented Nov 12, 2021

ktock commented Nov 12, 2021

ktock commented Nov 12, 2021

ktock commented Nov 12, 2021

Support distributing OCI images on IPFS #6175

Support distributing OCI images on IPFS #6175

Conversation

ktock commented Oct 29, 2021 • edited Loading

Moved to containerd/nerdctl#505

IPFS-enabled OCI Image

Code changes to containerd client

Next steps

Related discussions

theopenlab-ci bot commented Oct 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theopenlab-ci bot commented Oct 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AkihiroSuda Oct 29, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

theopenlab-ci bot commented Oct 29, 2021

dmcgowan commented Oct 29, 2021

ktock commented Oct 30, 2021

theopenlab-ci bot commented Nov 2, 2021

kzys commented Nov 12, 2021

ktock commented Nov 12, 2021

ktock commented Nov 12, 2021

ktock commented Nov 12, 2021

ktock commented Oct 29, 2021 •

edited

Loading

AkihiroSuda Oct 29, 2021 •

edited

Loading