Enable image distribution on IPFS #505

ktock · 2021-11-09T08:59:30Z

This commit enables image distribution on IPFS when ipfs:// prefix is specified before the image name.
(ipfs daemon needs running)

push: pushes the specified image to IPFS

The following example pushes ubuntu:20.04 (pulled from DockerHub with nerdctl pull ubuntu:20.04 in advance) to IPFS

# nerdctl push ipfs://ubuntu:20.04
INFO[0000] pushing image "ubuntu:20.04" to IPFS         
INFO[0000] ensuring image contents                      
INFO[0005] converted "application/vnd.docker.image.rootfs.diff.tar.gzip" to sha256:3645595d053b17b1910effafdcc6d924ff0f915b25ce81a07f76a7b0bb22c964 
bafkreigrkfhzm5bt5xbb7tvrnwzrkvulgipwupc4w24kwudysexmwzyske

pull: pulls the specified image from IFPS

# nerdctl pull ipfs://bafkreigrkfhzm5bt5xbb7tvrnwzrkvulgipwupc4w24kwudysexmwzyske
bafkreigrkfhzm5bt5xbb7tvrnwzrkvulgipwupc4w24kwudysexmwzyske:                      resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:cfbf2274a7499e2ccf1738793396cce136576786415d4f3da20e500491c21e95:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:78ef4cff96127a9c9302508ab8116d692e68f863f43598c00aa202ed216d961c: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:92978473846792f54f088079a8be7b26ab384e08271a3c6b2ca699f75c6a0b25:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3645595d053b17b1910effafdcc6d924ff0f915b25ce81a07f76a7b0bb22c964:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 1.2 s                                                                    total:  29.0 M (24.2 MiB/s)

run:

# nerdctl run --rm -it ipfs://bafkreigrkfhzm5bt5xbb7tvrnwzrkvulgipwupc4w24kwudysexmwzyske echo hello
hello

compose:

# mkdir /tmp/proj
# cat <<EOF > /tmp/proj/docker-compose.yml
version: "3.4"
services:
  ubuntu:
    image: ipfs://bafkreigrkfhzm5bt5xbb7tvrnwzrkvulgipwupc4w24kwudysexmwzyske
    command: echo hello
EOF
# nerdctl compose up -f /tmp/proj/docker-compose.yml
INFO[0001] Creating container proj_ubuntu_1             
INFO[0000] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4] 
INFO[0000] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844] 
INFO[0001] Attaching to logs                            
ubuntu_1 |hello
INFO[0001] Container "proj_ubuntu_1" exited             
INFO[0001] All the containers have exited               
INFO[0001] Stopping containers (forcibly)               
INFO[0001] Stopping container proj_ubuntu_1

encryption (needs to disable estargz conversoin):

# nerdctl pull ghcr.io/stargz-containers/ubuntu:20.04-org
# nerdctl image encrypt --recipient=jwe:/tmp/mypubkey.pem --platform=linux/amd64 ghcr.io/stargz-containers/ubuntu:20.04-org ghcr.io/stargz-containers/ubuntu:20.04-org-enc
sha256:524694948a388c3116478bbbe86c2934a158f51a1d9b6d28459ac9a5fa573e91
# nerdctl push --ipfs-no-estargz ipfs://ghcr.io/stargz-containers/ubuntu:20.04-org-enc
INFO[0000] pushing image "ghcr.io/stargz-containers/ubuntu:20.04-org-enc" to IPFS 
INFO[0000] ensuring image contents                      
bafkreid4rpdokddpdxkjplmbgijyilmdneqzl3vpxzgpsq4764d5rrm4qq
# nerdctl run --rm -it ipfs://bafkreid4rpdokddpdxkjplmbgijyilmdneqzl3vpxzgpsq4764d5rrm4qq echo hello
FATA[0000] failed to extract layer sha256:ccdbb80308cc5ef43b605ac28fac29c6a597f89f5a169bbedbb8dec29c987439: failed to get stream processor for application/vnd.oci.image.layer.v1.tar+gzip+encrypted: exec: "ctd-decoder": executable file not found in $PATH: unknown 
# nerdctl pull --unpack=false ipfs://bafkreid4rpdokddpdxkjplmbgijyilmdneqzl3vpxzgpsq4764d5rrm4qq echo hello
# nerdctl image decrypt --key=/tmp/mykey.pem ipfs://bafkreid4rpdokddpdxkjplmbgijyilmdneqzl3vpxzgpsq4764d5rrm4qq ubuntu:decrypted
sha256:2aa04b1e79ab1ca8af57a5bf67649871cb5fb6118e9cb07660cd6f1f435c5f80
# nerdctl run --rm -it ubuntu:decrypted echo hello
INFO[0000] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4] 
INFO[0000] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844] 
hello

~~This commit also adds nerdctl ipfs push and nerdctl ipfs pull subcommands as the low-level CLI can be used for debugging.~~

Lazy pulling with stagz snapshotter (doc) is enabled when --snapshotter=stargz is specified.

TODO:
- agreement on CLI design
- tests
- nerdctl build should support pulling base images from IPFS (can be a following-up PR)
  - We can achive this by having a proxy which converts OCI Distribution Spec API to IPFS.
  - blocker: Fix containerd fails to pull OCI image with non-http(s):// urls containerd#6221
    - must be downstreamed to BuildKit. [v0.9] Bump up containerd to the latest 1.5 moby/buildkit#2462
- have an option for image encryption (can be a following-up PR)

AkihiroSuda · 2021-11-09T09:50:33Z

cmd/nerdctl/ipfs_pull.go

+	ipfsPullCommand.PersistentFlags().Bool("all-platforms", false, "Pull content for all platforms")
+	// #endregion
+
+	ipfsPullCommand.PersistentFlags().String("scheme", "ipfs", "Scheme to pull the CID (\"ipfs\" or \"ipns\")")


nit: register flag completion func

AkihiroSuda · 2021-11-09T09:51:15Z

cmd/nerdctl/ipfs_pull.go

+
+func newIPFSPullCommand() *cobra.Command {
+	var ipfsPullCommand = &cobra.Command{
+		Use:           "pull",


Explain argument

AkihiroSuda · 2021-11-09T09:53:20Z

cmd/nerdctl/ipfs_push.go

+	ipfsPushCommand.PersistentFlags().Bool("all-platforms", false, "Push content for all platforms")
+	// #endregion
+
+	ipfsPushCommand.PersistentFlags().Bool("estargz", true, "Convert the image into eStargz")


If we want --estargz flag here, we should add that to the regular nerdctl image push too, and deduplicate the code across nerdctl image push and nerdctl ipfs push.
(But do we really need nerdctl ipfs * commands?)

AkihiroSuda · 2021-11-15T03:15:10Z

Dockerfile.d/test-integration-rootless.sh

+            echo "ipfs = true" > ${HOME}/.config/containerd-stargz-grpc/config.toml
+            systemctl --user restart stargz-snapshotter.service
+            sleep 3
+        fi


The logic looks too much complicated.

Maybe we should just run IPFS as a systemd service?

Rootless nerdctl (inside rootlesskit network namespace) can't access to IPFS daemon API (127.0.0.1:5001) on the host?

Right. I thought we could just use a public IP.

Wondering we should have containerd-rootless-setuptool.sh install-ipfs to run IPFS daemon in the netns

nerdctl/extras/rootless/containerd-rootless-setuptool.sh

Lines 397 to 419 in 5b52e3b

usage() {

echo "Usage: ${ARG0} [OPTIONS] COMMAND"

echo

echo "A setup tool for Rootless containerd (${CONTAINERD_ROOTLESS_SH})."

echo

echo "Commands:"

echo " check Check prerequisites"

echo " nsenter Enter into RootlessKit namespaces (mostly for debugging)"

echo " install Install systemd unit and show how to manage the service"

echo " uninstall Uninstall systemd unit"

echo

echo "Add-on commands (BuildKit):"

echo " install-buildkit Install the systemd unit for BuildKit"

echo " uninstall-buildkit Uninstall the systemd unit for BuildKit"

echo

echo "Add-on commands (fuse-overlayfs):"

echo " install-fuse-overlayfs Install the systemd unit for fuse-overlayfs snapshotter"

echo " uninstall-fuse-overlayfs Uninstall the systemd unit for fuse-overlayfs snapshotter"

echo

echo "Add-on commands (stargz):"

echo " install-stargz Install the systemd unit for stargz snapshotter"

echo " uninstall-stargz Uninstall the systemd unit for stargz snapshotter"

}

AkihiroSuda · 2021-11-15T08:41:30Z

cmd/nerdctl/compose_up_test.go

+	testutil.DockerIncompatible(t)
+	base := testutil.NewBase(t)
+	requiresStargz(base)
+	base.Env = append(os.Environ(), "CONTAINERD_SNAPSHOTTER=stargz")


We should test this for overlayfs as well

AkihiroSuda · 2021-11-15T08:43:17Z

cmd/nerdctl/push.go

@@ -47,6 +58,11 @@ func newPushCommand() *cobra.Command {
 	pushCommand.Flags().Bool("all-platforms", false, "Push content for all platforms")
 	// #endregion

+	pushCommand.PersistentFlags().Bool("estargz", false, "Convert the image into eStargz (always true in IPFS mode)")


Not sure why this has to be always true in IPFS mode.

AkihiroSuda · 2021-11-15T08:43:38Z

cmd/nerdctl/push.go

+	pushCommand.PersistentFlags().Bool("estargz", false, "Convert the image into eStargz (always true in IPFS mode)")
+
+	pushCommand.PersistentFlags().Bool("ipfs-no-estargz", false, "Disable conversion of the image into eStargz in IPFS mode")
+	pushCommand.PersistentFlags().Bool("ipfs-ensure-image", true, "Ensure the IPFS image is fully pulled to containerd")


s/pulled/pushed/ (?)

ktock · 2021-11-16T06:43:21Z

CI seems to be cancelled?

Error: The operation was canceled.

AkihiroSuda · 2021-11-16T09:36:19Z

go.mod

 	github.com/mattn/go-isatty v0.0.14
 	github.com/moby/sys/mount v0.2.0
 	github.com/opencontainers/go-digest v1.0.0
 	github.com/opencontainers/image-spec v1.0.2-0.20211102003311-9a7a9876500e
 	github.com/opencontainers/runtime-spec v1.0.3-0.20211101234015-a3c33d663ebc
+	github.com/pkg/errors v0.9.1


Shouldn't be needed

AkihiroSuda · 2021-11-16T09:38:06Z

docs/ipfs.md

+bafkreifajsysbvhtgd7fdgrfesszexdq6v5zbj5y2jnjfwxdjyqws2s3s4
+```
+
+You can pull the encrypted image from IPFS using `ipfs://` prefix and can decrypt it in the same way as described in [`/docs/ocycrypt.md`](/docs/ocicrypt.md).


/docs/ -> ./

AkihiroSuda · 2021-11-16T09:38:25Z

docs/ipfs.md

+hello
+```
+
+:information_source: Note that though the IPFS-enabled image is OCI compatible, some runtimes including [containerd](https://github.com/containerd/containerd/pull/6221) and [podman](https://github.com/containers/image/pull/1403)) had bugs and failed to pull that image. Containerd has already fixed this.


Suggested change

:information_source: Note that though the IPFS-enabled image is OCI compatible, some runtimes including [containerd](https://github.com/containerd/containerd/pull/6221) and [podman](https://github.com/containers/image/pull/1403)) had bugs and failed to pull that image. Containerd has already fixed this.

:information_source: Note that though the IPFS-enabled image is OCI compatible, some runtimes including [containerd](https://github.com/containerd/containerd/pull/6221) and [podman](https://github.com/containers/image/pull/1403) had bugs and failed to pull that image. Containerd has already fixed this.

AkihiroSuda · 2021-11-16T09:43:51Z

Looks good but a couple of nits.
This is awesome 👍

AkihiroSuda · 2021-11-16T12:01:04Z

Probably now we have to increase the CI timeout

nerdctl/.github/workflows/test.yml

Line 55 in f316402

timeout-minutes: 20

ktock · 2021-11-16T12:05:03Z

Probably now we have to increase the CI timeout

I'll fix it in this PR.

docs/ipfs.md

Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>

AkihiroSuda

Thanks

AkihiroSuda added this to the v0.14.0 milestone Nov 9, 2021

AkihiroSuda added enhancement New feature or request impact/major labels Nov 9, 2021

AkihiroSuda reviewed Nov 9, 2021

View reviewed changes

ktock force-pushed the ipfs branch 3 times, most recently from 3e1febb to a46e06a Compare November 10, 2021 02:59

AkihiroSuda modified the milestones: v0.13.1, v0.14.0 Nov 10, 2021

ktock mentioned this pull request Nov 12, 2021

Support distributing OCI images on IPFS containerd/containerd#6175

Closed

AkihiroSuda modified the milestones: vNextNext, vNext Nov 12, 2021

ktock force-pushed the ipfs branch 4 times, most recently from 97cba30 to 98edcd5 Compare November 15, 2021 03:06

AkihiroSuda reviewed Nov 15, 2021

View reviewed changes

ktock force-pushed the ipfs branch 2 times, most recently from e83b542 to 5dee562 Compare November 16, 2021 05:35

ktock force-pushed the ipfs branch 2 times, most recently from e2d5a58 to 5eacb67 Compare November 16, 2021 09:30

ktock marked this pull request as ready for review November 16, 2021 09:30

AkihiroSuda reviewed Nov 16, 2021

View reviewed changes

ktock force-pushed the ipfs branch from 5eacb67 to 0dee508 Compare November 16, 2021 11:39

AkihiroSuda approved these changes Nov 16, 2021

View reviewed changes

ktock force-pushed the ipfs branch 2 times, most recently from e085893 to fdc5335 Compare November 17, 2021 01:05

ktock commented Nov 17, 2021

View reviewed changes

docs/ipfs.md Outdated Show resolved Hide resolved

ktock force-pushed the ipfs branch from fdc5335 to 8e2881f Compare November 17, 2021 01:58

Enable to distribute images on IPFS

2ae16de

Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>

ktock force-pushed the ipfs branch from 8e2881f to 2ae16de Compare November 17, 2021 02:36

AkihiroSuda approved these changes Nov 17, 2021

View reviewed changes

AkihiroSuda merged commit 00c2ca6 into containerd:master Nov 17, 2021

AkihiroSuda mentioned this pull request Nov 17, 2021

update docs + nits #517

Merged

ktock deleted the ipfs branch November 17, 2021 14:37

AkihiroSuda mentioned this pull request Nov 22, 2021

IPFS (nerdctl run /ipfs/QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV) #465

Closed

AkihiroSuda added the area/ipfs IPFS label Nov 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable image distribution on IPFS #505

Enable image distribution on IPFS #505

ktock commented Nov 9, 2021 •

edited

AkihiroSuda Nov 9, 2021

AkihiroSuda Nov 9, 2021

AkihiroSuda Nov 9, 2021

AkihiroSuda Nov 15, 2021

ktock Nov 15, 2021

AkihiroSuda Nov 15, 2021

AkihiroSuda Nov 15, 2021

ktock Nov 15, 2021

AkihiroSuda Nov 15, 2021

AkihiroSuda Nov 15, 2021

AkihiroSuda Nov 15, 2021

ktock commented Nov 16, 2021

AkihiroSuda Nov 16, 2021

AkihiroSuda Nov 16, 2021

AkihiroSuda Nov 16, 2021

AkihiroSuda commented Nov 16, 2021

AkihiroSuda commented Nov 16, 2021

ktock commented Nov 16, 2021

AkihiroSuda left a comment

	usage() {
	echo "Usage: ${ARG0} [OPTIONS] COMMAND"
	echo
	echo "A setup tool for Rootless containerd (${CONTAINERD_ROOTLESS_SH})."
	echo
	echo "Commands:"
	echo " check Check prerequisites"
	echo " nsenter Enter into RootlessKit namespaces (mostly for debugging)"
	echo " install Install systemd unit and show how to manage the service"
	echo " uninstall Uninstall systemd unit"
	echo
	echo "Add-on commands (BuildKit):"
	echo " install-buildkit Install the systemd unit for BuildKit"
	echo " uninstall-buildkit Uninstall the systemd unit for BuildKit"
	echo
	echo "Add-on commands (fuse-overlayfs):"
	echo " install-fuse-overlayfs Install the systemd unit for fuse-overlayfs snapshotter"
	echo " uninstall-fuse-overlayfs Uninstall the systemd unit for fuse-overlayfs snapshotter"
	echo
	echo "Add-on commands (stargz):"
	echo " install-stargz Install the systemd unit for stargz snapshotter"
	echo " uninstall-stargz Uninstall the systemd unit for stargz snapshotter"
	}

	:information_source: Note that though the IPFS-enabled image is OCI compatible, some runtimes including [containerd](https://github.com/containerd/containerd/pull/6221) and [podman](https://github.com/containers/image/pull/1403)) had bugs and failed to pull that image. Containerd has already fixed this.
	:information_source: Note that though the IPFS-enabled image is OCI compatible, some runtimes including [containerd](https://github.com/containerd/containerd/pull/6221) and [podman](https://github.com/containers/image/pull/1403) had bugs and failed to pull that image. Containerd has already fixed this.

Enable image distribution on IPFS #505

Enable image distribution on IPFS #505

Conversation

ktock commented Nov 9, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ktock commented Nov 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AkihiroSuda commented Nov 16, 2021

AkihiroSuda commented Nov 16, 2021

ktock commented Nov 16, 2021

AkihiroSuda left a comment

Choose a reason for hiding this comment

ktock commented Nov 9, 2021 •

edited