Add 'umoci copy' subcommand #359

jdolitsky · 2021-02-08T19:34:12Z

Support for pulling oci:// URL into local store
Support for pushing from local store to registry
Downloads blobs only when not found in local store
Currently only supports OCI v1 manifest media type
Does not support docker:// URLs or Docker manifests
Does not yet support chunked blob uploading

Resolves #345

Demo

Get binaries for zot and umoci (modified), add to PATH

mkdir demo && cd demo/
mkdir bin/
git clone --depth 1 --branch v1.1.12 https://github.com/anuvu/zot.git
(cd zot && make binary && mv bin/zot ../bin/)
git clone --depth 1 --branch copy-subcommand https://github.com/bloodorangeio/umoci.git
(cd umoci && make && mv ./umoci ../bin/)
export PATH="${PWD}/bin:${PATH}"

Start zot server (port 8080)

echo "{\"storage\":{\"rootDirectory\":\"${PWD}/storage\"}}" > zot.yaml
zot serve ./zot.yaml

Use skopeo to fetch Docker manifest, convert to OCI, and push to zot

skopeo copy -f oci --dest-tls-verify=false \
  docker://opensuse/amd64:42.2 docker://127.0.0.1:8080/opensuse/amd64:42.2

umoci copy: remote to local

umoci --log=debug copy --plain-http oci://localhost:8080/opensuse/amd64:42.2 opensuse:42.2

output:

   • Registry address: http://localhost:8080, namespace: opensuse/amd64, tag: 42.2
   • Checking if manifest available in registry
   • Registry reports manifest with digest sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
   • Downloading manifest from registry
   • actual manifest digest matches expected digest (sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed)
   • blob saved to local store, digest: sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed, size: 349
   • Manifest layer list contains 1 item(s)
   • Copying layer 1/1 with digest sha256:f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8
   • actual layer digest matches expected digest (sha256:f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8)
   • blob saved to local store, digest: sha256:f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8, size: 48850323
   • Copying config with digest sha256:2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff
   • actual config digest matches expected digest (sha256:2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff)
   • blob saved to local store, digest: sha256:2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff, size: 720
   • Saving reference '42.2' to index in opensuse

inspect CAS:

$ tree opensuse
opensuse
├── blobs
│   └── sha256
│       ├── 2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff
│       ├── 7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
│       └── f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8
├── index.json
└── oci-layout
$ cat opensuse/index.json | jq
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed",
      "size": 349,
      "annotations": {
        "org.opencontainers.image.ref.name": "42.2"
      }
    }
  ]
}

umoci copy: local to remote

umoci --log=debug copy --plain-http opensuse:42.2 oci://localhost:8080/othersuse/amd64:othertag

output:

   • Registry address: http://localhost:8080, namespace: othersusex/amd64, tag: othertag
   • -> ws.recurse             digest=sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
   • <- ws.recurse             digest=sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
   • casext.ResolveReference(42.2) got these descriptors refs=[{[{application/vnd.oci.image.manifest.v1+json sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed 349 [] map[org.opencontainers.image.ref.name:42.2] <nil>}]}]
   • Reference '42.2' found in index, points to manifest sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
   • Manifest successfully loaded from local store
   • Manifest layer list contains 1 item(s)
   • Uploading layer 1/1 with digest sha256:f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8 from local store
   • Uploading config sha256:2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff
   • Successfully copied to remote localhost:8080

If you add the --trace-requests flag, raw HTTP output will be displayed:

umoci --log=debug copy --plain-http --trace-requests oci://localhost:8080/opensuse/amd64:42.2 opensuse:42.2

output:

   • Registry address: http://localhost:8080, namespace: opensuse/amd64, tag: 42.2
   • Checking if manifest available in registry
2021/02/08 15:58:15.750531 DEBUG RESTY
==============================================================================
~~~ REQUEST ~~~
HEAD  /v2/opensuse/amd64/manifests/42.2  HTTP/1.1
HOST   : localhost:8080
HEADERS:
	Accept: application/vnd.oci.image.manifest.v1+json
	User-Agent: umoci 0.4.6+dev~gitd2f6056482314a44724ac2151622843881a18c45
BODY   :
***** NO CONTENT *****
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 200 OK
RECEIVED AT  : 2021-02-08T15:58:15.749825-05:00
TIME DURATION: 4.758014ms
HEADERS      :
	Content-Length: 0
	Date: Mon, 08 Feb 2021 20:58:15 GMT
	Docker-Content-Digest: sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
BODY         :

==============================================================================
   • Registry reports manifest with digest sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
 ...

- Support for pulling oci:// URL into local store - Support for pushing from local store to registry - Downloads blobs only when not found in local store - Currently only supports OCI v1 manifest media type - Does not support docker:// URLs or Docker manifests - Does not yet support chunked blob uploading Signed-off-by: Josh Dolitsky <josh@dolit.ski>

Signed-off-by: Josh Dolitsky <josh@dolit.ski>

tych0 · 2021-03-10T23:25:21Z

cmd/umoci/copy-remote-to-local.go

+				fmt.Sprintf("Content-Type header for image manifest invalid: %s", mediaType))
+		}
+
+		d, s, err := engine.PutBlob(engineContext, bytes.NewReader(resp.Body()))


As far as I can tell, resp.Body() here is: https://pkg.go.dev/github.com/go-resty/resty/v2#Response.Body which will end up reading the whole layer into memory. Seems like you want RawBody() here?

https://pkg.go.dev/github.com/go-resty/resty/v2#Response.RawBody

Oh, derp. I meant to put this on the layer one below. For this one I think it's unavoidable: you definitely want to read the manifest bytes into memory.

tych0 · 2021-03-10T23:27:00Z

One small nit on this, but otherwise it looks like it might be useful to us. How close do you feel this is to merging?

cyphar · 2021-03-11T05:56:11Z

@tych0

One small nit on this, but otherwise it looks like it might be useful to us. How close do you feel this is to merging?

I've been chatting to @jdolitsky about this for a little bit -- my current plan is for me to rework this PR so that it's implemented as a cas.Engine backend once I get back from vacation next week.

tych0 · 2021-03-11T13:56:50Z

my current plan is for me to rework this PR so that it's implemented as a cas.Engine backend once I get back from vacation next week.

Excellent, thanks!

tych0 · 2021-09-07T14:35:16Z

What's the status of this? I'd like to get some features into containers/image soon-ish, but it would be better to have them here, if possible :)

cyphar · 2021-09-07T17:11:27Z

I haven't touched this in a while. I can try to revive my PoC using the CAS API -- what exactly did you want to get into containers/image? Switching to umoci copy would probably be somewhat non-trivial even if I did get it working soon.

tych0 · 2021-09-07T17:18:53Z

Mostly I don't want it to compress things automatically that aren't gzip :)

cyphar · 2021-09-21T06:22:10Z

I feel the the only slight complication is going to be chunked uploads and downloads which this PoC didn't implement. I started working on a PoC of this, it shouldn't be that bad on paper but implementing a generic umoci copy API might be slightly tricky and I think the CasEngineExt API needs to be reworked somewhat in order to be able to handle arbitrary CAS implementations to wrap).

jdolitsky · 2021-10-20T14:32:41Z

@cyphar - shall I close this for now?

jdolitsky added 2 commits February 8, 2021 14:30

Check in go.mod, go.sum, vendor/

d2f6056

Signed-off-by: Josh Dolitsky <josh@dolit.ski>

tych0 reviewed Mar 10, 2021

View reviewed changes

tych0 mentioned this pull request Mar 25, 2021

invent a way to refer to additional manifest annotations via a URI #366

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add 'umoci copy' subcommand #359

Add 'umoci copy' subcommand #359

jdolitsky commented Feb 8, 2021 •

edited

tych0 Mar 10, 2021

tych0 Mar 10, 2021

tych0 commented Mar 10, 2021

cyphar commented Mar 11, 2021

tych0 commented Mar 11, 2021

tych0 commented Sep 7, 2021

cyphar commented Sep 7, 2021 •

edited

tych0 commented Sep 7, 2021

cyphar commented Sep 21, 2021 •

edited

jdolitsky commented Oct 20, 2021

Add 'umoci copy' subcommand #359

Are you sure you want to change the base?

Add 'umoci copy' subcommand #359

Conversation

jdolitsky commented Feb 8, 2021 • edited

Demo

tych0 Mar 10, 2021

Choose a reason for hiding this comment

tych0 Mar 10, 2021

Choose a reason for hiding this comment

tych0 commented Mar 10, 2021

cyphar commented Mar 11, 2021

tych0 commented Mar 11, 2021

tych0 commented Sep 7, 2021

cyphar commented Sep 7, 2021 • edited

tych0 commented Sep 7, 2021

cyphar commented Sep 21, 2021 • edited

jdolitsky commented Oct 20, 2021

jdolitsky commented Feb 8, 2021 •

edited

cyphar commented Sep 7, 2021 •

edited

cyphar commented Sep 21, 2021 •

edited