DO NOT MERGE: OCI ImageDestination #102

runcom · 2016-06-14T00:38:25Z

This doesn't do much except 90b7bdd which moves manifest utils to a top level pkg.

The rest implements OCI ImageDestination which basically outputs to a directory following https://github.com/opencontainers/image-spec/blob/master/image-layout.md

Note that the OCI image spec it's still WIP itself.

This is still WIP but I greatly appreciate any comment @mtrmac (there are some ugliness just to make this work so don't bother about them I will fix it of course, I'd like to have an impression on the direction first)

It's already kind of working (expect reference naming which has an issue upstream as well):

$ ./skopeo copy docker://docker.io/fedora oci:docker.io/fedora
INFO[0001] Downloading library/fedora/blobs/sha256:8b53fe19c255ee721e5b2441b8469fc01a6324e8564de89bf8485387b34f95e6 
INFO[0045] Downloading library/fedora/blobs/sha256:a20665eb1fe2912accb3d5dadaed360430df0d1aa46874875886947d61d3d4ee 

$ tree docker.io/fedora 
docker.io/fedora
├── blobs
│   ├── sha256-8b53fe19c255ee721e5b2441b8469fc01a6324e8564de89bf8485387b34f95e6
│   ├── sha256-a20665eb1fe2912accb3d5dadaed360430df0d1aa46874875886947d61d3d4ee
│   └── sha256-fdf695e9003ee97fcec524f3f6c5aa70a32b256eab051a6f1cb621b7cf77352a
├── oci-layout
└── refs
    └── TODO

2 directories, 5 files

runcom · 2016-06-14T00:42:10Z

/cc @vbatts @rhatdan

the final directory isn't compressed or anything else because that's still missing definition in the spec from what I've understood

mtrmac · 2016-06-14T13:28:25Z

manifest/manifest.go

@@ -10,19 +10,31 @@ import (

 // FIXME: Should we just use docker/distribution and docker/docker implementations directly?

+// FIXME(runcom, mitr): should we havea mediatype pkg??


I don’t quite feel we need to have a separate package for everything. Sure, use packages to simplify dependencies and avoid loops and have clear separation between transports, conversions and signatures, and from time to time to simplify the callers’ notation to avoid too many words in a symbol reference, but at some point adding a complex difficult-to-navigate package structure could make understanding more difficult instead of simpler.

mtrmac · 2016-06-14T13:32:17Z

In mtrmac/skopeo:verify-on-pull I have already moved the layer parsing out of copy.go; it would be good to integrate this here. Shall I separate that part of the verify-on-pull branch into a separate, immediately mergeable, PR?
Moving skopeo/docker/utils to skopeo/manifest: sure, why not; but when dropping the Manifest part of function names, it would read cleaner to keep the module reference as manifest.Digest instead of mutils.Digest. etc.

mtrmac · 2016-06-14T13:33:58Z

oci/oci.go

+
+func (d *ociImageDestination) PutSignatures(signatures [][]byte) error {
+	// TODO
+	return nil


If this TODO comment is not a short-term thing, the method should return an error instead of silently throwing away data.

but wait, then skopeo copy can't work well because it always error out on failed to upload signatures. any idea?

if len(signatures) != 0 { return $someError }.

At the moment images with signatures are nonexistent, so the above would work well enough. Later we will need a skopeo copy --drop-signatures.

alright, that makes sense, wasn't sure about that

runcom · 2016-06-14T13:38:51Z

In mtrmac/skopeo:verify-on-pull I have already moved the layer parsing out of copy.go; it would be good to integrate this here. Shall I separate that part of the verify-on-pull branch into a separate, immediately mergeable, PR?

would be cool - pls do if you can

Moving skopeo/docker/utils to skopeo/manifest: sure, why not; but when dropping the Manifest part of function names, it would read cleaner to keep the module reference as manifest.Digest instead of mutils.Digest. etc.

right the issue here is that we have so many vars called manifest - I'll see if I can replace them all

mtrmac · 2016-06-14T13:56:41Z

In mtrmac/skopeo:verify-on-pull I have already moved the layer parsing out of copy.go; it would be good to integrate this here. Shall I separate that part of the verify-on-pull branch into a separate, immediately mergeable, PR?

would be cool - pls do if you can

#103 .

runcom · 2016-06-14T14:21:21Z

@mtrmac PTAL while I import #103 here after merged

runcom · 2016-06-14T14:22:26Z

oci/oci.go

+func NewOCIImageDestination(dir string) (types.ImageDestination, error) {
+	// oci dest follow docker's one - parse those here
+	// there's no clear direction on naming though but since image-spec is WIP just assume this for now..
+	ref, tag, err := parseDockerImageName(dir)


@mtrmac this is important but I guess it's fine as it is right now :(

In general paths are not valid Docker Image Names; Is even /home/mitr/some-dir valid? /home/mitr/I_LIKE_CAPS/some_dir definitely isn’t valid.

right, that's why we error out if the destination name in oci:docker.io/fedora isn't a valid reference.

AFAICS d.ref is only used for FullName (i.e. to turn it back into a string), this seems not worth all the effort; just find the last : and split into directory + tag?

(Have I ever mentioned that I hate mini-languages and strings and in-band signalling in general, a tiny bit? :) Yeah yeah, we can’t avoid strings for command line.)

(Have I ever mentioned that I hate mini-languages and strings and in-band signalling in general, a tiny bit? :) Yeah yeah, we can’t avoid strings for command line.)

haha now I know it

AFAICS d.ref is only used for FullName (i.e. to turn it back into a string), this seems not worth all the effort; just find the last : and split into directory + tag?

was smelling to me as well - I'll do it

just find the last : and split into directory + tag?

how do we make sure directory is valid? valid as in docker.io/fedora for instance, I think we should require oci: prefix to enforce a valid destination, or not?

mtrmac · 2016-06-14T16:22:48Z

Copying to preserve:

I guess PutManifest would really benefit from a MIME type parameter.

but should that parameter match the actual mime type in the manifest bytes? otherwise, it's just asking to covert bytes to the correct manifest right?

Yes; conversion, if any would happen in the caller. And the caller may not know the manifest type either (see GetManifest). But if the caller does know, it seems a bit pointless, and potentially dangerous, to throw that away and guess.

Right, we'll figure that out once I actually come here and hack on this TODO, can you move those comments in the main PR page instead of per commit? (sorry I don't want to lose them once I push force :()

runcom · 2016-06-14T16:53:24Z

@mtrmac ~~fixing latest stuff before trying to merge with #103 :)~~

Does it look relatively good to go and try to merge with #103? Otherwise, I though about merging this as is and I'll follow on carrying #103 and adapting it for OCI - just so we have some OCIness into skopeo master, does this sound resonable?

runcom · 2016-06-15T12:27:47Z

oci/oci_dest.go

+	if err := d.ensureParentDirectoryExists("refs"); err != nil {
+		return err
+	}
+	// TODO(runcom):


this is the only big todo left

@mtrmac ptal at the last commit

mtrmac · 2016-06-24T14:36:17Z

oci/oci_dest.go

+func (d *ociImageDestination) ensureParentDirectoryExists(parent string) error {
+	path := filepath.Join(d.dir, parent)
+	if _, err := os.Stat(path); err != nil && os.IsNotExist(err) {
+		if err := os.MkdirAll(path, 0700); err != nil {


Use 0755 perhaps? This is using 0700, and files within these directories are created using 0644. They don’t have to be consistent (the 0700 on the directory prevails) but it would be nice for them to be.

mtrmac · 2016-06-24T14:37:11Z

@mtrmac PTAL - I've added 2 more commits after rebasing

Read this line by line, that should be all.

runcom · 2016-06-24T14:39:48Z

On it, thanks a lot

runcom · 2016-06-25T09:22:07Z

@mtrmac test failures seems related?

mtrmac · 2016-06-27T13:37:02Z

@mtrmac test failures seems related?

Yeah; without actually placing any breakpoints in, it seems that this changes the manifest we get from the Docker Hub from v1s1 to v2s2, and the OpenShift registry does not support v2s2 and rejects it. Oops.

A short-term fix would be to either find s1-only images hosted on the Docker Hub (do they exist, or have they converted everything to s2?), or perhaps to modify the tests to pull some local images (fixtures?) from any one of the 3? in-container Docker registries instead of relying on the Docker Hub (which would be nice anyway).

Longer-term, we absolutely need to figure this out. I suppose it is easy enough to add SupportedDestinationMIMETypes to an ImageDestination, and to backpropagate that information through Image (which would need a new parameter) to ImageSource (which already supports it in GetManifest).

(It is also conceptually really awkward; a tagged Docker Reference, even if not retagged, does not specify a unique image, you can get something completely different depending on what destination you want to store the image in. sigh, no helping this now, I suppose.)

runcom · 2016-06-28T07:05:46Z

image/image.go

+	for _, layer := range m.Layers {
+		blobs = append(blobs, layer.Digest)
+	}
+	blobs = append(blobs, m.Config.Digest)


I'd like to sort this out before moving to containers/image - so when a v2s2 manifest is retrieved and we look for layers I'm appending the config digest here because it's needed (as it was in v2s1, but v2s1 had the config embedded into the manifest itself) so now we have to retrieve the config somehow. Would it be better by adding a Config() to the generalManifest interface? or is this ok? /cc @mtrmac

nvm tried a fix into containers/image#19

Yeah, see the conversation around containers/image#19.

runcom · 2016-06-28T14:00:06Z

I'll keep this open until we merge:

containers/image#19
containers/image#17

after those are merged, I'll split the OCI bits to containers/image and refactor code here

mtrmac · 2016-06-28T15:21:24Z

containers/image#17 / containers/image#19 as it is now does not, AFAICS, fix the integration test failures (pulling s2 from Docker Hub and trying to save it into OpenShift, which only supports s1).

runcom · 2016-06-28T15:26:48Z

they don't I'm just tracking progress in containers/image to go forward with this one

Signed-off-by: Antonio Murdaca <runcom@redhat.com>

runcom · 2016-06-28T17:20:29Z

eah; without actually placing any breakpoints in, it seems that this changes the manifest we get from the Docker Hub from v1s1 to v2s2, and the OpenShift registry does not support v2s2 and rejects it. Oops.

A short-term fix would be to either find s1-only images hosted on the Docker Hub (do they exist, or have they converted everything to s2?), or perhaps to modify the tests to pull some local images (fixtures?) from any one of the 3? in-container Docker registries instead of relying on the Docker Hub (which would be nice anyway).

right, using images with v2s1 only manifests should fix the test - I just pushed to test my fix out

runcom · 2016-06-28T17:45:34Z

right, test passed but we should look forward to fix this as you described in #102 (comment)

mtrmac · 2016-06-28T18:16:21Z

right, test passed but we should look forward to fix this as you described in #102 (comment)

Good to confirm that this is the culprit; if this is fixed, it would be nice to revert to using the library/ images again so that we can have confidence in this working for the common users and uses.

runcom · 2016-06-28T18:17:23Z

Right, we still have to fix the underlying issue though :( taking care of this tomorrow probably

Signed-off-by: Antonio Murdaca <runcom@redhat.com>

mtrmac · 2016-06-29T14:46:40Z

cmd/skopeo/copy.go


 	dest, err := parseImageDestination(context, context.Args()[1])
 	if err != nil {
 		return fmt.Errorf("Error initializing %s: %v", context.Args()[1], err)
 	}
 	signBy := context.String("sign-by")
+	src := image.FromSource(rawSource, dest.SupportedImageDestinationMIMEType())


Yay, it works! Stylistically it would be a bit more readable to do all of dest first and source second, then, so that we have a clear, simple dest paragraph and src paragraph.

+1 will do it after we merge containers/image#26 and do the full rebase here

runcom force-pushed the oci-2 branch 4 times, most recently from dcea9d0 to bb74d94 Compare June 14, 2016 12:03

mtrmac reviewed Jun 14, 2016
View reviewed changes

runcom force-pushed the oci-2 branch from bb74d94 to cc42b66 Compare June 14, 2016 14:20

runcom reviewed Jun 14, 2016
View reviewed changes

runcom force-pushed the oci-2 branch from cc42b66 to 14f8f73 Compare June 14, 2016 16:19

runcom force-pushed the oci-2 branch 2 times, most recently from ca327a8 to 947cc0e Compare June 14, 2016 16:52

runcom force-pushed the oci-2 branch 6 times, most recently from ab5cba6 to c3be1ba Compare June 15, 2016 12:27

runcom reviewed Jun 15, 2016
View reviewed changes

mtrmac reviewed Jun 24, 2016
View reviewed changes

runcom force-pushed the oci-2 branch from 156b7e7 to a01abf9 Compare June 24, 2016 16:11

runcom force-pushed the oci-2 branch 2 times, most recently from 6328231 to 49c2d26 Compare June 26, 2016 11:48

mtrmac mentioned this pull request Jun 27, 2016

Add a test against skopeo? containers/image#13

Closed

runcom reviewed Jun 28, 2016
View reviewed changes

runcom mentioned this pull request Jun 28, 2016

OCI ImageDestination containers/image#23

Merged

fix after review

c6e7a31

Signed-off-by: Antonio Murdaca <runcom@redhat.com>

runcom force-pushed the oci-2 branch from 49c2d26 to c6e7a31 Compare June 28, 2016 17:19

runcom changed the title ~~OCI ImageDestination~~ DO NOT MERGE: OCI ImageDestination Jun 28, 2016

runcom mentioned this pull request Jun 29, 2016

*: add SupportedImageDestinationMIMEType to types.ImageDestination containers/image#26

Merged

try opencontainers/image #26

7624571

Signed-off-by: Antonio Murdaca <runcom@redhat.com>

mtrmac reviewed Jun 29, 2016
View reviewed changes

runcom closed this in containers/image#26 Jun 29, 2016

runcom deleted the oci-2 branch June 29, 2016 17:17

runcom mentioned this pull request Jun 29, 2016

add possibility to download to OCI image-layout #133

Merged

owtaylor mentioned this pull request Aug 24, 2017

Don't expect image sources to convert images for us containers/image#331

Closed

github-actions bot added the locked - please file new issue/PR label Oct 10, 2023

github-actions bot locked as resolved and limited conversation to collaborators Oct 10, 2023

		@@ -10,19 +10,31 @@ import (

		// FIXME: Should we just use docker/distribution and docker/docker implementations directly?

		// FIXME(runcom, mitr): should we havea mediatype pkg??

DO NOT MERGE: OCI ImageDestination #102

DO NOT MERGE: OCI ImageDestination #102

Conversation

runcom commented Jun 14, 2016 • edited Loading

runcom commented Jun 14, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtrmac commented Jun 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

runcom commented Jun 14, 2016

mtrmac commented Jun 14, 2016

runcom commented Jun 14, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtrmac commented Jun 14, 2016

runcom commented Jun 14, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtrmac commented Jun 24, 2016

runcom commented Jun 24, 2016

runcom commented Jun 25, 2016

mtrmac commented Jun 27, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

runcom commented Jun 28, 2016

mtrmac commented Jun 28, 2016 • edited Loading

runcom commented Jun 28, 2016

runcom commented Jun 28, 2016

runcom commented Jun 28, 2016

mtrmac commented Jun 28, 2016

runcom commented Jun 28, 2016

mtrmac Jun 29, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

runcom commented Jun 14, 2016 •

edited

Loading

runcom commented Jun 14, 2016 •

edited

Loading

runcom commented Jun 14, 2016 •

edited

Loading

mtrmac commented Jun 28, 2016 •

edited

Loading

mtrmac Jun 29, 2016 •

edited

Loading