ctr import: strictly match platform

[ginglis13]

Currently, ctr import will use loose matching as defined by platforms.Only(..), meaning in the case of platform linux/amd64 as in issue #6441 , #6441 (comment), importing will also match linux/386 platform on the image-to-be-imported's index. However, that image-to-be-imported may not have both the linux/amd64 and linux/386 platform contents, resulting in a failure to unpack the image : ctr: content digest sha256:abc123xyz: not found, where the content not found is the image's manifest for the other platform.

This change makes that platform matching strict such that the requested platform to import for is the only platform content imported, avoiding implicit attempts to import content for another platform. ctr export will treat the platform option as strict, so this change makes ctr import consistent with that.

If it's requested to expand the import --platform option to allow specifying multiple platforms (as ctr image export does):

containerd/cmd/ctr/commands/images/export.go

Lines 72 to 84 in 68d9d46

	if pss := context.StringSlice("platform"); len(pss) > 0 {
	var all []ocispec.Platform
	for _, ps := range pss {
	p, err := platforms.Parse(ps)
	if err != nil {
	return fmt.Errorf("invalid platform %q: %w", ps, err)
	}
	all = append(all, p)
	}
	exportOpts = append(exportOpts, archive.WithPlatform(platforms.Ordered(all...)))
	} else {
	exportOpts = append(exportOpts, archive.WithPlatform(platforms.Default()))
	}

then I can include those changes in this PR.

resolves #6441

Signed-off-by: Gavin Inglis giinglis@amazon.com

[k8s-ci-robot]

Hi @ginglis13. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kzys]

The change looks good to me. Is it possible to notify the number of matched platforms and warn a user if no platforms are matched?

The matching behavior confused a lot of people and we had multiple iterations. So it may be better to tell end users what's happening exactly.

[ginglis13]

The change looks good to me. Is it possible to notify the number of matched platforms and warn a user if no platforms are matched?

Hmm I'm not so sure how that could be propagated back to the user.. Are you expecting this notification to come from the actual import and/or unpack method itself? i.e. within client.Import, or rather check the platforms of imgs returned by client.Import(ctx, r, opts...)?

platform matching calls FilterPlatforms

containerd/images/handlers.go

Line 261 in 68d9d46

func FilterPlatforms(f HandlerFunc, m platforms.Matcher) HandlerFunc {

maybe in this function we can have that notification? although FilterPlatforms is also used in push, pull, image ls ...

[kzys]

@ginglis13 Hmm I see. I'm fine if that is complicated.

[kzys]

@jonyhy96 @AkihiroSuda Can you take a look?

[jonyhy96]

LGTM Thanks for the contribution!
BTW, when looking for solution of Is it possible to notify the number of matched platforms and warn a user if no platforms are matched?. Find that the export may have some promblems.
Steps to reproduce:

$ ctr image pull --platform linux/amd64 docker.io/library/postgres:12
$ ctr image export --platform linux/amd64 postgres-12.tar docker.io/library/postgres:12
$ ctr image import --all-platform postgres-12.tar

content digest sha256:xxx: not found

[jonyhy96]

The change looks good to me. Is it possible to notify the number of matched platforms and warn a user if no platforms are matched?

The matching behavior confused a lot of people and we had multiple iterations. So it may be better to tell end users what's happening exactly.

https://github.com/containerd/containerd/blob/main/import.go#L199-L201
We can make this func return imgs,err and do error assertion with errors.Is(err,errdefs.ErrNotFound) on caller and do some warning base on imgs returned.

[ginglis13]

Steps to reproduce:

$ ctr image pull --platform linux/amd64 docker.io/library/postgres:12
$ ctr image export --platform linux/amd64 postgres-12.tar docker.io/library/postgres:12
$ ctr image import --all-platform postgres-12.tar

content digest sha256:xxx: not found

In the case of specifying --all-platforms, is it expected that import would succeed only for platform linux/amd64, and warn about failure to import for other platforms?

https://github.com/containerd/containerd/blob/main/import.go#L199-L201
We can make this func return imgs,err and do error assertion with errors.Is(err,errdefs.ErrNotFound) on caller and do some warning base on imgs returned.

In this specific case at least, only 1 image would be returned, and I'm not sure if we could extract useful information from that for a warning message (i.e., the platforms for which content could not be found).

@jonyhy96 The original content not found error in your example and the original issue can be traced to ReadBlob in images.Children

containerd/images/image.go

Lines 341 to 345 in e217c83

	case MediaTypeDockerSchema2Manifest, ocispec.MediaTypeImageManifest:
	p, err := content.ReadBlob(ctx, provider, desc)
	if err != nil {
	return nil, err
	}

this error will propagate all the way to WalkNotEmpty/Walk, and import would still fail. FilterPlatforms is where platform information could be extracted and check errors.Is(err,errdefs.ErrNotFound) to warn (without returning error) about failure to find content for a specific platform, i.e.:

if err != nil {
    if !errdefs.IsNotFound(err) {
        return children, err
    } else {
        log.G(ctx).Warnf("content for platform %s not found", platforms.Format(*desc.Platform))
    }
}

This would work for your provided example, but I'm not sure that logging directly from this function is acceptable, or the implications for other functions using FilterPlatforms.

[jonyhy96]

In the case of specifying --all-platforms, is it expected that import would succeed only for platform linux/amd64, and warn about failure to import for other platforms?
In the case of specifying --all-platforms, is it expected that import would succeed only for platform linux/amd64, and warn about failure to import for other platforms?

Not sure if it will warn about failure to import for other platforms, at least it should success on platforms that it exported with. IMO, all-platforms means all platforms this image supports.

FilterPlatforms is where platform information could be extracted and check errors.Is(err,errdefs.ErrNotFound) to warn (without returning error) about failure to find content for a specific platform, i.e.:

if err != nil {
    if !errdefs.IsNotFound(err) {
        return children, err
    } else {
        log.G(ctx).Warnf("content for platform %s not found", platforms.Format(*desc.Platform))
    }
}

This would work for your provided example, but I'm not sure that logging directly from this function is acceptable, or the implications for other functions using FilterPlatforms.

The desc.Platform could be nil due to export issue above.

[vishwanathjadhav]

@ginglis13, @kzys - Hi Guys, I am facing a similar issue while importing the busybox images. Is there a way we can export the image with more than one platform so that we can export it for both "linux/amd64" and "linux/386" .
Please let me know in which release version I should expect the fix.

[Kern--]

Not sure if it will warn about failure to import for other platforms, at least it should success on platforms that it exported with. IMO, all-platforms means all platforms this image supports.

ctr image export exports the original image index which contains all platforms. There's no component that "knows" what platforms were exported - that is implicitly handled by which blobs end up in the archive. So at least by the current logic the exported image supports all the platforms the original image supported.

Maybe helpful for others in the future, but this is the current logic without this PR for import/export:

Export:
no flags: Loose matching (e.g. linux/386 and linux/amd64 will match on an amd64 machine)
--platform $PLAT: Strict matching (e.g. linux/386 will not match --platform linux/amd64)
--all-platforms: Matches all platforms that the image supports

Import:
no flags: Loose matching (e.g. linux/386 and linux/amd64 will match on amd64 machine)
--platform $PLAT: Loose matching (e.g. linux/386 and linux/amd64 will match --platform linux/amd64)
--all-platforms: Matches all platforms in the original index

[kzys]

/ok-to-test

[estesp]

Since this is effectively a bug fix, we should put it into our supported release branches