Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot pull containers from mcr.microsoft.com due to platform matching #7439

Open
krotz-dieter opened this issue Sep 27, 2022 · 11 comments
Open

Cannot pull containers from mcr.microsoft.com due to platform matching #7439

krotz-dieter opened this issue Sep 27, 2022 · 11 comments
Labels
exp/intermediate kind/bug platform/windows Windows

Comments

@krotz-dieter
Copy link

Description

On Windows 11 Version 10.0.22000.978

docker pull mcr.microsoft.com/windows/nanoserver:ltsc2022

works fine, it pulls the image and
docker run --isolation process -m 4G -it --rm --name test-run mcr.microsoft.com/windows/nanoserver:ltsc2022
also works !
With containerd

crictl pull mcr.microsoft.com/windows/nanoserver:ltsc2022

we get the error:

E0927 11:53:53.142414   23840 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \"mcr.microsoft.com/windows/nanoserver:ltsc2022\": no match for platform in manifest: not found" image="mcr.microsoft.com/windows/nanoserver:ltsc2022"
time="2022-09-27T11:53:53+02:00" level=fatal msg="pulling image: rpc error: code = NotFound desc = failed to pull and unpack image \"mcr.microsoft.com/windows/nanoserver:ltsc2022\": no match for platform in manifest: not found"

The logs from containerd service:

time="2022-09-27T11:29:29.107877800+02:00" level=info msg="PullImage \"mcr.microsoft.com/windows/nanoserver:ltsc2022\""
time="2022-09-27T11:29:29.114833800+02:00" level=debug msg=resolving host=mcr.microsoft.com
time="2022-09-27T11:29:29.114833800+02:00" level=debug msg="do request" host=mcr.microsoft.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.6.8 request.method=HEAD url="https://mcr.microsoft.com/v2/windows/nanoserver/manifests/ltsc2022"
time="2022-09-27T11:29:29.285758600+02:00" level=debug msg="fetch response received" host=mcr.microsoft.com response.header.access-control-expose-headers=Docker-Content-Digest response.header.access-control-expose-headers.1=WWW-Authenticate response.header.access-control-expose-headers.2=Link response.header.access-control-expose-headers.3=X-Ms-Correlation-Request-Id response.header.content-length=350 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Tue, 27 Sep 2022 09:29:37 GMT" response.header.docker-content-digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755\"" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.strict-transport-security.1="max-age=31536000; includeSubDomains" response.header.x-cache=CONFIG_NOCACHE response.header.x-content-type-options=nosniff response.header.x-mcr-env=prod response.header.x-mcr-privacy="https://privacy.microsoft.com/en-us/privacystatement" response.header.x-ms-client-request-id= response.header.x-ms-correlation-request-id=7288f4f6-8a73-404f-aac0-3b2729856453 response.header.x-ms-request-id=dabae9ae-dcad-4a0c-88bd-46fc12c2b699 response.header.x-msedge-ref="Ref A: F822B534419B485F97D8620C44124705 Ref B: MUC30EDGE0415 Ref C: 2022-09-27T09:29:38Z" response.status="200 OK" url="https://mcr.microsoft.com/v2/windows/nanoserver/manifests/ltsc2022"
time="2022-09-27T11:29:29.285758600+02:00" level=debug msg=resolved desc.digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755" host=mcr.microsoft.com
time="2022-09-27T11:29:29.285758600+02:00" level=debug msg=fetch digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=350
time="2022-09-27T11:29:29.293765100+02:00" level=debug msg="do request" digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.accept="application/vnd.docker.distribution.manifest.list.v2+json, */*" request.header.user-agent=containerd/v1.6.8 request.method=GET size=350 url="https://mcr.microsoft.com/v2/windows/nanoserver/manifests/sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755"
time="2022-09-27T11:29:29.464871400+02:00" level=debug msg="fetch response received" digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.access-control-expose-headers=Docker-Content-Digest response.header.access-control-expose-headers.1=WWW-Authenticate response.header.access-control-expose-headers.2=Link response.header.access-control-expose-headers.3=X-Ms-Correlation-Request-Id response.header.content-length=350 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Tue, 27 Sep 2022 09:29:38 GMT" response.header.docker-content-digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755\"" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.strict-transport-security.1="max-age=31536000; includeSubDomains" response.header.x-cache=CONFIG_NOCACHE response.header.x-content-type-options=nosniff response.header.x-mcr-env=prod response.header.x-mcr-privacy="https://privacy.microsoft.com/en-us/privacystatement" response.header.x-ms-client-request-id= response.header.x-ms-correlation-request-id=107253db-1b65-4fd5-91c8-56324ae43208 response.header.x-ms-request-id=7ac3b319-bedf-466c-ae64-8512bac5c0b3 response.header.x-msedge-ref="Ref A: A03DFEAA8D1C446E92EA49CFEAF58449 Ref B: MUC30EDGE0215 Ref C: 2022-09-27T09:29:38Z" response.status="200 OK" size=350 url="https://mcr.microsoft.com/v2/windows/nanoserver/manifests/sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755"
time="2022-09-27T11:29:29.475871600+02:00" level=error msg="PullImage \"mcr.microsoft.com/windows/nanoserver:ltsc2022\" failed" error="rpc error: code = NotFound desc = failed to pull and unpack image \"mcr.microsoft.com/windows/nanoserver:ltsc2022\": no match for platform in manifest: not found"
time="2022-09-27T11:29:29.544241300+02:00" level=debug msg="remove content" key="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755"
time="2022-09-27T11:29:29.546883800+02:00" level=debug msg="schedule content cleanup"
time="2022-09-27T11:29:29.547921900+02:00" level=debug msg="removed content" digest="sha256:ac76db222126ed259f8ac1ed42e7e4564509ea869ae535e3faae866c01cd7755"
time="2022-09-27T11:29:29.547921900+02:00" level=debug msg="content garbage collected" d=1.0381ms
time="2022-09-27T11:29:29.548439700+02:00" level=debug msg="garbage collected" d=3.1736ms

The expectation would be that it would work also within containerd.

Steps to reproduce the issue

crictl pull mcr.microsoft.com/windows/nanoserver:ltsc2022

Describe the results you received and expected

The expectation woould be that it would work also within containerd.

What version of containerd are you using?

containerd github.com/containerd/containerd v1.6.8 9cd3357

Any other relevant information

crictl info

{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "C:\\k\\cni\\bin"
    ],
    "PluginConfDir": "C:\\k\\containerd\\cni\\conf",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "cbr0",
          "CNIVersion": "0.2.0",
          "Plugins": [
            {
              "Network": {
                "cniVersion": "0.2.0",
                "name": "cbr0",
                "type": "flannel",
                "capabilities": {
                  "dnsCapabilities": true,
                  "portMappings": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"dnsCapabilities\":true,\"portMappings\":true},\"cniVersion\":\"0.2.0\",\"delegate\":{\"AdditionalArgs\":[{\"Name\":\"EndpointPolicy\",\"Value\":{\"Settings\":{\"Exceptions\":[\"172.20.0.0/16\",\"172.21.0.0/16\",\"172.19.1.0/24\",\"192.168.44.0/24\"]},\"Type\":\"OutBoundNAT\"}}],\"dns\":{\"Nameservers\":[\"172.20.1.2\"]},\"optionalFlags\":{\"forceBridgeGateway\":true,\"loopbackDSR\":false},\"type\":\"sdnbridge\"},\"name\":\"cbr0\",\"type\":\"flannel\"}"
            }
          ],
          "Source": "{\"cniVersion\":\"0.2.0\",\"name\":\"cbr0\",\"plugins\":[{\"capabilities\":{\"dnsCapabilities\":true,\"portMappings\":true},\"cniVersion\":\"0.2.0\",\"delegate\":{\"AdditionalArgs\":[{\"Name\":\"EndpointPolicy\",\"Value\":{\"Settings\":{\"Exceptions\":[\"172.20.0.0/16\",\"172.21.0.0/16\",\"172.19.1.0/24\",\"192.168.44.0/24\"]},\"Type\":\"OutBoundNAT\"}}],\"dns\":{\"Nameservers\":[\"172.20.1.2\"]},\"optionalFlags\":{\"forceBridgeGateway\":true,\"loopbackDSR\":false},\"type\":\"sdnbridge\"},\"name\":\"cbr0\",\"type\":\"flannel\"}]}"
        },
        "IFName": "eth0"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "windows",
      "defaultRuntimeName": "runhcs-wcow-process",
      "defaultRuntime": {
        "runtimeType": "",
        "runtimePath": "",
        "runtimeEngine": "",
        "PodAnnotations": [],
        "ContainerAnnotations": [],
        "runtimeRoot": "",
        "options": {},
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimePath": "",
        "runtimeEngine": "",
        "PodAnnotations": [],
        "ContainerAnnotations": [],
        "runtimeRoot": "",
        "options": {},
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0
      },
      "runtimes": {
        "runhcs-wcow-process": {
          "runtimeType": "io.containerd.runhcs.v1",
          "runtimePath": "",
          "runtimeEngine": "",
          "PodAnnotations": [],
          "ContainerAnnotations": [],
          "runtimeRoot": "",
          "options": {},
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": "",
          "cniConfDir": "",
          "cniMaxConfNum": 0
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": false,
      "discardUnpackedLayers": false,
      "ignoreRdtNotEnabledErrors": false
    },
    "cni": {
      "binDir": "C:\\k\\cni\\bin",
      "confDir": "C:\\k\\containerd\\cni\\conf",
      "maxConfNum": 1,
      "confTemplate": "",
      "ipPref": ""
    },
    "registry": {
      "configPath": "",
      "mirrors": {},
      "configs": {},
      "auths": {},
      "headers": {
        "User-Agent": [
          "containerd/v1.6.8"
        ]
      }
    },
    "imageDecryption": {
      "keyModel": "node"
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 0,
    "sandboxImage": "mcr.microsoft.com/oss/kubernetes/pause:3.6",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": false,
    "disableHugetlbController": false,
    "device_ownership_from_security_context": false,
    "ignoreImageDefinedVolumes": false,
    "netnsMountsUnderStateDir": false,
    "enableUnprivilegedPorts": false,
    "enableUnprivilegedICMP": false,
    "containerdRootDir": "C:\\containerd\\root",
    "containerdEndpoint": "\\\\.\\pipe\\containerd-containerd",
    "rootDir": "C:\\containerd\\root\\io.containerd.grpc.v1.cri",
    "stateDir": "C:\\containerd\\state\\io.containerd.grpc.v1.cri"
  },
  "golang": "go1.17.13",
  "lastCNILoadStatus": "OK",
  "lastCNILoadStatus.default": "OK"
}

Show configuration if it is related to CRI plugin.

disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "C:\\containerd\\root"
state = "C:\\containerd\\state"
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "\\\\.\\pipe\\containerd-containerd"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = false
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "mcr.microsoft.com/oss/kubernetes/pause:3.6"
    selinux_category_range = 0
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = false
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "C:\\k\\cni\\bin"
      conf_dir = "C:\\k\\containerd\\cni\\conf"
      conf_template = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runhcs-wcow-process"
      disable_snapshot_annotations = false
      discard_unpacked_layers = false
      no_pivot = false
      snapshotter = "windows"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runhcs-wcow-process]
          base_runtime_spec = ""
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runhcs.v1"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runhcs-wcow-process.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "C:\\k\\containerd\\root\\opt"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["windows/amd64", "linux/amd64"]

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["windows", "windows-lcow"]

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "C:\\k\\containerd\\ocicrypt\\keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=C:\\k\\containerd\\ocicrypt\\ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "C:\\k\\containerd\\ocicrypt\\keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=C:\\k\\containerd\\ocicrypt\\ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0
@estesp estesp added the platform/windows Windows label Sep 27, 2022
@estesp
Copy link
Member

estesp commented Sep 27, 2022

Looks like a platform matching issue with your Windows version and the data in the image? @dcantah or @jterry75 ?

@krotz-dieter
Copy link
Author

Why is it working then with the docker daemon ? The container is the same: mcr.microsoft.com/windows/nanoserver:ltsc2022, it should behave the same with containerd also.

@estesp
Copy link
Member

estesp commented Sep 27, 2022

The Docker daemon has its own implementation of the registry client interactions; our platform matching code is probably different because of this. It's possible we have a bug or tighter restriction than Docker engine on the Windows matching; we will have to dig into that and understand why; that's why I copied in our Windows experts :)

@dcantah
Copy link
Member

dcantah commented Sep 27, 2022 •
edited

Yea Phil nailed it, the platform matcher for CRI (k8s entrypoint) matches to the hosts build exactly and will reject anything else. So in this case 20348 != 22000 (win11). Reason for the restriction being hyper-v containers weren't supported until very recently through the CRI plugin here, so pulling any other image wasn't valid as windows used to mandate the host <-> container image build to match. That was actually relaxed recently, so the ltsc2022 image should work fine on any host that's higher in build number, so we should probably put in some logic for this as what you're trying to do is completely valid.

@krotz-dieter
Copy link
Author

Thanks for the info, can you point me to the code of the platform matcher CRI ? Would like to try out myself to see if I can get it running.
Also, do you plan to incorporate such a change for Windows containers ?

@dcantah
Copy link
Member

dcantah commented Oct 5, 2022

@SyngoPredevelopment Sure, https://github.com/containerd/containerd/blob/main/platforms/defaults_windows.go. And yes! Need to work out any edge cases but yes, it'd be a shame to not allow this functionality if it's supported in the platform now

@krotz-dieter
Copy link
Author

OK, many thanks

@kzys kzys changed the title Cannot pull containers from mcr.microsoft.com Cannot pull containers from mcr.microsoft.com due to platform matching Oct 13, 2022
@fuweid
Copy link
Member

fuweid commented Jan 11, 2023

@dcantah @SyngoPredevelopment any update here?

@dcantah
Copy link
Member

dcantah commented Jan 11, 2023

Hey, no updates on my end but #7856 looks to address platform matching discrepancies here a bit

@krotz-dieter
Copy link
Author

We solved it in another way, we add in the manifest the supported OS versions.

@weipah
Copy link

weipah commented Mar 17, 2023 •
edited

Solution is cooking there, I guess: #8137

Dirty workaround is to match the os.version in manifest file to your client os version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
exp/intermediate kind/bug platform/windows Windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@kzys @estesp @weipah @fuweid @dcantah @krotz-dieter