Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

starting container fails with: The system cannot find the file specified.: unknown #6300

Closed
ckrueger1979 opened this issue Nov 30, 2021 · 5 comments
Closed

starting container fails with: The system cannot find the file specified.: unknown #6300

ckrueger1979 opened this issue Nov 30, 2021 · 5 comments
Labels
kind/bug platform/windows Windows

Comments

@ckrueger1979
Copy link

ckrueger1979 commented Nov 30, 2021 •
edited
Loading

Description

Hi,

I'm trying to run a container with containerd on windows server 2019.

I'm roughly using these guides:
https://blog.devgenius.io/gentle-containerd-on-windows-guide-for-you-50d4a46daccf
(link seems down right now, via archive.org
https://web.archive.org/web/20211007164324/https://blog.devgenius.io/gentle-containerd-on-windows-guide-for-you-50d4a46daccf?gi=1d353ea4a782
)
+
https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops

The container works perfectly fine with docker runtime but with containerd it's not starting

.\crictl.exe start $ContainerId

time="2021-11-30T09:28:59Z" level=fatal msg="starting the container \"544494bda2d94405c85025aadab41c328224baa1edd2055afb0fee163ae3c50d\": rpc error: code = Unknown desc = failed to start containerd task \"544494bda2d94405c85025aadab41c328224baa1edd2055afb0fee163ae3c50d\": hcs::System::CreateProcess 544494bda2d94405c85025aadab41c328224baa1edd2055afb0fee163ae3c50d: The system cannot find the file specified.: unknown"

Thoughts?

greetings
Carsten

Steps to reproduce the issue

  1. Install containerd according the guide above
  2. build the windows container according to the other guide
  3. try to run it

Describe the results you received and expected

.\crictl.exe start $ContainerId

time="2021-11-30T09:28:59Z" level=fatal msg="starting the container \"544494bda2d94405c85025aadab41c328224baa1edd2055afb0fee163ae3c50d\": rpc error: code = Unknown desc = failed to start containerd task \"544494bda2d94405c85025aadab41c328224baa1edd2055afb0fee163ae3c50d\": hcs::System::CreateProcess 544494bda2d94405c85025aadab41c328224baa1edd2055afb0fee163ae3c50d: The system cannot find the file specified.: unknown"

C:\ProgramData\containerd\root\panic.log

time="2021-11-30T08:28:03.003272900Z" level=error msg=Span duration=266.758ms endTime="2021-11-30 08:28:03.2700309 +0000 GMT m=+22.925468601" error="The system cannot find the file specified." name=HcsCreateProcess parentSpanID=0000000000000000 processParameters="{\"CommandLine\":\"\\\"cmd /S /C powershell .\\\\start.ps1\\\"\",\"WorkingDirectory\":\"C:\\\\azp\",\"CreateStdOutPipe\":true,\"CreateStdErrPipe\":true}" result="{\"Error\":-2147024894,\"ErrorMessage\":\"The system cannot find the file specified.\"}" spanID=26092effa36b8370 startTime="2021-11-30 08:28:03.0032729 +0000 GMT m=+22.658710601" traceID=8a2996a048fb2b9e72ad7101a0f17504
time="2021-11-30T08:28:03.270059900Z" level=info msg=Span duration=6.7854ms endTime="2021-11-30 08:28:03.2768453 +0000 GMT m=+22.932254001" name=HcsTerminateComputeSystem options= parentSpanID=b989b9f9cba43232 spanID=6fefde99356060e3 startTime="2021-11-30 08:28:03.2700599 +0000 GMT m=+22.925468601" traceID=3449be97f5b7238adfadb893d56f098f
time="2021-11-30T08:28:03.288492600Z" level=info msg=Span duration=0s endTime="2021-11-30 08:28:03.2884926 +0000 GMT m=+22.943899301" name=HcsUnregisterComputeSystemCallback parentSpanID=b8d58f34c7543d56 spanID=01bc40cf58491ac9 startTime="2021-11-30 08:28:03.2884926 +0000 GMT m=+22.943899301" traceID=b5f4848f18b77183678e2a3b596ba1ae
time="2021-11-30T08:28:01.294326900Z" level=error msg=Span cid=353e8f6be4777f9bb241a480ce8afb446847111b5c47857cb16b9a6df4749b28 duration=1.9961281s endTime="2021-11-30 08:28:03.290455 +0000 GMT m=+22.946087901" error="hcs::System::waitBackground 353e8f6be4777f9bb241a480ce8afb446847111b5c47857cb16b9a6df4749b28: hcsshim: the handle generating this notification has been closed" name="hcs::System::waitBackground" parentSpanID=0000000000000000 spanID=4c0a45f4f3dde1f3 startTime="2021-11-30 08:28:01.2943269 +0000 GMT m=+20.949959801" traceID=5b1ce195895ae5db3a90b3e2d1264908
time="2021-11-30T08:28:03.290168600Z" level=info msg=Span duration=2.8439ms endTime="2021-11-30 08:28:03.2930125 +0000 GMT m=+22.948419001" name=HcsCloseComputeSystem parentSpanID=b8d58f34c7543d56 spanID=4aa2f169ea3df73b startTime="2021-11-30 08:28:03.2901686 +0000 GMT m=+22.945575101" traceID=b5f4848f18b77183678e2a3b596ba1ae
time="2021-11-30T08:28:03.288492600Z" level=info msg=Span cid=353e8f6be4777f9bb241a480ce8afb446847111b5c47857cb16b9a6df4749b28 duration=5.0973ms endTime="2021-11-30 08:28:03.2935899 +0000 GMT m=+22.948996601" name="hcs::System::Close" parentSpanID=0000000000000000 spanID=b8d58f34c7543d56 startTime="2021-11-30 08:28:03.2884926 +0000 GMT m=+22.943899301" traceID=b5f4848f18b77183678e2a3b596ba1ae
time="2021-11-30T08:28:01.296175100Z" level=error msg=Span duration=1.9971888s eid= endTime="2021-11-30 08:28:03.2933639 +0000 GMT m=+22.948996601" error="hcs::System::CreateProcess 353e8f6be4777f9bb241a480ce8afb446847111b5c47857cb16b9a6df4749b28: The system cannot find the file specified." name=Start parentSpanID=70a3085f3ab055bf pod-id=c6d62533277e0ca34775da2b2c6bc1ddc3ef7f2d0a0a00d02910edc0d0259439 spanID=b989b9f9cba43232 startTime="2021-11-30 08:28:01.2961751 +0000 GMT m=+20.951807801" tid=353e8f6be4777f9bb241a480ce8afb446847111b5c47857cb16b9a6df4749b28 traceID=3449be97f5b7238adfadb893d56f098f
time="2021-11-30T08:28:01.296175100Z" level=info msg=Span duration=1.9971888s endTime="2021-11-30 08:28:03.2933639 +0000 GMT m=+22.948996601" name=containerd.task.v2.Task.Start parentSpanID=0000000000000000 spanID=70a3085f3ab055bf startTime="2021-11-30 08:28:01.2961751 +0000 GMT m=+20.951807801" traceID=3449be97f5b7238adfadb893d56f098f
time="2021-11-30T08:28:01.294879200Z" level=info msg=Span duration=1.9984846s eid=353e8f6be4777f9bb241a480ce8afb446847111b5c47857cb16b9a6df4749b28 endTime="2021-11-30 08:28:03.2933638 +0000 GMT m=+22.948996601" name="hcsExec::waitForContainerExit" parentSpanID=0000000000000000 spanID=ded6a62917c79bd9 startTime="2021-11-30 08:28:01.2948792 +0000 GMT m=+20.950512001" tid=353e8f6be4777f9bb241a480ce8afb446847111b5c47857cb16b9a6df4749b28 traceID=77ca33fd73bb64f07f79eb5baee2981d
time="2021-11-30T08:28:03.293590400Z" level=error msg="failed to wait for container shutdown" error="hcs::System::waitBackground 353e8f6be4777f9bb241a480ce8afb446847111b5c47857cb16b9a6df4749b28: hcsshim: the handle generating this notification has been closed" spanID=27bd57c4a8bb784c traceID=0a928e115bf174227ea3db8b5ca6864b
time="2021-11-30T08:28:03.294649900Z" level=error msg="failed to wait for container terminate" error="hcs::System::waitBackground 353e8f6be4777f9bb241a480ce8afb446847111b5c47857cb16b9a6df4749b28: hcsshim: the handle generating this notification has been closed" spanID=27bd57c4a8bb784c traceID=0a928e115bf174227ea3db8b5ca6864b
time="2021-11-30T08:28:03.308790700Z" level=error msg=Span all=true duration=0s eid= endTime="2021-11-30 08:28:03.3087907 +0000 GMT m=+22.964195201" error="rpc error: code = NotFound desc = exec: '353e8f6be4777f9bb241a480ce8afb446847111b5c47857cb16b9a6df4749b28' in task: '353e8f6be4777f9bb241a480ce8afb446847111b5c47857cb16b9a6df4749b28' not found: not found" name=Kill parentSpanID=6e3bb5d430106b07 pod-id=c6d62533277e0ca34775da2b2c6bc1ddc3ef7f2d0a0a00d02910edc0d0259439 signal=9 spanID=b721666fc204487a startTime="2021-11-30 08:28:03.3087907 +0000 GMT m=+22.964195201" tid=353e8f6be4777f9bb241a480ce8afb446847111b5c47857cb16b9a6df4749b28 traceID=5a1fde402aa4fa0ac8c1c6a4ea4eecab

What version of containerd are you using?

containerd github.com/containerd/containerd v1.5.8 1e5ef94

Any other relevant information

OS: Windows Server 2019

{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "C:\\Program Files\\containerd\\cni\\bin"
    ],
    "PluginConfDir": "C:\\Program Files\\containerd\\cni\\conf",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "nat",
          "CNIVersion": "0.2.0",
          "Plugins": [
            {
              "Network": {
                "cniVersion": "0.2.0",
                "name": "nat",
                "type": "nat",
                "capabilities": {
                  "dns": true,
                  "portMappings": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"dns\":true,\"portMappings\":true},\"cniVersion\":\"0.2.0\",\"ipam\":{\"routes\":[{\"gateway\":\"10.1.0.1\"}],\"subnet\":\"10.1.0.0/16\"},\"master\":\"Ethernet\",\"name\":\"nat\",\"type\":\"nat\"}"
            }
          ],
          "Source": "{\"cniVersion\":\"0.2.0\",\"name\":\"nat\",\"plugins\":[{\"capabilities\":{\"dns\":true,\"portMappings\":true},\"cniVersion\":\"0.2.0\",\"ipam\":{\"routes\":[{\"gateway\":\"10.1.0.1\"}],\"subnet\":\"10.1.0.0/16\"},\"master\":\"Ethernet\",\"name\":\"nat\",\"type\":\"nat\"}]}"
        },
        "IFName": "eth0"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "windows",
      "defaultRuntimeName": "runhcs-wcow-process",
      "defaultRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": [],
        "ContainerAnnotations": [],
        "runtimeRoot": "",
        "options": {},
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": [],
        "ContainerAnnotations": [],
        "runtimeRoot": "",
        "options": {},
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "runtimes": {
        "runhcs-wcow-process": {
          "runtimeType": "io.containerd.runhcs.v1",
          "runtimeEngine": "",
          "PodAnnotations": [],
          "ContainerAnnotations": [],
          "runtimeRoot": "",
          "options": {},
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": ""
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": false,
      "discardUnpackedLayers": false
    },
    "cni": {
      "binDir": "C:\\Program Files\\containerd\\cni\\bin",
      "confDir": "C:\\Program Files\\containerd\\cni\\conf",
      "maxConfNum": 1,
      "confTemplate": ""
    },
    "registry": {
      "configPath": "",
      "mirrors": {},
      "configs": {},
      "auths": {},
      "headers": {}
    },
    "imageDecryption": {
      "keyModel": "node"
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 0,
    "sandboxImage": "k8s.gcr.io/pause:3.6",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": false,
    "disableHugetlbController": false,
    "ignoreImageDefinedVolumes": false,
    "netnsMountsUnderStateDir": false,
    "containerdRootDir": "C:\\ProgramData\\containerd\\root",
    "containerdEndpoint": "\\\\.\\pipe\\containerd-containerd",
    "rootDir": "C:\\ProgramData\\containerd\\root\\io.containerd.grpc.v1.cri",
    "stateDir": "C:\\ProgramData\\containerd\\state\\io.containerd.grpc.v1.cri"
  },
  "golang": "go1.16.10",
  "lastCNILoadStatus": "OK"
}

Show configuration if it is related to CRI plugin.

disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "C:\\ProgramData\\containerd\\root"
state = "C:\\ProgramData\\containerd\\state"
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "\\\\.\\pipe\\containerd-containerd"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = false
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "k8s.gcr.io/pause:3.6"
    selinux_category_range = 0
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = false
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "C:\\Program Files\\containerd\\cni\\bin"
      conf_dir = "C:\\Program Files\\containerd\\cni\\conf"
      conf_template = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runhcs-wcow-process"
      disable_snapshot_annotations = false
      discard_unpacked_layers = false
      no_pivot = false
      snapshotter = "windows"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runhcs-wcow-process]
          base_runtime_spec = ""
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runhcs.v1"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runhcs-wcow-process.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]

    [plugin."io.containerd.grpc.v1.cri".registry.configs."tfs.test.lab".tls]
      insecure_skip_verify = true

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "C:\\ProgramData\\containerd\\root\\opt"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["windows/amd64", "linux/amd64"]

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["windows", "windows-lcow"]

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "C:\\Program Files\\containerd\\ocicrypt\\keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=C:\\Program Files\\containerd\\ocicrypt\\ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "C:\\Program Files\\containerd\\ocicrypt\\keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=C:\\Program Files\\containerd\\ocicrypt\\ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0
@mxpv mxpv added the platform/windows Windows label Nov 30, 2021
@ckrueger1979
Copy link
Author

ckrueger1979 commented Dec 7, 2021 •
edited
Loading

I found the root cause.
The problem is related to escape character "backslash"
https://docs.docker.com/engine/reference/builder/

Wha I'm not really getting is the following. If I run the container with docker there is no problem regarding the escaping, if I run it with containerd there is a problem

@ckrueger1979
Copy link
Author

ckrueger1979 commented Dec 7, 2021 •
edited
Loading

This is not working with containerd (but it's working with docker)

CMD powershell .\start.ps1

this is working with containerd:

CMD ["powershell", ".\\start.ps1"]

Is this is bug in containerd or am I understanding something wrong?

@luthermonson luthermonson mentioned this issue Dec 14, 2021
Merged
@kzys kzys mentioned this issue Jan 31, 2022
Merged
@BrunoJuchli
Copy link

Looks to me like this is a duplicate of #5067

@ckrueger1979
Copy link
Author

@BrunoJuchli

ACK

@MageshSrinivasulu MageshSrinivasulu mentioned this issue Aug 10, 2022
Closed
@kzys
Copy link
Member

kzys commented Oct 19, 2022

Resolving as duplicate. Let's use #5067.

@kzys kzys closed this as completed Oct 19, 2022
@hugobritobh hugobritobh mentioned this issue Apr 27, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug platform/windows Windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kzys @mxpv @BrunoJuchli @ckrueger1979