Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to use containerd with crictl on Windows Server 2022 #210

Closed
doggy8088 opened this issue Mar 22, 2022 · 17 comments
Closed

How to use containerd with crictl on Windows Server 2022 #210

doggy8088 opened this issue Mar 22, 2022 · 17 comments
Labels
bug Something isn't working Networking Connectivity and network infrastructure

Comments

@doggy8088
Copy link

doggy8088 commented Mar 22, 2022 •
edited
Loading

Here are the steps I tried to install containerd on Windows Server 2022.

  1. Install Windows Features

    Add-WindowsFeature Containers,Hyper-V,Hyper-V-Tools,Hyper-V-PowerShell -Restart -IncludeManagementTools

  2. Install containerd 1.6.1

    # Download containerd 1.6.1
curl.exe -LO https://github.com/containerd/containerd/releases/download/v1.6.1/containerd-1.6.1-windows-amd64.tar.gz

tar xvf containerd-1.6.1-windows-amd64.tar.gz
mkdir -force "C:\Program Files\containerd"
mv ./bin/* "C:\Program Files\containerd"
Remove-Item bin

. "C:\Program Files\containerd\containerd.exe" config default | Out-File "C:\Program Files\containerd\config.toml" -Encoding ascii

Add-MpPreference -ExclusionProcess "$Env:ProgramFiles\containerd\containerd.exe"

. "$Env:ProgramFiles\containerd\containerd.exe" --register-service

Start-Service containerd

$env:PATH = "C:\Program Files\containerd;" + $env:PATH

  3. Configure container networking

    mkdir -force "C:\Program Files\containerd\cni\bin"
mkdir -force "C:\Program Files\containerd\cni\conf"

    Download windows-container-networking-cni-amd64-v0.2.0.zip file from microsoft/windows-container-networking

    curl.exe -LO https://github.com/microsoft/windows-container-networking/releases/download/v0.2.0/windows-container-networking-cni-amd64-v0.2.0.zip
Expand-Archive windows-container-networking-cni-amd64-v0.2.0.zip -DestinationPath "C:\Program Files\containerd\cni\bin" -Force
Remove-Item windows-container-networking-cni-amd64-v0.2.0.zip

    Creating a nat network

    curl.exe -LO https://raw.githubusercontent.com/microsoft/SDN/master/Kubernetes/windows/hns.psm1
Import-Module ./hns.psm1

$subnet="10.0.0.0/16"
$gateway="10.0.0.1"
New-HNSNetwork -Type NAT -AddressPrefix $subnet -Gateway $gateway -Name "nat"

@"
{
    "cniVersion": "0.2.0",
    "name": "nat",
    "type": "nat",
    "master": "Ethernet",
    "ipam": {
        "subnet": "$subnet",
        "routes": [
            {
                "gateway": "$gateway"
            }
        ]
    },
    "capabilities": {
        "portMappings": true,
        "dns": true
    }
}
"@ | Set-Content "C:\Program Files\containerd\cni\conf\0-containerd-nat.conf" -Force

Running a container using ctr

  1. Check Windows version

    cmd /c ver

    Here is my Windows Server 2022 version: 10.0.20348.587

    Microsoft Windows [Version 10.0.20348.587]

  2. Pull mcr.microsoft.com/windows/nanoserver:ltsc2022 image

    ctr.exe image pull mcr.microsoft.com/windows/nanoserver:ltsc2022

  3. Running a container

    ctr.exe run --rm  mcr.microsoft.com/windows/nanoserver:ltsc2022 test cmd /c echo hello
    hello

    Using ctr.exe is all okay. No problem at all.

Running a Pod and Container using crictl

  1. Install crictl tool

    curl.exe -LO https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.23.0/crictl-v1.23.0-windows-amd64.tar.gz
tar xvf crictl-v1.23.0-windows-amd64.tar.gz
mv crictl.exe "C:\Program Files\containerd"

  2. Configure crictl config

    mkdir -Force "$home\.crictl"

@"
runtime-endpoint: npipe://./pipe/containerd-containerd
image-endpoint: npipe://./pipe/containerd-containerd
timeout: 10
#debug: true
"@ | Set-Content "$home\.crictl\crictl.yaml" -Force

crictl.exe info

  3. Pull Pause container image (k8s.gcr.io/pause:3.6)

    crictl pull k8s.gcr.io/pause:3.6

  4. Creating a sandbox / Pod

    @"
{
    "metadata": {
        "name": "nginx-sandbox",
        "namespace": "default",
        "attempt": 1,
        "uid": "hdishd83djaidwnduwk28bcsb"
    },
    "log_directory": "/tmp",
    "linux": {
    }
}
"@ | Set-Content "pod-config.json" -Force

crictl runp .\pod-config.json

    I stucked here. ( IP address is either invalid or not part of any configured subnet(s). )

    time="2022-03-23T01:19:40+08:00" level=fatal msg="run pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "035006a1d4b4fd6ab8244f4fef527fc5117b6b37188b61e321f37538f8f01b78": plugin type="nat" name="nat" failed (add): error creating endpoint hcnCreateEndpoint failed in Win32: IP address is either invalid or not part of any configured subnet(s). (0x803b001e) {"Success":false,"Error":"IP address is either invalid or not part of any configured subnet(s). ","ErrorCode":2151350302} : endpoint config &{ 035006a1d4b4fd6ab8244f4fef527fc5117b6b37188b61e321f37538f8f01b78_nat 94530bee-c40a-4f25-9caf-19bbe91e748f [] [{ 0}] { [] [] []} [{10.0.0.1 0.0.0.0/0 0}] 0 {2 0}}"

    I don't know how to fix the error!
@doggy8088 doggy8088 added the question Further information is requested label Mar 22, 2022
@ghost ghost added the triage New and needs attention label Mar 22, 2022
@daschott
Copy link

Have you also created a NAT network before trying to start containers with NAT networking?
Import HNS module: https://www.powershellgallery.com/packages/HNS/0.2.4
New-HnsNetwork -Type NAT -Name nat

@doggy8088
Copy link
Author

@daschott Please see my Creating a nat network section in my original post. I did and it created.

Get-HnsNetwork
ActivityId             : 58261574-A434-494B-BF39-A698343E225D
AdditionalParams       :
CurrentEndpointCount   : 0
Extensions             : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering Platform},
                         @{Id=F74F241B-440F-4433-BB28-00F89EAD20D8; IsEnabled=False; Name=Microsoft Azure VFP Switch Extension},
                         @{Id=430BDADD-BAB0-41AB-A369-94B67FA5BE0A; IsEnabled=True; Name=Microsoft NDIS Capture}}
Flags                  : 8
Health                 : @{LastErrorCode=0; LastUpdateTime=132924426122016395}
ID                     : 94530BEE-C40A-4F25-9CAF-19BBE91E748F
IPv6                   : False
LayeredOn              : 490CA4AE-EB76-4F4D-831C-F6580C4934A1
MacPools               : {@{EndMacAddress=00-15-5D-5F-CF-FF; StartMacAddress=00-15-5D-5F-C0-00}}
MaxConcurrentEndpoints : 0
Name                   : nat
NatName                : NAT9EB86A6C-8942-4B54-8F11-135F3849E2D5
Policies               : {@{Type=VLAN; VLAN=1}}
State                  : 1
Subnets                : {@{AdditionalParams=; AddressPrefix=10.0.0.0/16; Flags=0; GatewayAddress=10.0.0.1; Health=;
                         ID=AB6315D3-D415-4C69-9C2C-0E07E5995146; IpSubnets=System.Object[]; ObjectType=5; Policies=System.Object[]; State=0}}
TotalEndpoints         : 0
Type                   : NAT
Version                : 55834574851
Resources              : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=;
                         ID=58261574-A434-494B-BF39-A698343E225D; PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0;
                         parentId=67604C6B-D935-4FC5-8A40-282135C362DA}

@lippertmarkus
Copy link

I experience the same with ctr. Installing on both Windows Server 2019 and 2022 with the same steps:

curl.exe -LO "https://github.com/lippertmarkus/containerd-installer/releases/download/v0.0.3/containerd-installer.exe"
.\containerd-installer.exe --debug  --containerd-version "1.6.1"
[Environment]::SetEnvironmentVariable("Path", "$($env:path);C:\Program Files\containerd", [System.EnvironmentVariableTarget]::Machine)
$env:Path = [System.Environment]::GetEnvironmentVariable("Path","Machine")

Afterwards running a container on Windows Server 2019 works:

ctr i pull mcr.microsoft.com/windows/nanoserver:1809
ctr run --cni -rm mcr.microsoft.com/windows/nanoserver:1809 test curl.exe https://example.org

{"level":"debug","msg":"[cni-net] Plugin wcn-net version .","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"[net] Network interface: {Index:4 MTU:1500 Name:Ethernet HardwareAddr:00:22:48:5c:62:0f Flags:up|broadcast|multicast} with IP addresses: [fe80::e4be:a00:f1d5:266a/64 10.0.0.6/24]","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"[net] Network interface: {Index:1 MTU:-1 Name:Loopback Pseudo-Interface 1 HardwareAddr: Flags:up|loopback|multicast} with IP addresses: [::1/128 127.0.0.1/8]","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"[net] Network interface: {Index:13 MTU:1500 Name:vEthernet (nat) HardwareAddr:00:15:5d:bd:ee:5f Flags:up|broadcast|multicast} with IP addresses: [fe80::a14f:e9fd:1b46:c580/64 172.24.64.1/20]","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"[cni-net] Plugin started.","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"[cni-net] Processing ADD command with args {ContainerID:default-test Netns:ce201286-3dd9-4974-a8e4-108d3b104548 IfName:eth0 Args: Path:/opt/cni/bin}.","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"[cni-net] Read network configuration \u0026{CniVersion:0.2.0 Name:nat Type:nat Ipam:{Type: Environment: AddrSpace: Subnet:172.24.64.0/20 Address: QueryInterval: Routes:[{Dst:{IP:\u003cnil\u003e Mask:\u003cnil\u003e} GW:\u003cnil\u003e}]} DNS:{Nameservers:[] Domain: Search:[] Options:[]} OptionalFlags:{LocalRoutePortMapping:false AllowAclPortMapping:false} RuntimeConfig:{PortMappings:[] DNS:{Servers:[] Searches:[] Options:[]}} AdditionalArgs:[]}.","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"Parsing port mappings from []","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"[cni-net] Found network 407bddba-9e12-4191-b584-df13a6380a5d with subnet [{{172.24.64.0 fffff000} 172.24.64.1 []}].","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"[cni-net] Creating a new Endpoint","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"hcn::HostComputeEndpoint::Create id=","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"hcn::HostComputeEndpoint::Create JSON: {\"Name\":\"default-test_nat\",\"HostComputeNetwork\":\"407bddba-9e12-4191-b584-df13a6380a5d\",\"IpConfigurations\":[{}],\"Dns\":{},\"Routes\":[{\"NextHop\":\"172.24.64.1\",\"DestinationPrefix\":\"0.0.0.0/0\"}],\"SchemaVersion\":{\"Major\":2,\"Minor\":0}}","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"hcn::HostComputeEndpoint::AddNamespaceEndpoint id=13615a0d-5346-4ab7-b13f-87a38fd96ca0","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"hcn::HostComputeNamespace::ModifyNamespaceSettings id=ce201286-3dd9-4974-a8e4-108d3b104548","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"[cni-net] result: IP4:{IP:{IP:172.24.68.178 Mask:fffff000} Gateway:172.24.64.1 Routes:[]}, DNS:{Nameservers:[] Domain: Search:[] Options:[]}","time":"2022-03-22T18:04:26Z"}
{"level":"debug","msg":"[cni-net] Plugin stopped.","time":"2022-03-22T18:04:26Z"}


  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0   1653      0 --:--:-- --:--:-- --:--:--  1656

{"level":"debug","msg":"[cni-net] Plugin wcn-net version .","time":"2022-03-22T18:04:29Z"}
{"level":"debug","msg":"[net] Network interface: {Index:4 MTU:1500 Name:Ethernet HardwareAddr:00:22:48:5c:62:0f Flags:up|broadcast|multicast} with IP addresses: [fe80::e4be:a00:f1d5:266a/64 10.0.0.6/24]","time":"2022-03-22T18:04:29Z"}
{"level":"debug","msg":"[net] Network interface: {Index:1 MTU:-1 Name:Loopback Pseudo-Interface 1 HardwareAddr: Flags:up|loopback|multicast} with IP addresses: [::1/128 127.0.0.1/8]","time":"2022-03-22T18:04:29Z"}
{"level":"debug","msg":"[net] Network interface: {Index:13 MTU:1500 Name:vEthernet (nat) HardwareAddr:00:15:5d:bd:ee:5f Flags:up|broadcast|multicast} with IP addresses: [fe80::a14f:e9fd:1b46:c580/64 172.24.64.1/20]","time":"2022-03-22T18:04:29Z"}
{"level":"debug","msg":"[cni-net] Plugin started.","time":"2022-03-22T18:04:29Z"}
{"level":"debug","msg":"[cni-net] Processing DEL command with args {ContainerID:default-test Netns: IfName:eth0 Args: Path:/opt/cni/bin}","time":"2022-03-22T18:04:29Z"}
{"level":"debug","msg":"[cni-net] Read network configuration \u0026{CniVersion:0.2.0 Name:nat Type:nat Ipam:{Type: Environment: AddrSpace: Subnet:172.24.64.0/20 Address: QueryInterval: Routes:[{Dst:{IP:\u003cnil\u003e Mask:\u003cnil\u003e} GW:\u003cnil\u003e}]} DNS:{Nameservers:[] Domain: Search:[] Options:[]} OptionalFlags:{LocalRoutePortMapping:false AllowAclPortMapping:false} RuntimeConfig:{PortMappings:[] DNS:{Servers:[] Searches:[] Options:[]}} AdditionalArgs:[]}.","time":"2022-03-22T18:04:29Z"}
{"level":"debug","msg":"Parsing port mappings from []","time":"2022-03-22T18:04:29Z"}
{"level":"debug","msg":"hcn::HostComputeNamespace::RemoveNamespaceEndpoint id=13615a0d-5346-4ab7-b13f-87a38fd96ca0","time":"2022-03-22T18:04:29Z"}
{"level":"debug","msg":"hcn::HostComputeNamespace::ModifyNamespaceSettings id=ce201286-3dd9-4974-a8e4-108d3b104548","time":"2022-03-22T18:04:29Z"}
{"level":"debug","msg":"hcn::HostComputeEndpoint::Delete id=13615a0d-5346-4ab7-b13f-87a38fd96ca0","time":"2022-03-22T18:04:29Z"}
{"level":"debug","msg":"[cni-net] DEL succeeded.","time":"2022-03-22T18:04:29Z"}
{"level":"debug","msg":"[cni-net] Plugin stopped.","time":"2022-03-22T18:04:29Z"}

And on Windows Server 2022 it doesn't:

ctr i pull mcr.microsoft.com/windows/nanoserver:ltsc2022
ctr run --cni -rm mcr.microsoft.com/windows/nanoserver:ltsc2022 test curl.exe https://example.org

{"level":"debug","msg":"[cni-net] Plugin wcn-net version .","time":"2022-03-22T18:31:07Z"}
{"level":"debug","msg":"[net] Network interface: {Index:5 MTU:1500 Name:Ethernet HardwareAddr:60:45:bd:8b:6d:c2 Flags:up|broadcast|multicast} with IP addresses: [fe80::31b3:7fb1:5fe5:36ed/64 10.0.0.4/24]","time":"2022-03-22T18:31:07Z"}
{"level":"debug","msg":"[net] Network interface: {Index:1 MTU:-1 Name:Loopback Pseudo-Interface 1 HardwareAddr: Flags:up|loopback|multicast} with IP addresses: [::1/128 127.0.0.1/8]","time":"2022-03-22T18:31:07Z"}
{"level":"debug","msg":"[net] Network interface: {Index:13 MTU:1500 Name:vEthernet (nat) HardwareAddr:00:15:5d:7d:96:13 Flags:up|broadcast|multicast} with IP addresses: [fe80::f542:9c67:7f4:d2fc/64 172.20.160.1/20]","time":"2022-03-22T18:31:07Z"}
{"level":"debug","msg":"[cni-net] Plugin started.","time":"2022-03-22T18:31:07Z"}
{"level":"debug","msg":"[cni-net] Processing DEL command with args {ContainerID:default-test Netns: IfName:eth0 Args: Path:/opt/cni/bin}","time":"2022-03-22T18:31:07Z"}
{"level":"debug","msg":"[cni-net] Read network configuration \u0026{CniVersion:0.2.0 Name:nat Type:nat Ipam:{Type: Environment: AddrSpace: Subnet:172.20.160.0/20 Address: QueryInterval: Routes:[{Dst:{IP:\u003cnil\u003e Mask:\u003cnil\u003e} GW:\u003cnil\u003e}]} DNS:{Nameservers:[] Domain: Search:[] Options:[]} OptionalFlags:{LocalRoutePortMapping:false AllowAclPortMapping:false} RuntimeConfig:{PortMappings:[] DNS:{Servers:[] Searches:[] Options:[]}} AdditionalArgs:[]}.","time":"2022-03-22T18:31:07Z"}
{"level":"debug","msg":"Parsing port mappings from []","time":"2022-03-22T18:31:07Z"}
{"level":"debug","msg":"[cni-net] Endpoint was not found error, err:Endpoint Id  not found","time":"2022-03-22T18:31:07Z"}
{"level":"debug","msg":"[cni-net] Plugin stopped.","time":"2022-03-22T18:31:07Z"}
ctr: plugin type="nat" name="nat" failed (add): error creating endpoint hcnCreateEndpoint failed in Win32: IP address is either invalid or not part of any configured subnet(s). (0x803b001e) {"Success":false,"Error":"IP address is either invalid or not part of any configured subnet(s). ","ErrorCode":2151350302} : endpoint config &{ default-test_nat d41bb839-10a8-408c-b4c3-3ffe81e53845  [] [{ 0}] { [] [] []} [{172.20.160.1 0.0.0.0/0 0}]  0 {2 0}}

Without CNI it's also working on Windows Server 2022:

ctr run -rm mcr.microsoft.com/windows/nanoserver:ltsc2022 test curl.exe https://example.org

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: example.org

So I think the issue is rather somewhere in the NAT CNI plugin https://github.com/microsoft/windows-container-networking

@daschott
Copy link

My apologies @doggy8088 I missed that part. I am also able to reproduce this issue using Windows CNI plugins release v0.2.0.

There is a change missing because the binaries in v0.2.0 release are outdated. There needs to be a new CNI release made that includes needed changes that correct this behavior for Windows Server 2022 support.
cc @Keith-Mange

In the meantime, can you try to build the CNI plugins yourself? Alternatively, I can also share the compiled binaries with you, just let me know.

@lippertmarkus
Copy link

I can confirm it works when compiling the CNI plugins by myself.

@cwilhit cwilhit added bug Something isn't working Networking Connectivity and network infrastructure and removed question Further information is requested triage New and needs attention labels Mar 22, 2022
@doggy8088
Copy link
Author

@daschott
After re-compile nat CNI plugins, it works now. But...

  1. Creating a sandbox / Pod

    @"
{
    "metadata": {
        "name": "pause-sandbox",
        "namespace": "default",
        "attempt": 1,
        "uid": "hdishd83djaidwnduwk28bcsb"
    },
    "log_directory": "/tmp",
    "linux": {
    }
}
"@ | Set-Content "pod-config.json" -Force

$POD_ID=(crictl runp .\pod-config.json)

    It works now.

  2. Creating container

    @"
{
  "metadata": {
      "name": "k8s.gcr.io/pause:3.6"
  },
  "image":{
      "image": "k8s.gcr.io/pause:3.6"
  },
  "command": [
      ""
  ],
  "log_path":"pause.0.log",
  "linux": {
  }
}
"@ | Set-Content "container-config.json" -Force

$CONTAINER_ID=(crictl create $POD_ID .\container-config.json .\pod-config.json)

    It works here too.

  3. Start the container

    crictl start $CONTAINER_ID

    time="2022-03-23T09:22:30+08:00" level=fatal msg="starting the container "fd178939d36a209bf894bfcee8664fb25a7db4ddf01380e242fe12ad85f352b0": rpc error: code = Unknown desc = failed to create containerd task: failed to create container loggers: failed to create and open log file: The system cannot find the path specified."

    I stucked here again.

@daschott
Copy link

daschott commented Mar 23, 2022 •
edited
Loading

Thank you for confirming. At this point, I think this is a different issue. To cover the original CNI issue, could you file a new issue in https://github.com/microsoft/windows-container-networking/ requesting a new CNI release?

To cover the next issue, @cwilhit could you add the relevant CRI expects here to investigate? The CNI issue should be resolved with updated plugins @doggy8088 is currently using.

@lippertmarkus
Copy link

@doggy8088 you specified a log_directory in your pod config but that directory doesn't exist. You need to create it:

mkdir C:\tmp

@doggy8088
Copy link
Author

@lippertmarkus After I created the C:\tmp folder, the log file pause.0.log is generated after I started the container. Thanks!

But I still stuck at this issue:

time="2022-03-23T19:14:32+08:00" level=fatal msg="starting the container \"34234d60deb6babf29594f257e5bb7b67f4aadbf2a8b742a66198086c2c57bf0\": rpc error: code = Unknown desc = failed to start containerd task \"34234d60deb6babf29594f257e5bb7b67f4aadbf2a8b742a66198086c2c57bf0\": hcs::System::CreateProcess 34234d60deb6babf29594f257e5bb7b67f4aadbf2a8b742a66198086c2c57bf0: The parameter is incorrect.: unknown"

Do you have any idea what going on?

@doggy8088 doggy8088 mentioned this issue Mar 23, 2022
Closed
@lippertmarkus
Copy link

@doggy8088 your pod/container specs are tailored to Linux. Can you try the ones here (with ltsc2022 image instead): https://lippertmarkus.com/2022/01/22/containerd-ctr-windows/#running-containers-with-crictl

Those did work for me.

@doggy8088
Copy link
Author

doggy8088 commented Mar 23, 2022 •
edited
Loading

@lippertmarkus It works for me too. Thank you so much! 😊

All the detailed steps are been posted in my blog. Sorry it's Traditional Chinese only.

https://blog.miniasp.com/post/2022/03/23/Windows-Containers-in-containerd-using-Windows-Server-2022

@jonaskello jonaskello mentioned this issue May 22, 2022
Closed
@kkbruce
Copy link

kkbruce commented Jul 18, 2022

Hi All

PS C:\> containerd.exe -v
containerd github.com/containerd/containerd v1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1

I downloaded https://github.com/microsoft/windows-container-networking/releases/tag/v0.3.0 and resolved the path microsoft/windows-container-networking#70 , --cni is work. ( Thanks, @lippertmarkus )

PS C:\> ctr run --cni --rm mcr.microsoft.com/windows/nanoserver:ltsc2022 test ipconfig

Windows IP Configuration


Ethernet adapter vEthernet (default-test_nat):

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::c59d:f941:c0c6:74cb%17
   IPv4 Address. . . . . . . . . . . : 172.20.88.200
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 172.20.80.1

setup crictl from @doggy8088 blog:

PS C:\> crictl -v
crictl version v1.24.1

crictl info

PS C:\> crictl info
{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "C:\\Program Files\\containerd\\cni\\bin"
    ],
    "PluginConfDir": "C:\\Program Files\\containerd\\cni\\conf",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "nat",
          "CNIVersion": "0.2.0",
          "Plugins": [
            {
              "Network": {
                "cniVersion": "0.2.0",
                "name": "nat",
                "type": "nat",
                "capabilities": {
                  "dns": true,
                  "portMappings": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"dns\":true,\"portMappings\":true},\"cniVersion\":\"0.2.0\",\"ipam\":{\"routes\":[{\"gateway\":\"172.20.80.1\"}],\"subnet\":\"172.20.80.0/20\"},\"master\":\"Ethernet\",\"name\":\"nat\",\"type\":\"nat\"}"
            }
          ],
          "Source": "{\"cniVersion\":\"0.2.0\",\"name\":\"nat\",\"plugins\":[{\"capabilities\":{\"dns\":true,\"portMappings\":true},\"cniVersion\":\"0.2.0\",\"ipam\":{\"routes\":[{\"gateway\":\"172.20.80.1\"}],\"subnet\":\"172.20.80.0/20\"},\"master\":\"Ethernet\",\"name\":\"nat\",\"type\":\"nat\"}]}"
        },
        "IFName": "eth0"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "windows",
      "defaultRuntimeName": "runhcs-wcow-process",
      "defaultRuntime": {
        "runtimeType": "",
        "runtimePath": "",
        "runtimeEngine": "",
        "PodAnnotations": [],
        "ContainerAnnotations": [],
        "runtimeRoot": "",
        "options": {},
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimePath": "",
        "runtimeEngine": "",
        "PodAnnotations": [],
        "ContainerAnnotations": [],
        "runtimeRoot": "",
        "options": {},
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": "",
        "cniConfDir": "",
        "cniMaxConfNum": 0
      },
      "runtimes": {
        "runhcs-wcow-process": {
          "runtimeType": "io.containerd.runhcs.v1",
          "runtimePath": "",
          "runtimeEngine": "",
          "PodAnnotations": [],
          "ContainerAnnotations": [],
          "runtimeRoot": "",
          "options": {},
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": "",
          "cniConfDir": "",
          "cniMaxConfNum": 0
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": false,
      "discardUnpackedLayers": false,
      "ignoreRdtNotEnabledErrors": false
    },
    "cni": {
      "binDir": "C:\\Program Files\\containerd\\cni\\bin",
      "confDir": "C:\\Program Files\\containerd\\cni\\conf",
      "maxConfNum": 1,
      "confTemplate": "",
      "ipPref": ""
    },
    "registry": {
      "configPath": "",
      "mirrors": {},
      "configs": {},
      "auths": {},
      "headers": {
        "User-Agent": [
          "containerd/v1.6.6"
        ]
      }
    },
    "imageDecryption": {
      "keyModel": "node"
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 0,
    "sandboxImage": "k8s.gcr.io/pause:3.6",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": false,
    "disableHugetlbController": false,
    "device_ownership_from_security_context": false,
    "ignoreImageDefinedVolumes": false,
    "netnsMountsUnderStateDir": false,
    "enableUnprivilegedPorts": false,
    "enableUnprivilegedICMP": false,
    "containerdRootDir": "C:\\ProgramData\\containerd\\root",
    "containerdEndpoint": "\\\\.\\pipe\\containerd-containerd",
    "rootDir": "C:\\ProgramData\\containerd\\root\\io.containerd.grpc.v1.cri",
    "stateDir": "C:\\ProgramData\\containerd\\state\\io.containerd.grpc.v1.cri"
  },
  "golang": "go1.17.11",
  "lastCNILoadStatus": "OK",
  "lastCNILoadStatus.default": "OK"
}

PS C:\> crictl pull hello-world:nanoserver-ltsc2022
Image is up to date for sha256:c764b8ebd864571db3689163cd960f3631acafa81bb1f0d81dc629cc738994a3

Until the create Pod error:

PS C:\PodConf> $POD_ID=(crictl runp .\pod-config.json)
E0718 08:13:56.836281    3080 remote_runtime.go:201] "RunPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
time="2022-07-18T08:13:56Z" level=fatal msg="run pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded"

I repeated the test with Will and Markus's blog posts, but still the same error.

@kkbruce
Copy link

kkbruce commented Jul 18, 2022

Add some information

C:\PodConf>cmd /c ver

Microsoft Windows [Version 10.0.20348.825]

PS C:\PodConf> crictl -D run .\container.json .\pod.json
time="2022-07-18T10:02:03Z" level=debug msg="get image connection"
time="2022-07-18T10:02:03Z" level=debug msg="get runtime connection"
time="2022-07-18T10:02:03Z" level=debug msg="RunPodSandboxRequest: &RunPodSandboxRequest{Config:&PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:mycont-sandbox,Uid:hdishd83djaidwnduwk28basb,Namespace:default,Attempt:0,},Hostname:,LogDirectory:,DnsConfig:nil,PortMappings:[]*PortMapping{},Labels:map[string]string{},Annotations:map[string]string{},Linux:nil,Windows:nil,},RuntimeHandler:,}"
E0718 10:02:23.723036    1228 remote_runtime.go:201] "RunPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
time="2022-07-18T10:02:23Z" level=debug msg="RunPodSandboxResponse: "
time="2022-07-18T10:02:23Z" level=fatal msg="running container: run pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded"

@doggy8088 doggy8088 reopened this Jul 18, 2022
@lippertmarkus
Copy link

seems like you didn't set the runtime endpoint correctly:

$env:CONTAINER_RUNTIME_ENDPOINT="npipe:////./pipe/containerd-containerd"

@kkbruce
Copy link

kkbruce commented Jul 19, 2022

Script from your blog:

PS C:\PodConf> $env:CONTAINER_RUNTIME_ENDPOINT="npipe:////./pipe/containerd-containerd"
PS C:\PodConf> crictl pull mcr.microsoft.com/windows/nanoserver:ltsc2022
Image is up to date for sha256:e9a98dadcd0872c19ade490b9ed424af502bb022974bbf9029d6347e0c867652
PS C:\PodConf> cat .\pod.json
{
    "metadata": {
      "name": "mycont-sandbox",
      "namespace": "default",
      "uid": "hdishd83djaidwnduwk28basb"
    }
}
PS C:\PodConf> cat .\container.json
{
    "metadata": {
        "name": "mycont"
    },
    "image":{
        "image": "mcr.microsoft.com/windows/nanoserver:ltsc2022"
    },
    "command": ["cmd", "/c", "ping -t 127.0.0.1"]
  }
PS C:\PodConf> $POD_ID=(crictl runp .\pod.json)
E0719 02:34:28.959978    4100 remote_runtime.go:201] "RunPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
time="2022-07-19T02:34:28Z" level=fatal msg="run pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded"

screen shot

image

@kkbruce
Copy link

kkbruce commented Jul 21, 2022 •
edited
Loading

I know what's causing the problem.

When creating Pod for the first time, crictl will download the k8s.gcr.io/pause:3.6 image, if the download time is over crictl command timeout option will find the above DeadlineExceeded error.

@lippertmarkus, thanks for your help.

@ghost
Copy link

ghost commented Feb 17, 2023

After running all windows configuration here, we got this error , on kube-proxy for windows, I am using flannel

error while dialing open \\.\pipe\rancher_wins: The system cannot find the file specified."

can someonw help here please

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working Networking Connectivity and network infrastructure
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@doggy8088 @lippertmarkus @kkbruce @cwilhit @daschott