Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Calico CNI failed with hcnCreateEndpoint failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d) #8465

Closed
wizpresso-steve-cy-fan opened this issue Jan 31, 2024 · 7 comments
Closed

Calico CNI failed with hcnCreateEndpoint failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d) #8465

wizpresso-steve-cy-fan opened this issue Jan 31, 2024 · 7 comments
Labels
kind/support

Comments

@wizpresso-steve-cy-fan
Copy link

wizpresso-steve-cy-fan commented Jan 31, 2024 •
edited
Loading

Expected Behavior

Calico should work on Windows 11 with containerd

Current Behavior

It does not work on Windows 11

Possible Solution

Steps to Reproduce (for bugs)

  1. Install a k0s master
  2. Install a k0s worker on Windows 10/11 (not Windows Server)
  3. Install Calico with VXLAN mode only
  4. Wait until both Linux and Windows sides are healthy
  5. Create a Windows Container pod

Context

Here's the error

time="2024-01-31 14:52:57" level=info msg="I0131 14:52:57.786876    8516 kuberuntime_manager.go:436] \"Retrieved pods from runtime\" all=true" component=kubelet.exe stream=stderr
time="2024-01-31 14:52:57" level=info msg="E0131 14:52:57.936439    8516 remote_runtime.go:193] \"RunPodSandbox from runtime service failed\" err=\"rpc error: code = Unknown desc = failed to setup network for sandbox \\\"9f207b881af6caa158dd2b7251fc1f47b165de0d7395cfda370d66981189a279\\\": plugin type=\\\"calico\\\" name=\\\"Calico\\\" failed (add): failed to create the new HostComputeEndpoint: hcnCreateEndpoint failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d) {\\\"Success\\\":false,\\\"Error\\\":\\\"所提供的原則設定無效或缺少參數。 \\\",\\\"ErrorCode\\\":2151350285}\"" component=kubelet.exe stream=stderr
time="2024-01-31 14:52:57" level=info msg="E0131 14:52:57.936439    8516 kuberuntime_sandbox.go:72] \"Failed to create sandbox for pod\" err=\"rpc error: code = Unknown desc = failed to setup network for sandbox \\\"9f207b881af6caa158dd2b7251fc1f47b165de0d7395cfda370d66981189a279\\\": plugin type=\\\"calico\\\" name=\\\"Calico\\\" failed (add): failed to create the new HostComputeEndpoint: hcnCreateEndpoint failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d) {\\\"Success\\\":false,\\\"Error\\\":\\\"所提供的原則設定無效或缺少參數。 \\\",\\\"ErrorCode\\\":2151350285}\" pod=\"default/win-webserver-5cf6f5dd6f-g6xdp\"" component=kubelet.exe stream=stderr
time="2024-01-31 14:52:57" level=info msg="E0131 14:52:57.936962    8516 kuberuntime_manager.go:1171] \"CreatePodSandbox for pod failed\" err=\"rpc error: code = Unknown desc = failed to setup network for sandbox \\\"9f207b881af6caa158dd2b7251fc1f47b165de0d7395cfda370d66981189a279\\\": plugin type=\\\"calico\\\" name=\\\"Calico\\\" failed (add): failed to create the new HostComputeEndpoint: hcnCreateEndpoint failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d) {\\\"Success\\\":false,\\\"Error\\\":\\\" 所提供的原則設定無效或缺少參數。 \\\",\\\"ErrorCode\\\":2151350285}\" pod=\"default/win-webserver-5cf6f5dd6f-g6xdp\"" component=kubelet.exe stream=stderr
time="2024-01-31 14:52:57" level=info msg="I0131 14:52:57.936962    8516 kubelet.go:1697] \"SyncPod exit\" pod=\"default/win-webserver-5cf6f5dd6f-g6xdp\" podUID=\"2edbcee8-c2e1-48f6-a8f4-2102fa956c03\" isTerminal=false" component=kubelet.exe stream=stderr
time="2024-01-31 14:52:57" level=info msg="E0131 14:52:57.936962    8516 pod_workers.go:1300] \"Error syncing pod, skipping\" err=\"failed to \\\"CreatePodSandbox\\\" for \\\"win-webserver-5cf6f5dd6f-g6xdp_default(2edbcee8-c2e1-48f6-a8f4-2102fa956c03)\\\" with CreatePodSandboxError: \\\"Failed to create sandbox for pod \\\\\\\"win-webserver-5cf6f5dd6f-g6xdp_default(2edbcee8-c2e1-48f6-a8f4-2102fa956c03)\\\\\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\\\\\"9f207b881af6caa158dd2b7251fc1f47b165de0d7395cfda370d66981189a279\\\\\\\": plugin type=\\\\\\\"calico\\\\\\\" name=\\\\\\\"Calico\\\\\\\" failed (add): failed to create the new HostComputeEndpoint: hcnCreateEndpoint failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d) {\\\\\\\"Success\\\\\\\":false,\\\\\\\"Error\\\\\\\":\\\\\\\"所提供的原則設定無效或缺少參數。 \\\\\\\",\\\\\\\"ErrorCode\\\\\\\":2151350285}\\\"\" pod=\"default/win-webserver-5cf6f5dd6f-g6xdp\" podUID=\"2edbcee8-c2e1-48f6-a8f4-2102fa956c03\"" component=kubelet.exe stream=stderr
time="2024-01-31 14:52:57" level=info msg="I0131 14:52:57.936962    8516 event.go:307] \"Event occurred\" object=\"default/win-webserver-5cf6f5dd6f-g6xdp\" fieldPath=\"\" kind=\"Pod\" apiVersion=\"v1\" type=\"Warning\" reason=\"FailedCreatePodSandBox\" message=\"Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox \\\"9f207b881af6caa158dd2b7251fc1f47b165de0d7395cfda370d66981189a279\\\": plugin type=\\\"calico\\\" name=\\\"Calico\\\" failed (add): failed to create the new HostComputeEndpoint: hcnCreateEndpoint failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d) {\\\"Success\\\":false,\\\"Error\\\":\\\"所提供的原則設定無效或缺少參數。 \\\",\\\"ErrorCode\\\":2151350285}\"" component=kubelet.exe stream=stderr

The key error here is:

rpc error: code = Unknown desc = failed to setup network for sandbox \"9f207b881af6caa158dd2b7251fc1f47b165de0d7395cfda370d66981189a279\": plugin type=\"calico\" name=\"Calico\" failed (add): failed to create the new HostComputeEndpoint: hcnCreateEndpoint failed in Win32: The provided policy configuration is invalid or missing parameters. (0x803b000d)

Calico CNI config:

{
  "name": "Calico",
  "windows_use_single_network": true,

  "cniVersion": "0.3.1",
  "type": "calico",
  "mode": "vxlan",

  "vxlan_mac_prefix":  "0E-2A",
  "vxlan_vni": 4096,

  "policy": {
    "type": "k8s"
  },

  "log_level": "info",

  "windows_loopback_DSR": true,

  "capabilities": {"dns": true},

  "DNS":  {
    "Nameservers":  ["10.96.0.10"],
    "Search":  [
      "svc.cluster.local"
    ]
  },

  "nodename_file": "C:\\CalicoWindows\\libs\\calico\\..\\..\\nodename",

  "datastore_type": "kubernetes",

  "etcd_endpoints": "",
  "etcd_key_file": "",
  "etcd_cert_file": "",
  "etcd_ca_cert_file": "",

  "kubernetes": {
    "kubeconfig": "C:\\CalicoWindows\\calico-kube-config"
  },

  "ipam": {
    "type": "calico-ipam",
    "subnet": "usePodCidr"
  },

  "policies":  [
    {
      "Name":  "EndpointPolicy",
      "Value":  {
        "Type":  "OutBoundNAT",
        "ExceptionList":  [
          "10.96.0.0/12"
        ]
      }
    },
    {
      "Name":  "EndpointPolicy",
      "Value":  {
        "Type":  "SDNROUTE",
        "DestinationPrefix":  "10.96.0.0/12",
        "NeedEncap":  true
      }
    }
  ]
}

Also there are some error in Felix on Windows node:

2024-01-31 16:51:17.728 [WARNING][30888] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.244.178.63
2024-01-31 16:51:21.240 [WARNING][30888] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.244.178.0
2024-01-31 16:51:21.240 [WARNING][30888] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.244.178.1
2024-01-31 16:51:21.240 [WARNING][30888] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.244.178.2
2024-01-31 16:51:21.240 [WARNING][30888] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.244.178.63
2024-01-31 16:51:32.720 [WARNING][30888] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.244.178.0
2024-01-31 16:51:32.720 [WARNING][30888] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.244.178.1
2024-01-31 16:51:32.720 [WARNING][30888] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.244.178.2
2024-01-31 16:51:32.720 [WARNING][30888] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.244.178.63
2024-01-31 16:51:36.207 [WARNING][30888] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.244.178.0
2024-01-31 16:51:36.207 [WARNING][30888] felix/l3_route_resolver.go 662: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=10.244.178.1

I cannot ping from Windows to Linux and from Linux to Windows using the pod IPs either.

Your Environment

  • Calico version: 3.26.0
  • Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes
  • Operating System and version: Windows 11 and Linux
  • Link to your project (optional):
Open
@wizpresso-steve-cy-fan
Copy link
Author

Get-HnsPolicyList:

ActivityId         : 11903C90-501F-478F-A5C9-3B2A70BF51C2
AdditionalParams   :
Flags              : 0
Health             : @{LastErrorCode=0; LastUpdateTime=133511593158285187}
HostComputeNetwork : 93C59A4B-F5C0-42AA-BE09-E7480DE1FF3D
ID                 : D680D0E7-185B-46A7-B097-37A8B439F588
IsApplied          : False
Policies           : {@{ExternalPort=55680; InternalPort=55680; IsDSR=True; Protocol=6; SourceVIP=10.244.178.2; Type=ELB; VIPs=System.Object[]}}
References         : {/endpoints/c55b2c86-bd06-400e-bca7-8fb7de11e635}
State              : 2
Version            : 64424509440
Resources          : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=11903C90-501F-478F-A5C9-3B2A70BF51C2;
                     PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

ActivityId         : DE759193-759A-473F-93A7-7B290DD64380
AdditionalParams   :
Flags              : 0
Health             : @{LastErrorCode=0; LastUpdateTime=133511593158484942}
HostComputeNetwork : 93C59A4B-F5C0-42AA-BE09-E7480DE1FF3D
ID                 : F4F6A3DB-F234-49B2-960B-F0C45C25208B
IsApplied          : False
Policies           : {@{ExternalPort=6831; InternalPort=6831; IsDSR=True; Protocol=17; SourceVIP=10.244.178.2; Type=ELB; VIPs=System.Object[]}}
References         : {/endpoints/c55b2c86-bd06-400e-bca7-8fb7de11e635}
State              : 2
Version            : 64424509440
Resources          : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=DE759193-759A-473F-93A7-7B290DD64380;
                     PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

ActivityId         : 8F777234-3BD7-485D-9F39-FF26F8CD3603
AdditionalParams   :
Flags              : 0
Health             : @{LastErrorCode=0; LastUpdateTime=133511593159196171}
HostComputeNetwork : 93C59A4B-F5C0-42AA-BE09-E7480DE1FF3D
ID                 : 3E4A53E6-F0CF-497C-BCA1-2BDD9C369861
IsApplied          : False
Policies           : {@{ExternalPort=6832; InternalPort=6832; IsDSR=True; Protocol=17; SourceVIP=10.244.178.2; Type=ELB; VIPs=System.Object[]}}
References         : {/endpoints/c55b2c86-bd06-400e-bca7-8fb7de11e635}
State              : 2
Version            : 64424509440
Resources          : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=8F777234-3BD7-485D-9F39-FF26F8CD3603;
                     PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

ActivityId         : 557AF077-AAF8-465E-A43D-13B0962A0750
AdditionalParams   :
Flags              : 0
Health             : @{LastErrorCode=0; LastUpdateTime=133511593159311966}
HostComputeNetwork : 93C59A4B-F5C0-42AA-BE09-E7480DE1FF3D
ID                 : 4F12C540-151C-4BD8-9A00-87EDD52B7495
IsApplied          : False
Policies           : {@{ExternalPort=14250; InternalPort=14250; IsDSR=True; Protocol=6; SourceVIP=10.244.178.2; Type=ELB; VIPs=System.Object[]}}
References         : {/endpoints/c55b2c86-bd06-400e-bca7-8fb7de11e635}
State              : 2
Version            : 64424509440
Resources          : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=557AF077-AAF8-465E-A43D-13B0962A0750;
                     PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

ActivityId         : 3FBDE7BB-17CF-45FC-8586-12CE2A670DEF
AdditionalParams   :
Flags              : 0
Health             : @{LastErrorCode=0; LastUpdateTime=133511593244913954}
HostComputeNetwork : 93C59A4B-F5C0-42AA-BE09-E7480DE1FF3D
ID                 : 2551E744-B88C-42A0-9162-9B0652B2F02A
IsApplied          : False
Policies           : {@{ExternalPort=9187; InternalPort=9187; IsDSR=True; Protocol=6; SourceVIP=10.244.178.2; Type=ELB; VIPs=System.Object[]}}
References         : {/endpoints/dd58afad-bddb-4a68-98c7-e0a232331561}
State              : 2
Version            : 64424509440
Resources          : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=3FBDE7BB-17CF-45FC-8586-12CE2A670DEF;
                     PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

ActivityId         : 7688CAE8-55BA-4900-8701-B0BB622F0E46
AdditionalParams   :
Flags              : 0
Health             : @{LastErrorCode=0; LastUpdateTime=133511593254715718}
HostComputeNetwork : 93C59A4B-F5C0-42AA-BE09-E7480DE1FF3D
ID                 : E73D6794-6227-4FE3-A8D6-6C9B96BA5BE8
IsApplied          : False
Policies           : {@{ExternalPort=80; InternalPort=3000; IsDSR=True; Protocol=6; SourceVIP=10.244.178.2; Type=ELB; VIPs=System.Object[]}}
References         : {/endpoints/8f151b3c-3bc0-44b6-add6-aa2bcae68ce0}
State              : 2
Version            : 64424509440
Resources          : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=7688CAE8-55BA-4900-8701-B0BB622F0E46;
                     PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

ActivityId         : 4BF37B22-376E-4963-8D8A-0A3975ACA872
AdditionalParams   :
Flags              : 0
Health             : @{LastErrorCode=0; LastUpdateTime=133511593423306160}
HostComputeNetwork : 93C59A4B-F5C0-42AA-BE09-E7480DE1FF3D
ID                 : 4C554B6B-8D11-4A37-8B2A-9107047CD27A
IsApplied          : False
Policies           : {@{ExternalPort=6379; InternalPort=6379; IsDSR=True; Protocol=6; SourceVIP=10.244.178.2; Type=ELB; VIPs=System.Object[]}}
References         : {/endpoints/e209e477-7e82-4b6b-ae20-43bdec417c00}
State              : 2
Version            : 64424509440
Resources          : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=4BF37B22-376E-4963-8D8A-0A3975ACA872;
                     PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

ActivityId         : 9091FA8A-B03E-4AE1-A7E2-E5409A4143BD
AdditionalParams   :
Flags              : 0
Health             : @{LastErrorCode=0; LastUpdateTime=133511595879785581}
HostComputeNetwork : 93C59A4B-F5C0-42AA-BE09-E7480DE1FF3D
ID                 : EA3900A8-991F-47F5-A17E-8C7EF8232B3B
IsApplied          : False
Policies           : {@{ExternalPort=3100; InternalPort=3100; IsDSR=True; Protocol=6; SourceVIP=10.244.178.2; Type=ELB; VIPs=System.Object[]}}
References         : {/endpoints/4f4f1356-df35-48e7-b479-411760920ba1}
State              : 2
Version            : 64424509440
Resources          : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=9091FA8A-B03E-4AE1-A7E2-E5409A4143BD;
                     PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

ActivityId         : 2CC13A52-9D5F-41ED-A2C2-F83F2EA9F28E
AdditionalParams   :
Flags              : 0
Health             : @{LastErrorCode=0; LastUpdateTime=133511615801134364}
HostComputeNetwork : 93C59A4B-F5C0-42AA-BE09-E7480DE1FF3D
ID                 : 7A59CEAD-B1F2-421B-82C1-AE5E8913EB58
IsApplied          : False
Policies           : {@{ExternalPort=5433; InternalPort=7433; IsDSR=True; Protocol=6; SourceVIP=10.244.178.2; Type=ELB; VIPs=System.Object[]}}
References         : {/endpoints/2d182b92-68d0-4080-bbc6-61593bb075c7}
State              : 2
Version            : 64424509440
Resources          : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=2CC13A52-9D5F-41ED-A2C2-F83F2EA9F28E;
                     PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

@wizpresso-steve-cy-fan
Copy link
Author

Get-HnsNamespace:

ActivityId       : 23FDB4C2-7808-44A2-A92E-E2CF4D508DC0
AdditionalParams :
CompartmentId    : 0
Containers       : {}
Flags            : 0
Health           : @{LastErrorCode=0; LastUpdateTime=133511564334272922}
ID               : 37B9E55C-5EC5-45B6-969F-8585A89DC6E5
IsDefault        : False
Policies         : {}
SchemaVersion    : @{Major=0; Minor=0}
State            : 1
Version          : 64424509440
Resources        : @{AdditionalParams=; AllocationOrder=0; CompartmentOperationTime=0; Flags=0; Health=; ID=23FDB4C2-7808-44A2-A92E-E2CF4D508DC0; PortOperationTime=0; State=1;
                   SwitchOperationTime=0; VfpOperationTime=0}

ActivityId       : 108F5FD4-C87D-4D3C-9419-2683FE237F5C
AdditionalParams :
CompartmentGuid  : B1062982-2B18-4B4F-B3D5-A78DDB9CDD49
CompartmentId    : 1
Containers       : {}
Flags            : 0
Health           : @{LastErrorCode=0; LastUpdateTime=133508076462025898}
ID               : 910F7D92-BA2D-4C3F-98AE-7C0AC590D2DC
IsDefault        : True
Policies         : {}
State            : 1
Version          : 64424509440
Resources        :

ActivityId       : A8FC83A9-FF84-41BD-AEC7-776C5BC51724
AdditionalParams :
CompartmentId    : 0
Containers       : {}
Flags            : 0
Health           : @{LastErrorCode=0; LastUpdateTime=133511565456820649}
ID               : C32F1730-0CAC-4195-9F35-2DF3A55CF4CA
IsDefault        : False
Policies         : {}
SchemaVersion    : @{Major=0; Minor=0}
State            : 1
Version          : 64424509440
Resources        : @{AdditionalParams=; AllocationOrder=0; CompartmentOperationTime=0; Flags=0; Health=; ID=A8FC83A9-FF84-41BD-AEC7-776C5BC51724; PortOperationTime=0; State=1;
                   SwitchOperationTime=0; VfpOperationTime=0}

ActivityId       : 0B5BC1C1-B948-4515-81A6-78E02E6D76FA
AdditionalParams :
CompartmentId    : 0
Containers       : {}
Flags            : 0
Health           : @{LastErrorCode=0; LastUpdateTime=133511563884759372}
ID               : C400F8F0-3184-4A7F-A589-9E978C2B4E52
IsDefault        : False
Policies         : {}
SchemaVersion    : @{Major=0; Minor=0}
State            : 1
Version          : 64424509440
Resources        : @{AdditionalParams=; AllocationOrder=0; CompartmentOperationTime=0; Flags=0; Health=; ID=0B5BC1C1-B948-4515-81A6-78E02E6D76FA; PortOperationTime=0; State=1;
                   SwitchOperationTime=0; VfpOperationTime=0}

ActivityId       : 91258373-35D4-4F54-B91E-10601C09DBE4
AdditionalParams :
CompartmentId    : 0
Containers       : {}
Flags            : 0
Health           : @{LastErrorCode=0; LastUpdateTime=133511564204918777}
ID               : B687E0BC-66CF-428A-8911-D286EC1D4126
IsDefault        : False
Policies         : {}
SchemaVersion    : @{Major=0; Minor=0}
State            : 1
Version          : 64424509440
Resources        : @{AdditionalParams=; AllocationOrder=0; CompartmentOperationTime=0; Flags=0; Health=; ID=91258373-35D4-4F54-B91E-10601C09DBE4; PortOperationTime=0; State=1;
                   SwitchOperationTime=0; VfpOperationTime=0}

ActivityId       : 9F3A847D-F7D0-4826-9362-6E1003009882
AdditionalParams :
Containers       : {}
Flags            : 0
Health           : @{LastErrorCode=0; LastUpdateTime=133511565214011290}
ID               : B5F2B8D5-D10F-4DAD-A8D2-EC1B35A1A60F
IsDefault        : False
Policies         : {}
SchemaVersion    : @{Major=0; Minor=0}
State            : 1
Version          : 64424509440
Resources        : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=9F3A847D-F7D0-4826-9362-6E1003009882;
                   PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

ActivityId       : 28897455-5B27-4097-85E9-34CD6BCD727B
AdditionalParams :
Containers       : {}
Flags            : 0
Health           : @{LastErrorCode=0; LastUpdateTime=133511565002386173}
ID               : FDCFD6E5-88EB-4211-9D4E-7CBE17F904C6
IsDefault        : False
Policies         : {}
SchemaVersion    : @{Major=0; Minor=0}
State            : 1
Version          : 64424509440
Resources        : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=28897455-5B27-4097-85E9-34CD6BCD727B;
                   PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

ActivityId       : 9B531535-C97F-4465-95E8-E5B820FF6FC5
AdditionalParams :
CompartmentId    : 0
Containers       : {}
Flags            : 0
Health           : @{LastErrorCode=0; LastUpdateTime=133511565160811581}
ID               : AEC99FB3-D67C-47D6-B043-B5D9FD3F1616
IsDefault        : False
Policies         : {}
SchemaVersion    : @{Major=0; Minor=0}
State            : 1
Version          : 64424509440
Resources        : @{AdditionalParams=; AllocationOrder=0; CompartmentOperationTime=0; Flags=0; Health=; ID=9B531535-C97F-4465-95E8-E5B820FF6FC5; PortOperationTime=0; State=1;
                   SwitchOperationTime=0; VfpOperationTime=0}

ActivityId       : C11DECEF-49A1-4D5F-B4C6-6EA9F6549829
AdditionalParams :
Containers       : {}
Flags            : 0
Health           : @{LastErrorCode=0; LastUpdateTime=133511565491101643}
ID               : 9A2AD8D1-15DF-4B99-96D1-459C78CF2E07
IsDefault        : False
Policies         : {}
SchemaVersion    : @{Major=0; Minor=0}
State            : 1
Version          : 64424509440
Resources        : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=C11DECEF-49A1-4D5F-B4C6-6EA9F6549829;
                   PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0}

ActivityId       : F8CA953C-B580-4A38-B9A7-4F3D7A85F9AF
AdditionalParams :
CompartmentId    : 0
Containers       : {}
Flags            : 0
Health           : @{LastErrorCode=0; LastUpdateTime=133511654365588728}
ID               : 4482981E-E2AF-43F0-992E-A03898B224D8
IsDefault        : False
Policies         : {}
SchemaVersion    : @{Major=0; Minor=0}
State            : 1
Version          : 64424509440
Resources        : @{AdditionalParams=; AllocationOrder=0; CompartmentOperationTime=0; Flags=0; Health=; ID=F8CA953C-B580-4A38-B9A7-4F3D7A85F9AF; PortOperationTime=0; State=1;
                   SwitchOperationTime=0; VfpOperationTime=0}

@wizpresso-steve-cy-fan
Copy link
Author

Get-HnsNetwork:

ActivityId             : C4D8511F-6959-4162-A30F-F12EC759A942
AdditionalParams       :
CurrentEndpointCount   : 1
Extensions             : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows 篩選平台}, @{Id=F74F241B-440F-4433-BB28-00F89EAD20D8; IsEnabled=False;
                         Name=Microsoft Azure VFP Switch Filter Extension}, @{Id=430BDADD-BAB0-41AB-A369-94B67FA5BE0A; IsEnabled=True; Name=Microsoft NDIS Capture}}
Flags                  : 8
Health                 : @{LastErrorCode=0; LastUpdateTime=133508076463076342}
ID                     : 55FEDB88-4B11-4DA4-8CD1-59970718A804
IPv6                   : False
LayeredOn              : B0BEC74D-4BD5-4E66-ACEA-F8A7B44CA38E
MacPools               : {@{EndMacAddress=00-15-5D-F2-0F-FF; StartMacAddress=00-15-5D-F2-00-00}}
MaxConcurrentEndpoints : 1
Name                   : nat
NatName                : NATF7437238-4A6F-4E71-8718-78A394A7F770
Policies               : {@{Type=VLAN; VLAN=1}}
State                  : 1
Subnets                : {@{AdditionalParams=; AddressPrefix=172.29.144.0/20; Flags=0; GatewayAddress=172.29.144.1; Health=; ID=8989068C-581F-4B05-8D93-D1A9E8D5CFF8;
                         IpSubnets=System.Object[]; ObjectType=5; Policies=System.Object[]; State=0}}
SwitchGuid             : 55FEDB88-4B11-4DA4-8CD1-59970718A804
TotalEndpoints         : 3
Type                   : nat
Version                : 64424509440
Resources              : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=C4D8511F-6959-4162-A30F-F12EC759A942;
                         PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0; parentId=21350A50-2C46-458D-9470-57D22753F2FF}

ActivityId             : 18440B9B-FE11-43BE-BDF4-5091BDAE9702
AdditionalParams       :
CurrentEndpointCount   : 0
DNSServerCompartment   : 4
DrMacAddress           : 00-15-5D-63-50-D5
Extensions             : {@{Id=F74F241B-440F-4433-BB28-00F89EAD20D8; IsEnabled=True; Name=Microsoft Azure VFP Switch Filter Extension}, @{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A;
                         IsEnabled=False; Name=Microsoft Windows 篩選平台}, @{Id=430BDADD-BAB0-41AB-A369-94B67FA5BE0A; IsEnabled=True; Name=Microsoft NDIS Capture}}
Flags                  : 0
Health                 : @{LastErrorCode=0; LastUpdateTime=133511511245535391}
ID                     : 93C59A4B-F5C0-42AA-BE09-E7480DE1FF3D
IPv6                   : False
LayeredOn              : 566A5F10-9D33-4D5A-A1AF-5641C0388D25
MacPools               : {@{EndMacAddress=00-15-5D-E2-AF-FF; StartMacAddress=00-15-5D-E2-A0-00}}
ManagementIP           : 172.30.0.3
MaxConcurrentEndpoints : 0
Name                   : Calico
Policies               : {@{Type=HostRoute}, @{DestinationPrefix=10.244.199.0/26; DistributedRouterMacAddress=66-ee-c8-00-d3-e1; IsolationId=4096; ProviderAddress=172.30.0.13;
                         Type=RemoteSubnetRoute}, @{DestinationPrefix=10.244.199.64/26; DistributedRouterMacAddress=66-ee-c8-00-d3-e1; IsolationId=4096; ProviderAddress=172.30.0.13;
                         Type=RemoteSubnetRoute}, @{DestinationPrefix=10.244.199.128/26; DistributedRouterMacAddress=66-ee-c8-00-d3-e1; IsolationId=4096; ProviderAddress=172.30.0.13;
                         Type=RemoteSubnetRoute}}
State                  : 1
Subnets                : {@{AdditionalParams=; AddressPrefix=10.244.178.0/26; Flags=0; GatewayAddress=10.244.178.1; Health=; ID=E17AF071-4186-452A-A86F-3AD89660AEDD;
                         IpSubnets=System.Object[]; ObjectType=5; Policies=System.Object[]; State=0}}
SwitchGuid             : 36FD5159-3DC3-4097-9154-67AE063DD803
TotalEndpoints         : 0
Type                   : Overlay
Version                : 64424509440
Resources              : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=18440B9B-FE11-43BE-BDF4-5091BDAE9702;
                         PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0; parentId=0D86ADC0-B0C2-4612-A33D-3A40B95ADF88}

ActivityId             : 0173C593-C43B-4951-9107-42829048B350
AdditionalParams       :
CurrentEndpointCount   : 0
DNSServerCompartment   : 3
DrMacAddress           : 00-15-5D-63-50-D5
Extensions             : {@{Id=F74F241B-440F-4433-BB28-00F89EAD20D8; IsEnabled=True; Name=Microsoft Azure VFP Switch Filter Extension}, @{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A;
                         IsEnabled=False; Name=Microsoft Windows 篩選平台}, @{Id=430BDADD-BAB0-41AB-A369-94B67FA5BE0A; IsEnabled=True; Name=Microsoft NDIS Capture}}
Flags                  : 0
Health                 : @{LastErrorCode=0; LastUpdateTime=133511550474556356}
ID                     : 08BD0B90-8153-4AE0-A6D4-44E606D4AFED
IPv6                   : False
LayeredOn              : 566A5F10-9D33-4D5A-A1AF-5641C0388D25
MacPools               : {@{EndMacAddress=00-15-5D-5D-8F-FF; StartMacAddress=00-15-5D-5D-80-00}}
ManagementIP           : 172.30.0.3
MaxConcurrentEndpoints : 0
Name                   : External
Policies               : {}
State                  : 1
Subnets                : {@{AdditionalParams=; AddressPrefix=192.168.255.0/30; Flags=0; GatewayAddress=192.168.255.1; Health=; ID=49982329-3B8B-425E-A20D-DDD0F26A9A2B;
                         IpSubnets=System.Object[]; ObjectType=5; Policies=System.Object[]; State=0}}
SwitchGuid             : 36FD5159-3DC3-4097-9154-67AE063DD803
TotalEndpoints         : 0
Type                   : Overlay
Version                : 64424509440
Resources              : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=0173C593-C43B-4951-9107-42829048B350;
                         PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0; parentId=0D86ADC0-B0C2-4612-A33D-3A40B95ADF88}

ActivityId             : 07D4BE14-BCDF-497C-BE6D-EBE7BC24F598
AdditionalParams       :
CurrentEndpointCount   : 0
Extensions             : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows 篩選平台}, @{Id=F74F241B-440F-4433-BB28-00F89EAD20D8; IsEnabled=False;
                         Name=Microsoft Azure VFP Switch Filter Extension}, @{Id=430BDADD-BAB0-41AB-A369-94B67FA5BE0A; IsEnabled=True; Name=Microsoft NDIS Capture}}
Flags                  : 11
GatewayMac             : 00-15-5D-01-67-00
Health                 : @{AddressNotificationMissedCount=0; AddressNotificationSequenceNumber=0; DHCPNotificationMissedCount=0; DNSCacheNotificationMissedCount=0;
                         DNSCacheNotificationSequenceNumber=0; DNSNotificationMissedCount=0; DNSNotificationSequenceNumber=0; InterfaceNotificationMissedCount=0;
                         InterfaceNotificationSequenceNumber=0; LastErrorCode=0; LastUpdateTime=133511554129543789; MacAddressNotificationMissedCount=0;
                         MacAddressNotificationSequenceNumber=0; NeighborNotificationMissedCount=0; NeighborNotificationSequenceNumber=0; RouteNotificationMissedCount=0;
                         RouteNotificationSequenceNumber=0; XlatNotificationMissedCount=0; XlatNotificationSequenceNumber=0}
ID                     : C08CB7B8-9B3C-408E-8E30-5E16A3AEB444
IPv6                   : False
LayeredOn              : BDFFBBAE-CB1F-4452-B307-053D6A79E288
MacPools               : {@{EndMacAddress=00-15-5D-D2-9F-FF; StartMacAddress=00-15-5D-D2-90-00}}
MaxConcurrentEndpoints : 0
Name                   : Default Switch
NatName                : ICS03DF7903-538F-485D-B1A0-0A81EEA8F816
Policies               : {}
State                  : 1
Subnets                : {@{AdditionalParams=; AddressPrefix=172.19.16.0/20; Flags=0; GatewayAddress=172.19.16.1; Health=; ID=A6C3F491-0ECC-47B3-BBE9-2CB294171826;
                         IpSubnets=System.Object[]; ObjectType=5; Policies=System.Object[]; State=0}}
SwitchGuid             : C08CB7B8-9B3C-408E-8E30-5E16A3AEB444
SwitchName             : Default Switch
TotalEndpoints         : 0
Type                   : ICS
Version                : 64424509440
Resources              : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; CompartmentOperationTime=0; Flags=0; Health=; ID=07D4BE14-BCDF-497C-BE6D-EBE7BC24F598;
                         PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0; parentId=AAABDEBC-1B6C-4696-8228-7B33C036175F}

@wizpresso-steve-cy-fan
Copy link
Author

I have confirmed by setting natOutgoing to false, then the policies are valid:

apiVersion: crd.projectcalico.org/v1
kind: IPPool
metadata:
  name: default-ipv4-ippool
spec:
  allowedUses:
    - Workload
    - Tunnel
  blockSize: 26
  cidr: 10.244.0.0/16
  ipipMode: Never
  natOutgoing: false
  nodeSelector: all()
  vxlanMode: Always

@wizpresso-steve-cy-fan
Copy link
Author

I created a specific block for Windows nodes, and it is clearly a bug now:

apiVersion: crd.projectcalico.org/v1
kind: IPPool
metadata:
  name: default-ipv4-ippool-windows
spec:
  allowedUses:
    - Workload
    - Tunnel
  blockSize: 26
  cidr: 10.244.253.0/24
  natOutgoing: false
  nodeSelector: kubernetes.io/os==windows
  vxlanMode: Always

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f651efd2845430dd29351e0f1e72cb8c8d6060f55214e53be849daf9633a9edb": plugin type="calico" name="Calico" failed (add): Expected string

@wizpresso-steve-cy-fan
Copy link
Author

I noticed I cannot ping to 10.245.253.1 on both the Windows node, which is supposed to be the first VXLAN for the host, but I can ping to the VXLAN IP on the Linux node.
So this is my Linux node

projectcalico.org/IPv4VXLANTunnelAddr=10.244.199.167

And I can ping from Windows node to Linux node:

(base) PS C:\Users\Administrator> ping 10.244.199.167

Ping 10.244.199.167 (使用 32 位元組的資料):
回覆自 10.244.199.167: 位元組=32 時間<1ms TTL=64
回覆自 10.244.199.167: 位元組=32 時間<1ms TTL=64
回覆自 10.244.199.167: 位元組=32 時間<1ms TTL=64
回覆自 10.244.199.167: 位元組=32 時間<1ms TTL=64

10.244.199.167 的 Ping 統計資料:
    封包: 已傳送 = 4，已收到 = 4, 已遺失 = 0 (0% 遺失)，
大約的來回時間 (毫秒):
    最小值 = 0ms，最大值 = 0ms，平均 = 0ms

@coutinhop
Copy link
Contributor

Hi @wizpresso-steve-cy-fan, Windows desktop versions are not supported by either calico (https://docs.tigera.io/calico/latest/getting-started/kubernetes/windows-calico/requirements#windows-platform-requirements) or kubernetes itself for that matter (https://kubernetes.io/docs/concepts/windows/intro/#windows-os-version-support). The exact details escape me at the moment, but there is functionality exclusive to Windows server versions which is needed for k8s.

@coutinhop coutinhop closed this as not planned Won't fix, can't repro, duplicate, stale Feb 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@coutinhop @wizpresso-steve-cy-fan