Skip to content

Conversation

@huntergregory
Copy link
Contributor

@huntergregory huntergregory commented Jan 10, 2023

Updates to enable NPM on CAPZ Windows Testing for use in run-capz-e2e.sh. This setup uses Calico CNI.

Changes

1/3: kubeconfig Location

CAPZ stores the default kubeconfig in a different location, so we must have a different DaemonSet command to copy from this location.

Files:

  • new .ps1 file
  • new NPM yaml using this ps1 in a DaemonSet command
  • in Dockerfile, copy over .ps1

2/3: Toggle for Windows Network Name

Can now configure "calico" HNS network (as well as the standard "azure").

3/3: Add Connectivity

Add Base ACLs to Endpoints

Without Calico NetPol, a Calico CNI Endpoint looks like:

{
    "ExceptionList":  [
                            "10.96.0.0/12",
                            "192.168.0.0/16"
                        ],
    "Type":  "OutBoundNAT"
},
{
    "DestinationPrefix":  "10.96.0.0/12",
    "NeedEncap":  true,
    "Type":  "ROUTE"
},
{
    "PA":  "10.1.0.5",
    "Type":  "PA"
},
{
    "Action":  "Block",
    "Direction":  "In",
    "Priority":  65500,
    "Protocols":  "256",
    "RuleType":  "Switch",
    "Scope":  0,
    "Type":  "ACL"
}

We will add the below ACLs, which are all of the ACLs on Azure CNI Endpoints. Note we have to increase the priority on two of the ACLs (from 65500 to 65499).

{
    "Id": "azure-acl-baseazurewireserver",
    "Action":  "Block",
    "Direction":  "Out",
    "Priority":  200,
    "Protocols":  "6",
    "RemoteAddresses":  "168.63.129.16/32",
    "RemotePorts":  "80",
    "RuleType":  "Switch",
    "Type":  "ACL"
},
{
    "Id": "azure-acl-baseallowinswitch",
    "Action":  "Allow",
    "Direction":  "In",
    "Priority":  65499,
    "Type":  "ACL"
},
{
    "Id": "azure-acl-baseallowoutswitch",
    "Action":  "Allow",
    "Direction":  "Out",
    "Priority":  65499,
    "Type":  "ACL"
},

We also need to include the following ACLs, which are a subset of ACLs seen on Endpoints where Calico CNI and NetPol are enabled. Without these ACLs, we get this error when trying to connect Pods: dial tcp 192.168.240.198:80: connectex: An attempt was made to access a socket in a way forbidden by its access permissions..

Note 1: to use existing NPM code, have to disregard the following fields in the original Calico ACLs: "Protocol", "LocalPort", "RemotePort", and "InternalPort".

{
    "Id": "azure-acl-baseallowinhost",
    "Action":  "Allow",
    "Direction":  "In",
    "LocalAddresses":  "",
    "Priority":  0,
    "RemoteAddresses":  "",
    "RuleType":  "Host",
},
{
    "Id": "azure-acl-baseallowouthost",
    "Action":  "Allow",
    "Direction":  "Out",
    "LocalAddresses":  "",
    "Priority":  0,
    "RemoteAddresses":  "",
    "RuleType":  "Host",
}

Note 2: interestingly, the result in HNS for the above ACLs looks like the below. For instance, "RuleType" is not displayed. The above socket error does not occur though.

{
    "Action":  "Allow",
    "Direction":  "In",
    "Id":  "azure-acl-baseallowinhost",
    "Scope":  0,
    "Type":  "ACL"
},

ACL Appendix

HNS Policies on an Azure CNI Endpoint

{
    "ExceptionList":  [
                            "10.224.0.0/16",
                            "10.224.0.0/12"
                        ],
    "Type":  "OutBoundNAT"
},
{
    "Action":  "Block",
    "Direction":  "Out",
    "Priority":  200,
    "Protocols":  "6",
    "RemoteAddresses":  "168.63.129.16/32",
    "RemotePorts":  "80",
    "RuleType":  "Switch",
    "Scope":  0,
    "Type":  "ACL"
},
{
    "Action":  "Allow",
    "Direction":  "In",
    "Priority":  65500,
    "Scope":  0,
    "Type":  "ACL"
},
{
    "Action":  "Allow",
    "Direction":  "Out",
    "Priority":  65500,
    "Scope":  0,
    "Type":  "ACL"
},
{
    "Destinations":  [
                            "10.224.0.116"
                        ],
    "Type":  "OutBoundNAT"
},
{
    "Type":  "L2Driver"
}

HNS Policies on a Calico CNI Endpoint (with NetPol enabled)

{
    "ExceptionList":  "10.96.0.0/12 192.168.0.0/16",
    "Type":  "OutBoundNAT"
},
{
    "DestinationPrefix":  "10.96.0.0/12",
    "NeedEncap":  true,
    "Type":  "ROUTE"
},
{
    "PA":  "10.1.0.4",
    "Type":  "PA"
},
{
    "Action":  "Allow",
    "Direction":  "In",
    "Id":  "allow-host-to-endpoint",
    "InternalPort":  0,
    "LocalAddresses":  "",
    "LocalPort":  0,
    "Priority":  900,
    "Protocol":  256,
    "RemoteAddresses":  "10.1.0.4/32",
    "RemotePort":  0,
    "RuleType":  "Switch",
    "Scope":  0,
    "ServiceName":  "",
    "Type":  "ACL"
},
{
    "Action":  "Allow",
    "Direction":  "In",
    "Id":  "profile-kns.test-DuT2ob60L3yhj_Gv-0",
    "InternalPort":  0,
    "LocalAddresses":  "",
    "LocalPort":  0,
    "Priority":  1000,
    "Protocol":  256,
    "RemoteAddresses":  "",
    "RemotePort":  0,
    "RuleType":  "Switch",
    "Scope":  0,
    "ServiceName":  "",
    "Type":  "ACL"
},
{
    "Action":  "Allow",
    "Direction":  "Out",
    "Id":  "profile-kns.test-DuT2ob60L3yhj_Gv-0",
    "InternalPort":  0,
    "LocalAddresses":  "",
    "LocalPort":  0,
    "Priority":  1000,
    "Protocol":  256,
    "RemoteAddresses":  "",
    "RemotePort":  0,
    "RuleType":  "Switch",
    "Scope":  0,
    "ServiceName":  "",
    "Type":  "ACL"
},
{
    "Action":  "Block",
    "Direction":  "In",
    "InternalPort":  0,
    "LocalAddresses":  "",
    "LocalPort":  0,
    "Priority":  1001,
    "Protocol":  256,
    "RemoteAddresses":  "",
    "RemotePort":  0,
    "RuleType":  "Switch",
    "Scope":  0,
    "ServiceName":  "",
    "Type":  "ACL"
},
{
    "Action":  "Allow",
    "Direction":  "In",
    "InternalPort":  0,
    "LocalAddresses":  "",
    "LocalPort":  0,
    "Priority":  0,
    "Protocol":  256,
    "RemoteAddresses":  "",
    "RemotePort":  0,
    "RuleType":  "Host",
    "Scope":  0,
    "ServiceName":  "",
    "Type":  "ACL"
},
{
    "Action":  "Block",
    "Direction":  "Out",
    "Id":  "azure-wireserver",
    "InternalPort":  0,
    "LocalAddresses":  "",
    "LocalPort":  0,
    "Priority":  200,
    "Protocol":  6,
    "RemoteAddresses":  "168.63.129.16/32",
    "RemotePort":  0,
    "RemotePorts":  "80",
    "RuleType":  "Switch",
    "Scope":  0,
    "ServiceName":  "",
    "Type":  "ACL"
},
{
    "Action":  "Block",
    "Direction":  "Out",
    "InternalPort":  0,
    "LocalAddresses":  "",
    "LocalPort":  0,
    "Priority":  1001,
    "Protocol":  256,
    "RemoteAddresses":  "",
    "RemotePort":  0,
    "RuleType":  "Switch",
    "Scope":  0,
    "ServiceName":  "",
    "Type":  "ACL"
},
{
    "Action":  "Allow",
    "Direction":  "Out",
    "InternalPort":  0,
    "LocalAddresses":  "",
    "LocalPort":  0,
    "Priority":  0,
    "Protocol":  256,
    "RemoteAddresses":  "",
    "RemotePort":  0,
    "RuleType":  "Host",
    "Scope":  0,
    "ServiceName":  "",
    "Type":  "ACL"
}

@huntergregory huntergregory added npm Related to NPM. ci Infra or tooling. labels Jan 10, 2023
@huntergregory huntergregory requested a review from a team as a code owner January 10, 2023 23:39
@huntergregory huntergregory requested review from ck319 and removed request for a team January 10, 2023 23:39
@huntergregory huntergregory force-pushed the hgregory/capz-kubeconfig branch from 6480039 to cdfa27a Compare January 11, 2023 18:27
@huntergregory huntergregory force-pushed the hgregory/capz-kubeconfig branch from cdfa27a to 149a0f9 Compare January 11, 2023 18:57
@huntergregory huntergregory force-pushed the hgregory/capz-kubeconfig branch 2 times, most recently from 9b56cff to 3df9d9f Compare January 19, 2023 00:42
@huntergregory huntergregory force-pushed the hgregory/capz-kubeconfig branch from 3df9d9f to 7964a89 Compare January 19, 2023 22:50
@huntergregory huntergregory force-pushed the hgregory/capz-kubeconfig branch 3 times, most recently from 683864e to 33758c2 Compare January 20, 2023 00:28
@huntergregory huntergregory force-pushed the hgregory/capz-kubeconfig branch from 33758c2 to de3b8f3 Compare January 20, 2023 00:51
@huntergregory huntergregory changed the title test: [NPM-WIN] support for CAPZ windows testing feat: [NPM-WIN] support for CAPZ windows testing Jan 20, 2023
@vakalapa vakalapa merged commit 2832b50 into master Mar 2, 2023
@vakalapa vakalapa deleted the hgregory/capz-kubeconfig branch March 2, 2023 21:24
rjdenney pushed a commit that referenced this pull request Mar 13, 2023
* set kubeconfig on capz

* update dockerfile

* test network name Calico

* add base acls

* add WindowsNetworkName toggle and revert hard coded Calico parts

* update base acls for calico and add UTs

* capitalize calico network name

* fix connectivity. try with host allow acls

* revert change to policy_windows.go

* more UTs and add base ACLs for other "new endpoint" scenario

* run all UTs

* update npm image to .42

* add log line

* allow traffic going inter-node

* Revert "allow traffic going inter-node"

This reverts commit e101482.

* add long-runner pod for testing vfp tags in capz

* fix lints
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Infra or tooling. npm Related to NPM.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants