Release/v1.4 Remove AKS-Engine (#2212)

* build azure-vnet-telemetry and azure-vnet-ipam in dropgz-test (#1846) build azure-vnet-telemetry and azure-vnet-ipam in dropgz-test for parity with release image Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> (cherry picked from commit f619259) * ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <tamanoha@microsoft.com> (cherry picked from commit 024819d) * CI: [CNI] Replace the bash scripts for CNI load testing with golang test cases (#2003) CI:[CNI] Replace the bash scripts with the golang test cases (cherry picked from commit 008ae45) * ci: [CNI] Move Nightly Cilium Pipeline test to ACN (#1963) * CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: Eliminating duplicate lines * ci: Add update permission for ciliumidentity * fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> * test for panic in pool monitor Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> --------- Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <tamanoha@microsoft.com> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> * test for panic in pool monitor Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> --------- Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <tamanoha@microsoft.com> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: File Directory * style: Comments * Addressing Comments --------- Co-authored-by: Paul Johnston <35265851+pjohnst5@users.noreply.github.com> (cherry picked from commit 1514d95) * ci:[CNI] Add windows CNIv1 datapath test (#2016) * ci: Transfer files * test: Working Datapath Test * test: apierror Tests * style: Datapath Package * test: Deployment timing * fix: Error check * fix: Lint (cherry picked from commit 390977d) * fix: [CNI] CNI load test failing due to namespace already created (#2031) fix: CNI load test failing due to namespace already created (cherry picked from commit c10900e) * ci:[CNI] Windows cniv1 load test pipeline (#2024) CI:[CNI] Windows cniv1 load test pipeline (cherry picked from commit e45ad21) * ci: [CNI] Adding aks cluster creation steps for k8s e2e test (#2052) * ci: [CNI] Adding aks cluster creation steps for k8s e2e test * Add validate step to the pipeline * Adding the telemetry config to the cluster (cherry picked from commit 846e508) * ci:[CNI] Replace AKS-Engine Tests with k8s conformance tests (#2062) * Initial Commit * Add attempts to prevent flakyness * Add taint for windows tests * Add k8s e2e tests * Testing vmSizes * Artifact k8se2e binary * Remove NPM E2E * Add testing and increase processes * Addressing comments (cherry picked from commit 451c691) * CI: Removing AKS engine related code (#2089) (cherry picked from commit b45c2c7) * feat: [dropgz] Dropgz for windows (#2075) * feat: [dropgz] Dropgz for windows * Removing the code for killing the process from dropgz for windows (cherry picked from commit 7a41178) * ci: Update dns tests for k8s conformance (#2104) Update dns tests for k8s v1.26 (cherry picked from commit bbf2fd4) * ci: adding cni package as a trigger (#2108) (cherry picked from commit e6a8ea6) * ci: add packages for submodule trigger (#2154) (cherry picked from commit 4aecfd6) * set mellanox reg key (#1768) (cherry picked from commit fa2de6d) --------- Co-authored-by: Evan Baker <rbtr@users.noreply.github.com> Co-authored-by: Camryn Lee <31013536+camrynl@users.noreply.github.com> Co-authored-by: Vipul Singh <vipul21sept@gmail.com> Co-authored-by: Rajvi <107083915+rajvinar@users.noreply.github.com>
Azure · Sep 22, 2023 · 5d25eb1 · 5d25eb1
1 parent c325121
commit 5d25eb1
Show file tree

Hide file tree

Showing 84 changed files with 3,669 additions and 1,173 deletions.
diff --git a/.pipelines/cni/cilium/cilium-cni-load-test.yaml b/.pipelines/cni/cilium/cilium-cni-load-test.yaml
@@ -16,7 +16,7 @@ stages:
               inlineScript: |
                 set -ex
                 make -C ./hack/swift azcfg AZCLI=az REGION=$(LOCATION)
-                make -C ./hack/swift overlay-byocni-up AZCLI=az REGION=$(LOCATION) SUB=$(SUBSCRIPTION_ID) CLUSTER=${RESOURCE_GROUP} NODE_COUNT=10 VM_SIZE=Standard_DS4_v2
+                make -C ./hack/swift overlay-no-kube-proxy-up AZCLI=az REGION=$(LOCATION) SUB=$(SUBSCRIPTION_ID) CLUSTER=${RESOURCE_GROUP}-$(make revision) NODE_COUNT=10 VM_SIZE=Standard_DS4_v2
             name: "CreateAksCluster"
             displayName: "Create AKS Cluster"
   - stage: install_cilium
@@ -35,7 +35,7 @@ stages:
               inlineScript: |
                 set -ex
                 az extension add --name aks-preview
-                make -C ./hack/swift set-kubeconf AZCLI=az CLUSTER=${RESOURCE_GROUP}
+                make -C ./hack/aks set-kubeconf AZCLI=az CLUSTER=${{ parameters.clusterName }}-$(make revision)
                 ls -lah
                 pwd
                 kubectl cluster-info
@@ -48,6 +48,9 @@ stages:
                 echo "install Cilium onto Overlay Cluster"
                 kubectl apply -f test/integration/manifests/cilium/cilium-agent
                 kubectl apply -f test/integration/manifests/cilium/cilium-operator
+                # Passes Cilium image to daemonset and deployment
+                envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/daemonset.yaml | kubectl apply -f -
+                envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/deployment.yaml | kubectl apply -f -
                 kubectl get po -owide -A
                 echo "deploy ip-masq-agent for overlay"
                 kubectl create -f test/integration/manifests/ip-masq-agent/ip-masq-agent.yaml --validate=false
@@ -62,20 +65,11 @@ stages:
     jobs:
       - job: deploy_pods
         steps:
-          - task: AzureCLI@1
-            displayName: "Pod Deployment"
-            inputs:
-              azureSubscription: $(TEST_SUB_SERVICE_CONNECTION)
-              scriptLocation: "inlineScript"
-              scriptType: "bash"
-              addSpnToEnvironment: true
-              inlineScript: |
-                set -ex
-                az extension add --name aks-preview
-                make -C ./hack/swift set-kubeconf AZCLI=az CLUSTER=${RESOURCE_GROUP}
-                kubectl apply -f hack/manifests/pod.yaml
-                kubectl apply -f hack/manifests/hostprocess.yaml
-                bash hack/scripts/scale_deployment.sh
+          - template: ../load-test-templates/pod-deployment-template.yaml
+            parameters:
+              clusterName: ${{ parameters.clusterName }}
+              scaleup: 2400
+              os: linux
   - stage: validate_state
     dependsOn: pod_deployment
     displayName: "Validate State"
@@ -89,9 +83,9 @@ stages:
               scriptType: "bash"
               addSpnToEnvironment: true
               inlineScript: |
-                make -C ./hack/swift set-kubeconf AZCLI=az CLUSTER=${RESOURCE_GROUP}
+                make -C ./hack/swift set-kubeconf AZCLI=az CLUSTER=${RESOURCE_GROUP}-$(make revision)
                 kubectl get pods -A
-                bash hack/scripts/validate_state.sh
+                make test-validate-state
             name: "ValidateState"
             displayName: "Validate State"
             retryCountOnTaskFailure: 3
@@ -109,18 +103,22 @@ stages:
               addSpnToEnvironment: true
               inlineScript: |
                 echo "Scale up the pods and immediated restart the nodes"
-                make -C ./hack/swift set-kubeconf AZCLI=az CLUSTER=${RESOURCE_GROUP}
+                make -C ./hack/swift set-kubeconf AZCLI=az CLUSTER=${RESOURCE_GROUP}-$(make revision)
                 make -C ./hack/swift azcfg AZCLI=az REGION=$(LOCATION)
+                cd test/integration/load
                 echo "Scaling the pods down to 100 per node"
-                bash ./hack/scripts/scale_deployment.sh -n 0 -u 1000 -s
+                go test -count 1 -timeout 30m -tags load -run ^TestScaleDeployment$ -tags=load -scaleup=1000 -skip-wait=true
+                cd ../../../
                 echo "Restarting the nodes"
                 vmss_name=$(az vmss list -g MC_${RESOURCE_GROUP}_${RESOURCE_GROUP}_$(LOCATION) --query "[].name" -o tsv)
-                make -C ./hack/swift restart-vmss AZCLI=az CLUSTER=${RESOURCE_GROUP} REGION=$(LOCATION) VMSS_NAME=$vmss_name
-                bash ./hack/scripts/scale_deployment.sh -n 0 -u 1000 -c
+                make -C ./hack/swift restart-vmss AZCLI=az CLUSTER=${RESOURCE_GROUP}-$(make revision) REGION=$(LOCATION) VMSS_NAME=$vmss_name
+                cd test/integration/load
+                go test -count 1 -timeout 30m -tags load -run ^TestScaleDeployment$ -tags=load -replicas=1000
             name: "RestartNodes"
             displayName: "Restart Nodes"
           - script: |
-              bash hack/scripts/validate_state.sh
+              export RESTART_CASE=true
+              make test-validate-state
             name: "ValidateState"
             displayName: "Validate State"
             retryCountOnTaskFailure: 3
@@ -148,11 +146,11 @@ stages:
               addSpnToEnvironment: true
               inlineScript: |
                 set -ex
-                make -C ./hack/swift set-kubeconf AZCLI=az CLUSTER=${RESOURCE_GROUP}
+                make -C ./hack/aks set-kubeconf AZCLI=az CLUSTER=${{ parameters.clusterName }}-$(make revision)
             name: "GetCluster"
             displayName: "Get AKS Cluster"
           - script: |
-              k delete deployment container -n default
+              kubectl delete ns load-test
               cilium connectivity test
             retryCountOnTaskFailure: 6
             name: "CiliumConnectivityTests"
@@ -175,9 +173,9 @@ stages:
                 if [ "$(DELETE_RESOURCES)" ]
                 then
                   echo "Deleting Cluster and resource group"
-                  make -C ./hack/swift set-kubeconf AZCLI=az CLUSTER=${RESOURCE_GROUP}
-                  make -C ./hack/swift azcfg AZCLI=az REGION=$(LOCATION)
-                  make -C ./hack/swift down AZCLI=az REGION=$(LOCATION) SUB=$(SUBSCRIPTION_ID) CLUSTER=${RESOURCE_GROUP}
+                  make -C ./hack/aks set-kubeconf AZCLI=az CLUSTER=${{ parameters.clusterName }}-$(make revision)
+                  make -C ./hack/aks azcfg AZCLI=az REGION=$(LOCATION)
+                  make -C ./hack/aks down AZCLI=az REGION=$(LOCATION) SUB=$(SUBSCRIPTION_ID) CLUSTER=${{ parameters.clusterName }}-$(make revision)
                   echo "Cluster and resources down"
                 else
                   echo "Deletion of resources is False"

diff --git a/.pipelines/cni/cilium/nightly-release-test.yml b/.pipelines/cni/cilium/nightly-release-test.yml
@@ -0,0 +1,54 @@
+pr: none
+trigger: none
+
+stages:
+  - stage: init
+    displayName: "Build and Push Cilium Image"
+    jobs:
+      - job: build_and_push_cilium_image
+        steps:
+          - bash: |
+              set -ex
+              cd .pipelines/
+              git clone https://github.com/cilium/cilium.git
+              cd cilium
+              make docker-cilium-image
+              make docker-operator-generic-image
+            name: BuildCiliumImage
+            displayName: "Build Cilium Image"
+          - task: Docker@2
+            displayName: Login
+            inputs:
+              containerRegistry: $(CONTAINER_REGISTRY)
+              command: "login"
+          - task: AzureCLI@1
+            inputs:
+              azureSubscription: $(TEST_SUB_SERVICE_CONNECTION)
+              scriptLocation: "inlineScript"
+              inlineScript: |
+                set -ex
+                docker tag quay.io/cilium/cilium:$(DOCKER_IMAGE_TAG) $(ACR).azurecr.io/cilium/cilium:$(DOCKER_IMAGE_TAG)
+                docker tag quay.io/cilium/operator-generic:$(DOCKER_IMAGE_TAG) $(ACR).azurecr.io/cilium/operator-generic:$(DOCKER_IMAGE_TAG)
+                docker push $(ACR).azurecr.io/cilium/cilium:$(DOCKER_IMAGE_TAG)
+                docker push $(ACR).azurecr.io/cilium/operator-generic:$(DOCKER_IMAGE_TAG)
+            name: "PushCiliumImage"
+            displayName: "Push Cilium Image"
+          - task: Docker@2
+            displayName: Logout
+            inputs:
+              containerRegistry: $(CONTAINER_REGISTRY)
+              command: "logout"
+  - stage: cilium_nightly
+    displayName: E2E - Cilium Nightly
+    variables:
+      GOPATH: "$(Agent.TempDirectory)/go" # Go workspace path
+      GOBIN: "$(GOPATH)/bin" # Go binaries path
+      modulePath: "$(GOPATH)/src/github.com/Azure/azure-container-networking"
+    jobs:
+      - job: cilium_nightly
+        steps:
+        - template: ../../singletenancy/cilium-overlay/cilium-overlay-e2e-step-template.yaml
+          parameters:
+            name: "cilium_nightly"
+            testDropgz: ""
+            clusterName: "ciliumnightly"
diff --git a/.pipelines/cni/k8s-e2e/k8s-e2e-job-template.yaml b/.pipelines/cni/k8s-e2e/k8s-e2e-job-template.yaml
@@ -0,0 +1,111 @@
+parameters:
+  clusterName: ""
+  os: ""
+  dependsOn: ""
+  sub: ""
+
+
+jobs:
+  - job: cni_k8se2e
+    displayName: "CNI k8s E2E"
+    dependsOn: ${{ parameters.dependsOn }}
+    pool:
+      name: $(BUILD_POOL_NAME_DEFAULT)
+    steps:
+      - task: AzureCLI@1
+        inputs:
+          azureSubscription: ${{ parameters.sub }}
+          scriptLocation: "inlineScript"
+          scriptType: "bash"
+          addSpnToEnvironment: true
+          inlineScript: |
+            set -e
+            make -C ./hack/aks set-kubeconf AZCLI=az CLUSTER=${{ parameters.clusterName }}-$(make revision)
+
+            # sig-release provides test suite tarball(s) per k8s release. Just need to provide k8s version "v1.xx.xx"
+            # pulling k8s version from AKS.
+            eval k8sVersion="v"$( az aks show -g ${{ parameters.clusterName }}-$(make revision) -n ${{ parameters.clusterName }}-$(make revision) --query "currentKubernetesVersion")
+            curl -L https://dl.k8s.io/$k8sVersion/kubernetes-test-linux-amd64.tar.gz -o ./kubernetes-test-linux-amd64.tar.gz
+
+            # https://github.com/kubernetes/sig-release/blob/master/release-engineering/artifacts.md#content-of-kubernetes-test-system-archtargz-on-example-of-kubernetes-test-linux-amd64targz-directories-removed-from-list
+            # explictly unzip and strip directories from ginkgo and e2e.test
+            tar -xvzf kubernetes-test-linux-amd64.tar.gz --strip-components=3 kubernetes/test/bin/ginkgo kubernetes/test/bin/e2e.test
+
+        displayName: "Setup Environment"
+      - ${{ if eq(parameters.datapath, true) }}:
+        - template: ../k8s-e2e/k8s-e2e-step-template.yaml
+          parameters:
+            testName: Datapath
+            name: datapath
+            clusterName: ${{ parameters.clusterName }}
+            ginkgoFocus: '(.*).Networking.should|(.*).Networking.Granular|(.*)kubernetes.api'
+            ginkgoSkip: 'SCTP|Disruptive|Slow|hostNetwork|kube-proxy|IPv6'
+            os: ${{ parameters.os }}
+            processes: 8
+            attempts: 10
+      - ${{ if eq(parameters.dns, true) }}:
+        - template: ../k8s-e2e/k8s-e2e-step-template.yaml
+          parameters:
+            testName: DNS
+            name: dns
+            clusterName: ${{ parameters.clusterName }}
+            ginkgoFocus: '\[sig-network\].DNS.should'
+            ginkgoSkip: 'resolv|256 search'
+            os: ${{ parameters.os }}
+            processes: 8
+            attempts: 3
+      - ${{ if eq(parameters.portforward, true) }}:
+        - template: ../k8s-e2e/k8s-e2e-step-template.yaml
+          parameters:
+            testName: Kubectl Portforward
+            name: portforward
+            clusterName: ${{ parameters.clusterName }}
+            ginkgoFocus: '\[sig-cli\].Kubectl.Port'
+            ginkgoSkip: ''
+            os: ${{ parameters.os }}
+            processes: 8
+            attempts: 3
+      - ${{ if eq(parameters.loadBalancer, true) }}:
+        - template: ../k8s-e2e/k8s-e2e-step-template.yaml
+          parameters:
+            testName: Load Balancers
+            name: load
+            clusterName: ${{ parameters.clusterName }}
+            ginkgoFocus: '\[sig-network\].LoadBalancers'
+            ginkgoSkip: 'ESIPP|Serial'
+            os: ${{ parameters.os }}
+            processes: 8
+            attempts: 3
+      - ${{ if eq(parameters.service, true) }}:
+        - template: ../k8s-e2e/k8s-e2e-step-template.yaml
+          parameters:
+            testName: Service Conformance
+            name: service
+            clusterName: ${{ parameters.clusterName }}
+            ginkgoFocus: 'Services.*\[Conformance\].*'
+            ginkgoSkip: ''
+            os: ${{ parameters.os }}
+            processes: 8
+            attempts: 3
+      - ${{ if eq(parameters.hostport, true) }}:
+        - template: ../k8s-e2e/k8s-e2e-step-template.yaml
+          parameters:
+            testName: Host Port
+            name: hostport
+            clusterName: ${{ parameters.clusterName }}
+            ginkgoFocus: '\[sig-network\](.*)HostPort|\[sig-scheduling\](.*)hostPort'
+            ginkgoSkip: 'SCTP|exists conflict' # Skip slow 5 minute test
+            os: ${{ parameters.os }}
+            processes: 1 # Has a short serial test
+            attempts: 3
+      - ${{ if and(eq(parameters.hybridWin, true), eq(parameters.os, 'windows')) }}:
+        - template: ../k8s-e2e/k8s-e2e-step-template.yaml
+          parameters:
+            testName: Hybrid Network
+            name: hybrid
+            clusterName: ${{ parameters.clusterName }}
+            ginkgoFocus: '\[sig-windows\].Hybrid'
+            ginkgoSkip: ''
+            os: ${{ parameters.os }}
+            processes: 8
+            attempts: 3
diff --git a/.pipelines/cni/k8s-e2e/k8s-e2e-step-template.yaml b/.pipelines/cni/k8s-e2e/k8s-e2e-step-template.yaml
@@ -0,0 +1,53 @@
+parameters:
+  testName: ""
+  name: ""
+  clusterName: ""
+  ginkgoFocus: ""
+  ginkgoSkip: ""
+  os: ""
+  processes: "" # Number of parallel processes
+  attempts: ""
+
+
+steps:
+  - script: |
+      set -ex
+
+      # ginkgoSkip cant handle only |LinuxOnly. Need to have check
+      if ${{ lower(and(ge(length(parameters.ginkgoSkip), 1), eq(parameters.os, 'windows'))) }}
+      then
+        SKIP="|LinuxOnly"
+      elif ${{ lower(eq(parameters.os, 'windows')) }}
+      then
+        SKIP="LinuxOnly"
+      fi
+
+      # Taint Linux nodes so that windows tests do not run on them
+      if ${{ lower(eq(parameters.os, 'windows')) }}
+      then
+        kubectl taint nodes -l kubernetes.azure.com/mode=system node-role.kubernetes.io/control-plane:NoSchedule
+      fi
+
+      # Depreciating flags. Change once k8s minimum version supported is > 1.24
+      # nodes -> procs
+      # flakeAttempts -> flake-attempts
+      # dryRun -> dry-run
+
+      ./ginkgo --nodes=${{ parameters.processes }} \
+      ./e2e.test -- \
+      --num-nodes=2 \
+      --provider=skeleton \
+      --ginkgo.focus='${{ parameters.ginkgoFocus }}' \
+      --ginkgo.skip="${{ parameters.ginkgoSkip }}$SKIP" \
+      --ginkgo.flakeAttempts=${{ parameters.attempts }} \
+      --ginkgo.v \
+      --node-os-distro=${{ parameters.os }} \
+      --kubeconfig=$HOME/.kube/config
+
+      # Untaint Linux nodes once testing is complete
+      if ${{ lower(eq(parameters.os, 'windows')) }}
+      then
+        kubectl taint nodes -l kubernetes.azure.com/mode=system node-role.kubernetes.io/control-plane:NoSchedule-
+      fi
+    name: ${{ parameters.name }}
+    displayName: k8s E2E - ${{ parameters.testName }}
diff --git a/.pipelines/cni/load-test-templates/create-cluster-template.yaml b/.pipelines/cni/load-test-templates/create-cluster-template.yaml
@@ -0,0 +1,20 @@
+parameters:
+  clusterType: ""
+  clusterName: ""
+  nodeCount: ""
+  vmSize: ""
+  windowsVMSize: ""
+
+steps:
+  - task: AzureCLI@1
+    inputs:
+      azureSubscription: $(TEST_SUB_SERVICE_CONNECTION)
+      scriptLocation: "inlineScript"
+      scriptType: "bash"
+      addSpnToEnvironment: true
+      inlineScript: |
+        set -ex
+        make -C ./hack/aks azcfg AZCLI=az REGION=$(LOCATION)
+        make -C ./hack/aks ${{ parameters.clusterType }} AZCLI=az REGION=$(LOCATION) SUB=$(SUBSCRIPTION_ID) CLUSTER=${{ parameters.clusterName }}-$(make revision) NODE_COUNT=${{ parameters.nodeCount }} VM_SIZE=${{ parameters.vmSize }} WINDOWS_VM_SKU=${{ parameters.windowsVMSize }} WINDOWS_USERNAME=${WINDOWS_USERNAME} WINDOWS_PASSWORD=${WINDOWS_PASSWORD}
+    name: "CreateAksCluster"
+    displayName: "Create AKS Cluster"
diff --git a/.pipelines/cni/load-test-templates/pod-deployment-template.yaml b/.pipelines/cni/load-test-templates/pod-deployment-template.yaml
@@ -0,0 +1,20 @@
+parameters:
+  clusterName: ""
+  scaleup: 1000
+  os: ""
+  iterations: 4
+
+steps:
+  - task: AzureCLI@1
+    displayName: "Pod Deployment"
+    inputs:
+      azureSubscription: $(TEST_SUB_SERVICE_CONNECTION)
+      scriptLocation: "inlineScript"
+      scriptType: "bash"
+      addSpnToEnvironment: true
+      inlineScript: |
+        set -ex
+        az extension add --name aks-preview
+        make -C ./hack/aks set-kubeconf AZCLI=az CLUSTER=${{ parameters.clusterName }}-$(make revision)
+        cd test/integration/load
+        go test -timeout 30m -tags load -run ^TestLoad$ -tags=load -iterations=${{ parameters.iterations }} -scaleup=${{ parameters.scaleup }} -os=${{ parameters.os }}