ci: [NPM] scale test pipeline using KWOK #1915

huntergregory · 2023-04-14T18:13:08Z

Reason for Change:
Automate NPM scale tests.

Issue Fixed:

Requirements:

uses conventional commit messages
includes documentation
adds unit tests

Notes:

Test Steps

1. Scale Up Large (~1K fake Pods, 30 real Pods)

Create Fake + Real Pods.
Label them.
Add NetPols.

2. Test NPM Bootup Latency and Connectivity

Must achieve initial connectivity within 15 minutes, and apply a new NetworkPolicy within 30 seconds.

Restart NPM to see performance improvements from perf: [WIN-NPM] fast bootup #1900.
Check connectivity. Fail if a timeout is exceeded.
- Create two pinger Pods.
- Check initial connectivity. If unsuccessful, retry until timeout exceeded.
  - Allowed: pinger <-> pinger
  - Allowed: real <-> real
  - Blocked: real <-> pinger
- Add NetPol and assert connectivity. If unsuccessful, retry until timeout exceeded.
  - Allowed: real <-> pinger

3. General CRUD at Medium Scale (~30 fake Pods, 10 real Pods)

Delete previous namespaces (including Pods and Netpols).
Recreate everything in step 1.
Delete and re-add labels.
Delete and re-create Pods.
Delete and re-create NetPols.

4. Test Connectivity After CRUD

Must achieve initial connectivity within 10 minutes, and apply a new NetworkPolicy within 20 seconds.

Check connectivity like in step 2.

.pipelines/npm/npm-conformance-tests.yaml

huntergregory · 2023-04-14T19:35:26Z

/azp run

azure-pipelines · 2023-04-14T19:35:37Z

Azure Pipelines successfully started running 1 pipeline(s), but failed to run 1 pipeline(s).

…ure-container-networking into hgregory/04-14-pipeline

…ing)

huntergregory · 2023-04-19T20:12:42Z

.pipelines/npm/npm-conformance-tests.yaml

@@ -345,14 +345,14 @@ jobs:
          done

          exit $exitCode
-        displayName: "Run Test Suite and Get Logs"
+        displayName: "Run Test Suite and Get Logs - $(PROFILE)"


this change lets us distinguish job failures in our pipeline report

huntergregory · 2023-04-19T20:17:09Z

.pipelines/npm/npm-conformance-tests.yaml

@@ -463,7 +463,169 @@ jobs:
        name: cyclonus
        displayName: "Run Cyclonus Test"
        failOnStderr: false
+        condition: succeeded()


used to run cyclonus even if previous step failed

vakalapa

Make sure you add times on why it is taking almost 2hrs for the connectivity tests to start ?

huntergregory · 2023-05-08T18:22:46Z

Make sure you add times on why it is taking almost 2hrs for the connectivity tests to start ?

Now takes 6 minutes. Kwok wasn't running, so the fake Pods weren't being scheduled

…ure-container-networking into hgregory/04-14-pipeline

This reverts commit 7b21125.

vakalapa · 2023-05-09T21:16:33Z

.pipelines/npm/npm-scale-test.yaml

+            echo "waiting for NPM to start running..."
+            kubectl wait --for=condition=Ready pod -l k8s-app=azure-npm -n kube-system --timeout=20m
+            echo "sleep 1m to let NPM restart in case of bootup failure due to HNS errors"
+            sleep 1m


windows pods can take longer than 1m sometimes it think ?

now sleep for longer

matmerr · 2023-05-09T22:11:04Z

.pipelines/npm/npm-scale-test.yaml

+          inlineScript: |
+            set -e
+            mkdir -p $(System.DefaultWorkingDirectory)/$(RESOURCE_GROUP)-$(PROFILE)
+            ./kwok --kubeconfig ~/.kube/config \


this is pretty useful for other scenarios, maybe after this add a make recipe to attach kwok nodes

huntergregory · 2023-05-09T22:20:02Z

going to hold off on merging until HNS fixes are in prod (probably next week)

* wip * temporarily disable most conf runs * update readme * back to raw yamls and clone the branch to run scale test * fix raw yaml URLs * fix inline script * fix length of rg name * uncomment all conf again * comment out everything unnecessary for testing * remove commented out dependencies * use master branch for pipeline * label nodes * multiple nodes * uncomment rest of conformance pipeline (originally commented for testing) * fix print out for time taken in test-connectivity.sh * fix: run kwok command in background * mkdir for kwok log * try azure cli 1 to fix login error * Revert "try azure cli 1 to fix login error" This reverts commit f1671e3. * move scale test to new pipeline yaml * remove scale test from conformance pipeline yaml * revert name change for cyclonus job * remove unnecessary image build and variable * error codes and display names * change sleep and wait for npm logic * look at directory * use pre-cloned repo * fix directory path * install kubectl first * FIXME: comment out succeeded condition * kubectl binary arg * kubectl for scale test * fix label selector * fix kubectl path * fix kubectl binary arg * fix kwok, more steps * FIXME: temporarily use custom fast image * fix kwok pid and add comment * 10m timeout for connectivity after crud * fix kwok command invocation * bump up timeouts for testing * higher memory limit * add note to connectivity script * fix sed * no need to curl npm yaml * tmp: comment things out to test final step * only check if kwok pods are running, not necessarily ready * Revert "tmp: comment things out to test final step" This reverts commit 7b21125. * update registry keys to fix HNS reliability * update regkey code * sleep to let NPM restart in case of bootup failure * adaptive wait timeout * change some errors to warnings * log date * make sure all pods are labeled * delete and readd labels after deleting pods * tmp: skip large scale up and connectivity check for testing * fix overwrite arg * rename tasks and uncomment things * update command for updating reg key * make timeout logic simpler * back to reg add command for regkeys * official timeouts instead of test values * delete task updating registry keys and stop hardcoding npm image * increase sleep

huntergregory added 2 commits April 14, 2023 11:07

wip

1570c7f

temporarily disable most conf runs

a558f25

huntergregory added npm Related to NPM. ci Infra or tooling. windows labels Apr 14, 2023

huntergregory requested review from matmerr and vakalapa April 14, 2023 18:13

huntergregory commented Apr 14, 2023

View reviewed changes

.pipelines/npm/npm-conformance-tests.yaml Outdated Show resolved Hide resolved

huntergregory added 3 commits April 14, 2023 11:19

update readme

21e8444

back to raw yamls and clone the branch to run scale test

d0638fc

fix raw yaml URLs

a34d959

Merge branch 'master' into hgregory/04-14-pipeline

2335b4c

huntergregory added 5 commits April 14, 2023 12:47

fix inline script

411dfe4

Merge branch 'hgregory/04-14-pipeline' of https://github.com/Azure/az…

bea399d

…ure-container-networking into hgregory/04-14-pipeline

fix length of rg name

a1eff33

uncomment all conf again

a054655

comment out everything unnecessary for testing

507b25b

huntergregory marked this pull request as draft April 14, 2023 20:28

huntergregory added 5 commits April 14, 2023 13:57

remove commented out dependencies

2ea61c0

use master branch for pipeline

dd8ede1

label nodes

3f62591

multiple nodes

4a6f33a

uncomment rest of conformance pipeline (originally commented for test…

90cf575

…ing)

huntergregory commented Apr 19, 2023

View reviewed changes

Merge branch 'master' into hgregory/04-14-pipeline

d30d754

huntergregory marked this pull request as ready for review April 19, 2023 20:18

vakalapa previously approved these changes Apr 20, 2023

View reviewed changes

Merge branch 'master' into hgregory/04-14-pipeline

65b669c

huntergregory added 18 commits May 8, 2023 12:34

only check if kwok pods are running, not necessarily ready

beeeb27

Merge branch 'hgregory/04-14-pipeline' of https://github.com/Azure/az…

31f8c62

…ure-container-networking into hgregory/04-14-pipeline

Revert "tmp: comment things out to test final step"

dea4f2f

This reverts commit 7b21125.

update registry keys to fix HNS reliability

324038c

update regkey code

3913a8b

sleep to let NPM restart in case of bootup failure

c0c8435

adaptive wait timeout

5454707

change some errors to warnings

dda02d0

log date

ccd22e6

make sure all pods are labeled

73caee8

delete and readd labels after deleting pods

d5c9bdf

tmp: skip large scale up and connectivity check for testing

ac82f42

fix overwrite arg

1744113

rename tasks and uncomment things

1171b94

update command for updating reg key

16ee0ac

make timeout logic simpler

26289d6

back to reg add command for regkeys

e130457

official timeouts instead of test values

6c5b8c2

huntergregory requested a review from a team as a code owner May 9, 2023 16:57

delete task updating registry keys and stop hardcoding npm image

362cc83

vakalapa reviewed May 9, 2023

View reviewed changes

increase sleep

e4a46fa

matmerr reviewed May 9, 2023

View reviewed changes

vakalapa approved these changes May 18, 2023

View reviewed changes

vakalapa merged commit ce11a8d into master May 18, 2023
37 of 45 checks passed

vakalapa deleted the hgregory/04-14-pipeline branch May 18, 2023 16:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: [NPM] scale test pipeline using KWOK #1915

ci: [NPM] scale test pipeline using KWOK #1915

huntergregory commented Apr 14, 2023 •

edited

Loading

huntergregory commented Apr 14, 2023

azure-pipelines bot commented Apr 14, 2023

huntergregory Apr 19, 2023

huntergregory Apr 19, 2023

vakalapa left a comment

huntergregory commented May 8, 2023 •

edited

Loading

vakalapa May 9, 2023

huntergregory May 16, 2023

matmerr May 9, 2023

huntergregory commented May 9, 2023

ci: [NPM] scale test pipeline using KWOK #1915

ci: [NPM] scale test pipeline using KWOK #1915

Conversation

huntergregory commented Apr 14, 2023 • edited Loading

Test Steps

1. Scale Up Large (~1K fake Pods, 30 real Pods)

2. Test NPM Bootup Latency and Connectivity

3. General CRUD at Medium Scale (~30 fake Pods, 10 real Pods)

4. Test Connectivity After CRUD

huntergregory commented Apr 14, 2023

azure-pipelines bot commented Apr 14, 2023

huntergregory Apr 19, 2023

Choose a reason for hiding this comment

huntergregory Apr 19, 2023

Choose a reason for hiding this comment

vakalapa left a comment

Choose a reason for hiding this comment

huntergregory commented May 8, 2023 • edited Loading

vakalapa May 9, 2023

Choose a reason for hiding this comment

huntergregory May 16, 2023

Choose a reason for hiding this comment

matmerr May 9, 2023

Choose a reason for hiding this comment

huntergregory commented May 9, 2023

huntergregory commented Apr 14, 2023 •

edited

Loading

huntergregory commented May 8, 2023 •

edited

Loading