ci: [NPM] scale test pipeline using KWOK (#1915)

* wip

* temporarily disable most conf runs

* update readme

* back to raw yamls and clone the branch to run scale test

* fix raw yaml URLs

* fix inline script

* fix length of rg name

* uncomment all conf again

* comment out everything unnecessary for testing

* remove commented out dependencies

* use master branch for pipeline

* label nodes

* multiple nodes

* uncomment rest of conformance pipeline (originally commented for testing)

* fix print out for time taken in test-connectivity.sh

* fix: run kwok command in background

* mkdir for kwok log

* try azure cli 1 to fix login error

* Revert "try azure cli 1 to fix login error"

This reverts commit f1671e3.

* move scale test to new pipeline yaml

* remove scale test from conformance pipeline yaml

* revert name change for cyclonus job

* remove unnecessary image build and variable

* error codes and display names

* change sleep and wait for npm logic

* look at directory

* use pre-cloned repo

* fix directory path

* install kubectl first

* FIXME: comment out succeeded condition

* kubectl binary arg

* kubectl for scale test

* fix label selector

* fix kubectl path

* fix kubectl binary arg

* fix kwok, more steps

* FIXME: temporarily use custom fast image

* fix kwok pid and add comment

* 10m timeout for connectivity after crud

* fix kwok command invocation

* bump up timeouts for testing

* higher memory limit

* add note to connectivity script

* fix sed

* no need to curl npm yaml

* tmp: comment things out to test final step

* only check if kwok pods are running, not necessarily ready

* Revert "tmp: comment things out to test final step"

This reverts commit 7b21125.

* update registry keys to fix HNS reliability

* update regkey code

* sleep to let NPM restart in case of bootup failure

* adaptive wait timeout

* change some errors to warnings

* log date

* make sure all pods are labeled

* delete and readd labels after deleting pods

* tmp: skip large scale up and connectivity check for testing

* fix overwrite arg

* rename tasks and uncomment things

* update command for updating reg key

* make timeout logic simpler

* back to reg add command for regkeys

* official timeouts instead of test values

* delete task updating registry keys and stop hardcoding npm image

* increase sleep

.pipelines/npm/npm-conformance-tests.yaml

@@ -345,7 +345,7 @@ jobs:
           done
 
           exit $exitCode
-        displayName: "Run Test Suite and Get Logs"
+        displayName: "Run Test Suite and Get Logs ($(PROFILE))"
         failOnStderr: false
 
       - publish: $(System.DefaultWorkingDirectory)/npmLogs_$(AZURE_CLUSTER)
@@ -464,7 +464,7 @@ jobs:
         name: cyclonus
         displayName: "Run Cyclonus Test"
         failOnStderr: false
-        condition: always()
+        condition: succeeded()
 
       - bash: |
           export CLUSTER_NAME=$(RESOURCE_GROUP)-$(PROFILE)