ci: Improve CNI|NPM integration test #2498

jpayne3506 · 2024-01-10T22:53:49Z

Reason for Change:

Window NPM testing takes an increased amount of time to complete then linux due to custom changes upstream. kubernetes/kubernetes@master...huntergregory:kubernetes:sleep-before-probing

Fixed taint being left on windows nodes if job timeout occurs.

Issue Fixed:

Requirements:

uses conventional commit messages
includes documentation
adds unit tests
relevant PR labels added

Notes:

huntergregory · 2024-01-10T23:49:59Z

.pipelines/npm/npm-cni-integration-test.yaml

    displayName: "NPM k8s E2E"
    dependsOn: ${{ parameters.dependsOn }}
    condition: and( and( not(canceled()), not(failed()) ), ${{ or(contains(parameters.os_version, '2022'), eq(parameters.os, 'linux') ) }} , or( contains(variables.CONTROL_SCENARIO, 'npm') , contains(variables.CONTROL_SCENARIO, 'all') ) )
+    timeoutInMinutes: 180 # This is for testing windows, due to the 3m between the 14 tests -> results in 42m of wasted time


Noticing that that change to the k8s fork was in July fyi

If desired, we could also create another branch with a smaller timeout. I don't know why we changed to two minutes, but I feel like it's probably unnecessary at this point (Perhaps there was a latency issue in HNS that's been fixed)

I tested with a shorter timeout from your fork and got similar results. (Roughly 20 minutes faster). I would rather keep this in and have consistent results and avoid flakes due to HNS variability.

Does it actually take 180 mins to complete ? Can we pull npm test in parallel to all other tests (if not already)

It takes roughly 75 - 105 minutes to complete... Main problem is that it takes 140-160 minutes to fail. If a job timeout happens before failure then we lose a considerable amount of results and debugging logs.

pull npm test in parallel

Do you mean do we create the e2e binary so we can pull the same binary in parallel? If so, yes. See artifact: npm*

vipul-21 · 2024-01-11T03:33:04Z

.pipelines/npm/npm-cni-integration-test.yaml

    displayName: "NPM k8s E2E"
    dependsOn: ${{ parameters.dependsOn }}
    condition: and( and( not(canceled()), not(failed()) ), ${{ or(contains(parameters.os_version, '2022'), eq(parameters.os, 'linux') ) }} , or( contains(variables.CONTROL_SCENARIO, 'npm') , contains(variables.CONTROL_SCENARIO, 'all') ) )
+    timeoutInMinutes: 180 # This is for testing windows, due to the 3m between the 14 tests -> results in 42m of wasted time


Does it actually take 180 mins to complete ? Can we pull npm test in parallel to all other tests (if not already)

deps: bump github.com/spf13/viper from 1.16.0 to 1.18.2 (#2495) Bumps [github.com/spf13/viper](https://github.com/spf13/viper) from 1.16.0 to 1.18.2. - [Release notes](https://github.com/spf13/viper/releases) - [Commits](spf13/viper@v1.16.0...v1.18.2) --- updated-dependencies: - dependency-name: github.com/spf13/viper dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> change config to be uniform ci: increase dependabot PR limit to 10 (#2497) ci: constrain cilium to linux on Release Test Pipeline (#2489) ci: constrain cilium to linux ci: Improve CNI|NPM integration test (#2498) * ci: increase timeout for CNI|NPM intergration test * add: retry to generate NPM logs deps: bump golang.org/x/sys from 0.15.0 to 0.16.0 (#2500) Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.15.0 to 0.16.0. - [Commits](golang/sys@v0.15.0...v0.16.0) --- updated-dependencies: - dependency-name: golang.org/x/sys dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> deps: bump github.com/prometheus/client_golang from 1.16.0 to 1.18.0 (#2501) Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.16.0 to 1.18.0. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md) - [Commits](prometheus/client_golang@v1.16.0...v1.18.0) --- updated-dependencies: - dependency-name: github.com/prometheus/client_golang dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> deps: bump google.golang.org/protobuf from 1.31.0 to 1.32.0 (#2502) Bumps google.golang.org/protobuf from 1.31.0 to 1.32.0. --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> deps: bump golang.org/x/crypto from 0.17.0 to 0.18.0 (#2492) Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.17.0 to 0.18.0. - [Commits](golang/crypto@v0.17.0...v0.18.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> deps: bump k8s.io/klog/v2 from 2.110.1 to 2.120.0 (#2503) Bumps [k8s.io/klog/v2](https://github.com/kubernetes/klog) from 2.110.1 to 2.120.0. - [Release notes](https://github.com/kubernetes/klog/releases) - [Changelog](https://github.com/kubernetes/klog/blob/main/RELEASE.md) - [Commits](kubernetes/klog@v2.110.1...v2.120.0) --- updated-dependencies: - dependency-name: k8s.io/klog/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> ci: verbose output for make test-load (#2481) ci: make test-load verbose output fix review comments deps: bump google.golang.org/grpc from 1.59.0 to 1.60.1 (#2494) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.59.0 to 1.60.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.59.0...v1.60.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> ci: add OS scale timeout (#2490) ci: add kubeproxy restart before windows k8s conformance testing (#2496) * ci: ensure kubeproxy is running on windows * ci: add set -e Signed-off-by: John Payne <89417863+jpayne3506@users.noreply.github.com> --------- Signed-off-by: John Payne <89417863+jpayne3506@users.noreply.github.com> ci: Improve TestValidateState calls from load package (#2484) * ci: improve TestValidateState * chore: address comments fix: joining multiple dns server lists (#2506) * fix: joining multiple dns server lists * address comments * add 0 check fix: downgrade k8s components to 1.28 (#2505) Signed-off-by: GitHub <noreply@github.com>

* ci: increase timeout for CNI|NPM intergration test * add: retry to generate NPM logs

ci: increase timeout for CNI|NPM intergration test

2502fe7

jpayne3506 added npm Related to NPM. cni Related to CNI. ci Infra or tooling. labels Jan 10, 2024

jpayne3506 self-assigned this Jan 10, 2024

jpayne3506 requested a review from a team as a code owner January 10, 2024 22:53

jpayne3506 requested review from huntergregory and jackieluc January 10, 2024 22:53

add: retry to generate NPM logs

020253d

jpayne3506 force-pushed the jpayne3506/npmload branch from 295cc94 to 020253d Compare January 10, 2024 22:56

jpayne3506 changed the title ~~Jpayne3506/npmload~~ ci: Improve CNI|NPM integration test Jan 10, 2024

jpayne3506 mentioned this pull request Jan 10, 2024

ci: Add Windows CNIv2 to CNI Load Test Pipeline #2279

Merged

4 tasks

jpayne3506 enabled auto-merge January 10, 2024 23:44

huntergregory reviewed Jan 10, 2024

View reviewed changes

huntergregory approved these changes Jan 10, 2024

View reviewed changes

jpayne3506 added this pull request to the merge queue Jan 10, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jan 11, 2024

jpayne3506 added this pull request to the merge queue Jan 11, 2024

vipul-21 approved these changes Jan 11, 2024

View reviewed changes

Merged via the queue into master with commit ba5faa5 Jan 11, 2024

jpayne3506 deleted the jpayne3506/npmload branch January 11, 2024 04:40

matmerr pushed a commit that referenced this pull request Jan 17, 2024

ci: Improve CNI|NPM integration test (#2498)

a124e02

* ci: increase timeout for CNI|NPM intergration test * add: retry to generate NPM logs

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: Improve CNI|NPM integration test #2498

ci: Improve CNI|NPM integration test #2498

Uh oh!

jpayne3506 commented Jan 10, 2024 •

edited

Loading

Uh oh!

huntergregory Jan 10, 2024

Uh oh!

jpayne3506 Jan 10, 2024

Uh oh!

vipul-21 Jan 11, 2024

Uh oh!

jpayne3506 Jan 11, 2024 •

edited

Loading

Uh oh!

Uh oh!

vipul-21 Jan 11, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ci: Improve CNI|NPM integration test #2498

ci: Improve CNI|NPM integration test #2498

Uh oh!

Conversation

jpayne3506 commented Jan 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huntergregory Jan 10, 2024

Choose a reason for hiding this comment

Uh oh!

jpayne3506 Jan 10, 2024

Choose a reason for hiding this comment

Uh oh!

vipul-21 Jan 11, 2024

Choose a reason for hiding this comment

Uh oh!

jpayne3506 Jan 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vipul-21 Jan 11, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jpayne3506 commented Jan 10, 2024 •

edited

Loading

jpayne3506 Jan 11, 2024 •

edited

Loading