Kubernetes CI Policy: release-blocking jobs must run in dedicated cluster #18549

spiffxp · 2020-07-30T21:56:21Z

Part of #18551

Why it's necessary:

we believe declaring Guaranteed Pod QOS jobs may not be defended against Best Effort or Burstable Pods hogging all resources on the same node
a cluster that only has Guaranteed pods is far more likely to respect resource requirements

What decisions do we need to make:

For all of the jobs being migrated to k8s-infra-prow-build, we're going to use the same node pool as everything else. To pin to a dedicated node pool will require much more boilerplate (or possibly augmenting prow's preset feature). If after migrating everything we find we do need a dedicated nodepool, then we'll pay the added cost.
Jobs that can't be migrated quickly will remain in google.com-owned k8s-prow-builds until we can overcome whatever obstacles are keeping them there. This means no community visibility into resource consumption, and test-infra-oncall will have to be relied upon.

Boskos projects to be added:

Identify quota changes needed for gpu jobs, create pool of gpu projects k8s.io#1095 - gpu-project

Jobs to be migrated

TODO:

Add test to enforce critical jobs run on k8s-infra-prow-build #18576 - write a test that enforces this policy, but logs instead of errors
determine which jobs may not be able to be migrated quickly, decide if / how to mitigate
migrate all jobs that can be migrated quickly
migrate jobs blocked on Migrate jobs away from gs://kubernetes-release-dev k8s.io#846
migrate jobs that need to be migrated away from RBE
jobs: enforce kubernetes CI policy for blocking jobs #20989 - make policy enforcement test blocking (fails instead of logs)
declare we are finished (or done enough to move on)

BenTheElder · 2020-07-31T04:52:18Z

NodePool is probably fine, though it has the downside that you have to explicitly set a matching node selector and taint toleration in every prowjob podspec if you want to actually make it dedicated. there's no abstraction for this, and they're a little verbose.

we've done this before when experimenting with ubuntu nodes for kind IPv6 before we'd standardized on that.

spiffxp · 2020-08-01T04:06:17Z

/sig testing
/sig release
/wg k8s-infra
/area release-eng
/area jobs

spiffxp · 2020-08-08T20:10:46Z

Current status (not counting issues that are held open to monitor jobs)

spiffxp@spiffxp-macbookpro:test-infra (master %)$ go test -v -count=1 ./config/tests/jobs
# ...
=== RUN   TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: ci-kubernetes-build: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: ci-kubernetes-build-1-19: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: ci-kubernetes-build-fast: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: ci-kubernetes-build-stable1: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: ci-kubernetes-build-stable2: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: ci-kubernetes-build-stable3: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: periodic-kubernetes-bazel-build-1-16: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: periodic-kubernetes-bazel-build-1-17: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: periodic-kubernetes-bazel-build-1-18: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: periodic-kubernetes-bazel-build-1-19: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: periodic-kubernetes-bazel-build-master: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: periodic-kubernetes-bazel-test-1-16: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: periodic-kubernetes-bazel-test-1-17: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: periodic-kubernetes-bazel-test-1-18: should run on cluster: k8s-infra-prow-build, found: default
    TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild: jobs_test.go:1084: periodic-kubernetes-bazel-test-1-19: should run on cluster: k8s-infra-prow-build, found: default
--- PASS: TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild (0.02s)

Will need to decide how we'd like to define exceptions for the build jobs, and/or dedicate resources to them.

Duplicate of ci-ingress-gce-e2e that validates the job can run on k8s-infra-prow-build. Ref: kubernetes/k8s.io#1093 Part of : kubernetes#18549 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

justaugustus · 2020-11-04T14:24:56Z

Assigning Dan to TL completion of this.
/assign @hasheddan

spiffxp · 2020-11-05T01:27:17Z

Updated description to enumerate all release-blocking jobs and the umbrella issues tracking their progress or the PRs that migrated them

Ref : kubernetes#19483 Part of: kubernetes#18549 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

spiffxp · 2021-01-08T16:50:55Z

Current status...

spiffxp@spiffxp-macbookpro:test-infra (master %)$ go test -v -count=1 ./config/tests/jobs/...
# ...
=== RUN   TestKubernetesReleaseBlockingJobsShouldRunOnK8sInfraProwBuild
    jobs_test.go:1049: ci-kubernetes-build: should run on cluster: k8s-infra-prow-build, found: default
    jobs_test.go:1049: ci-kubernetes-build-1-19: should run on cluster: k8s-infra-prow-build, found: default
    jobs_test.go:1049: ci-kubernetes-build-1-20: should run on cluster: k8s-infra-prow-build, found: default
    jobs_test.go:1049: ci-kubernetes-build-stable2: should run on cluster: k8s-infra-prow-build, found: default
    jobs_test.go:1049: ci-kubernetes-build-stable3: should run on cluster: k8s-infra-prow-build, found: default
    jobs_test.go:1049: periodic-kubernetes-bazel-build-1-17: should run on cluster: k8s-infra-prow-build, found: default
    jobs_test.go:1049: periodic-kubernetes-bazel-build-1-18: should run on cluster: k8s-infra-prow-build, found: default
    jobs_test.go:1049: periodic-kubernetes-bazel-build-1-19: should run on cluster: k8s-infra-prow-build, found: default
    jobs_test.go:1049: periodic-kubernetes-bazel-build-1-20: should run on cluster: k8s-infra-prow-build, found: default
    jobs_test.go:1049: periodic-kubernetes-bazel-build-master: should run on cluster: k8s-infra-prow-build, found: default

We have "canary" versions of these jobs that run on k8s-infra-prow-build, and write to gs://k8s-release-dev.

However, there is much out there that depends on the existing jobs that write to gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images (ref: #19483 (comment)). I'm considering a KEP for deprecation/migration/removal.

I think full completion of the deprecation process is out of scope for this issue. So I would like us to identify what the minimum set of jobs/scripts/etc should be pulling from gs://k8s-release-dev to call this done.

spiffxp · 2021-02-02T00:38:30Z

/milestone v1.21

spiffxp · 2021-02-08T17:37:52Z

/priority important-soon

spiffxp · 2021-02-23T07:59:12Z

I think the following PRs can close this out:

Log or enforce kubernetes jobs pull from k8s-infra buckets #20885 - enforced that all release-blocking jobs pull from gs://k8s-release-dev
Move periodic-kubernetes-bazel-build jobs to release-informing #20963 - proposes moving bazel-build jobs to release-informing
Deprecate google-hosted kubernetes-build jobs #20964 - proposes renaming build jobs to build-deprecated and moving to release-informing, renaming build-canary jobs to build and moving to release-blocking
jobs: enforce kubernetes CI policy for blocking jobs #20989 - TestKubernetesReleaseBlockingJobsCIPolicy passes for all release-blocking jobs and will enforce any new additions

spiffxp · 2021-02-23T08:00:48Z

/assign
/assign @justaugustus
Can you think of anything else that needs to happen to call this done?

spiffxp · 2021-02-24T17:48:48Z

Tangentially related: followup work to deprecate google-hosted artifacts related to release-blocking builds will be tracked under kubernetes/k8s.io#1571

spiffxp · 2021-02-24T18:33:34Z

/close
Calling this done!

k8s-ci-robot · 2021-02-24T18:33:42Z

@spiffxp: Closing this issue.

In response to this:

/close
Calling this done!

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This was referenced Jul 30, 2020

Kubernetes CI Policy: merge-blocking jobs must run in dedicated cluster #18550

Closed

Kubernetes CI Policy (Umbrella issue) #18551

Open

Migrate release-master-blocking jobs to k8s-infra-prow-build kubernetes/k8s.io#841

Closed

spiffxp mentioned this issue Aug 1, 2020

Add test to enforce critical jobs run on k8s-infra-prow-build #18576

Merged

k8s-ci-robot added sig/testing Categorizes an issue or PR as relevant to SIG Testing. sig/release Categorizes an issue or PR as relevant to SIG Release. wg/k8s-infra area/release-eng Issues or PRs related to the Release Engineering subproject area/jobs labels Aug 1, 2020

This was referenced Oct 7, 2020

release-blocking jobs must run in dedicated cluster: ci-kubernetes-build #19483

Closed

release-blocking jobs must run in dedicated cluster: ci-kubernetes-build-fast #19484

Closed

ameukam mentioned this issue Oct 14, 2020

Create ci-kubernetes-build-canary job #19574

Merged

ameukam mentioned this issue Oct 29, 2020

Add canary job for ci-ingress-gce-e2e #19762

Merged

justaugustus mentioned this issue Nov 4, 2020

releng: Teach jobs using latest-fast marker to extract from K8s Infra #19841

Merged

k8s-ci-robot assigned hasheddan Nov 4, 2020

justaugustus mentioned this issue Nov 5, 2020

ci-kubernetes-e2e-gce-scale-performance is continuously testing the same, stale k8s version since 10-29 #19838

Closed

ameukam added a commit to ameukam/test-infra that referenced this issue Nov 10, 2020

Add canary job for ci-kubernetes-build jobs variants.

a769ca3

Ref : kubernetes#19483 Part of: kubernetes#18549 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

ameukam mentioned this issue Nov 10, 2020

Add canary jobs for ci-kubernetes-build job variants. #19904

Merged

ameukam added a commit to ameukam/test-infra that referenced this issue Nov 10, 2020

Add canary job for ci-kubernetes-build jobs variants.

c13249d

Ref : kubernetes#19483 Part of: kubernetes#18549 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

ameukam added a commit to ameukam/test-infra that referenced this issue Nov 10, 2020

Add canary job for ci-kubernetes-build jobs variants.

3d2c36e

Ref : kubernetes#19483 Part of: kubernetes#18549 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

ameukam added a commit to ameukam/test-infra that referenced this issue Nov 10, 2020

Add canary job for ci-kubernetes-build jobs variants.

9f0d9c8

Ref : kubernetes#19483 Part of: kubernetes#18549 Signed-off-by: Arnaud Meukam <ameukam@gmail.com>

justaugustus mentioned this issue Nov 11, 2020

Enforce / monitor / document version marker usage #19922

Closed

k8s-ci-robot added this to the v1.21 milestone Feb 2, 2021

k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Feb 8, 2021

This was referenced Feb 23, 2021

Move periodic-kubernetes-bazel-build jobs to release-informing #20963

Merged

Deprecate google-hosted kubernetes-build jobs #20964

Merged

k8s-ci-robot assigned justaugustus and spiffxp Feb 23, 2021

spiffxp mentioned this issue Feb 24, 2021

jobs: enforce kubernetes CI policy for blocking jobs #20989

Merged

k8s-ci-robot closed this as completed Feb 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubernetes CI Policy: release-blocking jobs must run in dedicated cluster #18549

Kubernetes CI Policy: release-blocking jobs must run in dedicated cluster #18549

spiffxp commented Jul 30, 2020 •

edited

Loading

BenTheElder commented Jul 31, 2020

spiffxp commented Aug 1, 2020

spiffxp commented Aug 8, 2020

justaugustus commented Nov 4, 2020

spiffxp commented Nov 5, 2020

spiffxp commented Jan 8, 2021

spiffxp commented Feb 2, 2021

spiffxp commented Feb 8, 2021

spiffxp commented Feb 23, 2021 •

edited

Loading

spiffxp commented Feb 23, 2021

spiffxp commented Feb 24, 2021

spiffxp commented Feb 24, 2021

k8s-ci-robot commented Feb 24, 2021

Kubernetes CI Policy: release-blocking jobs must run in dedicated cluster #18549

Kubernetes CI Policy: release-blocking jobs must run in dedicated cluster #18549

Comments

spiffxp commented Jul 30, 2020 • edited Loading

BenTheElder commented Jul 31, 2020

spiffxp commented Aug 1, 2020

spiffxp commented Aug 8, 2020

justaugustus commented Nov 4, 2020

spiffxp commented Nov 5, 2020

spiffxp commented Jan 8, 2021

spiffxp commented Feb 2, 2021

spiffxp commented Feb 8, 2021

spiffxp commented Feb 23, 2021 • edited Loading

spiffxp commented Feb 23, 2021

spiffxp commented Feb 24, 2021

spiffxp commented Feb 24, 2021

k8s-ci-robot commented Feb 24, 2021

spiffxp commented Jul 30, 2020 •

edited

Loading

spiffxp commented Feb 23, 2021 •

edited

Loading