Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reducing Kubernetes Build Maintenance #2420

Closed
12 tasks done
BenTheElder opened this issue Feb 4, 2021 · 40 comments
Closed
12 tasks done

Reducing Kubernetes Build Maintenance #2420

BenTheElder opened this issue Feb 4, 2021 · 40 comments
Assignees
Labels
area/release-eng Issues or PRs related to the Release Engineering subproject kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. sig/release Categorizes an issue or PR as relevant to SIG Release. sig/testing Categorizes an issue or PR as relevant to SIG Testing. stage/stable Denotes an issue tracking an enhancement targeted for Stable/GA status tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team

Comments

@BenTheElder
Copy link
Member

BenTheElder commented Feb 4, 2021

Enhancement Description

Additional detailed PR listing below #2420 (comment)

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/assign
/sig testing
/sig release

@k8s-ci-robot k8s-ci-robot added sig/testing Categorizes an issue or PR as relevant to SIG Testing. sig/release Categorizes an issue or PR as relevant to SIG Release. labels Feb 4, 2021
@BenTheElder BenTheElder changed the title Reducing Build Maintenance Reducing Kubernetes Build Maintenance Feb 4, 2021
@justaugustus justaugustus added this to Backlog in Release Engineering via automation Feb 4, 2021
@justaugustus justaugustus moved this from Backlog to To Do in 1.21 in Release Engineering Feb 4, 2021
@BenTheElder BenTheElder added this to the v1.21 milestone Feb 4, 2021
@annajung annajung added the tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team label Feb 5, 2021
@spiffxp
Copy link
Member

spiffxp commented Feb 8, 2021

Initial draft PR: #2421

@BenTheElder
Copy link
Member Author

BenTheElder commented Feb 8, 2021

followup for implementable #2469

@BenTheElder
Copy link
Member Author

BenTheElder commented Feb 9, 2021

Discussed today in SIG Testing meeting, no concerns there. (this has also been circulated in SIG Testing and Release previously).

@BenTheElder
Copy link
Member Author

BenTheElder commented Feb 25, 2021

UPDATE 02/24/2021

We moved pull-kubernetes-e2e-kind-ipv6 to build without bazel, so far so good. kubernetes/test-infra#21009

We're now running pull-kubernetes-unit (non blocking) which runs ~make test with the race detector enabled as an approximate equivilant to pull-kubernetes-bazel-test (our heaviest bazel dependency in CI). This is working well. kubernetes/test-infra#21008

Nominally speaking we are on track to not depend on this build in CI, opening the path to notify and remove.


However there are a handful of test targets in pull-kubernetes-bazel-test that are not golang unit tests and therefore will not be covered by make test.
I produced the following list by inspecting the junit output and ignoring :go_default_test targets (go unit tests under bazel).

  • 1. //cluster:common_test

Per BUILD:

# These tests just verify that bash can interpret the file.

This one I think we can ignore. This doesn't appear to function as intended and we don't normally do this.
EDIT: kubernetes/kubernetes#99486 to remove. See details there.

  • 2. //cluster:kube-util_test

Per BUILD:

# These tests just verify that bash can interpret the file.

This one I think we can ignore. This doesn't appear to function as intended and we don't normally do this.
EDIT: kubernetes/kubernetes#99486 to remove. See details there.

  • 3. //hack/boilerplate:boilerplate_test IMPORTANT

Python script that ensures the boilerplate headers are in place.

This one is already covered by pull-kubernetes-verify / make verify so we don't need to do anything.

  • 4. //pkg/util:verify-util-pkg

This is a sh_test that basically ensures that ./pkg/util/ only contains code in subdirectories, not ./pkg/util/foo.go

I'm not sure why we couldn't just let approvers handle this, it seems a bit heavyweight, but if we still need it we can move it to pull-kubernetes-verify / make verify.
EDIT: Another approach we could take here is to use prow's blockade plugin to block ^pkg/util/[^/]*\.go by automatically adding do-not-merge/blocked-paths label and posting a configurable explanation comment.
EDIT: Fixed by kubernetes/test-infra#21037 (blockade)
EDIT: kubernetes/kubernetes#99486 to remove.

See: kubernetes/kubernetes#15634

  • 5. //test/conformance:conformance_test IMPORTANT

This is a sh_test that ensures the generated conformance.yaml is up to date.

We should probably rework this to run under make verify / pull-kubernetes-verify.
EDIT: fixed in kubernetes/kubernetes#99549 to work under make verify and hack/update-conformance-yaml.sh

@BenTheElder BenTheElder added the lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. label Feb 25, 2021
@dims
Copy link
Member

dims commented Feb 25, 2021

cc @navidshaikh

@BenTheElder
Copy link
Member Author

BenTheElder commented Feb 26, 2021

Update 02/25/2021

  1. pull-kubernetes-unit is looking pretty good https://prow.k8s.io/?job=pull-kubernetes-unit
    pkg/controlplane sometimes times out at the current 240s limit we're setting for make test, we could raise it, but we should fix that test Umbrella issue: slow/timing-out unit tests kubernetes#98486
=== FAIL: pkg/controlplane  (0.00s)
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
	A future release of golang/protobuf will delete this package,
	which has long been excluded from the compatibility promise.
FAIL	k8s.io/kubernetes/pkg/controlplane	240.803s
  1. pull-kubernetes-e2e-kind-ipv6 is working fine building with make with ~30m total runtime https://prow.k8s.io/?job=pull-kubernetes-e2e-kind-ipv6. Writing a PR to switch more jobs currently.

  2. Filed add blockade to kubernetes for pkg/util test-infra#21037 for 4) on the list above Reducing Kubernetes Build Maintenance #2420 (comment), checking that one off.

  3. Filed Cleanup old test targets kubernetes#99486 to cleanup 1) 2) 4) from above.

@spiffxp
Copy link
Member

spiffxp commented Sep 21, 2021

Could also remove code that detects no re-introduction of bazel build files

Do we need to maintain greenhouse at the same high instance setting? Or could we scale it down a bit?

@spiffxp
Copy link
Member

spiffxp commented Sep 22, 2021

/priority important-longterm

@k8s-ci-robot k8s-ci-robot added the priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. label Sep 22, 2021
@BenTheElder
Copy link
Member Author

BenTheElder commented Sep 23, 2021

n.b.: #2420 (comment) is a note from the bi-weekly SIG testing meeting, so I agree with my own comment ;-)

just waiting for 1.19 turndown from SIG release

@BenTheElder
Copy link
Member Author

BenTheElder commented Nov 4, 2021

More cleanup PRs:

We are on-track to stable with the turn down of 1.19 related Kubernetes project resources. There's a bit more to do but we're nearly done.

Next up: We run a reasonably large custom bazel cache (greenhouse) instance in each CI cluster primarily aimed at Kubernetes/Kubernetes, we should either turn this down or at least reduce the capacity / operational costs significantly.

@kevindelgado
Copy link
Contributor

kevindelgado commented Nov 8, 2021

Thanks for the update @BenTheElder! Still gonna give you the copy/pasta with all the deets:

Hi @BenTheElder 👋

Checking in once more as we approach 1.23 code freeze at 6:00 pm PST on Tuesday, November 16.

Please ensure the following items are completed:

  • All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
  • All PRs are fully merged by the code freeze deadline.
  • Have a documentation placeholder PR open by Thursday, November 18.

As always, we are here to help should questions come up.

Thanks!!

@spiffxp
Copy link
Member

spiffxp commented Nov 24, 2021

We're past code freeze and test freeze at this point, I'm hoping / assuming nothing remains to be done in kubernetes/kubernetes itself. What tasks do we need to complete to call this done?

@BenTheElder
Copy link
Member Author

BenTheElder commented Nov 24, 2021

There's some pretty minor stuff but we could probably close it out now that we've done the rest.

  • turn down greenhouse test-infra#24247
  • there's still a few errant references in kubernetes/kubernetes, off the top of my head the publishing bot is still find-and-removing bazel related files still, because we took longer to turn down the 1.19 branch (and also missed turning up 1.23 ...)

@BenTheElder
Copy link
Member Author

BenTheElder commented Nov 24, 2021

none of those references are in production release binary code or release / build scripts though, just in tests, publishing bot config, that sort of thing.

@gracenng
Copy link
Member

gracenng commented Jan 9, 2022

Hi @BenTheElder ,

1.24 Enhancements Lead here. Could you please update this enhancements's KEP status to implemented, then close this issue?

Thanks :)

@gracenng gracenng added tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team and removed tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team labels Jan 9, 2022
@gracenng gracenng removed this from the v1.23 milestone Jan 9, 2022
@BenTheElder
Copy link
Member Author

BenTheElder commented Feb 4, 2022

filed #3209, we can follow up with any additional minor cleanup.

@BenTheElder BenTheElder moved this from Blocked to Done in sig-testing issues Feb 4, 2022
Release Engineering automation moved this from In Review (Mostly PRs) to Done Feb 4, 2022
@BenTheElder
Copy link
Member Author

BenTheElder commented Feb 4, 2022

@gracenng sorry for the delay, this is done!

coutinhop pushed a commit to coutinhop/kind that referenced this issue Aug 18, 2022
Preparing to rework node image building, dropping bazel as we no longer have a reason to support this.

See: kubernetes/enhancements#2420
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/release-eng Issues or PRs related to the Release Engineering subproject kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. sig/release Categorizes an issue or PR as relevant to SIG Release. sig/testing Categorizes an issue or PR as relevant to SIG Testing. stage/stable Denotes an issue tracking an enhancement targeted for Stable/GA status tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team
Projects
Development

No branches or pull requests