Publishing Kubernetes packages on community infrastructure

[justaugustus]

Enhancement Description

• One-line enhancement description (can be used as a release note): Publishing Kubernetes packages on community infrastructure
• Kubernetes Enhancement Proposal: keps/sig-release/1731-publishing-packages/README.md
• Discussion Link: https://docs.google.com/spreadsheets/d/1Vgpb7HPg8WgoFtDuddaRD6j7jQLoqNK6vUu6Lb0ea28
• Primary contact (assignee): @justaugustus @saschagrunert
• Responsible SIGs: SIG Release
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): v1.28
  • Beta release target (x.y):
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s):
    Add KEP for publishing packages in provisional state #843
    KEP-1731 : Publishing Kubernetes packages on community infrastructure #3434
    KEP-1731: Extend the KEP with the OBS implementation #3750
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

---

Milestones and Tasks Checklist

Milestone 1.0—Code Deliverable

• Success looks like: debs and rpms are in multiple locations (Google infra and third-party hosting service(s)) @detiber
  • Define User Scenarios and Requirements (Configuration, management, monitoring, etc.)
  • Describe proposed Flow (what are key actions that the user will take? How will they take them?)
  • Choose the third-party hosting service/Set up a package host of some sort @detiber, @leonardpahlke helping
    • Identified some options to explore
    • Outreach to reps of those options to see if they'll work with us on this WIP
  • Do a spike test to see if the third-party hosting service meets our needs @detiber, @leonardpahlke helping
  • Publish the debs/rpms to the third-party hosting service @detiber

Milestone 1.0—Documentation Deliverable

• Success looks like: Entire deb/rpms move to community infra plan is outlined in a KEP
  • Update this KEP: @RobertKielty (made a PR July 15, 2022)
  • Review the KEP @saschagrunert, @ameukam, @detiber
  • Approve the KEP
  • Get an understanding of the top-level script: what type of infra we're going to run, who pays for this, who can access to this, what type of account we have, tools available @RobertKielty
    - Script reviewed by Robert Kielty and discussed on SIG Release channel with Ben Elder. This KEP will drive outstanding infra/billing questions for the new implementation.
  • Clarify what "top-level script" means (linked in cell above) @RobertKielty
  • Finalize defining the user personas for this work, with group review and sign-off @detiber, @LappleApple

Milestone 1.0—Risk Mitigation

• Success looks like: All risk matters for this milestone are accounted for, plans in place to mitigate
  • Risk: A new GPG key will be issued incompatible with the existing GPG key managed by Google. This is a breaking change for the existing infrastructures using DEB/RPM packages as installation artifacts

Milestone 1.0—Questions resolved

• Resolve if this is needed: Give access to the sandbox env to release eng to work on it @ameukam, @onlydole helping
• Develop sandbox env (GCP project) and give access to Rel Eng to work on it @ameukam, @onlydole helping
• Needed more discussion: Meeting to specifically discuss if we are doing GPG or if we are owning the packages; then who owns new GPG keys and how and when we store them. @ameukam, @onlydole helping
• Generate new keys : Sign kubernetes system packages with GPG release#2627
• Decide on plan: Be able to issue new gcp trusted keys for entire ecosystem/whoever is relying on those packages to deploy/install/upgrade K8s project
• Find/line up people who can build packages for debian, CentOS to work on build tooling (@upodroid to help build Debian packages)

Milestone 2.0—Code deliverable

• Success looks like: implementation (source code) in our repositories, such that we can publish releases by ourselves without having to rely on Google people at all
  • Define User Scenarios and Requirements (Configuration, management, monitoring, etc.
  • Proposed Flow (what are key actions that the user will take? How will they take them?)
  • Refine the current debs/rpms built by kubepkg
  • Resume discussing how we rethink / rewrite the build tools
  • Rebase the existing kubepkg work into something krel can use directly
  • Krel, the kubernetes release tool, should start including a package build step running the build scripts
  • Phase out publishing to the google controlled host, for having krel publish to our new host for 1-2 releases
  • Users migrate from the Google registry to the community registry, adding the signing key

Milestone 2.0—Documentation Deliverable

• Success looks like: we've published docs on how process works and when it's not working as expected, what to do (for release managers)
  • Expand documentation on the top-level script to drive knowledge transfer
  • Expand documentation on the package build scripts for Debian and RPM to drive knowledge transfer

Milestone 2.0—Questions resolved

• Success looks like: @saschagrunert and @justaugustus have provided context and knowledge about the package build scripts for Debian and rpms
  • Answer: what type of infra we're going to run
  • Answer: who pays for it
  • Answer: what type of account do we have
  • Answer: what tools do/will we have available

Milestone 3.0—Documentation deliverable

• Success looks like: We've updated the google build+publish+release script to download packages uploaded by the upstream release process

Milestone 4.0— Documentation Deliverable

• Success looks like: We've published docs for end users about old infra and new/migrated (more static)
  • Write email to community regarding the required migration to the community repository
  • Send email to community regarding the required migration to the community repository
  • SIG Release informs the community so we don't break people over just one milestone: "these are the keys we'll use to release 1.2X..."

---

Why is this needed?

• It's part of the effort for the community to fully own the infrastructure related to the Kubernetes project.
• Ensure all aspects of project releases can be staffed across companies
• We don’t want to be dependent on a Build Admin any longer (which tends to slow releases)
• We seek more control to eventually choose to build packages for prereleases, extend what packages are shipped, etc.

Why is it needed now?

• The existing infrastructure of this relies on a workstation inside Google offices—essentially one person. It’s not reliable or sustainable.

Who needs it? (User Personas WIP)

• Google Build Team as the existing team building the system packages.
• What does “done” look like/acceptance criteria?
• Automated builds of deb and rpm Kubernetes packages within community infrastructure

Related open issues

• #913

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[jeremyrickard]

/milestone v1.28

[ruheenaansari34]

Hello @justaugustus 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 1:00 UTC on Friday 16th June 2023.

This enhancement is targeting for stage alpha for 1.28 (correct me, if otherwise)

Here’s where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.28
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria.
• KEP has a production readiness review that has been completed and merged into k/enhancements.

With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[Rishit-dagli]

Hello @justaugustus 👋, 1.28 Docs Lead here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[xmudrii]

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

Yes. I'll follow up on this with folks and I'll make sure that we have everything in place by the deadline.

[ramrodo]

Hello @xmudrii, 1.28 Comms here.

Is this enhancement planned to have an opt-in process for Feature Blog delivery?

The deadline for opt-in is on July 19th, 2023, so please consider submitting a place-holder PR at kubernetes/website for this to be considered.

Thank you!

[xmudrii]

@ramrodo Is just creating a PR enough to opt-in or is there anything else that we should do?

[ramrodo]

@ramrodo Is just creating a PR enough to opt-in or is there anything else that we should do?

Yes. Creating the placeholder PR is enough to opt-in for now.

[Rishit-dagli]

@xmudrii

Yes. I'll follow up on this with folks and I'll make sure that we have everything in place by the deadline.

A reminder on this since there is 1 week to the deadline, this can even be a draft PR right now.

[xmudrii]

@Rishit-dagli Thank you, I'll make sure to create placeholders by the end of the week.

[xmudrii]

@Rishit-dagli @ramrodo I created placeholder PRs for both docs and feature blog: kubernetes/website#42022 and kubernetes/website#42023

[ruheenaansari34]

Hey again @justaugustus 👋
Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023.

I don't see any code (k/k) update PR(s) in the issue description so if there are any k/k related PR(s) that we should be tracking for this KEP please link them in the issue description above.

As always, we are here to help if any questions come up. Thanks!

[xmudrii]

@ruheenaansari34 This is an out of tree KEP, so as of now, it doesn't require any code changes in k/k.

[Atharva-Shinde]

Hey @justaugustus @xmudrii this enhancement is now marked as tracked for the v1.28 Code freeze.

[Rishit-dagli]

Hello @justaugustus @xmudrii wave: please take a look at Documenting for a release - PR Ready for Review to get your docs PR ready for review before Tuesday 25th July 2023. Thank you!

Ref: kubernetes/website#42022

[sftim]

This is an out of tree KEP, so as of now, it doesn't require any code changes in k/k.

For the alpha, is there any change to document?

[xmudrii]

For the alpha, is there any change to document?

Yes, we plan to mention OBS in docs, but we're still trying to figure out the best way for that.

[sftim]

How about on https://k8s.dev/docs/ - and then revisit for beta?

[xmudrii]

@sftim I'm not sure if that's visible enough. Also, I don't think this KEP will be going through standard alpha/beta/stable criteria, but that's also something to discuss.

[npolshakova]

/remove-label lead-opted-in

[sftim]

This is still labelled alpha; is that appropriate?

[xmudrii]

We're hopefully going to graduate it to beta, I'll see with leads about that

[npolshakova]

Hello @justaugustus @xmudrii, 1.29 Enhancements team here! Is this enhancement targeting 1.29? If it is, can you follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board? Thanks!