Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publishing Kubernetes packages on community infrastructure #1731

Open
8 of 49 tasks
justaugustus opened this issue Apr 30, 2020 · 12 comments
Open
8 of 49 tasks

Publishing Kubernetes packages on community infrastructure #1731

justaugustus opened this issue Apr 30, 2020 · 12 comments
Assignees
Labels
sig/release Categorizes an issue or PR as relevant to SIG Release. stage/alpha Denotes an issue tracking an enhancement targeted for Alpha status tracked/out-of-tree Denotes an out-of-tree enhancement issue, which does not need to be tracked by the Release Team

Comments

@justaugustus
Copy link
Member

justaugustus commented Apr 30, 2020

Enhancement Description


Milestones and Tasks Checklist

Milestone 1.0—Code Deliverable

  • Success looks like: debs and rpms are in multiple locations (Google infra and third-party hosting service(s)) @detiber
    • Define User Scenarios and Requirements (Configuration, management, monitoring, etc.)
    • Describe proposed Flow (what are key actions that the user will take? How will they take them?)
    • Choose the third-party hosting service/Set up a package host of some sort @detiber, @leonardpahlke helping
      • Identified some options to explore
      • Outreach to reps of those options to see if they'll work with us on this WIP
    • Do a spike test to see if the third-party hosting service meets our needs @detiber, @leonardpahlke helping
    • Publish the debs/rpms to the third-party hosting service @detiber

Milestone 1.0—Documentation Deliverable

Milestone 1.0—Risk Mitigation

  • Success looks like: All risk matters for this milestone are accounted for, plans in place to mitigate
    • Risk: A new GPG key will be issued incompatible with the existing GPG key managed by Google. This is a breaking change for the existing infrastructures using DEB/RPM packages as installation artifacts

Milestone 1.0—Questions resolved

  • Resolve if this is needed: Give access to the sandbox env to release eng to work on it @ameukam, @onlydole helping
  • Develop sandbox env (GCP project) and give access to Rel Eng to work on it @ameukam, @onlydole helping
  • Needed more discussion: Meeting to specifically discuss if we are doing GPG or if we are owning the packages; then who owns new GPG keys and how and when we store them. @ameukam, @onlydole helping
  • Generate new keys : Sign kubernetes system packages with GPG release#2627
  • Decide on plan: Be able to issue new gcp trusted keys for entire ecosystem/whoever is relying on those packages to deploy/install/upgrade K8s project
  • Find/line up people who can build packages for debian, CentOS to work on build tooling (@upodroid to help build Debian packages)

Milestone 2.0—Code deliverable

  • Success looks like: implementation (source code) in our repositories, such that we can publish releases by ourselves without having to rely on Google people at all
    • Define User Scenarios and Requirements (Configuration, management, monitoring, etc.
    • Proposed Flow (what are key actions that the user will take? How will they take them?)
    • Refine the current debs/rpms built by kubepkg
    • Resume discussing how we rethink / rewrite the build tools
    • Rebase the existing kubepkg work into something krel can use directly
    • Krel, the kubernetes release tool, should start including a package build step running the build scripts
    • Phase out publishing to the google controlled host, for having krel publish to our new host for 1-2 releases
    • Users migrate from the Google registry to the community registry, adding the signing key

Milestone 2.0—Documentation Deliverable

  • Success looks like: we've published docs on how process works and when it's not working as expected, what to do (for release managers)
    • Expand documentation on the top-level script to drive knowledge transfer
    • Expand documentation on the package build scripts for Debian and RPM to drive knowledge transfer

Milestone 2.0—Questions resolved

  • Success looks like: @saschagrunert and @justaugustus have provided context and knowledge about the package build scripts for Debian and rpms
    • Answer: what type of infra we're going to run
    • Answer: who pays for it
    • Answer: what type of account do we have
    • Answer: what tools do/will we have available

Milestone 3.0—Documentation deliverable

  • Success looks like: We've updated the google build+publish+release script to download packages uploaded by the upstream release process

Milestone 4.0— Documentation Deliverable

  • Success looks like: We've published docs for end users about old infra and new/migrated (more static)
    • Write email to community regarding the required migration to the community repository
    • Send email to community regarding the required migration to the community repository
    • SIG Release informs the community so we don't break people over just one milestone: "these are the keys we'll use to release 1.2X..."

Why is this needed?

  • It's part of the effort for the community to fully own the infrastructure related to the Kubernetes project.
  • Ensure all aspects of project releases can be staffed across companies
  • We don’t want to be dependent on a Build Admin any longer (which tends to slow releases)
  • We seek more control to eventually choose to build packages for prereleases, extend what packages are shipped, etc.

Why is it needed now?

  • The existing infrastructure of this relies on a workstation inside Google offices—essentially one person. It’s not reliable or sustainable.

Who needs it? (User Personas WIP)

  • Google Build Team as the existing team building the system packages.
  • What does “done” look like/acceptance criteria?
  • Automated builds of deb and rpm Kubernetes packages within community infrastructure

Related open issues

@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Apr 30, 2020
@justaugustus justaugustus added the sig/release Categorizes an issue or PR as relevant to SIG Release. label Apr 30, 2020
@k8s-ci-robot k8s-ci-robot removed the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Apr 30, 2020
@justaugustus justaugustus added this to Backlog in SIG Release via automation Apr 30, 2020
@justaugustus justaugustus moved this from Backlog to In progress in SIG Release Apr 30, 2020
@justaugustus justaugustus added the tracked/out-of-tree Denotes an out-of-tree enhancement issue, which does not need to be tracked by the Release Team label Apr 30, 2020
@fejta-bot
Copy link

fejta-bot commented Jul 29, 2020

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 29, 2020
@LappleApple LappleApple moved this from In progress to In Progress, but no activity in >=14 days in SIG Release Aug 7, 2020
@fejta-bot
Copy link

fejta-bot commented Aug 29, 2020

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 29, 2020
@fejta-bot
Copy link

fejta-bot commented Sep 28, 2020

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Sep 28, 2020

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@LappleApple LappleApple moved this from In Progress, but no activity in >=14 days to Blocked/waiting for feedback in SIG Release Oct 7, 2020
@LappleApple LappleApple moved this from Blocked/waiting for feedback to Done (1.20) in SIG Release Oct 7, 2020
@justaugustus justaugustus removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Dec 3, 2020
@justaugustus justaugustus reopened this Dec 3, 2020
SIG Release automation moved this from Done/Closed (1.20) to In progress Dec 3, 2020
@justaugustus justaugustus added this to the v1.21 milestone Dec 3, 2020
@justaugustus justaugustus removed this from In progress in SIG Release Dec 3, 2020
@justaugustus justaugustus added this to Backlog in Artifact Management (SIG Release) via automation Dec 3, 2020
@justaugustus justaugustus moved this from Backlog to KEPs in Artifact Management (SIG Release) Dec 3, 2020
@justaugustus justaugustus self-assigned this Dec 3, 2020
@justaugustus justaugustus added the stage/alpha Denotes an issue tracking an enhancement targeted for Alpha status label Dec 3, 2020
@fejta-bot
Copy link

fejta-bot commented Mar 3, 2021

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 3, 2021
@justaugustus justaugustus removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 26, 2021
@fejta-bot
Copy link

fejta-bot commented Jun 24, 2021

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 24, 2021
@k8s-triage-robot
Copy link

k8s-triage-robot commented Jul 26, 2021

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 26, 2021
@k8s-triage-robot
Copy link

k8s-triage-robot commented Aug 25, 2021

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue or PR with /reopen
  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Aug 25, 2021

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue or PR with /reopen
  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Artifact Management (SIG Release) automation moved this from KEPs to Done (1.21) Aug 25, 2021
@saschagrunert saschagrunert self-assigned this Jul 15, 2022
@saschagrunert saschagrunert reopened this Jul 15, 2022
Artifact Management (SIG Release) automation moved this from Done (1.21) to In Progress Jul 15, 2022
@saschagrunert saschagrunert removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jul 15, 2022
@saschagrunert saschagrunert removed this from the v1.21 milestone Jul 15, 2022
@saschagrunert saschagrunert changed the title Publishing packages Publishing Kubernetes packages on community infrastructure Jul 15, 2022
@saschagrunert
Copy link
Member

saschagrunert commented Jul 15, 2022

@RobertKielty @LappleApple feel free to propose an update to the existing KEP

@xmudrii
Copy link
Member

xmudrii commented Oct 21, 2022

Summary

We started evaluating OpenBuildService (OBS) hosted by openSUSE (https://build.opensuse.org/) as a solution for building, signing, publishing, and hosting our packages. OBS handles Debian (deb) and RPM packages for many of the most popular distros (Ubuntu, Debian, CentOS, RHEL...).

The initial proof-of-concept phase went very well:

  • packages are being built successfully using the same spec files with no or very minor changes
  • it's possible to install those packages and successfully provision a cluster
  • it's possible to migrate to the new OBS repos and then upgrade a cluster

The initial proof-of-concept covered both debs and rpms on amd64.

Open questions

Several open questions are still to be investigated/tested or decided upon.

  • How do we want to handle multi-arch builds?
    • Option 1: try to parametrize spec files to choose the correct binary from a tarball with all binaries for all supported architectures
      • Potential issue is that such a tarball might be big, but that might be mitigated by using a higher compression rate
    • Option 2: push sources to OBS and build binaries for the target architecture in OBS
      • There are three problems with this option:
        • Kubernetes sources are also big, ranging from 1 GB to 2 GB depending on if .git is included
        • Unless we get reproducible builds working in OBS, the resulting binary will be different than the binary we distribute via release buckets
        • The binary itself will not be signed (e.g. with cosign). This might not be a blocking issue because we sign the APT repository or the RPM package
  • How do we want to organize our spec files?
    • OBS works in a way that you have a Project that holds packages. One package is published to multiple repositories and there's a repository for each supported/chosen operating system
    • For Debian packages, a package on OBS is needed for each Debian package -- you can't define multiple packages in one spec file
    • For RPM packages, one package on OBS can be used for building/hosting multiple RPM packages -- you can define multiple packages in the same spec file (that's what we do right now)
    • We have multiple options:
      • Option 1: continue what we're doing right now, however, having one single RPM spec is harder to maintain and harder for users to consume. We are also not consistent with debs
      • Option 2: break the current RPM spec into multiple specs (already done by kubepkg) -- this is currently the most favorable option
      • Option 3: use debbuild to build deb packages from RPM specs. The disadvantage is that deb packages are not "natively" defined in this case -- someone building deb packages might not have an idea bout how RPM works. This also introduces a new dependency
    • Note: we have the option to change our spec files, but we should try to keep the same results if that's possible -- so that users can migrate to the new packages without too much hassle

Community infrastructure concerns

The solution we're evaluating right now is hosted by openSUSE. openSUSE is willing to sponsor us for all our needs, so we can use their infrastructure. This is not "truly" community infrastructure, but this helps us a lot as we don't have to worry about building and publishing/hosting packages ourselves.

Additionally, OpenBuildService is an open source platform, so we can always decide to host it ourselves if we don't want to use openSUSE's infra for any reason. To make this possible, we'll look into building proxy/redirects from apt.kubernetes.io/yum.kubernetes.io to the openSUSE OBS infra.

Package/repository signing concerns

One of the biggest concerns we had is how are going to manage GPG keys for signing packages (in the case of rpm) and repositories (in the case of APT). The OBS platform fully manages the GPG keys -- the private key is not exposed to anyone using the platform. In those terms, we can look at OBS as an API that we can give access to without worrying that someone might get access to the key.

Moving building packages from Google to the community

Another concern is that we want to move building packages from Google to the community. We are planning to accomplish this by building packages on OBS and letting Google Build Admins download packages from there to be published to the Google (current) infra.

Next steps and timeline

The proposed timeline would be something along the following lines:

Pre-alpha

  • Get answers to the open questions
  • Extend the current proof-of-concept with multi-arch builds
  • Final decision on do we want to proceed with OBS

Alpha

  • Integrate OBS with the current release pipeline
    • Start building and publishing packages from OBS in parallel to doing that on the Google infra
  • Work on tests to ensure the quality of created packages
  • Communicate the progress, timelines, and upcoming changes to the community

Beta

  • Publish packages built by OBS to the Google infra
  • Another round of comms to the community

Stable

  • Gradually phase out Google infra in favor of OBS

Alternatives considered

We considered building packages manually in our pipeline and publishing and hosting packages independently instead of using OBS. That however has much increased complexity, as we have to maintain the whole infra, and we also have to build a management system for GPG keys, which is very complex.

@saschagrunert
Copy link
Member

saschagrunert commented Nov 1, 2022

  • Get answers to the open questions

What do you think would be the best way to achieve this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/release Categorizes an issue or PR as relevant to SIG Release. stage/alpha Denotes an issue tracking an enhancement targeted for Alpha status tracked/out-of-tree Denotes an out-of-tree enhancement issue, which does not need to be tracked by the Release Team
Projects
Development

No branches or pull requests

6 participants