RFE: Support for CentOS Stream Koji

[LorbusChris]

I'd like to be able to automate builds for CentOS 9 Stream RPMs with packit.
This should mirror the build in-koji functionality for builds on Fedora's Koji instance that already exists: https://packit.dev/docs/cli/build/koji/

CentOS 9 Stream's Koji instance runs at https://kojihub.stream.centos.org/koji/
The respective dist-git repos live on GitLab: https://gitlab.com/redhat/centos-stream/rpms/

Thank you for considering this RFE!

Edit(TomasTomecek)
TODO:

• Reach out to the CentOS Stream team and pitch this idea
• Obtain permissions to build in the CentOS Koji instance(s)
  • Ideally reuse our FAS Packit account
• Koji build jobs can configure koji URLs
  • Implement in Packit if this is missing
• Once everything is deployed in production, verify via Packit Staging instance you can submit builds in a custom CentOS koji instance
• Work with @LorbusChris and other stakeholders whether their use case is addressed

[TomasTomecek]

Thank you for opening this RFE! If this is only about doing a (scratch) koji build in Stream koji, then this is straightforward. We'd just need to get credentials for Packit in there (or does it support FAS?)

[LorbusChris]

Hey @TomasTomecek, apologies for the late reply.

I think what we need are "standard" non-scratch release builds (although having the ability to do scratch builds definitely won't hurt).
In the future we might want to do those builds under a custom Koji tag too, but it's not a requirement at this time.

[TomasTomecek]

I would love to set the builds first in Copr using the CentOS Stream 9 target and once that works well, we can plan next steps and introduce the ability to build in the official CentOS Stream koji; would that work?

[LorbusChris]

In general COPR builds using the C9S targets work well, see e.g. https://copr.fedorainfracloud.org/coprs/g/OKD/okd/build/5165963/ (from https://github.com/cri-o/cri-o/blob/main/.packit.yaml#L19-L20)

(However, there seem to be some other, possibly unrelated issues in more recent builds: https://copr.fedorainfracloud.org/coprs/g/OKD/okd/builds/)

[TomasTomecek]

@LorbusChris thanks for the pointer, opened a PR to resolve the cri-o build failures: cri-o/cri-o#6553

[amoralej]

Note that the koji instance for CentOS SIGs builds, the right koji instance is CBS https://cbs.centos.org/koji/ , not https://kojihub.stream.centos.org/koji/

Builds in CBS can be based on srpms or from source, in that case the distgits must be located in https://git.centos.org/ as for example https://git.centos.org/rpms/openvswitch

[LorbusChris]

Ok, it seems my request was uninformed. We need access to the SIG instance: https://cbs.centos.org/koji/

[TomasTomecek]

It would be great to discuss this request with the CPE team during their planning session in March so that all our expectation match

[oglok]

Is there any update on the planning session with the CPE team? Thanks!

[TomasTomecek]

Thank you for the reminder! We're still yet to discuss the outcome and plan our next quarter. It may take us a few weeks to reach a consensus. We will update this issue afterwards. CentOS Stream integration is a huge ask and we need to scope all the work properly. Sorry it takes so long.

[LorbusChris]

@TomasTomecek should I open a separate issue for the CentOS CBS Koji instance? (That's the instance where we want the automation for CentOS SIG builds - automation on the CentOS Stream instance isn't required by us, but I'm sure it'll be super useful too)

[cathay4t]

I am working on this RFE. Will reach out CentOS team and update work blocker list here once confirmed.

[lsm5]

This would be great for the podman / container tools team. /cc @jnovy

[lbarcziova]

Hi @LorbusChris ! We have recently discussed this issue at our architecture meeting together with Adam Samalik and Aleksandra Fedorova and decided to wait with the implementation in the Packit Service and start with the #1979 as the problems were:

• You need to be in Red Hat Network to build in CentOS Stream Koji.
• The process is not as straightforward as for Fedora, so we would first need to discuss this with other stakeholders.

But reading through the discussion in this issue, I would like to clarify, are you interested in building in Koji via Packit as a service or only via the command line manually?

[LorbusChris]

I'm particularly interested in fully automated builds on the CBS instance (not the Stream one). There is no need to be on the RH network for CBS builds.

[lbarcziova]

Thanks for the clarification! So, in that case, how would you expect the Packit functionality to work, what should be the event that should trigger the builds in CBS instance? (For Fedora, we react to the new changes in dist-git, see)

[cathay4t]

https://docs.centos.org/en-US/stream-contrib/techinfo/buildsystem/ indicate the CBS is not for CentOS stream 9+. The CBS is for SIG and deprecated CentOS Linux.

And the CentOS stream 9+ contributes in gitlab is the old source of package which will be shipped to RHEL 9.

[LorbusChris]

@lbarcziova downstream-propose as well as on-dist-git-commit build triggers would be excellent.

@cathay4t the packages built on CBS by the SIGs are not part of the CentOS Stream repos (and neither downstream RHEL), but the builds from CBS are still targeted at C9S, not just at deprecated CentOS versions.
CBS is part of the official CentOS infrastructure, and it shares its lookahead cache with Fedora (using a different namespace).
As mentioned in #1979, CBS can consume dist-git repos from both git.centos.org and gitlab.com/CentOS

[LorbusChris]

CBS allows for flexibly building and shipping multiple (conflicting/preview) versions simultaneously through different yum repos.

For example in the OKD project, we consume openvswitch built by SIG NFV and cri-o built by SIG Cloud. Neither of those packages are part of C9S/RHEL.

[jkonecny12]

Hi, it would be super helpful to us too (Anaconda). Because it's taking a lot of time to do the centos stream builds.

If we / I can help somehow with this, please feel free to contact me.

[lbarcziova]

hi @jkonecny12! Just for clarity, you would be interested in builds in https://kojihub.stream.centos.org/koji/ (there is the CBS instance also and there was created a separate issue #2002 for that)?

[lachmanfrantisek]

After a team discussion, we didn't pick this as a top Packit team priority for the next quarter and preferred epics with bigger user benefits. Sadly, we (as the Packit team) have limited resources... If anyone wants to help us with this, we will be really glad. We are open to any collaboration and have already successfully implemented/started multiple affords thanks to people outside of the Packit team.

I know this will probably not help everyone, but one of the priorities we did pick is packit/packit-service#2106

[jkonecny12]

hi @jkonecny12! Just for clarity, you would be interested in builds in https://kojihub.stream.centos.org/koji/ (there is the CBS instance also and there was created a separate issue #2002 for that)?

We are looking for similar automation as we have now for Fedora (propose_downstream) but to official CentOS Stream repositories. Basically, we would be glad if something would handle PR creation to downstream build.

Right now, the process of downstream releasing is almost a one day job, would be great to have automation at least for some parts.

[lbarcziova]

@jkonecny12 thanks for the clarification! That is actually in progress (creating the PR part) and propose_downstream for CentOS Stream is already implemented, we just need to document it and will be glad if you can onboard to using Packit for that! The progress of the epic in general is tracked in packit/packit-service#2106.

Here is a snippet of an example config that you could get inspired by (but we will definitely document this in more detail):

packages:
  nispor-fedora:
    actions:
      post-upstream-clone:
        - wget https://src.fedoraproject.org/rpms/nispor/raw/rawhide/f/nispor.spec
  nispor-centos:
    pkg_tool: centpkg
    actions:
      post-upstream-clone:
        - wget https://gitlab.com/redhat/centos-stream/rpms/nispor/-/raw/c9s/nispor.spec

jobs:
  - job: propose_downstream
    trigger: release
    packages: [nispor-fedora]
    dist_git_branches:
      - fedora-all

  - job: propose_downstream
    trigger: release
    packages: [nispor-centos]
    dist_git_branches:
      - c9s

[jkonecny12]

Sounds great! We are, unfortunately, right now quite overloaded by other work but sounds like something which hopefully shouldn't be that hard to use and could improve our life.

[HuijingHei]

@lbarcziova thanks for working on this, does it support c10s now (or in plan)? The doc includes dist_git_branches as c8s/c9s.

[lbarcziova]

@HuijingHei the implementation is not dependent on the branches, so it should work, but no one has yet tested it (and you are right, we should then update the docs as well).

Currently, few users were successful in general with running propose-downstream from CLI; with running this in service (automatically triggered by release), there is currently the downside that the lookaside upload step is skipped.

Let us know if you would like any help with onboarding!