Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Always Green CI - Owners of CI jobs - Your help is needed #403

Closed
chavafg opened this issue Sep 17, 2021 · 26 comments
Closed

Always Green CI - Owners of CI jobs - Your help is needed #403

chavafg opened this issue Sep 17, 2021 · 26 comments
Labels
enhancement Improvement to an existing feature needs-review Needs to be assessed by the team.

Comments

@chavafg
Copy link
Contributor

chavafg commented Sep 17, 2021

As we talked in last AC meeting, we are proposing a number of ideas to improve the results that we currently have in our CI. One of the first items that we would like to have is a list of owners/maintainers that will help us maintain the good state of the jobs reporting to the CI.
In this issue, I want to document the jobs that are currently reporting back to the CI.

Scope:
Jenkins Jobs that are reporting back to GitHub in every PR.

Out of scope:
GitHub actions, Jenkins nightly jobs, jobs that do not report back to GitHub

jobs in kata-containers and tests repositories currently Required Job Type ($CI_JOB) Owners
jenkins-ci-ARM-ubuntu-18-04 No CRI_CONTAINERD_K8S @jongwu, @justin-he
jenkins-ci-centos-8 No CRIO_K8S @snir911, @c3d
jenkins-ci-debian-10 Yes CRIO_K8S
jenkins-ci-fedora-32-crio Yes CRIO_K8S @snir911, @c3d
jenkins-ci-fedora-32-crio-k8s-e2e-minimal No CRIO_K8S_MINIMAL @snir911, @c3d
jenkins-ci-firecracker-ubuntu-18-04 No FIRECRACKER @ananos
jenkins-ci-pmem-ubuntu-18-04 Yes PMEM @devimc, @GabyCT (Intel)
jenkins-ci-s390x-ubuntu-20-04 No CRI_CONTAINERD_K8S @BbolroC, @stevenhorsman, @davidhay1969
jenkins-ci-ubuntu-18-04-initrd No CRI_CONTAINERD_K8S_INITRD
jenkins-ci-ubuntu-1804-clh-containerd-k8s-e2e-minimal Yes CLOUD-HYPERVISOR-K8S-CONTAINERD-MINIMAL @intel
jenkins-ci-ubuntu-1804-cloud-hypervisor-k8s-containerd-2.0 No CLOUD-HYPERVISOR-K8S-CONTAINERD @jcvenegas (Intel)
jenkins-ci-ubuntu-1804-containerd-k8s-e2e-minimal Yes CRI_CONTAINERD_K8S_MINIMAL
jenkins-ci-ubuntu-20-04 Yes CRI_CONTAINERD_K8S @GabyCT, @jcvenegas (Intel)
jenkins-ci-ubuntu-clh-crio-18-04 No CLOUD-HYPERVISOR-K8S-CRIO @jcvenegas (Intel)
jenkins-metrics-ubuntu-20-04 Yes METRICS @GabyCT, @dborquez (Intel)
jenkins-qat-ubuntu-20-04 Yes QAT @GabyCT, @devimc (Intel)
jenkins-vfio-host:ubuntu18-04-vm:fedora32 Yes VFIO @devimc, @GabyCT (Intel)

Job reporting to CRI-O repository:

cri-o repo currently Required Job Type ($CI_JOB) Owners
jenkins-kata-containers-crio YES CRIO_K8S @fgiudici, @littlejawa and @fidencio

Jobs that are WIP to be stable that would like to report back to GitHub:

jobs in kata-containers and tests repositories currently Required Job Type ($CI_JOB) Owners
jenkins-ci-Power9-ubuntu-20-04 No N/A @Amulyam24

I have added some names in the jobs above, including the ones that Intel would be able to maintain.

@jongwu, @Jakob-Naucke, @ananos, I added your names there as I understand you are currently working in those areas, so please confirm that you can be taken as owners or feel free to change them. It would be great if you could also identify backups for these jobs.

@ariel-adam If you can take a look at the jobs that need owner with your team, it would be great. There are some cri-o, fedora and centos related jobs that I think are important for RedHat.

For anyone else in the community, please consider joining this effort. Feel free to leave a comment below if you would like to be added and I'll update the table. Your help is really appreciated!

To understand which tests are being executed in each of the jobs, I have added the column Job Type. You can look at the tests that are executed in that job depending on the value of the CI_JOB variable. Details in the run.sh script.

Owners for tests

One of the comments that we received in the AC meeting discussion was that it would also be good to have owners of the tests that are run in the CI. For this, I think we can make use of: Areas of Interest wiki page, where we have documented areas of interest of the different community members. The wiki page needs to be updated, so my other ask to everyone is to open the link and add your names where it make sense. There, we would be able to track who can help if a certain test has become unstable.

/cc @kata-containers/architecture-committee @wainersm @marcel-apf @GabyCT @jcvenegas @egernst @c3d @fidencio

Related:
google doc proposal

@chavafg chavafg added enhancement Improvement to an existing feature needs-review Needs to be assessed by the team. labels Sep 17, 2021
@jongwu
Copy link

jongwu commented Sep 18, 2021

Hello @chavafg -, thanks for figure out these. I will take job of maintaining "jenkins-ci-ARM-ubuntu-18-04".

@Amulyam24
Copy link
Contributor

Hi @chavafg, Thank you for the proposal. I want to add that I'm working on making the jenkins-ci-Power9-ubuntu-20-04 job stable and it shall hopefully be enabled on the upstream repos soon :)

Can the job be added to the above list? I will take the responsible of maintaining it. However, I would be requiring the necessary permissions(jenkins) for managing the job. Can someone please help me with it?

@fidencio
Copy link
Member

Can the job be added to the above list? I will take the responsible of maintaining it. However, I would be requiring the necessary permissions(jenkins) for managing the job. Can someone please help me with it?

You'll be given the permissions / access to the jenkins.

@Amulyam24
Copy link
Contributor

Thanks @fidencio! :)

@Jakob-Naucke
Copy link
Member

@chavafg yes, I would like to become first maintainer of jenkins-ci-s390x-ubuntu-20-04 (and maybe more s390x in the future, but let's fully stabilize that first :)). Let me get back to you about backups.

@Jakob-Naucke
Copy link
Member

@chavafg please list @stevenhorsman and @davidhay1969 as second & third owners for s390x CI. Unfortunately, we cannot cover other time zones at this time.

@chavafg
Copy link
Contributor Author

chavafg commented Sep 20, 2021

@jongwu thanks for confirming. is there someone else that may cover you in case you are on PTO?

@Amulyam24 I have added Power 9 job to the list above and specified it is still WIP. Let us know when you need to do modifications to Jenkins job and we can provide you correct permissions.

@Jakob-Naucke Thanks for confirming and for providing backups. I have added them in the table above.

@wainersm
Copy link
Contributor

@chavafg hello! just a heads-up: @ariel-adam has worked internally to figure out the jobs we gonna maintain. By tomorrow I believe we will have it figured out.

@fidencio
Copy link
Member

@chavafg hello! just a heads-up: @ariel-adam has worked internally to figure out the jobs we gonna maintain. By tomorrow I believe we will have it figured out.

Just remember that Today & Tomorrow are holidays in Israel. :-)

@wainersm
Copy link
Contributor

@chavafg hello! just a heads-up: @ariel-adam has worked internally to figure out the jobs we gonna maintain. By tomorrow I believe we will have it figured out.

Just remember that Today & Tomorrow are holidays in Israel. :-)

hehehe so probably tomorrow not. @fidencio nice to see your memory is working very good ;)

@ananos
Copy link
Member

ananos commented Sep 20, 2021

@jongwu, @Jakob-Naucke, @ananos, I added your names there as I understand you are currently working in those areas, so please confirm that you can be taken as owners or feel free to change them. It would be great if you could also identify backups for these jobs.

happy to take on the FC-related CI tests. I think we should plan for all the supported archs/variants too (IIRC, FC officially supports Intel; AMD support is flaky on some CPU versions and aarch64 is still considered experimental).

We'll have a chat about backups and get back to you once I get an update.

@jongwu
Copy link

jongwu commented Sep 21, 2021

@jongwu thanks for confirming. is there someone else that may cover you in case you are on PTO?

@Amulyam24 I have added Power 9 job to the list above and specified it is still WIP. Let us know when you need to do modifications to Jenkins job and we can provide you correct permissions.

@Jakob-Naucke Thanks for confirming and for providing backups. I have added them in the table above.

@chavafg -, Yeah, you can add @justin-he as the second maintainer of Arm CI.

@fidencio
Copy link
Member

@chavafg, please, would be possible to add to the table the CI that CRI-O relies on? That's http://jenkins.katacontainers.io/job/kata-containers-2-crio-PR/ and the maintainers will be @fgiudici and @littlejawa from the Red Hat side, and myself from the Intel side.

@cmaf
Copy link

cmaf commented Sep 28, 2021

If we have a project for "Green CI" already this could be linked to it.

@marcel-apf
Copy link

@chavafg Hi, can you please add me as owner of the jobs:

  • jenkins-ci-centos-8
  • jenkins-ci-fedora-32-crio
  • jenkins-ci-fedora-32-crio-k8s-e2e-minimal
  • jenkins-vfio-host:ubuntu18-04-vm:fedora32 (co maintain)

I will maintain them temporary until we will discuss it internally.

@chavafg
Copy link
Contributor Author

chavafg commented Sep 30, 2021

@fidencio I have added the cri-o job with the three maintainers. Thanks!

@marcel-apf I have also added you as maintainer of the items you described above, let me know if there is a change and I can modify the table. Thanks!

@cmaf, thanks for the recommendation, I think it is worth tracking this in a project.

@GabyCT
Copy link
Contributor

GabyCT commented Sep 30, 2021

any idea when the jobs that do not have maintainer will be removed from the CI?
/cc @chavafg @ariel-adam

@wainersm
Copy link
Contributor

@chavafg could you please list me the maintainer of following jobs?

kata-containers-2.0-qemu-x86_64
kata-containers-2.0-kernel-vanilla-x86_64-nightly
kata-containers-2.0-kernel-experimental-x86_64-nightly

They are used by the other jobs, so they need to be maintained. Any volunteer(s) to co-maintain them with me?

@cmaf
Copy link

cmaf commented Sep 30, 2021

@cmaf, thanks for the recommendation, I think it is worth tracking this in a project.

It was actually @c3d's suggestion!

@wainersm
Copy link
Contributor

any idea when the jobs that do not have maintainer will be removed from the CI? /cc @chavafg @ariel-adam

Good question @GabyCT ! Our Jenkins today have a bunch of what seems garbage.

One related topic... after the clean up and if we continue to manage the jobs via the web UI, then I bet we will end up on the same situation (i.e. a bunch of unmaintained jobs) soon.

I suggest that we threat the jobs as code by adopting the Jenkins Job Builder (JJB) (unless someone has others suggestions of tools). I provide a proof-of-concept (PoC) in #359, and what I need right now is a go/no-go from the community to continue integrating JJB into our workflow.

@GabyCT
Copy link
Contributor

GabyCT commented Oct 7, 2021

@chavafg @ananos which Firecracker job is going to be maintained the FC jailer or the FC job ?

@fidencio
Copy link
Member

fidencio commented Oct 7, 2021

@chavafg @ananos which Firecracker job is going to be maintained the FC jailer or the FC job ?

We can delete the FC jailer job, @GabyCT.
The "normal" one has the jailer enabled by default now.

@marcel-apf
Copy link

As I previously stated, I was the interim maintainer until we will find the right people.
@chavafg can you please replace me with:
kata-containers-2.0-ubuntu-20.04-VFIO-main-baseline - Please remove me, maybe @dgibson is interested to be the 3rd maintainer?
kata-containers-2.0-centos-8-CRIO_K8S-main-baseline - @snir911 and @c3d
kata-containers-2.0-fedora-32-CRIO_K8S-main-baseline - @snir911 @c3d
kata-containers-2.0-fedora-32-CRIO_K8S_MINIMAL-main-baseline - @snir911 and @c3d

@GabyCT
Copy link
Contributor

GabyCT commented Dec 15, 2021

Closing this issue as we already defined owners and baselines

@Jakob-Naucke
Copy link
Member

@chavafg I'll be going on a leave of absence in April. Can we have @BbolroC as owner for s390x?

@chavafg
Copy link
Contributor Author

chavafg commented Apr 4, 2022

@Jakob-Naucke sorry for my late reply. I added @BbolroC name here and in the Jenkins view.
Thanks and good luck!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improvement to an existing feature needs-review Needs to be assessed by the team.
Projects
Issue backlog
  
Done
Development

No branches or pull requests

10 participants