2.x - Introduce Jenkins Job Builder for CI #359

wainersm · 2020-12-18T12:50:53Z

This introduces the files and scripts to use with Jenkins
Job Builder to manage our CI jobs.

As a proof of concept, it is generated only four jobs:
kata-containers-main-kernel-experimental-x86_64
kata-containers-main-kernel-vanilla-x86_64
kata-containers-main-qemu-x86_64
kata-containers-stable-2.2-qemu-x86_64

Fixes #343

Signed-off-by: Wainer dos Santos Moschetta wainersm@redhat.com

wainersm · 2020-12-18T13:00:10Z

Hi @jodh-intel , I'm sending this PR for the 2.0-dev branch as a simplified version of #344 . I re-evaluated the approach and now I think I should start with a simple version and growing on the scope with small steps.

So this PR will only generate jobs for the kernel nightly build for 2.x (we don't have such as jobs yet, so the install script always build the kernel - consequently taking more time). Next in my list is:

Create a job "update-jobs" which will self-update Jenkins whatever a file in jobs-builder/* is merged on this repository
Generate the jobs for nightly hypervisors builds (qemu, firecracker...etc)
Generate the jobs for nightly images. More complicated since identifying the cache is dirty for the image files seems more complex.
Then work on the remain of the jobs which are triggered on pull request for kata-containers repo.

Sound as a good plan?

wainersm · 2020-12-18T13:02:03Z

@chavafg @GabyCT Hi people! I appreciate if you could give your opinion and reviews about this proposal.

jobs-builder/jjb.conf.template

chavafg

thanks @wainersm, this looks good, just a couple of comments.

jobs-builder/publish_jobs.sh

jobs-builder/README.md

jodh-intel

Thanks @wainersm. A few comments.

jobs-builder/README.md

jodh-intel · 2021-01-04T10:45:19Z

jobs-builder/jobs/dependencies.yaml

+    # Template default variables
+    #
+    arch: x86_64
+    os: ubuntu1804


Why not 20.04?

The QEMU and Kernel jobs are running on Ubuntu 18.04. I don't want to change it now.

jodh-intel · 2021-01-04T10:47:48Z

jobs-builder/jobs/dependencies.yaml

+          single-use: True
+      - timestamps
+      - timeout:
+          timeout: 5


Is this in seconds? Why 5 ooi?

Increased to 180.

I just realized the timeout in the yaml is in minutes but the job builder will convert to seconds.

jobs-builder/jobs/dependencies.yaml

jodh-intel · 2021-01-04T10:49:21Z

jobs-builder/jobs/dependencies.yaml

+          case "$flavor" in
+            "vanilla") export experimental_kernel=false;;
+            "experimental") export experimental_kernel=true;;
+            *) echo "Do not know $flavor flavor" && exit 1


Suggested change

*) echo "Do not know $flavor flavor" && exit 1

*) echo >&2 "ERROR: Unknown flavour: '$flavour'"; exit 1

jodh-intel · 2021-01-04T10:50:40Z

jobs-builder/jobs/include/os2node.yaml.inc

+{% elif os == "fedora32" -%}
+fedora32_azure


F32 goes EOL in a few months - could this be changed to F33?

As we add new jobs I plan to update the os2node.yaml.inc file. For now, it is good enough for the qemu and kernel jobs.

jodh-intel · 2021-01-04T10:52:46Z

jobs-builder/jobs/include/os2node.yaml.inc

+{% elif os == "ubuntu1804" -%}
+ubuntu1804_azure
+{% elif os == "ubuntu1804_ARM" -%}
+arm_node && arm-ubuntu-1804


Again, can this be Ubuntu 20.04?

jodh-intel · 2021-02-24T11:24:58Z

Any update on this @wainersm?

ariel-adam · 2021-05-18T14:44:05Z

@wainersm any updates on this PR :-)?

jodh-intel · 2021-07-12T10:20:16Z

pr:
  yaml-is-better-than-xml: true
  pr-needs-update: true
  i-would-love-to-see-this-land: true
  person-to-ping: "@wainersm"

wainersm · 2021-09-23T03:14:53Z

hey @jodh-intel I managed to resume that work! There are still small details here and there which I plan to improve on the next days.

wainersm · 2021-09-23T18:33:38Z

Ok, now I think this is ready for review (again). As I mentioned elsewhere this is just a PoC to show the jenkins job builder capabilities.

On the version I just pushed I:

added a section on README.md explaining how someone could check the generated jobs on a local Jenkins instance
I changed jobs/dependencies.yaml for the purpose of showing the power of jinja2 templates as well as on how YAML parts can be reused
Likely changed other stuffs that I don't remember now :)

wainersm · 2021-09-23T18:41:07Z

@chavafg @GabyCT I'm might be tendentious saying that but I believe we will need soon something like what is being proposed on this PR. With the "always green CI" initiative flying, we will need to have a disciplined way of managed the jobs. Using a tool like Jenkins Job Builder allow us to threat the jobs as yet another piece of source code; people will be able to review the change more properly; we can ensure that what is in the repo is what you see on the UI, and so on.

@jodh-intel I am sure you are eager for this :)

Cc @ariel-adam

c3d

Having heard about this PR today, I took a quick look. I really like the idea, but I feel I am not the right person to approve it. @GabyCT @cpmeadors would you mind giving your opinion?

jodh-intel

Thanks @wainersm !

/cc @GabyCT and @jcvenegas for input.

jobs-builder/jobs/dependencies.yaml

jobs-builder/publish_jobs.sh

This introduces the files and scripts to use with Jenkins Job Builder to manage our CI jobs. As a proof of concept, it is generated only four jobs: kata-containers-main-kernel-experimental-x86_64 kata-containers-main-kernel-vanilla-x86_64 kata-containers-main-qemu-x86_64 kata-containers-stable-2.2-qemu-x86_64 Fixes kata-containers#343 Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

jodh-intel

Thanks @wainersm !

lgtm but let's get input from @chavafg, @GabyCT, @jcvenegas, @cpmeadors.

jodh-intel · 2021-10-25T13:05:40Z

Ping y'all - this is ready to go but should really have another ack I feel.

GabyCT reviewed Dec 18, 2020

View reviewed changes

jobs-builder/jjb.conf.template Show resolved Hide resolved

chavafg reviewed Dec 18, 2020

View reviewed changes

jobs-builder/publish_jobs.sh Outdated Show resolved Hide resolved

jobs-builder/README.md Show resolved Hide resolved

wainersm force-pushed the kata-2_x-introduce_jjb branch from a56f544 to eed2531 Compare December 18, 2020 19:25

jodh-intel mentioned this pull request Jan 4, 2021

[WIP] Introduce Jenkins Job Builder for CI #344

Closed

jodh-intel reviewed Jan 4, 2021

View reviewed changes

wainersm mentioned this pull request Jan 22, 2021

Ease the management of CI jobs kata-containers/kata-containers#1298

Open

jodh-intel mentioned this pull request Aug 9, 2021

jenkins: Add s390x tests job #391

Merged

wainersm force-pushed the kata-2_x-introduce_jjb branch from eed2531 to 98ce60d Compare September 23, 2021 03:06

wainersm force-pushed the kata-2_x-introduce_jjb branch from 98ce60d to bc18546 Compare September 23, 2021 18:24

wainersm mentioned this pull request Sep 30, 2021

Always Green CI - Owners of CI jobs - Your help is needed #403

Closed

c3d reviewed Oct 5, 2021

View reviewed changes

jodh-intel reviewed Oct 14, 2021

View reviewed changes

jobs-builder/jobs/dependencies.yaml Outdated Show resolved Hide resolved

jobs-builder/publish_jobs.sh Outdated Show resolved Hide resolved

wainersm force-pushed the kata-2_x-introduce_jjb branch from bc18546 to a22049c Compare October 14, 2021 20:41

jodh-intel approved these changes Oct 19, 2021

View reviewed changes

GabyCT approved these changes Oct 25, 2021

View reviewed changes

GabyCT merged commit fe46d70 into kata-containers:2.0-dev Oct 25, 2021

This was referenced Nov 10, 2021

main | Introduce Jenkins Job Builder for CI #406

Merged

jobs-builder: plan of migration - your help is needed #407

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2.x - Introduce Jenkins Job Builder for CI #359

2.x - Introduce Jenkins Job Builder for CI #359

wainersm commented Dec 18, 2020 •

edited

Loading

wainersm commented Dec 18, 2020

wainersm commented Dec 18, 2020

chavafg left a comment

jodh-intel left a comment

jodh-intel Jan 4, 2021

wainersm Sep 23, 2021

jodh-intel Jan 4, 2021

wainersm May 7, 2021

wainersm Sep 23, 2021

jodh-intel Jan 4, 2021

jodh-intel Jan 4, 2021

wainersm Sep 23, 2021

jodh-intel Jan 4, 2021

wainersm Sep 23, 2021

jodh-intel commented Feb 24, 2021

ariel-adam commented May 18, 2021

jodh-intel commented Jul 12, 2021

wainersm commented Sep 23, 2021

wainersm commented Sep 23, 2021

wainersm commented Sep 23, 2021

c3d left a comment

jodh-intel left a comment

jodh-intel left a comment

jodh-intel commented Oct 25, 2021

	*) echo "Do not know $flavor flavor" && exit 1
	*) echo >&2 "ERROR: Unknown flavour: '$flavour'"; exit 1

2.x - Introduce Jenkins Job Builder for CI #359

2.x - Introduce Jenkins Job Builder for CI #359

Conversation

wainersm commented Dec 18, 2020 • edited Loading

wainersm commented Dec 18, 2020

wainersm commented Dec 18, 2020

chavafg left a comment

Choose a reason for hiding this comment

jodh-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jodh-intel commented Feb 24, 2021

ariel-adam commented May 18, 2021

jodh-intel commented Jul 12, 2021

wainersm commented Sep 23, 2021

wainersm commented Sep 23, 2021

wainersm commented Sep 23, 2021

c3d left a comment

Choose a reason for hiding this comment

jodh-intel left a comment

Choose a reason for hiding this comment

jodh-intel left a comment

Choose a reason for hiding this comment

jodh-intel commented Oct 25, 2021

wainersm commented Dec 18, 2020 •

edited

Loading