Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a nightly pipeline for GA/EUS releases #4046

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from
+94 −26
Draft

Create a nightly pipeline for GA/EUS releases #4046

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
310ec58
Create new nightly pipeline for GA/EUS images
tkoscieln Mar 28, 2024
180f4c3
Add exclude rule and rename the pipeline
tkoscieln Apr 29, 2024
14f786e
Rename the slack jobs
tkoscieln Apr 29, 2024
7dd1124
Change parameters in slack notification
tkoscieln Apr 29, 2024
0fb6966
Add old GA exclusion, rename pipeline and add exceptions on jobs
tkoscieln Apr 30, 2024
fed83ee
Exclude the GA jobs from PR pipeline - fix
tkoscieln May 2, 2024
dddb540
Edit comments and simplify rule
tkoscieln May 6, 2024
b34676c
Rename slack notification message
tkoscieln May 6, 2024
0d4a7d4
Return SaaS related tests back to PR pipeline
tkoscieln Jun 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
114 changes: 89 additions & 25 deletions .gitlab-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,17 +54,21 @@ init:

.build_rules:
rules:
- if: '$CI_PIPELINE_SOURCE != "schedule" && $SKIP_CI == "false"'
- if: '$CI_PIPELINE_SOURCE != "schedule" && $SKIP_CI == "true"'
- if: '$CI_PIPELINE_SOURCE != "schedule" && $SKIP_CI == "false" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'
- if: '$CI_PIPELINE_SOURCE != "schedule" && $SKIP_CI == "true" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'
when: manual

.upstream_rules_all:
rules:
- if: '$CI_PIPELINE_SOURCE != "schedule"'
- if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'

.upstream_rules_x86_64:
rules:
- if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER =~ "/^.*(x86_64).*$/"'
- if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER =~ "/^.*(x86_64).*$/" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'
atodorov marked this conversation as resolved.
Show resolved Hide resolved

.upstream_rules_ga_all:
rules:
- if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER'

.nightly_rules_all:
rules:
Expand All @@ -74,6 +78,16 @@ init:
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-9.5-[^ga][\S]+/ && $RUNNER =~ "/^.*(x86_64).*$/" && $NIGHTLY == "true" && $RHEL_MAJOR == "9"'

.ga_rules_all:
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $NIGHTLY== "false"'
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $NIGHTLY == "false"'

.ga_rules_x86_64:
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $RUNNER =~ "/^.*(x86_64).*$/" && $NIGHTLY== "false"'
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $RUNNER =~ "/^.*(x86_64).*$/" && $NIGHTLY == "false"'

Comment on lines +81 to +90
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's sufficient to rely on NIGHTLY == false in addition to being a scheduled pipeline. Why not add a new variable like GA == true and check that instead? If we add a new schedule which will also set NIGHTLY=false but will do something else it would trigger these jobs as well.

Also the rules are duplicate, just extra space on the second one.

In addition to this we also want to run all these jobs on main just like before which I don't think will work now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not add a new variable like GA == true and check that instead?

@jrusz see #4046 (comment).

If we add a new schedule which will also set NIGHTLY=false but will do something else it would trigger these jobs as well.

That's true but let's worry about it when it happens. Currently this is not what we intend to do.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok fair enough, let's keep that part as it is now.


.RPM_RUNNERS_RHEL: &RPM_RUNNERS_RHEL
RUNNER:
Expand All @@ -87,6 +101,7 @@ RPM:
extends: .terraform
rules:
- !reference [.build_rules, rules]
- !reference [.ga_rules_all, rules]
script:
- sh "schutzbot/mockbuild.sh"
parallel:
Expand All @@ -102,6 +117,8 @@ RPM:
- aws/rhel-8.9-ga-aarch64
- aws/rhel-9.3-ga-x86_64
- aws/rhel-9.3-ga-aarch64
- aws/centos-stream-8-x86_64
- aws/centos-stream-8-aarch64
- aws/centos-stream-9-x86_64
- aws/centos-stream-9-aarch64
- <<: *RPM_RUNNERS_RHEL
Expand All @@ -125,7 +142,7 @@ Container:
stage: rpmbuild
extends: .terraform
rules:
- !reference [.build_rules, rules]
- !reference [.ga_rules_all, rules]
script:
- sh "schutzbot/containerbuild.sh"
parallel:
Expand All @@ -137,7 +154,7 @@ Packer:
stage: test
extends: .terraform
rules:
- if: '$CI_PIPELINE_SOURCE != "schedule"'
- !reference [.ga_rules_all, rules]
script:
- tools/ci-build-worker-packer.sh
variables:
Expand Down Expand Up @@ -169,6 +186,7 @@ Base:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/base_tests.sh
Expand Down Expand Up @@ -198,6 +216,7 @@ Base:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/${SCRIPT}
Expand Down Expand Up @@ -239,23 +258,28 @@ regression-composer-works-behind-satellite:

regression-excluded-dependency:
extends: .regression
rules:
# WHITELIST
- if: $RUNNER =~ "/^.*(rhel-8.*|rhel-9.*|centos-stream-9).*$/" && $CI_PIPELINE_SOURCE != "schedule"
rules:
# WHITELIST & BLACKLIST: excluding GA runners from the PR pipeline
- if: $RUNNER =~ "/^.*(rhel-8.*|rhel-9.*|centos-stream-9).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
variables:
SCRIPT: regression-excluded-dependency.sh

regression-include-excluded-packages:
extends: .regression
rules:
# BLACKLIST: Skipped on fedora systems
- if: $RUNNER !~ "/^.*(fedora).*$/" && $CI_PIPELINE_SOURCE != "schedule"
# BLACKLIST: Skipped on fedora systems & excluding GA runners from the PR pipeline
- if: $RUNNER !~ "/^.*(fedora).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to my comment above. The rule addition is applied correctly here as it is a blacklist rule just please update the comment above.

- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
variables:
SCRIPT: regression-include-excluded-packages.sh

regression-old-worker-new-composer:
rules:
- !reference [.upstream_rules_ga_all, rules]
- !reference [.ga_rules_all, rules]
parallel:
matrix:
- RUNNER:
Expand All @@ -269,19 +293,21 @@ regression-old-worker-new-composer:
regression-insecure-repo:
extends: .regression
rules:
# WHITELIST
- if: $RUNNER =~ "/^.*(rhel-*).*$/" && $CI_PIPELINE_SOURCE != "schedule"
# WHITELIST & BLACKLIST: excluding GA runners from the PR pipeline
- if: $RUNNER =~ "/^.*(rhel-*).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update to a whitelist rule. See comment above for details.

- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
variables:
SCRIPT: regression-insecure-repo.sh


regression-no-explicit-rootfs-definition:
extends: .regression
rules:
# BLACKLIST: Skipped on fedora systems
- if: $RUNNER !~ "/^.*(fedora).*$/" && $CI_PIPELINE_SOURCE != "schedule"
# BLACKLIST: Skipped on fedora systems & excluding GA runners from the PR pipeline
- if: $RUNNER !~ "/^.*(fedora).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the comment. See my comment above for details.

- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
variables:
SCRIPT: regression-no-explicit-rootfs-definition.sh

Expand Down Expand Up @@ -318,6 +344,7 @@ Trigger-rhel-edge-ci:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/${SCRIPT}
Expand Down Expand Up @@ -360,7 +387,8 @@ koji.sh (cloud upload):
stage: test
extends: .terraform
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.upstream_rules_ga_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/koji.sh cloud-upload ${CLOUD_TARGET} ${IMAGE_TYPE}
Expand Down Expand Up @@ -389,7 +417,8 @@ koji.sh (cloudapi):
extends: .integration
# Not supported in nightly pipelines
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.upstream_rules_ga_all, rules]
- !reference [.ga_rules_all, rules]
variables:
SCRIPT: koji.sh
parallel:
Expand All @@ -403,8 +432,7 @@ koji.sh (cloudapi):
aws.sh:
extends: .integration
rules:
# Skip rhel-8.4-ga-x86_64
- if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-8.4-[\S]+/'
- if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-9.5-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "9"'
variables:
SCRIPT: aws.sh
Expand All @@ -414,6 +442,7 @@ oci.sh:
rules:
# Run only on x86_64
- !reference [.upstream_rules_x86_64, rules]
- !reference [.ga_rules_x86_64, rules]
variables:
SCRIPT: oci.sh

Expand All @@ -435,6 +464,7 @@ azure.sh:
# Run only on x86_64
- !reference [.upstream_rules_x86_64, rules]
- !reference [.nightly_rules_x86_64, rules]
- !reference [.ga_rules_x86_64, rules]
variables:
SCRIPT: azure.sh

Expand All @@ -444,6 +474,7 @@ azure.sh_hyperv_gen2:
# Run only on x86_64
- !reference [.upstream_rules_x86_64, rules]
- !reference [.nightly_rules_x86_64, rules]
- !reference [.ga_rules_x86_64, rules]
variables:
SCRIPT: azure_hyperv_gen2.sh

Expand All @@ -453,6 +484,7 @@ gcp.sh:
rules:
- !reference [.upstream_rules_x86_64, rules]
- !reference [.nightly_rules_x86_64, rules]
- !reference [.ga_rules_x86_64, rules]
variables:
SCRIPT: gcp.sh

Expand All @@ -462,6 +494,7 @@ vmware.sh_vmdk:
# Run only on x86_64
- !reference [.upstream_rules_x86_64, rules]
- !reference [.nightly_rules_x86_64, rules]
- !reference [.ga_rules_x86_64, rules]
variables:
SCRIPT: vmware.sh vmdk

Expand All @@ -471,6 +504,7 @@ vmware.sh_ova:
# Run only on x86_64
- !reference [.upstream_rules_x86_64, rules]
- !reference [.nightly_rules_x86_64, rules]
- !reference [.ga_rules_x86_64, rules]
variables:
SCRIPT: vmware.sh ova

Expand Down Expand Up @@ -507,7 +541,8 @@ API:
stage: test
extends: .terraform
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.upstream_rules_ga_all, rules]
- !reference [.ga_rules_all, rules]
# note: cloud API is not supported for on-prem installations so
# don't run this test case for nightly trees
script:
Expand Down Expand Up @@ -539,7 +574,8 @@ API-module-hotfixes:
stage: test
extends: .terraform
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.upstream_rules_ga_all, rules]
- !reference [.ga_rules_all, rules]
# note: cloud API is not supported for on-prem installations so
# don't run this test case for nightly trees
script:
Expand Down Expand Up @@ -567,6 +603,7 @@ API-module-hotfixes:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/${SCRIPT}
Expand All @@ -593,6 +630,7 @@ ubi-wsl.sh:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/ubi-wsl.sh
Expand All @@ -609,6 +647,7 @@ weldr-distro-dot-notation+aliases:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/weldr-distro-dot-notation-and-aliases.sh
Expand All @@ -623,8 +662,9 @@ weldr-distro-dot-notation+aliases:
extends: .libvirt_integration
rules:
# BLACKLIST
- if: $RUNNER !~ "/^.*(rhel-9.5).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $NIGHTLY != "true"
- if: $RUNNER !~ "/^.*(rhel-9.5).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $NIGHTLY != "true" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]

generic_s3_http.sh:
extends: .generic_s3
Expand All @@ -650,7 +690,8 @@ RHEL 9 on 8:
stage: test
extends: .terraform
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.upstream_rules_ga_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/koji.sh
Expand Down Expand Up @@ -691,7 +732,7 @@ NIGHTLY_FAIL:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $NIGHTLY == "true"'
when: on_failure
script:
- schutzbot/slack_notification.sh FAILED ":big-sad:"
- schutzbot/slack_notification.sh FAILED ":big-sad:" nightly

NIGHTLY_SUCCESS:
stage: finish
Expand All @@ -700,14 +741,34 @@ NIGHTLY_SUCCESS:
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $NIGHTLY == "true"'
script:
- schutzbot/slack_notification.sh SUCCESS ":partymeow:"
- schutzbot/slack_notification.sh SUCCESS ":partymeow:" nightly

GA_FAIL:
stage: finish
tags:
- shell
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $NIGHTLY== "false"'
when: on_failure
script:
- schutzbot/slack_notification.sh FAILED ":big-sad:" ga

GA_SUCCESS:
stage: finish
tags:
- shell
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $NIGHTLY== "false"'
script:
- schutzbot/slack_notification.sh SUCCESS ":partymeow:" ga

Installer:
stage: test
extends: .terraform/openstack
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/installers.sh
Expand All @@ -722,6 +783,7 @@ ContainerUpload:
extends: .terraform
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/container-upload.sh
Expand All @@ -737,6 +799,7 @@ ContainerEmbedding:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/container-embedding.sh
Expand All @@ -753,6 +816,7 @@ WorkerExecutor:
extends: .terraform
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/worker-executor.sh
Expand Down
6 changes: 5 additions & 1 deletion schutzbot/slack_notification.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,11 @@ fi

COMPOSE_ID=$(cat COMPOSE_ID)
COMPOSER_NVR=$(cat COMPOSER_NVR)
MESSAGE="\"Nightly pipeline execution on *$COMPOSE_ID* with *$COMPOSER_NVR* finished with status *$1* $2 \n QE: @atodorov, @jrusz\n Link to results: $CI_PIPELINE_URL\n For edge testing status please see https://url.corp.redhat.com/edge-pipelines \""
if [ "$3" == "ga" ]; then
MESSAGE="\"GA composes pipeline execution finished with status *$1* $2 \n QE: @atodorov, @jrusz, @tkosciel\n Link to results: $CI_PIPELINE_URL \""
else
MESSAGE="\"Nightly pipeline execution on *$COMPOSE_ID* with *$COMPOSER_NVR* finished with status *$1* $2 \n QE: @atodorov, @jrusz, @tkosciel\n Link to results: $CI_PIPELINE_URL\n For edge testing status please see https://url.corp.redhat.com/edge-pipelines \""
fi

curl \
-X POST \
Expand Down