Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a nightly pipeline for GA/EUS releases #4046

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
+82 −22
Draft

Create a nightly pipeline for GA/EUS releases #4046

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b036aa7
Create new nightly pipeline for GA/EUS images
tkoscieln Mar 28, 2024
d1d2404
Add exclude rule and rename the pipeline
tkoscieln Apr 29, 2024
8ab3ede
Rename the slack jobs
tkoscieln Apr 29, 2024
ff7276f
Change parameters in slack notification
tkoscieln Apr 29, 2024
4a1fd3b
Add old GA exclusion, rename pipeline and add exceptions on jobs
tkoscieln Apr 30, 2024
7e784fd
Exclude the GA jobs from PR pipeline - fix
tkoscieln May 2, 2024
90449ab
Edit comments and simplify rule
tkoscieln May 6, 2024
098e858
Rename slack notification message
tkoscieln May 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 73 additions & 16 deletions .gitlab-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,17 +54,17 @@ init:

.build_rules:
rules:
- if: '$CI_PIPELINE_SOURCE != "schedule" && $SKIP_CI == "false"'
- if: '$CI_PIPELINE_SOURCE != "schedule" && $SKIP_CI == "true"'
- if: '$CI_PIPELINE_SOURCE != "schedule" && $SKIP_CI == "false" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'
- if: '$CI_PIPELINE_SOURCE != "schedule" && $SKIP_CI == "true" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'
when: manual

.upstream_rules_all:
rules:
- if: '$CI_PIPELINE_SOURCE != "schedule"'
- if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'

.upstream_rules_x86_64:
rules:
- if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER =~ "/^.*(x86_64).*$/"'
- if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER =~ "/^.*(x86_64).*$/" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'
atodorov marked this conversation as resolved.
Show resolved Hide resolved

.nightly_rules_all:
rules:
Expand All @@ -76,6 +76,16 @@ init:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-9.4-[^ga][\S]+/ && $RUNNER =~ "/^.*(x86_64).*$/" && $NIGHTLY == "true" && $RHEL_MAJOR == "9"'
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-8.10-[^ga][\S]+/ && $RUNNER =~ "/^.*(x86_64).*$/" && $NIGHTLY == "true" && $RHEL_MAJOR == "8"'

.ga_rules_all:
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $NIGHTLY== "false"'
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $NIGHTLY == "false"'

.ga_rules_x86_64:
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $RUNNER =~ "/^.*(x86_64).*$/" && $NIGHTLY== "false"'
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/ && $RUNNER =~ "/^.*(x86_64).*$/" && $NIGHTLY == "false"'

Comment on lines +79 to +88
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's sufficient to rely on NIGHTLY == false in addition to being a scheduled pipeline. Why not add a new variable like GA == true and check that instead? If we add a new schedule which will also set NIGHTLY=false but will do something else it would trigger these jobs as well.

Also the rules are duplicate, just extra space on the second one.

In addition to this we also want to run all these jobs on main just like before which I don't think will work now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not add a new variable like GA == true and check that instead?

@jrusz see #4046 (comment).

If we add a new schedule which will also set NIGHTLY=false but will do something else it would trigger these jobs as well.

That's true but let's worry about it when it happens. Currently this is not what we intend to do.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok fair enough, let's keep that part as it is now.


.RPM_RUNNERS_RHEL: &RPM_RUNNERS_RHEL
RUNNER:
Expand All @@ -91,6 +101,7 @@ RPM:
extends: .terraform
rules:
- !reference [.build_rules, rules]
- !reference [.ga_rules_all, rules]
script:
- sh "schutzbot/mockbuild.sh"
parallel:
Expand Down Expand Up @@ -131,7 +142,7 @@ Container:
stage: rpmbuild
extends: .terraform
rules:
- !reference [.build_rules, rules]
- !reference [.ga_rules_all, rules]
script:
- sh "schutzbot/containerbuild.sh"
parallel:
Expand All @@ -143,7 +154,7 @@ Packer:
stage: test
extends: .terraform
rules:
- if: '$CI_PIPELINE_SOURCE != "schedule"'
- !reference [.ga_rules_all, rules]
script:
- tools/ci-build-worker-packer.sh
variables:
Expand Down Expand Up @@ -176,6 +187,7 @@ Base:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/base_tests.sh
Expand Down Expand Up @@ -209,6 +221,7 @@ Base:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/${SCRIPT}
Expand Down Expand Up @@ -256,23 +269,27 @@ regression-composer-works-behind-satellite:

regression-excluded-dependency:
extends: .regression
rules:
rules:
# WHITELIST
- if: $RUNNER =~ "/^.*(rhel-8.*|rhel-9.*|centos-stream-8|centos-stream-9).*$/" && $CI_PIPELINE_SOURCE != "schedule"
- if: $RUNNER =~ "/^.*(rhel-8.*|rhel-9.*|centos-stream-8|centos-stream-9).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment above # WHITELIST means that the rule specified here is defining which runners are allowed to run this job. The rule you're adding is doing the opposite, it's defining which runner to NOT run this on which would be a # BLACKLIST rule.

We have added these comments in the past above these seemingly complicated rules to indicate what they're trying to do to help anyone looking at this understand what's the rule trying to do. This change you're adding would confuse the user now.

- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
variables:
SCRIPT: regression-excluded-dependency.sh

regression-include-excluded-packages:
extends: .regression
rules:
# BLACKLIST: Skipped on fedora systems
- if: $RUNNER !~ "/^.*(fedora).*$/" && $CI_PIPELINE_SOURCE != "schedule"
- if: $RUNNER !~ "/^.*(fedora).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to my comment above. The rule addition is applied correctly here as it is a blacklist rule just please update the comment above.

- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
variables:
SCRIPT: regression-include-excluded-packages.sh

regression-old-worker-new-composer:
rules:
- !reference [.ga_rules_all, rules]
parallel:
matrix:
- RUNNER:
Expand All @@ -287,8 +304,9 @@ regression-insecure-repo:
extends: .regression
rules:
# WHITELIST
- if: $RUNNER =~ "/^.*(rhel-*).*$/" && $CI_PIPELINE_SOURCE != "schedule"
- if: $RUNNER =~ "/^.*(rhel-*).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update to a whitelist rule. See comment above for details.

- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
variables:
SCRIPT: regression-insecure-repo.sh

Expand All @@ -297,8 +315,9 @@ regression-no-explicit-rootfs-definition:
extends: .regression
rules:
# BLACKLIST: Skipped on fedora systems
- if: $RUNNER !~ "/^.*(fedora).*$/" && $CI_PIPELINE_SOURCE != "schedule"
- if: $RUNNER !~ "/^.*(fedora).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the comment. See my comment above for details.

- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
variables:
SCRIPT: regression-no-explicit-rootfs-definition.sh

Expand Down Expand Up @@ -337,6 +356,7 @@ Trigger-rhel-edge-ci:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/${SCRIPT}
Expand Down Expand Up @@ -382,7 +402,7 @@ koji.sh (cloud upload):
stage: test
extends: .terraform
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/koji.sh cloud-upload ${CLOUD_TARGET} ${IMAGE_TYPE}
Expand Down Expand Up @@ -412,6 +432,7 @@ koji.sh (cloudapi):
# Not supported in nightly pipelines
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.ga_rules_all, rules]
variables:
SCRIPT: koji.sh
parallel:
Expand All @@ -426,7 +447,7 @@ aws.sh:
extends: .integration
rules:
# Skip rhel-8.4-ga-x86_64
- if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-8.4-[\S]+/'
- if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-8.4-[\S]+/ && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this rule was just skipping 8.4 GA and your new rule skips ALL GA releases it should include that one as well and could be replaced instead of adding it on top.

- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-9.4-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "9"'
- if: '$CI_PIPELINE_SOURCE == "schedule" && $RUNNER =~ /[\S]+rhel-8.10-[^ga][\S]+/ && $NIGHTLY == "true" && $RHEL_MAJOR == "8"'
variables:
Expand All @@ -437,6 +458,7 @@ oci.sh:
rules:
# Run only on x86_64
- !reference [.upstream_rules_x86_64, rules]
- !reference [.ga_rules_x86_64, rules]
variables:
SCRIPT: oci.sh

Expand All @@ -458,6 +480,7 @@ azure.sh:
# Run only on x86_64
- !reference [.upstream_rules_x86_64, rules]
- !reference [.nightly_rules_x86_64, rules]
- !reference [.ga_rules_x86_64, rules]
variables:
SCRIPT: azure.sh

Expand All @@ -467,6 +490,7 @@ azure.sh_hyperv_gen2:
# Run only on x86_64
- !reference [.upstream_rules_x86_64, rules]
- !reference [.nightly_rules_x86_64, rules]
- !reference [.ga_rules_x86_64, rules]
variables:
SCRIPT: azure_hyperv_gen2.sh

Expand All @@ -476,6 +500,7 @@ gcp.sh:
rules:
- !reference [.upstream_rules_x86_64, rules]
- !reference [.nightly_rules_x86_64, rules]
- !reference [.ga_rules_x86_64, rules]
variables:
SCRIPT: gcp.sh

Expand All @@ -485,6 +510,7 @@ vmware.sh_vmdk:
# Run only on x86_64
- !reference [.upstream_rules_x86_64, rules]
- !reference [.nightly_rules_x86_64, rules]
- !reference [.ga_rules_x86_64, rules]
variables:
SCRIPT: vmware.sh vmdk

Expand All @@ -494,6 +520,7 @@ vmware.sh_ova:
# Run only on x86_64
- !reference [.upstream_rules_x86_64, rules]
- !reference [.nightly_rules_x86_64, rules]
- !reference [.ga_rules_x86_64, rules]
variables:
SCRIPT: vmware.sh ova

Expand Down Expand Up @@ -531,6 +558,7 @@ API:
extends: .terraform
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.ga_rules_all, rules]
# note: cloud API is not supported for on-prem installations so
# don't run this test case for nightly trees
script:
Expand Down Expand Up @@ -564,6 +592,7 @@ API-module-hotfixes:
extends: .terraform
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.ga_rules_all, rules]
# note: cloud API is not supported for on-prem installations so
# don't run this test case for nightly trees
script:
Expand Down Expand Up @@ -591,6 +620,7 @@ API-module-hotfixes:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/${SCRIPT}
Expand Down Expand Up @@ -619,6 +649,7 @@ ubi-wsl.sh:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/ubi-wsl.sh
Expand All @@ -632,6 +663,7 @@ weldr-distro-dot-notation+aliases:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/weldr-distro-dot-notation-and-aliases.sh
Expand All @@ -646,8 +678,9 @@ weldr-distro-dot-notation+aliases:
extends: .libvirt_integration
rules:
# BLACKLIST
- if: $RUNNER !~ "/^.*(rhel-8.10|rhel-9.4).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $NIGHTLY != "true"
- if: $RUNNER !~ "/^.*(rhel-8.10|rhel-9.4).*$/" && $CI_PIPELINE_SOURCE != "schedule" && $NIGHTLY != "true" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]

generic_s3_http.sh:
extends: .generic_s3
Expand All @@ -674,6 +707,7 @@ RHEL 9 on 8:
extends: .terraform
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/koji.sh
Expand Down Expand Up @@ -714,7 +748,7 @@ NIGHTLY_FAIL:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $NIGHTLY == "true"'
when: on_failure
script:
- schutzbot/slack_notification.sh FAILED ":big-sad:"
- schutzbot/slack_notification.sh FAILED ":big-sad:" nightly

NIGHTLY_SUCCESS:
stage: finish
Expand All @@ -723,14 +757,34 @@ NIGHTLY_SUCCESS:
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $NIGHTLY == "true"'
script:
- schutzbot/slack_notification.sh SUCCESS ":partymeow:"
- schutzbot/slack_notification.sh SUCCESS ":partymeow:" nightly

GA/EUS_FAIL:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover from rename, please change to just GA

stage: finish
tags:
- shell
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $NIGHTLY== "false"'
when: on_failure
script:
- schutzbot/slack_notification.sh FAILED ":big-sad:" ga

GA/EUS_SUCCESS:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover from rename, please change to just GA

stage: finish
tags:
- shell
rules:
- if: '$CI_PIPELINE_SOURCE == "schedule" && $NIGHTLY== "false"'
script:
- schutzbot/slack_notification.sh SUCCESS ":partymeow:" ga

Installer:
stage: test
extends: .terraform/openstack
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/installers.sh
Expand All @@ -746,6 +800,7 @@ ContainerUpload:
extends: .terraform
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/container-upload.sh
Expand All @@ -761,6 +816,7 @@ ContainerEmbedding:
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.nightly_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/container-embedding.sh
Expand All @@ -780,6 +836,7 @@ WorkerExecutor:
extends: .terraform
rules:
- !reference [.upstream_rules_all, rules]
- !reference [.ga_rules_all, rules]
script:
- schutzbot/deploy.sh
- /usr/libexec/tests/osbuild-composer/worker-executor.sh
Expand Down
6 changes: 5 additions & 1 deletion schutzbot/slack_notification.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,11 @@ fi

COMPOSE_ID=$(cat COMPOSE_ID)
COMPOSER_NVR=$(cat COMPOSER_NVR)
MESSAGE="\"Nightly pipeline execution on *$COMPOSE_ID* with *$COMPOSER_NVR* finished with status *$1* $2 \n QE: @atodorov, @jrusz\n Link to results: $CI_PIPELINE_URL\n For edge testing status please see https://url.corp.redhat.com/edge-pipelines \""
if [ "$3" == "ga" ]; then
MESSAGE="\"GA/EUS composes pipeline execution finished with status *$1* $2 \n QE: @atodorov, @jrusz, @tkosciel\n Link to results: $CI_PIPELINE_URL \""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leftover from rename, please change to just GA

else
MESSAGE="\"Nightly pipeline execution on *$COMPOSE_ID* with *$COMPOSER_NVR* finished with status *$1* $2 \n QE: @atodorov, @jrusz, @tkosciel\n Link to results: $CI_PIPELINE_URL\n For edge testing status please see https://url.corp.redhat.com/edge-pipelines \""
fi

curl \
-X POST \
Expand Down
Loading