fix: replace broken centos 8 appstream repo with alma linux repos #227

supershal · 2022-02-01T01:27:28Z

What problem does this PR solve?:
The centos 8 appstream repo is deprecated and not available anymore. this breaks builds at multiple points.

replace centos 8 appstream repo with alma linux repos
download "epel-release" repos for OS/VERSION family
refactor "package" role to remove duplicate repo handling code for "RedHat" family
Fix "save-images" target to accept any overrides including "overrides/fips.yaml" to download fips compatible k8s images.

Deprecation reference links:
https://centos.org/cl-vs-cs/
https://forums.centos.org/viewtopic.php?f=54&t=78026

Which issue(s) does this PR fix?:

https://jira.d2iq.com/browse/D2IQ-NUMBER

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

github-actions · 2022-02-01T01:44:51Z

File	Coverage
All files	`3%`	❌
pkg/ansible/runner.go	`0%`	❌
pkg/app/artifacts.go	`0%`	❌
pkg/app/build.go	`7%`	❌
pkg/app/errors.go	`0%`	❌
pkg/app/provision.go	`0%`	❌
pkg/app/root.go	`0%`	❌
pkg/app/validate.go	`0%`	❌
pkg/appansible/io.go	`0%`	❌
pkg/appansible/playbook.go	`0%`	❌
pkg/logging/logger.go	`0%`	❌
pkg/packer/manifest.go	`0%`	❌
pkg/packer/packer.go	`0%`	❌
pkg/stringutil/rand.go	`0%`	❌
pkg/version/info.go	`0%`	❌

Minimum allowed coverage is 75%

Generated by 🐒 cobertura-action against 7a447f5

ansible/roles/packages/tasks/repo-redhat.yaml

dkoshkin · 2022-02-01T15:56:49Z

~~Im a little confused by the epel-release changes, will this deprecation also impact konvoy1 https://github.com/mesosphere/konvoy/search?q=epel?~~

It does:

[+] Building 14.9s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                     0.0s
 => => transferring dockerfile: 244B                                                                                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/centos:8.2.2004                                                                                                                                                                                       1.1s
 => [internal] load build context                                                                                                                                                                                                                        0.0s
 => => transferring context: 1.47kB                                                                                                                                                                                                                      0.0s
 => [1/3] FROM docker.io/library/centos:8.2.2004@sha256:4062bbdd1bb0801b0aa38e0f83dece70fb7a5e9bce223423a68de2d8b784b43b                                                                                                                                12.5s
 => => resolve docker.io/library/centos:8.2.2004@sha256:4062bbdd1bb0801b0aa38e0f83dece70fb7a5e9bce223423a68de2d8b784b43b                                                                                                                                 0.0s
 => => sha256:4062bbdd1bb0801b0aa38e0f83dece70fb7a5e9bce223423a68de2d8b784b43b 762B / 762B                                                                                                                                                               0.0s
 => => sha256:fd84102fc72960dd1b8da0ee3b4c13e3b0c1d2a085de118bc4c97821cd986e02 529B / 529B                                                                                                                                                               0.0s
 => => sha256:831691599b88ad6cc2a4abbd0e89661a121aff14cfa289ad840fd3946f274f1f 2.18kB / 2.18kB                                                                                                                                                           0.0s
 => => sha256:6910e5a164f725142d77994b247ba20040477fbab49a721bdbe8d61cf855ac23 74.87MB / 74.87MB                                                                                                                                                         8.8s
 => => extracting sha256:6910e5a164f725142d77994b247ba20040477fbab49a721bdbe8d61cf855ac23                                                                                                                                                                3.6s
 => [2/3] COPY *.repo /etc/yum.repos.d/                                                                                                                                                                                                                  0.4s
 => ERROR [3/3] RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm &&     yum -y install yum-utils &&     yum clean all                                                                                           0.6s
------
 > [3/3] RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm &&     yum -y install yum-utils &&     yum clean all:
#7 0.590 CentOS-8 - AppStream                            179  B/s |  38  B     00:00
#7 0.594 Error: Failed to download metadata for repo 'AppStream': Cannot prepare internal mirrorlist: No URLs in mirrorlist
------
executor failed running [/bin/sh -c yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm &&     yum -y install yum-utils &&     yum clean all]: exit code: 1
make[3]: *** [rpm-artifacts-getter] Error 1
make[2]: *** [rpm-artifacts-getter.check] Error 2
make[1]: *** [download-rpms] Error 2
make: *** [download-all-rpms] Error 2

ansible/roles/packages/tasks/repo-redhat.yaml

supershal · 2022-02-01T16:15:20Z

Wherever we use Centos 8 docker image or AMI, the "appstream" and "baseos" mirror list are broken. We have to either replace with some other repo or use fixed working "base_url" instead of mirror list.

faiq · 2022-02-01T20:57:15Z

ansible/roles/packages/tasks/repo-redhat.yaml

+    # The correct epel-release rpms will be downloaded for Centos, RHEL and oracle 7 repos from in-built extra repos
+    # epel-release for Centos/RHEL/Oracle 8 will be installed using Alma linux "extra" repository
+    - name: install epel-release
+      yum:
+        name: epel-release
+        state: present


Where are the in-built extra repos coming from? Just a little confused Oracle7 seems to be failing on this for some reason

in-built extra repos are installed with the OS in /etc/yum.repos.d/ . each distribution has its own base, extra and appstream repos. Those are not always in sync 100% with their upstream counter part.

added back centos 7 repositories. The changes in this PR only affects centos 8 based repos.

supershal · 2022-02-02T00:42:53Z

ansible/roles/version/tasks/main.yaml

    sed 's/$/{{ '_' + kubernetes_build_tag if kubernetes_build_tag }}/' |
-    sed 's/^\(.\+pause:.\+\)_{{ kubernetes_build_tag }}$/\1/'
+    sed 's/^\(.\+pause:.\+\)_{{ kubernetes_build_tag }}$/\1/' |
+    sed 's#^\(.\+coredns:\)\(.\+\)_{{ kubernetes_build_tag }}#{{ coredns_image_registry_repository }}:\2#'


a bit ugly fix to make etcd and coredns images work for both FIPS and non-FIPS versions.
I intend to rewrite it in future.

Non FIPS images:

(item=k8s.gcr.io/kube-apiserver:v1.21.6) (item=k8s.gcr.io/kube-controller-manager:v1.21.6) (item=k8s.gcr.io/kube-scheduler:v1.21.6) (item=k8s.gcr.io/kube-proxy:v1.21.6) (item=k8s.gcr.io/pause:3.4.1) (item=k8s.gcr.io/etcd:3.4.13-0) (item=k8s.gcr.io/coredns/coredns:v1.8.0) (item=k8s.gcr.io/pause:3.2) (item=docker.io/mesosphere/kfips:v0.1.0)

FIPS changes:

(item=docker.io/mesosphere/kube-apiserver:v1.21.6_fips.0) (item=docker.io/mesosphere/kube-controller-manager:v1.21.6_fips.0) (item=docker.io/mesosphere/kube-scheduler:v1.21.6_fips.0) (item=docker.io/mesosphere/kube-proxy:v1.21.6_fips.0) (item=docker.io/mesosphere/pause:3.4.1) (item=docker.io/mesosphere/etcd:v3.4.13_fips.0) (item=k8s.gcr.io/coredns/coredns:v1.8.0) (item=k8s.gcr.io/pause:3.2) (item=docker.io/mesosphere/kfips:v0.1.0)

supershal · 2022-02-02T02:24:53Z

@faiq @dkoshkin The builds were breaking at multiple points because of two issues: Centos 8 deprecation and bug in pulling fips images.

replaces centos 8 repos with alma linux's extra repo
added back logic to install epel repo for centos/oracle 7
fixed the dynamic image parsing logic to pull correct version of etcd and coredns for both fips and non-fips mode
removed makefile target for centos-8 and centos-8-nvidia buids

supershal · 2022-02-02T02:36:31Z

Should we move "E2E CentOS 7 NVIDIA (Konvoy Image Builder" builds on Main branch only or nightly build? Or we can try to build images in different AWS region so that PRs not blocked by the lack of p2 instances?

dkoshkin

Thanks!

fix: replace broken centos 8 appstream repo with alma linux repos

39bca83

github-actions bot added fix and removed fix labels Feb 1, 2022

fix: use tab instead of space in Makefile target

e883bff

shalin patel added 2 commits January 31, 2022 18:02

fix: add extra args to save-image target only if provided

7c617cd

fix: remove built in appstream repo from image

66eef33

supershal force-pushed the shalin/fix-epel branch from 0870ae3 to 66eef33 Compare February 1, 2022 02:28

shalin patel added 2 commits January 31, 2022 19:19

add kuberentes-version flag when downloading images

0add68c

add back offline repo for redhat

0db3ae8

faiq reviewed Feb 1, 2022

View reviewed changes

ansible/roles/packages/tasks/repo-redhat.yaml Outdated Show resolved Hide resolved

dkoshkin reviewed Feb 1, 2022

View reviewed changes

ansible/roles/packages/tasks/repo-redhat.yaml Outdated Show resolved Hide resolved

dkoshkin reviewed Feb 1, 2022

View reviewed changes

ansible/roles/packages/tasks/repo-redhat.yaml Outdated Show resolved Hide resolved

ansible/roles/packages/tasks/repo-redhat.yaml Outdated Show resolved Hide resolved

faiq reviewed Feb 1, 2022

View reviewed changes

fix: add back EPEL repo for centos 7

267aa78

supershal commented Feb 2, 2022

View reviewed changes

shalin patel added 2 commits February 1, 2022 16:53

fix: remove centos 8 build targets

d51d4c4

fix: parameterize alma linux repo details

7a447f5

supershal requested review from dkoshkin and faiq February 2, 2022 02:38

supershal marked this pull request as ready for review February 2, 2022 02:38

dkoshkin approved these changes Feb 2, 2022

View reviewed changes

faiq approved these changes Feb 2, 2022

View reviewed changes

faiq merged commit ada2ca9 into main Feb 2, 2022

faiq deleted the shalin/fix-epel branch February 2, 2022 15:51

github-actions bot mentioned this pull request Feb 2, 2022

chore(main): release 1.6.0 #204

Merged

github-actions bot mentioned this pull request May 18, 2022

chore(main): release 1.14.0 #346

Merged

mesosphere-ci mentioned this pull request Feb 8, 2023

chore(main): release 2.0.0 #660

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: replace broken centos 8 appstream repo with alma linux repos #227

fix: replace broken centos 8 appstream repo with alma linux repos #227

supershal commented Feb 1, 2022 •

edited

Loading

github-actions bot commented Feb 1, 2022 •

edited

Loading

dkoshkin commented Feb 1, 2022 •

edited

Loading

supershal commented Feb 1, 2022

faiq Feb 1, 2022

supershal Feb 1, 2022

supershal Feb 2, 2022

supershal Feb 2, 2022

supershal Feb 2, 2022

supershal commented Feb 2, 2022

supershal commented Feb 2, 2022

dkoshkin left a comment

fix: replace broken centos 8 appstream repo with alma linux repos #227

fix: replace broken centos 8 appstream repo with alma linux repos #227

Conversation

supershal commented Feb 1, 2022 • edited Loading

github-actions bot commented Feb 1, 2022 • edited Loading

dkoshkin commented Feb 1, 2022 • edited Loading

supershal commented Feb 1, 2022

faiq Feb 1, 2022

Choose a reason for hiding this comment

supershal Feb 1, 2022

Choose a reason for hiding this comment

supershal Feb 2, 2022

Choose a reason for hiding this comment

supershal Feb 2, 2022

Choose a reason for hiding this comment

supershal Feb 2, 2022

Choose a reason for hiding this comment

supershal commented Feb 2, 2022

supershal commented Feb 2, 2022

dkoshkin left a comment

Choose a reason for hiding this comment

supershal commented Feb 1, 2022 •

edited

Loading

github-actions bot commented Feb 1, 2022 •

edited

Loading

dkoshkin commented Feb 1, 2022 •

edited

Loading