Test failed because of 'elrepo' repository URL has been changed for CentOS7

[juliayakovlev]

AMI ami-0d5aee59acfb93c8f) (version 4.0.11) problem:

elrepo repository URL has been changed for CentOS7. Due this change yum update failes:

Command: 'sudo yum update -y --skip-broken'
Exit code: 1
Stdout:
Transaction Summary
================================================================================
Install    2 Packages (+1 Dependent package)
Upgrade  117 Packages
Total size: 297 M
Total download size: 65 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
Stderr:
Trying other mirror.
http://mirror.imt-systems.com/elrepo/archive/kernel/el7/x86_64/RPMS/python-perf-5.9.1-1.el7.elrepo.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
Error downloading packages:
python-perf-5.9.1-1.el7.elrepo.x86_64: [Errno 256] No more mirrors to try.
kernel-ml-5.9.1-1.el7.elrepo.x86_64: [Errno 256] No more mirrors to try.
kernel-ml-devel-5.9.1-1.el7.elrepo.x86_64: [Errno 256] No more mirrors to try.

Failed test: https://jenkins.scylladb.com/view/nexts/job/enterprise-2020.1/job/rolling-upgrade/job/rolling-upgrade-ami-test/40/console

[penberg]

@juliayakovlev When you say "elrepo repository URL" has changed, what do you mean? There's no recent changes in this area.

[juliayakovlev]

@juliayakovlev When you say "elrepo repository URL" has changed, what do you mean? There's no recent changes in this area.

If run, for example, yum repolist on the instance that was created with this ami - repodata is not found:

[centos@ip-10-0-12-182 ~]$ yum repolist
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: d36uatko69830t.cloudfront.net
 * extras: d36uatko69830t.cloudfront.net
 * updates: d36uatko69830t.cloudfront.net
http://mirror.imt-systems.com/elrepo/archive/kernel/el7/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found

Same problem when yum update -y --skip-broken runs during the test - received error

http://mirror.imt-systems.com/elrepo/archive/kernel/el7/x86_64/RPMS/python-perf-5.9.1-1.el7.elrepo.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
Error downloading packages:
python-perf-5.9.1-1.el7.elrepo.x86_64: [Errno 256] No more mirrors to try.
kernel-ml-5.9.1-1.el7.elrepo.x86_64: [Errno 256] No more mirrors to try.
kernel-ml-devel-5.9.1-1.el7.elrepo.x86_64: [Errno 256] No more mirrors to try.

and test fails

If open the link http://mirror.imt-systems.com/elrepo/archive/kernel/el7/x86_64/repodata/repomd.xml - page not found

[penberg]

I think the issue is in the http://mirror.imt-systems.com/elrepo/ mirror, but I wonder why it did not try other mirrors automatically...

[penberg]

Perhaps /etc/yum.repos.d/elrepo.repo only has this mirror configured?

[juliayakovlev]

I think the issue is in the http://mirror.imt-systems.com/elrepo/ mirror, but I wonder why it did not try other mirrors automatically...

AFAIK repo configuration is part of the ami? @bentsi can you advise?

[penberg]

Yeah:

[centos@ip-172-31-47-66 ~]$ cat /etc/yum.repos.d/elrepo-archive.repo
[kernel-ml-archive]
name=kernel-ml-archive
baseurl=http://mirror.imt-systems.com/elrepo/archive/kernel/el7/x86_64/
enabled=1
gpgcheck=0

@juliayakovlev So the situation is as follows: the 4.0 AMI has a hard-coded URL to mirror.imt-systems.com, which for whatever reason no longer hosts the packages. This could be a transient error, but for now the 4.0 AMI won't work for upgrades. This is not a regression we can fix in the code, but buggy code that relies on an external service that has now changed.

As a workaround, the broken repository can be disabled with:

sudo yum-config-manager --disable kernel-ml-archive

but it's not clear to me if it will break something else.

Let me CC @syuu1228 and @slivne too.

[syuu1228]

elrepo-archive.repo is created by this commit: 7084433
The commit is already reverted on recent releases, but this issue is about upgrading from older AMI, the file is still exists.
I guess we should document to suggest removing /etc/yum.repos.d/elrepo-archive.repo and then upgrade latest kernel-ml kernel.

see: scylladb/scylladb#7036

[slivne]

Thanks so this is not a blocker

• lets test that the solution provided works
• we need a knowledgebase / faq / release notes so this will be fixed

[penberg]

The following steps unblock the upgrade for me:

sudo rm /etc/yum.repos.d/elrepo-archive.repo
sudo yum install https://www.elrepo.org/elrepo-release-7.el7.elrepo.noarch.rpm

@syuu1228, can you please confirm that this is the kind of workaround you were thinking of?

[syuu1228]

@penberg Yes, suggested commands should fix the problem.
But even the problem fixed, the environment still has outdated kernel we used to pinning for workaround.
To resolve this, better to run
'yum --enablerepo=elrepo-kernel install kernel-ml' and 'reboot'.

[yaronkaikov]

Version 4.0.11 is EOL long time ago and we already switched to AMI's based on ubuntu:20.04