Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Amazon Linux 2 channels Core x86_64 and Core aarch64 Permission denied - Uyuni server 2022.11 #6244

Closed
eins opened this issue Dec 1, 2022 · 6 comments
Closed

Amazon Linux 2 channels Core x86_64 and Core aarch64 Permission denied - Uyuni server 2022.11 #6244

eins opened this issue Dec 1, 2022 · 6 comments
Labels
bug Something isn't working P5

Comments

@eins
Copy link

eins commented Dec 1, 2022 •
edited

Problem description

Howdy All

I update my uyuni server to version 2022-11 and I have permission errors for amazonlinux core channels, Core x86_64 and
Core aarch64

# Core x86_64
Error syncing the channel: Amazon Linux 2 Core x86_64
Command '[/usr/bin/spacewalk-repo-sync, --channel, amazonlinux2-core-x86_64, --type, yum, --non-interactive]' exited with error code 1: 02:23:47 ======================================
02:23:47 | Channel: amazonlinux2-core-x86_64
02:23:47 ======================================
02:23:47 Sync of channel started.
Retrieving repository 'amazonlinux2-core-x86_64' metadata [.error]
02:23:49 RepoMDError: Cannot access repository.
Repository 'amazonlinux2-core-x86_64' is invalid.
[amazonlinux2-core-x86_64|http://amazonlinux.default.amazonaws.com/2/core/latest/x86_64/mirror.list] Valid metadata not found at specified URL
History:
 - [|] Error trying to read from 'http://amazonlinux.default.amazonaws.com/2/core/latest/x86_64/mirror.list'
 - Permission to access 'http://amazonlinux.default.amazonaws.com/2/core/latest/x86_64/mirror.list/content' denied.

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'amazonlinux2-core-x86_64' because of the above error.
Could not refresh the repositories because of errors.

02:23:49 Total time: 0:00:01


# Core aarch64
Error syncing the channel: Amazon Linux 2 Core aarch64
Command '[/usr/bin/spacewalk-repo-sync, --channel, amazonlinux2-core-aarch64, --type, yum, --non-interactive]' exited with error code 1: 02:34:08 ======================================
02:34:08 | Channel: amazonlinux2-core-aarch64
02:34:08 ======================================
02:34:08 Sync of channel started.
Retrieving repository 'amazonlinux2-core-aarch64' metadata [.error]
02:34:10 RepoMDError: Cannot access repository.
Repository 'amazonlinux2-core-aarch64' is invalid.
[amazonlinux2-core-aarch64|http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list] Valid metadata not found at specified URL
History:
 - [|] Error trying to read from 'http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list'
 - Permission to access 'http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list/content' denied.

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'amazonlinux2-core-aarch64' because of the above error.
Could not refresh the repositories because of errors.

02:34:10 Total time: 0:00:01

I update uyuni server from 2022-08 to 2022-11 and after it I applied the patch described in this ticket #6193

I also check the current amazon linux uyuni documentation client registration and the names look valid:
https://www.uyuni-project.org/uyuni-docs/en/uyuni/client-configuration/clients-amazon.html

spacewalk-common-channels -l | grep  amazon
 amazonlinux2-core:   x86_64, aarch64
 amazonlinux2-extra-docker: x86_64, aarch64
 amazonlinux2-uyuni-client: x86_64, aarch64
 amazonlinux2-uyuni-client-devel: x86_64, aarch64

I also notice that the other two amazonlinux channels do not have errors

==> amazonlinux2-uyuni-client-aarch64.log <==
2022/11/30 03:22:11 -00:00   Patches in repo: 0.
2022/11/30 03:22:12 -00:00 Sync completed.
2022/12/01 03:05:13 -00:00 Command: ['/usr/bin/spacewalk-repo-sync', '--channel', 'amazonlinux2-uyuni-client-aarch64', '--type', 'yum', '--non-interactive']
2022/12/01 03:05:13 -00:00 Sync of channel started.
2022/12/01 03:05:16 -00:00 Repo URL: https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/
2022/12/01 03:05:16 -00:00     Packages in repo:               284
2022/12/01 03:05:17 -00:00     No new packages to sync.
2022/12/01 03:05:17 -00:00
2022/12/01 03:05:17 -00:00   Patches in repo: 0.
2022/12/01 03:05:18 -00:00 Sync completed.

==> amazonlinux2-uyuni-client-x86_64.log <==
2022/11/30 03:23:52 -00:00   Regenerating bootstrap repositories.
2022/11/30 03:25:00 -00:00 Sync completed.
2022/12/01 02:42:25 -00:00 Command: ['/usr/bin/spacewalk-repo-sync', '--channel', 'amazonlinux2-uyuni-client-x86_64', '--type', 'yum', '--non-interactive']
2022/12/01 02:42:25 -00:00 Sync of channel started.
2022/12/01 02:42:28 -00:00 Repo URL: https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/
2022/12/01 02:42:28 -00:00     Packages in repo:               284
2022/12/01 02:42:31 -00:00     No new packages to sync.
2022/12/01 02:42:31 -00:00
2022/12/01 02:42:31 -00:00   Patches in repo: 0.
2022/12/01 02:42:31 -00:00 Sync completed.

any advice related to this is highly appreciated

Steps to reproduce

  1. uyuni server version 2022.11 with the patch https://www.uyuni-project.org/pages/patches.html
  2. you need to have Linux 2 channels for X6_64 and aarch64
  3. sync them
    ...

Uyuni version

zypper info Uyuni-Server-release
Loading repository data...
Reading installed packages...


Information for package Uyuni-Server-release:
---------------------------------------------
Repository     : Uyuni Server Stable
Name           : Uyuni-Server-release
Version        : 2022.11-220400.193.4.uyuni2
Arch           : x86_64
Vendor         : obs://build.opensuse.org/systemsmanagement:Uyuni
Support Level  : Level 3
Installed Size : 1.4 KiB
Installed      : Yes
Status         : up-to-date
Source package : Uyuni-Server-release-2022.11-220400.193.4.uyuni2.src
Summary        : Uyuni Server
Description    :
    Uyuni lets you efficiently manage physical, virtual,
    and cloud-based Linux systems. It provides automated and cost-effective
    configuration and software management, asset management, and system

Uyuni proxy version (if used)

no uyuni proxy involved

Useful logs

tail -f amazonlinux2-*
==> amazonlinux2-core-aarch64.log <==
Repository 'amazonlinux2-core-aarch64' is invalid.
[amazonlinux2-core-aarch64|http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list] Valid metadata not found at specified URL
History:
 - [|] Error trying to read from 'http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list'
 - Permission to access 'http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list/content' denied.

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'amazonlinux2-core-aarch64' because of the above error.
Could not refresh the repositories because of errors.


==> amazonlinux2-core-x86_64.log <==
Repository 'amazonlinux2-core-x86_64' is invalid.
[amazonlinux2-core-x86_64|http://amazonlinux.default.amazonaws.com/2/core/latest/x86_64/mirror.list] Valid metadata not found at specified URL
History:
 - [|] Error trying to read from 'http://amazonlinux.default.amazonaws.com/2/core/latest/x86_64/mirror.list'
 - Permission to access 'http://amazonlinux.default.amazonaws.com/2/core/latest/x86_64/mirror.list/content' denied.

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'amazonlinux2-core-x86_64' because of the above error.
Could not refresh the repositories because of errors.


==> amazonlinux2-uyuni-client-aarch64.log <==
2022/11/30 03:22:11 -00:00   Patches in repo: 0.
2022/11/30 03:22:12 -00:00 Sync completed.
2022/12/01 03:05:13 -00:00 Command: ['/usr/bin/spacewalk-repo-sync', '--channel', 'amazonlinux2-uyuni-client-aarch64', '--type', 'yum', '--non-interactive']
2022/12/01 03:05:13 -00:00 Sync of channel started.
2022/12/01 03:05:16 -00:00 Repo URL: https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/
2022/12/01 03:05:16 -00:00     Packages in repo:               284
2022/12/01 03:05:17 -00:00     No new packages to sync.
2022/12/01 03:05:17 -00:00
2022/12/01 03:05:17 -00:00   Patches in repo: 0.
2022/12/01 03:05:18 -00:00 Sync completed.

==> amazonlinux2-uyuni-client-x86_64.log <==
2022/11/30 03:23:52 -00:00   Regenerating bootstrap repositories.
2022/11/30 03:25:00 -00:00 Sync completed.
2022/12/01 02:42:25 -00:00 Command: ['/usr/bin/spacewalk-repo-sync', '--channel', 'amazonlinux2-uyuni-client-x86_64', '--type', 'yum', '--non-interactive']
2022/12/01 02:42:25 -00:00 Sync of channel started.
2022/12/01 02:42:28 -00:00 Repo URL: https://download.opensuse.org/repositories/systemsmanagement:/Uyuni:/Stable:/CentOS7-Uyuni-Client-Tools/CentOS_7/
2022/12/01 02:42:28 -00:00     Packages in repo:               284
2022/12/01 02:42:31 -00:00     No new packages to sync.
2022/12/01 02:42:31 -00:00
2022/12/01 02:42:31 -00:00   Patches in repo: 0.
2022/12/01 02:42:31 -00:00 Sync completed.

Additional information

No response
@eins eins added bug Something isn't working P5 labels Dec 1, 2022
@juliogonzalez
Copy link
Member

Anything else you see at /var/log/rhn itself?

The way I see it, either Amazon changed something on their side (strange), either we have a regression reading mirror.list files.

That is, of course, assuming your Uyuni Server does not have network issues connecting with the AWS repos for some reason.

Can you try to run curl http://amazonlinux.default.amazonaws.com/2/core/latest/x86_64/mirror.list on the Uyuni server and paste the results?

@eins
Copy link
Author

eins commented Dec 1, 2022

I checked in /var/log/rhn

# file: rhn_taskomatic_daemon.log
2022-12-01 20:20:32,296 [DefaultQuartzScheduler_Worker-14] INFO  com.redhat.rhn.taskomatic.task.RepoSyncTask - Syncing repos for channel: Amazon Linux 2 Core aarch64
2022-12-01 20:20:34,879 [Thread-28929] INFO  com.redhat.rhn.taskomatic.task.RepoSyncTask - 20:20:33 ======================================
20:20:33 | Channel: amazonlinux2-core-aarch64
20:20:33 ======================================
20:20:33 Sync of channel started.
Retrieving repository 'amazonlinux2-core-aarch64' metadata [.error]
20:20:34 RepoMDError: Cannot access repository.
Repository 'amazonlinux2-core-aarch64' is invalid.
[amazonlinux2-core-aarch64|http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list] Valid metadata not found at specified URL
History:
 - [|] Error trying to read from 'http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list'
 - Permission to access 'http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list/content' denied.

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'amazonlinux2-core-aarch64' because of the above error.
Could not refresh the repositories because of errors.

20:20:34 Total time: 0:00:01

2022-12-01 20:20:34,901 [DefaultQuartzScheduler_Worker-14] ERROR com.redhat.rhn.taskomatic.task.RepoSyncTask - Command '[/usr/bin/spacewalk-repo-sync, --channel, amazonlinux2-core-aarch64, --type, yum, --non-interactive]' exited with error code 1: 20:20:33 ======================================
20:20:33 | Channel: amazonlinux2-core-aarch64
20:20:33 ======================================
20:20:33 Sync of channel started.
Retrieving repository 'amazonlinux2-core-aarch64' metadata [.error]
20:20:34 RepoMDError: Cannot access repository.
Repository 'amazonlinux2-core-aarch64' is invalid.
[amazonlinux2-core-aarch64|http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list] Valid metadata not found at specified URL
History:
 - [|] Error trying to read from 'http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list'
 - Permission to access 'http://amazonlinux.default.amazonaws.com/2/core/latest/aarch64/mirror.list/content' denied.

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'amazonlinux2-core-aarch64' because of the above error.
Could not refresh the repositories because of errors.

20:20:34 Total time: 0:00:01

2022-12-01 20:21:34,987 [DefaultQuartzScheduler_Worker-1] INFO  com.redhat.rhn.taskomatic.task.RepoSyncTask - Syncing repos for channel: Ubuntu 18.04 LTS AMD64 Universe for Uyuni
2022-12-01 20:25:38,434 [Thread-28941] INFO  com.redhat.rhn.taskomatic.task.RepoSyncTask - 20:21:36 ======================================
20:21:36 | Channel: ubuntu-1804-amd64-universe-uyuni
20:21:36 ======================================
20:21:36 Sync of channel started.
20:21:40 Repo URL: http://archive.ubuntu.com/ubuntu/dists/bionic/universe/binary-amd64/
20:21:40     Packages in repo:             53596
20:25:38     No new packages to sync.
20:25:38
20:25:38   Patches in repo: 0.
20:25:38   Updating overview of 5 systems
20:25:38 Sync completed.
20:25:38 Total time: 0:04:02

2022-12-01 20:26:00,272 [DefaultQuartzScheduler_Worker-11] INFO  com.redhat.rhn.taskomatic.task.SystemOverviewUpdateQueue - In the queue: 5
2022-12-01 20:30:00,385 [DefaultQuartzScheduler_Worker-3] INFO  com.redhat.rhn.taskomatic.task.SessionCleanup - 420 stale session(s) deleted
2022-12-01 21:01:00,262 [DefaultQuartzScheduler_Worker-7] INFO  com.redhat.rhn.taskomatic.task.SystemOverviewUpdateQueue - In the queue: 19
2022-12-01 21:09:50,540 [DefaultQuartzScheduler_Worker-1] INFO  com.redhat.rhn.taskomatic.task.RepoSyncTask - Syncing repos for channel: Ubuntu 18.04 LTS AMD64 Base for Uyuni
2022-12-01 21:09:51,405 [Thread-41] WARN  com.redhat.rhn.taskomatic.core.SchedulerKernel - Reinitializing ssh-push-default, found 1 runs in the future.
2022-12-01 21:09:51,481 [Thread-41] WARN  com.redhat.rhn.taskomatic.core.SchedulerKernel - Reinitializing errata-cache-default, found 2 runs in the future.
2022-12-01 21:09:52,006 [Thread-41] WARN  com.redhat.rhn.taskomatic.core.SchedulerKernel - Number of interrupted runs: 3
2022-12-01 21:09:53,845 [Thread-55] INFO  com.redhat.rhn.taskomatic.task.RepoSyncTask - 21:09:52 ======================================
21:09:52 | Channel: ubuntu-18.04-pool-amd64-uyuni
21:09:52 ======================================
21:09:52 Sync of channel started.
21:09:53 Repo URL: http://localhost/pub/repositories/empty-deb/?uniquekey=1804-uyuni
21:09:53     Packages in repo:                 0
21:09:53     No new packages to sync.
21:09:53
21:09:53   Patches in repo: 0.
21:09:53   Updating overview of 5 systems
21:09:53 Sync completed.

curl http://amazonlinux.default.amazonaws.com/2/core/latest/x86_64/mirror.list

 curl http://amazonlinux.default.amazonaws.com/2/core/latest/x86_64/mirror.list
 https://cdn.amazonlinux.com/2/core/2.0/x86_64/0cac95da63306270fbedb235008cfaf4f04477723dc1966e2abadb274a0edd44

this is really weird because I do not see netowrking errors in my uyuni server

I found an issue with one service called apparmor-parser but this should no be related .

systemctl status apparmor.service
× apparmor.service - Load AppArmor profiles
     Loaded: loaded (/usr/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2022-11-29 20:31:55 UTC; 2 days ago
   Main PID: 537 (code=exited, status=1/FAILURE)

Nov 29 20:31:55 uyuni apparmor.systemd[537]: Restarting AppArmor
Nov 29 20:31:55 uyuni apparmor.systemd[537]: Reloading AppArmor profiles
Nov 29 20:31:55 uyuni apparmor.systemd[542]: Warning from stdin (line 1): Cache: failed to add read only location '/usr/share/apparmor/cache', does not contain valid cache directory for the specified feature set
Nov 29 20:31:55 uyuni apparmor.systemd[551]: Found reference to variable HOME, but is never declared
Nov 29 20:31:55 uyuni apparmor.systemd[567]: Warning from stdin (line 1): Cache: failed to add read only location '/usr/share/apparmor/cache', does not contain valid cache directory for the specified feature set
Nov 29 20:31:55 uyuni apparmor.systemd[568]: Found reference to variable HOME, but is never declared
Nov 29 20:31:55 uyuni apparmor.systemd[537]: Error: At least one profile failed to load
Nov 29 20:31:55 uyuni systemd[1]: apparmor.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 20:31:55 uyuni systemd[1]: apparmor.service: Failed with result 'exit-code'.
Nov 29 20:31:55 uyuni systemd[1]: Failed to start Load AppArmor profiles.

this is really funny because apparmor.service was not installed . the package installed was apparmor-parser
I removed it and the systemctl has no degraded errors.
I do not use docker in my uyuni server and docker daemon was not running at all.

zypper rm apparmor-parser
Reading installed packages...
Resolving package dependencies...

The following 5 packages are going to be REMOVED:
  apparmor-parser apparmor-parser-lang docker docker-bash-completion docker-zsh-completion

5 packages to remove.
After the operation, 133.6 MiB will be freed.
Continue? [y/n/v/...? shows all options] (y): y
(1/5) Removing apparmor-parser-lang-3.0.4-150400.5.3.1.noarch ......................................................................................................................................................................................................................................................................................................................................................................[done]
(2/5) Removing docker-bash-completion-20.10.17_ce-150000.169.1.noarch ..............................................................................................................................................................................................................................................................................................................................................................[done]
(3/5) Removing docker-zsh-completion-20.10.17_ce-150000.169.1.noarch ...............................................................................................................................................................................................................................................................................................................................................................[done]
(4/5) Removing docker-20.10.17_ce-150000.169.1.x86_64 ..............................................................................................................................................................................................................................................................................................................................................................................[done]
Removed /etc/systemd/system/multi-user.target.wants/apparmor.service.
(5/5) Removing apparmor-parser-3.0.4-150400.5.3.1.x86_64 ...........................................................................................................................................................................................................................................................................................................................................................................[done]
There are running programs which still use files and libraries deleted or updated by recent upgrades. They should be restarted to benefit from the latest updates. Run 'zypper ps -s' to list these programs.

after the reboot I got other repo sync errors like this screenshot
uyuni2022-11_error-sync01

zzaimeche added a commit to zzaimeche/uyuni that referenced this issue Dec 2, 2022
@zzaimeche
Add 'octet-stream' to accepted mirrorlist content-types
3792333 
This commit adds 'octet-stream' to the list of acceptable content-types for
urls pointing to mirrorlists. This fixes:
uyuni-project#6244
and ensures that amazonlinux channels will continue to work.
@zzaimeche zzaimeche mentioned this issue Dec 2, 2022
Merged
12 tasks
zzaimeche added a commit to zzaimeche/uyuni that referenced this issue Dec 2, 2022
@zzaimeche
Add 'octet-stream' to accepted mirrorlist content-types
b1ce636 
This commit adds 'octet-stream' to the list of acceptable content-types for
urls pointing to mirrorlists. This fixes:
uyuni-project#6244
and ensures that amazonlinux channels will continue to work.
@juliogonzalez
Copy link
Member

Reopening. Bug is still present, so we need to decide what to do: Either we tell users to fix this manually (not an approach I like), or we need to prepare a patch we can release, and in that case the PR for master needs to be revoked, and we need a new one against the last Uyuni version.

@juliogonzalez juliogonzalez reopened this Dec 2, 2022
zzaimeche added a commit to zzaimeche/uyuni that referenced this issue Dec 2, 2022
@zzaimeche
Add 'octet-stream' to accepted mirrorlist content-types
002b71c 
This commit adds 'octet-stream' to the list of acceptable content-types for
urls pointing to mirrorlists. This fixes:
uyuni-project#6244
and ensures that amazonlinux channels will continue to work.
@zzaimeche zzaimeche mentioned this issue Dec 2, 2022
Merged
12 tasks
@juliogonzalez
Copy link
Member

@eins I just released the patch prepared by @zzaimeche, but we still need to announce it.

It should be on the mirrors soon, in case you want to apply it before we announce. The patch is for the server, and can be applied following the usual patching procedure at https://www.uyuni-project.org/pages/patches.html

@juliogonzalez
Copy link
Member

And the official announcement: https://lists.opensuse.org/archives/list/announce@lists.uyuni-project.org/thread/K5FY2MRWKOBGJUJN776KRXZ6RSYTXARC/

@eins
Copy link
Author

eins commented Dec 2, 2022

@juliogonzalez @zzaimeche thanks a lot for the quick reply. I'm applying the patch right now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working P5
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@juliogonzalez @eins