Repositories that fail to connect block all rpm-ostree operations

[csoriano1618]

Host system details

[csoriano@localhost ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181002.n.0 (2018-10-03 01:59:25)
                BaseCommit: ece2fdd70ddc40c18e2ccc7d7f4de4c27f47b198a549e60a3b246f931539abd1
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
           LayeredPackages: baobab cairo-devel chromium chromium-libs chromium-libs-media chromium-libs-media-freeworld file-roller fuse-devel gdb git-extras
                            gnome-shell-extension-topicons-plus gnome-tweaks go gtk3-devel krb5-workstation meson mono-devel nant npm nunit python3-devel unar vim
                            xorg-x11-server-Xvfb
             LocalPackages: rpmfusion-nonfree-release-29-0.5.noarch redhat-internal-openvpn-profiles-0.1-30.el7.csb.noarch flash-plugin-29.0.0.171-release.x86_64
                            nautilus-dropbox-2015.10.28-1.fc10.x86_64 rpmfusion-free-release-29-0.5.noarch redhat-internal-cert-install-0.1-7.el7.csb.noarch

  ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181002.n.0 (2018-10-03 01:59:25)
                BaseCommit: ece2fdd70ddc40c18e2ccc7d7f4de4c27f47b198a549e60a3b246f931539abd1
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
           LayeredPackages: baobab cairo-devel chromium chromium-libs chromium-libs-media chromium-libs-media-freeworld file-roller fuse-devel gdb git-extras gnome-tweaks go
                            gtk3-devel krb5-workstation meson mono-devel nant npm nunit python3-devel unar vim xorg-x11-server-Xvfb
             LocalPackages: rpmfusion-nonfree-release-29-0.5.noarch redhat-internal-openvpn-profiles-0.1-30.el7.csb.noarch flash-plugin-29.0.0.171-release.x86_64
                            nautilus-dropbox-2015.10.28-1.fc10.x86_64 rpmfusion-free-release-29-0.5.noarch redhat-internal-cert-install-0.1-7.el7.csb.noarch
[csoriano@localhost ~]$ 

Expected vs actual behavior

I have some repositories installed by Dropbox and RHEL rhpkg that sometimes they fail to connect to their servers. These repositories block any other operation by rpm-ostree.

Ideally these repositories would be skipped if they fail to connect to their remote, as dnf does.

Steps to reproduce it

...

Would you like to work on the issue?

Not really, but thanks if you do :)

[dustymabe]

can you post the /etc/yum.repos.d/ files of the repos in question? Do the repos have skip_if_unavailable=True in them?

[csoriano1618]

Yes:

[csoriano@localhost ~]$ cat /etc/yum.repos.d/rcm-tools-fedora.repo
[rcm-tools-fedora-rpms]
name=RCM Tools for Fedora $releasever (RPMs)
baseurl=http://download.devel.redhat.com/rel-eng/RCMTOOLS/latest-RCMTOOLS-2-F-28/compose/Everything/$basearch/os/
enabled=0
gpgcheck=0
skip_if_unavailable=true

[cgwalters]

This gets to a truly fundamental difference between rpm-ostree and yum/dnf (as well as apt and most other systems): rpm-ostree is at heart an image system, and intentionally carries forward only explicitly specified state.

That explicit state is things like the LayeredPackages. We always compute the new state as a function of the new base plus requested changes.

In contrast, yum/apt etc default to treating all of your currently installed packages as state to preserve. (Both have a concept of "user installed" packages bolted on top though)

For yum, implementing skip_if_unavailable is pretty easy, it can just default to preserving your existing packages.

For us...when a skip_if_unavailable repo failed, we'd have to have the high level code look and say "OK, the previous state has cached packages from these repos" and preserve them. Today we very intentionally again have very little of this "look at previous state" code.

[fbruetting]

I have a related issue, which might be relevant as well:

I had the same problem and tried to deactivate that unavailable repo by setting enabled=0 in the repo file. But then I still couldn’t upgrade everything else:

sudo rpm-ostree upgrade
1 metadata, 0 content objects fetched; 569 B transferred in 4 seconds                                                                       Checking out tree f21ec65... done
Enabled rpm-md repositories: updates-testing updates fedora rpmfusion-free-updates-testing rpmfusion-free rpmfusion-nonfree-updates-testing Updating metadata for 'heliocastro-hack-fonts': [=============] 100%
rpm-md repo 'heliocastro-hack-fonts'; generated: 2018-05-19 17:21:46
Updating metadata for 'nalimilan-julia': [=============] 100%
rpm-md repo 'nalimilan-julia'; generated: 2018-10-18 11:29:15
Importing metadata [=============] 100%
error: Packages not found: gnome-encfs-manager

[dustymabe]

I'm building F33 FCOS now (finally). When I got booted up into the system I tried to layer a package and I get an error:

[core@cosa-devsh yum.repos.d]$ sudo rpm-ostree install htop 
Checking out tree ace8f23... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates-testing updates fedora
Updating metadata for 'fedora-cisco-openh264'... done
error: Updating rpm-md repo 'fedora-cisco-openh264': cannot update repo 'fedora-cisco-openh264': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-33&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]; Last error: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-33&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org]

The fedora-cisco-openh264 repo has skip_if_unavailable=True and in this case it truly is not required so it struck me as a bit odd that we were failing to move forward.

For yum, implementing skip_if_unavailable is pretty easy, it can just default to preserving your existing packages.

For us...when a skip_if_unavailable repo failed, we'd have to have the high level code look and say "OK, the previous state has cached packages from these repos" and preserve them. Today we very intentionally again have very little of this "look at previous state" code.

Alternatively the transaction fails if it can't find a package it needs because a repo isn't available. Of course my suggestion here gets a little more complicated when you have a repo setup like we have in Fedora with a fedora, and fedora-updates repo where you could end up getting older versions of packages if fedora-updates is not available. However, in practice this won't happen because fedora-updates is set to skip_if_unavailable=False.

[dustymabe]

The error itself I'm seeing is because DNS apparently isn't working inside my cosa run VM, but I think the argument still stands.

[cgwalters]

Short term you can just set enabled=0 in /etc/yum.repos.d/fedora-cisco.openh264.conf right?

I think fundamentally skip_if_unavailable creates more problems than it solves - specifically it causes unpredictability and really masks errors.

The problem particularly for rpm-ostree is that we have rpm-ostree status - correctly and usefully displaying state when skip_if_unavailable is in play is certainly possible but far from trivial.

Your case is really the easy one, where you don't have any packages from that repo installed.

[cheese]

Our machine don't have Internet access and don't have any layered package. rpm-ostree upgrade --check and some other commands are not usable, even we have an OSTree archive repo in our intranet.

[jlebon]

@cheese Please provide the output of rpm-ostree status -v and rpm-ostree upgrade. If you don't have any layering or overrides, yum repos should not be involved at all.

[cheese]

I found out I have some layered packages in the booted deploy. I ran rpm-ostree reset to remove all the layered packages in the pending deploy. And then running 'rpm-ostree upgrade --check` will fail since some yum repos were not accessible.