Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

split base/layered versionlocked package problem (krb5-workstation) #415

Open
cgwalters opened this issue Aug 1, 2016 · 50 comments

Comments

@cgwalters
Copy link
Member

commented Aug 1, 2016

In wstree right now krb5-libs is a dependency of various things in the base OS. krb5-workstation is an app that uses it - however, it's version locked.

This introduces a problem with package layering, since if the compose server picks up a newer version from package mirror A, but the client only has an older version from mirror B, upgrades break. (Or if it's not installed, it can't be).

There are obviously a few solutions to this...likely krb5-workstation should be more flexible, and only introduce a hard requirement on the -libs package version if it needs it. Second, we may need to consider a way to retrieve rpm-md data from the same place as the ostree commit, or vice versa. Basically having unified metadata/data management.

--

EDIT (2019-09-08): rojig is a much bigger hammer attempting to solve this problem.

@jlebon

This comment has been minimized.

Copy link
Member

commented Aug 8, 2016

Second, we may need to consider a way to retrieve rpm-md data from the same place as the ostree commit, or vice versa. Basically having unified metadata/data management.

Isn't this completely up to the content provider, though? Doesn't seem like something we can do much about in rpm-ostree.

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Oct 7, 2016

I think if we provided the infrastructure to link to the rpm-md repos from the ostree commit, infra providers could use that. We basically have all the data already from the treecompose side. The infra provider would need to retain multiple versions of the rpm-md repo data though.

That said I think the general solution to this is to relax the strict version requirements.

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Mar 7, 2017

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Apr 19, 2017

Hit this again today trying to layer virt-manager; through a depchain virt-manager-commonpython-libxml2 (not in the tree) → libxml2:

error: Could not depsolve transaction; 1 problem detected:
0. package python-libxml2-2.9.3-4.fc25.x86_64 requires libxml2 = 2.9.3-4.fc25, but none of the providers can be installed

And at the time:

$ rpm -q libxml2
libxml2-2.9.4-2.fc25.x86_64
@dustymabe

This comment has been minimized.

Copy link
Member

commented Apr 20, 2017

doesn't this problem go away if we can replace content in the base layer? #485

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Apr 20, 2017

I think we should definitely allow it, but doing that invalidates the concept that the tree is tested and run as a unit. I'm not sure I'd want it to just happen by default.

Particularly in this case, it turned out the problem is I was being served a very out of date (rpm-md) mirror.

@dustymabe

This comment has been minimized.

Copy link
Member

commented Apr 20, 2017

I think we should definitely allow it, but doing that invalidates the concept that the tree is tested and run as a unit. I'm not sure I'd want it to just happen by default.

right, but don't we invalidate that when we allow package layering at all? I know technically we are just "adding to" rather than "changing" but adding things breaks things sometimes too.

Maybe for base package overrides we default to spitting out a message about the fact that this transaction will override a base package and thus the integrity of the tree may suffer and then allow the user to (y/n). Also provide a cli flag for forcing this.

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented May 10, 2017

Another good example of this is emacs-filesystem (pulled into base) and emacs.

@kallisti5

This comment has been minimized.

Copy link

commented Oct 9, 2017

vim-atom

Seeing this here as well with vim... this is a painful bug.

@miabbott

This comment has been minimized.

Copy link
Collaborator

commented Oct 10, 2017

Like @kallisti5 above, I ran into the problem with vim-common. (Going to paste my session here for searchability later)

$ sudo rpm-ostree status                                                                                                                                                                     
[sudo] password for miabbott:
State: idle                                         
Deployments:                                                                                             
● atomicws:fedora/x86_64/workstation                                                                     
                   Version: 26.201 (2017-10-06 17:57:57)                                                                                                                                                           
                BaseCommit: c458ed1025d135b88f611eb169c515fd478d8751bcd742cb05284a929ab07a56
           LayeredPackages: chrome-gnome-shell ffmpeg-libs krb5-workstation libvirt-client sshpass tmux vagrant-libvirt vim virt-install virt-manager
                                                                                                                                                                                                                   
  atomicws:fedora/x86_64/workstation                                                                     
                   Version: 26.187 (2017-10-02 09:58:23)
                BaseCommit: 5637588b7331a5b49f4c417e4f5eca706f067d9b47911d5849fe9981ec30c3c9
           LayeredPackages: chrome-gnome-shell ffmpeg-libs krb5-workstation libvirt-client tmux vagrant-libvirt vim virt-manager                                                                                   
[/var/home/miabbott *]$ sudo rpm-ostree upgrade     

Updating metadata for 'updates': [==========================] 100%
rpm-md repo 'updates'; generated: 2017-10-09 14:42:40                                                                                                                                                              
                                                                                                         

Updating metadata for 'fedora': [==========================] 100%
rpm-md repo 'fedora'; generated: 2017-07-05 20:31:38                                                                                                                                                               
                                                                                                         

Updating metadata for 'rpmfusion-nonfree': [==================================] 100%
rpm-md repo 'rpmfusion-nonfree'; generated: 2017-07-10 12:08:07                                                                                                                                                    
                                                                                                         

Updating metadata for 'region51-chrome-gnome-shell': [============================================] 100%
rpm-md repo 'region51-chrome-gnome-shell'; generated: 2017-10-10 07:03:04                                                                                                                                          
                                                                                                         

Updating metadata for 'walters-syslinux-stub-i686': [========================================] 100%
rpm-md repo 'walters-syslinux-stub-i686'; generated: 2017-10-10 09:05:48                                                                                                                                           
                                                                                                         

Updating metadata for 'walters-syslinux-stub': [==================================== 100%
rpm-md repo 'walters-syslinux-stub'; generated: 2017-10-10 09:05:46                                                                                                                                                
                                                                                                         

Updating metadata for 'rpmfusion-nonfree-updates': [=========================================] 100%
rpm-md repo 'rpmfusion-nonfree-updates'; generated: 2017-10-06 17:41:43                                                                                                                                            
                                                                                                         

Updating metadata for 'rpmfusion-free': [===============================] 100%
rpm-md repo 'rpmfusion-free'; generated: 2017-07-10 12:06:27                                                                                                                                                       
                                                    
                                                    
Updating metadata for 'rpmfusion-free-updates': [======================================] 100%

                                                                                                                                                                                                                   
Importing metadata [======================] 100%
Resolving dependencies... done                      
Will download: 2 packages (196.2 kB)

  Downloading from updates: [========================] 100%

Importing: [=======================] 100%
Applying 134 overlays... error: Checking out vim-common-2:8.0.1171-1.fc26.x86_64: Hardlinking c0/679f8a0eefe6ddb91b60afe7ff885aa8ee2b39fea8a0dbf342d7758733a7d8.file to ex.1.gz: File exists
@jlebon

This comment has been minimized.

Copy link
Member

commented Oct 10, 2017

Right, this is the same issue though the output is different because v2017.9 has #974. A hacky workaround is to manually download the matching vim packages (https://koji.fedoraproject.org/koji/buildinfo?buildID=969318) and locally install that. Hacky because on the next upgrade that bumps vim-minimal again, you'll have to uninstall them and possibly get the latest RPMs if they're still not in sync (though if the tree is freshly composed, that shouldn't be an issue).

@jlebon

This comment has been minimized.

Copy link
Member

commented Oct 10, 2017

@miabbott, I think you're hitting something different. I opened a separate issue for your error here: #1047.

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Jan 10, 2018

This problem will be comprehensively fixed by jigdo ♲📦 #1081 because the rpm-md and ostree commit will move at the same cadence by default.

@dustymabe

This comment has been minimized.

Copy link
Member

commented Feb 1, 2018

This problem will be comprehensively fixed by jigdo ♲📦 #1081 because the rpm-md and ostree commit will move at the same cadence by default.

Can you explain this a little more? if yesterday's bodhi run had a new fedora-atomic-host rpm in it (i.e. a new release), and today's run includes new packages (and new base packages that are deps) then won't an rpm-ostree install still fail in the same way if a base package is part of the update?

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Feb 1, 2018

The jigdoRPM should be generated from the same rpm-md reposet as contains it. That's how it works today in FAHC.

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Feb 1, 2018

Yes, this means you need to do: createrepo_c, compose tree, copy in jigdoRPM, createrepo_c.

@jlebon

This comment has been minimized.

Copy link
Member

commented Feb 1, 2018

But that still means you need to update to the new jigdoRPM first, right? (Which also assumes that there's a new jigdoRPM to update to). Just like OSTree mode right now?

Jigdo does improve the situation in the sense that, at the moment that the jigdoRPM is released, it is in sync with the rest of the RPMs, which is not a guarantee right now. But otherwise, this split issue is still something people can hit, right?

@dustymabe

This comment has been minimized.

Copy link
Member

commented Feb 1, 2018

Yes, this means you need to do: createrepo_c, compose tree, copy in jigdoRPM, createrepo_c.

yeah I figured that much. Specifically for this issue, though, tomorrows bodhi run fires and no new fedora-atomic-host rpm is in that, now there are newer rpms in the rpm-md than what was used to build the fedora-atomic-host rpm and we have this same problem all over again?

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Feb 1, 2018

But that still means you need to update to the new jigdoRPM first, right?

Yes, but we do that by default on upgrade right?

Specifically for this issue, though, tomorrows bodhi run fires and no new fedora-atomic-host rpm is in that, now there are newer rpms in the rpm-md than what was used to build the fedora-atomic-host rpm and we have this same problem all over again?

I don't quite understand that. Again the design is that the jigdoRPM is contained within and versioned by the rpm-md repo. The jigdoRPM should be regenerated whenever the RPM content changes. However what you may be getting at here is that today the compose tree process doesn't yet support "pure jigdo" mode; the canonical history is still stored in an ostree repo.

@jlebon

This comment has been minimized.

Copy link
Member

commented Feb 1, 2018

The jigdoRPM should be regenerated whenever the RPM content changes.

Right, assuming that all Bodhi updates are batched and each batch has a jigdoRPM, then there's no issue. But e.g. right now for FAH, we release every two weeks. Whereas Bodhi batches are every week, right? So there's potentially a one week window during which we can get mismatches. Isn't that the same issue we're hitting right now with the stable vs updates ref?

@dustymabe

This comment has been minimized.

Copy link
Member

commented Feb 1, 2018

I don't quite understand that.

Let me explain. Today we have the stable ref fedora/27/x86_64/atomic-host and the updates ref that gets updated every day fedora/27/x86_64/updates/atomic-host. In jigdo future we have the stable jigdoRPM fedora-atomic-host and an updates jigdoRPM fedora-atomic-host-updates. If we release the stable fedora-atomic-host jigdoRPM today then the rpm-md it was generated from will be out of date tomorrow, just like the stable ostree ref that we use today fedora/27/x86_64/atomic-host.

@dustymabe

This comment has been minimized.

Copy link
Member

commented Feb 1, 2018

Isn't that the same issue we're hitting right now with the stable vs updates ref?

yes, i think so

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Feb 1, 2018

But e.g. right now for FAH, we release every two weeks. Whereas Bodhi batches are every week, right?

Yes; making full use of jigdo requires that we synchronize those in actuality.

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Feb 1, 2018

I see two paths:

  1. We don't differ from Bodhi at all - this may mean we have an ostree update 3 days in a row if one of the RPMs changes each of those days
  2. We create /etc/yum.repos.d/fedora-atomic.repo
@jlebon

This comment has been minimized.

Copy link
Member

commented Feb 1, 2018

Alternatively, we could also add a kind of --allow-replacements switch so we automatically add replace overrides.

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Feb 1, 2018

Alternatively, we could also add a kind of --allow-replacements switch so we automatically add replace overrides.

Yeah true, but I really like the protection we have today against the depsolver deciding to swap out something from the base by default. By default we shouldn't offer depsolve errors and a magic switch "make it work" right?

@dustymabe

This comment has been minimized.

Copy link
Member

commented Feb 1, 2018

Ok, just so i'm clear on everything.

  • this is a problem with jigdo AND ostree transports (i.e. jigdo doesn't solve this problem)
  • the most desirable solution is actually in the releng side to make sure rpm repos sync with when we do releases
  • an alternative solution is to allow forcing overrides in the base via a command line switch

Some more responses:

We don't differ from Bodhi at all - this may mean we have an ostree update 3 days in a row if one of the RPMs changes each of those days

This takes us back to where we were before we 'slowed down' the cadence. Not saying that is a bad thing, but it is different.

We create /etc/yum.repos.d/fedora-atomic.repo

The problem with this is we don't get all of the rpms from the vast ecosystem that is Fedora. Is there a solution to this that would allow us to keep pulling rpms from the Everything package set but still have a subset of rpms in a separate repo?

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented May 3, 2018

For Fedora, one can work around this today by explicitly using versioned repositories client side, like this:

[fedora-compose]
baseurl=https://kojipkgs.fedoraproject.org/compose/updates/Fedora-28-updates-20180503.0/compose/Everything/$basearch/os
gpgcheck=1
enabled=1

Disable the updates repo.

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Sep 13, 2018

Yeah, I don't think I want to go back to having upgrades everyday. No one wants to upgrade everyday.

I used to agree, but...once one goes to staged automatic updates, it's more about when you want to reboot, and that's obviously user controlled.

this is a problem with jigdo AND ostree transports (i.e. jigdo doesn't solve this problem)

And let me emphasize again - topics like this https://discussion.fedoraproject.org/t/forbidden-base-package-replacements/387/1 wouldn't come up because the things would be in sync always.

Today we cache rpm-md and don't update it within the expiry period, but that's not true of ostree repos, etc.

And yes, big picture to really gain parity we need to version the rpm-md repos too. As I've mentioned elsewhere, in practice Fedora does this today via pungi but libdnf has no idea what a compose is - there's no dnf distro-sync --version 20180831.0/ etc. But fixing that would benefit everything, including container builds.

@dustymabe

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

And let me emphasize again - topics like this https://discussion.fedoraproject.org/t/forbidden-base-package-replacements/387/1 wouldn't come up because the things would be in sync always.

i guess I fundamentally don't understand what mechanism is making them be in sync. Is it the cached rpm-md you mention below?

Today we cache rpm-md and don't update it within the expiry period, but that's not true of ostree repos, etc.

What is preventing us from caching rpm-md at the time that we do an rpm-ostree upgrade from an ostree repo and achieving the same goal?

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Sep 13, 2018

i guess I fundamentally don't understand what mechanism is making them be in sync. Is it the cached rpm-md you mention below?

With rojig, there's only one place content comes from - so there's nothing to go out of sync.

@dustymabe

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

With rojig, there's only one place content comes from - so there's nothing to go out of sync.

This is the part that i'd like to explore:

Let's say I installed my system with the latest version of rojig rpm/ostree two days ago 29.20180911.0 and today I decide to rpm-ostree install htop. In yesterday's compose (29.20180912.0) new glibc got updated (and let's say htop from today's compose depends on that new version of glibc). Where does htop in the rpm-ostree install htop operation come from? The options are 29.20180911.0 repo, 29.20180912.0 repo, 29.20180913.0 repo, or latest repo (what's currently on the mirrors today).

@jlebon

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

I used to agree, but...once one goes to staged automatic updates, it's more about when you want to reboot, and that's obviously user controlled.

Yeah, it's true that AutomaticUpdatePolicy=stage changes the picture a bit.

I think we discussed this at some point, but... another way of solving this would be if Fedora didn't prune previous versions from the repos, right? Which could be easier to implement than "officially" publishing versioned repos. (Versioned metadata could still be a layer on top of that.)

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Sep 13, 2018

and let's say htop from today's compose depends on that new version of glibc

Ah. I see what you're getting at. It is likely indeed that with rojig we probably want install to really be rpm-ostree upgrade --install.

And to follow up...with automatic updates enabled, you probably already have the updated OS data staged.

@dustymabe

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

Ah. I see what you're getting at. It is likely indeed that with rojig we probably want install to really be rpm-ostree upgrade --install.

good to know 👍

Assumption: Fedora updates yum repos and OSTree repo gets updated at approximately the same time, which is the case today.

In the case where you say we want install to be rpm-ostree upgrade --install I don't think rojig is going to help us with this exact problem over what we have today since (I think, please tell me if I'm wrong) that rpm-ostree upgrade --install would solve our problems today with the OSTree repo (at least throughout a normal fedora cycle (exceptions include when we have stable/beta|final/testing composes going on at the same time)).

Could we just solve most of our problem today by detecting a conflict and telling the user to try rpm-ostree upgrade --install instead ? or we could give them a Installing this RPM would require updateing the base OSTree. Continue? (y/N)

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Oct 1, 2018

Assumption: Fedora updates yum repos and OSTree repo gets updated at approximately the same time, which is the case today.

Yeah, but this whole issue is about the current gaps in "approximately". Not to mention that when e.g. enabling testing repositories, users currently need to understand how to simutaneously enable the updates-testing repostiory and rebase to a testing ref.

I could imagine teaching rpm-ostree and the server side enough intelligence to "tie together" these things - something like each ref having metadata about which rpm-md repos generated it. But IMO the solution is rojig.

or we could give them a Installing this RPM would require updateing the base OSTree. Continue? (y/N)

In general, I am not a fan of these sorts of dynamic interactive questions - I think they're very often indicative of bad design.

See also https://ometer.com/preferences.html

That doesn't mean we shouldn't have any questions like this, but in this specific case, I think we should take the "hard path" (rojig) and save our users confusing questions.

(Another important thing here is that any questions like this would have to be reimplemented in gnome-software and Cockpit)

@dustymabe

This comment has been minimized.

Copy link
Member

commented Oct 2, 2018

Not to mention that when e.g. enabling testing repositories, users currently need to understand how to simutaneously enable the updates-testingrepostiory and rebase to a testing ref.

I could imagine teaching rpm-ostree and the server side enough intelligence to "tie together" these things - something like each ref having metadata about which rpm-md repos generated it.

yeah that would be nice

or we could give them a Installing this RPM would require updateing the base OSTree. Continue? (y/N)

In general, I am not a fan of these sorts of dynamic interactive questions - I think they're very often indicative of bad design.

See also https://ometer.com/preferences.html

That doesn't mean we shouldn't have any questions like this, but in this specific case, I think we should take the "hard path" (rojig) and save our users confusing questions.

(Another important thing here is that any questions like this would have to be reimplemented in gnome-software and Cockpit)

Then we just make it not an option and upgrade the user by default just like in rojig. Solves the same problem. Then all that's left is the updates vs updates-testing problem mentioned above.

@dustymabe

This comment has been minimized.

Copy link
Member

commented Oct 2, 2018

Then we just make it not an option and upgrade the user by default just like in rojig.

although I think this would render rpm-ostree install && rpm-ostree ex livefs useless because we would always be updating the base and requiring a reboot, right?

@cgwalters cgwalters referenced this issue Dec 11, 2018
0 of 3 tasks complete
@edwintorok

This comment has been minimized.

Copy link

commented Dec 27, 2018

Edit: this actually seems to be a problem with the repository itself, libxcrypt-devel-4.4.2-3.fc29 is claimed to be not available at all, although it is available on koji: #1724

It would be useful if rpm-ostree printed more details on which package is causing the problem, I just ran into this problem today with a very generic Some base packages would be replaced:

sudo rpm-ostree override reset fuse-overlayfs
sudo rpm-ostree upgrade
note: automatic updates (stage) are enabled
1 metadata, 0 content objects fetched; 569 B transferred in 7 seconds
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
Updating metadata for 'fedora'... done
rpm-md repo 'fedora'; generated: 2018-10-24T22:20:15Z
Updating metadata for 'updates'... done
rpm-md repo 'updates'; generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced

I tried using --uninstall, but it still complained:

sudo rpm-ostree upgrade --uninstall ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools latencytop libreoffice mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam pass perf perl-Pod-Html python3-psutil ripgrep stow strace tig weston zsh
[sudo] password for edwin: 
note: automatic updates (stage) are enabled
1 metadata, 0 content objects fetched; 569 B transferred in 5 seconds
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced

Finally I pinned the current deployment, uninstalled all layers, upgraded, used rpm-ostree install -n to narrow down the package (in this case ocaml):

sudo rpm-ostree uninstall --all
[...]
sudo rpm-ostree upgrade
note: automatic updates (stage) are enabled
1 metadata, 0 content objects fetched; 569 B transferred in 3 seconds
Staging deployment... done
Upgraded:
  container-selinux 2:2.76-1.git87fae85.fc29 -> 2:2.77-1.git2c57a17.fc29
  libxcrypt 4.4.2-1.fc29 -> 4.4.2-3.fc29
  runc 2:1.0.0-59.dev.gitccb5efd.fc29 -> 2:1.0.0-66.dev.gitbbb17ef.fc29
  zchunk-libs 0.9.17-1.fc29 -> 1.0.0-1.fc29
[...]
for i in bcc-tools dconf-editor evince evolution fedora-toolbox firefox-wayland gnome-photos gnome-tweak-tool kernel-tools latencytop mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam perf ripgrep stow strace tig weston zsh; do echo $i; sudo rpm-ostree install -n $i; done
[...]
sudo rpm-ostree install ocaml
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced

However it still doesn't show which package in the base packages would be the problem...

Here is my current status:

sudo rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
  ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181227.0 (2018-12-27T01:09:18Z)
                BaseCommit: a6d00a98f2bfb16e3cdae4f89f306a5035c7e67ae15012ccf3d757c11c4b8624
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
           LayeredPackages: ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools
                            latencytop libreoffice mozilla-ublock-origin ncurses-term neovim pass perf perl-Pod-Html python3-psutil ripgrep stow strace tig weston zsh

● ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181225.0 (2018-12-25T01:13:26Z)
                BaseCommit: 9c2273bb044c6b0a36c0e1549011091afd79f95700f27a568f5964ac12c7eb32
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
      ReplacedBasePackages: fuse-overlayfs 0.1-6.dev.git3d48bf9.fc29 -> 0.1-8.dev.git91bb401.fc29
           LayeredPackages: ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools
                            latencytop libreoffice mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam pass perf perl-Pod-Html python3-psutil ripgrep stow strace
                            tig weston zsh
                    Pinned: yes

  ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181224.0 (2018-12-24T01:17:14Z)
                BaseCommit: a071235dd160f942385a1c2472521ef1906e066b26083e69fce734e1ac71a803
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
      ReplacedBasePackages: fuse-overlayfs 0.1-6.dev.git3d48bf9.fc29 -> 0.1-8.dev.git91bb401.fc29
           LayeredPackages: ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools
                            latencytop libreoffice mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam pass perf perl-Pod-Html python3-psutil ripgrep stow strace
                            tig weston zsh

AvailableUpdate:
        Version: 29.20181227.0 (2018-12-27T01:09:18Z)
         Commit: a6d00a98f2bfb16e3cdae4f89f306a5035c7e67ae15012ccf3d757c11c4b8624
   GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
           Diff: 4 upgraded, 232 removed

Edit: in this case the problem was libxcrypt-devel (needed by glibc-devel, needed by gcc, needed by ocaml-runtime, needed by ocaml):

sudo rpm-ostree install -n libxcrypt
error: "libxcrypt" is already provided by: libxcrypt-4.4.2-3.fc29.x86_64. Use --allow-inactive to explicitly require it.
edwin@bolt:~ % sudo rpm-ostree install -n libxcrypt-devel
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced

Note that rpm-ostree refresh-md and rpm-ostree cleanup -m didn't fix the problem, libxcrypt-devel is still uninstallable.

@jistr

This comment has been minimized.

Copy link

commented Mar 4, 2019

I'm seeing this when trying to install gcc on Atomic host 29.20190219.0:

Forbidden base package replacements:
  libgcc 8.2.1-6.fc29 -> 8.3.1-2.fc29 (updates)
  libxcrypt 4.4.3-2.fc29 -> 4.4.3-10.fc29 (updates)
  libgomp 8.2.1-6.fc29 -> 8.3.1-2.fc29 (updates)

As soon as the update repos get out of sync from the latest base layer, installing some RPMs becomes impossible. (rpm-ostree cleanup -m doesn't help.) IMO this issue shouldn't be tagged just "enhancement", it's a significant usability bug.

Until a real fix is implemented, would it help to build Atomic host base layers with a nightly cadence?

@cgwalters

This comment has been minimized.

Copy link
Member Author

commented Mar 4, 2019

Until a real fix is implemented, would it help to build Atomic host base layers with a nightly cadence?

That exists; just rpm-ostree rebase fedora-atomic:fedora/29/x86_64/updates/atomic-host

(But I'd recommend doing development in a container, not the host)

@jistr

This comment has been minimized.

Copy link

commented Mar 5, 2019

@cgwalters Thanks! Rebasing to the updates base layer helped.

(Re development in container i agree, we're perhaps using Atomic in a weird way here. We're trying to make a multi-user workstation without forcing everyone to work in containers all the time. We picked Atomic mainly because multiple users have admin rights on the machine, and in this situation Atomic should accumulate less bit rot over time than traditional distros.)

@dustymabe

This comment has been minimized.

Copy link
Member

commented Mar 5, 2019

@cgwalters Thanks! Rebasing to the updates base layer helped.

@jistr - I do want to point out that using the updates base layer (fedora/29/x86_64/updates/atomic-host) works, but it goes through less testing than the stable ref (fedora/29/x86_64/atomic-host) that only gets releases every two weeks. Just something to consider when making this decision.

@fbruetting

This comment has been minimized.

Copy link

commented Apr 28, 2019

I don’t have the time to read through all of this (nor do I understand all those details mentioned here), but I’d like to reference my proposal I wrote in #1817, where I was told to repost here in this more appropriate thread:

Better handling of upgrade abortions due to inconsistency

At updating, I more and more often get the error “some base packages would be replaced”. As far as I know, this is due to the fact that the build server has not completely iterated over everything, which is why package versions differ from those of the base image. This is not the nicest user experience, and the users then have to try updating several times, until a consistent state is encountered.

Wouldn’t it be a good idea to instead leave out parts of the updates, which already “progressed too far”? So for example, if the dnf package got bumped, but the base image doesn’t reflect that yet, rpm-ostree could run the update successfully with updating to the second most recent version of the dnf package. For small update deltas, this likely means to just leave the dnf package at the current version, but at least everything else could be updated.

Another solution might be to gate all updates until the build server ran through everything, so that consistency is ensured at all times. I assume that this is easier, but has the downside that security updates would be deferred.

Additionally, I yesterday was inspired to one more proposal:

I wanted to install ansible, but as the upgrades of the days before were not successful, my staging deployment (correct term?) was “polluted” by being in a partly upgraded state, which didn’t let me install ansible. So I saw the recommendation to do rpm-ostree cleanup -m for getting rid of that polluted staging deployment (which helped).

Against to this current way things work, I wished this was done implicitly, because I (with very limited knowlegde of ostree) don’t see a use case in having those partly upgraded images.

As far as I know, those downloaded, incomplete applied delta upgrades stay in the cache until being purged or fully applied. So with that intermediate image state not being of use to me at all, purging that image implicitly and installing ansbile correctly would have met my expectations very well. With the delta upgrades residing in the cache, I also would not have lost download volume.

If the deltas do – against my assumptions – not stay cached on the systems, I propose doing that. 😜

Thoughts? 😊

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.