Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No updates since 39.20231205.0 #516

Closed
chrisawi opened this issue Dec 9, 2023 · 27 comments
Closed

No updates since 39.20231205.0 #516

chrisawi opened this issue Dec 9, 2023 · 27 comments
Labels
bug Something isn't working f38 Related to Fedora 38 f39 Related to Fedora 39

Comments

@chrisawi
Copy link

chrisawi commented Dec 9, 2023

The composes seem to be finishing correctly, but rpm-ostree upgrade isn't finding them. Notably, this means the fix for containers/podman#20872 hasn't gone out yet.

https://kojipkgs.fedoraproject.org/compose/updates/Fedora-39-updates-20231208.0/logs/x86_64/Everything/ostree-1/create-ostree-repo.log

Side note:

NOTICE: Running rpm-ostree compose tree without --unified-core is deprecated.
Please add --unified-core to the command line and ensure your content
works with it. For more information, see coreos/rpm-ostree#729

Is that expected? I thought F39 was using unified core.

@chrisawi chrisawi added the bug Something isn't working label Dec 9, 2023
@walking-octopus
Copy link

walking-octopus commented Dec 9, 2023

Can reproduce on Kinoite.

stas@thinkpad-t14 ~> sudo rpm-ostree upgrade
[sudo] password for stas: 
note: automatic updates (stage) are enabled
2 metadata, 0 content objects fetched; 788 B transferred in 3 seconds; 0 bytes content written
Checking out tree 694f995... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-archive
Updating metadata for 'updates'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2023-03-14T10:57:01Z solvables: 4
rpm-md repo 'updates'; generated: 2023-12-09T02:29:32Z solvables: 15530
rpm-md repo 'fedora' (cached); generated: 2023-11-01T00:12:39Z solvables: 70825
rpm-md repo 'updates-archive'; generated: 2023-12-09T02:55:17Z solvables: 15912
Resolving dependencies... done
No upgrade available.
stas@thinkpad-t14 ~> rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: last run 15h ago
Deployments:
● fedora:fedora/39/x86_64/kinoite
                  Version: 39.20231205.0 (2023-12-05T01:04:12Z)
               BaseCommit: 694f995d4b7ca42e87b393ff3058949dad660e5f2d692338dacaaceab06cd754
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: 'gcc-c++' adb alsa-lib-devel automake cmake duperemove ffmpeg-free fish gcc htop
                           jetbrains-mono-fonts-all make mozilla-openh264 podman-compose podman-docker
                           powertop systemd-devel tmux xeyes

  fedora:fedora/39/x86_64/kinoite
                  Version: 39.20231205.0 (2023-12-05T01:04:12Z)
               BaseCommit: 694f995d4b7ca42e87b393ff3058949dad660e5f2d692338dacaaceab06cd754
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: 'gcc-c++' adb alsa-lib-devel automake cmake duperemove ffmpeg-free fish gcc htop
                           jetbrains-mono-fonts-all make mozilla-openh264 podman-compose podman-docker
                           powertop systemd-devel tmux xeyes

  fedora:fedora/38/x86_64/kinoite
                  Version: 38.20231123.0 (2023-11-23T02:20:11Z)
               BaseCommit: 9b3d24e0139e07a6f334af5565761393b0a4d2b310a4bcfee77c2e0f5c4fc6d8
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
          LayeredPackages: 'gcc-c++' automake cmake duperemove ffmpeg-free fish gcc htop
                           jetbrains-mono-fonts-all make mozilla-openh264 podman-compose podman-docker
                           powertop tmux xeyes
                   Pinned: yes

AvailableUpdate:
           Diff: 6 upgraded

@gerryxiao
Copy link

same here:
gerryxiao@fedora ~> toolbox
Error: failed to get the Podman version
gerryxiao@fedora /> bat /etc/os-release
1 │ NAME="Fedora Linux"
2 │ VERSION="39.20231205.0 (Sericea)"
3 │ ID=fedora
4 │ VERSION_ID=39
5 │ VERSION_CODENAME=""
6 │ PLATFORM_ID="platform:f39"
7 │ PRETTY_NAME="Fedora Linux 39.20231205.0 (Sericea)"
8 │ ANSI_COLOR="0;38;2;60;110;180"
9 │ LOGO=fedora-logo-icon
10 │ CPE_NAME="cpe:/o:fedoraproject:fedora:39"
11 │ DEFAULT_HOSTNAME="fedora"
12 │ HOME_URL="https://fedoraproject.org/sericea/"
13 │ DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-sericea/"
14 │ SUPPORT_URL="https://ask.fedoraproject.org/"
15 │ BUG_REPORT_URL="https://gitlab.com/fedora/sigs/sway/SIG/-/issues"
16 │ REDHAT_BUGZILLA_PRODUCT="Fedora"
17 │ REDHAT_BUGZILLA_PRODUCT_VERSION=39
18 │ REDHAT_SUPPORT_PRODUCT="Fedora"
19 │ REDHAT_SUPPORT_PRODUCT_VERSION=39
20 │ SUPPORT_END=2024-11-12
21 │ VARIANT="Sericea"
22 │ VARIANT_ID=sericea
23 │ OSTREE_VERSION='39.20231205.0'

@travier travier added f38 Related to Fedora 38 f39 Related to Fedora 39 labels Dec 10, 2023
@travier
Copy link
Member

travier commented Dec 10, 2023

This is indeed super weird. We'll need to file an infra issue.

@RobusTetus
Copy link

This is probably unrelated, although I am experiencing same issue, but on top of that GNOME Software reports glibc-langpack-cs can be updated. After installing and rebooting, it is still there as if it did not get installed.

@travier
Copy link
Member

travier commented Dec 11, 2023

@fizzyizzy05
Copy link

This is probably unrelated, although I am experiencing same issue, but on top of that GNOME Software reports glibc-langpack-cs can be updated. After installing and rebooting, it is still there as if it did not get installed.

I'm having the same thing with glibc-langpack-en

GNOME software shows there's an available update, and appears to be installing it and offers to reboot, but it does nothing and the update is still there on reboot. Running rpm-ostree upgrade --preview shows there's an upgrade available to 2.38-14, but running rpm-ostree upgrade (no --preview or --check) shows it's fine.

@chrisawi
Copy link
Author

Running rpm-ostree upgrade --preview shows there's an upgrade available to 2.38-14, but running rpm-ostree upgrade (no --preview or --check) shows it's fine.

This is a known issue in rpm-ostree: coreos/rpm-ostree#1579

It pops up whenever base image updates are interrupted.

See also: https://gitlab.gnome.org/GNOME/gnome-software/-/issues/1728

@bitestring
Copy link

Can confirm. Two of my Fedora Silverblue instances stopped receiving updates since version 39.20231205.0.

Also rpm-ostree is constantly busy

╰─>$ rpm-ostree status
State: busy
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: inactive
Transaction: refresh-md
  Initiator: caller :1.72

doing nothing. Even rpm-ostree cancel does not work.

@travier
Copy link
Member

travier commented Dec 12, 2023

Three options for workarounds until this is resolved:

  • Switch to the unofficial container images (they are updated daily):
$ rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/fedora-ostree-desktops/silverblue:39
$ rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/fedora-ostree-desktops/kinoite:39
$ rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/fedora-ostree-desktops/sericea:39
$ rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/fedora-ostree-desktops/onyx:39
  • Replace podman by a working version:
# 4.8.1 from stable repos:
$ rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2023-e8cfbfbdd9
# or 4.8.2 from testing repos:
$ rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2023-68ad6ada3b
  • If you don't mind losing all containers, images, volumes:
$ podman system reset

@travier
Copy link
Member

travier commented Dec 12, 2023

Likely fixed with https://pagure.io/fedora-infra/ansible/pull-request/1697. We'll see tomorrow.

@rscm
Copy link

rscm commented Dec 13, 2023

Likely fixed with https://pagure.io/fedora-infra/ansible/pull-request/1697. We'll see tomorrow.

not yet, I tried updating and I'm still on the same BaseCommit


also this is not the first time it happens #475

❯ rpm-ostree status -b
State: idle
BootedDeployment:
● fedora:fedora/39/x86_64/silverblue
                  Version: 39.20231205.0 (2023-12-05T00:55:28Z)
               BaseCommit: ae1afb462f69a9170bdfa24ef9fc3059b2366480c4fbcfac2a15c0fc65487d8a
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
      RemovedBasePackages: firefox firefox-langpacks 120.0.1-1.fc39 gnome-software gnome-software-rpm-ostree 45.1-1.fc39
          LayeredPackages: akmod-nvidia android-tools distrobox gnome-tweaks kernel-tools pipewire-codec-aptx printer-driver-brlaser rpmfusion-free-release rpmfusion-nonfree-release starship steam-devices tailscale terminator xboxdrv
                           xorg-x11-drv-nvidia

~ 

Can confirm. Two of my Fedora Silverblue instances stopped receiving updates since version 39.20231205.0.

Also rpm-ostree is constantly busy

╰─>$ rpm-ostree status
State: busy
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: inactive
Transaction: refresh-md
  Initiator: caller :1.72

doing nothing. Even rpm-ostree cancel does not work.

that happened to me, at least in my case was gnome-software monopolizing the process (in a bad way)

there are two solutions/workarounds for that:

  • Permanent: disable/remove gnome-software (gnome-software and gnome-software-rpm-ostree)
  • Temporary: restart rpm-ostreed service and be quick to write the command you need before gnome-software hijack the service again, repeat if it does

@moljac024
Copy link

I am still having this issue:

❯ rpm-ostree update --preview
1 metadata, 0 content objects fetched; 592 B transferred in 0 seconds; 0 bytes content written
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora rpmfusion-nonfree-nvidia-driver google-chrome rpmfusion-nonfree-steam phracek-PyCharm updates-archive
Updating metadata for 'fedora-cisco-openh264'... done
Updating metadata for 'updates'... done
Updating metadata for 'fedora'... done
Updating metadata for 'rpmfusion-nonfree-nvidia-driver'... done
Updating metadata for 'google-chrome'... done
Updating metadata for 'rpmfusion-nonfree-steam'... done
Updating metadata for 'phracek-PyCharm'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264'; generated: 2023-03-14T10:57:01Z solvables: 4
rpm-md repo 'updates'; generated: 2023-12-13T01:20:39Z solvables: 15915
rpm-md repo 'fedora'; generated: 2023-11-01T00:12:39Z solvables: 70825
rpm-md repo 'rpmfusion-nonfree-nvidia-driver'; generated: 2023-11-30T17:20:04Z solvables: 29
rpm-md repo 'google-chrome'; generated: 2023-12-12T20:06:21Z solvables: 3
rpm-md repo 'rpmfusion-nonfree-steam'; generated: 2023-08-10T16:27:35Z solvables: 2
rpm-md repo 'phracek-PyCharm'; generated: 2023-08-10T15:35:19Z solvables: 5
rpm-md repo 'updates-archive'; generated: 2023-12-13T01:38:01Z solvables: 16689
Note: --check and --preview may be unreliable.  See https://github.com/coreos/rpm-ostree/issues/1579
AvailableUpdate:
       Upgraded: glibc-langpack-en 2.38-11.fc39 -> 2.38-14.fc39
                 gnutls-dane 3.8.2-1.fc39 -> 3.8.2-2.fc39
                 gnutls-utils 3.8.2-1.fc39 -> 3.8.2-2.fc39
❯ rpm-ostree update
2 metadata, 0 content objects fetched; 788 B transferred in 1 seconds; 0 bytes content written
Checking out tree ae1afb4... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora rpmfusion-nonfree-nvidia-driver google-chrome rpmfusion-nonfree-steam phracek-PyCharm updates-archive
Updating metadata for 'fedora-cisco-openh264'... done
Updating metadata for 'updates'... done
Updating metadata for 'fedora'... done
Updating metadata for 'rpmfusion-nonfree-nvidia-driver'... done
Updating metadata for 'google-chrome'... done
Updating metadata for 'rpmfusion-nonfree-steam'... done
Updating metadata for 'phracek-PyCharm'... done
Updating metadata for 'updates-archive'... done
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264'; generated: 2023-03-14T10:57:01Z solvables: 4
rpm-md repo 'updates'; generated: 2023-12-13T01:20:39Z solvables: 15915
rpm-md repo 'fedora'; generated: 2023-11-01T00:12:39Z solvables: 70825
rpm-md repo 'rpmfusion-nonfree-nvidia-driver'; generated: 2023-11-30T17:20:04Z solvables: 29
rpm-md repo 'google-chrome'; generated: 2023-12-12T20:06:21Z solvables: 3
rpm-md repo 'rpmfusion-nonfree-steam'; generated: 2023-08-10T16:27:35Z solvables: 2
rpm-md repo 'phracek-PyCharm'; generated: 2023-08-10T15:35:19Z solvables: 5
rpm-md repo 'updates-archive'; generated: 2023-12-13T01:38:01Z solvables: 16689
Resolving dependencies... done
No upgrade available.

@RobusTetus
Copy link

I don't know if the today's compose happened yet. But I can confirm, still nothing over here too. 😢

@Mershl
Copy link

Mershl commented Dec 13, 2023

I don't know if the today's compose happened yet. But I can confirm, still nothing over here too. 😢

You can query for the latest available image on the repo using
ostree remote summary fedora | grep -A3 "fedora/39/x86_64/silverblue"

Version (ostree.commit.version): 39.20231205.0 -> 5th December 2023

@travier
Copy link
Member

travier commented Dec 13, 2023

Building with unified core is fixed. The builds are still not pushed to the repo. Still needs more investigation.

@RobusTetus
Copy link

Building with unified core is fixed. The builds are still not pushed to the repo. Still needs more investigation.

My bad I thought that it will get built today.

Likely fixed with https://pagure.io/fedora-infra/ansible/pull-request/1697. We'll see tomorrow.

I misunderstood your message.

@SnuggleCovenant
Copy link

SnuggleCovenant commented Dec 13, 2023

Three options for workarounds until this is resolved:

* Switch to the unofficial container images (they are updated daily):
$ rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/fedora-ostree-desktops/silverblue:39
$ rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/fedora-ostree-desktops/kinoite:39
$ rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/fedora-ostree-desktops/sericea:39
$ rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/fedora-ostree-desktops/onyx:39
* Replace podman by a working version:
# 4.8.1 from stable repos:
$ rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2023-e8cfbfbdd9
# or 4.8.2 from testing repos:
$ rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2023-68ad6ada3b
* If you don't mind losing all containers, images, volumes:
$ podman system reset

I tried the second option.

$ rpm-ostree override replace https://bodhi.fedoraproject.org/updates/FEDORA-2023-e8cfbfbdd9
$ systemctl reboot
$ rpm-ostree cleanup -m
$ rpm-ostree upgrade
[...] No upgrade available.
$ ostree remote summary fedora | grep -A3 "fedora/39/x86_64/silverblue"
* fedora/39/x86_64/silverblue
    Latest Commit (61,0 kB):
      ae1afb462f69a9170bdfa24ef9fc3059b2366480c4fbcfac2a15c0fc65487d8a
   Version (ostree.commit.version): 39.20231205.0

I also tried the first option which is rebasing on the latest nightly. ostree remote summary still shows the base from 5 dec 23 so that makes me think i might have to keep upgrading by rebasing regularly?

I think I'll switch to ublue since they are unaffected and signed.

I'm surprised how little attention this issue seems to be getting from what I can gather as a simple user.

@travier
Copy link
Member

travier commented Dec 13, 2023

ublue images are based on the images listed in the first workaround.

The second and third options are not going to get you updates but only a working podman if you are affected by the issue from the 4.8.0 release: #513.

@Froggy232
Copy link

Hi,
Is there an ETA for when it will be solved?
Thanks you

@miabbott
Copy link
Member

Is there an ETA for when it will be solved?

No ETA at this time; you can follow progress in the Fedora ticket - https://pagure.io/releng/issue/11827

@sn4201
Copy link

sn4201 commented Dec 15, 2023

I'm experiencing this same behaviour with the glibc update showing as always available but not receiving anything isnce 12-05-2023

So am I correct in understanding that we simply wait for the usual update mechanism to have a fix pushed remotely? Will it simply start working again or will there be some manual intervention required afterward?

@Mershl
Copy link

Mershl commented Dec 15, 2023

39.20231215.1 just popped up on the repo. Happy updating everybody.

@rene-geile
Copy link

Hi, successfully upgraded now with lots of Packages to 39.20231215.1 Previous attempt ~~19:20 CET failed ( online 1 Package SDL2 Update) so it seems directly related to the fix.

@travier
Copy link
Member

travier commented Dec 15, 2023

Should be good for Fedora 39. Fedora 38 will get its update tomorrow.

@RobusTetus
Copy link

Should be good for Fedora 39. Fedora 38 will get its update tomorrow.

Can confirm it's all good for me on Silverblue. Successfully updated using the rpm-ostree update command. Thank you for your hard work. 👍

@travier
Copy link
Member

travier commented Dec 16, 2023

Should be fixed for F38 & F39 now. Closing.

@travier travier closed this as completed Dec 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working f38 Related to Fedora 38 f39 Related to Fedora 39
Projects
None yet
Development

No branches or pull requests

16 participants