ostree admin kargs edit-in-place fails with `error: Attempting to remove booted deployment` #2664

travier · 2022-07-01T11:24:00Z

Tested using ostree from https://download.copr.fedorainfracloud.org/results/@CoreOS/continuous/fedora-36-x86_64/04587999-ostree/, on a Fedora Silverblue 36 system.

Preparation:

[silverblue@fedora ~]$ sudo rpm-ostree override replace ./*.rpm
Checking out tree 847aa9f... done
Enabled rpm-md repositories: fedora-cisco-openh264 fedora-modular updates-modular updates fedora rpmfusion-free-updates rpmfusion-free rpmfusion-nonfree-updates rpmfusion-nonfree updates-archive
Importing rpm-md... done
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2022-04-07T16:52:38Z solvables: 4
rpm-md repo 'fedora-modular' (cached); generated: 2022-05-04T21:12:01Z solvables: 825
rpm-md repo 'updates-modular' (cached); generated: 2022-06-14T02:11:42Z solvables: 1155
rpm-md repo 'updates' (cached); generated: 2022-07-01T00:58:10Z solvables: 14977
rpm-md repo 'fedora' (cached); generated: 2022-05-04T21:16:11Z solvables: 67992
rpm-md repo 'rpmfusion-free-updates' (cached); generated: 2022-06-29T16:49:23Z solvables: 71
rpm-md repo 'rpmfusion-free' (cached); generated: 2022-05-04T04:48:11Z solvables: 506
rpm-md repo 'rpmfusion-nonfree-updates' (cached); generated: 2022-06-29T17:08:56Z solvables: 51
rpm-md repo 'rpmfusion-nonfree' (cached); generated: 2022-05-04T05:11:55Z solvables: 225
rpm-md repo 'updates-archive' (cached); generated: 2022-07-01T01:28:34Z solvables: 17284
Resolving dependencies... done
error: Could not depsolve transaction; 2 problems detected:
 Problem 1: package flatpak-1.12.7-3.fc36.x86_64 requires ostree-libs(x86-64) >= 2020.8, but none of the providers can be installed
  - cannot install both ostree-libs-0.15.0.4.g0d911bf9-2.fc36.x86_64 and ostree-libs-2022.4-2.fc36.x86_64
  - cannot install both ostree-libs-0.15.0.4.g0d911bf9-2.fc36.x86_64 and ostree-libs-2022.1-2.fc36.x86_64
  - cannot install both ostree-libs-0.15.0.4.g0d911bf9-2.fc36.x86_64 and ostree-libs-2022.2-1.fc36.x86_64
  - cannot install both ostree-libs-0.15.0.4.g0d911bf9-2.fc36.x86_64 and ostree-libs-2022.3-3.fc36.x86_64
  - conflicting requests
 Problem 2: package flatpak-libs-1.12.7-3.fc36.x86_64 requires ostree(x86-64) >= 2020.8, but none of the providers can be installed
  - cannot install both ostree-0.15.0.4.g0d911bf9-2.fc36.x86_64 and ostree-2022.4-2.fc36.x86_64
  - cannot install both ostree-0.15.0.4.g0d911bf9-2.fc36.x86_64 and ostree-2022.1-2.fc36.x86_64
  - cannot install both ostree-0.15.0.4.g0d911bf9-2.fc36.x86_64 and ostree-2022.2-1.fc36.x86_64
  - cannot install both ostree-0.15.0.4.g0d911bf9-2.fc36.x86_64 and ostree-2022.3-3.fc36.x86_64
  - conflicting requests
[silverblue@fedora ~]$ sudo rpm-ostree usroverlay
[silverblue@fedora ~]$ sudo rpm -ivh --nodeps --force ostree-libs-0.15.0.4.g0d911bf9-2.fc36.x86_64.rpm ostree-0.15.0.4.g0d911bf9-2.fc36.x86_64.rpm ostree-grub2-0.15.0.4.g0d911bf9-2.fc36.x86_64.rpm
[root@fedora ~]# rpm-ostree status
State: idle
Deployments:
● fedora:fedora/36/x86_64/silverblue
                   Version: 36.20220701.0 (2022-07-01T00:41:36Z)
                BaseCommit: 847aa9f009338d2945ac0bfe50a7445ac59cb3a14c528eedfa16bfcb9e81c83d
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
           LayeredPackages: rpmfusion-free-release rpmfusion-nonfree-release
                  Unlocked: development

  fedora:fedora/36/x86_64/silverblue
                   Version: 36.20220617.0 (2022-06-17T00:45:03Z)
                BaseCommit: 5bf39cb84b31d36af49e07dfb57a5c531ad7c13a8b2df91397494bea9bf6e871
              GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
           LayeredPackages: rpmfusion-free-release rpmfusion-nonfree-release

Error:

[silverblue@fedora ~]$ sudo ostree admin kargs edit-in-place --append-if-missing=rw -v
OT: Deployment a0a9938406fd9cce905703b7d24b44824fd2c19862e2d0925c45cdf90f657862.0 unlocked=1
OT: Deployment 6970d469a22e9b0386f2033440195470f588fd91fa0be86066b6184e5e61218d.0 unlocked=0
error: Attempting to remove booted deployment

Other info:

[root@fedora ~]# cat /boot/loader/entries/ostree-*
title Fedora Linux 36.20220617.0 (Silverblue) (ostree:1)
version 1
options rhgb quiet root=UUID=32b22ab6-32ee-4c5f-942d-7623c0b92fee rootflags=subvol=root ostree=/ostree/boot.1/fedora/13005bbee767bff2c2dfd52deaf0d8a6c06a30dc532c8ddc283cd066deea2656/0
linux /ostree/fedora-13005bbee767bff2c2dfd52deaf0d8a6c06a30dc532c8ddc283cd066deea2656/vmlinuz-5.18.5-200.fc36.x86_64
initrd /ostree/fedora-13005bbee767bff2c2dfd52deaf0d8a6c06a30dc532c8ddc283cd066deea2656/initramfs-5.18.5-200.fc36.x86_64.img
title Fedora Linux 36.20220701.0 (Silverblue) (ostree:0)
version 2
options rhgb quiet root=UUID=32b22ab6-32ee-4c5f-942d-7623c0b92fee rootflags=subvol=root ostree=/ostree/boot.1/fedora/80d20c9fe3480ca8444a456ba373fcc54f0b9dc8fc9ac6bd209d84150b5032e8/0
linux /ostree/fedora-80d20c9fe3480ca8444a456ba373fcc54f0b9dc8fc9ac6bd209d84150b5032e8/vmlinuz-5.18.7-200.fc36.x86_64
initrd /ostree/fedora-80d20c9fe3480ca8444a456ba373fcc54f0b9dc8fc9ac6bd209d84150b5032e8/initramfs-5.18.7-200.fc36.x86_64.img

References:

The text was updated successfully, but these errors were encountered:

travier · 2022-07-01T11:24:10Z

CC @HuijingHei

HuijingHei · 2022-07-01T13:22:08Z

Thanks @travier for your info, I will look at it

HuijingHei · 2022-07-05T11:24:03Z

Can reproduce when install new ostree-0.15.0.4.g0d911bf9-2.fc36.x86_64.rpm with 36.20220618.1.1

$ sudo ostree admin kargs edit-in-place --append-if-missing=test1 -v
OT: remounted /sysroot writable
OT: remounted /boot writable
OT: Deployment 977c43d14beab7d5dbb4f7dcd64ae64775223db39af393479c8572bf8ca01681.0 unlocked=1
error: Attempting to remove booted deployment

The error is similar with #2599.
I read about ostree_sysroot_write_deployments_with_options refer to https://github.com/ostreedev/ostree/blob/main/src/libostree/ostree-sysroot-deploy.c#L2621, not quite understand the error, any pointer is appreciated, thanks!

cgwalters · 2022-07-05T13:29:34Z

OK right, I think we lost the thread/issue about the main API ostree_sysroot_write_deployments not supporting mutate-in-place.

Hmm can you test out this patch?

diff --git a/src/libostree/ostree-sysroot-deploy.c b/src/libostree/ostree-sysroot-deploy.c
index 802e262a..e6f27b14 100644
--- a/src/libostree/ostree-sysroot-deploy.c
+++ b/src/libostree/ostree-sysroot-deploy.c
@@ -3620,18 +3620,19 @@ ostree_sysroot_deployment_set_kargs (OstreeSysroot     *self,
   g_autofree char *new_options = ostree_kernel_args_to_string (kargs);
   ostree_bootconfig_parser_set (new_bootconfig, "options", new_options);
 
-  g_autoptr(GPtrArray) new_deployments = g_ptr_array_new_with_free_func (g_object_unref);
-  for (guint i = 0; i < self->deployments->len; i++)
-    {
-      OstreeDeployment *cur = self->deployments->pdata[i];
-      if (cur == deployment)
-        g_ptr_array_add (new_deployments, g_object_ref (new_deployment));
-      else
-        g_ptr_array_add (new_deployments, g_object_ref (cur));
-    }
+  g_autofree char *bootconf_name =
+    g_strdup_printf ("ostree-%d-%s.conf",
+                     self->deployments->len - ostree_deployment_get_index (deployment),
+                     ostree_deployment_get_osname (deployment));
+
+  g_autofree char *bootconfdir = g_strdup_printf ("loader.%d/entries", self->bootversion);
+  glnx_autofd int bootconf_dfd = -1;
+  if (!glnx_opendirat (self->boot_fd, bootconfdir, TRUE, &bootconf_dfd, error))
+    return FALSE;
 
-  if (!ostree_sysroot_write_deployments (self, new_deployments,
-                                         cancellable, error))
+  if (!ostree_bootconfig_parser_write_at (ostree_deployment_get_bootconfig (deployment),
+                                          bootconf_dfd, bootconf_name,
+                                          cancellable, error))
     return FALSE;
 
   return TRUE;

Right now the unit tests don't actually appear booted, so to cover this path we'll want to add a test case to tests/kolainst/destructive.

HuijingHei · 2022-07-05T14:01:16Z

Thanks @cgwalters, I will try

HuijingHei · 2022-07-08T15:16:02Z

Build ostree refer to https://ostreedev.github.io/ostree/contributing-tutorial/#testing-in-a-container, and with patch #2669

run ./ci/build.sh in container
run make install DESTDIR=$(pwd)/installed/rootfs
copy cp -r /srv/ostree/installed/rootfs/* overrides/rootfs/ to cosa override dir, start cosa container to build disk.

Test locally it works.

Add func to set kernel arguments in place, instead of create new deployment Fix ostreedev#2664

HuijingHei · 2022-07-11T13:35:34Z

@cgwalters how about adding test into https://github.com/coreos/fedora-coreos-config instead of tests/kolainst/destructive in ostree? Any suggestions?

cgwalters · 2022-07-11T20:11:37Z

Yes, this is the long-running tension between "tests in the project git" versus "centralized tests". We have a good issue for this somewhere..

coreos/coreos-assembler#1159

Ah ah no wait I found it: coreos/fedora-coreos-config#677

The core problem here is that if you try to add a test to fedora-coreos-config right now, it will start failing because ostree doesn't have the fix.

Also, logically speaking, this is an ostree bug fix independent of Fedora CoreOS; and it'd be good to have our tests at least in theory be executable on other test frameworks and platforms.

HuijingHei · 2022-07-12T00:57:48Z

Yes, you are correct.
Sorry for confusion, build ostree including #2669, it works. (And update #2664 (comment))

Add func to set kernel arguments in place, instead of create new deployment Fix ostreedev#2664

HuijingHei · 2022-07-12T07:52:49Z

Also, logically speaking, this is an ostree bug fix independent of Fedora CoreOS; and it'd be good to have our tests at least in theory be executable on other test frameworks and platforms.

OK, I will add test in tests/kolainst/destructive.

Add func to set kernel arguments in place, instead of create new deployment Fix ostreedev#2664

travier · 2022-07-25T13:42:42Z

I'm testing this right now, looks like this is working for me. Thanks!

travier · 2022-07-25T13:59:58Z

Hum, this fails when we have pending deployments:

Jul 25 13:57:17 fedora ostree-sysroot-readonly[2417]: + ostree admin kargs edit-in-place --append-if-missing=rw
Jul 25 13:57:17 fedora audit[2427]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=2427 comm="ostree" exe="/usr/bin/ostree" sig=6 res=1
Jul 25 13:57:17 fedora ostree-sysroot-readonly[2427]: **
Jul 25 13:57:17 fedora ostree-sysroot-readonly[2427]: OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:3618:ostree_sysroot_deployment_set_kargs_in_place: assertion failed: (!ostree_deployment_is_staged (deployment))
Jul 25 13:57:17 fedora ostree-sysroot-readonly[2427]: Bail out! OSTree:ERROR:src/libostree/ostree-sysroot-deploy.c:3618:ostree_sysroot_deployment_set_kargs_in_place: assertion failed: (!ostree_deployment_is_staged (deployment))
Jul 25 13:57:17 fedora systemd[1]: Created slice system-systemd\x2dcoredump.slice - Slice /system/systemd-coredump.
Jul 25 13:57:17 fedora audit: BPF prog-id=131 op=LOAD
Jul 25 13:57:17 fedora audit: BPF prog-id=132 op=LOAD
Jul 25 13:57:17 fedora audit: BPF prog-id=133 op=LOAD
Jul 25 13:57:17 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@0-2428-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=>
Jul 25 13:57:17 fedora systemd[1]: Started systemd-coredump@0-2428-0.service - Process Core Dump (PID 2428/UID 0).
Jul 25 13:57:17 fedora [2430]: [164B blob data]
Jul 25 13:57:17 fedora systemd-coredump[2429]: [🡕] Process 2427 (ostree) of user 0 dumped core.
                                               
                                               Module linux-vdso.so.1 with build-id 539b6f5c64df6de29594035a5c6820a79cbe1e44
                                               Module libcrypt.so.2 with build-id 9924d3a82ca8fa6b94bc574a6520bcf84ccf831f
                                               Stack trace of thread 2427:
                                               #0  0x00007fccfb34dc4c __pthread_kill_implementation (libc.so.6 + 0x8ec4c)
                                               #1  0x00007fccfb2fd9c6 raise (libc.so.6 + 0x3e9c6)
                                               #2  0x00007fccfb2e77f4 abort (libc.so.6 + 0x287f4)
                                               #3  0x00007fccfb5ced18 g_assertion_message.cold (libglib-2.0.so.0 + 0x1ed18)
                                               #4  0x00007fccfb62ebae g_assertion_message_expr (libglib-2.0.so.0 + 0x7ebae)
                                               #5  0x00007fccfb9a856d ostree_sysroot_deployment_set_kargs_in_place (libostree-1.so.1 + 0x8256d)
                                               #6  0x0000556a2e4d7e08 ot_admin_kargs_builtin_edit_in_place (ostree + 0x32e08)
                                               #7  0x0000556a2e4d4f47 ot_admin_builtin_kargs (ostree + 0x2ff47)
                                               #8  0x0000556a2e4cd540 ostree_builtin_admin (ostree + 0x28540)
                                               #9  0x0000556a2e4c0ad3 main (ostree + 0x1bad3)
                                               #10 0x00007fccfb2e8550 __libc_start_call_main (libc.so.6 + 0x29550)
                                               #11 0x00007fccfb2e8609 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29609)
                                               #12 0x0000556a2e4c0d75 _start (ostree + 0x1bd75)
                                               ELF object binary architecture: AMD x86-64

travier · 2022-07-25T14:02:24Z

Moved to #2679

travier · 2022-07-25T14:05:54Z

It works without pending deployment.

HuijingHei added a commit to HuijingHei/ostree that referenced this issue Jul 8, 2022

Fix ostreedev#2664

e408b95

HuijingHei added a commit to HuijingHei/ostree that referenced this issue Jul 8, 2022

Fix ostreedev#2664

98b05a4

HuijingHei added a commit to HuijingHei/ostree that referenced this issue Jul 11, 2022

Fix ostree admin kargs edit-in-place fails issue

5ca9d73

Add func to set kernel arguments in place, instead of create new deployment Fix ostreedev#2664

HuijingHei mentioned this issue Jul 11, 2022

Fix ostree admin kargs edit-in-place fails issue #2669

Merged

HuijingHei added a commit to HuijingHei/ostree that referenced this issue Jul 12, 2022

Fix ostree admin kargs edit-in-place fails issue

eb652b0

Add func to set kernel arguments in place, instead of create new deployment Fix ostreedev#2664

HuijingHei added a commit to HuijingHei/ostree that referenced this issue Jul 12, 2022

Fix ostree admin kargs edit-in-place fails issue

4d2549b

Add func to set kernel arguments in place, instead of create new deployment Fix ostreedev#2664

HuijingHei added a commit to HuijingHei/ostree that referenced this issue Jul 12, 2022

Fix ostree admin kargs edit-in-place fails issue

7b7b6d7

Add func to set kernel arguments in place, instead of create new deployment Fix ostreedev#2664

HuijingHei added the jira label Jul 12, 2022

cgwalters closed this as completed in #2669 Jul 13, 2022

bpmooch pushed a commit to render-se/ostree that referenced this issue Jul 15, 2022

Fix ostree admin kargs edit-in-place fails issue

7f9f87d

Add func to set kernel arguments in place, instead of create new deployment Fix ostreedev#2664

travier mentioned this issue Jul 25, 2022

ostree admin kargs edit-in-place assertion reached when deployments are pending #2679

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ostree admin kargs edit-in-place fails with `error: Attempting to remove booted deployment` #2664

ostree admin kargs edit-in-place fails with `error: Attempting to remove booted deployment` #2664

travier commented Jul 1, 2022 •

edited

travier commented Jul 1, 2022

HuijingHei commented Jul 1, 2022

HuijingHei commented Jul 5, 2022 •

edited

cgwalters commented Jul 5, 2022

HuijingHei commented Jul 5, 2022

HuijingHei commented Jul 8, 2022 •

edited

HuijingHei commented Jul 11, 2022

cgwalters commented Jul 11, 2022

HuijingHei commented Jul 12, 2022

HuijingHei commented Jul 12, 2022

travier commented Jul 25, 2022

travier commented Jul 25, 2022

travier commented Jul 25, 2022

travier commented Jul 25, 2022

ostree admin kargs edit-in-place fails with error: Attempting to remove booted deployment #2664

ostree admin kargs edit-in-place fails with error: Attempting to remove booted deployment #2664

Comments

travier commented Jul 1, 2022 • edited

travier commented Jul 1, 2022

HuijingHei commented Jul 1, 2022

HuijingHei commented Jul 5, 2022 • edited

cgwalters commented Jul 5, 2022

HuijingHei commented Jul 5, 2022

HuijingHei commented Jul 8, 2022 • edited

HuijingHei commented Jul 11, 2022

cgwalters commented Jul 11, 2022

HuijingHei commented Jul 12, 2022

HuijingHei commented Jul 12, 2022

travier commented Jul 25, 2022

travier commented Jul 25, 2022

travier commented Jul 25, 2022

travier commented Jul 25, 2022

ostree admin kargs edit-in-place fails with `error: Attempting to remove booted deployment` #2664

ostree admin kargs edit-in-place fails with `error: Attempting to remove booted deployment` #2664

travier commented Jul 1, 2022 •

edited

HuijingHei commented Jul 5, 2022 •

edited

HuijingHei commented Jul 8, 2022 •

edited