handling recommends 🗣

[cgwalters]

libdnf defaults to installing Recommends (aka weak deps). There is DNF_IGNORE_WEAK_DEPS which wraps the libsolv handling.

Hence, today we default to installing weak dependencies. I think for compat with existing treefiles we'll need to continue to do so by default. Though an evaluation of what would drop out of e.g. Atomic Host if we dropped Recommends would be interesting.

However, for layered packages, I think we should support omitting them. The reason I'm filing this is right now in Fedora gdb has a Recommends on dnf.

[cgwalters]

One thing that makes this trickier for us is our anti-hysteresis - there's dnf --setopt=install_weak_deps=0 install foo. And that decision will "stick" naturally due to the way traditional package managers like dnf only upgrade what's installed.

We could pretty easily have a single global flag to ignore weak dependencies, but I'm not sure how easy it'd be to implement "per package" weak state. It seems like we'd need to do something like do a depsolve with all weak off, then for every package that isn't weak, readd it to the goal with recommends on.

[cgwalters]

dracut recommends: https://bugzilla.redhat.com/show_bug.cgi?id=1452348

[jlebon]

Hence, today we default to installing weak dependencies. I think for ABI reasons we'll need to continue to do so

Can you expand on this? If an application has a weak dep on a shared lib, then surely it can't unconditionally link against it, right? Or are you talking about something else?

[cgwalters]

I meant compat with existing treefiles. Updated the summary above.

[cgwalters]

I am thinking now that it should be pretty easily to explicitly do e.g.:

remove-recommended-packages: ['grubby']

Somehow we'd try to verify that everything that depends on it was solely via weak deps. Alternatively, we could just do remove-packages. Also a hazard here is none of this applies to e.g. RHEL7; in fact AFAICS there it's the kernel.rpm which Requires: grubby.

[nullr0ute]

It would be useful to be able to globally turn this on or off depending on the use case. For IoT we explicitly don't want weak dependencies enabled because it brings in a lot of extra things and we want ot be able to control that, for example it pulls in linux-atm-libs which we won't ever have a requirement for. We need to be able to explicitly control what we have due to size, updates etc in particular from a security PoV (if it's not shipped it can't be compromised) and from a change delta PoV (there's a lot of IoT usecases where there will be constrained bandwidth available so every bit counts (also in cases where we could potentially have millions of devices) which for a lot of the current OSTree use cases which are in datacentres where there's usually multiple Gb available.

[cgwalters]

I looked at this again briefly, but it looks like libdnf needs API here to support this. Today at least our libdnf does:

gboolean
dnf_transaction_depsolve(DnfTransaction *transaction,
...
    if (!dnf_goal_depsolve(goal, DNF_ALLOW_UNINSTALL, error))
        return FALSE;

Yet we need to pass DNF_IGNORE_WEAK there.

What I had in a local branch, rebased:

diff --git a/docs/manual/treefile.md b/docs/manual/treefile.md
index 74f96517..75e6f426 100644
--- a/docs/manual/treefile.md
+++ b/docs/manual/treefile.md
@@ -41,6 +41,11 @@ It supports the following parameters:
  * `install-langs`: Array of strings, optional.  This sets the RPM
    _install_langs macro.  Set this to e.g. `["en_US", "fr_FR"]`.
 
+ * `recommends`: Boolean, optional: Defaults to `true`.  For systems that support
+   the RPM `Recommends` flag, this may be used to globally turn off installing them.
+   Use with caution, as you may likely need to add some recommended packages to the
+   explicit `packages` list.
+
  * `mutate-os-release`: String, optional.  This causes rpm-ostree to
     change the `VERSION` and `PRETTY_NAME` fields to include the ostree
     version, and adds a specific `OSTREE_VERSION` key that can be easier
diff --git a/src/libpriv/rpmostree-core.c b/src/libpriv/rpmostree-core.c
index 630e7b84..b8c3754a 100644
--- a/src/libpriv/rpmostree-core.c
+++ b/src/libpriv/rpmostree-core.c
@@ -271,14 +271,18 @@ rpmostree_treespec_new_from_keyfile (GKeyFile   *keyfile,
   if (g_key_file_get_boolean (keyfile, "tree", "skip-sanity-check", NULL))
     g_variant_builder_add (&builder, "{sv}", "skip-sanity-check", g_variant_new_boolean (TRUE));
 
-  { gboolean documentation = TRUE;
-    g_autofree char *value = g_key_file_get_value (keyfile, "tree", "documentation", NULL);
+  /* Booleans that default to on */
+  const char *bound_bools[] = { "documentation", "recommends" };
+  for (guint i = 0; i < G_N_ELEMENTS (bound_bools); i++)
+    {
+      const char *key = bound_bools[i];
+      gboolean val = TRUE;
 
-    if (value)
-      documentation = g_key_file_get_boolean (keyfile, "tree", "documentation", NULL);
+      if (g_key_file_has_key (keyfile, "tree", key, NULL))
+        val = g_key_file_get_boolean (keyfile, "tree", key, NULL);
 
-    g_variant_builder_add (&builder, "{sv}", "documentation", g_variant_new_boolean (documentation));
-  }
+      g_variant_builder_add (&builder, "{sv}", key, g_variant_new_boolean (val));
+    }
 
   ret->spec = g_variant_builder_end (&builder);
   ret->dict = g_variant_dict_new (ret->spec);

[nullr0ute]

Is there a bug reported with libdnf for the API request?

[nullr0ute]

@cgwalters can you provide the reference to the libdnf bug/RFE?

[nullr0ute]

https://bugzilla.redhat.com/show_bug.cgi?id=1582939

[cgwalters]

PR in #1513

[heyakyra]

I need to remove a recommended package, libreswan, to be able to use a VPN for work. Is this possible now?
nm-l2tp/NetworkManager-l2tp#123 (comment)

I have strongswan installed as a replacement:

$ rpm -q strongswan
strongswan-5.8.2-5.fc32.x86_64

But ipsec still reports using libreswan:

$ ipsec --version
Linux Libreswan 3.30 (netkey) on 5.6.0-0.rc5.git0.2.fc32.x86_64

I don't know how to remove libreswan. Nothing requires or suggests it, but NetworkManager-l2tp recommends it:

$ rpm -q --whatdepends libreswan
rpm: --whatdepends: unknown option
$ rpm -q --whatsuggests libreswan
no package suggests libreswan
$ rpm -q --whatrecommends libreswan
NetworkManager-l2tp-1.8.0-5.fc32.x86_64

[jlebon]

@heyakyra You can try using rpm-ostree override remove libreswan. There may be other dependencies it pulls in that you'll need to remove as well.

[heyakyra]

Thanks for the tip, but it says it is not found:

$ rpm-ostree override remove libreswan
error: Package "libreswan" not found
$ rpm -q libreswan
libreswan-3.30-1.fc32.x86_64

Edit: This gets things working temporarily, but doesn't persist through reboots:

$ sudo rpm-ostree usroverlay
[sudo] password for kyra: 
Development mode enabled.  A writable overlayfs is now mounted on /usr.
All changes there will be discarded on reboot.

[jlebon]

Thanks for the tip, but it says it is not found:

Ahh right, it's an overlay. Hmm, if NetworkManager-l2tp did something more like Recommends: libreswan or strongswan then it'd work to do rpm-ostree install NetworkManager-l2tp strongswan.

I'm afraid I don't have a good answer for you. This is what this issue is tracking. :)

I'm not familiar with those packages, though I'd be surprised if there weren't a way to tell the software to use a specific backend.

[LorbusChris]

It seems to me the only part missing for recommends handling is the client side.

As a note, related BZ was closed: https://bugzilla.redhat.com/show_bug.cgi?id=1582939

Jaroslav Mracek 2020-12-04 16:00:42 CET
I suggest that the issue is resolved by commandline option for microdnf --setopt=install_weak_deps=false. In case that it
doesn't work like expected, please feel free to reopen the bug report.

[heyakyra]

This also appears to be blocking rebase to F35 because of recommends on glibc-gconv-extra
fedora-silverblue/issue-tracker#210

[LukeShortCloud]

I am trying to do a build with the legendary package added but that adds wine as an optional dependency (which I do not need) and eventually leads to a failed build due to this issue with installing wine: #4653

[LukeShortCloud]

Looking over this issue again, it seems to be split into two different problems:

(1) rpm-ostree installs recommended dependencies. I see the Treefile operation of recommends: false was added (true by default). That solved my issue.

(2) Recommended dependencies cannot be managed on a per-package basis. This is still lingering and why this GitHub Issue is still open.

[LukeShortCloud]

Actually, the suggested (but never created) remove-recommended-packages: workaround may be addressed by the exclude-packages: Treefile option that was implemented.

#1980

[valpackett]

I'm currently using rpm-ostree native container builds, how do I disable the installation of recommended packages in this scenario, without any treefiles?

Current use case in particular: doing a multi-stage build in the Containerfile with a separate stage for building the nvidia kernel module, to avoid all the dev dependencies in the final image; rpm-ostree install -y akmod-nvidia pulls in the userland driver in the build stage which wastes time, network bandwidth and disk space during the build.

[jlebon]

Ideally, the base image you're deriving from has dnf (if it doesn't yet, it should soon). Then you can use the regular dnf config option for this (by writing to /etc/dnf/dnf.conf or passing it in via the CLI, see e.g. https://docs.fedoraproject.org/en-US/packaging-guidelines/WeakDependencies/#_weak_dependencies).

[giusepperagusa]

I'm under the impression that the above hints and new features unfortunately do not address a basic scenario: layering a minimal set of extra packages from the commandline using "rpm-ostree install pkg1 pkg2 pkg3" on top of a running CoreOS instance.
I tried (under Fedora CoreOS 38.20231027.3.2) both creating a /etc/dnf/dnf.conf file with "install_weak_deps=False" in the "[main]" section and adding the same option to all existing repos under /etc/yum.repos.d/*.repo but to no effect: all recommended packages are installed anyway and removing the unwanted ones afterwards can become really cumbersome.
Are there any current or in roadmap solutions addressing this use case?

[jmpolom]

This is a broken/incomplete feature apparently. Fedora dependency sets and worse still recommendations are bloated. Having no control over this via rpm-ostree install is a major problem.

[jmpolom]

Ideally, the base image you're deriving from has dnf (if it doesn't yet, it should soon). Then you can use the regular dnf config option for this (by writing to /etc/dnf/dnf.conf or passing it in via the CLI, see e.g. https://docs.fedoraproject.org/en-US/packaging-guidelines/WeakDependencies/#_weak_dependencies).

@jlebon are you suggesting it should be possible in an ostree container to install dnf, install packages via dnf and then ostree container commit the changes and they will become deployable with the resulting image? What are you suggesting here and can you please provide an example workflow?

I have observed what is going on with the experimental Fedora builds over in the gitlab area (https://gitlab.com/fedora/bootc/base-images-experimental/-/blob/main/Containerfile?ref_type=heads) so hopefully you are not talking about that method. That method looks disastrous right now, if you ask me. A step back for sure. I assume we are not seeing finished features there though.

For clarity's sake I have sought to build my own base images and not "derive from" the upstream images under development as they do not meet my needs. So all this should be working nicely for those of us who are opting to use these tools as building blocks from ground 0.

[jlebon]

Ideally, the base image you're deriving from has dnf (if it doesn't yet, it should soon). Then you can use the regular dnf config option for this (by writing to /etc/dnf/dnf.conf or passing it in via the CLI, see e.g. docs.fedoraproject.org/en-US/packaging-guidelines/WeakDependencies/#_weak_dependencies).

@jlebon are you suggesting it should be possible in an ostree container to install dnf, install packages via dnf and then ostree container commit the changes and they will become deployable with the resulting image? What are you suggesting here and can you please provide an example workflow?

When doing a layered build, you can use dnf install -y --setopt=install_weak_deps=false to install packages with weak opts disabled. E.g.

FROM quay.io/fedora/fedora-bootc:40
RUN dnf install --setopt=install_weak_deps=false -y NetworkManager-l2tp && dnf clean all
# notice it doesn't pull in libreswan

For clarity's sake I have sought to build my own base images and not "derive from" the upstream images under development as they do not meet my needs. So all this should be working nicely for those of us who are opting to use these tools as building blocks from ground 0.

When doing a base build by tweaking e.g https://gitlab.com/fedora/bootc/base-images, you can set recommends: false (which is currently off in tier-0 and on in tier-1).

[jmpolom]

Ideally, the base image you're deriving from has dnf (if it doesn't yet, it should soon). Then you can use the regular dnf config option for this (by writing to /etc/dnf/dnf.conf or passing it in via the CLI, see e.g. docs.fedoraproject.org/en-US/packaging-guidelines/WeakDependencies/#_weak_dependencies).

@jlebon are you suggesting it should be possible in an ostree container to install dnf, install packages via dnf and then ostree container commit the changes and they will become deployable with the resulting image? What are you suggesting here and can you please provide an example workflow?

When doing a layered build, you can use dnf install -y --setopt=install_weak_deps=false to install packages with weak opts disabled. E.g.

FROM quay.io/fedora/fedora-bootc:40
RUN dnf install --setopt=install_weak_deps=false -y NetworkManager-l2tp && dnf clean all
# notice it doesn't pull in libreswan

Right, I found this works, but obviously only if you have dnf (or dnf5 in my case) in the image. Problem is if I'm creating an image where I want rpm-ostree present for future ad-hoc package layering (maybe I want this door open) I now have an additional need for dnf/dnf5 if I want to make lean downstream derivative builds. This is really suboptimal and also very confusing.

It seems like rpm-ostree really should have a way to disable pulling in weak deps and obey the same options dnf does, or, dnf5 should grow a feature to handle a transactional package layering with ostree. These tools perform similar activities and should behave the same. Having to have both side by side is not a solution to this problem. Unless there is something I'm missing the "solution" is basically a work around to accommodate limitations of both rpm-ostree and dnf.

What are the prospects for a more sane fix here? Is this on the horizon at all? These limitations should be documented more formally if they are to persist indefinitely. Given the seeming plans to divorce more from rpm-ostree I presume there are not really any changes planned to actually resolve this problem.

For clarity's sake I have sought to build my own base images and not "derive from" the upstream images under development as they do not meet my needs. So all this should be working nicely for those of us who are opting to use these tools as building blocks from ground 0.

When doing a base build by tweaking e.g https://gitlab.com/fedora/bootc/base-images, you can set recommends: false (which is currently off in tier-0 and on in tier-1).

I already figured this out by reading the treefile docs. It works but only at compose time. Not really a solution to the issue described.

[RichardFevrier]

A way to remove all previously installed recommends?

[travier]

rpm-ostree reset then set the config option, then re-install only the ones you need.

[RichardFevrier]

Since you are here @travier, are recommends installed on base fedora images like Silverblue or they only contains requiered packages?

[travier]

Fedora Atomic Desktops have recommended packages included by default.

[RichardFevrier]

Thanks for the info! Do you think it will be the other way around someday or it's made on purpose?

[0rzech]

@RichardFevrier @travier AFAIU, according to #4974 (comment), there's no need to reset overlayed packages. Once the knob is set, recommended packages will be removed with a new deployment.

Do you think it will be the other way around someday or it's made on purpose?

Given that package layering is meant to be a last resort, I think Atomic Desktops should have Recommends=false by default too. Now I have over 250 packages layered mostly because of how much is "recommended".

[travier]

there's no need to reset overlayed packages. Once the knob is set, recommended packages will be removed with a new deployment.

Ah, indeed, no need for the reset step.

Do you think it will be the other way around someday or it's made on purpose?

Given that package layering is meant to be a last resort, I think Atomic Desktops should have Recommends=false by default too. Now I have over 250 packages layered mostly because of how much is "recommended".

Let's move that discussion to an Atomic Desktops SIG issue.

[RichardFevrier]

I meant without layering anything. Atomic desktops could only include strict minimum package to work and not recommends.
But I also agree about having Recommends=false could be the default.

[0rzech]

Let's move that discussion to an Atomic Desktops SIG issue.

Moved here.

[jmpolom]

Is Red Hat pulling all their projects from github into gitlab.com moving forward?

[cgwalters]

Is Red Hat pulling all their projects from github into gitlab.com moving forward?

This is a pretty random place to ask that question 😄
I think the more proximate place to ask is around channels for https://communityblog.fedoraproject.org/2024-git-forge-evaluation/ e.g.

Red Hat is a large company with many sub-groups. You may observe that RHEL clearly chose to put things in https://gitlab.com/redhat/centos-stream for example. You may also observe https://github.com/openshift and how many things are there and how e.g. Prow is hardwired to Github...