A proposal for "features/extras" packaging (Rust, Python, etc.) without subpackages

[ferdnyc]

This is me floating an idea I had for creating packages that have variable dependencies based on what features are requested, that doesn't rely on creating a unique subpackage per feature definition. Any and all feedback, criticism, corrections, etc. welcome. Please poke as many holes in this as you can find. (Bonus points if you have an idea for how said hole can be plugged.)

The problem

IMHO the explosion of feature-subpackages creates distracting amounts of clutter in the package collection, as well as overly verbose package names (inserting the feature name into the package name for each feature provided), when all of those feature packages contain nothing other than unique Provides: and Requires: dependencies (no actual files), meaning there'd be no reason for the subpackages at all if the main package could express the necessary information.

An example

Take the rust-hyper package in Fedora 44. Like many Rust packages, it consists of a main package (rust-hyper-devel) that contains all of the files for the package, plus EIGHT subpackages for its various features.

The subpackages look like this:

$ for sub in default server http1 http2 client full capi tracing; \
  do name="rust-hyper+$sub-devel"; \
  echo "\n$name\n---------------"; \
  echo "===== Files"; rpm -ql $name; \
  echo "===== Provides"; rpm -q --provides $name; \
  echo "===== Requires"; (rpm -q --requires $name | grep -v rpmlib); \
  echo; done

rust-hyper+default-devel
---------------
===== Files
/usr/share/cargo/registry/hyper-1.11.0/Cargo.toml
===== Provides
crate(hyper/default) = 1.11.0
rust-hyper+default-devel = 1.11.0-1.fc44
===== Requires
cargo
crate(hyper) = 1.11.0

rust-hyper+server-devel
---------------
===== Files
/usr/share/cargo/registry/hyper-1.11.0/Cargo.toml
===== Provides
crate(hyper/server) = 1.11.0
rust-hyper+server-devel = 1.11.0-1.fc44
===== Requires
(crate(httpdate/default) >= 1.0.0 with crate(httpdate/default) < 2.0.0~)
(crate(pin-project-lite/default) >= 0.2.4 with crate(pin-project-lite/default) < 0.3.0~)
(crate(smallvec/const_generics) >= 1.12.0 with crate(smallvec/const_generics) < 2.0.0~)
(crate(smallvec/const_new) >= 1.12.0 with crate(smallvec/const_new) < 2.0.0~)
(crate(smallvec/default) >= 1.12.0 with crate(smallvec/default) < 2.0.0~)
cargo
crate(hyper) = 1.11.0

rust-hyper+http1-devel
---------------
===== Files
/usr/share/cargo/registry/hyper-1.11.0/Cargo.toml
===== Provides
crate(hyper/http1) = 1.11.0
rust-hyper+http1-devel = 1.11.0-1.fc44
===== Requires
(crate(atomic-waker/default) >= 1.1.2 with crate(atomic-waker/default) < 2.0.0~)
(crate(futures-channel/default) >= 0.3.0 with crate(futures-channel/default) < 0.4.0~)
(crate(futures-core/default) >= 0.3.31 with crate(futures-core/default) < 0.4.0~)
(crate(httparse/default) >= 1.9.0 with crate(httparse/default) < 2.0.0~)
(crate(itoa/default) >= 1.0.0 with crate(itoa/default) < 2.0.0~)
cargo
crate(hyper) = 1.11.0

rust-hyper+http2-devel
---------------
===== Files
/usr/share/cargo/registry/hyper-1.11.0/Cargo.toml
===== Provides
crate(hyper/http2) = 1.11.0
rust-hyper+http2-devel = 1.11.0-1.fc44
===== Requires
(crate(atomic-waker/default) >= 1.1.2 with crate(atomic-waker/default) < 2.0.0~)
(crate(futures-channel/default) >= 0.3.0 with crate(futures-channel/default) < 0.4.0~)
(crate(futures-core/default) >= 0.3.31 with crate(futures-core/default) < 0.4.0~)
(crate(h2/default) >= 0.4.14 with crate(h2/default) < 0.5.0~)
cargo
crate(hyper) = 1.11.0

rust-hyper+client-devel
---------------
===== Files
/usr/share/cargo/registry/hyper-1.11.0/Cargo.toml
===== Provides
crate(hyper/client) = 1.11.0
rust-hyper+client-devel = 1.11.0-1.fc44
===== Requires
(crate(pin-project-lite/default) >= 0.2.4 with crate(pin-project-lite/default) < 0.3.0~)
(crate(smallvec/const_generics) >= 1.12.0 with crate(smallvec/const_generics) < 2.0.0~)
(crate(smallvec/const_new) >= 1.12.0 with crate(smallvec/const_new) < 2.0.0~)
(crate(smallvec/default) >= 1.12.0 with crate(smallvec/default) < 2.0.0~)
(crate(want/default) >= 0.3.0 with crate(want/default) < 0.4.0~)
cargo
crate(hyper) = 1.11.0

rust-hyper+full-devel
---------------
===== Files
/usr/share/cargo/registry/hyper-1.11.0/Cargo.toml
===== Provides
crate(hyper/full) = 1.11.0
rust-hyper+full-devel = 1.11.0-1.fc44
===== Requires
cargo
crate(hyper) = 1.11.0
crate(hyper/client) = 1.11.0
crate(hyper/http1) = 1.11.0
crate(hyper/http2) = 1.11.0
crate(hyper/server) = 1.11.0

rust-hyper+capi-devel
---------------
===== Files
/usr/share/cargo/registry/hyper-1.11.0/Cargo.toml
===== Provides
crate(hyper/capi) = 1.11.0
rust-hyper+capi-devel = 1.11.0-1.fc44
===== Requires
cargo
crate(hyper) = 1.11.0

rust-hyper+tracing-devel
---------------
===== Files
/usr/share/cargo/registry/hyper-1.11.0/Cargo.toml
===== Provides
crate(hyper/tracing) = 1.11.0
rust-hyper+tracing-devel = 1.11.0-1.fc44
===== Requires
(crate(tracing) >= 0.1.0 with crate(tracing) < 0.2.0~)
(crate(tracing/std) >= 0.1.0 with crate(tracing/std) < 0.2.0~)
cargo
crate(hyper) = 1.11.0

The proposal

What I would propose, instead of the current glut of subpackages, is a way of defining dependencies "for" only specific Provides:. All of the various feature Provides: (crate(hyper/server), crate(hyper/default), etc.) would be defined on the main package, instead of in subpackages, and the dependencies for each Provides: would be set as Requires: conditional on a specific Provides:.

So, for example, the .spec syntax, wrapped for readability, might be (and this is of course subject to change, it's just an example of one possible format — also, in reality these Requires: would be created by the Rust build tooling, only the Provides: for the feature crates would be defined in the .spec file, in place of the current subpackages):

Provides: crate(hyper/server) = 1.11.0
Requires: (crate(httpdate/default) >= 1.0.0
           with crate(httpdate/default) < 2.0.0~)
          for crate(hyper/server)
Requires: (crate(pin-project-lite/default) >= 0.2.4
           with crate(pin-project-lite/default) < 0.3.0~)
          for crate(hyper/server)
Requires: (crate(smallvec/const_generics) >= 1.12.0
           with crate(smallvec/const_generics) < 2.0.0~)
          for crate(hyper/server)
Requires: (crate(smallvec/const_new) >= 1.12.0
           with crate(smallvec/const_new) < 2.0.0~)
          for crate(hyper/server)
Requires: (crate(smallvec/default) >= 1.12.0
           with crate(smallvec/default) < 2.0.0~)
          for crate(hyper/server)

Provides: crate(hyper/client) = 1.11.0
Requires: (crate(pin-project-lite/default) >= 0.2.4
           with crate(pin-project-lite/default) < 0.3.0~)
          for crate(hyper/client)
Requires: (crate(smallvec/const_generics) >= 1.12.0
           with crate(smallvec/const_generics) < 2.0.0~)
          for crate(hyper/client)
Requires: (crate(smallvec/const_new) >= 1.12.0
           with crate(smallvec/const_new) < 2.0.0~)
          for crate(hyper/client)
Requires: (crate(smallvec/default) >= 1.12.0
           with crate(smallvec/default) < 2.0.0~)
          for crate(hyper/client)
Requires: (crate(want/default) >= 0.3.0
           with crate(want/default) < 0.4.0~)
          for crate(hyper/client)

Provides: crate(hyper/full) = 1.11.0
Requires: crate(hyper/client) = 1.11.0 for crate(hyper/full)
Requires: crate(hyper/http1) = 1.11.0 for crate(hyper/full)
Requires: crate(hyper/http2) = 1.11.0 for crate(hyper/full)
Requires: crate(hyper/server) = 1.11.0 for crate(hyper/full)

The idea is that, installing the package rust-hyper-devel-1.11.0-1.fc44.noarch either by package name or via its crate(hyper) Provides: would install the package files, and pull in the unconditional Requires: (the ones without for %{some_provide} conditions) as dependencies.

Installing, or using as a dependency, one of the crate(hyper/foo) provides, would also pull in any Requires: listed as for crate(hyper/foo).

Open questions

1. Is this remotely workable, from an RPM package definition / database perspective? (Obviously it would require new features developed for both RPM and DNF5, but does it seem possible to write that code within the current codebase / APIs?)

2. Is this remotely implementable, in the Rust or Python build tooling? (Since the Provides: for extras subpackages are autogenerated based on the packaging metadata, that tooling would need to be able to generate conditional Requires: ... for metadata instead.)

3. Are there any feature/extra packages that contain files other than the %ghost copy of their package definition? For example, source files that are unique to the feature in question?

   If so, that would probably torpedo this, but I think the answer is "No". Even if such files could theoretically exist (in terms of the native package definition), I don't believe our current tooling has any means of expressing that kind of relationship. The complete set of source files is included in the main package. (I believe. PLEASE correct me if I'm wrong!)

4. Would the conditional Requires: ... for dependency definitions be sufficient for this to work, or would the RPM definition of "installed" need to change to separately track individual Provides:?

   Currently if a package is installed, it's assumed that all of its Provides: are available, and all of its Requires: are installed (or RPM flags the package as missing dependencies). But for these packages, having the physical package installed means that not all Provides: will necessarily be "active", and so not all of their Requires: ... for dependencies will necessarily be installed. A conditional Requires: ... for %{some_provide} only needs to be installed if there's some installed package that Requires: %{some_provide}.

   So, would it be sufficient to have the Requires: ... for dependencies only activate when the Provides: is activated as a dependency (or an explicit request, like sudo dnf install %{some_provide}), or would the RPM database need to track which Provides: of a given package have been installed, and which have not? (Or is there some other way of making this work?)

5. Perhaps the most important question, as it makes the others moot if the answer is no: Would there be any interest in migrating some future Fedora / RHEL release's Rust, Python, etc. packaging to use these features, instead of the current glut-o'-subpackages setup?