[Feature request] Optional runtime dependencies

[BlackLotus]

Ok this is basically https://github.com/voidlinux/xbps/issues/119
A package should define optional dependencies, maybe a xbps-install option could also be added for this --as-dependency or something like this. If you install a package that optionally depends on $foo you could then simply install it through

xbps-install -Sy $foo --as-dependency

and when removing the package $bar that is the only package that has a (optional) dependency on it it can be handled as normal dependency and remove it with the package.

[Duncaen]

You can use xbps-pkgdb to change the installed package mode to auto to achieve this behavior.

xbps-install $foo
xbps-pkgdb -m auto $foo

[fosslinux]

.... What if a package can't be split into subpackages? I personally think that it'd be a good idea to have this option.

[Duncaen]

There are usecases for it, that is not the question, more like how it should behave, I think pacmans implementation is not very good.

Here is what I was thinking about:

2019-07-08 19:45:08     duncaen was thinking about adding a flag to xbps-install like -o/--optional-depends which makes xbps-install ask for uninstalled optional dependencies
2019-07-08 19:45:53     leah2   as long as it's off by default
2019-07-08 19:46:04     duncaen yep thats my main requirement
2019-07-08 19:46:35     duncaen was thinkaing about making -o ask [y/n] for new uninstalled optional dependencies
2019-07-08 19:46:56     duncaen maybe another one like -O to install all or maybe just combine it with -oy
2019-07-08 19:47:17     duncaen optional dependencies installed thorugh this would be marked as auto installed
2019-07-08 19:48:09     duncaen obsolete detection would also look if the package is used in any optional dpeendencies
2019-07-08 19:51:55     duncaen xbps-remove -o and xbps-query -o wouldn't remove/list them
2019-07-08 19:52:29     duncaen but there would be a new xbps-query flag that lists automatic installed packages iwthout "hard" dependencies
2019-07-08 19:52:53     duncaen xbps-remove optdep would still work
2019-07-08 20:13:17     maldridge       I dislike this
2019-07-08 20:13:30     maldridge       but I'm dealing with a prod problem
2019-07-08 20:13:33     maldridge       so I'll have to expand later

@the-maldridge disliked the idea, but I don't think it was discussed why.

[the-maldridge]

It took me longer than I wanted to get back to this. The gist of what I dislike here is that it adds bloat where there wasn't previously bloat. In my experience the number of places where you could absolutely confine a dependency as optional is fairly small. In that case, it seems reasonable to expect a well informed user to install the packages they so desire.

On a more general and philosophical note, I'm against adding any more features to xbps until the ones we already have work better (see repo signing, staging detection, general speed, etc).

[eli-schwartz]

Why? well, with subpkgs we don't need to have bins that are useless unless a dependency is installed.

This is technically the case with pacman too. In the case of pacman, there is also a culture that "more packages are bad, because users shouldn't need to install a dozen packages just to get one thing".

On the other hand, often it is simply utterly impossible to split things out.

$ pacman -Qi gvim
[...]
Optional Deps   : python2: Python 2 language support [installed]
                  python: Python 3 language support [installed]
                  ruby: Ruby language support [installed]
                  lua: Lua language support [installed]
                  perl: Perl language support [installed]
                  tcl: Tcl language support [installed]
[...]

These optional dependencies are implemented via dlopen(), no matter what you do there's still exactly one binary, and if that binary cannot successfully dlopen the script interpreter shared library in question it will simply emit error messages that the relevant vim functions are unavailable because the support library could not be loaded.

In that case, it seems reasonable to expect a well informed user to install the packages they so desire.

$ pacman -Qi diffoscope
[...]
Optional Deps   : acl: access control list utilities support [installed]
                  binutils: binary utilities support [installed]
                  binwalk: binwalk support [installed]
                  bzip2: bzip2 utilities support [installed]
                  cdrtools: ISO utilities support
                  colord: ICC profiles support [installed]
                  cpio: cpio archive support [installed]
                  diffutils: diff utilities support [installed]
                  docx2txt: docx comparing support
                  dtc: device tree comparing support
                  e2fsprogs: Ext2/3/4 filesystem utilities support [installed]
                  enjarify: Android dex file support
                  imagemagick: ImageMagick identify support [installed]
                  fpc: Free Pascal utilities support
                  java-environment: java utilities support [installed]
                  libcaca: image compare support [installed]
                  llvm: LLVM bitcode files support [installed]
                  lz4: lz4 compression support [installed]
                  odt2txt: odt comparing support
                  fontforge: bitmap font utilities support
                  gettext: GNU internationalization utilities support [installed]
                  ghc: haskell utilities support
                  giflib: gifbuild utilities support [installed]
                  gnupg: GNU privacy guard support [installed]
                  gnumeric: GNOME Spreadsheet support
                  mono: mono support [installed]
                  openssh: OpenSSH key comparing support [installed]
                  poppler: PDF utilities support [installed]
                  r: R language support
                  sqlite: SQLite support [installed]
                  squashfs-tools: squashfs filesystem support [installed]
                  python-argcomplete: completion support
                  python-jsbeautifier: javascript beautifier support [installed]
                  python-progressbar: show progressbar support [installed]
                  tcpdump: pcap matching support [installed]
                  python-tlsh: fuzzy matching supprt
                  unzip: zip utilities support [installed]
                  gzip: gzip utilities support [installed]
                  tar: tar utilities support [installed]
                  xxd: xxd utilities support [installed]
                  xz: XZ and LZMA utilities support [installed]

"Well-informed user" seems doubtful, most well-informed users won't very well know what options are here, they will simply know that diffoscope spits out some output, which turns out to not be as detailed as it could have been because most of the comparators are autodisabled.

I think pacmans implementation is not very good.

I would tend to agree on that. Some old notes on making pacman better are here: https://wiki.archlinux.org/index.php/User:Allan/Pacman_OptDepends
But the status of optdepends in pacman can still be largely said to be "informational only", which is... of decreased usefulness. It's most likely possible to do quite a bit better.

(It is of course a valid point that there may be other far greater priorities.)

[Duncaen]

It took me longer than I wanted to get back to this. The gist of what I dislike here is that it adds bloat where there wasn't previously bloat. In my experience the number of places where you could absolutely confine a dependency as optional is fairly small. In that case, it seems reasonable to expect a well informed user to install the packages they so desire.

I don't think this is bloat and wouldn't add a lot of code and wouldn't affect anything unless you "enable" this feature.

My proposal for now would be to split optional dependencies into two stages.
First I would just add the metadata to xbps-create and make xbps-query show this metadata.

This way we could start adding the metadata through xbps-src and then evaluate if there are enough packages requiring this feature to add a few more lines to handle them automatically.
I think this would already greatly benefit users if they have a way to xbps-query -S a package and see that there are some optional dependencies they might want to install, allowing us to remove some INSTALL.msg files that currently do this.

On a more general and philosophical note, I'm against adding any more features to xbps until the ones we already have work better (see repo signing, staging detection, general speed, etc).

The proposed improvements are all a lot more work compared to adding optional dependencies, I'm going to open another RFC issue for repo signing and staging because they both are related and imho require a new tool for publishing repos, adding this to xbps-rindex would interfer with how xbps-src manages repositories and I think a new tool to manage "published" repositories out of simple xbps-src/xbps-rindex repositories would be the best way to go forward.

General speed improvements also require a lot more work, first step would be to download, checksum and verify packages at the same time instead of how it currently writes to the disk and then later reads the package file again to create a checksum. I already have a PR for this #149.
Other ways to imrpove performance would probably require rewriting the solving algorithm, which is something I'm also working on, but its something that will take a while until its finished.