sync: pick up a "provides" package from the command line

[CyberShadow]

Consider: aur sync bitlbee-libpurple-git bitlbee-discord-git

bitlbee-libpurple-git provides bitlbee, and bitlbee-discord-git depends on bitlbee, so one would expect that bitlbee-discord-git is built against bitlbee-libpurple-git. However, aurutils pulls in the bitlbee package instead.

Not sure if this is a defect or maybe there is a good reason for this. However, it is at least inconsistent with the behavior when bitlbee-discord-git is available in a repository; then, aurutils does not pull in bitlbee, but just uses bitlbee-discord-git which provides it.

[AladW]

While your expectation is perfectly valid, it's a difficult problem because aurweb cannot resolve provides. (A patch has been posted over 2 years ago, but it didn't get anywhere.) As such, one would have to resolve all possible candidates, i.e. bitlbee from depends and bitlbee-libpurple-git from the command-line, and then check for provides locally.

In case the provider is already in a repository, it is handled by aur-repo-filter. Other cases were not implemented, though aur-graph could likely be leveraged for this (using .SRCINFO).

[AladW]

Now that aurweb is fully based on Python, perhaps someone could revive/adapt https://patchwork.archlinux.org/project/aur-dev/patch/20180601025055.6938-1-afg984@gmail.com/.

[AladW]

https://gitlab.archlinux.org/archlinux/aurweb/-/issues/307

[AladW]

The following works (at the cost of a large file download):

$ time aur pkglist --info | jq -r '.[] | select(.Provides[]? | contains("libcubeb.so")) | .Name'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8459k  100 8459k    0     0  23.8M      0 --:--:-- --:--:-- --:--:-- 23.8M
cubeb-git

real	0m2.849s
user	0m3.455s
sys	0m0.293s

after caching of the .json archive (most time taken by jq):

$ time aur pkglist --info | jq -r '.[] | select(.Provides[]? | contains("libcubeb.so")) | .Name'
cubeb-git

real	0m2.358s
user	0m3.262s
sys	0m0.213s

So what would the best UI for this be? I can think of the following cases:

1. a dependency name is both a package name, and provided by a package of a different name (as for bitlbee and bitlbee-libpurple-git)
2. a dependency name is only a package name
3. a dependency name is only provided by a package of a different name

For case 1. and 3. there can either be one or multiple packages that provide the package name. Approach for case 1:

• If the dependency name is a package name, default to it as before.
• Query command-line arguments through the provides interface above. (Preferably in a single pass, jq is very slow to operate on large files)
• If there are matches (e.g. bitlbee-libpurple-git -> bitlbee) replace them in the queue. (This should not be done unconditionally; packages adding their own name to provides is considered a packaging error.)

For case 3, since the build process cannot continue without the user specifying a provides:

• If none is specified on the command-line, print a diagnostic with all possible choices. (An option like --use-first-provider which selects the first candidate can also be added.)
• Otherwise, proceed as with case 1.

Because of low performance, the above should be explicitely enabled by the user. When RPCv6 eventually supports search by provides, it could be made the default. i.e. the aur-pkglist pipline would look like:

aur query -t search -b provides libcubeb.so | jq -r '.results[].Name'

although having 1 request per provides search is far from optimal. (Having a blow-up of requests only to resolve provides is certainly not something I want.)

edit: https://gitlab.archlinux.org/archlinux/aurweb/-/issues/307#note_63764

---

Arguably the implementation also belongs in aur-depends instead of aur-sync.

[VorpalBlade]

This is also an issue with .so provides on the command line. Example command which triggered issues:

aur sync -c qtforkawesome qtutilities c++utilities syncthingtray

qtforkawesome fails to build because (according to alad on IRC):

<alad> Vorpal, oh, I see now for 1): error: target not found: libqtforkawesome.so - this would likely by solved by using aur-graph to resolve the dependencies a second time, using the .SRCINFO

[AladW]

provides on the command line are addressed with 9680df6

[AladW]

So, with aur-format merged you can create a shlibs file as follows:

aur pkglist --info | aur format -f '%n\t%P\n' --delim=, \
    | awk -F'\t' '{ split($2, provs, ","); for (p in provs) { sub(/[<>=].*/, "", provs[p]); printf("%s\t%s\n", provs[p], $1); }; }' \
    | sort -k1,1b -u > shlibs

This file can then be used to resolve provides to AUR targets. Sample output:

libaacs.so      libaacs-git
libacl.so       acl-git
libad9361       libad9361-git
libad9361-git   libad9361-git
libad9361-iio   libad9361-iio-git
libad9361.so    libad9361-git
libadalang      libadalang-git
libadlmidi      libadlmidi-git
libadvo libadvo
libadwaita      libadwaita-git
libadwaita-1.so libadwaita-git

It might make sense to remove targets provided by repo packages from this output:

datamash -W check < shlibs  # check for malformed packages
cut -f1 shlibs | aur repo-filter --sync > shlibs_repo
grep -wvxf shlibs_repo shlibs > shlibs_aur

repo-filter is however much slower than format 🤔

Benchmark 1: cut -f1 shlibs_all | aur repo-filter --sync
  Time (mean ± σ):     17.253 s ±  0.659 s    [User: 17.452 s, System: 0.689 s]
  Range (min … max):   16.292 s … 18.378 s    10 runs

Benchmark 1: aur pkglist --info --ttl 3600 | aur format -f %nt%Pn --delim=,
  Time (mean ± σ):      1.088 s ±  0.006 s    [User: 0.846 s, System: 0.140 s]
  Range (min … max):    1.079 s …  1.097 s    10 runs

[VorpalBlade]

repo-filter is however much slower than format

Should it not be possible to cache this result for, say, 24 hours? In which case it doesn't really matter. You could set it up as a systemd timer (user one perhaps).

[AladW]

Sure but there should be faster alternatives available. For example:

1. Filter out any provides which already have an AUR pkgname with aur-pkglist.
2. Use expac -S '%P' and filter out repository provides by hand.

[AladW]

It took a rewrite in perl to solve this fully: #1041

Any provides specified on the command-line are used transparently. --table also adds a Self line for any target mentioned on the command-line. Demonstration:

# Case 1: primary target specified
$ aur depends bitlbee-discord-git --table
bitlbee-discord-git	bitlbee-discord-git	bitlbee-discord-git	0.4.3-1	Self
bitlbee	bitlbee-discord-git	bitlbee	3.6-2	Depends

# Case 2: primary target and dependency specified
$ aur depends bitlbee-discord-git bitlbee --table
bitlbee	bitlbee-discord-git	bitlbee	3.6-2	Depends
bitlbee	bitlbee	bitlbee	3.6-2	Self
bitlbee-discord-git	bitlbee-discord-git	bitlbee-discord-git	0.4.3-1	Self

# Case 3: primary target and provider specified
$ aur depends bitlbee-libpurple-git bitlbee-discord-git --table
bitlbee-discord-git	bitlbee-discord-git	bitlbee-discord-git	0.4.3-1	Self
bitlbee-libpurple-git	bitlbee-discord-git	bitlbee-libpurple-git	3.5.1.r80.gc3599e03-1	Depends
bitlbee-libpurple-git	bitlbee-libpurple-git	bitlbee-libpurple-git	3.5.1.r80.gc3599e03-1	Self

Additional logic makes it works with providers that don't have an existing pkgname. Demonstration:

$ diff -u <(aur depends --table yuzu | sort -u) <(aur depends --table yuzu cubeb-git | sort -u)
--- /dev/fd/63	2022-11-28 15:43:32.151803449 +0100
+++ /dev/fd/62	2022-11-28 15:43:32.151803449 +0100
@@ -1,7 +1,12 @@
+celt	jack	celt	0.11.3-5	MakeDepends
 cpp-httplib-compiled	yuzu	cpp-httplib-compiled	0.11.2-1	Depends
 cpp-httplib-compiled	yuzu	cpp-httplib-compiled	0.11.2-1	MakeDepends
+cubeb-git	cubeb-git	cubeb-git	0.2.r1395.g28c8aa4-1	Self
+cubeb-git	yuzu	cubeb-git	0.2.r1395.g28c8aa4-1	Depends
+cubeb-git	yuzu	cubeb-git	0.2.r1395.g28c8aa4-1	MakeDepends
 dynarmic	yuzu	dynarmic	6.3.1-1	Depends
 dynarmic	yuzu	dynarmic	6.3.1-1	MakeDepends
+jack	cubeb-git	jack	0.126.0-5	MakeDepends
 xbyak	dynarmic	xbyak	6.66-1	MakeDepends
 xbyak	yuzu	xbyak	6.66-1	MakeDepends
 yuzu	yuzu	yuzu	mainline.0.1245-1	Self

Note that #1041 does not update aur-sync to account for aur-depends changes yet. This should be a routine change though.

[CyberShadow]

Sweet, nice work!

[AladW]

Note that #1041 does not update aur-sync to account for aur-depends changes yet. This should be a routine change though.

This was also implemented in #1041