How best to make aurutils' repo only contain exactly a given list of packages and their dependencies?

[CyberShadow]

Background: My aurutils repo has accumulated lots of packages that take a lot of time to build (when using aur sync -u) which I never even use. I already use a configuration manager to manage the list of installed packages, so I would like to integrate that with aurutils so that the set of AUR packages installed on any machine is the same set of packages that are built (plus build dependencies).

So, the goal is to make the repo consist of the exact set of packages that are "needed", i.e. packages that are in some list plus their dependencies (runtime and build-time). aur-sync should then never build a package without an immediate reason to do so.

The approach I'm currently experimenting with is:

1. Create the desired list of root packages.
2. Nuke and recreate a temporary repo, to be used by aur-sync. The repo should be in the same directory as the permanent aurutils repo, which pacman will fetch from.
3. Run aur-sync, pointing it at the temporary repo, and instead of -u, give it the desired list of packages its the command line.
   What enables this to work is that aur-build will skip files which already exist. lib/aur-build: Add existing packages to the database when skipping them #839 improves on that and makes these files available as dependencies for the following builds.
4. Should aur-sync complete successfully, promote the temporary repo to a permanent one, and make it available for pacman -Sy etc.

Though I ran into a few complications, I mostly got this working. Is there perhaps a simpler way to achieve this?

As I understand, in NixOS you get this automatically, as package building is integrated with the package manager, and the system configuration pulls in the exact set of packages that need to be built.

[maximbaz]

I use a combo of things, maybe you'll find some of these ideas useful to you:

• paccache -vruk0 -c /var/cache/pacman/your-aurutils-repo from pacman-contrib will delete cache files from packages that are no longer installed on your system (entry will still be present in the repo)
• repoctl update will delete entries from the repo for which the cache files do not exist (i.e. cleaned up by the command above)
• you can make a shell alias that removes a package via pacman and also removes it from the repo

as long you keep your repo clean (using shell alias or repoctl), aurutils will not try to build that package anymore 🙂

[AladW]

aur sync -u forces you into an all-or-nothing approach. It's probably best to replace it with your own custom logic. If you have a list of packages you want, that could be:

xargs -a my-package-list.txt aur sync <args>

To keep the repository and your list in lockstep, I'd suggest taking complements. Recreating the repository every time isn't going to scale well - repo-add is too slow for that. For example:

grep -Fxvf my-package-list.txt <(aur repo --list | cut -f1) | xargs -r repoctl rm

[AladW]

I've played around with aconfmgr and I think I get the idea better now. I guess it would look something like

AddRepoPackage custom foo1
AddRepoPackage custom foo2
...

A while ago I made a repoctl replacement to handle these kind of tasks and more: https://github.com/AladW/repoutils

repoutils is in a very early stage, but I believe it'd make more sense to address your issue there.

[AladW]

An updated version using aur-view of the script I posted on IRC.

#!/bin/bash
set -e
packages_argv=("$@")

# 0. Create a scratch space and switch to it
tmp=$(mktemp -d)
cd "$tmp"

# 0. Retrieve local repository information
# Adjust accordingly if more than one local repository are used.
{ read -r repo
  read -r root
  read -r path
} < <(aur repo --status)
wait $!

# 1. Resolve the dependency tree of given packages
# Can be either done locally (with a custom set of PKGBUILDs) or
# remotely (for AUR-published PKGBUILDS)
aur depends --pkgname --no-checkdepends "${packages_argv[@]}" | tee queue

# 2. Avoid concurrent use
(
    flock -n 200 || exit 1

    # 3. Operate on copy of existing database
    db_file=$(basename -- "$path")
    cp -v "$path" "$db_file"

    # 4. Remove packages which are not in the dependency tree
    aur repo -d "$repo" --list | grep -Fxvf queue | while read -r pkg; do
        repo-remove "$path" "$pkg"
    done

    # 5. Generate pacman.conf with temporary database
    # This is made with host builds in mind, container builds should
    # be adjusted accordingly.
    { printf '[options]\n'
      pacconf --raw --options

      pacconf --repo-list | grep -Fxv "$repo" | while read -r repo_static; do
          printf '[%s]\n' "$repo_static"
          pacconf --raw --repo="$repo_static"
      done

      printf '[%s]\n' "$repo"
      printf 'Server = %s\n' "$PWD/$db_file"
      # Include other options as desired
    } > pacman.conf

    # 6. Retrieve AUR repositories
    # It is advised to do this in a different directory than /tmp,
    # because this is where the build process will take place.
    AURDEST=${XDG_CACHE_HOME:$HOME/.cache}/aurutils/sync
    cd "$AURDEST"
    aur fetch - < queue

    # 7. Inspect build files and diffs (viewed commits are stored in
    # $XDG_CONFIG_HOME/aurutils/view) in a temporary directory
    aur view -a queue

    # 8. If we arrive here, the user has confirmed has confirmed the
    # package files are OK, and we can start the build process.
    aur build -a queue --pacman-conf="$tmp/pacman.conf" --noconfirm

    # 7. If we arrive here, the repository is in the desired
    # state. Copy it over, with a backup just in case.
    cp -bv "$PWD/db_file" "$path"
    
) 200> /tmp/lockfile

Feedback welcome.

[AladW]

Any news on the script above? I don't know how or if to incorporate this in an existing script, but it might be a possible workflow to document.

[AladW]

I've added the script and a description to examples. If there's new ideas, feel free to propose them.

[AladW]

Here's a fun idea, tar --from-files:

$ cat list.txt
aconfmgr-git
alatools
apulse
archutils-git
astroid-git

# scratch space
tmp=$(mktemp -d)
# select db entries matching list.txt
tar -xvf /var/lib/aurutils/alad/alad.db.tar.gz -C "$tmp" --from-files <(aur repo --list | grep -Fwf list.txt | cut -f1,2 --output-delimiter=-)
# create new archive
cd "$tmp"
tar czvf alad-new.db.tar.gz *

$ tar -tvf alad-new.db.tar.gz
drwxr-xr-x archie/archie     0 2022-03-09 16:14 aconfmgr-git-r799.023ff64-1/
-rw-r--r-- archie/archie   706 2022-03-09 16:14 aconfmgr-git-r799.023ff64-1/desc
drwxr-xr-x archie/archie     0 2022-02-09 16:57 alatools-2018.12-3/
-rw-r--r-- archie/archie   449 2022-02-09 16:57 alatools-2018.12-3/desc
drwxr-xr-x archie/archie     0 2022-02-09 16:57 apulse-0.1.13-1/
-rw-r--r-- archie/archie   465 2022-02-09 16:57 apulse-0.1.13-1/desc
drwxr-xr-x archie/archie     0 2022-01-27 12:48 archutils-git-1:156.2201271248-1/
-rw-r--r-- archie/archie   682 2022-01-27 12:48 archutils-git-1:156.2201271248-1/desc
drwxr-xr-x archie/archie     0 2022-02-18 11:31 astroid-git-v0.16.r1.g4d6b06a-1/
-rw-r--r-- archie/archie   950 2022-02-18 11:31 astroid-git-v0.16.r1.g4d6b06a-1/desc

No more running repo-remove in a loop 🎉

I realize there's some other things in the example script that need addressing - e.g. only the repository is copied, but the existing packages remain in the repository root. Symlinks might work (or just operate in the original directory).

To help manipulating package files, I plan to introduce aur repo --field:

$ aur repo --field filename
aconfmgr-git-r799.023ff64-1-any.pkg.tar.zst
alatools-2018.12-3-any.pkg.tar.zst
apulse-0.1.13-1-x86_64.pkg.tar.zst
archutils-git-1:156.2201271248-1-any.pkg.tar.zst
astroid-git-v0.16.r1.g4d6b06a-1-x86_64.pkg.tar.zst
...

[AladW]

6bf2986 but note #964 (comment)