Add option to use pkgctl build instead of makepkg or similar

[ultimatespirit]

Bit of an enhancement thing, more than a few times you'll run into some AUR package that gets really finicky with regards to what's on your system. For those cases, or just in general for better sandboxing, it's really convenient to be able to use the relatively new system of pkgctl build to automatically create a proper sandbox within which solely the dependencies for the build get installed, and build the package there.

Simply having an option to use pkgctl would be a nice improvement for those edgecases, even better if it's able to store a configuration option saying "hey package foo needs pkgctl" to use during the next aura -Akuax operation.

If aura could go further and also handle more complicated cases like chained AUR dependencies, that could be really cool too. I.e. if AUR package foo depends upon package bar and it's available both in the main repos and the AUR (or only in the AUR), as far as I know you need to explicitly include the already built package bar to pkgctl build with the -I... option. So if this hypothetical option were able to make aura build bar first with pkgctl and then foo with bar included that would be really cool.

[bb010g]

(pkgctl build was introduced in devtools commit 1d7f997.)

[fosskers]

Thanks for the suggestion guys, I've been out of town and this is the first I've heard of pkgctl.

[fosskers]

I just gave it a shot and I think this is a good development. As you suggested, I will start by adding support for this in config only, as a list of packages that should be built via pkgctl build. Since it takes a while to set up the chroot, turning it on for all packages might not be what the user wants. However, if people ask for it later, I'll consider such an option as well.

[bb010g]

Since it takes a while to set up the chroot, turning it on for all packages might not be what the user wants.

I know aurutils manages to set up chroots pretty quickly, but it also doesn't use pkgctl build. aur build --chroot uses its sibling aur chroot instead, which manually invokes mkarchroot(1) (on aur chroot --create), arch-nspawn(1) (on aur chroot --update), and makechrootpkg(1) (on aur chroot --build).

I don't use Arch often anymore, but I want to note that aurutils's chroot building has been consistently reliable every time I've used it. (I don't have experience with pkgctl build to compare to.) If you want to offer more optimized chroot builds alongside pkgctl build, I'd highly recommend reimplementing aurutils's chroot logic in Aura.

[fosskers]

pkgctl build does the chroot building itself.

[ultimatespirit]

Since it takes a while to set up the chroot, turning it on for all packages might not be what the user wants.

I haven't tested it with a lot of builds, but with the few that I have done (a few rebuilds of packages / set of packages), it seemed like your very first pkgctl build would spend a while doing chroot setup, but then all following builds would be able to much more quickly setup the chroot. Basically taking only as long as it takes to install the required packages that package needed to build. So just in case you're referring to the first setup period, that cost seems to be a one time type. If you're referring to the general setup of required packages, yea that's basically unavoidable I think, and in general understandable for it to at most be some option a user could turn on (with some accompanied warnings about level of overkill / setup time probably).

I guess for some cases you could have it do the slightly dirty "keep reusing the chroot" method, like what's listed here: https://wiki.archlinux.org/title/DeveloperWiki:Building_in_a_clean_chroot#Handling_major_rebuilds

Though that's probably more work than it's worth...

[bb010g]

pkgctl build (ignoring pkgctl build --offload) actually hands off the chroot building to an archbuild(1) command (source), which ends up also calling mkarchroot, arch-nspawn, and makechrootpkg. So, performance once pkgctl build hits e.g. extra-x86_64-build should be about the same as aur chroot --create --update --build.

@fosskers, I'd be curious how aur build performs for you compared to pkgctl build. AFAICT, both of these options shouldn't be much slower than building outside of a chroot, save for the time spent updating the chroot. The biggest slowdown I could see pkgctl build introducing is via pkgctl build --clean, which calls ${pkgrepo}${arch:+-}${arch:-}-build --clean, which would require running mkarchroot for each build. I'm actually curious how aur build avoids leaving installed packages around between builds now, because that seems possible if you're just running aur chroot or archbuild without --clean. I'm hopefully missing something that automatically uninstalls irrelevant packages, or reverts chroot state. EDIT: makechrootpkg(1) takes care of that in sync_chroot() on makechrootpkg -c.

See also for aurutils's rationale behind aur chroot instead of archbuild:

• scripts: readd aur-chroot aurutils/aurutils#655

EDIT 2: archbuild(1) always calls mkchrootpkg(1) with at least -c -n -C (clean the chroot before building, run namcap(1) on the build package, run checkpkg(1) on the build package), so I believe Aura should not call pkgctl build --clean in non-exceptional circumstances. Initial build times will be longer, but subsequent builds should be pretty quick.

[fosskers]

Thanks for the follow-up guys. Indeed I only tried it once - subsequent chroot setups might have been much faster. I will begin implementing this tomorrow.

[bb010g]

Notes on the linked aur chroot rationale:

This is a compromise between the previous aur-chroot and a full delegation to archbuild. In particular, a dedicated pacman.conf is used for the chroot (defaulting to /usr/share/devtools/pacman-extra.conf, with extra as a configurable prefix) with no additional processing.

Here are aur chroot's current pacman.conf(5) and makepkg.conf(5) candidate file lists:

# The pacman configuration in the chroot may contain a local repository that
# is not configured on the host. Therefore, $db_name is only used for the
# default paths below when specified on the command-line or through `AUR_REPO`.
if [[ -v suffix ]]; then
    default_pacman_paths=("$etcdir/pacman-$suffix.conf"
                          "$etcdir/pacman-$machine.conf"
                          "$shrdir/pacman.conf.d/$suffix.conf"
                          "$shrdir/pacman.conf.d/aurutils-$machine.conf")

    default_makepkg_paths=("$etcdir/makepkg-$suffix.conf"
                           "$etcdir/makepkg-$machine.conf"
                           "$shrdir/makepkg.conf.d/$suffix.conf"
                           "$shrdir/makepkg.conf.d/$machine.conf")

The first candidate that exists is used. Also note that aurutils used to support $shrdir/pacman.conf.d/aurutils.conf, but that support was removed in 2023-09. Compare to archbuild(1)'s current behavior (where @pkgdatadir@ becomes /usr/share/devtools):

pacman_config="@pkgdatadir@/pacman.conf.d/${repo}.conf"
if [[ -f @pkgdatadir@/pacman.conf.d/${repo}-${arch}.conf ]]; then
    pacman_config="@pkgdatadir@/pacman.conf.d/${repo}-${arch}.conf"
fi
makepkg_config="@pkgdatadir@/makepkg.conf.d/${arch}.conf"
if [[ -f @pkgdatadir@/makepkg.conf.d/${repo}-${arch}.conf ]]; then
    makepkg_config="@pkgdatadir@/makepkg.conf.d/${repo}-${arch}.conf"
fi

Support for chroot-specific configs was committed on 2019-08-09 and first released in 20190821, so I'm not sure why Alad brought this up as a difference, aside from aurutils's deprecated aurutils.conf. I think archbuild's current behavior is reasonable for an AUR helper, if the helper documents it.

aur-chroot differs as follows from archbuild:

• Include directives are supported;

I don't see how this is accomplished. Though, I did find that aur chroot detects file:// URLs using pacman-conf --config <file> (which does admittedly resolve Include directives).

# bind mount file:// paths to container (#461)
# required for update/build steps
while read -r key _ value; do
    case $key=$value in
        Server=file://*)
            bindmounts_rw+=("${value#file://}") ;;
    esac
done < <(pacman-conf --config "$pacman_conf")

pkgctl build and archbuild don't do similar scanning, but archbuild does pull mounts from /usr/share/devtools/mount.d/${arch}.

# Architecture-specific Mount
arch_mounts=()
if [[ -f "@pkgdatadir@/mount.d/${arch}" ]]; then
	mapfile -t arch_mounts < "@pkgdatadir@/mount.d/${arch}"
fi
for arch_mount in "${arch_mounts[@]}"; do
	if [[ $arch_mount = rw* ]]; then
		arch_mount=${arch_mount#rw }
		in_array "$arch_mount" "${makechrootpkg_args[@]}" || makechrootpkg_args+=("-d" "$arch_mount")
	elif [[ $arch_mount = ro* ]]; then
		arch_mount=${arch_mount#ro }
		in_array "$arch_mount" "${makechrootpkg_args[@]}" || makechrootpkg_args+=("-D" "$arch_mount")
	fi
done

In any case, that's all specific to bind mounts. I don't see where aur chroot is more robust than archbuild when they both read pacman.conf(5) configuration. As far as I can tell, both tools leave that up to mkarchroot and arch-nspawn.

• makechrootpkg arguments can be overridden from the command line;

Also supported by archbuild, though the initial arguments are always -c -n -C. aur chroot always uses initial arguments of -c -u.

• the distinction between multilib-devel and base-devel is derived from the supplied prefix instead of $0;

I'd hope archbuild relying on $0 won't be an issue in practice? We're also invoking archbuild through pkgctl build, if that's relevant for long-term maintenance.

• the arch-nspawn, mkarchroot and makechrootpkg steps can be used seperately;

This is something archbuild & pkgctl build still don't support. I don't see that being an issue for Aura, but Alad should be able to speak to the utility of this 5 years into current aur chroot.

• both makepkg.conf and pacman.conf can be specified from the command-line.

This is something archbuild & pkgctl build still don't support. I also don't see that being an issue for Aura.

---

I would appreciate @AladW's thoughts on any & all of this.

[AladW]

The main motivation for including a custom aur-chroot instead of directly using *-x86_64-build is that the latter works poorly with local (file://) repositories. If you don't support those, you can use extra-x86_64-build or pkgctl --build directly, although I see no benefit in using pkgctl --build over extra-x86_64-build.

Dependencies then need to be installed manually into the (working copy of the) container with arch-nspawn or makechrootpkg -I.

[fosskers]

See also for aurutils's rationale behind aur chroot instead of archbuild:

I think archbuild's current behavior is reasonable for an AUR helper, if the helper documents it.

I will forego this level of control for Aura and rely on the default behaviour of pkgctl build.

I haven't tested it with a lot of builds, but with the few that I have done (a few rebuilds of packages / set of packages), it seemed like your very first pkgctl build would spend a while doing chroot setup, but then all following builds would be able to much more quickly setup the chroot.

I tried it again this morning, and sure enough subsequent builds are much faster. There is of course the question of the --clean flag which does a full chroot rebuild each time, but perhaps that isn't necessary here?

I think I want the transition to pkgctl build to be gradual. For now, the default will remain makepkg-based building within ~/.cache/aura/builds/, as these are easier to inspect. pkgctl use will be enabled in config for specifically indicated packages. If this goes well, we can consider making it the default for all builds.

[fosskers]

Done and released as Beta 4. Give it a shot boys.