Support or documentation for non-AUR packages

[dhouck]

[Suggested label: question and/or enhancement]

This is a great tool for building Arch packages and putting them in a repository for Pacman. Despite the name, this is useful for more than just the AUR; is there a way to use most of this same infrastructure for non-AUR packages? As far as I can tell, there is not, but primarily because aur-depends is called unconditionally by aur-sync and I have to drop down to aur-build, and do things like git pulling the packages manually.

As a specific example, both for concreteness and to forestall the “just submit it to the AUR” objection:

I like the Terminus font, which is available as a package in community/terminus-font. However, it supports a number of variant character appearances, and I like a few of the options more than the defaults. I could do what several other people have done and upload a custom configuration to the AUR, but nobody else would use it and it would be harder to make sure I stayed in sync with upstream, and wouldnʼt actually make things any easier.

So unless I end up switching to Gentoo where the equivalent package has USE flags for this kind of thing, Iʼm better off installing packages that donʼt come from the AUR. But Iʼd still like to use as much of this as still applies, which is a lot.

Before filing a bug, check the following:

1. Do you use the latest release of aurutils and its dependencies? If so,
   is the issue reproducible from the master branch? Yes
   Other questions deleted as not relevant

__If yes to all, create a debug log:__with

AUR_DEBUG=1 aur <name> <arguments> >aurutils.log

N/A

[AladW]

As far as I can tell, there is not, but primarily because aur-depends is called unconditionally by aur-sync and I have to drop down to aur-build, and do things like git pulling the packages manually.

You don't need to git pull manually, because there's aur-fetch which aur sync uses as well. The only unique aspect about aur sync is its file inspection mechanism. Since that one is already decoupled, I planned to move it to a seperate script (aur-view).

As a specific example, both for concreteness and to forestall the “just submit it to the AUR” objection:

One specific aim of the modular nature of aurutils is that you can use it for other tasks than just the AUR, so I don't object.

[dhouck]

You don't need to git pull manually, because there's aur-fetch which aur sync uses as well.

Ah, I thought aur-fetch was also AUR-specific, but that seems to be only for cloning new packages, , but it looks like thatʼs controlled by a flag, so Iʼll try that next time.

So it looks like my workflow for modified official packages would be

1. asp update
2. aur fetch all my non-AUR packages
3. Parse the results file to figure out what changed, check for rebase conflicts
4. Review any untrusted PKGBUILDs (to be moved to aur view)
5. If everything looks good, pass the results to aur-build

I guess only steps 2, 4, and 5 are the same between this and the AUR workflow, since aur sync uses AUR information to determine which packages have updates, before it does aur fetch, so just adding an option to entirely skip aur depends in aur sync wouldnʼt simplify the workflow that much.

[AladW]

Ah, I thought aur-fetch was also AUR-specific, but that seems to be only for cloning new packages, , but it looks like thatʼs controlled by a flag, so Iʼll try that next time.

Well aur-fetch tries to git clone if it didn't find a git repository for your package. It could gain a flag --clone-url to point the URL to https://github.com/archlinux/svntogit-{packages,community}, but you'd have to handle the distinction between packages and community (asp does that already). Another option would be --existing to disable git clone entirely.

About 3, the official repositories has a limited API to check for package updates: https://wiki.archlinux.org/title/Official_repositories_web_interface
You could make your own wrapper for that (or use kittypack) and pass the results to aur vercmp. Though not all PKGBUILD changes result in a pkgver or pkgrel increase, so your approach of parsing the results file may be more reliable.

About 4, if PKGBUILDs are strictly from the official repositories, they have a trusted nature and it's not strictly needed to review them. Of course aur-view can come in handy to review diffs, when rebasing changes.

[dhouck]

Yeah, I agree that I could simplify the procedure for official repositories. I donʼt see myself using non-AUR non-official PKGBUILDs Iʼm not completely writing myself, so Iʼd probably skip some of these steps.

[AladW]

Did you come up with an example script? My plan is to add one to the examples directory, including a short description.

[AladW]

I've come up with a sample script (see below the horizontal line). I would like input on some points:

1. Save custom commits in master or in a separate branch, and whether to view build files before or after git-rebase.
2. How to best avoid checksum mismatches between modified and original package. When naming the packages differently, pkgrel~suffix compares higher than pkgrel, but lower than pkgrel+1 (and the modified package would be automatically pulled in by pacman -Syu, regardless of ordering of repositories in pacman.conf).

The general workflow is similar to #860 (comment).

---

#!/bin/bash
set -e
argv0=sync-arch
XDG_CACHE_HOME=${XDG_CACHE_HOME:-$HOME/.cache}
XDG_DATA_HOME=${XDG_DATA_HOME:-$HOME/.local/share}
ASPROOT=${ASPROOT:-$XDG_CACHE_HOME/asp}
ARCH_VIEW_DB=${ARCH_VIEW_DB:-$XDG_DATA_HOME/aurutils/view_arch}

# Scratch space for intermediary results.
tmp=$(mktemp -d --tmpdir "aurutils-$UID/$argv0.XXXXXXXX")
trap 'rm -rf "$tmp"' EXIT

# The source directory is assumed to have the following layout:
# .
# ├── paraview
# |   ├── .git
# │   ├── repos
# │   │   └── community-x86_64
# │   │       ├── ...
# │   │       └── PKGBUILD
# │   └── trunk
# │       ├── ...
# │       └── PKGBUILD
# └── ...
cd "$1"

# Retrieve new upstream revisions (FETCH_HEAD) and display them with
# aur-view(1). Checkouts have to be updated individually.
mapfile -t targets < <(asp list-local) # get all tracked packages
asp update # update ASPROOT/cache
aur fetch "${targets[@]}" --existing --results "$tmp"/fetch_results # update */{trunk,repos}

# If the changes are approved (exit 0), write FETCH_HEAD to a file in
# ARCH_VIEW_DB. Note that build files are shown at their previous
# revisions; they are merged in the next step.
mkdir -p "$ARCH_VIEW_DB"
env AUR_VIEW_DB="$ARCH_VIEW_DB" -C \
    aur view "${repo_targets[@]}" --revision FETCH_HEAD

targets_new=()
while IFS=: read -r mode rev_old rev path; do
    path=${path#file://} name=${path##*/}

    # A package is considered a target if HEAD differs from FETCH_HEAD.
    if [[ $mode == "fetch" ]] && [[ $rev != "$rev_old" ]]; then
        targets_new+=("$name")
    fi
done <"$tmp"/fetch_results

# Rebase custom changes. Assumes custom changes are done as commits to the
# master branch.
# XXX: 1. check if this sets FETCH_HEAD == HEAD
#      2. do this in a custom branch (or keep a different branch for upstream)
for t in "${targets_new[@]}"; do
    env -C "$t" git rebase --verbose
done

# Build packages in dependency order. `printsrcinfo` is a small helper
# script which uses `libmakepkg` to write .SRCINFO. This skips any
# linting checks done by makepkg to increase performance.
printf './%s/trunk\0' "${targets_new[@]}" \
    | xargs -0 -I{} env -C {} printsrcinfo \
    | aur graph | tsort | tac | awk '{print $0 "/trunk"}' >"$tmp"/queue

# XXX: the modified package has the same version string as the
# original one, which may cause issues if the original package is
# cached. Workaround: use setconf to bump "subpkgrel"
for t in "${targets_new[@]}"; do
    ( cd "$t"; pkgrel=$(pacini PKGBUILD "$pkgrel")
      setconf PKGBUILD pkgrel=$pkgrel~custom
    )
done

# Build in a chroot so that the modified package can be updated in a
# single transaction.
aur build -a "$tmp"/queue --chroot

The used printsrcinfo can be something simple like:

#!/bin/bash
set -e
source /usr/share/makepkg/srcinfo.sh

source_safe PKGBUILD
write_srcinfo

[dhouck]

Iʼll look at this later today; Iʼve been a bit busy recently but it should be calming down soon.

[AladW]

The script above had some typos - those should be fixed now.

[SahAssar]

Could someone perhaps let me know how to use the script above? As in if I have a git url, how to I use the above to build a package to my package repo? I'm guessing I first need to build and install the package manually since it uses asp list-local to find what targets to build, right?

Thanks!

[eclairevoyant]

@SahAssar FYI the script above was meant specifically for modified packages from the official repos. Also, the script will need updating since asp is no longer in use. asp used to keep a list of locally tracked packages from the Arch repos, but with the new tool (pkgctl) I believe that's no longer the case.

If I get some time I might write my own script, since I do keep some PKGBUILDs that aren't uploaded to the AUR.