Ability to create multiple repositories for cpu specific optimized packages

[recover-sanity]

Feature summary

Currently a single Ahriman instance (to my knowledge) only allows you to make a single repository. Not only that it only allows you to specify a build command for different architectures e.g x86-64, i686, ARM, RISC-V, etc. What I'd like to request is the ability to make multiple repositories that specifically use a custom build command and or makepkg.conf to allow building of cpu specific optimized packages e.g x86-64-v3, zen3, skylake.

Cause of the feature request

I have multiple systems running Arch running on various cpus. I currently have my package build system (TeamCities + bash scripts) build optimized packages for these systems (zen3, skylake, sandybridge, x86-64-v2) and push them to multiple repositories (repo-zen3, repo-skylake, repo-sandybridge, repo-x86-64-v2). Ahriman doesn't appear to support this as it can only do one repository and only allows you to specify the architecture (x86, arm, risc-v).

Proposed changes and/or features

Allow multiple repositories each with their own name e.g [repository:MY-x86-64-REPO]

Move [build] settings to inside of [repository:NAME]

Potentially? allow use of a user specified makepkg.conf for each repository e.g makepkg_conf=/path/to/makepkg.conf

[arcan1s]

Having an ability to specify repository name might be good feature for case when you are using it for managing own disto 🤔 However, I didn't really get use case of having same repository with architectures as suffixes. Back in time when arch supported i686, repositories had the same name (as well as in case of other distros - mostly)

Potentially? allow use of a user specified makepkg.conf for each repository e.g makepkg_conf=/path/to/makepkg.conf

oh, sadly at the moment I'm locked with devtools features (as I don't really want yet to write own alternative). IIRC there were some good feature requests to pkgctl which can be adopted later, but so far it is very behind in features than current scripts.

Nevertheless, devtools supports separated makepkg.conf's, however, it is impossible to specify their locations. I did some research some time ago https://ahriman.readthedocs.io/en/latest/faq.html#non-x86-64-architecture-setup, see p2 in physical server setup chapter.

[recover-sanity]

I didn't really get use case of having same repository with architectures as suffixes.

I probably could've worded it better but my idea was to have [repository:NAME] be any name not just an architecture or optimization. One could have [repository:farts] with it's own custom build settings if they so desired.

Potentially? allow use of a user specified makepkg.conf for each repository e.g makepkg_conf=/path/to/makepkg.conf

oh, sadly at the moment I'm locked with devtools features (as I don't really want yet to write own alternative). IIRC there were some good feature requests to pkgctl which can be adopted later, but so far it is very behind in features than current scripts.

Nevertheless, devtools supports separated makepkg.conf's, however, it is impossible to specify their locations. I did some research some time ago ahriman.readthedocs.io/en/latest/faq.html#non-x86-64-architecture-setup, see p2 in physical server setup chapter.

Oh I didn't realise that. Thank you for mentioning this.

[arcan1s]

I see. Let me check how it can be implemented without significant reworking and/or breaking changes

[arcan1s]

changes have been merged into master. But I'm going to test it additionally on my repository first

[recover-sanity]

Thanks for working on this! However, would it be possible for the web interface to support multiple repositories? Currently using the Docker container and when trying to run the web interface with multiple repos I'm getting: ahriman.core.exceptions.MultipleArchitecturesError: Multiple architectures/repositories are not supported by subcommand web, got [RepositoryId(architecture='x86_64', name='my_x86_64_test'), RepositoryId(architecture='x86_64', name='my_zen3_test')] Only other option I can see is to have multiple web interfaces running for each and every repository which is annoying.

[arcan1s]

let me check how it can be implemented

[arcan1s]

sorry bg3 took longer than i probably expected, I'm still working on your case, but so far it is mostly implemented, just need more checks, test coverage, docs update etc

[recover-sanity]

Thanks for the reply! I've built the docker image from the latest commit on the feature/multi-web branch and I'm facing a strange issue. Unless I'm doing something wrong it seems that while the new web interface does show all available repositories it does not seem to work for building in other repositories besides the first one in the list. E.g I run the docker image with: docker run -it --rm --privileged --net=host -e AHRIMAN_PORT=6783 -e AHRIMAN_HOST=192.168.1.32 -e AHRIMAN_REPOSITORY_ROOT=/var/lib/ahriman/ahriman -e AHRIMAN_REPOSITORY="my_x86_64_test" -v $PWD/data/repo:/var/lib/ahriman -v $PWD/config/ahriman.ini:/etc/ahriman.ini localbuild/ahriman:latest --log-handler=console --repository "my_zen3_test" --repository "my_sandybridge_test" web and instead of being able to add packages and build them on all three repositories it only works on the first one in the list (my_x86_64_test) trying it on the others however results in this error message:

[ERROR 2023-09-29 23:12:16,484] [ahriman.core.repository.repository.Repository]: cpufetch-git (x86_64) build exception
Traceback (most recent call last):
  File "/usr/lib/python3.11/configparser.py", line 805, in get
    value = d[option]
            ~^^^^^^^^
  File "/usr/lib/python3.11/collections/__init__.py", line 1006, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/collections/__init__.py", line 998, in __missing__
    raise KeyError(key)
KeyError: 'build_command'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/ahriman/core/repository/executor.py", line 100, in process_build
    build_single(single, Path(dir_name), packager.packager_id)
  File "/usr/lib/python3.11/site-packages/ahriman/core/repository/executor.py", line 83, in build_single
    task = Task(package, self.configuration, self.architecture, self.paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ahriman/core/build_tools/task.py", line 67, in __init__
    self.build_command = configuration.get("build", "build_command")
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/configparser.py", line 808, in get
    raise NoOptionError(option, section)
configparser.NoOptionError: No option 'build_command' in section: 'build' 

If you need more logs I can try to provide some more. If I've messed something up feel free to let me know :-)

[arcan1s]

Unless I'm doing something wrong it seems that while the new web interface does show all available repositories it does not seem to work for building in other repositories besides the first one in the list

In current implementation there are tabs above, which controls in which repository actions will be run (the only exception is pgp key dialog which is common for any repo). I'm thinking about adding dialog (at least for package addition) to choose in which repositories it has to be run

docker run -it --rm --privileged --net=host -e AHRIMAN_PORT=6783 -e AHRIMAN_HOST=192.168.1.32 -e AHRIMAN_REPOSITORY_ROOT=/var/lib/ahriman/ahriman -e AHRIMAN_REPOSITORY="my_x86_64_test" -v $PWD/data/repo:/var/lib/ahriman -v $PWD/config/ahriman.ini:/etc/ahriman.ini localbuild/ahriman:latest --log-handler=console --repository "my_zen3_test" --repository "my_sandybridge_test" web

the --repository options and AHRIMAN_REPOSITORY are redundant for web service atm btw. It scans filesystem to create tree instead

trying it on the others however results in this error message

another issue which is not yet addressed is that setup command doesn't work in this case correctly. It generates configuration only for one repository which leads to inability to run actions on different architectures. It will be fixed before merge, however, it is possible to pass own configuration directly for all available repositoriies

[recover-sanity]

the --repository options and AHRIMAN_REPOSITORY are redundant for web service atm btw. It scans filesystem to create tree instead

Good to know

another issue which is not yet addressed is that setup command doesn't work in this case correctly. It generates configuration only for one repository which leads to inability to run actions on different architectures. It will be fixed before merge, however, it is possible to pass own configuration directly for all available repositories

Glad to hear it'll be fixed. Excited to see this working.

[arcan1s]

I've merged another mr and updated [https://ahriman-demo.arcanis.me/](demo page) accordingly. Also I wrote small example of how-to in docker (the difference is in AHRIMAN_POSTSETUP_COMMAND/AHRIMAN_PRESETUP_COMMAND variables).

Basically differences from the original implementation are the following:

• only single --repository and --architecture flags are allowed now (consider them as filters)
• web subcommand itself doesn't require architecture/repo specification and its configuration is global (though docker still requires it, because it also performs setup command)
• any "additional" repository for web in docker should be mentioned as setup command in AHRIMAN_POSTSETUP_COMMAND variable (; separated as usual in shell)

So considering example you posted above, the command should be something like

docker run -it --rm --privileged --net=host -e AHRIMAN_PORT=6783 -e AHRIMAN_HOST=192.168.1.32 -e AHRIMAN_REPOSITORY_ROOT=/var/lib/ahriman/ahriman -e AHRIMAN_REPOSITORY="my_x86_64_test" -e AHRIMAN_OUTPUT=console -e AHRIMAN_POSTSETUP_COMMAND="ahriman --architecture x86_64 --repository my_zen3_test --log-handler console service-setup --build-as-user ahriman --packager 'ahriman bot <ahriman@example.com>'" -v $PWD/data/repo:/var/lib/ahriman -v $PWD/config/ahriman.ini:/etc/ahriman.ini localbuild/ahriman:latest web

[recover-sanity]

Awesome! It appears to be working so far 😄 . One thing I can't seem to figure out is how to add a package to a specific repository via repo-pull. Following the FAQ I've added a remote git repo to pull PKGBUILDs from but there doesn't seem to be a way to add a specific package to a specific repository. It seems to just add all package to all repositories.

And one more thing (I'm sorry if I'm asking for too much) Would it be possible in the future to somehow build a specific package using a specific makepkg.conf or some other way of changing the build flags like the previously mentioned Devtools? I've been playing with compiler optimizations with Clang and I've found that while most packages would compile with certain optimizations others would fail. Ideally one could change the settings per package to fix the ones that fail.

[arcan1s]

One thing I can't seem to figure out is how to add a package to a specific repository via repo-pull. Following the FAQ I've added a remote git repo to pull PKGBUILDs from but there doesn't seem to be a way to add a specific package to a specific repository. It seems to just add all package to all repositories.

there was a "feature" which has been fixed in 158d01a, which automatically add packages to repository from local cache. Because cache is shared between repositories (which is mostly correct), it would attempt to add packages in all repositories during the next check

Would it be possible in the future to somehow build a specific package using a specific makepkg.conf or some other way of changing the build flags like the previously mentioned Devtools?

mhm, there are some features which makepkg supports ,but devtools - doesn't. E.g. it is possible to pass settings via environment variables, but devtools only allows whitelisted ones. Also as I've mentioned above, it is impossible too specify config path.

Let me check it more closely though, I've tried some naive hacks, but none of them works sadly

[arcan1s]

Would it be possible in the future to somehow build a specific package using a specific makepkg.conf or some other way of changing the build flags like the previously mentioned Devtools?

aight, here you go:

[root@9af2c52041d3 /]# sudo -u ahriman ahriman patch-add ix PKGEXT
Post new function or variable value below. Press Ctrl-D to finish:
.pkg.tar.xz

and then

[root@9af2c52041d3 /]# sudo -u ahriman ahriman --log-handler console -r aur-clone --force add ix --now
[DEBUG 2023-10-22 19:11:04,457] [root]: using console_handler logger
...
[DEBUG 2023-10-22 19:11:17,775] [ahriman.core.alpm.repo.Repo]: ==> Adding package '/var/lib/ahriman/ahriman/repository/aur-clone/x86_64/ix-1.0-1.18-any.pkg.tar.xz'
...

PKGEXT is just an example of variable from makepkg, which is quite easy to test. However, in your context it is required to add (optional) repository id for patches. And ma-a-aybe add ability to use "shadow" patches (as in case of pkgrel bump), but it breaks transparency (c) and reproducible builds (tm)

[arcan1s]

soo, I've added ability to apply simple patches with package-add command (via --variable), which covers the case you described. Also I've added some small patch controls to web forms (you can add variables during package addition or remove them from package info modal). Also I've added some more information and controls to package info modal (you can see some WIP screenshots in #115)

Upd: demo site has been updated as well to the latest master commit

[arcan1s]

the feature has been also released. I'm closing this issue, feel free to reopen and/or create another one in case of feedback/additional changes required.

[recover-sanity]

Awesome, Thank you.