alpine: add backend

[Cogitri]

fixes #73

[Cogitri]

It seems to index the packages now, but crashes during metadata generation with:

object.Exception@../src/asgen/zarchive.d(318): File /usr/lib/libreoffice/share/xdg/math.desktop was not found in the archive /pack/alpine/repo/edge/community/x86_64/libreoffice-math-6.4.3.2-r1.apk.
object.Exception@../src/asgen/zarchive.d(318): File /usr/lib/libreoffice/share/xdg/impress.desktop was not found in the archive /pack/alpine/repo/edge/community/x86_64/libreoffice-impress-6.4.3.2-r1.apk.
object.Exception@../src/asgen/zarchive.d(318): File /usr/lib/libreoffice/share/xdg/base.desktop was not found in the archive /pack/alpine/repo/edge/community/x86_64/libreoffice-base-6.4.3.2-r1.apk.
object.Exception@../src/asgen/zarchive.d(318): File /usr/lib/libreoffice/share/xdg/calc.desktop was not found in the archive /pack/alpine/repo/edge/community/x86_64/libreoffice-calc-6.4.3.2-r1.apk.
object.Exception@../src/asgen/zarchive.d(318): File /usr/lib/libreoffice/share/xdg/draw.desktop was not found in the archive /pack/alpine/repo/edge/community/x86_64/libreoffice-draw-6.4.3.2-r1.apk.
object.Exception@../src/asgen/zarchive.d(318): File /usr/lib/libreoffice/share/xdg/writer.desktop was not found in the archive /pack/alpine/repo/edge/community/x86_64/libreoffice-writer-6.4.3.2-r1.apk.

I guess I'm adding the wrong contents to some package 🤔

Also, turns out I can cope without the dependency on other D packages, so hooray :)

[Cogitri]

Hm, I can't really figure out why I'm getting those errors - I only add those files to the libreoffice-common package, but for some reason it's also searching for them in the specific libreoffice packages.

Libreoffice-common has the following .desktop files:

usr/lib/libreoffice/share/xdg/base.desktop
usr/lib/libreoffice/share/xdg/calc.desktop
usr/lib/libreoffice/share/xdg/draw.desktop
usr/lib/libreoffice/share/xdg/impress.desktop
usr/lib/libreoffice/share/xdg/math.desktop
usr/lib/libreoffice/share/xdg/startcenter.desktop
usr/lib/libreoffice/share/xdg/writer.desktop
usr/lib/libreoffice/share/xdg/xsltfilter.desktop

libreoffice-draw only has one:

usr/share/applications/libreoffice-draw.desktop

The backtrace isn't overly helpful either:

Program terminated with signal SIGABRT, Aborted.
#0  __restore_sigs (set=set@entry=0x7ffe99dfbb40) at ./arch/x86_64/syscall_arch.h:40
40	./arch/x86_64/syscall_arch.h: No such file or directory.
[Current thread is 1 (LWP 30822)]
#0  __restore_sigs (set=set@entry=0x7ffe99dfbb40) at ./arch/x86_64/syscall_arch.h:40
#1  0x00007f805d191b3d in raise (sig=sig@entry=6) at src/signal/raise.c:11
#2  0x00007f805d169719 in abort () at src/exit/abort.c:13
#3  0x00007f805b961fba in _d_throw_exception () at /usr/lib/libdruntime-ldc-shared.so.90
#4  0x00007f805bc8978a in _D3std11parallelism16submitAndExecuteFCQBlQBk8TaskPoolMDFZvZv () at /usr/lib/libphobos2-ldc-shared.so.90
#5  0x00005632ac8c11c3 in _D3std11parallelism__T15ParallelForeachTAC5asgen8backends10interfaces7PackageZQCe7opApplyMFMDFKQCcZiZi (this=0x7f6059831800, dg=...) at /usr/include/d/std/parallelism.d:4088
#6  0x00005632ac8c10a9 in _D5asgen6engine6Engine15processPackagesMFKACQBq8backends10interfaces7PackageCQCx8handlers11iconhandler11IconHandlerZv (this=0x7f805a610380, pkgs=..., iconh=0x7f605426d8f0) at engine.d:153
#7  0x00005632ac8c57b4 in _D5asgen6engine6Engine19processSuiteSectionMFSQBs6config5SuitexAyaCQCn15reportgenerator15ReportGeneratorZb (this=0x7f805a610380, suite=..., section=..., rgen=0x7f805a613200) at engine.d:674
#8  0x00005632ac8c6da6 in asgen.engine.Engine.run(immutable(char)[]) (this=0x7f805a610380, suiteName=...) at engine.d:832
#9  0x00005632ac843265 in app._main(immutable(char)[][]) (args=...) at app.d:205
#10 0x00007f805b9619ac in _D2rt6dmain212_d_run_main2UAAamPUQgZiZ6runAllMFZv () at /usr/lib/libdruntime-ldc-shared.so.90
#11 0x00007f805b96178a in _d_run_main2 () at /usr/lib/libdruntime-ldc-shared.so.90
#12 0x00007f805b9615ee in _d_run_main () at /usr/lib/libdruntime-ldc-shared.so.90
#13 0x00005632ac842e3f in main (argc=5, argv=0x7ffe99dfda88) at app.d:102

line 153 is foreach (ref pkg; parallel (pkgs)) {, why would it look for the archive at that point (or is that the backtrace being weird because it does things in parallel?)

[ximion]

I think what is going on here is that the generator follows a symlink from the /usr/share/applications/ directory to a different location in the same package, but that doesn't exist.
Symlinks in any directory are only supported when they point to a file in the same directory, and otherwise are ignored - unless you have a symlink in /usr/share/applications, that one is special.

In Debian & Co., package maintainers are strongly advised to keep their .desktop files and binaries together. If people don't do that, asgen will attribute an application to the wrong package (usually a -data package) which leads to user confusion.

For this particular error, I am not sure how to resolve this - a workaround would be to return a null string in case that symlink can't be followed within the same package, which will result in the package being ignored entirely. Another solution would be to exclude the symlinks from the package contents index.

As a stop-gap measure, I made this issue non-fatal, so you can continue working on the code and testing the result. But this is an actual issue that needs to be resolved in some way by the backend or distribution.

[Cogitri]

As a stop-gap measure, I made this issue non-fatal, so you can continue working on the code and testing the result. But this is an actual issue that needs to be resolved in some way by the backend or distribution.

FWIW that triggers a warning from glib, in case you intend to keep this behaviour:

(process:21393): GLib-CRITICAL **: 14:17:25.106: g_key_file_load_from_data: assertion 'data != NULL || length == 0' failed

I'll look into fixing this on Alpine's side.

[Cogitri]

Alright, I just generated archives for all of Alpine's repos and that seems to work, thanks for the work on appstream-generator, making the Alpine backend really was a lot easier than I had expected! :)

Just a few questions about the generated data:

1. I set that asgen generates XML appdata for us since I wasn't really sure what to choose and that seems like what other distros also use (other than Debian). Is there an advantage to using YAML?

2. Should we keep installing the appdata with the package once you install them? It seems like installed software shows up twice in GNOME Software with that (but maybe that's a bug in my Alpine Linux plugin for GNOME Software).

3. I see Arch only installs 64x64 and 128x128 icons, should we also do that?

[ximion]

making the Alpine backend really was a lot easier than I had expected

Neat :-) While originally made just for Debian, many other distributions being interested and making backends certainly helped with that :-)

I set that asgen generates XML appdata for us since I wasn't really sure what to choose and that seems like what other distros also use (other than Debian). Is there an advantage to using YAML?

The only advantage of YAML is that the Debian FTP masters accepted this format in the archive as metadata. So a regular apt update will automatically also have the latest AppStream metadata, no package installation or any other task needed (and XML was a format they didn't want to see). Other than that, the XML and YAML versions are pretty much identical, so choose whetever you prefer, and if in doubt, choose the XML variant.

Should we keep installing the appdata with the package once you install them? It seems like installed software shows up twice in GNOME Software with that (but maybe that's a bug in my Alpine Linux plugin for GNOME Software).

It definitely shouldn't show up twice, that's some kind of bug. How are you installing the metadata the appstream-generator has produced?
Also, metainfo files should of course continued to be installed (unless there is a crazy need to save every little bit of disk space, even a few kb. Even in that case, the metainfo files need to remain in the packages though for asgen to find them)

I see Arch only installs 64x64 and 128x128 icons, should we also do that?

The 64x64 icons must be present, the specification requires that. All other sizes are optional. In Debian, we let the package that needs the icons (or the user) choose, e.g. if you install GNOME Software, apt will only download the 64x64 icons, if you install KDE Discover it will fetch the 64x64 and 128x128 icons, etc. If you have no GUI tool, no icons will be downloaded. (AppStream info is still useful to fetch console apps though, e.g. via appstreamcli)

[Cogitri]

How are you installing the metadata the appstream-generator has produced?

I install them like so:

usr/share/app-info/icons/alpinelinux/128x128/<many icons>
usr/share/app-info/icons/alpinelinux/64x64/<many icons>
usr/share/app-info/xmls/community.xml.gz
usr/share/app-info/xmls/main.xml.gz
usr/share/app-info/xmls/testing.xml.gz

I guess this probably is a bug in my GNOME Software plugin though, I'll ask the Software folks about it tomorrow :)

Also, metainfo files should of course continued to be installed (unless there is a crazy need to save every little bit of disk space, even a few kb)

Ah no, there's no need for that, I just wanted to make sure nothing bad happens if both the package's appstream data and what asgen generates is installed.

The 64x64 icons must be present, the specification requires that. All other sizes are optional. In Debian, we let the package that needs the icons (or the user) choose, e.g. if you install GNOME Software, apt will only download the 64x64 icons, if you install KDE Discover it will fetch the 64x64 and 128x128 icons, etc. If you have no GUI tool, no icons will be downloaded. (AppStream info is still useful to fetch console apps though, e.g. via appstreamcli)

Alright, thanks for the info. Considering the entire dir with both 64x64 and 128x128 icons is 7.6MBs big I think we're all set with having both icon sizes installed for now, maybe I'll make a subpackage later on.

[ximion]

The data is installed correctly (as long as the data origin= tag is set to "alpinelinux".

I guess this probably is a bug in my GNOME Software plugin though, I'll ask the Software folks about it tomorrow :)

If you have a separate plugin that bypasses AppStream data, then it's a safe bet that that is indeed the culprit :-) (I know nothing about that plugin though, so from my side it's just a guess)

Ah no, there's no need for that, I just wanted to make sure nothing bad happens if both the package's appstream data and what asgen generates is installed.

No, that's perfectly normal and legal. The AppStream metadata provided by the distributor is used by default, but when launching gnome-software with --prefer-local the metainfo files are preferred (useful for upstream authors who want to test things).

The ideal solution for shipping collection metadata (the stuff asgen produces) is to make it part of the archive metadata, like Debian and a few other distros do, so the package manager keeps it up-to-date automatically. The second best option is to package it and not forget to keep the package up to date (especially not with screenshots on a CDN).

[Cogitri]

The data is installed correctly (as long as the data origin= tag is set to "alpinelinux".

Ah, thanks for the hint, the origin tag is alpinelinux-$suite-$section. However, even after putting it in the right location: /usr/share/app-info/icons/alpinelinux-edge-testing which GNOME Software also mentions when I run it with--verbose`:

07:44:25:0585 Gs  GsApp: [0x5574745c0330]
kind:                desktop
state:               available
quirk:               provenance,not-launchable
id:                  veracrypt
unique-id:           system/package/alpine/desktop/veracrypt/*
scope:               system
bundle-kind:         package
kudos:               has-keywords
kudo-percentage:     5
name:                VeraCrypt
icon-kind:           stock
icon-name:           veracrypt
icon-prefix:         /usr/share/app-info/icons/alpinelinux-edge-testing
version:             
update-version:      1.22-r6
summary:             Disk encryption with strong security based on TrueCrypt
description:         disk encryption with strong security based on TrueCrypt
source-00:           veracrypt
url{homepage}:       https://www.veracrypt.fr/
launchable{desktop-id}: veracrypt.desktop
management-plugin:   apk
origin:              alpine
origin-appstream:    alpinelinux-edge-testing
origin-hostname:     alpinelinux.org
reviews:             0
provides:            0
size-installed:      3.6 MB
size-download:       1.3 MB
category:            System
{appstream::source-file}: /usr/share/app-info/xmls/testing.xml.gz
{apk::name}:         veracrypt
{GnomeSoftware::PackagingFormat}: apk

It doesn't display an icon for it.

Thanks for all the help! :)

[Cogitri]

Ah, of course that was a bug in my plugin, which I've fixed in the meantime. It seems like everything works now and I was able to generate a tarball with asgen for all of Alpine's repos. I've also fixed the Libreoffice package now - but I suppose it'd be nice if the thing about missing desktop files stayed a warning so that it's in the HTML report of asgen :)

[ximion]

Except for minor nitpicks, the patch looks good to me now! And there is a release in a few hours, so I'll merge this now so this change can get in last-minute.

Thanks for the patch! :-)