RFC: Generating AppStream Metadata

[matthewbauer]

Issue

This is coming up working on getting "gnome-software" working on Nix. AppStream is an XML data format to provide meta data listings on "apps" available to a distribution that is used by "gnome-software" to provide app listings in a distro agnostic way.

AppStream Metadata

To work with AppStream, a Linux distribution needs to provide an XML file that has a list of "components". The resulting XML looks like this:

​<?xml version="1.0"?>
​<components version="0.6">
​  <component type="application">
​    <id>firefox.desktop</id>
​    <pkgname>firefox-bin</pkgname>
​    <name>Firefox</name>
​    <name lang="en_GB">Firefoux</name>
​    <summary>Web browser</summary>
​    <summary lang="fr_FR">Navigateur web</summary>
​    <project_license>MPL-2</project_license>
​    <keywords>
​      <keyword>internet</keyword>
​      <keyword>web</keyword>
​      <keyword>browser</keyword>
​      <keyword lang="fr_FR">navigateur</keyword>
​    </keywords>
​    <icon type="stock">web-browser</icon>
​    <icon type="cached">firefox.png</icon>
​    <categories>
​      <category>network</category>
​      <category>web</category>
​    </categories>
​    <mimetypes>
​      <mimetype>text/html</mimetype>
​      <mimetype>text/xml</mimetype>
​      <mimetype>application/xhtml+xml</mimetype>
​      <mimetype>application/vnd.mozilla.xul+xml</mimetype>
​      <mimetype>text/mml</mimetype>
​      <mimetype>application/x-xpinstall</mimetype>
​      <mimetype>x-scheme-handler/http</mimetype>
​      <mimetype>x-scheme-handler/https</mimetype>
​    </mimetypes>
​    <url type="homepage">http://www.mozilla.com</url>
​    <screenshots>
​      <screenshot type="default">
​        <image type="source" width="800" height="600">http://www.awesomedistro.example.org/en_US/firefox.desktop/main.png</image>
​        <image type="thumbnail" width="200" height="150">http://www.awesomedistro.example.org/en_US/firefox.desktop/main-small.png</image>
​      </screenshot>
​    </screenshots>
​    <provides>
​      <binary>firefox</binary>
​    </provides>
​  </component>
​  <component>
​    <id>pulseaudio</id>
​    <name>PulseAudio</name>
​    <summary>The PulseAudio sound server</summary>
​    <url type="homepage">http://www.freedesktop.org/wiki/Software/PulseAudio/</url>
​    <project_license>GPLv2</project_license>
​    <provides>
​      <library>libpulse-simple.so.0</library>
​      <library>libpulse.so.0</library>
​      <binary>start-pulseaudio-kde</binary>
​      <binary>start-pulseaudio-x11</binary>
​    </provides>
​    <release version="2.0"/>
​  </component>
​  <component type="font">
​    <id>LinLibertine_M.otf</id>
​    <name>Libertine</name>
​    <summary>Linux Libertine Open fonts</summary>
​    <font_classifier>Mono</font_classifier>
​    <font_parent>Libertine</font_parent>
​  </component>
​  <!-- more components here! -->
​</components>

Some of the fields like "id", "packagename", "url", and "summary" we already have analogues in Nix. The issue going forward is going to be generating the rest of the data. Luckily most upstream apps already provide this data in the form of "appdata" XML files.

Upstream metadata

Most GNOME apps and some other apps already provide most of this data in the form of "appdata" XML. This is a short file that has some info on metadata like what libraries, fonts, binaries, etc it provides. The result looks like this:

​<?xml version="1.0" encoding="UTF-8"?>
​<component>
​  <id>foobar</id>
​  <name>Foo Bar</name>
​  <summary>A foo-ish bar</summary>
​  <url type="homepage">http://www.example.org</url>
​  <metadata_license>CC0-1.0</metadata_license>
​
​  <provides>
​    <library>libfoobar.so.2</library>
​    <font>foo.ttf</font>
​    <binary>foobar</binary>
​  </provides>
​  <releases>
​    <release version="1.2" date="2015-02-16" />
​  </releases>
​  <developer_name>FooBar Team</developer_name>
​</component>

In a normal distribution, each XML metadata file is installed either in "/usr/share/metainfo/" or "/usr/share/appdata/". With Nix, these files should already be installed in relevant derivations under /nix/store/.../share/appdata/ and /nix/store/.../share/metainfo. In addition, there are some other files that provide data that would be useful to keep track of:

• /share/appdata/
• /share/metainfo/
• /share/pixmaps/
• /share/icons/
• /share/${app_name}/icons/
• /share/applications/
• /share/applications/kde4/

Derivations that generate these folders need to be added to some index to keep track of AppStream "apps". This is going to be similar to how we were doing "command-not-found" support before issue NixOS/nixos-channel-scripts#4.

Proposal

So, my basic thinking right now is that Hydra would generate a list of all files within a derivation and upload it in the form of a *.narlist as in issue NixOS/nixos-channel-scripts#4. A NixOS channel script could then be run like "generate-programs-index.pl" that would generate the above AppStream metadata. Each channel would need to have its own AppStream list to ensure a correct index.

Each derivation that has any of the above file paths in it would cause a "generate-appstream-index.pl" script to be run with the derivation's out path. Each derivation would generate a "component" like above from the different folders output in the derivation. Each component would be added to the list of components in an "appstream.xml" file. That appstream.xml file could then be added to the "nixexprs.tar.xz" like "programs.sqlite" was previously. The generated appstream.xml would then be available at "/nix/var/nix/profiles/per-user/root/channels/nixos/appstream.xml".

The challenge would be making sure that "appstream-glib" is able to detect our appstream.xml file. Because it's not technically a derivation, this is basically going to require hard coding that path in like command-not-found does. This would require modifying the appstream-glib derivation to trick it into looking in /nix/var/nix/profiles/per-user/root/channels/nixos/.

Remaining Issues

There is other data that would be useful for appstream to have available. This includes the "/usr/share/applications/" and "/usr/share/icons/". Both of these are going to be much trickier because there's no easy way to put them in an XML file like with "appstream.xml" above. In addition, we don't want to put too much in "nixexprs.tar.xz" and things like icons are inevitably going to bloat the file size.

Conclusion

I am not completely familiar with how Hydra and NixOS are built so any suggestions or corrections are welcome for this proposal. It requires some extra steps in the build process but I think it should be doable.

[vcunat]

I didn't get into any details for now, but the big points seem OK to me.

[matthewbauer]

Okay great. It's probably a longer write up than it needs to be but it's kind of confusing. The important thing to know is that some way or another the appstream data must "derive" from the set of all nixpkgs. So it needs to be "above" the nixpkgs repo in some way and can't be handled just like any other derivation.

[vcunat]

I expect the mechanism won't be much different from the command-not-found database, but we haven't resolved that one yet ;-)

[matthewbauer]

I'm working on nix2appstream. It takes in a package name and outputs an AppStream collection.

[vcunat]

Any Idea if this is useful for the purpose? https://github.com/ximion/appstream-generator

[matthewbauer]

Sort of... You need a backend for appstream-generator to work.

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[worldofpeace]

We still need this.

Since we'll soon have a working packagekit nix backend with #94055 so this makes it extra desirable to have.

Not sure about making a home grown solution, it does seem more standard to use the tool other downstreams use https://github.com/ximion/appstream-generator/, the only real hurdle there is writing the backend which needs to be in DLang. And I believe it remains that it will need to be published into the channel.

[matthewbauer]

We still need this.

Since we'll soon have a working packagekit nix backend with #94055 so this makes it extra desirable to have.

Not sure about making a home grown solution, it does seem more standard to use the tool other downstreams use https://github.com/ximion/appstream-generator/, the only real hurdle there is writing the backend which needs to be in DLang. And I believe it remains that it will need to be published into the channel.

Yeah that sounds much easier. Most of the original issue is not relevant since we now have NixOS/nix@542ae5c and can just query stuff directly from Hydra.

[stale]

I marked this as stale due to inactivity. → More info

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/brainstorming-generating-appstream-metadata/19273/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/the-linux-experiment-every-linux-distro-should-have-these-features/21955/1