Skip to content

RFC: Generating AppStream Metadata #15932

Open
Open
RFC: Generating AppStream Metadata#15932
Labels
0.kind: enhancementAdd something new or improve an existing system.6.topic: freedesktopInteroperability and shared base technology for X, Wayland and others6.topic: user experienceNixOS / Nixpkgs end user experience

Description

@matthewbauer
matthewbauer
opened on Jun 2, 2016

Issue

This is coming up working on getting "gnome-software" working on Nix. AppStream is an XML data format to provide meta data listings on "apps" available to a distribution that is used by "gnome-software" to provide app listings in a distro agnostic way.

AppStream Metadata

To work with AppStream, a Linux distribution needs to provide an XML file that has a list of "components". The resulting XML looks like this:

​<?xml version="1.0"?>
​<components version="0.6">
​  <component type="application">
​    <id>firefox.desktop</id>
​    <pkgname>firefox-bin</pkgname>
​    <name>Firefox</name>
​    <name lang="en_GB">Firefoux</name>
​    <summary>Web browser</summary>
​    <summary lang="fr_FR">Navigateur web</summary>
​    <project_license>MPL-2</project_license>
​    <keywords>
​      <keyword>internet</keyword>
​      <keyword>web</keyword>
​      <keyword>browser</keyword>
​      <keyword lang="fr_FR">navigateur</keyword>
​    </keywords>
​    <icon type="stock">web-browser</icon>
​    <icon type="cached">firefox.png</icon>
​    <categories>
​      <category>network</category>
​      <category>web</category>
​    </categories>
​    <mimetypes>
​      <mimetype>text/html</mimetype>
​      <mimetype>text/xml</mimetype>
​      <mimetype>application/xhtml+xml</mimetype>
​      <mimetype>application/vnd.mozilla.xul+xml</mimetype>
​      <mimetype>text/mml</mimetype>
​      <mimetype>application/x-xpinstall</mimetype>
​      <mimetype>x-scheme-handler/http</mimetype>
​      <mimetype>x-scheme-handler/https</mimetype>
​    </mimetypes>
​    <url type="homepage">http://www.mozilla.com</url>
​    <screenshots>
​      <screenshot type="default">
​        <image type="source" width="800" height="600">http://www.awesomedistro.example.org/en_US/firefox.desktop/main.png</image>
​        <image type="thumbnail" width="200" height="150">http://www.awesomedistro.example.org/en_US/firefox.desktop/main-small.png</image>
​      </screenshot>
​    </screenshots>
​    <provides>
​      <binary>firefox</binary>
​    </provides>
​  </component>
​  <component>
​    <id>pulseaudio</id>
​    <name>PulseAudio</name>
​    <summary>The PulseAudio sound server</summary>
​    <url type="homepage">http://www.freedesktop.org/wiki/Software/PulseAudio/</url>
​    <project_license>GPLv2</project_license>
​    <provides>
​      <library>libpulse-simple.so.0</library>
​      <library>libpulse.so.0</library>
​      <binary>start-pulseaudio-kde</binary>
​      <binary>start-pulseaudio-x11</binary>
​    </provides>
​    <release version="2.0"/>
​  </component>
​  <component type="font">
​    <id>LinLibertine_M.otf</id>
​    <name>Libertine</name>
​    <summary>Linux Libertine Open fonts</summary>
​    <font_classifier>Mono</font_classifier>
​    <font_parent>Libertine</font_parent>
​  </component>
​  <!-- more components here! -->
​</components>

Some of the fields like "id", "packagename", "url", and "summary" we already have analogues in Nix. The issue going forward is going to be generating the rest of the data. Luckily most upstream apps already provide this data in the form of "appdata" XML files.

Upstream metadata

Most GNOME apps and some other apps already provide most of this data in the form of "appdata" XML. This is a short file that has some info on metadata like what libraries, fonts, binaries, etc it provides. The result looks like this:

​<?xml version="1.0" encoding="UTF-8"?>
​<component>
​  <id>foobar</id>
​  <name>Foo Bar</name>
​  <summary>A foo-ish bar</summary>
​  <url type="homepage">http://www.example.org</url>
​  <metadata_license>CC0-1.0</metadata_license>
​
​  <provides>
​    <library>libfoobar.so.2</library>
​    <font>foo.ttf</font>
​    <binary>foobar</binary>
​  </provides>
​  <releases>
​    <release version="1.2" date="2015-02-16" />
​  </releases>
​  <developer_name>FooBar Team</developer_name>
​</component>

In a normal distribution, each XML metadata file is installed either in "/usr/share/metainfo/" or "/usr/share/appdata/". With Nix, these files should already be installed in relevant derivations under /nix/store/.../share/appdata/ and /nix/store/.../share/metainfo. In addition, there are some other files that provide data that would be useful to keep track of:

  • /share/appdata/
  • /share/metainfo/
  • /share/pixmaps/
  • /share/icons/
  • /share/${app_name}/icons/
  • /share/applications/
  • /share/applications/kde4/

Derivations that generate these folders need to be added to some index to keep track of AppStream "apps". This is going to be similar to how we were doing "command-not-found" support before issue NixOS/nixos-channel-scripts#4.

Proposal

So, my basic thinking right now is that Hydra would generate a list of all files within a derivation and upload it in the form of a *.narlist as in issue NixOS/nixos-channel-scripts#4. A NixOS channel script could then be run like "generate-programs-index.pl" that would generate the above AppStream metadata. Each channel would need to have its own AppStream list to ensure a correct index.

Each derivation that has any of the above file paths in it would cause a "generate-appstream-index.pl" script to be run with the derivation's out path. Each derivation would generate a "component" like above from the different folders output in the derivation. Each component would be added to the list of components in an "appstream.xml" file. That appstream.xml file could then be added to the "nixexprs.tar.xz" like "programs.sqlite" was previously. The generated appstream.xml would then be available at "/nix/var/nix/profiles/per-user/root/channels/nixos/appstream.xml".

The challenge would be making sure that "appstream-glib" is able to detect our appstream.xml file. Because it's not technically a derivation, this is basically going to require hard coding that path in like command-not-found does. This would require modifying the appstream-glib derivation to trick it into looking in /nix/var/nix/profiles/per-user/root/channels/nixos/.

Remaining Issues

There is other data that would be useful for appstream to have available. This includes the "/usr/share/applications/" and "/usr/share/icons/". Both of these are going to be much trickier because there's no easy way to put them in an XML file like with "appstream.xml" above. In addition, we don't want to put too much in "nixexprs.tar.xz" and things like icons are inevitably going to bloat the file size.

Conclusion

I am not completely familiar with how Hydra and NixOS are built so any suggestions or corrections are welcome for this proposal. It requires some extra steps in the build process but I think it should be doable.

Activity

  1. vcunat commented on Jun 2, 2016

    @vcunat
    vcunat
    on Jun 2, 2016
    Member

    I didn't get into any details for now, but the big points seem OK to me.

  2. vcunat
    added
    0.kind: enhancementAdd something new or improve an existing system.
    6.topic: usability
    on Jun 2, 2016

  3. matthewbauer commented on Jun 2, 2016

    @matthewbauer
    matthewbauer
    on Jun 2, 2016
    MemberAuthor

    Okay great. It's probably a longer write up than it needs to be but it's kind of confusing. The important thing to know is that some way or another the appstream data must "derive" from the set of all nixpkgs. So it needs to be "above" the nixpkgs repo in some way and can't be handled just like any other derivation.

  4. vcunat commented on Jun 2, 2016

    @vcunat
    vcunat
    on Jun 2, 2016
    Member

    I expect the mechanism won't be much different from the command-not-found database, but we haven't resolved that one yet ;-)

  5. vcunat
    added
    6.topic: user experienceNixOS / Nixpkgs end user experience
    and removed
    6.topic: usability
    on Oct 20, 2016

  6. matthewbauer commented on Jan 12, 2017

    @matthewbauer
    matthewbauer
    on Jan 12, 2017
    MemberAuthor

    I'm working on nix2appstream. It takes in a package name and outputs an AppStream collection.

  7. vcunat commented on Jan 13, 2017

    @vcunat
    vcunat
    on Jan 13, 2017
    Member

    Any Idea if this is useful for the purpose? https://github.com/ximion/appstream-generator

  8. matthewbauer commented on Feb 7, 2017

    @matthewbauer
    matthewbauer
    on Feb 7, 2017
    MemberAuthor

    Sort of... You need a backend for appstream-generator to work.

  9. jtojnar
    added
    6.topic: freedesktopInteroperability and shared base technology for X, Wayland and others
    on May 9, 2018

  10. stale commented on Jun 4, 2020

    @stale
    stalebot
    on Jun 4, 2020

    Thank you for your contributions.

    This has been automatically marked as stale because it has had no activity for 180 days.

    If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

    Here are suggestions that might help resolve this more quickly:

    1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
    2. Ask on the NixOS Discourse.
    3. Ask on the #nixos channel on irc.freenode.net.
  11. stale
    added
    2.status: stalehttps://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md
    on Jun 4, 2020

  12. worldofpeace commented on Jul 29, 2020

    @worldofpeace
    worldofpeace
    on Jul 29, 2020
    Contributor

    We still need this.

    Since we'll soon have a working packagekit nix backend with #94055 so this makes it extra desirable to have.

    Not sure about making a home grown solution, it does seem more standard to use the tool other downstreams use https://github.com/ximion/appstream-generator/, the only real hurdle there is writing the backend which needs to be in DLang. And I believe it remains that it will need to be published into the channel.

  13. stale
    removed
    2.status: stalehttps://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md
    on Jul 29, 2020

  14. matthewbauer commented on Jul 29, 2020

    @matthewbauer
    matthewbauer
    on Jul 29, 2020
    MemberAuthor

    We still need this.

    Since we'll soon have a working packagekit nix backend with #94055 so this makes it extra desirable to have.

    Not sure about making a home grown solution, it does seem more standard to use the tool other downstreams use https://github.com/ximion/appstream-generator/, the only real hurdle there is writing the backend which needs to be in DLang. And I believe it remains that it will need to be published into the channel.

    Yeah that sounds much easier. Most of the original issue is not relevant since we now have NixOS/nix@542ae5c and can just query stuff directly from Hydra.

  15. stale commented on Jan 25, 2021

    @stale
    stalebot
    on Jan 25, 2021

    I marked this as stale due to inactivity. → More info

  16. stale
    added
    2.status: stalehttps://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md
    on Jan 25, 2021

  17. nixos-discourse commented on May 22, 2022

    @nixos-discourse
    nixos-discourse
    on May 22, 2022

    This issue has been mentioned on NixOS Discourse. There might be relevant details there:

    https://discourse.nixos.org/t/brainstorming-generating-appstream-metadata/19273/1

  18. stale
    removed
    2.status: stalehttps://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md
    on May 22, 2022

  19. nixos-discourse commented on Sep 24, 2022

    @nixos-discourse
    nixos-discourse
    on Sep 24, 2022

    This issue has been mentioned on NixOS Discourse. There might be relevant details there:

    https://discourse.nixos.org/t/the-linux-experiment-every-linux-distro-should-have-these-features/21955/1

  20. nixpkgs-ci
    added
    2.status: stalehttps://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md
    on Jul 3, 2025

  21. nixos-discourse commented on Feb 19, 2026

    @nixos-discourse
    nixos-discourse
    on Feb 19, 2026

    This issue has been mentioned on NixOS Discourse. There might be relevant details there:

    https://discourse.nixos.org/t/nixos-plugin-for-gnome-software/75453/2

  22. nixpkgs-ci
    removed
    2.status: stalehttps://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md
    on Feb 19, 2026

  23. bahrom04 commented on Jun 10, 2026

    @bahrom04
    bahrom04
    on Jun 10, 2026
    Member

    Still actual topic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    0.kind: enhancementAdd something new or improve an existing system.6.topic: freedesktopInteroperability and shared base technology for X, Wayland and others6.topic: user experienceNixOS / Nixpkgs end user experience

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions