[WIP] Allow AppStream to parse recommendations

[lucasmoura]

This MR add the feature to AppStream to allow it to parse recommendation files. A recommendation file is defined by a XML document in the following format:

<?xml version="1.0"?>
    <recommendations>
        <recommendation>
            <id>vim.desktop</id>
            <because>
                <id>emacs.desktop</id>
            </because>
        </recommendation>
        <recommendation>
            <id>ipython.desktop</id>
            <because>
                <id>eric.desktop</id>
                <id>winpdb.desktop</id>
            </because>
        </recommendation>
    </recommendations>

Where id is the package being recommended and because is the user packages that generated this recommendation.

Currently, it is already working the construction and parsing of a valid XML recommendation, however there are some necessary improvements to be done:

• There is a lot of code that equals the one on the core AppStream modules and a refactoring is necessary to fix that.
• It is necessary to validate a recommendation object, because if the recommended package is not indexed by AppStream, the recommendation object should be invalid.
• Add an appstreamcli function to parse and display the recommendations.
• Validate the XML that represents a recommendation.

Therefore, this MR is manly to collect feedback from the AppStream community regarding this feature.

[ximion]

Nice! I'll be a bit busy the next days (KDE AppStore meeting and other things), but I think I can give it a quick review tonight. One obvious thing to resolve would be fixing the test issues ;-) (minor stuff, as far as I can see).
Some quick initial questions:

1. Where is that recommendation data supposed to come from? Are upstream projects supposed to ship it in their metainfo file? Or doe distributions seed that stuff?

2. Regarding the XML example:

<recommendations>
        <recommendation>
            <recommended>vim.desktop</recommended>
            <because>
                <pkg>emacs.desktop</pkg>
            </because>

You want to define a condition here, right? So, something like "if Vim present, suggest Emacs". The "because" should therefore probably be an "if". Also, I think the whole block is a bit strange... Why not put the recommendation into the component-block of the thing recommending it? So, something like:

<component type="desktop">
  <id>emacs.desktop</id>
  <recommendations>
    <recommendation>emacs.desktop</recommendation>
  </recommendations>
</component>

That would look far more simple to me, and would prevent the need for declaring logic in XML. Aside from that, your pkg tag is actually referring to an AppStream-ID (probably a leftover from when it was package-based).

I need to read the code this evening to fully understand the concept behind this and give some better feedback though ;-)

[lucasmoura]

No problem. I will look why the build is failing. It is currently presenting this error:

gpg: no valid OpenPGP data found.var/lib/app-inf

But I will look it up to fix it.

1. Originally, I thought that the recommendation application could generate the file and place it a common place for AppStream to parse the recommendations. Something like /usr/share/recommendations. But I still don't know if that is the best approach.

2. I think I have not explained the because tag properly. The recommendation would not be something like if the user has emacs.desktop then recommends vim.desktop, it would be because the user already has emacs.desktop then recommends vim.desktop. The recommender application would be the one responsible for asserting which packages the user has and control this representation.

But you are right, the tag names provided by your XML file are way better than the ones I have provided. I will create an XML file with better names and update the code accordingly.

[ximion]

Sidenote: I really want a reply function in the GitHub UI! Making replys manually is annoying...

gpg: no valid OpenPGP data found.var/lib/app-inf
But I will look it up to fix it.

No worries, that was a temporary failure due to the LLVM APT repo not being available. I took that as an argument to make the whole build process happen in a Docker container now, which allows us to be independent of the outdated Travis base system.
So, to fix this for you, just rebase on master.

The recommender application would be the one responsible for asserting which packages the user has and control this representation.

Okay, then why have this because information in the first place? Wouldn't simply dropping a list of recommended AppStream-IDs into some directory in /var/cache be more useful? (and have AS pick that up then)

[lucasmoura]

Okay, I have applied the rebase and I will check again if there is another problem during the build process.

But about the because tag. Originally, I though that it would be good for an user to know which of his packages generated the one being recommended. It would made the recommendation more trustworthy, since the user would see the reason behind it. Therefore, I thought that the applications that would use this AppStream data could display this extra information for their users.

However, maybe on the context of AppStream this extra information is really not necessary or I can make it optional.

[lucasmoura]

I have updated some of the XML tag names and also added a method to check if the recommendation object is valid. By valid, I mean that the package referenced by the id variable is a package indexed by AppStream.

[lucasmoura]

I have added the "merge" type for a component class. I have also made a verification on the
as_data_pool_merge_components method, to only merge components if the src_cpt has the merge type.

[ximion]

Should rather be cpt_ids of utf8

[ximion]

On second thought, I also wonder if a GHashTable isn't a better idea here, since that will prevent us from having duplicate entries...

[ximion]

I fixed up the first two patches and committed the result to master.
Can you please rebase the remaining bits on top of latest master? (there were quite a few changes, for example the inclusion of the merge component, and a clean set of patches will be easier to review).

Bonus points for adding some validator hints (e.g. having a type=heuristic component in metainfo files is an error).

[ximion]

You are leaking memory here.

[ximion]

The contents of this PR are now merged - there is still work to do, mainly support for writing the suggests tag and for the YAML part is completely missing.
Also, there are no tests for reading the data from an XML file / writing it yet.

[ximion]

Thanks for submitting this PR!

[lucasmoura]

Sorry for that huge amount of memory leaks. I will be more careful on my next merge request. I think I will now allow AppRecommender to write the XML files and look into the YAML part as well.

Thanks for the patience on reviewing this feature :)

[ximion]

I already took care of the YAML world (and pretty much all other bits) - the remaining piece is getting the merge logic work for (almost) all fields, see the as_merge_components function in AsDataPool.

I hope I can do a new AppStream release ASAP.
Please pay attention that @hughsie and I changed how component merges are done after a brief discussion on IRC, check the example file in the testsuite for how it's done now :)

The libappstream library is capable of writing the necessary XML or YAML files for merge components now.