Doesn't export uncompressed files but apt requires them

[neilmayhew]

I had trouble figuring out why apt-get update wasn't fetching any of the dep11 files from my repo, even though the files were mentioned in the Release file. Eventually I discovered that it looks for the uncompressed Components and icons files, and then downloads the compressed ones. (It actually downloads .xz, then recompresses as .gz.) So currently I'm having to generate the uncompressed files in a separate step after running asgen.

This is more of a bug in apt than in asgen but since apt is out there like this in Xenial right now, it's probably a good idea to match what it expects.

[ximion]

This is definitely not a bug in APT or asgen, because it works on Debian and Ubuntu with compressed-files only ;-)
APT basically doesn't care which files it gets, it happily recompresses to whatever intermediate form is needed.
Take a look at ftp://ftp.debian.org/debian/dists/sid/Release and ftp://ftp.debian.org/debian/dists/sid/InRelease - the checksums for the uncompressed version are stored, but not that version itself.
So, this must be a bug in something else (Xenial is definitely fine with compressed-only files). Not compressing the files is both slow for downloading them, and also for loading them from disk (which is why we keep them GZip-compressed even post-download).

[neilmayhew]

It certainly doesn't work in Xenial unless the uncompressed checksum is in the Release file. It's true that the file itself doesn't have to exist, but it seems a bit weird to be putting an entry in the release for a file that doesn't actually exist.

I guess you could say that the process I'm using for putting the dep11 checksums into the Release file needs to be improved. However, I'm using a reprepro export hook that simply adds all the files that exist in the dep11 subdirectory. The checksums are generated by reprepro after the hook lists the extra files to be included, so I'd still need the uncompressed file to exist so reprepro can calculate its sum.

I'm not sure what I should do here. My company runs its own repo (packages.sil.org) and we also use PPAs to distribute packages that aren't in the main repos. We're now required to use AppStream to make our apps visible to regular users in Xenial (due to the switch to Gnome Software) but I'm finding it hard to get things working. I have now succeeded in getting the apps from our own repo to show up in appstreamcli, but they're not showing up in Gnome Software, presumably because there's something unacceptable in the metadata. (I have changed org.gnome.software.official-sources in dconf but that keeps getting reset, so maybe that's the problem.) However, there's no way of making apps in PPAs show up because DEP-11 isn't supported in PPAs and currently there are no plans to change this.

[ximion]

Unfortunate...
That reprepro doesn't add the uncompressed entry is a bug in reprepro, in my opinion, since this is common practice. In the Debian repo, the uncompressed entries for every file are present, and there are no uncompressed versions shipped.
So maybe file a bug against reprepro? I could also ask the Debian ftp-masters about this, just to be sure.

For the metadata, I think you are running into hughsie/appstream-glib#98 which - as soon as it's fixed - I will propose for SRU at Ubuntu.
I really don't want to add a legacy-compatibility mode for the generator, but if this doesn't get fixed in Xenial, I fear there will be no way around that. I will push Richard a bit to consider solving that issue ;-) (or maybe solve it myself, it's really not a complicated fix)

The other issue you could be running into is that your metadata priority is too low, but if appstreamcli shows the stuff, then this cause is unlikely.

[ximion]

For Ubuntu, you can follow https://bugs.launchpad.net/ubuntu/+source/appstream-glib/+bug/1576780 now ;-)
Debian is fixed.

[neilmayhew]

I built a package with that diff and tried it. The icons now show up in appstreamcli dump whereas they didn't before, so I think that part of it is working. However, my apps still don't show up when I search in Gnome Software, so it must be some other problem. The value I'm putting in org.gnome.software.official-sources in dconf keeps getting reset, which seems suspicious, but I haven't figured out when and why yet.

Thanks very much for your help with this.

[ximion]

You really shouldn't modify org.gnome.software.official-sources, that is only for distributor sources.
The data is in the repo? If so, I can test why it doesn't appear tomorrow, probably just a simple issue.
Maybe killing the GNOME Software background daemon will do something.
Also ensure that origin and media_baseurl are properly set in your XML / generator settings.

[neilmayhew]

You really shouldn't modify org.gnome.software.official-sources, that is only for distributor sources.

I get the impression that the provenance plugin in Gnome Software will ignore apps that aren't from an official source. Or does it just mark them in some way and still show them? I haven't been able to get any third-party apps to show up, so I can't really test it :-)

The data is in the repo? If so, I can test why it doesn't appear tomorrow, probably just a simple issue.

The data isn't fully in the repo yet. I've been working with a local copy of the repo so I could play around and not destabilize things. However, if you're going to take a look at it I'll put my current data in the real one.

Maybe killing the GNOME Software background daemon will do something.

I've made sure to do that each time, after seeing what you wrote in Ubuntu 1576780.

Also ensure that origin and media_baseurl are properly set in your XML / generator settings.

There was a problem with that, due to my switching between live and test repos, but even after fixing MediaBaseUrl in the config it doesn't seem to help. I'm not sure what you mean by "origin", though. Would this be the same as ProjectName? I currently have that set to something other than ubuntu since it's a third-party repo. When I run gnome-software --verbose I can see it tagging my apps as follows:

(gnome-software:13295): GsPlugin-DEBUG: origin pso-ubuntu-xenial-main provides 2 apps (0%)
(gnome-software:13295): GsPlugin-DEBUG: origin ubuntu-xenial-multiverse provides 50 apps (3%)
(gnome-software:13295): GsPlugin-DEBUG: origin pso-ubuntu-xenial-experimental-main provides 3 apps (0%)
(gnome-software:13295): GsPlugin-DEBUG: origin ubuntu-xenial-main provides 104 apps (6%)
(gnome-software:13295): GsPlugin-DEBUG: origin fwupd provides 6 apps (0%)
(gnome-software:13295): GsPlugin-DEBUG: origin ubuntu-xenial-universe provides 1649 apps (90%)
(gnome-software:13295): GsPlugin-DEBUG: origin ubuntu-xenial-restricted provides 7 apps (0%)
...
(gnome-software:13295): GsPlugin-DEBUG: Adding keyword 'pso-ubuntu-xenial-experimental-main' to fieldworks-flex.desktop
(gnome-software:13295): GsPlugin-DEBUG: Adding keyword 'pso-ubuntu-xenial-experimental-main' to fieldworks-te.desktop
(gnome-software:13295): GsPlugin-DEBUG: Adding keyword 'pso-ubuntu-xenial-experimental-main' to scripture-app-builder.desktop

That's why I was messing with org.gnome.software.official-sources, to allow the additional origin to be used.

When I run apt-get update, the 50appstream hook runs appstreamcli refresh which produces the following message:

AppStream cache update completed, but some metadata was ignored due to errors.

When I run it manually with full output, I get:

$ sudo appstreamcli refresh --force --verbose --details
** (appstreamcli:30752): DEBUG: Refreshing AppStream cache
** (appstreamcli:30752): DEBUG: Reading: /usr/share/app-info/xmls/org.freedesktop.fwupd.xml
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial_main_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_main_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_main_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/bach.wycliffe.ca_packages_ubuntu_dists_xenial_main_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/bach.wycliffe.ca_packages_ubuntu_dists_xenial_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/bach.wycliffe.ca_packages_ubuntu_dists_xenial_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/bach.wycliffe.ca_packages_ubuntu_dists_xenial_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/bach.wycliffe.ca_packages_ubuntu_dists_xenial-experimental_main_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/bach.wycliffe.ca_packages_ubuntu_dists_xenial-experimental_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/bach.wycliffe.ca_packages_ubuntu_dists_xenial-experimental_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/bach.wycliffe.ca_packages_ubuntu_dists_xenial-experimental_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:30752): DEBUG: Reading: /var/cache/app-info/xmls/fwupd.xml
** (appstreamcli:30752): DEBUG: Replaced 'language-selector.desktop' with data of higher priority.
** (appstreamcli:30752): DEBUG: Replaced 'fieldworks-flex.desktop' with data of higher priority.
** (appstreamcli:30752): DEBUG: Replaced 'fieldworks-te.desktop' with data of higher priority.
** (appstreamcli:30752): DEBUG: Replaced 'scripture-app-builder.desktop' with data of higher priority.
** (appstreamcli:30752): DEBUG: Detected colliding ids: plan.desktop was already added with the same priority.
** (appstreamcli:30752): DEBUG: Detected colliding ids: flcheckers.desktop was already added with the same priority.
** (appstreamcli:30752): DEBUG: Detected colliding ids: flblocks.desktop was already added with the same priority.
** (appstreamcli:30752): DEBUG: Detected colliding ids: flsudoku.desktop was already added with the same priority.
** (appstreamcli:30752): DEBUG: zathura-pdf-poppler.desktop extends zathura.desktop, but zathura.desktop was not found.
** (appstreamcli:30752): DEBUG: WARNING: Skipped component 'com.steelseries.rival-legacy.firmware': The component is invalid.
** (appstreamcli:30752): DEBUG: Removing old rebuild-dir from previous database rebuild.
AppStream cache update completed, but some metadata was ignored due to errors.

bach.wycliffe.ca is my local copy of the packages.sil.org repo used for testing. fieldworks-flex.desktop, fieldworks-te.desktop, scripture-app-builder.desktop are the three apps we have in Xenial so far. There will be others but they're not tested and working in Xenial yet. The reason for the replacement is probably that the same package occurs in both xenial and xenial-experimental at the moment.

So it looks like the appstream subsystem isn't finding an error with our apps, but gnome-software is. I'll get my data posted asap so you can have a look at it.

[neilmayhew]

Hmmm, not sure what I've done wrong. I copied my data to the real repo and disabled the test one on the machine I'm using for testing (a slow laptop). The appstreamcli refresh run by the hook from apt-get update ran at 100% CPU for 10 minutes and didn't finish so I aborted it. I ran it again with --verbose --details and it gets stuck with the following output:

** (appstreamcli:31276): DEBUG: Refreshing AppStream cache
** (appstreamcli:31276): DEBUG: Reading: /usr/share/app-info/xmls/org.freedesktop.fwupd.xml
** (appstreamcli:31276): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial_main_dep11_Components-amd64.yml.gz
** (appstreamcli:31276): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_main_dep11_Components-amd64.yml.gz
** (appstreamcli:31276): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:31276): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:31276): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:31276): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_main_dep11_Components-amd64.yml.gz
** (appstreamcli:31276): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:31276): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:31276): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:31276): DEBUG: Reading: /var/lib/app-info/yaml/packages.sil.org_ubuntu_dists_xenial_main_dep11_Components-amd64.yml.gz
** (appstreamcli:31276): DEBUG: Reading: /var/lib/app-info/yaml/packages.sil.org_ubuntu_dists_xenial_universe_dep11_Components-amd64.yml.gz

The problem might be that /var/lib/app-info/yaml/packages.sil.org_ubuntu_dists_xenial_universe_dep11_Components-amd64.yml.gz contains just:

---
File: DEP-11
Version: '0.8'
Origin: pso-ubuntu-xenial-universe
MediaBaseUrl: http://packages.sil.org/ubuntu/appstream/media

The one it's processed just before does contain some reasonable-looking data.

I don't understand why I didn't hit this before when I was using the test repo.

[neilmayhew]

I've also copied the validation output to the repo server. There are some issues, but I don't think any of them are enough to cause major problems.

[ximion]

Urgh... Well, I get everything to work perfectly well here. I just add the repo, stuff shows up in GNOME Software.
But the following new patch is needed: hughsie/appstream-glib@cba31a5
I will ensure this one gets included in asglib ;-)
The 10min delay in appstreamcli sounds really scary, if you are able to reproduce that somehow, please report a bug against AppStream: https://github.com/ximion/appstream/issues

Thanks for your patience :) There is nothing wrong with the data itself, everything is fine except for the uncompressed files thing, which is IMHO is a reprepro issue.
Also, your apps should have a long description ;-)
Additionally, you seem to lack the JavaScript files for the generator, maybe run make js once, or just download the stuff from elsewhere and copy it into the statis/js directory.

Sorry for the inconvenience, and thank you for pioneering this! (together with Elementary, which seems to hit less issues at time ^^)
This is very much appreciated, because it helps me to iron out bugs in the stack.

[ximion]

As for the "does GS need provenance for the apps to be visible": No, it doesn't - if the data comes from 3rd-party sources, GS will just display a 3rd-party banner below that app.
But since I didn't even see that, maybe that function is disabled in GS...

If this now works for you with the new patch, please give some test feedback on https://bugs.launchpad.net/ubuntu/+source/appstream-glib/+bug/1576780 to get that change in faster :-)

[neilmayhew]

Thanks, great work! My apps show up in Gnome Software now. I added the cba31a5 change to my locally-built package and I was able to see my apps.

I see that the fix has already gone into sid and yakkety. Hopefully the xenial SRU will kick in soon. I'll add some feedback to LP 1576780 to help things along.

It's been good working with you on this, and I appreciate your responsiveness. I'm glad it's helped you get bugs ironed out in the stack. We all benefit in the end.

[neilmayhew]

The 10min delay in appstreamcli sounds really scary, if you are able to reproduce that somehow, please report a bug against AppStream

I did some more testing, and I found that it goes wrong only if I run the refresh command directly with sudo. If I run it from a root shell obtained with sudo -s, or I wrap it with bash (sudo bash -c '...') then it runs successfully in just a few seconds. So it must be something to do with effective ids.

The problem is 100% reproducible with my current data files. I had to mess with /proc to get it to dump core in this situation, but I was able to do that successfully, so I'll open an issue against AppStream as you suggest.