@neilmayhew
Copy link
Contributor

I have a third-party Ubuntu repo at http://packages.sil.org/ubuntu. I'm using a Xenial test machine. After refreshing sources with apt-get update, I run appstreamcli as follows:

sudo tools/appstreamcli refresh --force --verbose --details

It then hangs, using 100% CPU. I've left it as long as 10 minutes and it still doesn't finish. The output is:

** (appstreamcli:13992): DEBUG: Refreshing AppStream cache
** (appstreamcli:13992): DEBUG: Reading: /usr/share/app-info/xmls/org.freedesktop.fwupd.xml
** (appstreamcli:13992): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial_main_dep11_Components-amd64.yml.gz
** (appstreamcli:13992): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_main_dep11_Components-amd64.yml.gz
** (appstreamcli:13992): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:13992): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:13992): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-updates_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:13992): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_main_dep11_Components-amd64.yml.gz
** (appstreamcli:13992): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_restricted_dep11_Components-amd64.yml.gz
** (appstreamcli:13992): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_universe_dep11_Components-amd64.yml.gz
** (appstreamcli:13992): DEBUG: Reading: /var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_xenial-security_multiverse_dep11_Components-amd64.yml.gz
** (appstreamcli:13992): DEBUG: Reading: /var/lib/app-info/yaml/packages.sil.org_ubuntu_dists_xenial_main_dep11_Components-amd64.yml.gz
** (appstreamcli:13992): DEBUG: Reading: /var/lib/app-info/yaml/packages.sil.org_ubuntu_dists_xenial_universe_dep11_Components-amd64.yml.gz

If I run it from a root shell (sudo -s) or wrap it with a shell (sudo bash -c '...') I get the same result. Previously, it would run successfully like this, but the behaviour has now changed. The only difference I can think of is that I've installed build dependencies for appstream and have built it from source. However, I haven't done make install and it doesn't matter whether I run the built-from-source version or the installed package.

Very annoyingly, if I try to run appstreamcli from gdb, it runs successfully! So I can't step through the code and see exactly what's happening. Instead, I'm having to rely on creating a core dump, and because it's being run from a set-uid program, I have to set /proc/sys/fs/suid_dumpable and /proc/sys/kernel/core_pattern appropriately. I can then use ^\ to send SIGQUIT and use gdb on the resulting core dump.

I get the following backtrace:

#0  0x00007f912e1a18b4 in yaml_event_delete () from /usr/lib/x86_64-linux-gnu/libyaml-0.so.2
#1  0x00007f912fa767e0 in as_yamldata_parse_distro_data (ydt=0x1612d20, 
    data=0x1712580 "---\nFile: DEP-11\nVersion: '0.8'\nOrigin: pso-ubuntu-xenial-universe\nMediaBaseUrl: http://packages.sil.org/ubuntu/appstream/media\n\360\273p\001", error=0x7ffff1342c20) at /home/mayhewn/src/appstream/appstream/src/as-yamldata.c:1932
#2  0x00007f912fa77063 in as_metadata_parse_yaml (metad=0x161ed40, 
    data=0x1712580 "---\nFile: DEP-11\nVersion: '0.8'\nOrigin: pso-ubuntu-xenial-universe\nMediaBaseUrl: http://packages.sil.org/ubuntu/appstream/media\n\360\273p\001", error=0x7ffff1342c20) at /home/mayhewn/src/appstream/appstream/src/as-metadata.c:255
#3  0x00007f912fa77486 in as_metadata_parse_file (metad=0x161ed40, file=0x1619d40, error=0x7ffff1342c20)
    at /home/mayhewn/src/appstream/appstream/src/as-metadata.c:357
#4  0x00007f912fa7cc2b in as_data_pool_load_metadata (dpool=0x1612c80) at /home/mayhewn/src/appstream/appstream/src/as-data-pool.c:465
#5  0x00007f912fa7ce29 in as_data_pool_update (dpool=0x1612c80, error=0x7ffff1342d00)
    at /home/mayhewn/src/appstream/appstream/src/as-data-pool.c:512
#6  0x00007f912fa6c76b in as_cache_builder_refresh (builder=0x1612c30, force=0, error=0x7ffff1342d78)
    at /home/mayhewn/src/appstream/appstream/src/as-cache-builder.c:537
#7  0x0000000000405114 in ascli_refresh_cache (dbpath=0x0, datapath=0x0, forced=0)
    at /home/mayhewn/src/appstream/appstream/tools/ascli-actions-mdata.c:61
#8  0x0000000000403d61 in as_client_run (argv=0x7ffff1343128, argc=2) at /home/mayhewn/src/appstream/appstream/tools/appstream-cli.c:202
#9  0x000000000040405b in main (argc=4, argv=0x7ffff1343128) at /home/mayhewn/src/appstream/appstream/tools/appstream-cli.c:255

The content of /var/lib/app-info/yaml/packages.sil.org_ubuntu_dists_xenial_universe_dep11_Components-amd64.yml.gz is:

---
File: DEP-11
Version: '0.8'
Origin: pso-ubuntu-xenial-universe
MediaBaseUrl: http://packages.sil.org/ubuntu/appstream/media

It looks like as_metadata_parse_yaml is being given some extra junk characters on the end of the real data (\360\273p\001). These are definitely not in the real file (I checked with zcat ... | hexdump -C).

The output of ldd tools/appstreamcli is:

    linux-vdso.so.1 =>  (0x00007ffe1313f000)
    libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f0869571000)
    libgio-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x00007f08691e8000)
    libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007f0868f95000)
    libappstream.so.3 => /home/mayhewn/src/appstream/appstream/build/src/libappstream.so.3 (0x00007f0868d22000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0868958000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f08686e8000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f08684cb000)
    libgmodule-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007f08682c6000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f08680ac000)
    libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f0867e8a000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f0867c6e000)
    libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f0867a66000)
    libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f08676ac000)
    libyaml-0.so.2 => /usr/lib/x86_64-linux-gnu/libyaml-0.so.2 (0x00007f086748c000)
    libxapian.so.22 => /usr/lib/x86_64-linux-gnu/libxapian.so.22 (0x00007f0867090000)
    libprotobuf-lite.so.9 => /usr/lib/x86_64-linux-gnu/libprotobuf-lite.so.9 (0x00007f0866e5f000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0866adc000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f08668c6000)
    /lib64/ld-linux-x86-64.so.2 (0x00005635c4a6a000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f08666c2000)
    libicuuc.so.55 => /usr/lib/x86_64-linux-gnu/libicuuc.so.55 (0x00007f086632d000)
    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f086610b000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0865e02000)
    libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f0865bfc000)
    libicudata.so.55 => /usr/lib/x86_64-linux-gnu/libicudata.so.55 (0x00007f0864145000)

So it's picking up system libraries for everything except libappstream.so.3, as it should.