Chokes on repo components that don't have any packages in them

[neilmayhew]

Eg, xenial-updates/main/binary-amd64/Packages.gz doesn't have any packages in it yet.

The error is:

object.Exception@source/archive.d(84): Unable to open compressed file '.../Packages.gz'

FYI, I'm not actually running this on the real Ubuntu repo. I just gave that as an example. We have a company-specific repo that I'm trying to generate data for. I've had to remove the empty components from asgen-config.json temporarily.

[ximion]

Weird, the code shouldn't even get that far...
Can you try with the patch above applied and - if that doesn't solve it - attach the full backtrace?

[ximion]

Okay, this issue seems fixed, tried it here with a small archive and a fake section.
Thanks for the bug report!
If this issue persists, please report back se we can reopen this bug report!

[ximion]

I also tried this with an empty Packages.gz file (content: zero), without producing the crash you describe - could it be that your Packages.gz file is actually broken? If it isn't, and it's small enough, can you attach it?

[neilmayhew]

I rebuilt from the tip of master (which now includes the commit you mention above) and re-ran the generation. Here's the stack backtrace:

2016-04-22 15:13:43 - INFO: Scanning new packages.
object.Exception@source/archive.d(84): Unable to open compressed file '/srv/rsync/packages/ubuntu/dists/xenial-updates/main/binary-amd64/Packages.gz'
----------------
0x45cedd immutable(char)[] ag.archive.decompressFile(immutable(char)[])
    source/archive.d:84
0x462a4b void ag.backend.debian.tagfile.TagFile.open(immutable(char)[])
    source/backends/debian/tagfile.d:46
0x460502 ag.backend.intf.Package[] ag.backend.debian.pkgindex.DebianPackageIndex.loadPackages(immutable(char)[], immutable(char)[], immutable(char)[])
    source/backends/debian/deb-package-index.d:124
0x460ae4 ag.backend.intf.Package[] ag.backend.debian.pkgindex.DebianPackageIndex.packagesFor(immutable(char)[], immutable(char)[], immutable(char)[])
    source/backends/debian/deb-package-index.d:161
0x51f713 void ag.engine.Engine.seedContentsData(ag.config.Suite)
    source/engine.d:134
0x521f55 void ag.engine.Engine.run(immutable(char)[])
    source/engine.d:353
0x453aa8 _Dmain
    source/app.d:112
0x5ff60e _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv
    ../../../../src/libphobos/libdruntime/rt/dmain2.d:411
0x5ff73e void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())
    ../../../../src/libphobos/libdruntime/rt/dmain2.d:386
0x5ffb18 void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll()
    ../../../../src/libphobos/libdruntime/rt/dmain2.d:411
0x5ff73e void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())
    ../../../../src/libphobos/libdruntime/rt/dmain2.d:386
0x5ff8c5 _d_run_main
    ../../../../src/libphobos/libdruntime/rt/dmain2.d:419
0x45362a main
    /usr/lib/gcc/x86_64-linux-gnu/5/include/d/__entrypoint.di:62
0x7f4aace6260f __libc_start_main
    ???:0
0x453538 _start
    ???:0
0xffffffffffffffff ???
    ???:0

And here's the Packages.gz in question. It was generated by reprepro:

Packages.gz.zip

(I had to zip it to make github accept it as an attachment. The original file was Packages.gz, not Packages.gz.zip)

[neilmayhew]

For context, I'm on Debian Stretch, without libappstream-dev installed, and a copy of https://github.com/ximion/appstream built from 57917bd installed to /opt/appstream. I then run it with:

LD_LIBRARY_PATH=/opt/appstream/lib build/appstream-generator process ...

I used ldd to check that it's picking up the correct copy of libappstream.so.3.

I modified dub.json to enable it to pick up the locally-built libappstream during the build of asgen:

diff --git a/dub.json b/dub.json
index 52dac6b..3ca1a39 100644
--- a/dub.json
+++ b/dub.json
@@ -28,7 +28,7 @@
     "x-sourcePaths": ["source", "build/bindings"],
     "x-importPaths": ["build/bindings"],

-    "dflags": ["-Wl,--push-state,-no-as-needed", "-lcurl", "-Wl,--pop-state"],
+    "dflags": ["-Wl,--push-state,-no-as-needed", "-lcurl", "-Wl,--pop-state", "-I/opt/appstream/include/", "-Wl,-L/opt/appstream/lib/"],
     "systemDependencies": "LMDB, AppStream, GLib2, GIO, GObject, libarchive, GDLib, LibRSVG2",
     "libs": ["lmdb", "archive", "glib-2.0", "gobject-2.0", "gio-2.0", "appstream", "cairo", "gdk-pixbuf-2.0", "librsvg-2.0"]
 }

I tried doing it by setting PKG_CONFIG_PATH but for some reason gdc doesn't pass the -L option to ld properly and I get link errors (undefined reference to 'as_metadata_set_architecture').

[ximion]

That's fine ;-)
At Debian, we build the generator without patch, but wrap its call in a script which sets LD_LIBRARY_PATH to find a more recent libappstream than what is available in Jessie.

The cause of this bug was libarchive's auto-detection failing on 0-byte gzip compressed files. So I implemented a workaround for this now.
Since we will switch to using the XZ compressed files by default soon anyway, this shouldn't be an issue for long anymore :)

Thanks for reporting this!

[neilmayhew]

Thanks for the quick response! It's now fixed for me.

[neilmayhew]

FYI, I created an issue against libarchive: libarchive/libarchive#706

[neilmayhew]

Since we will switch to using the XZ compressed files by default soon anyway, this shouldn't be an issue for long anymore

Actually, libarchive has the same problem with xz-compressed empty files, and I've pointed this out in the issue.

[ximion]

Okay, then I need to investigate why it worked in my quick test.
Thanks for walking the extra mile on this and reporting the issue upstream and making a testcase! That's excellent style and highly appreciated!

[neilmayhew]

Based on the explanation in libarchive/libarchive#706 (comment), there is a better solution to this. See PR #8.