Random icons missing from exported data

[antonio-rojas]

Every time I generate metadata for Archlinux results in some 10/15 icons missing from the exported data. The missing icons are different every time, so it doesn't seem to be a problem of any specific package but rather some kind of race condition. There are no visible errors in the asgen output.

Attached is the used asgen-config.json. Best way to reproduce it is to run the generator a few times in a row, and do a diff on the generated icon tarballs.
asgen-config.json.txt

[ximion]

A verbose log of the generator as well as the resulting icons tarball would be very helpful here. Are the icons only missing from the tarball, or also from the pool and OS suite directories?

[antonio-rojas]

Attached requested data. Some missing icons this time are kdesvn_kdesvn.png and kfloppy_kfloppy.png (in 48x48 size, other sizes are ok). The icons are present in the pool, only missing from the tarball.

verbose.log.txt
extra-icons-48x48-20210905.tar.gz
extra-icons-64x64-20210905.tar.gz
extra-icons-128x128-20210905.tar.gz

[ximion]

Can you maybe also attach the generated metadata and possibly the media pool as well? That's a lot of data, but it will be helpful to determine if there is an error and if so, in which component it likely is.
If the icon is in the pool but not in the icon tarball, then that's likely an issue with tar creation, which isn't threaded at all, so that would be odd.

[antonio-rojas]

Here's the full export dir
https://pkgbuild.com/~arojas/export.tar.gz

[antonio-rojas]

More info, which may or may not be useful:

• Adding some debug output shows that the icons are already missing from the loop that adds them to the tarball (which rules out a problem with libarchive)
• Compiling asgen with ldc or dmd doesn't make any difference (which rules out a compiler bug)
• Tried reverting 2eb3d1b, which looks like it could be related: no difference.

[antonio-rojas]

I removed threading in exportData:

diff --git a/src/asgen/engine.d b/src/asgen/engine.d
index ee81861..11ccae5 100644
--- a/src/asgen/engine.d
+++ b/src/asgen/engine.d
@@ -368,7 +368,7 @@ public:
         string[string] cidGcidMap;
         bool firstHintEntry = true;
         logDebug ("Building final metadata and hints files.");
-        foreach (ref pkg; parallel (pkgs, 40)) {
+        foreach (ref pkg; pkgs) {
             immutable pkid = pkg.id;
             auto gcids = dstore.getGCIDsForPackage (pkid);
             if (gcids !is null) {

and didn't experience any icon loss in 5 consecutive asgen runs (and didn't notice any performance loss, this step is almost instant). As soon as I went back to unpatched asgen it started losing icons again.

[ximion]

@antonio-rojas Can you check if this commit solves this for you? I did some testing here and didn't see in 40 runs, but even before this was a very rare issue on my machine (got it only once in 40 runs...).
iconTarFiles not being protected was definitely a bug, and it would absolutely explain the missing icons - since I'm at a conference for the first half of next week I originally wanted to write a "be patient, I'll get to it" reply, but you basically pointed me right at the (or at least a) issue, thanks for all the investigation work!

[ximion]

Btw, in your config file disabling validation is not ideal, as it may hide issues with your metadata - the issue why you disabled it in the past should have long been fixed by now :-)

[antonio-rojas]

Yes, this works, thanks for the quick fix!