Base suite packages are scanned every time if the base suite isn't itself in the config

[iainlane]

After I fixed #31, I noticed that every run scans all packages in xenial:

2016-11-15 10:30:26 - INFO: Scanning new packages for xenial-proposed/main [amd64]
2016-11-15 10:30:35 - INFO: Scanned xchat-gnome/1:0.30.0~git20141005.816798-0ubuntu9/amd64, could be interesting.
2016-11-15 10:30:35 - INFO: Scanned gnome-session-common/3.18.1.2-1ubuntu1/amd64, could be interesting.
2016-11-15 10:30:35 - INFO: Scanned transmission-gtk/2.84-3ubuntu3/amd64, could be interesting.
2016-11-15 10:30:35 - INFO: Scanned gnome-system-monitor/3.18.2-1/amd64, could be interesting.

and so on for all packages in xenial.

I think it's because here we would normally skip the package if it's been seen before, but we don't because it's not in dstore. AFAICS that is because only the packages for the actual suite are processed.

Is that the bug? Do we need to process base suite packages? What do we need base suite packages for - just icons? If so, do they need to be in seedContentsData at all?

[iainlane]

Any thoughts? Just poked the logs and was reminded that this eats up a bit of time.

It'd make less work for the generator to drop the base suite from seedContentsData, but it would be safer to process base suite packages.

[ximion]

I need to look at this again, but I currently thing that the line pkgs ~= pkgIndex.packagesFor (suite.baseSuite, section, arch); is wrong there - the function should be called to load and cache the package data temporarily, but its result shouldn't be appended to the pkgs array, because that array will only contain the to-be-processed packages.
But I'll need to look at this again to be sure changing the line doesn't have unwanted other sideeffects.
Normally baseSuite is really just for finding icons, and for that asgen relies on contents information provided in the data store.
So I think the software might actually be operating correctly there, and that we maybe shouldn't allow to define a baseSuite that isn't listed in the config.
Or maybe there is a different, even better solution... Anyway, this I need to think about this again and toy around with a minimal archive when I am less tired ^^

[iainlane]

I think we can do something like process the base suite in a separate pass in seedContentsData, but only for cstore, no dstore or interestingFound. Will look more after the holidays.

[iainlane]

Testing the referenced branch on appstream.staging.ubuntu.com now.

[ximion]

That patch looks good in itself, and it looks sane for fixing the issue - sorry that I haven't followed up on this, but I was (and still am) overly busy with other things. At least the Debian freeze is now, so that means a bit less stuff to do :) (if there are no RC bugs)

Btw, there will be a new asgen release in a relatively short time, thanks to https://bugzilla.gdcproject.org/show_bug.cgi?id=251 I ignored a failing CI so the last release has had some preventable bugs (patched in the 0.9.2 Debian package, but providing a sane upstream tarball for other distros would be useful). For the .3 release I also want to add the option to move Media/Html/etc. directories to custom locations (would make some rsync'ing unnecessary)

[iainlane]

Works well - saves quite a few minutes - thanks!