Filtering misleading appstream components

[aleixpol]

I stumbled upon this issue:
https://bugs.archlinux.org/task/49439

In addition to the fact that packagekit::resolve(pkgids) fails if 1 package is missing (with alpm at least), I have discover failing on ArchLinux.

Would it be possible to filter those out at appstreamcli refresh time?

[ximion]

This is impossible to do at cache creation time, because:

• PackageKit is really slow, and the cache must be generated quickly
• Getting data for all the packages is really wasteful, it is much more efficient to have software centers load them on-demand (= for the stuff they really display on screen).
• The cache generation usually runs at times when we can't or shouldn't access the package cache, e.g. as part of a global package-cache update or as trigger/postinst script invocation during package installations.

This actually looks like a PackageKit bug, if you resolve something invalid we should tell you that, but still send back all the other results that resolved correctly.
Furthermore, it would help if Arch could update their data automatically, so it is more up-to-date... (but that's more a sidequest, the real problem is in PK)

[aleixpol]

Isn't the whole point of having a cache that the software center can focus on leveraging it rather than having to proof it?

I agree that:

• ArchLinux shouldn't provide a faulty entry
• PackageKit shouldn't plany refuse to give any packages if one is missing

It can be implemented in the software center as well for sure, but it goes against the principle of having this intermediate cache. If PackageKit is too slow, we can allow a way to ask the package manager directly.

[ximion]

Patches welcome for that - it's much more complex than you think ;-) This would basically mean that I need to replicate parts of PackageKit, with backends for different distributions, which I would rather like to avoid.
The point for having the cache is also not only to provide every information (well, it does aggregate stuff from different sources, if the data can be obtained quickly enough). It's rather:

• Be memory efficient by just caching the data required on the current system, and loading it quickly on-demand.
• Safeguard against faulty data.
• Provide very efficient and fast search functionality, with advanced features such as stemming.
• Combine different data sources and make them easily accessible for client applications

If I would add PackageKit resolve support to ascli, then apt update would deadlock, or in a less-worse case it would just take between 10sec up to 2min more time, which I really don't want to do.
You could say that I can do the resolving if the PackageKit backend is fast enough, but in that case client applications can not rely on information being in the cache, which is also something I really want to avoid.
And even if I would rebuild the cache with package-info, I would need to do this again and again with every package update that is done, so everytime the user installs or removes a package or updates the cache. That's an insane amount of rebuilds, and given the resolve performance, will lead to a very poor user experience.

Frontends don't have all of these issues, because they only need to resolve what the user can see, which is way faster with PackageKit.

[aleixpol]

Why do you need to refresh the database every time you un/install something?

[ximion]

I was thinking local packages there specifically, and also thought about caching version numbers / the installed state. If that isn't in the cache, you would need to call PK anyway in the client again, which makes this really wasteful, since the first resolve done for the cache rebuild isn't really useful if you have to resolve again in the client anyway.