I understand that speed and cost are an issue for generating metadata. How would it be if you only looked at newly-added packages? You already have data about the existing packages cached in the database, so they don't all need to be rescanned. The job that processes incoming packages could tell asgen which packages to consider by passing it the .changes files to look at. That would avoid the expensive loading of indices. (This would be a good option in my infrastructure, where we could call asgen from the cron job that scans the incoming directories for changes.)

It would still be necessary to regenerate the whole of the Components and icons files, but this would be done from cached data in the database and would be considerably less expensive than reading the indices since there are far fewer apps than packages. Alternatively, it could be done by a cron job, but that could be run more frequently since this part of the operation is less costly.

I propose that the current process operation be split into two separate operations, scan and generate. The scan operation would take a list of .changes or .deb files to look at. There would still be a process operation to scan an entire suite, but it would use these two other operations internally.