alpine: Add support for remote repositories

[Cogitri]

Hello,

our (Alpine's) infra team has notified me that we may receive new, tiered mirrors in the future, meaning that we won't be able to just mount the entire repository as a folder into a docker container to run asgen on it. As such they suggested something rsync (or something similiar), so that asgen still only downloads changed packages, but doesn't need the entire repository available to generate metadata. I'm not sure how much effort it'd be to implement a different storage backend in asgen and I guess this is kind of a special requirement to Alpine, so I'm not sure if this is something that you'd want in asgen though :)

[ximion]

This is already supported, but unfortunately not in a generic way (so this needs explicit support of the respective backend).
In Debian, we have a read-only network mount for the whole archive, which is a good and transparent choice to not require to mirror the whole archive (in most cases, even a normal sshfs mount will do).
Canonical however is running their instance of appstream-generator on their cloud infrastructure. There, asgen will download all the stuff it needs on demand and throw away downloaded packages after extraction.
All of this support is implemented in the Debian and Ubuntu backends, so all Alpine would need is to implement the same (which should be a relatively easy task, as all needed building blocks should be in the common API).

See https://github.com/ximion/appstream-generator/blob/master/src/asgen/backends/debian/debpkg.d#L101 for some of the needed code (both the archive index use the global Downloader instance to both download files on demand to the temporary directory of the current run, and also clean up the packages once they are no longer needed (so disk usage doesn't get excessive for bigger runs))

[Cogitri]

Oh great, that makes things easier :)

There, asgen will download all the stuff it needs on demand and throw away downloaded packages after extraction.

Yup, that was the problem for us as well. Am I understanding the Downloader code correctly that it'll just fetch all packages via HTTP(S) too? I guess that won't be too much of a problem though, diskspace is more of a limiting factor to us than bandwidth

[Cogitri]

Fixed by #80