Alternative, more resource efficient puller

[calmh]

When implementing the new puller, we could keep in mind that it might be useful with a completely different, less resource intensive but also less efficient algorithm.

The current puller, and the future graph based one, will require keeping essentially the entire file set in memory. We need this for ordering (sorting), dependencies, rename detection, and so on.

But we could also use an altogether simpler approach where we don't do rename detection or dependencies or anything fancy and just directly work out of the database, item by item. It'll suck in some respects (delete and re-download instead of rename, fail and have to retry for some changes, etc). But it could also work in a very small amount of RAM.

As an option. [X] Use a sucky puller but spare my RAM

[AudriusButkevicius]

or the graph could be on disk.
the sucky puller is pretty much the existing puller, minus some stuff that goes into ram (such as the job queue)

[calmh]

That's actually a good point. Maybe, possibly, we could use something like the disk-spilling db thing we use for sorting to keep the small cases fast and the large cases memory efficient.