Feature Request: Multiple KSP Instances -> share downloads

[valuial]

I have more than one KSP instance, at least one Main copy, a testing copy for bug-hunting, and soonish one to test new mods for compatibility without feeding my saves to the Kraken.

Most mods are shared between them - even in the same versions.
I'd like ckan to store the downloaded archives in a centralized location (config setting for this? env i.e. $CKAN-DOWNLOADS) so I don't have to download a mod 3 times, when one time is enough, also saves HD storage.

Short term workaround could be to have ckan look in the other known installs for the required download.

[mgsdk]

Duplicate of https://github.com/KSP-CKAN/CKAN-core/issues/10, which should probably be moved over here (or will be as the repos merge again).

[mgsdk]

I can also see I had some old code for this, might need a bit of checking before it can be merged in. For anyone who wants to see it: https://github.com/mgsdk/CKAN-core/tree/cache_downloads.

[valuial]

I didn't mentioin it explicitly, but I wanted to imply using metadata across KSP install paths as well.

[mgsdk]

So you want to update the repository once, and have it updated for all KSP installs?

[valuial]

Yes. Unless there is a reason to have a different metadata set for a KSP install? Then there should be some way to configure this I guess.

[Dazpoet]

Just throwing it out there that some of us use different repositories for different installs, e.g. my main install use the standard and MechJeb2-dev repos while some of my testinstalls use all the canonical repositories. If the same metadata list was used for all installs wouldn't this behaviour become impossible?

[pjf]

Sharing metadata between installs is harder than it looks, not least because one may have different (and multiple) upstreams for each install.

We do have some potential ways to make it more efficient to fetch metadata. As a first pass, flipping from .zip to .tar.gz downloads (I know we've got a ticket for this somewhere), but further in the future we'll likely have a bot that cooks the metadata to only return compatible and/or latest versions.

But seriously, switching to .tar.gz will drop metadata download size by a lot, as it seriously takes advantage of the repetition that exists between files.

[netkan-bot]

Hey there! I'm a fun-loving automated bot who's responsible for making sure old support tickets get closed out. As we haven't seen any activity on this ticket for a while, we're hoping the problem has been resolved and I'm closing out the ticket automaically. If I'm doing this in error, please add a comment to this ticket to let us know, and we'll re-open it!

[valuial]

@netkan-bot s/automaically/automatically/

[politas]

This would be good to have. The number of times I've had CKAN crash while downloading the same files I know I already have downloaded for another install...

[valuial]

Not sure if metadata is required, but what about only sharing the mod downloads? Downloaded that mod for another instance, so you just get the file from the common folder, and merge it with the active instance...

[godarklight]

I already do this on linux by symlinking the downloads/ folder

[politas]

If installs have different upstreams, wouldn't that make the hashes different, anyway, so the downloads should co-exist without issues, just as different KSP versions will likely use many different downloads. So even if metadata sharing isn't possible, mod download sharing is?

[HebaruSan]

A crazy thought just hit me: It ought to be possible to approximate this without re-architecting by simply making the cache search span all the caches. So if I have three instances, and I want to install ModuleManager-3.0.7, I look in all three of those instances' caches for it (using the current algorithm of looking for a file with a matching 8-char hex prefix). If it's found, install from that file; otherwise download and save in the cache of the current instance.

This would eliminate redundant downloads and reduce disk usage, just like a true "global" cache. And we wouldn't have to worry about merging separate caches or making the path configurable, etc. The saving code could be left completely alone.

I'd still prefer a clean scheme where all instances share one configurable cache folder, but this option does intrigue me.

Well look at that, the OP does mention that. Just ignore me...