Improve options for private app package distribution

[ocdtrekkie]

Someone on IRC would like to be able to pre-install a private app for all users on their server. This is hard because Pre-installed Apps depends on the app index to populate available apps, and likely needs the app index to provide the correct packageId to preinstall. My guess is that even MongoDB-mangling will not accomplish this goal.

The ideal case is probably for the user to operate their own App Index, but this strategy has flaws. Presently, a Sandstorm server can use a single App Index, and the App Index contains all of the data for all app packages the Index knows about including historically. I believe the official one is about 40 GB... mostly of Wekan updates. If one was to run their own App Index, they'd have to manage getting all updates for all apps they want users to be able to use into their private App Index and approving them, which is useful for enterprise IT, but burdensome for "I want to offer one private app".

I think the best case is to fix this with an App Index feature: The App Index should have an option to configure an "upstream App Index". When it does this, it should presumably copy the entire upstream index metadata including approvals. Private apps could then be spk published to such a downstream App Index.

I think in the ideal case, a downstream App Index should not actually have to store all SPK data for all versions of known packages (though it should be able to do so). It should either direct a Sandstorm server to pull the package from the upstream App Index or it should cache the package from the upstream App Index the first time it's requested through the downstream Index.

Finally, if we are to support server owners operating their own App Index, we probably should also make the jump to publishing the App Index in the Market, so that servers who maintain their own App Index receive updates to it.

[jimpick]

I wanted to see if I could get one of my "private" apps to appear on the "Admin / Pre-installed apps" screen. (I want to pre-populate sandstorm machines with some custom software before I invite people to use them)

As an experiment, I wrote a simple Node.js reverse proxy with Fastify that fetches then modifies the /apps/index.json response to include my package, and also provides a route to download the spk file. I put the proxy behind a SSL reverse proxy on a DNS name I control (using Caddy and Let's Encrypt).

https://github.com/jimpick/sandstorm-app-index-playground/blob/main/proxy/index.js

I also had to hack the interval for the "updateAppIndex" scan in Sandstorm itself to pick up the changes (it only scans once a day).

sandstorm/shell/imports/server/00-startup.js

Line 70 in 50c6ae7

SandstormAutoupdateApps.updateAppIndex(globalDb);

Finally, I was able to see my app in the "Admin / Pre-installed apps" screen, and it shows up in the list of installed apps for new users!