Netkan: Self-Published Metadata

[dbent]

This proposal is to allow mod authors to self-publish authoritative metadata for their mods by creating recursive .netkan files. The way I envisage it working is thusly:

A .netkan file is created in the NetKAN metadata repository as follows:

{
    "spec_version": 1,
    "$kref": "#/ckan/http/https://www.example.com/AwesomeMod.netkan"
}

This .netkan is actually recursive, it simply references another .netkan file, except this one is published by the mod author.

I've chosen to overload the HTTP $kref here, and indicate the recursive mode by having the URL end with .netkan as a matter of convention. If you want to be more explicit, a new $kref scheme could be used:

{
    "spec_version": 1,
    "$kref": "#/ckan/netkan/https://www.example.com/netkan?mod=AwesomeMod"
}

In either case, the NetKAN bot would download the .netkan periodically and use it exactly as if it were in the NetKAN metadata repository. Within their own .netkan file, Mod authors could choose to specify a Kerbal Stuff $kref or a GitHub $kref, or just use all explicit data.

For added flexibility the original .netkan (henceforth referred to as the "MetaNetKAN"), could append/override data in the Mod-authored .netkan. So for instance, if a mod author adds a ksp_version_min but fails to add a ksp_version_max, and subsequent updates cause the mod to break, a CKAN team member could change the MetaNetKAN to be:

{
    "spec_version": 1,
    "$kref": "#/ckan/netkan/https://www.example.com/netkan?mod=AwesomeMod",
    "ksp_version_max": "x.y.z"
}

And all is well.

The main benefits I see this having are twofold:

1. Mod authors have more control and responsibility for their mods, which means more robust, accurate, and timely metadata. This also means mod authors can be more proactive about issues and less reliant on the CKAN team.
2. CKAN team spends less time keeping on top of changes to upstream (a problem a lot of Linux distributions have).

[dbent]

As a side note, under this scheme it may be considered best practice (but not required) that the MetaNetKAN file contain a hardcoded identifier field, so a malicious mod author can't "spoof" another mod.

[birdspider]

this screams for a check on lookup-circles/loops [A->B->C->A->B...] and/or limits to the recur-depth

[dbent]

this screams for a check on lookup-circles/loops [A->B->C->A->B...] and/or limits to the recur-depth

I should have made it explicit, but my assumption is that the NetKAN bot would refuse to follow a MetaNetKAN more than one level (if the result of fetching from a MetaNetKAN is another MetaNetKAN, the code should bail out, set the server on fire, and walk away in slow motion).

[pjf]

I ducking love this. I'd tweak it a little so we have some way of the author providing version-ranged metadata, so they can update their .netkan file first, but not have that get picked up by the bot until the actual corresponding release that goes with it, but that's really a nice-to-have are than an essential. This is better than having mod authors embed .ckan files in their releases, since the metadata can be adjusted after a release.

The main gotchas are that if a mod author becomes unresponsive or uncooperative then we'll have an increased workload because users will keep reporting bugs to us. In that case I'd guess we'd have to resume control of the metadata, but I imagine the most users of this will be the really awesome mod authors who already send us PRs when they make new releases, so this really just saves everyone a step.

Adding this to our enhancement queue.

[BasharMilesTeg]

I think this would be especially useful to @Ezriilc and possibly anyone who uses http-style .netkans as they could write a new version of their self-hosted .netkan as part of their release process. I think it might even be useful for ModuleManager-dev but that requires more thought.
In addition, using HTTP headers ('If-Modified-Since' or 'Last-Modified' maybe?) would valuable to consider, depending on server support.

[dbent]

Implemented in #1218.