RFC: Automated CKAN file generation.

[pjf]

Discussion

I've been asked many times how one might go about adding CKAN support into existing build processes. This is essential for everyone; updating things by hand is unnecessary and error-prone.

Embedding the CKAN files in the archives themselves is an option, and easy for us, but it also sucks because it means mod authors have to manually update the version number.

A much better system is to hook into existing release systems. The big ones¹ are:

• Github Releases ( Allow CKAN files to be generated from github releases #2 )
• KerbalStuff ( KerbalStuff support #69 )
• AVC

I propose that authors can include a CKAN fragment file in their releases, which is essentially all the CKAN meta-data, but without the data we can derive from the release system (which is at least the version number for Github, and potentially the KSP target for KS and AVC).

Since most CKAN data changes rarely, the vast majority of mods will just include their fragment file and forget about it.

Spec

I'd suggest the following fields:

• version_from
• ksp_version_from

Each of which is an enum that may contain the values Github, KerbalStuff, or AVC. In each case, we enforce that the appropriate details are listed in the resources section, so our bot knows where to look.

It would be an error to have both version and version_from, and ksp_version and ksp_version_from in the same file.

After compiling the new file from the fragment and discovered meta-info, we would verify that it validates, and has the identifier we expect it to, before automatically adding it to the CKAN.

Questions:

• Does this sound like a good idea?
• What do we call the fragment file? (Not .ckan, because it won't have a full set of meta-data. .ckaf maybe ?)
• Any other comments / suggestions?

---

¹ I don't believe curse gives us APIs, or makes our lives easy, but I'd be delighted to find out otherwise.

[pjf]

Particularly interested in thoughts from @NathanKell, @erendrake, @godarklight, and @BobPalmer, should any of you find the time. ;)

[pjf]

Looking at KerbalStuff, the following are all potentially auto-generate friendly fields:

• abstract
• name
• author

[godarklight]

I don't use github releases but I could, I believe KS was planning github integration in some form as well - so this seems like a good option for open source mods.

I use kerbalstuff, but not everyone does. The good news with KS is that it's open source, and you might be able to get KS to generate CKAN files for you ;)

You'll also have to think about dev builds (if you decide to support them). Atm I'm in a situation where I'm pushing 1-3 updates a day, so I don't mind adding a little script to spit out a json file for dev builds.

[pjf]

@godarklight : For dev builds, the spec currently has a release_status field, which allows something to be marked as stable or not. (Currently it's 'alpha', 'beta', 'stable', but it will probably change to 'stable', 'testing', 'unstable', and 'experimental' to match the Debian standard, which gives these well-defined meanings).

Unstable and experimental releases probably won't go in the main repo; but I'd like to see users able to ckan repo add DMP/testing, which simply means a ckan update will grab an additional set of metadata, which can include those releases. In this way mod authors and braver users can test in-progress releases, but the typical user has no chance of ending up with them by accident.

Longer term, having KS generate CKAN files for us would be ideal, as we can then have KS as one of the default repos the default client can read from. Tagging @SirCmpwn.

[pjf]

I'm now wondering if we can just leave the embedded file as a .ckan file (rather than some funny syntax), and instead simply use JSON pointers inside it. For example:

{
    "spec_version"   : 1,
    "name"           : "#/kerbalstuff/name",
    "identifier"     : "MuchRocketSuchBoomWow",
    "abstract"       : "#/kerbalstuff/short_description",
    "download"       : "#/kerbalstuff/download_path",
    "license"        : "CC-BY",
    "author"         : "#/kerbalstuff/author",
    "version"        : "#/kerbalstuff/friendly_version",
    "ksp_version"    : "#/kerbalstuff/kspversion",
    "install" : [
        {
            "file"       : "Doge",
            "install_to" : "GameData"
        }
    ]
}

KS doesn't seem to export the license in its API, but I believe that's just an oversight.

One of the nice things about this is that we can process it using existing tools, and if we're introducing JSON references, then we can just include a reference that populates all the other fields. This means the actual file mod authors will need to include will instead look like this:

{
    "spec_version"   : 1,
    "identifier"     : "MuchRocketSuchBoomWow",
    "$ref"           : "#/ckan/kerbalstuff",
    "install" : [
        {
            "file"       : "Doge",
            "install_to" : "GameData"
        }
    ]
}

That gives me a happy just looking at it.

[AlexanderDzhoganov]

Please keep in mind that it is unfair to Kerbal Stuff (or any other host) that we use their bandwidth without redirecting the user to the website. Worst case scenario, they do something so you can't hotlink files, which would be detrimential to the community. It may be best to host our own repository and just scrape kerbal stuff and such for new releases.

[ddevault]

We wouldn't have made the API if we didn't want people to use it. You're fine.

[pjf]

@AlexanderDzhoganov : Oh! The generated CKAN files will be hosted in our own repo; although even the largest only come in at ~ 2kb, so 1000 files is only 2Mb uncompressed. We'll just pinging the KS for new releases of mods which we know include CKAN meta-files. That will involve downloading the mod the first time we see it, to generate the download size, fingerprint, and of course to extract the embedded CKAN fragment.

If number of downloads from KS becomes an issue, I would personally go hunting for sponsorship for a either a KS or a CKAN mirror¹. As in I will start pinging my personal contacts; as a FOSS project that's distributing (F)OSS mods, I don't view this as an insurmountable problem.

@SirCmpwn : You're the best! Thank you! :)

---

¹ The possibility we may need to have a CKAN mirror for all the mods themselves is one of the reasons why we have mandatory license fields.

[pjf]

Just a quick note that I'm changing $ref to $kref in our format. Actual JSON pointers replace themselves with what they point to, however this could potentially result in duplicate keys in our documents. That would normally be fine, because we have a reference first, and then have overrides for it afterwards, but this causes a problem for the resources field where we're likely to only want to override certain sections. We know that resources is going to get extended a lot, which is why it has its own little structure.

A $kref field (Kerbalised Reference) will instead merge the data it pulls in, and won't overwrite existing fields. This means we can keep the fragment files easy to understand, rather than requiring mod authors to insert multiple references if they want to supply their own homepage or other details.

Under the hood, we may end up converting $kref fields into a series of $ref fields, and then using standard tools on that.

[pjf]

The docs for this are written up, and we support GitHub and KS now. AVC is a nice to have, but it has its own ticket, so I'm closing this unless anyone thinks it should remain open.