GitLab kref #3661
GitLab kref #3661
Conversation
HebaruSan
added
Enhancement
Pull request
Spec
| /// <summary> | ||
| /// True to use source ZIP for a release. | ||
| /// Note that this MUST be true because GitLab only provides source ZIPs! | ||
| /// If they add other assets in the future, this requirement can be relaxed. | ||
| /// </summary> | ||
| [JsonProperty("use_source_archive")] | ||
| public readonly bool UseSourceArchive = false; |
There was a problem hiding this comment.
I might be misunderstanding the help text here, it says MUST be true and we're setting it to false?
There was a problem hiding this comment.
Yeah, it's kind-of supposed to be weird. The idea is to force all GitLab netkans to add the below metadata:
x_netkan_gitlab:
use_source_archive: true... until we can add the option of removing that. The = false default above only takes effect if the value isn't set in the netkan.
Let's look at this repo's releases to illustrate why:
https://gitlab.com/djdisodo/kspremote/-/releases
The app-debug.apk is the packaged file that the user is meant to download, but it's not provided by the API in a machine-readable form; it's just a Markdown-formatted link embedded in the description field:
https://gitlab.com/api/v4/projects/djdisodo%2Fkspremote/releases
[
{
"name": "release-1.1.0",
"tag_name": "release-1.1.0",
"description": "[app-debug.apk](/uploads/4ae81fe659c9d621ee64099522324c31/app-debug.apk)",
"created_at": "2020-05-01T10:40:09.568Z",
"released_at": "2020-05-01T10:40:09.567Z",
"upcoming_release": false,
"author": {
"id": 2563867,
"username": "djdisodo",
"name": "lee jisoo",
"state": "active",
"avatar_url": "https://secure.gravatar.com/avatar/ed5d23ffb1070c0ed33cc8c6dead8d11?s=80&d=identicon",
"web_url": "https://gitlab.com/djdisodo"
},
"commit": {
"id": "a4a7d1112b2953dbcfa113fa9c68c8aebd3f751d",
"short_id": "a4a7d111",
"created_at": "2020-05-01T19:31:16.000+09:00",
"parent_ids": [
"66388dcd8b8881107b50e13545bfa6b4461b209a",
"6359357ea2f3d18157bf04b18a63d6988d971616"
],
"title": "Merge branch 'master' of https://gitlab.com/djdisodo/kspremote",
"message": "Merge branch 'master' of https://gitlab.com/djdisodo/kspremote\n",
"author_name": "djdisodo",
"author_email": "djdisodo@gmail.com",
"authored_date": "2020-05-01T19:31:16.000+09:00",
"committer_name": "djdisodo",
"committer_email": "djdisodo@gmail.com",
"committed_date": "2020-05-01T19:31:16.000+09:00",
"trailers": {},
"web_url": "https://gitlab.com/djdisodo/kspremote/-/commit/a4a7d1112b2953dbcfa113fa9c68c8aebd3f751d"
},
"commit_path": "/djdisodo/kspremote/-/commit/a4a7d1112b2953dbcfa113fa9c68c8aebd3f751d",
"tag_path": "/djdisodo/kspremote/-/tags/release-1.1.0",
"assets": {
"count": 4,
"sources": [
{
"format": "zip",
"url": "https://gitlab.com/djdisodo/kspremote/-/archive/release-1.1.0/kspremote-release-1.1.0.zip"
},
{
"format": "tar.gz",
"url": "https://gitlab.com/djdisodo/kspremote/-/archive/release-1.1.0/kspremote-release-1.1.0.tar.gz"
},
{
"format": "tar.bz2",
"url": "https://gitlab.com/djdisodo/kspremote/-/archive/release-1.1.0/kspremote-release-1.1.0.tar.bz2"
},
{
"format": "tar",
"url": "https://gitlab.com/djdisodo/kspremote/-/archive/release-1.1.0/kspremote-release-1.1.0.tar"
}
],
"links": []
},
"evidences": [],
"_links": {
"self": "https://gitlab.com/djdisodo/kspremote/-/releases/release-1.1.0"
}
},
{
"name": "release-beta4",
"tag_name": "release-beta4",
"description": "[app-debug.apk](/uploads/5db2c5ac589d55abd37c4dcd18942f6e/app-debug.apk)",
"created_at": "2020-02-15T17:13:28.144Z",
"released_at": "2020-02-15T17:13:28.134Z",
"upcoming_release": false,
"author": {
"id": 2563867,
"username": "djdisodo",
"name": "lee jisoo",
"state": "active",
"avatar_url": "https://secure.gravatar.com/avatar/ed5d23ffb1070c0ed33cc8c6dead8d11?s=80&d=identicon",
"web_url": "https://gitlab.com/djdisodo"
},
"commit": {
"id": "6359357ea2f3d18157bf04b18a63d6988d971616",
"short_id": "6359357e",
"created_at": "2020-02-15T16:51:00.000+00:00",
"parent_ids": [
"af7fa415f0fd741c8ab5894dec7f8b9d09e5d746"
],
"title": "Update README.md",
"message": "Update README.md",
"author_name": "lee jisoo",
"author_email": "djdisodo@gmail.com",
"authored_date": "2020-02-15T16:51:00.000+00:00",
"committer_name": "lee jisoo",
"committer_email": "djdisodo@gmail.com",
"committed_date": "2020-02-15T16:51:00.000+00:00",
"trailers": {},
"web_url": "https://gitlab.com/djdisodo/kspremote/-/commit/6359357ea2f3d18157bf04b18a63d6988d971616"
},
"commit_path": "/djdisodo/kspremote/-/commit/6359357ea2f3d18157bf04b18a63d6988d971616",
"tag_path": "/djdisodo/kspremote/-/tags/release-beta4",
"assets": {
"count": 4,
"sources": [
{
"format": "zip",
"url": "https://gitlab.com/djdisodo/kspremote/-/archive/release-beta4/kspremote-release-beta4.zip"
},
{
"format": "tar.gz",
"url": "https://gitlab.com/djdisodo/kspremote/-/archive/release-beta4/kspremote-release-beta4.tar.gz"
},
{
"format": "tar.bz2",
"url": "https://gitlab.com/djdisodo/kspremote/-/archive/release-beta4/kspremote-release-beta4.tar.bz2"
},
{
"format": "tar",
"url": "https://gitlab.com/djdisodo/kspremote/-/archive/release-beta4/kspremote-release-beta4.tar"
}
],
"links": []
},
"evidences": [],
"_links": {
"self": "https://gitlab.com/djdisodo/kspremote/-/releases/release-beta4"
}
}
]If we want to index a mod from there using the API, we have 4 choices: source zip, source tar.gz, source tar.bz2, and source tar. That's why it says in various comments here that we can only support source archives for GitLab (and why we couldn't index that specific mod, even after this PR). The hope is that someday GitLab will improve their API to support non-source assets, and if/when that happens, we can remove this limitation.
| this.http = http; | ||
| this.token = token; |
There was a problem hiding this comment.
Do we need to to setup a gitlab account for the indexer? Or will we get by until we reach their API limit?
There was a problem hiding this comment.
I think we will want to create a GitLab token for this just for completeness/robustness, but I tested it without a token and it still worked. If I'm reading this table correctly, a token boosts the rate limiting from 500 to 2000 requests per minute:
https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits
Motivation
See KSP-CKAN/NetKAN#9325, the author of a mod hosted on GitLab has requested to have it added to CKAN. Currently we don't have a way to do that without hard coding the download URL, which is bad for maintenance.
Changes
netkan.exe --gitlab-tokencommand line option is now added and works just like--github-tokenbut for GitLabPurgeDownloadsfails after a failed inflation, we ignore it so the original error can be printed#/ckan/gitlab/<account>/<project>is added for indexing GitLab projectsGitlabTransfomeruses aGitlabApiand accompanying data classes from aNetkan/Sources/Gitlabfolder to inflate modules with GitLab krefsAfter this, we can inflate a netkan for the mod from KSP-CKAN/NetKAN#9325.
Fixes #2554.
Known limitations
Whereas GitHub allows the user to custom-build a special ZIP and attach it as a release asset, GitLab appears to only support source ZIPs (which we can support for GitHub as well using
x_netkan_github.use_source_archive). This means that mods with build steps such as compiling a plugin may not be indexable from GitLab, but some mods without such steps can be correctly installed from a source ZIP if the repo is structured the same as the installed folder.While it would be great if GitLab added support for non-source assets, we don't have to wait for that to get some value out of a GitLab kref. However, we do need to make sure that any future changes don't break the mods we index in the meantime. For this reason,
x_netkan_gitlab.use_source_archivemust be set totruefor now! This will ensure that we only index GitLab mods that can be properly installed from a source ZIP, while allowing for future enhancements to the API for non-source assets.