The index: how the catalogue is stored, updated, and fetched

[Maximilian-Nesslauer]

The scope RFC (#25) sets the constraints of how the catalogue is stored, updated, and fetched:
an index format of our own definition, free infrastructure only, and no hosting of content files.

Prompted by @KlaasWhite's question on Discord: "My biggest question at the moment is if its still the plan to do some form of desentrelized index or have it central in some way" .

This discussion settles the shape of the index within the constraints of the scope RFC before it becomes an RFC.

Please reference the numbers so the thread stays followable.

1. One canonical index, or many?

My position: one canonical community index, with clients free to consume additional sources.

The argument for one canonical catalogue:
mod identity is the folder name, the game overwrites anything a file declares (Mod.MakeUsing, see research/ksa-mod-loading.md), so the id namespace is global and needs an arbiter for uniqueness.
It is also what makes dependency resolution possible without downloading archives first: SpaceDock cannot answer "what does this mod depend on" without fetching the archive and reading mod.toml since SpaceDock itself just does not carry that metadata.

The argument against going fully decentralized: without a shared catalogue there is no uniqueness arbiter, no offline resolution, and discovery becomes crawling.

Multi-source stays a client capability either way (Borea already tags every entry with its source), so this is about which index is authoritative.

2. Storage layout

The reference is the authored/generated split (see research/prior-art-ckan.md): one small authored template per mod, one generated file per release, produced by tooling.

Open: one repository or two?
Branch protection is per branch, not per folder, so "humans need review on authored files" and "the bot pushes generated files unattended" do not mix well in one repo, and bot commits bury the authored history.

3. The fetch contract

Clients need a stable URL that survives storage reorganization.
Open: full-index download (the CKAN tarball model), per-file fetch, or both.
Published metadata must stay evaluable offline, consistent with RFC 0017's self-contained compatibility data.

4. How it stays current

A watcher bot picks up new releases from GitHub and SpaceDock and commits the generated files.

Manual pull requests remain the fallback and the path for mods hosted anywhere the bot does not watch.

5. Id uniqueness policy

If the index arbitrates ids: first come, first served, uniqueness case-insensitive (Windows folders are case-insensitive, Linux folders are not), casing preserved for display.
The full id rules belong to the metadata format RFC (#15); here the question is only that the index enforces them at publish time.

Push back on any of these, and add what I have missed.

[MrJeranimo]

1. One canonical index, or many?

One for the stated reason of the indexer being able to guarantee unique Mod Ids. I would assume a centralized index would make it easier on other creators as they only need to go to one place and learn only one system.

If we want to be more decentralized, this could be an index with just basic mod metadata, so no per-version metadata, and that's it. Essentially could be where you use for searching and guarantee that your Mod Id is unique.

4. How it stays current

Would these bots be able to detect if a version of a mod needs new dependencies through these sources?

My thoughts are that there is a tool for creating a mod and uploading a new version of a mod that can check for missing/incorrect data, and can potentially upload the mod to SpaceDock per SpaceDock's API (if they don't use GitHub for releases).

This tool could potentially be added to Borea as an additional package that the user must explicitly install as it would not be included in the base release. To make it easier to find, there can be a notice/way to download the package while using Borea if you try to create/update a mod. As for tracking who owns a mod, link it to the GitHub account (either listed in the metadata or through commit history) so Borea doesn't need to handle account verification, just a "Link GitHub" and maybe "Link SpaceDock" or something.

This format takes more effort on the creator, but I think it can be streamlined enough to make it worth it for consistent and correct data in the indexer.

5. Id uniqueness policy

We can use the indexer as the source of truth.

Additionally, the KSA Forums as a potential helper in disputing claims. This would be if in #15 the KSA Forums Post metadata property is accepted as required as it means all mods must have their creation announced on the forums. In the case someone uploads to the forums but not the indexer, this can be used to solve Mod Id disputes where the Forum Post that came first gets the Mod Id.

[Maximilian-Nesslauer]

On 4, your bot question: yes partially.

The watcher downloads each release archive anyway to compute the checksum, and the [StarMap] section of mod.toml is machine-readable, so new or removed code dependencies ([[StarMap.ModDependencies]], including Optional) are detectable per release automatically.

What a bot cannot derive would be the version bounds (the StarMap section deliberately carries none as per @KlaasWhite comment in discord), non-code relationships like conflicts or "use X instead".

So the model I would aim for is a bot-derived baseline with authored overrides on top.

That also answers a point @averageksp raised on Discord, that in CKAN somebody has to hand-write dependencies and install_to.

Install directives are derivable too when the archive has the standard layout, a top-level folder containing mod.toml; only unusual layouts need authoring.

[Maximilian-Nesslauer]

Three points from Discord so they do not get lost.

6. Dependencies that are not listed

@averageksp raised this: in CKAN, a mod cannot be indexed while one of its dependencies is unlisted, which guarantees resolvable installs but also creates the review queue he deals with daily.

I would not copy the hard block.
My leaning: allow the listing, record the dependency by id, and mark it unlisted.
Clients treat it like unknown compatibility, warn, point at wherever the author says the dependency lives, and let the user proceed.

7. The review and moderation model

Also from @averageksp since he has experience working for CKAN. https://discord.com/channels/1439383096813158702/1532471092252381244/1533944347597213787
Putting the sketch from Discord here for the record:

• A new release of an already listed mod needs no human: the watcher commits the generated file unattended, the same way builds.json updates itself today.
• A new listing is validated automatically, schema, id rules, ownership via the linked account, archive inspection, license present, and green plus verified ownership auto-merges. Listing your own mod is self-service.
• Humans handle only disputes, takedowns, and delisting.

The moderation duty that comes with even a metadata-only index would be that we need a documented takedown path and the ability to delist an entry instantly.

[Maximilian-Nesslauer]

Closing out my own open points so this can become the index RFC.

2. Storage layout: two repositories

I have come around to two repositories, and I would fix that now.

• An authored repo holding the authored TOML files, one per listing. Humans write here, branch protection and CODEOWNERS apply, and the self-service listing flow (point 7) auto-merges a validated new listing.
• A generated repo holding the stamped release files, one JSON file per release. Only the watcher writes here, unattended, the same way builds.json commits itself today.

The reasons, mostly said in the opening post but now with a decision attached:

• Branch protection is per branch, not per folder, so "humans need review" and "the bot pushes unattended" cannot both hold inside one repo.
• the authored history stays clean and reviewable.
• Release commits outnumber authored commits by an order of magnitude, and in one repo they bury the history of the half humans actually need to read.

The cost is that a listing's data lives in two places joined by id.

3. The fetch contract

Clients get one stable URL serving a single snapshot artifact, and never read the repos directly.

• A scheduled job in the generated repo builds a snapshot of the whole index, authored plus generated files merged into one fetchable artifact, and publishes it at a stable address.
• Clients poll that address with normal HTTP caching (ETag / If-None-Match), so an unchanged index costs one 304.
• Per-file raw fetch stays possible as a secondary path for tooling, but it is explicitly not the contract, so we can reorganize storage without breaking anyone.

I propose the single artifact because offline-first clients cheap: download once, resolve everything locally, consistent with RFC 0017 and RFC 0031 keeping compatibility evaluable offline.

Whether the artifact is a tarball of files or one merged JSON document is an implementation detail the RFC can leave to the tooling, as long as the address and the caching behavior are pinned.

On the decentralization question

Picking up @KlaasWhite's question and @averageksp's follow-up, here is the line I would draw: central in authority, decentralized in survivability.

Central in authority, because the id namespace needs exactly one arbiter (point 5), and dependency resolution needs one catalogue that is allowed to say what exists.

Decentralized in survivability, because everything above is plain git plus static files on free infrastructure:

• Mirroring the entire index is git clone, and anyone can do it at any time without asking.

If nobody objects to these three, I would write the index RFC on this basis: two repos, snapshot fetch contract, watcher plus self-service listing plus the moderation model from point 7.

[Maximilian-Nesslauer]

8. What a GitHub Actions incident does to this design

We got a live test of this yesterday: GitHub had an Actions incident, and it hit PR #31 .
The runs for one push were cancelled after 15 minutes without ever starting, and the next push produced no check suite at all: the pull_request event was dropped, nothing retries a dropped event, and the PR sat yellow until an intentional empty commit re-triggered it.

I believe that this is important for our design:

The scheduled watcher is resilient by shape.
Polling depends on no single event and a cancelled or dropped cron tick just means the next tick, some minutes later, scans everything that appeared since the last stamped release.
The state ("what is stamped") lives in the repo itself, not in an event queue, so an incident costs latency, never data.

The event-driven half needs a sweep.
The self-service listing flow (point 7) rides on pull_request events, so an incident can leave an author's listing PR with no validation run and no auto-merge, exactly what happened to PR #31.
We should make it so that each watcher tick also sweeps open listing PRs that have no check suite for their head commit and re-triggers their validation.
That way the event-driven path inherits the polling path's self-healing.

The watcher must be idempotent.