Game Package Manager Specification

[MythicManiac]

The Game Package Manager (or GPM) for short is a CLI tool which primary purposes are to:

• Download mod packages
• Install mod packages to the game
• Uninstall mod packages from the game
• Package mods
• Upload packaged mods

This issue is meant to discuss the interface we wish to expose from this CLI

[Aelto]

Hi, a conversation already occured in a different place so i will do a recap of the ideas we shared and of the ideas we discussed but did not agree on yet. Anything written below is to be interpreted as an RFC draft and is subject to changes based on feedback.

The main goal

The CLI (Command Line Interface) tool, that will be called CLI for the rest of this message for simplicity has one and simple goal: Allow the user to install, uninstall, download and manage the mods for many games. The first implementation of this CLI will be done to support the game Cyberpunk2077, but will eventually evolve to more than just this game.

The words used in this document

• REST API is the web service that listens for HTTP requests and manages the full mod database. The REST API shall also be mentionned as the Store

• the CLI tool is the tool we are talking about

• the GUI is something we have in mind, a graphical application that could sit on top of the CLI tool and could leverage it to make mod management easier for non-programmers. It would not implement anything and would just serve as an interface between the user and the complex commands the CLI will accept.

• The words should and must will be used and are there to imply that during the implementation of a program the following information has to be exactly as stated in order for the program to be considered standard

What is a CLI tool

First let's agree on what a CLI tool should do and should not do. A CLI tool is a tool that is often run through the command line to execute a piece of code. For example the following CLI tool cat allows the user to run the cat tool with a parameter to get the content of a file. The result of such a command would be cat my-file.txt. The result we get out of the CLI tool (what is called stdout) is caught by the command line interface of the user and is displayed at the screen.

So basically, we have a program/binary (the CLI tool) that can accept many commands like install, remove, publish, etc... (not real commands) that will output text of any form. This text can then be captured by anything that ran the program (our command line interface or even a GUI) and parse the result and act accordingly.

The GPM CLI

Now that we can agree on the idea of what a CLI should do, let's start talking about the basic commands the Games Package Manager cli should have to be considered a minimum viable product:

• install for downloading mods from the internet by contacting a Store. The choice of store could be offered to the user through an optional parameter if needed, something like install package-name --store new-store-address.
• remove to remove locally installed mods from the local repository
• init to help the user bootstrap a project for his future mods. It should by default act as an interactive command with prompts to fill the default values of the configuration file. Such a feature should be skippable through an optional parameter, something like init -y to skip the prompts and use the default values everywhere.
• enable to enable a mod that is in the local mod repository so the game will detect the new mod
• disable to disable a mod that is currently enabled and is in the local mod repository so the game won't detect the mod anymore
• publish to bundle and publish a mod that follows the convention required by the store. For an easy setup the user should refer to the init command. The publish is expected to be run inside the root directory of the mod to publish, but this point is up to the person implementing the CLI of course as it could also use optional parameters to supply a path to a mod directory.

The CLI tool has the responsibility to download, install and manage the mods without any user interaction (except from the commands, obviously).

The CLI tool should directly interact with the end-user filesystem to manage and install the mods.

The CLI tool should work entirely on the filesystem and the metadata files the mods have to manage the mods and should not use any database to store which mod is installed or not. The use of symlinks is recommended to avoid duplicated data on the disk of the end-user.

The CLI tool should only compare what's in the game directories and what's in the local mods repository and act accordingly in order to avoid any loss of track if the game install is modified by an external tool or by the user itself.

[marius851000]

About multiple store: as said in the other thread, multiple store should be allowed a way or another. The method this will be done is unsure, but I will draft something about it (based on @Aelto comment in the other thread). Will assume as of now that this will work in a way similar to debian (multiple store that are seen like they are one, assuming that multiple store who have the same mod (identified by id) will result in a unique being selected based on version):
we would need to have command to manage those list of store. Suggested change:
install will now search by default in all the allowed store, and install the more up-to-date version avalaible in any one store (where to look for those dependancy are unspecified, either all the trusted source or only the source wich host the selected mod). Only --store will restrict the search to one remote.
add:
store add <store_id> add the <store_id> to the list of trusted store, searched on by default (store_id will most likely be the store URL)
store list list all the store installed.
store remove <store_id> remove a trusted store. Keep mod installed via it. Allow removing the default(s) one.

there should be a/some default store trusted by the default.

[marius851000]

unrelated: allow installing from local file via install ./mod.zip or ~/mod.zip or /path/mod.zip or anything://path.zip (the last one for windows, but could maybe also include https)

[MythicManiac]

Slightly formatted excerpt of what I posted on Discord:

So I'm thinking we could make developers and users use the same interface with the tool

• Akin to how npm works, the directory in which you call the commands is your active project/context
• If you add a package, what it really would do is add it as a dependency to that project (by adding it to the project dependencies). Removing a package would remove it from the same configuration.
• Any and all commands we'd have would (by default) operate on the currently active project, which is equal to the workdir you're in, and would save whatever state changes have to be done inside that location
• We'd have a command you can use to start the game, which would look at the currently active configuration, and install/enable all the defined dependencies before launching the game.
• This way developers get to also test their WIP mods easily with dependencies automatically taken care of
• The end-user facing interface can re-use the functionality
• Additionally the end-user facing interface has built in support for mod profiles, given that we don't have a global state anywhere (aside for package cache)
• The only thing we'd have to keep in mind is that we don't want it to be necessary to define full package metadata when you just want to run the game, and not build and publish a package
  • which means we'd need the tool to be able to operate without full metadata

Thoughts?

[marius851000]

@MythicManiac This seem a good idea, in particular when we want to be able to easily share mod definition. Some issue that could happen :

• We need a better to specify dependancy. For example, if we want to install a mod from a local directory, the depency should point to it (the folder it has been unpacked to). This add the need for multiple dependancy source (but is otherwise a good idea).
• We should need more global state about application, mainly thinking for mod that replace/add file. Uninstallation should need to remove/restore them.
• Not sure I would like to cd into my standard mod folder every time I want to just a simple mod. Can be circumented via a GUI that abstract this, or a CLI flag that specify some global profile (this is not too important, and can be thought about later)

[MythicManiac]

We need a better to specify dependancy. For example, if we want to install a mod from a local directory, the depency should point to it (the folder it has been unpacked to). This add the need for multiple dependancy source (but is otherwise a good idea).

If we are to continue the earlier idea on how we handle multiple sources for packages, this does not seem like the correct way to handle it, as we'd be hardcoding information about where a package should be obtained from to the dependency. Instead of doing that we could for example support defining a local package repository (which is just a directory with packages somewhere) and use packages from there as a higher priority, or simply have a parameter (e.g. --from-file) when installing the packages. Thoughs?

Continuing, maybe we could take git-like approach, where you can define remotes (or local file usage) on a per-repository basis? this should allow for convenient local file usage if you want without requiring you to write it to the dependency file (that will be built into a package when publishing)

We should need more global state about application, mainly thinking for mod that replace/add file. Uninstallation should need to remove/restore them.

I think the current philosophy is that we do not want to store state (because it will get inconsistent), but what packages are installed should be obvious from looking at the files themselves. In reality this might be rather challenging, and I see two ways to go about it:

1. The virtualized filesystem approach that is currently being worked on, where we only apply modifications on launch time to a virtual filesystem, which leaves the original game files unmodified. This way we don't need to track state, because our launcher will apply the appropriate modifications to the game files during launch/load time in a virtualized fashion.
2. Each install strategy will be responsible of tracking their own state, and before launching we simply tell the strategies to synchronize the game files to match a desired state. This is a lot more unclean and risks integrity errors, but it's a fallback we'll have to take if the virtualized filesystem approach does not work for some reason

Not sure I would like to cd into my standard mod folder every time I want to just a simple mod. Can be circumented via a GUI that abstract this, or a CLI flag that specify some global profile (this is not too important, and can be thought about later)

Yeah agreed. We could have global contexts, or parametrize filepaths, or the default profile, etc. many solutions should exist to make the usage convenient on top of the base solution proposed here.

[MythicManiac]

My current thoughts on the command structure:

Command	Description
init	Initialize a new project to the current workdir
start	Start the project's game with the selected dependencies installed
add	Add a dependency to the project
remove	Remove a dependency from the project
list	List the current project dependencies
search	Search packages from the configured remotes
build	Build the project into a package
publish	Build the project into a package and publish it to a remote
remote list	List remotes
remote add	Add a remote
remote remove	Remove a remote
cache list	List packages in the local package cache
cache add	Download and add a package to the local package cache
cache remove	Delete a package from the local package cache

This is still lacking several things, and assumes we'll take a lot of parameters to each of the commands for more fine-grained control if needed. Some considerations for missing features:

• Listing only outdated dependencies
• Upgrading dependencies to their latest version
• Maybe commands for managing the caches the tool might generate

Also worth considering how do we configure what game is the project targeting or supposed to launch. We could have a gpm init --game cyberpunk2077 style approach for example.

[Aelto]

Also worth considering how do we configure what game is the project targeting or supposed to launch. We could have a gpm init --game cyberpunk2077 style approach for example.

I suppose the packages will have a field to specify which game they work on. So if a package supports only 1 game it could use it as the default value. Otherwise your --game cyberpunk2077 idea sounds good

[MythicManiac]

@Aelto I don't think it's a smart idea to require packages to define the game they support, although they could have a "preferred" game or a list of preferred games. Reasoning for this is that some packages might be applicable to multiple games (e.g. imagine you just distribute a 3d model). SOME WAY of figuring out whether or not a package is compatible with a certain game would be very nice, but I'm still unsure what that could be

[Cryotechnic]

Briefly read this over through a post in #wiki-updates on Discord and it's quite late in my area so I may not have the full picture but, nonetheless, here are my thoughts on this:

I don't think it's a smart idea to require packages to define the game they support, although they could have a "preferred" game or a list of preferred games.

@MythicManiac The way I see it, the only valid reason to have packages require explicit declaration of which game(s) they support would be if GPM would be used in the future to support more than 1 game at a time. Best way to do this would be to assign a game ID code to a game (eg. Cyberpunk = 1, Game2 = 2, Game3 = 3, 0 being "automatic", which would default it to Cyberpunk only for the time being, but could change in the future). It would then be up to the mod developer to include that flag in the mod's config/files, so that the mod manager could identify it when loading mods from multiple games.

I suppose the packages will have a field to specify which game they work on. So if a package supports only 1 game it could use it as the default value. Otherwise your --game cyberpunk2077 idea sounds good

I would use this more as a "force the package manager to create cyberpunk2077-based bootstrap", which would be different from the default environment. However, if GPM will exclusively be used for CP2077, then there is no need to implement any of those flags, because it will load all the mods in the same way.

Thoughts?

[marius851000]

@Cryotechnic About the first point, it may be a good idea to specify what the mod is compatible with with a list of String. For example (taking an exemple of a mod that work with both the withe 3 and cyberpunk 2077) [ "witcher3", "cyberpunk2077" ]. Maybe add group, that could be used as in ["openmw-engine"] actually mean ["morrowind-openmw", "morrowind-tes3mp", "othergame-openmw"] (tes3mp is a multiplayer form of openmw.)

[MythicManiac]

Multi-game support is something I think we want to do given the quality of the tool we're building. I agree we need some kind of a game identification system, and like @marius851000 suggested also I'd personally rather use strings that are clear (e.g. cyberpunk2077) than a cryptic numbering system.

Let's think about what's the game-specific information we need to track, because that's the primary relevant part. What comes to mind for me is at least:

• Game install path, or discovery rules to find it
• Game executable path, or discovery rules to find it

So if we take a step back and look at this, what we really care about is the paths. IMO it would make sense to create discovery rules with a game identifier that autodetect these paths, but also leave it possible to manually configure them. This would make it so we don't need to "bind" a project/profile into a single game.

So how do packages get installed if they don't know what game they get installed to? That problem should be solved by the pluggable install strategies, which handle the installation given a package and a game install path. If we have game or engine specific install strategies, they could validate that the configured game path matches their expectations, and error if they're incompatible with it. Some strategies might opt in to do their "installation", which if applied to the wrong game, would just end up doing nothing. I'd feel like this is an important capability to have to create a good ecosystem, as longer term we could see packages that simply share models for example that get used across multiple games.

So to summarize my opinions:

• String IDs for games rather than numeric ones would feel more intuitive
• CLI tool should not care about what game is in question, but rather the paths it should operate on
• We should have a database or set of hardcoded rules for automatically discovering game paths
• Install strategies should not care about what game is in question either, but may raise an error if the supplied install path doesn't match their expectations

The conclusion is that there's no technical reason we need to care about what game a package belongs to. The primary purpose of being able to relate a package to certain game or games is to make it easier to discover by the end users. This information however does not need to be baked in to the package (which is immutable), and could instead be configurable as dynamic metadata on the package repository website/API to enable appropriate search filtering.

Thoughts?

[marius851000]

@MythicManiac @Cryotechnic I really think that the list of game should be a in the metadata, be it for discoverability (and warning the user, with a non-fatal message). It should be better if we want to just add an existing into a package simply (without having to fill supported games) (and then the remote read the list of supported game from the package).

[MythicManiac]

@marius851000 the primary problem I see with that is that we'd have to pre-define each game to have a schema to validate against, or alternatively we'd just accept freeform strings, which would make it very close to the tags field.

Another problem is that now you're writing this information to something immutable, which means if you ever want to add a supported game, you have to bump your package version for practically no reason.

IMO it would be a good idea to separate dynamic metadata that might evolve over time from immutable metadata that we want to bake in, because immutable metadata changes will always require a package version bump.

So this really comes down to a design choice over anything else, we don't have a technical requirement for it but it could be useful. Can we come up with a couple of example scenarios where and how this metadata would be used to illustrate the need of the field (or lack of)?

[marius851000]

I personally think that any information that end up on the user computer after downloading and closing the application should be stored in the mod file. That indeed mean we will need to update the file to add support for a new game even if it is just to update the list of supported game.
I see them as a list of reconized id (like [game1, game2], but they can also be [game1, engine1] for matching the game engine. In this case, if either the game we install the mod into is game1 or user engine1, no warning is displayed, otherwise, a warning is printed. For mod that use multi-engine file format (like standarized gltf), we can also use [format1, format2], and if the game is registered for at least one of those format (in the same way for the engine), no warning is displayed. We can end up with stuff like:

["engine-openmw", "format-dae"]

[MythicManiac]

I personally think that any information that end up on the user computer after downloading and closing the application should be stored in the mod file.

I agree with this point, mod packages should self-contain all the required information for their operation. This is also partly why I don't think we should have a required game field in there that completely dictates it's operation, since:

1. It's impossible to predict how a package will be used. At best we can indicate the intended usage.
2. We will need to have a database of game IDs that evolves over time, which by definition means we're either:
   3.a Not going to have any validation on the game field, which will make it equivalent in functionality to the Tags field, aside for it's intended usage
   3.b Have validation that will block creation of packages to games we don't have in our database yet (be it hardcoded into the CLI utility or hosted online somewhere)

I see them as a list of reconized id (like [game1, game2], but they can also be [game1, engine1] for matching the game engine. In this case, if either the game we install the mod into is game1 or user engine1, no warning is displayed, otherwise, a warning is printed. For mod that use multi-engine file format (like standarized gltf), we can also use [format1, format2], and if the game is registered for at least one of those format (in the same way for the engine), no warning is displayed. We can end up with stuff like:
["engine-openmw", "format-dae"]

This seems like a overlap of responsibilities with the InstallStrategy field. If we list supported engines, what's the actual difference between listing supported install strategies (which are bound to be engine-dependant for some, some might be generic)

Could we focus on what is the end goal of the would-be game field and explore all options that could be used? Right now there's a lot of points being made for having a game field, but a case hasn't been made for why it's needed and what it should be used for exactly, which makes proposing alternatives difficult. I feel like at the moment I don't have the required information to reject or approve of the idea; I can only point out constraints it implies.

[marius851000]

The main use case I can see for them if for the user filtering them in the GUI. For example, we want to list all downloaded mod that can be enabled for a specific profile. For example, hiding mod for cyberpunk2077 while displaying those that can work for skyrim (assuming we want to mod skyrim).

[MythicManiac]

Some elaboration of my current thoughts:

Game

A game has:

• An identifier (e.g. cyberpunk2077)
• Install path location
• Binary location (to launch the game from)
• Rules for automatically discovering install location
• Rules for automatically discovering binary location

Project / Profile

The project term is interchangeable with profile, usually called profile when talking about mod manager context, and project when talking about development tool context.

A project is a working directory, under which all of the project's dependencies are stored in a configuration file. When the project is launched, GPM makes sure the all of the selected dependencies are appropriately installed in to the game

Each project is configured to target a specific Game, which in practice means they store the target game install directory and launch executable. The project configuration is stored in a configuration file within the project, e.g. akin to how git repositories retain configuration within the .git folder.

Install Strategy

Install strategy is a module, which given a mod package and a game install directory, will install the supplied mod package to the provided game install directory.

Install strategies should also provide a way to check if they are compatible with a specific game install directory. This means that the install strategy module is given a path (to the game directory), and it has to return whether or not it is applicable to that directory.

Filtering available packages to match the user's game

Given the above assumptions, we know that:

• Every profile/project is targeting a specific game, and has information about their game install directory known
• Every mod package has defined install strategies they support
• The GPM tool is able to query install strategies if they are compatible with the configured game

Using these design guarantees, we can follow the following operation model:

1. When a user creates a new profile, they specify the game they want the profile to be created for
2. The mod manager uses GPM to search for known autodiscovery rules to discover the game's install location, and configures those for the profile
3. GPM queries known install strategies, checking which of them are compatible with the game selected for the profile
4. The mod manager will show the end user only the packages that have been confirmed compatible with the selected game

[marius851000]

Each project is configured to target a specific Game, which in practice means they store the target game install directory and launch executable. The project configuration is stored in a configuration file within the project, e.g. akin to how git repositories retain configuration within the .git folder.

You should replace project with profile. It would be nice if only profile contain installation specific information, and to remove them when publishing/packaging the file.

about the filtering : If i understand well, we do something like:

• for each installation method :
  • query if it work on the installed game folder
  • if it does, add it to the list supported_install_method
• show as mod supported only mod that have at least one supported_install_method in the install method.

This can lead to issue for generic install method like VFS, I think (after all, all directory that contain a game should be a directory that can be patched via a VFS)

[MythicManiac]

You should replace project with profile. It would be nice if only profile contain installation specific information, and to remove them when publishing/packaging the file.

This is actually why I drew an analogy to .git, because the configurations under the .git are local only, so I'd assume any such configuration (target game, paths, etc) would also be local-only in our case. Good for pointing that out, should have mentioned that in the description 👍

This can lead to issue for generic install method like VFS, I think (after all, all directory that contain a game should be a directory that can be patched via a VFS)

Yeah it might cause issues, hard to tell before we know what the VFS implementation ends up being like. I do think we have a fairly layered filesystem approach already which should be compatible with this plan, but let's see how will it turn out. Another thing we can do on the long term is have different options aside for just VFS, the important part is that the install strategies wouldn't know they run in a VFS, they just operate on files like any other tool. But maybe we should open a new issue about the VFS implementation details 😄

Anyway, on a conceptual level my main philosophy has been that install strategies should be the primary contract between a package and how it should be installed, and any compatibility checks would be performed by the install strategy. When a package developer chooses what they target with their package, it shouldn't be a specific game, but a specific install strategy. Install strategies get to define their own guarantees and limitations, and could target a specific game, an engine, or even multiple engines.

If the VFS prevents our current approach somehow, I still believe keeping the separation of responsibilities above is a good idea, and we should explore other ways to make it work.

This way we can nicely support generic packages as well as very tightly coupled game-specific packages, as it's all up to the install strategies. To draw an analogy, install strategies would work as almost the same way "build targets" do in a more traditional programming terminology.

[marius851000]

Now, we need to know how to handle the fact that profile use only a single identifier is specified in the project file, yet we want the end user to specify a specific version. I propse:

• We add a minimal and maximal supported version for depency by a package (default to -inf and +inf , the maximal supported version should only be used sparingly).
• There is a lock file in the user-side (profile.lock/.profile.lock, or config.lock, or embed in the profile file). These store the exact version of the depency, as well as potentially a hash of the mod files (once decompressed, if we want to support multiple compression format in the future).
• When starting the game, it check that all the exact depency in the lock file are present and use them, downloading them if necessary.
• For selecting the version of a depency, it try to find something that match the requirement of every depency, otherwise it select a specific version with an unspecifed algoritm (the latest version ?) and add this version to the lock file.

Specifying the depency (that's similar to how rust does them:

[dependancies]
mod_any_version = {}
mod_some_version = {min_version="1.0.0"}

and in rust:

HashMap<String, DepencyParamater> // first string is id
struct DepencyParameter {
    min_version = Option<String>,
    max_version = Option<String>,
}

[MythicManiac]

That sounds good to me, and something like that is probably necessary if we want to support version ranges (which I think is a good idea). So the lockfile would be used during runtime, but we probably want to build the ranges into the package for distribution, rather than the lockfile?

[marius851000]

@MythicManiac yes, the range is included in the project file (and default to nothing). I'll start implementing this later today.

[Cryotechnic]

yes, the range is included in the project file (and default to nothing). I'll start implementing this later today.

I do think there needs to be some sort of "default", base version of a package that all mods need to have in order to be discoverable by GPM? This would be useful in the event of a game update, breaking some mods. We would add a patch to the GPM and bump the version up so that users would not see mods that are not compatible with the current version of the game.

Another option would be to specify which version of the game (the lowest version needed) the mod(s) need in order to run inside the lockfile/project file.

[marius851000]

For something totally different : Is it still a good idea to use both json and toml for files ? We can use toml for every file, only keeping json for communication with the GUI (if the GUI does parse the output on stdin of the command line after adding the --json flag instead of using interopability tool (like the C interability))

[MythicManiac]

I would still keep JSON as the serialization format for metadata built in the package zip. Reason is that GPM is not the only tool that will read package information, and as discussed before, JSON is by far the easiest serialization format to be a consumer of.

We definitely could use TOML, but I'd like to hear what the major advantages of doing so are, instead of doing it just for the sake of it. IMO there are very clear advantages to using JSON as the build target format.

[marius851000]

Okay. So I'll keep it so only human edited format are toml (and so keep the json for the lock file). (ps: with rust and serde, json and toml are as easy to use)