Mod Management Specification (Draft) #1
Comments
|
Maybe manifest should be TOML. Easy for humans to write and read, and still easy for machines to parse. Also, maybe support different compression algorithms for packages? If a mod has the possibility to be big, formats like |
|
Whoops, misclick 😄 @aspenluxxxy we briefly discussed metadata serialization formats on the discord server earlier today, to recap a bit what I remember: We discussed the possibility of using XML, JSON, TOML, and YAML. Nothing was decided, but I think the points that were made are somewhat as follows:
My personal opinion is that we should have a tool that builds the mod packages for us, converting either a YAML or TOML file into a JSON file. We would get the best of both worlds with this strategy, as it allows creators to use the tools best suited for writing configuration while still allowing for easy consumption of the generated package. |
|
TOML's difficulty depends on what the tooling will be made, although I wouldn't mind a converter. As long as it's not XML, I've had enough XML to last me a lifetime 😆 |
|
Yeah I've started liking the idea of converting either YAML or TOML to JSON when building the package the more I think about it. It's highly likely we will have a tool that can build and upload packages automatically, so applying a conversion step in that stage would be easy enough. JSON on the other hand is just so easy to parse with any language that in my opinion it deserves some pretty serious reasoning against using it as the distribution format. |
|
|
|
That's a very good idea, I wasn't familiar with what the spec is called so good to know. Updated the draft. |
|
YAML also has a downside as mentionned by @avail i think, it is indentation dependent. Making it much more error prone for inexperienced users who may confuse tabs with spaces for example or just anyone who is not aware of this fact. TOML doesn't use indentation, it can work with it but it is not needed. And as far as i know the language [TOML] has support for parsers in most of the languages, i found libraries in python, csharp, cpp, javascript and rust obviously. Not that this is not an exhaustive list. JSON could have been the candidate if it had support for comments in my opinion. But sadly it doesn't, unless we use a subversion of the language but again the issue of parsers arises. And i think JSON may appears as verbose as XML for inexperienced users who would like to create a basic mod that has nothing to do with programming. There is also the INI language (wikipedia page), TOML's ancestor basically. Any language in the world has a parser for it, basic syntax and support for comments too. In it we can add sections, and basic key-value pairs. Making it easy for to edit. Anyone, experienced or not, could edit such a file without risking a parser error. |
|
Good points @Aelto. I think it pushed my vote towards using TOML for the configuration language we write, but I still think we should distribute JSON just because of how prevalent it is nowadays. At least the python toml library basically hands you JSON as the output if you parse a TOML file, so it would feel natural to ship it as JSON, which overall is a less complex format. If we were distributing configuration files I would probably vote for TOML in distribution as well, but right now we're talking about something immutable, which is why I think JSON is still best suited for the job. |
I agree with you on this point, if the data is immutable there is no reason to use something like TOML and JSON would suit perfectly. What do you have in mind when you say configuration and distribution? I thought the configuration file was distributed alongside the mod to serve as a small metadata file for the mod manager and the api. Do you intend to have a translation phase where something would convert the configuration file to something the API can read? So my vote would have been for TOML too, but from what i understand JSON makes everything simpler so i vote for JSON too |
|
I'm not sure if I understand the question correctly, but what I'm saying is that it'd be nice to have developers write TOML, and a tool that builds the mod package zip file to convert the TOML into JSON before placing it into a file inside the zip. When the mod package gets uploaded to the site, we can validate the metadata json to match the spec we outline here. After the package has been published, we can expose the metadata in the JSON file content-wise 1:1 over the API if we want. When downstream users download mods, initially they just receive a zip file. Their mod manager will read the metadata json from within the zip file and install the mod according to the instructions. Additionally it can use data from the metadatafile to display mod information in the UI and so on. So the metadata fields we're outlining here are very much immutable once they're built into a mod package. User-editable configuration file standardization is another topic I suspect we might want to open up in another issue. Did this answer your question? 😄 |
|
I think what was suggested was for the devs not to have to write anything manually, but instead have a cli tool (such as npm init) prepare basic variables (author, mod name, [...]) and when packing the mod up the cli tool would take care of mapping the rest |
|
Alright @MythicManiac i understand what you meant now. So there are currently two ideas, use JSON everywhere and create an interactive CLI to ease up the mod creation, or use an easy language for the configuration file that is then translated into JSON. Or both perhaps? But i must say that i find the idea of translating a TOML configuration file to a JSON configuration a bit much. I'd rather go for the interactive CLI and JSON everywhere route instead, because it's just creating an unnecessary layer of abstraction for the sake of "simplicity" at this point. What do you think? |
|
Either sounds good to me. Interactive configuration isn't incompatible with TOML, but if you feel like using TOML as the in-development configuration format is too much hassle, let's go with JSON. I'd imagine a lot of people are familiar with JSON based package configurations anyway, given that npm also uses |
|
It might be nice, for dependancies, to add information about how the dependancy could be obtained. This could, for example, be a list of dictionary. Each entry of the list is a dependancy, and each dictionary have at least the key "id" (or "identifier") to specify the id. There then could have a sources entry, with a list of way to get the mod (could be list of string, or a list of dictionary that specify the fetch method and some other data). I image it like: [
{"id": "mod1", "sources": [
{"kind": "http", "url": "http://example.org/mod3"},
{...other fetch method...}
]},
{...entry for mod2...}
]having multiple sources would allow to augment reliability if multiple source avalaible but one is not avalaible. Other possibility (based on https://github.com/OpenMW/openmw/blob/master/docs/openmw-stage1.md) |
|
You're right, we did not think of how the tool would handle dependencies. I don't think the mod author should be able to specify where to download the dependencies, because it could be a potential security hole. Dependency management is always a hard topic to be honest because we always want a decentralized system where any dependency can be downloaded from anywhere, but at the same time we cannot allow the CLI to download files from non certified urls. I think lots of package managers do it in a centralized way, where the manager decides where to download the archive. Of course we can allow the user to register other mod repositories, but it is done by the user and not on a per-mod basis and by the mod authors. |
|
Having some formatting for the description would be nice. Maybe some markdown. Could allow to include picture, at the cost of rendering complexity (html would work, too, but then it would be uselessly harder to render). @Aelto After a bit of thinking, I came to this idea : When listing dependancy, also have a list of remote that have this dependancy. They could be identified by domain name, digital signature, or some other thing. This would allow the user to allow or block some remote (using an allowlist). When adding a mod, the mod manager see if the dependancy is in one of the allowed remote. If not, it ask the remote specified by the depency if they have it. It then list the remote who have the depency, and allow the user to select one, that could either be permanently trusted or only trusted for this one depency, or finally do not download it at all (and let the user fetch it himself). This should be both secure, and allow mod to specify the location of mod without needing the user to search what remote have the depency (or have a list of default depency remote already allowed -- or at least making the choose to allow or not some remote by default less biased due to will some random mod added via github or similar work out of the box) |
My idea was that the description is a short description (think 250 chars or so) that can be included in small UI spaces. On top of that we could standardize longer descriptions e.g. a README.md to ship with tha packages that could have any amount of other content, or even more advanced solutions whatever we come up with. Either way I'd imagine there's value in having a short description that can be shown in smaller UI spaces. Regarding the download sources |
|
@marius851000, an issue was created for directo discussion about the CLI tool itself. If you would like to continue discussing the interesting subject you brought up, i think it would be better suited in the issue #2 about the Game Package Manager tool (the cli). And i agree that your idea of allowlists and denylists (or whitelists and blacklists if we use the old and now outdated terms) should solve the issue. And the way you presented the solution seems like a perfect solution. An allowlist where the CLI prompts the user about new unknown hosts is perfect for this case. Do you think the CLI tool should have a command to easily manage the lists, like Please, quote and answer this post in the other issue as i completely forgot to do it myself 👍 |
|
@MythicManiac I effectively see the issue that I think I have an answer for that : |
|
Could allow mods to be signed with ed25519 keys, and hardcode the public key into manifest files. So only download |
|
(ed25519 is an asymetric digital signing method) |
Yeah, the dependency field could specify "only grab the dependency signed with |
|
One obvious downside to multiple sources is identification of identical packages. I don't think it's too far fetched to assume mod authors might serve their mods from multiple different sources, (e.g. github and the default repository), and if we no longer can match packages by package identifiers (which is a problem introduced by multiple download sources), we will need some other way to detect content-wise identical packages. It would be troublesome to have package A depend on package C from source 1, and have package B depend on the same package C from source 2. With the proposed solution we would have to treat both dependencies as unique, even if they were the same package. Now, you might think we could easily solve the issue by including somthing like a hash/signature as the package ID and avoid duplicate package installs that way, but how about if we try to do version range dependencies? I'm sure we could come up with a technical solution to make multiple package sources work as expected, but I'm starting to wonder is the extra complexity worth it, or should we sacrifice some integrity guarantees in favor of simplicity? Maybe we could always depend on a identifier only, and allow for listing of more download sources. We would then have the manager download packages from the first repository it finds that matches the package name, and the repositories could be priority ordered. Something akin to what system level package managers do on the linux world. |
|
You're right @MythicManiac. Maybe the simplest solution is to create a centralized repository and offer the support for optional repositories. But if the user chooses to use another repository it's his own choice and we should not make everyone's life more complicated just because we support 3rd party repos. Maybe we could make it in a dumb and simple way, if a mod is in a mod repository it means all of its dependencies are from the same repository. Roughly the same idea that mastodon (the decentralized twitter) uses for its instances. An instane is basically a independant version of the database, and it's up to the instance manager to synchronize the new instance to the other instances if they want to. It may not be realistic to think about a fully decentralized solution inside our CLI at the moment. The CLI should stay simple and do one thing, store an address for the current mod repository and then download everything from it. |
|
@Aelto @MythicManiac
Also, Will assume the same mod can't have multiple enabled version. also, the linux distro I use actually hash the version, package name and all the build instruction together to generate an hash that will identify the package -- allowing multiple version of the same software to be installed. If your interested, it's called nixos. This actually assume that a mod with the same id on multiple remote correspond to the same mod. |
|
about InstallStrategie: |
|
Exactly, so the idea is that it's up to the user to use a different remote than the default one. But it is not the mod author's responsibility to do it. Did i understand correctly? And i don't think we could compare two packages from two different remote instances, even with an ID or a GUID or anything because we cannot know if the instance manager changed the files in a malicious way without changing the IDs. Maybe comparing hashes to check the file integrity could solve the issue though Can you give example of various Install strategies you suggested? I can only find |
I was thinking the My thought process is that the package standard itself should not care what the install strategies do or what their parameters are, we just need to know what strategies the package is flagged as supporting. The install strategies themselves can have their own specs on how to define parameters; some could be entirely parameterless and function by convention, where others could have their own metadata files. For clarification, an Install Strategy is an identifier for the logic to use to extract/install the package into the game. The goal is to not lock in how packages are actually installed in the package specification, and allow the expansion of those rules and methods later on without requiring a package metadata change. An example is how the current packages define the following metadata: "extra_data": {
"mm_install_strategy": "extract-gamedir"
}I've hardcoded the MythicModManager to identify packages that define their |
|
@Aelto We should actually be able to compare mod from multiple different remote if we trust them all (and we keep the default list small). Including the hash shouldn't be a good idea, as it would prevent update to the mod (but asymetric cryptography could work at the cost of complexity). I suppose the idea of trusting every allowed remote is good, as from the moment you download any mod from one of this website, the mod could allow Arbitrary Code Execution. |
|
Yes, what scared me first was the fact the mod authors could specify remotes on a per mod basis or that a mod could have dependencies on different remotes too. But if we exclude these ideas, i don't see any issue with downloading packages from different remotes too. Except that it could create fragmentation in the packages world, but that is the cost of decentralization i suppose. And if we keep a nice and frequently updated default store there should be no issue. But i really don't see the point in comparing packages from different remotes. Unless we want the stores (remotes) to be able to synchronize their packages |
|
@Aelto comparing depency is usefull for when some remote are not up to date (althought limiting fetching dependancy from the same source as the mod is a good idea if the source is a store -- as opposed to a manually downloaded .zip file). Another use case could be using experimental package. The user just add a dev remote by the mod author, and tada nightly/whatever unstable release. Also, I suppose it is useless to specify that we only download depency if they are already installed locally (and no update are avalaible). |
|
It seems to me the way you would handle nightly-releases for dependencies is wrong. Are we going to create a remote just for nightly releases or will we ask every single author to create their own remote that the user will have to validate just for their nightly-release? I think we should manage nightly releases differently, i don't know how yet, perhaps in the And okay, yes i understand now why a way to compare packages between two remotes is expected. So how will it work, first it will fetch the package by its |
|
I think the most popular way to handle this kind of scenarios is to synchronize the available package index from each configured remote, and check which one(s) have the most latest versions available. This however does mean we'll idelly need a local package index cache that we can use to look up packages (think |
|
@Aelto @MythicManiac About the index: Assuming the number of mod to install in one go should be pretty small (the number of mod to install should be of some at a time) and we have a small number of remote (let's say 5) this shouldn't be too much of an issue to make 20 http query. The number of request augment linearly with both the number of mod to install and the number of remote. So having to have a local cache might not be required. Just query the version the remote have for each of those mod (or compact that in one http request, like tell me what's the version(s) of As for nightly handling: making it part of the the version doesn't seem a good idea, as we may want to rather use the first 7 seven character of the git revision, as is commonly done. As for the name, this could lead to some complexity for handling depency ( |
|
As someone fairly active in various modding communities, my suggestion would be to look at MinecraftForge's mod info/versioning file: Mainly three things: It supports multi-mod-single-archive mods, this would be useful for authors who modularise their mods but would like to offer an "all-in-one" solution. It has explicit support for dependencies and if they're mandatory or not. And lastly, the version spec is that of Maven's which supports a whole range of different things. From the basic Of course, this is merely a suggestion that aims to help with figuring things out, and implementing some of these things may be too much for right now, but I thought it would be a good example of something used in another unofficial modding API/toolset. |
|
Version range support for defining dependencies seems important to have, this was also just commented on #2 (comment) For the version numbering scheme, I have a small soft requirement, which is that it should be possible to parse the following information from a "full" package name, and that the full package name should be filsesystem safe:
The full package name would be formed by This is not a hard requirement, but it will give us some nice guarantees, e.g. that we can place all packages downloaded into a single package cache side-by-side without worrying about collisions, and that we can simply by looking at the filename determine what package is in question. The bigger picture philosophy is that packages are namespaced by the creator's identifier, akin to how repositories work in GitHub. |
|
I'd like to throw out there that a lot of this will be dependent on what services you plan on using. I've spent the last 16 months or so building a community around a project like this for the Bethesda game community. Wabbajack (https://github.com/wabbajack-tools/wabbajack) has had to handle a lot of these sorts of issues. In that time we've seen 100k installs using the software and about 40k users. While I'm all for adding support for Cyberpunk to Wabbajack, I recognize that green-field development can also have its perks. I would however recommend keeping version numbers string-free and machine readable. If you are going for |
|
Related: thunderstore-io/Thunderstore#138 In my opinion, requiring a People may be confused when viewing a manifest in a Git repo as there isn't any real link so it might appear to be a project specific file instead of a file used by a mod repository. If an install link is included in the README and the user looks at other mods they may realise that the file is a standard Some other file names are very explicit (e.g. In the short term, |
|
Would there be a set list of tags or can users select any tag they want? Would it be |
Agreed, keeping things parseable is a requirement. I was merely suggesting that in addition to keeping the fields separate, we can have guarantees in place that allows the use of a single-string identifier (by ensuring all components can still be parsed from it). If we look at the way Thunderstore for example handles things, the guarantee currently exists.
I'd say tags are just a list of arbitrary strings, so no list of set tags. We can restrict characters to something sane like proposed, though I'd allow at least |
|
On second thought I don't think we need to restrict tag characters either, my biggest concern allowing anyting would be if someone adds zalgo there or some super weird utf-8 characters. No idea what the allowed character set should be like to only leave out the weird stuff. |
|
Hey guys you might want to grab some ideas from this package manager: The description and bucket system seems to be quite straight forward ! |
Package format
All mod deliverables are to be shipped in a standard package format. The packages are zip files that contain all of the mod's content as well as a small amount of metadata to instruct how the package should be installed.
Currently the following fields for a metadata file in the package are planned:
Additionally, the following metadata could be included in some shape or form:
Tooling
Responsibilities of a mod manager
Install Strategies
The text was updated successfully, but these errors were encountered: