Simple Mod Framework v3

[atampy25]

Development of version 3.0.0 of SMF is currently in progress. As a new major version, it brings with it breaking changes which allow flexibility in improving the SMF experience. Here's a summary of the changes made so far.

TL;DR: v3 of the Simple Mod Framework:

• deploys mods in seconds (1.5 seconds per mod in fact, on average)
• supports Linux natively
• supports every version of the WoA engine (not just HITMAN 3)
• fixes all existing errors/bugs in v2
• brings major improvements to the mod development experience and can catch many common issues
• makes scripting significantly more powerful, allowing for interesting and more compatible mods
• comes with breaking changes that will break a small portion of mods and require some getting used to for mod developers

Breaking Changes

Update checking overhaul

Update URLs have been removed, in favour of a new top-level url key in the manifest. All mods must now specify a URL that links to their mod page. This link will be shown in the GUI and also used for mod update checking.

If the URL links to a page on Nexus Mods (https://nexusmods.com/hitman3/mods/453), users will be notified when the Nexus version changes, and will have to manually redownload the mod to receive updates. This means that while Nexus update checking is now possible, it's still better to use one of the alternative URLs which provide fully automatic updating.

If the URL links to a GitHub repository (https://github.com/Notexe/Portable-Chair), users will be notified of new releases (assuming the release is tagged with a version like 1.0.0 - no v at the start) and the framework will be able to automatically update the mod.

If the URL links to a ModWorkshop page (https://modworkshop.net/mod/45444), users will be notified of new releases and the framework will be able to automatically update the mod. Unfortunately, ModWorkshop doesn't support a proper changelog system, so the framework won't be able to provide a list of changes to the user.

{
    "id": "Notex.PortableChair",
-   "updateCheck": "https://github.com/Notexe/Portable-Chair/releases/latest/download/updates.json"
+   "url": "https://github.com/Notexe/Portable-Chair"
}

Framework data moved to data key

Properties like contentFolders and localisation have now been moved into the top-level data key (this applies to options as well, so the option now has a data key instead of directly having contentFolders).

{
    "id": "Notex.PortableChair",
-   "contentFolders": ["content"],
+   "data": {
+       "contentFolders": ["content"]
+   }
}

Options overhaul

Mod options have been completely overhauled. The easiest way by far to understand the new options system will be through the use of a schema and an editor like VS Code, but here are the basics.

Options now require IDs. These generally follow a kebab-case convention (i.e. my-option).

Most options now have the ability to be referenced within content files. This is done through a simple find-and-replace. "bla": "#{option:the-option-id}" will be transformed into "bla": true for a boolean option, or "bla": 1 for a number, etc. If a find-and-replace is inside of a string, like "bla": "something #{option:the-option-id} something", it will be replaced inline, like "bla": "something 1 something".

Option groups now exist, which are sections of options that can be displayed based on a condition (such as another option being on or off, or another mod being enabled).

Option tooltips have been replaced with descriptions, which may or may not be displayed differently, depending on how the GUI is ultimately designed.

Removal of thumbs

The thumbs manifest key no longer exists. It was previously used by some mods to disable dynamic resources to make REPO mods work online. Instead, the framework now decides automatically when to disable dynamic resources, and gives the user control over whether to allow this.

Automatic dependency porting

The dependencies key has been removed. Necessary resources are now automatically made available to whatever files they are referenced in, without having to manually specify dependencies to port.

You can, if absolutely necessary, port resources manually using the portResources key:

{
-   "dependencies": [
-       "00123456789ABCDE",
-       { "runtimeID": "00123456789ABCDE", "toChunk": 0, "portFromChunk1": true }
-   ],
+   "portResources": [
+       "00123456789ABCDE",
+       { "resource": "00123456789ABCDE", "forPartition": "chunk0" }
+   ]
}

This is unnecessary if the porting is just so a dependency will work; SMF v3 will port those automatically.

Only resources inaccessible by the given partition will be ported, an improvement over v2's unnecessary porting of some non-chunk0/1 dependencies.

Mod ID version range syntax change

The requirements, incompatibilities, loadBefore and loadAfter keys in v2 accept specific version ranges for the referenced mods. These are now mandatory, and the syntax for defining them has changed.

{
-   "requirements": [
-       ["SomeoneElse.TheirMod"],
-       ["SomeoneElse.TheirOtherMod", "^1.3.1"]
-   ],
+   "requirements": ["SomeoneElse.TheirMod@1.0.0", "SomeoneElse.TheirOtherMod@1.3.1"]
}

As before, the meaning of version ranges is interpreted according to standard convention, so 1.3.1 is equivalent to ^1.3.1, meaning "any version above 1.3.1 which is still major version 1". By making versions mandatory, this prevents mods from having obsolete incompatibilities defined that other mods can do nothing about, and additionally encourages mods to correctly use semantic versioning.

Packagedefinition changes

The packagedefinition key has been renamed to packageDefinition and support for custom partitions has been removed, meaning the type key in its entries has been removed as well (since it would always be entity).

Additionally, the framework now automatically adds scenes and bricks specified by entity.json files to packagedefinition, removing the need for explicitly specifying them most of the time.

blobsFolders renamed to blobFolders

Self-explanatory.

Condition changes and script overhaul

Scripts and conditions have both been changed to use the Rune language. This means they are now fully sandboxed and therefore safe for users; the framework will no longer give a warning when scripts are used in a mod. Scripts are now significantly more powerful as well, with the ability to dynamically alter game files based on any condition you wish.

The script interface has changed significantly.

Old

export async function analysis()
export async function preDeploy()
export async function postDeploy()
export const cachePolicy

New

fn data(config: Config, data: ManifestData) -> ManifestData;
fn operations(config: Config, data: ManifestData) -> Vec<(String, Operation)>;

The first function, data, receives the config and the manifest data (after options have been merged into it), and has the opportunity to modify the manifest data. For example, it can add a content folder based on a complex condition. The second function, operations, receives the same information but has the ability to return a series of (String, Operation) tuples. At least one of the two must be defined.

The mention of Operation may be confusing. It's due to the revamp of framework internals, which will be talked about later.

Removal of delta special file type

The XYZ.delta special file type has been removed due to it being largely useless and highly brittle to any change in the file. It was not used in any mod on the Nexus.

QuickEntity 3.2

A new version of QuickEntity is also under development, and should include improvements to the patch format as well as some behind-the-scenes changes for memory and performance improvements.

Removal of all QuickEntity files prior to version 3.2

Support for QuickEntity files (entity.json and entity.patch.json) which use a version prior to 3.2 has been removed. The framework will automatically upgrade these files when adding v2 mods.

Mod archive format change

Mod archives (e.g. ZIP) no longer include a folder inside them which contains the manifest. Instead, the manifest.json file exists at the root of the archive.

Localisation format change

The format for specifying manifest localisation has changed to keep multiple translations of the same string together.

{
    "localisation": {
-        "english": {
-            "UI_XYZ": "Some localisation"
-        },
-        "french": {
-            "UI_XYZ": "Some localisation"
-        }
+        "UI_XYZ": {
+            "english": "Some localisation",
+            "french": "Some localisation"
+        }
    }
}

New material JSON format

The old RPKG Tool material.json file format has been replaced with a new, simpler material format where material entities (MATT/MATB; now material.entity.json) and material instances (MATI; material.json) are separated. This is the format GlacierKit previews MATI files in already.

Like the other changes, this will be automatically upgraded by the mod manager.

Chunk folders are now named by partition name

The chunk folders inside a content folder are now named after the partition they represent to make it clearer what each chunk contains.

  content
- -> chunk0
+ -> super
- -> chunk1
+ -> base

Localisation overrides replaced with new localisation.patch.json special file type

The localisationOverrides manifest key was previously the only manifest key which patches a specific file. This task is better suited to content files (as seen with the other .patch.json files), so there is a new localisation.patch.json filetype following the format:

{
    "id": "[assembly:/localization/hitman6/conversations/ui/pro/online/repository/outfits_hero_s3.sweetmenutext].pc_localized-textlist",
    "lines": {
        "CHAR_REWARD_SAPIENZA_HERO_SUPERFAN_M_NAME": {
            "french": "Fan ultime",
            "german": "Superfan"
        }
    }
}

When automatically upgrading an older mod, the mod manager will place these new files in the first available content folder (which is part of the option/top-level manifest data). If no content folder is available, the mod manager will create one.

Texture changes

texture.tga files have been separated into two parts: the texture metadata in a new texture.json format and the texture itself as a PNG, TGA or DDS file with the same name. This also means you no longer have to specify the texture hash in the filename, allowing descriptive names like suit_diffuse.texture.json/suit_diffuse.texture.png.

supportedPlatforms format

The supportedPlatforms key now accepts version-platform pairs in the format h3-steam. You can additionally specify a version only, to allow deploying on any platform of that version (e.g. h2 to allow HITMAN 2 on Steam or Epic).

Improvements

So, you've heard all the reasons why the new update will require changes to your mods. What's the benefit? The framework has been completely redesigned.

Graph model

The framework no longer simply deploys mods one by one. All mods are now deployed, all at once. You might naturally ask whether this impacts the load order functionality; deployment order is still respected because the framework now represents each deploy as a dependency graph.

Essentially, instead of deploying files, the framework now applies operations like ApplyQNPatch or OverwriteMaterial in parallel across a graph which represents deploy order as dependencies (a later mod in the order depends on an earlier mod's patches finishing in order to perform its patches).

This also allows for more granular caching of data (such as specific manifest keys rather than the whole manifest being re-deployed). The framework will also now be able to store the whole cache in a flat, compressed folder, instead of the multi-gigabyte sprawling cache folder it previously had.

This is also where those two functions from the script interface come in. An Operation is in effect a node in the dependency graph, and the String its ID. A mod which wants to apply a dynamically-built patch, for example, can return an ApplyQNPatch operation. The ID is used for caching, which is another thing the framework could not previously do for scripts.

Safety and error reporting

In addition to scripts now being safe, other improvements have been made. Internal framework data structures can no longer be constructed at all with invalid data, meaning that a manifest with an invalid path will now immediately throw an error with a clear message explaining why.

Many conditions that can cause issues in-game are now caught immediately by the framework and prevented; for example, a dependency which is missing or a texture which is the wrong size and will cause infinite loading screens.

Speed

The framework can now deploy all supported mods on the Nexus in a total of 11 minutes. The cache additionally now deploys all cached files in a fraction of a second, meaning no matter the size of your mod list, adding a new mod will only take however long that mod takes to deploy.

Platform support

v3 is intended to have first-class native Linux support, distributed alongside the Windows build.

Game support

The Simple Mod Framework now supports all game/engine versions in the World of Assassination trilogy, meaning you can now design and deploy mods against HITMAN™ (2016), HITMAN 2 or HITMAN 3.

A note on the GUI

The Mod Manager is also receiving a redesign as part of this update. Much of it is similar, but it has been given a cleaner and faster interface and now uses Tauri rather than Electron.

So what?

This post has been made to ask for your feedback. If you have any issues with the ideas above, any ideas of your own, any new features you'd like to see added or any issues you have with current SMF, feel free to share below!

Also, polls will be running on some changes in the Glacier 2 Modding discord server, which you can join here.

Other note

If you'd like to play around with the current iteration of the new manifest format, you can use this JSON schema with VS Code or in an online editor.

[martesi]

The present Mod Manager takes forever to load mods with large mount of files.

E.g., Any NPC Replaces Any Suit. Each mod has full index for every suit in game, the only difference between them would be the replace target. For better experience, can we expect at least one of below:

A dynamic replace behavior

Mods can require a form schema in the manifest, which would create a visual form in Mod Manager for this mod. In previous scenario, mod creator should supply a list for replace from, and convert the current indexes to list replace to. This design should be flexible for future expansion.

Shared index

If the previous one is hard to implement, maybe a shared index key can be specified so Mod Manager will reuse instead of recreate.

Non-blocking loading

It's understandable mods with lots of file take more time on I/O, but the UI blocking is really annoying. Maybe a background work queue can be introduced, all the file system operation can be avoided in UI? I mean the UI really only needs a small parts of the Mod information, which can be transformed from worker_threads(just example) to main then to UI.

And what's the latest as for Jul 2024?

[atampy25]

I have no idea what you're talking about with regards to a "shared index key" or what replace from and replace to are.

None of the code, or performance issues, from v2's Mod Manager are retained in v3. They stem from large amounts of IO done by the frontend, which is JS and which has to reimplement many parts of the framework and is therefore slow. The mod manager in v3, being a Tauri app, is integrated with the same logic used for deploy and can perform necessary tasks like validating mods more efficiently.

[martesi]

Replace from and replace to

For example, replace the signature suit to Abel De silva:

• replace from: the signature suit
• replace to: Abel De Silva

So we can have a form like:

<Form>
	{USER_CAN_ADD.map(() => (
		<Row>
			<Form.Item name={"from"}>
				<Select value={"The Signature Suit"} options={DEVELOPER_PROVIDED_0} />
			</Form.Item>
			<Form.Item name={"to"}>
				<Select value={"Abel De Silva"} options={DEVELOPER_PROVIDED_1} />
			</Form.Item>
		</Row>
	))}
</Form>

This will leads to a config passed to mods like:

const config = {
	form: {
		// the key data is config by developer
		// value of from/to can be different from their labels
		data: [{ from: "The Signature Suit", to: "Abel De Silva" }]
	}
}

Mods can use this config and return the actual deploy payload using operations() mentioned in your changes.

Shared index key

Never mind.
It assumes the Mod Manager processed every mod with their individual content/blob on start up. I just roughly looked the code, it doesn't do this on start up, and Deploy process seems only take the mod option as its target to patch. They looks fine.

IO done by frontend

This is what I'm talking about. I don't think fs should be used directly from the frontend though it's easier to operate without more verbose ipc communication. But it's sync and in the main thread, which might be the cause to the frontend freeze.

It's also wired since I can only see the frontend reads mod manifest on start up, yet the speed is really frustrating.

Imagine a download manager, with multiple download tasks executing in the same time. You can see how far did it go, interrupt and restart with new config. For example, running multiple mod installation concurrently. This is what's I am suggesting regarding non-blocking.

[atampy25]

The first part you've described is just how the options system already works in v3.

I know what non blocking IO is. The frontend in v3, being a Tauri app, does not do any filesystem operations because the point of Tauri is to offload most model and controller concerns to Rust via asynchronous IPC. In regards to "running multiple mod installation concurrently", the v2 architecture cannot support it because load order must be respected, not because of synchronous IO, and the v3 architecture already does that as explained in the "Graph model" section.

[martesi]

I was concerning about the direct usage for window.__TAURI__.fs, which is just the replacement in Tauri. Good to know that it's not the case. I thought the graph model is describing deploy only. What I mentioned is actually the add mod operation, which is different and way easier to implement.

It's excited to know that v3 has so much improvement, can we get the first public test version within this year?

[NeetBux-Hash]

I know the ui will be revamped but will it be possible to have multiple check boxes in a group instead of having to make a select type group? I think the ability to group in category would look nice. Example of my current solution. It's an eyestrain

[atampy25]

Option groups serve this purpose in v3, so you can create sections under headings with options inside them.

[TheChilliPL]

Has there been any progress with v3? The current version seems to be working really slow for me, and there's plenty of issues that happen since I've moved to Linux (and am running the mod manager through Proton). Moving to Tauri as well as some other mentioned performance improvements seem like they could really improve the experience of the mod framework

[atampy25]

There is still much work to go, but the current test build is on par with v2's feature set. If you like, I can add you to the testing Discord - send me a DM at @Atampy26 (or join the Glacier 2 Modding discord linked in the main post).

[dastarruer]

Curious what the progress is on this?