Suggestion: INI layering or diffs

[replicaJunction]

Problem

Mods that make changes to configuration files currently need to include a full copy of the files they change. This means they're incompatible with each other. If a user wants to use more than one mod that changes the same file, they need to manually merge the contents of the two files.

Proposed solution

Allow mods to provide a partial INI file, containing only the keys they want to modify. This program would handle overwriting the specific keys and values provided in those files without changing the other values.

Mods could also be expected to provide a different format (in case something like JSON was easier to deserialize and parse), as long as the program ended up writing the correctly formatted INI file to the config directory.

Use cases

As a user / player of the Mass Effect games, it would be really nice to be able to apply multiple small patches to INI configuration files without doing the manual work of tracking down what changed and trying to merge changes from a couple of different items.

As a developer, it would be great to be able to say "this mod is fully compatible with other config mods that don't mess with the key [key name]."

Examples

ME1Controller, Improved MAKO, and Gameplay Overhaul all modify BioGame.ini.

ME1Controller and Skip Dialogue Without Choosing Answer both modify BioInput.ini. The latter is explicitly flagged "only works with ME1Controller" because it used the former mod's config file as a base; what if I want to use KB&M controls and apply the Skip Dialogue mod?

[Mgamerz]

How do you handle duplicate keys? Unreal ini files will contain many duplicate keys. How do you suppose you would target a specific one for operating on?

…

On Fri, Sep 4, 2020, 8:25 PM Joshua T ***@***.***> wrote: Problem Mods that make changes to configuration files currently need to include a full copy of the files they change. This means they're incompatible with each other. If a user wants to use more than one mod that changes the same file, they need to manually merge the contents of the two files. Proposed solution Allow mods to provide a partial INI file, containing only the keys they want to modify. This program would handle overwriting the specific keys and values provided in those files without changing the other values. Mods could also be expected to provide a different format (in case something like JSON was easier to deserialize and parse), as long as the program ended up writing the correctly formatted INI file to the config directory. Use cases ##As a user / player of the Mass Effect games,** it would be really nice to be able to apply multiple small patches to INI configuration files without doing the manual work of tracking down what changed and trying to merge changes from a couple of different items. *As a developer*, it would be great to be able to say "this mod is fully compatible with other config mods that don't mess with the key [key name]." Examples ME1Controller <https://www.nexusmods.com/masseffect/mods/60>, Improved MAKO <https://www.nexusmods.com/masseffect/mods/115>, and Gameplay Overhaul <https://www.nexusmods.com/masseffect/mods/96> all modify BioGame.ini. ME1Controller <https://www.nexusmods.com/masseffect/mods/60> and Skip Dialogue Without Choosing Answer <https://www.nexusmods.com/masseffect/mods/200> both modify BioInput.ini. The latter is explicitly flagged "only works with ME1Controller" because it used the former mod's config file as a base; what if I want to use KB&M controls and apply the Skip Dialogue mod? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#141>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAU4VFFSFMCHYQTYM43M5A3SEGOSNANCNFSM4QZ74KTQ> .

[mirh]

Putting aside that ME1Controller really is a fucked up mod (and Dybuk did a herculean job in having it working at all), I asked about something similar a month ago.

Would it be possible to make ME1RE like open source? ... in the sense that changes are in some plain format on, say, github
the real problem if any would be this is probably as distant as possible from the final package that you'd have to ship, if I can explain
and therefore a lot of reprocessing and tooling in-between would be needed

can't you see everything the mod does by opening the mod files in ME3explorer?

But you are missing out on the collaborative aspect of the thing if people can just "cooperate" by sharing binary files

And eventually @SirCxyrtyx told me this

are you suggesting decompiling package files into a textual format?
it's an interesting idea mirh. Some things would have to stay in binary format (textures, meshes, etc) but most of a package file (properties) could be turned into text in a way that would be possible to recompile

[Mgamerz]

I don't see how there is anything wrong with ME1Controller. This topic is about ini files, not comparing package files, there is no such thing as closed source dlc mods, to view the changes you'd use the exact same tools the developer used to make the mod.

[Kinkojiro]

A couple of issues with making a mod like ME1RE open source (even if we could):

• Firstly I think it would be really difficult to make only ME1RE open source. Effectively we would have to make the underlying files, including our edits, open source. I am not sure that is something EA/Bioware or Github would want or condone.
• Secondly problem is quality control. As a modder I can control the content of ME1RE and take responsibility for the bugs and fix them. If you start adding other peoples edits this quickly can get out of hand and I would be the person that players come to if something gets broken.
• Third issue is why would I, as the creator, want to do this. I don't make any donation points on ME1RE. But somebody could take my open source work, put it on Nexus and then start collecting money. They can legitimately say - open source - I recompiled it mine now.

[mirh]

1) Edits are your creation somewhat, and anyway I don't see how that's any different from redistributing the binaries
2) Putting aside that I really don't see hordes of people flocking to contribute, you are still free to reject what you don't like at the end of the day (if any, this would lead to *less* burden if you ask me)
3) For the same reasons they already can not? Stealing is bad (I can hardly think to a license that doesn't assert attribution), forking is something different. And it's the same thing you basically did with ME3Explorer, isn't it?

4) I mean, clearly ME1Controller is subpar if switching between m+kb and controller also requires switching files. But for the moment it's the best that it was possible to pull off from the limited ME1 engine (but I'm digressing).

This topic is about ini files, not comparing package files

I still feel like they are both part of the same problem (of course then, tinkering with a file that is plain text to begin with, is leaps and bounds easier than with upks)

[Mgamerz]

3. ME3Explorer was licensed under GPL by requirement of it using LZO, which is licensed under GPL. Mods are under no such license. I was perfectly free to fork the project, in fact I even offered my changes upstream, which were rejected. So I just continued my fork until they fully abandoned upstream. That's how open source development works under GPL.

4. This happens on all controller mods. It's pretty rude of you to just assume our mods are subpar because of the inability to switch back and forth when you don't even know how the system works. Moonshine, Dybuk and I put in countless hours of research for those mods.

None of this discussion has anything to do with the original topic of this issue so move it somewhere else if you want to continue the discussion.

[mirh]

I was perfectly free to fork the project, in fact I even offered my changes upstream, which were rejected.

And I see nothing wrong with that (I mean, they could have accepted them but so is life I guess?). What's the deal?
ME1RE specifically is off topic here then, but in the sense that I stated above the core issues are still more or less the same.

It's pretty rude of you to just assume our mods are subpar because of the inability to switch back and forth when you don't even know how the system works.

The features are subpar with respect to "ideal functionality". I don't know why that would disrespect anyone (except perhaps those monsters at demiurge/epic)
I was also there back in the days and I'm not sure how I could underline any more my admiration for the gods.
Why.. how's half of what I say always interpreted aggressively?

[replicaJunction]

Just to be totally clear, this feature request isn't specifically about either ME1RE or ME1Controller. I only referenced ME1Controller because it was a convenient example.

How do you handle duplicate keys? Unreal ini files will contain many duplicate keys. How do you suppose you would target a specific one for operating on?

I'm not intimately familiar with UE3, but most INI files I've seen don't allow duplicate keys in the same section. The key could be uniquely identified by the file name, section name, and key name. I know the INI format doesn't have a really firm spec, though; does this not work for Unreal?

As far as multiple mods changing the same key, I've got a couple different ideas, depending on how elaborate you wanted to go:

1. No specific handling - most recently applied mod wins. (This could also be included in part of the "batch mod" functionality.)
2. Provide the user a notification that there's a conflict and ask which version they want to use.

This might actually make it easier to provide compatibility patches if two mods legitimately need to change the same key, since the compatibility patch would only need to contain the conflicts. A dev could manually merge them once and publish the patch, and the user would apply the patch after both of the original mods.

[Mgamerz]

Unreal for me1 and 2 supports duplicate keys. Me3 has its own system for handling this type of scenario as they do it in dlc, you have to supply the line to remove and the new one is appended to the end. Underlying system is technically the same.

Keybinds for example use duplicate keys.

This idea is simple in practice but based on my work supporting RCW mods for me2 (essentially the same thing but made by a third party tool) has shown me how difficult this problem is to solve. RCW avoided it by simply basing everything on a vanilla file, so if you apply a mod, it resets and applys to vanilla. I'm not sure how the program handled conflicts, if any. Leaving things up to users to determine the right thing to do on technical issues is a bad idea. Many of my users couldn't figure out how to apply a compatibility pack in mod manager, I cannot see them understanding unreal ini key names.

It may be possible to extend RCW format to me1 since they are similar. ME1 presents unique challenges as the game and user settings are in the same files. Additionally, me1 config files are localized, so what may work on one localization may not work on another. Another issue would be building a file - I don't have time nor the inclination to build a UI builder like RCW - it would have to be all be done by hand (which isn't too tedious, honestly).

I'm not sure how this would work with the fun little iniversion issue me1 seems to have that reset the ini files.

[mirh]

Very simple idea: why couldn't you use the standard unidiff patch format?
The optional "functions name" heading would be replaced by the class name, so you can't really screw with duplicates.
This format could even be somewhat shared with other kind of data (like for example in-game text adjusements), and a ; comment could even help the user figure out what set what whenever two mods conflict.

...
Or why would you even differences at all (be it from a vanilla game file, or a previous version of the same value)?
AFAICT with inis, you are only just concerned with setting values (or at most removing them).
You can just ship your desired changes/lines, and only them, and if they already exist you change their values, otherwise you append the new value (through the previously mentioned ; comment mechanism you could still detect whenever two mods are fighting for the possession of the same value)

[Mgamerz]

For example, the game ini files (in documents/bioware/me1) are modified by the ME1DLC mod enabler. Duplicate lines are added to let the game find the movie files in multiple locations. This is computed and written at game boot. I'm not opposed to this idea, in fact I like it, but I am opposed to introducing yet another mod standard, as I would have to be the one to foot the research and development time plus the support of it. A diff of two files may work, but there is no guarantee the user will have the same files or ones close to the original that was used. I look through the alot diags people post and some people pack the game to the gills and wonder why something doesn't work. The duplicate keys is what I think would be the most difficult, it is the same problem I had to work around for RCW and a long time ago when working with ModMaker XML. I think looking at the current RCW implementation in M3 would be best, it is an established 'standard'. The one weakness it has is that it will not support adding a duplicate key of one does not already exist, if I recall (e.g. it could not add a second copy of a key, but it could add a third). Coalesced.ini is essentially the same thing as the config files but bundled into a single serialized file (it's not an actual ini file), it would not be too difficult to extend this to work. Conflicts are still an issue - in my RCW implementation, errors are only logged if things to subtract can't be found. Adding keys isn't really an issue.

…

On Sat, Sep 5, 2020, 3:56 PM mirh ***@***.***> wrote: Very simple idea: why couldn't you use the standard unidiff <https://www.gnu.org/software/diffutils/manual/html_node/Detailed-Unified.html#Detailed-Unified> patch format? The optional "functions name" heading would be replaced by the class name, so you can't really screw with duplicates. This format could even be somewhat shared with other kind of data (like for example translations), and a ; comment could even help the user figure out what set what whenever two mods conflict. ... Or why would you even differences at all (be it from a vanilla game file, or a previous version of the same value)? AFAICT with inis, you are only just concerned with setting values (or at most removing them). You can just ship your desired changes/lines, and only them, and if they already exist you change their values, otherwise you append the new value (through the previously mentioned ; comment mechanism you could still detect whenever two mods are fighting for the possession of the same value) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAU4VFEBWY4IY7V5AEW276TSEKX2DANCNFSM4QZ74KTQ> .

[mirh]

https://xkcd.com/927/
The risk is this, as always.
But if there's somebody that may actually pull out something better and comprehensive it is you (plural).
If the games modding/internals understanding itself is still WIP though, clearly it may still be too early to discuss this.
And anyway, of course just like with anything else, this is just yet another TODO on the wishful thinking list. You shouldn't feel pressed to work on anything you don't feel like to.

Duplicate lines are added to let the game find the movie files in multiple locations.

Mhh? Multiple locations? I thought you just appended (once) the right folders in [Core.System] and profit.
If any, what I had been missing out is that the right *order* of variables is also important here.
But I don't really think this could ever be truly fixed.

[mirh]

https://github.com/ME3Tweaks/ME3TweaksModManager/releases/tag/7.0.0.118
https://github.com/ME3Tweaks/ME3TweaksModManager/blob/master/documentation/merge_mods.md
Stoked to see this coming.

[Mgamerz]

I'm closing this in favor of #255, which is far more tenable from a support perspective.