Enhance patching feature

[Elpisdev]

Description of change

This update introduces new features and improvements that make the patch management system more user-friendly, flexible, and efficient. Users can now easily fetch patches from an URL, select union patches, and directly apply patches to the game files in the standalone version of the program.

• Changed the way how game versions are identified using the PE header.

• Added support for a new type of patch called "Union Patch", which allows selecting one patch from a group of patches.

• Introduced the ability to specify a URL from where patches can be fetched, making it easier to keep patches up to date.

• Added a "Hard Apply" button in the standalone version of the program, allowing users to directly write the selected patches to the game files. (Useful for cabinet users)

• Removed Herobrine.

Testing

Tested with SDVX, IIDX, DDR, jubeat, POPN, DRS. Used my own URL to fetch patches.

Patch file

git diff
forsen.patch

[LupinThidr]

The formatting changes unfortunately make a reading this patch extremely difficult, but for now:

Why did you remove the old patches and patches.json loading entirely? Internet patches should just extend the currently loaded ones. (don't break existing behavior, everybody isn't playing the newest games)
Patch URL would be better as an fmt string so people don't have to conform to xxx/resources/id.json i.e https://example.com/retrieve.php?ver={}
Is there a timeout / does patch retrieving lock up if there's any issues? (DNS for example)

[Elpisdev]

The formatting changes unfortunately make a reading this patch extremely difficult, but for now:

Why did you remove the old patches and patches.json loading entirely? Internet patches should just extend the currently loaded ones. (don't break existing behavior, everybody isn't playing the newest games) Patch URL would be better as an fmt string so people don't have to conform to xxx/resources/id.json i.e https://example.com/retrieve.php?ver={} Is there a timeout / does patch retrieving lock up if there's any issues? (DNS for example)

• each version corresponds to a single json with this implementation. along with this change, i've also removed datecode entries which renders the already included patch.json unusable so it needs to be updated and split for each version.

• alright, updated the patch diff for new url format:
  forsen.patch

• there is no software side timeout in the code. i haven't tried to spam the query either but i don't see how it could break at all.

[sp2xdev]

I agree that the built in patches.json should not be removed. Plenty of people rely on this especially for offline cab use.

Have you also tested if a patches.json file (using existing format), placed next to spicecfg.exe also continues to be loaded & parsed correctly? That's another feature that people use today.

Especially the game detection logic - we should be really careful not to randomly break existing installs that rely on datecode-based matching.

Strongly suggest reverting any formatting changes unrelated to the feature. It pollutes source control blame history & makes things difficult to diff during reviews.

[Elpisdev]

I agree that the built in patches.json should not be removed. Plenty of people rely on this especially for offline cab use.

Have you also tested if a patches.json file (using existing format), placed next to spicecfg.exe also continues to be loaded & parsed correctly? That's another feature that people use today.

Especially the game detection logic - we should be really careful not to randomly break existing installs that rely on datecode-based matching.

Strongly suggest reverting any formatting changes unrelated to the feature. It pollutes source control blame history & makes things difficult to diff during reviews.

reading from local files can be added back. but as said in the previous response, it would not work with the built in ancient patches. it could be adapted if pe header values are identified for each version. the whole purpose of switching from datecode matching to pe header identification is not cosmetic, but for actual version detection rather than having to rely on ea3 config and such, but if that's "randomly breaking things" or "unrelated" for you, i have nothing to say. i'm closing this issue since absolutely no one wants change.

[Elpisdev]

for the whole time i thought you guys were refering to my changes on the json formatting instead of the code indentation at unnecessary places. sorry for the misunderstanding and the response earlier. i can fix these no problem. about the built in patches, as said before i would need pe values for all versions in there or i could just add in the old json logic just for the built in patches. so its either someone provides me the necessary dll files or we store two json logics in the code

[sp2xdev]

Yep, I was referring to code style changes (such as indentation) and not the JSON changes. Apologies as I wasn't entirely clear. I still think this is a cool feature, to move the patches out of spice but to external services that can be updated more frequently.

Please do keep the old logic in tact, for both datecode matching and parsing of the old JSON format, and keeping the built in patches as is. This new JSON format and PE-header based matching should be added as separate logic. It's OK if we have some duplicated code.

If you have a sample of the new JSON format, it would be appreciated for easier understanding of the code.

If there is a conflict; e.g., if a game matches based both PE-header based matching and the datecode-based matching - we can prefer the new patches, or perhaps they can be smartly merged somehow (although that might be messy).

Another interesting point for testing: Spice Companion actually also ships with patches.json, so that you can remotely apply patches. We would want to ensure that this feature continues to work (as the receiving end / API server anyway, the clients are assumed to be dead and no modifications made for years).

Due to personal reasons I have not had all that much time for this project. Since this is a significant contribution, it would be really appreciated if others can help with code review, feedback, and testing.

[Elpisdev]

Yep, I was referring to code style changes (such as indentation) and not the JSON changes. Apologies as I wasn't entirely clear. I still think this is a cool feature, to move the patches out of spice but to external services that can be updated more frequently.

Please do keep the old logic in tact, for both datecode matching and parsing of the old JSON format, and keeping the built in patches as is. This new JSON format and PE-header based matching should be added as separate logic. It's OK if we have some duplicated code.

If you have a sample of the new JSON format, it would be appreciated for easier understanding of the code.

If there is a conflict; e.g., if a game matches based both PE-header based matching and the datecode-based matching - we can prefer the new patches, or perhaps they can be smartly merged somehow (although that might be messy).

Another interesting point for testing: Spice Companion actually also ships with patches.json, so that you can remotely apply patches. We would want to ensure that this feature continues to work (as the receiving end / API server anyway, the clients are assumed to be dead and no modifications made for years).

Due to personal reasons I have not had all that much time for this project. Since this is a significant contribution, it would be really appreciated if others can help with code review, feedback, and testing.

• added back the old logic to accomodate the embedded patch json

• added the option to read a local folder for patch source

• cleaned the code format which resulted in a much smaller diff file

git diff
forsen.patch

example json that utilizes union type
17115126627070840.json

[LupinThidr]

alright, updated the patch diff for new url format:

I meant so the user can supply a full URL to be formatted, with retrieve.php as an example so a "URL format" doesn't need to exist.
So the user can input https://example.com/resources/{}.json or https://example.com/retrieve.php?ver={}

there is no software side timeout in the code. i haven't tried to spam the query either but i don't see how it could break at all.

The concern is if the patch server is offline, DNS trouble, or the user is offline, it shouldn't infinitely wait trying to retrieve patches.

[sp2xdev]

Can the new JSON patch format contain the DLL identifier string?

As in,
the old format looks like this:

{
    "gameCode": "M32",
    "dateCode": 2022102400,
    "name": "Skip Tutorial",
    "description": "",
    "type": "memory",
    "preset": true,
    "patches": [
      {
        "dllName": "game.dll",
        "dataDisabled": "0F87",
        "dataEnabled": "90E9",
        "dataOffset": 946042
      }
    ]

your example looks like this:

    {
        "name": "Shared mode WASAPI",
        "patches": [
            {
                "offset": 5686953,
                "dllName": "soundvoltex.dll",
                "dataDisabled": "04",
                "dataEnabled": "00"
            }
        ],
        "gameCode": "KFC",
        "type": "memory"
    },

Can we have this instead?

    {
        "name": "Shared mode WASAPI",
        "patches": [
            {
                "offset": 5686953,
                "dllName": "soundvoltex.dll",
                "dataDisabled": "04",
                "dataEnabled": "00"
            }
        ],
        "gameCode": "KFC",
        "pe_identifier": "17115126627070840" <---------------------------------------------
        "type": "memory"
    },

This way, we are not relying on the file name (or URL).

Why is this needed? It's because it's legal to have a giant patches.json file locally that has a mix of patches that apply to the current game version, and patches that do not apply. Therefore, the identification needs to be done at per-patch level, not per-file level. It would also help us to distinguish between old and new versions of the patch format.

[sp2xdev]

One more point to consider: I understand the convenience of this and many people want it, no disagreeing there....

But one concern I have is that this is technically one giant (intentional) Remote Code Execution. If someone compromises your patcher site, they can push bad code to your DLL, and people will unknowingly run them with admin privileges.

Obviously, they can still do this today, just with extra steps - someone can still hijack your patcher, inject bad code into the page, and let people download bad DLLs - but it shifts the responsibility to spice.

I'm not saying we need to stop this, but we might need to put some safeguards in place. I can't think of anything other than a big red warning label at the moment, but feedback from others is appreciated.

[LupinThidr]

RCE was also possible with patches.json and the spice API (password required), I don't think there's much you can do other than massive warnings and confirmations.

[sp2xdev]

Well, at least with patches.json, the user had to download a file, and with spice API, it required a password. But I agree, I don't think there is much that can be done.

[LupinThidr]

I think the original maintainer had similar concerns with integrating automatic updates / checking. Maybe a warning or confirmation + patch view button (ImGui hex editor) for patches longer than x bytes / x offsets? Most are usually a couple bytes in one or two offsets, then there's reflec groovin upper / DDR X3 song unlocks... and very partial 'translations'...

[sp2xdev]

SDVX Plus patches tend to be very long and complicated, but also widely used at the same time.

[sp2xdev]

My suggestion for the UI would be to:

• Have a button for "Import from URL"
• This opens a dialog window with giant warning about risks, plus a text input box where you enter the URL.
• Press Import to load the JSON once and save as a local patches_imported.json file on disk, next to spice executables.
• In subsequent boots of the game, check for existence of this patches_imported.json file and load patches as needed.

We should not be automatically pulling patches from the URL every time and applying them on launch. I would go as far as not saving the URL in the config file and instead require the user to enter it every time they want to import. Introducing additional manual steps as above serves sufficient warning for the user what is going on, and provide an opportunity for more advanced users to inspect the patches file before letting them be applied to the game before running.

I understand we want things to "just work", but I also don't want a situation where someone malicious creates a legitimate patcher website initially & convince many people to set it up as auto-patching, and then later change the patches to distribute malware, and be in the middle of it as a distributor. This would get us a lot of bad press in the community & and risk getting flagged by anti-virus software (not just by accident but added explicitly as malware medium). Last thing we want is another moral panic over how spice is a crpto miner...

I have some mixed feelings about the Hard Apply button as well... one of the original stated goals of spicetools is to run on clean data. It feels like it goes against that. If the runtime memory patching works, why do we need to write to the file? At least a confirmation dialog would be nice to have, explaining that original DLLs will be overwritten with no backups made. You'd be surprised how many people never make any backups of their game data ever, and re-download the whole thing if they mess up.

[SalimOfShadow]

I edited mon's patcher .js file and added a download JSON button,this works for both the standard spice2x JSON schema and the newly added Elpis one...I'll make a pull request to Mon's repo tomorrow

[Elpisdev]

Changes

• changed the url logic so user can enter whatever url path they want instead of just the main url

• added tooltip near the version string to show identifier(s)

• added more games that is relying on multiple modules

Q&A

• in the case lupin mentioned it doesn't infinitely try but rather expectedly freezes for a few seconds then gives data error

• hard apply is useful for cab owners. however as a precaution i will make the program take a backup of the module(s) with adding ".bak" extension to it and copy before applying hard patches. with this implementation, we don't need to add yet another warning box.

Ideas/Stuff to implement

• i will remove the text box from the main page

• upon clicking "set source", a dialog window featuring the warning and our text box will appear along with a "cancel" and "set source" button.

• i will add the pe identifier entry to the json structure for each patch

• i will make the program load "patches.json" with priority, treating it as a file that has larger set of patches featuring multiple versions. if no patches match the currently loaded module(s) or "patches.json" doesn't exist, the program will fall back to my idea where json files are named after each version identifier.

• after setting source, if the json fetching is successful, it will create a subfolder near spice executables called "patches" and it will be populated with either "patches_imported.json" file or "identifier_imported.json" file(s) depending on the occasion. the game will load from these until the user tries to set source again.

• the patch source won't be saved to the config anymore so the user will have to enter it manually every time

any objections?

[sp2xdev]

Sounds good to me.

One question though, what is the reason for cab owners not applying patches at runtime?

[Elpisdev]

Sounds good to me.

One question though, what is the reason for cab owners not applying patches at runtime?

this seems like more of a preference topic

they seem to find the requirement of plugging in a mouse/keyboard to the cabinet to set patches rather overwhelming. i also hear the "cab users might not use spice" excuse but i think this is too subjective and i thought you could think "then they shouldn't use spice at all". another thing is for example, for games like jubeat that run on XP, they will not be able to operate with spice2x at all. for now i'm not really sure what to do with this functionality so i'll wait for more opinions.

also more clarification on the new implementations:

there will be two extra folders on the spice directory. "remotepatches" for well, to store remote patches from given URL. and user-specified local folder. remotepatches will only process "_imported" json files, priority being on the "patches_imported.json". if "patches_imported.json" doesn't exist or doesn't have any corresponding patches to the loaded module, spice will attempt looking for "{identifier}_imported.json" instead. as for a local folder, spice will only process non-imported files with the priority being on "patches.json" then "{identifier}.json". this logic prevents conflict if the user ever decides to put a json file in the folder it doesn't belong to.

to keep track of which folder to use when loading patches, a boolean entry will be added to the patch config json. this only changes if the user changes the type of set patch source.

[sp2xdev]

So we have

.\patches.json (existing feature)
.\patches\patches.json
.\patches\12345.json
.\remotepatches\patches_imported.json
.\remotepatches\45678_imported.json

Is that correct?

I think we can drop the _imported from the file names, if they're going to be in a separate directory.

By patch config json you mean spicetools_patch_manager.json right? That's the right place.

[Elpisdev]

So we have

.\patches.json (existing feature)
.\patches\patches.json
.\patches\12345.json
.\remotepatches\patches_imported.json
.\remotepatches\45678_imported.json

Is that correct?

I think we can drop the _imported from the file names, if they're going to be in a separate directory.

By patch config json you mean spicetools_patch_manager.json right? That's the right place.

yes that's the json i was referring to. and actually now that i think about it, if we completely drop the "imported" naming on json files we can just use a single folder for every json and completely forget about the url/local classification.

[Elpisdev]

Changes

• Changed how the URL is entered and added a warning. URL won't be saved anymore.
• Prioritized patch.json and added the identifier entry to its structure so it can accomodate multiple versions (Haven't touched json files named with the identifier because that would be useless)
• All patches are saved to the "patches" subfolder. User can also place their own.
• Backup is taken in case of hard apply
• Games with multiple modules are handled correctly. (Ark engine loves it)

git diff:
forsen424.patch

[sp2xdev]

Do you have a URL that can be used to test?

Couple issues I found:

• During testing, I noticed that patches.json using the old format doesn't get loaded. I tried putting it next to spicecfg.exe, and also putting into patches folder - neither worked, but I expected both paths to work.
  • I used this example file: https://spice2x.github.io/patches.json which works with current public release of spicecfg.
• patches directory seems to be being created unconditionally. This should only really be created when importing patches from URL and saving a file.

Built-in patch database does seem to load properly.

[Elpisdev]

Do you have a URL that can be used to test?

Couple issues I found:

• During testing, I noticed that patches.json using the old format doesn't get loaded. I tried putting it next to spicecfg.exe, and also putting into patches folder - neither worked, but I expected both paths to work.

  • I used this example file: https://spice2x.github.io/patches.json which works with current public release of spicecfg.

• patches directory seems to be being created unconditionally. This should only really be created when importing patches from URL and saving a file.

Built-in patch database does seem to load properly.

• i am using my own url "https://p.eagate.turksigara.net/resources/". currently it supports a couple of recent versions for a few games

• the old format of patches.json only works with the embedded "IDR_PATCHES". so this is intentional. the reason for this is to completely discard the usage of datecode based version detection and migrate to an actual version detection which scans the module. i don't see a valid reason to keep using the datecode method.

• sure i can adjust that

[sp2xdev]

Just want to say up top that your contribution & patience through the review process is appreciated.

i am using my own url "https://p.eagate.turksigara.net/resources/". currently it supports a couple of recent versions for a few games

I have not been able to access https://p.eagate.turksigara.net/resources/. I tried with a handful of new-ish games but it results in 404. Can you let me know what version of a game I can try out? Maybe the full URL for the JSON as well?

the old format of patches.json only works with the embedded "IDR_PATCHES". so this is intentional. the reason for this is to completely discard the usage of datecode based version detection and migrate to an actual version detection which scans the module. i don't see a valid reason to keep using the datecode method.

I already explained this previously, but this isn't really up for a debate. It's easy to introduce breaking changes as a contributor, but as a maintainer I must avoid it at all costs. This may be an obvious thing to state, but as an open source project, the burden is on the maintainers to maintain submitted code, forever.

We already have people using patches.json written against the existing format. Some people rely on this to boot their game, or at least get it to a playable state. If they upgrade to a version of spice that doesn't support this existing patches.json file, their games would stop working, or behave differently. Most likely this will result in issues being filed here, or people asking in various discord servers why their game stopped working. The way to "fix" it would be to:

1. Roll back to previous version of spice2x, and remain on it forever, not receiving updates
2. Find an online patcher that has the specific version & specific hex edit they need (which may not always be available; e.g., for things like omnimix patch on an older IIDX version)
3. Update the patches.json file to use the new PE header ID format, which requires some level of technical knowledge (which you cannot expect form most people who just want to play a game)

There are also patcher websites that let users download .json files, again, using the existing format. All of that would just stop working. Sure, they can also start adopting the new JSON format, but now they have to tell everyone to upgrade spice2x first before they can use the new JSON files. Again, this is unnecessary burden on other to provide (free) troubleshooting & support for others.

Your code already has logic to parse the old JSON format (load_embedded_patches). Can we not run this logic against local patches.json file, and then run load_local_patches_for_dll again to check for patches the new format?

Until this issue is resolved, I can't accept this change into a release. If you really feel strongly about not adding support for this, that's fine, just let us know, and we'll find someone to add that support back.

Couple other things I want to mention:

• We have someone on the team working on UI polish. This will get integrated later, once the main logic is accepted.
• I'm looking to you (@Elpisdev) to provide documentation for the request and response format. Namely, what the algorithm is to come up with the URL (PE header logic), and what the JSON format is expected to look like, along with a small sample JSON. If you can provide a doc in markdown, I can just add it to the wiki. This would be much appreciated so that other people can write their own patcher sites without having to decipher the parsing logic.

[aixxe]

In case it helps, I've uploaded some of the generated files from my conversion script for IIDX 30 final, which should work when using https://ldj.mempatch.pages.dev/ as the URL. Briefly tested with #161 (comment) applied and seems okay.

Here are direct links to the JSON files:

• 169338878917004372.json (LDJ-003-2023090500)
• 16933884898669968.json (LDJ-010-2023090500)
• 16933891397830976.json (LDJ-012-2023090500)

The patch filename is determined by: (using LDJ-003-2023090500 as an example)

• IMAGE_FILE_HEADER->TimeDateStamp: 64EF0FF5 => to decimal => 1693388789
• IMAGE_OPTIONAL_HEADER->AddressOfEntryPoint: 1037754 => to decimal => 17004372

Which is then concatenated to produce the final filename: 169338878917004372.json

[Elpisdev]

alright i'm thinking of two ways to re-support datecode method:

• scan patches.json. if no identifier entry is found, look for datecode instead

• in the new "patches" folder create and use a subfolder called "legacy" and inside only read patches.json which will be treated the usual current way. ignore anything else that has anything to do with this entire feature request

i don't think people would mind moving their existing json to a new location
i'm still not sure on allowing spice to scan patchfiles in modules or within spice directory. for the sake of organizing i'd say we could only use the "patches" folder
let me know which one you prefer or let me know if you can think of something better

as for documentating the logic, i think aixxe has it all covered on the comment above. let me know if another thing has to be addressed

[sp2xdev]

i don't think people would mind moving their existing json to a new location

I don't agree with this. The old format JSON in the root should continue to work.

i'm still not sure on allowing spice to scan patchfiles in modules or within spice directory. for the sake of organizing i'd say we could only use the "patches" folder

I agree that new patches should stay in the patches folder.

Can we use old logic for \patches.json, and assume everything in patches folder uses the new logic? I don't think we should have a folder for legacy.

If that's too confusing for users, maybe the folder should be called pe_patches or something to differentiate.

as for documentating the logic, i think aixxe has it all covered on the comment above.

Agreed.

[Elpisdev]

i don't think people would mind moving their existing json to a new location

I don't agree with this. The old format JSON in the root should continue to work.

i'm still not sure on allowing spice to scan patchfiles in modules or within spice directory. for the sake of organizing i'd say we could only use the "patches" folder

I agree that new patches should stay in the patches folder.

Can we use old logic for \patches.json, and assume everything in patches folder uses the new logic? I don't think we should have a folder for legacy.

If that's too confusing for users, maybe the folder should be called pe_patches or something to differentiate.

as for documentating the logic, i think aixxe has it all covered on the comment above.

Agreed.

so you're telling me patches.json can be located near spice or modules folder but this json will be parsed with old logic. and new jsons with the pe stuff will be located in the patches folder. is it correct?

[sp2xdev]

This is where we currently check for patches.json, turns out there are three:

patch_manager.cpp:427

        // automatic patch file detection
        std::filesystem::path autodetect_paths[] {
                "patches.json",
                MODULE_PATH / "patches.json",
                std::filesystem::path("..") / "patches.json",
        };

and new patches can be in the new patches folder (or pe_patches folder if you want to differentiate)

[Elpisdev]

the priority has been changed to:

1. patches/{id}.json(s) (new)
2. patches/patches.json (new)
3. patches.json (old) (same locations)

git diff
forsen0428.patch

[sp2xdev]

Thanks. I'm going to be making the following changes:

• undo some of the indentation changes
• undo linter changes (namely a lot this-> got removed from member functions)
• add back in some UI polish work

after some testing, will let you know here when it's integrated, and put out a release shortly.

[sp2xdev]

I'm making slight changes to preserve backwards compatibility:

from

    std::filesystem::path LOCAL_PATCHES_JSON_PATH_OF("patches.json");
    std::filesystem::path LOCAL_PATCHES_JSON_PATHS_NF[] {
        "patches/patches.json",
        MODULE_PATH / "patches.json",
        std::filesystem::path("..") / "patches.json" };

to

    // possible locations of old format JSON patches
    std::filesystem::path LOCAL_PATCHES_JSON_PATHS_OF[] = {
        "patches.json",
        MODULE_PATH / "patches.json",
        std::filesystem::path("..") / "patches.json" 
        };

    // possible locations of new format (PE header ID) JSON patches
    std::filesystem::path LOCAL_PATCHES_JSON_PATHS_NF[] = {
        "patches/patches.json"
        };

[sp2xdev]

Found another issue - when reading in a local patches.json using the old format, datecodes aren't being checked, so it loads in all the patches (as long as the three character gamecode matches). Working on adding this logic back in.

[sp2xdev]

One more issue - we're looking for offset field in the old JSONs, but it should be dataOffset. Just going add in logic for to checking both.

[sp2xdev]

Issues above are resolved now, but I found a few more issues that I need to work through:

1. the log always contains W:patchmanager: Failed to open file: <full path to modules directory> Not sure where this is coming from, needs to be debugged. -> should be resolved now.

2. union patches always generate an entry in spicetools_patch_manager.json, which is auto loaded unconditionally. I would like to avoid this. All union patches should have a default option which is to not patch any bytes in, and this should not rely on the patch server sending the right JSON.

I'm not sure how to fix this. I would have liked the UI to look like this:

[ ] Choose Custom TDJ Timing/Adapter FPS: [-]"

[v] Choose Custom TDJ Timing/Adapter FPS: [60 fps]"

but patchers right now assume one of the drop-down options contain the default option, which means the checkbox will be always selected - needs a bit more thought. I don't think solving this is high priority.

Alternatively we could add a Reset button for non-default settings, and clicking on it reverts to what's in the DLL:

[v] Choose Custom TDJ Timing/Adapter FPS: [60 fps] [Reset]"

3. union patches saved in spicetools_patch_manager.json only have the game code (e.g., LDJ) and not the date code or the pe header. This means patches from one version "leak" into other versions of the same game (e.g., iidx 24 vs. iidx 30). It really should have just used the hashing algorithm (modification of patch_hash function). Needs to be worked out. -> resolved.

For example, I might want to have Choose Skip Monitor Check FPS: set to `120fps" and auto-loaded for iidx30 LDJ-010, but not for iidx30 LDJ-003. Currently, it's not possible to do so, which is unexpected (when compared to the Memory patches).

4. for non-union patches auto loaded from new style JSON, patch_hash doesn't include pe identifier, which means patches will again leak into other versions of the same game if they have the matching name/description. -> resolved.

Same as above. Patches in iidx29 should not be auto-applied to iidx30.

5. When DLL contents do not match any of the options in union patches, the no entry is selected in the drop-down, with no indication that there is a mismatch. I think this could use better UI. Unresolved but lower priority.

[sp2xdev]

Here is the archive of the source file so far: (removed - see alpha release below)

Issues #2-4 mentioned above need to be resolved before it gets integrated into a release. I don't have the time or energy to resolve all of them.

[KuuaDev]

It's not a high priority right now I think, but I tried writing mockup Wiki documentation for the new format. Some descriptions are missing, and perhaps some things are wrong or might still change, but I hope it would be a good starting point for when the feature is integrated into a release.
patching_format.md

Edit 2024-04-30_17:21;
patching_format.md

[sp2xdev]

Here is an alpha / pre-release version:

https://github.com/spice2x/spice2x.github.io/releases/download/24-04-30/spice2x-24-04-30_r0000.zip

Please report bugs in this thread.

[sp2xdev]

Slight tweaks, calling this r0001:

https://github.com/spice2x/spice2x.github.io/releases/download/24-04-30/spice2x-24-04-30_r0001.zip

• PE identifier format changed. 169338878917004372 Is now LDJ-64ef0ff5_1037754 (3-char game code, timestamp in lower case hex, and entry point address in lower case hex). Applies to peIdentifier in JSON and also the URL. This is a breaking change, hopefully this isn't too much of a hassle for @aixxe and @Elpisdev.
• Copy button added next to the identifier in the patches tab.
• Changed strings for (Locked)

An example is hosted on https://spice2x.github.io/patches/ (example file hosted at https://spice2x.github.io/patches/M32-634910da_15daec.json)

[LupinThidr]

@sp2xdev would you be willing to allow patch submissions to this repo / occasionally convert mainline bemanipatcher commits to the new format, or is that going to be purely for example only?
though there's aixxe's mirror, so it may be unnecessary

[sp2xdev]

@sp2xdev would you be willing to allow patch submissions to this repo / occasionally convert mainline bemanipatcher commits to the new format, or is that going to be purely for example only? though there's aixxe's mirror, so it may be unnecessary

Just an example, no plans to host and maintain patches.

[aixxe]

• PE identifier format changed. 169338878917004372 Is now LDJ-64ef0ff5_1037754 (3-char game code, timestamp in lower case hex, and entry point address in lower case hex). Applies to peIdentifier in JSON and also the URL. This is a breaking change, hopefully this isn't too much of a hassle for @aixxe and @Elpisdev.

No problem. I've updated my mirror repository to use the new format. Briefly tested with r0001 and appears to be working.

[KuuaDev]

I think I found something which might be unexpected behaviour (r0001);

1. Used a patches.json in the old format and applied patches.
   
2. Imported from @aixxe 's mirror repository. patches/LDJ-64ef0ff5_1037754.json was correctly written.
3. The patch titles were not green anymore, indicating the patches weren't going to be applied, which I'd expect given the difference in hashes. However, the checkmarks are still present.
   
4. Relaunching spicecfg made the checkmarks disappear.
   

[sp2xdev]

I think I found something which might be unexpected behaviour (r0001);

Thanks for reporting. Should be fixed here:

https://github.com/spice2x/spice2x.github.io/releases/download/24-04-30/spice2x-24-05-01-r0002.zip

also contains some minor UI / logging updates and code refactor.

[aixxe]

I've got two minor patches for the spice2x-24-05-01-r0002 version.

The first is more of a refactoring around the SECOND_DLL_FOR logic to avoid repeating the game-specific comparison stuff in multiple places, but also adds coin.dll to the jubeat 'extra DLLs' list, since there's a jubeat saucer fulfill patcher that modifies it.

The second adds in PE identifier comparisons for the old patches.json format. Ran into this while trying to combine all the converted patches into a single file, but using peIdentifier rather than dateCode for version matching. On r0002 it would seemingly load all patches as long as the gameCode matched, and if you had any unions set, probably crash on memcmp trying to read some offset from a different version.

Attached the combined patches.json for reference. Tested with a clean Docker build, but not too extensively just yet.

• 0001-Refactor-SECOND_DLL_FOR-to-allow-more-than-one-addit.patch
• 0002-Add-filtering-by-PE-identifier-in-old-patches.json-f.patch
• patches.json

[sp2xdev]

I've got two minor patches for the spice2x-24-05-01-r0002 version.

The first is more of a refactoring around the SECOND_DLL_FOR logic to avoid repeating the game-specific comparison stuff in multiple places, but also adds coin.dll to the jubeat 'extra DLLs' list, since there's a jubeat saucer fulfill patcher that modifies it.

The second adds in PE identifier comparisons for the old patches.json format. Ran into this while trying to combine all the converted patches into a single file, but using peIdentifier rather than dateCode for version matching. On r0002 it would seemingly load all patches as long as the gameCode matched, and if you had any unions set, probably crash on memcmp trying to read some offset from a different version.

Attached the combined patches.json for reference. Tested with a clean Docker build, but not too extensively just yet.

* [0001-Refactor-SECOND_DLL_FOR-to-allow-more-than-one-addit.patch](https://github.com/spice2x/spice2x.github.io/files/15178563/0001-Refactor-SECOND_DLL_FOR-to-allow-more-than-one-addit.patch)

* [0002-Add-filtering-by-PE-identifier-in-old-patches.json-f.patch](https://github.com/spice2x/spice2x.github.io/files/15178564/0002-Add-filtering-by-PE-identifier-in-old-patches.json-f.patch)

* [patches.json](https://github.com/spice2x/spice2x.github.io/files/15178567/patches.json)

I've integrated the first patch.

For the second patch, what's the delta between load_from_new_patches_json and load_from_old_patches_json now? Feels like we could just use the same logic for both locations (patches.json and patches\patches.json).

[aixxe]

For the second patch, what's the delta between load_from_new_patches_json and load_from_old_patches_json now? Feels like we could just use the same logic for both locations (patches.json and patches\patches.json).

That makes sense. It was a bit unintuitive that the file would have the same name but be parsed differently depending on which folder it's in. I've tried combining the two loading functions and haven't noticed any side-effects yet from some brief testing.

• 0003-Combine-patches.json-loading-functions.patch

The combined JSON from my previous comment loads as expected when placed in either patches.json or patches/patches.json now. The more specific PE-based filename patches still load first if they exist, which seems correct.

Tried using the old build/patches.json too, which also loads from all the same paths. I'm not too familiar with what's different between 'old format' and 'new format' though, so I might've missed something obvious.

[sp2xdev]

r0003: https://github.com/spice2x/spice2x.github.io/releases/download/24-04-30/spice2x-24-05-02-r0003.zip

• Integrated 3 patches above from aixxe
• Update UI for union patches, clicking on the partially-checked checkbox resets drop-down to default value

[sp2xdev]

I put out a beta release on the site with these changes. Thanks everyone for contributing code, submitting suggestion, and reporting bugs. Especially @Elpisdev for patiently (and positively) responding to feedback.

https://github.com/spice2x/spice2x.github.io/releases/tag/24-05-04

[mon]

This is a pretty nifty feature, and I'm here to ask some questions about making bemanipatcher more compatible with it. Feel free to @ me in future patch conversations if bemanipatcher is making your life difficult :P

1. Am I reading the summary correctly, that old spice cannot/fails to interpret the new format?
2. ...and new spice can correctly read both formats?
3. Would it be beneficial to make bemanipatcher load a .json, so that spice2x and bemanipatcher are in sync? (without needing an explicit conversion step/download button).
4. Is the current metadata in bemanipatcher's patch description sufficient, or does it need expanding? Based on aixxe's converter, it seems like it needs to extract the gamecode to work properly, or was that requirement removed?

This is mostly due to Salim's PR causing a bunch of "hmmmmm" questions - it uses heuristics to try and extract the gamecode/datecode, and not having to do that would be beneficial.

[sp2xdev]

1. Am I reading the summary correctly, that old spice cannot/fails to interpret the new format?

Correct.

2. ...and new spice can correctly read both formats?

Yes.

3. Would it be beneficial to make bemanipatcher load a .json, so that spice2x and bemanipatcher are in sync? (without needing an explicit conversion step/download button).

There isn't a 1:1 conversion right now, so it might not be possible to convert your entire site to the JSON format. For example, spice2x doesn't understand "number range" format used for some offset patches.

4. Is the current metadata in bemanipatcher's patch description sufficient, or does it need expanding? Based on aixxe's converter, it seems like it needs to extract the gamecode to work properly, or was that requirement removed?

The new format introduces a hash generated using the PE header of the DLL. This allows DLLs to be detected better (e..g, if you have 003 DLL vs. 010 DLL for IIDX) without relying on datecode. This is optional though.

You can still use the old datecode matching & allow the user to download a patches.json file. Users will need to somehow figure out what DLL they have, so that they can download the correct JSON file... which could be a bit confusing.

[mon]

For example, spice2x doesn't understand "number range" format used for some offset patches.

Does it just ignore them, or does it fail to load/log_fatal? I'm wondering if I can include everything now and support comes later.

You can still use the old datecode matching & allow the user to download a patches.json file

Is the datecode/mcode required, or can I just generate patches and trust the user (ha)?

This is an extreme edge case, but translating strings inside the .dll by adding a new section header will probably break the PE header hash. Would spice ever use bemanipatcher's "just throw every patch set at the wall until something sticks" approach?

[sp2xdev]

Does it just ignore them, or does it fail to load/log_fatal? I'm wondering if I can include everything now and support comes later.

Just tried it and it just shows up in the UI as (Error) and cannot be applied, which seems good enough.

Is the datecode/mcode required, or can I just generate patches and trust the user (ha)?

For patches.json: you need the gameCode and dateCode at the minimum (this is the behavior since old spicetools).

For files in the patches directory: as long as the filename is correct (e.g., M32-634910da_15daec.json), any patches inside it will show up, you only need the gameCode.

This is an extreme edge case, but translating strings inside the .dll by adding a new section header will probably break the PE header hash. Would spice ever use bemanipatcher's "just throw every patch set at the wall until something sticks" approach?

To be accurate, it's not a hash of the whole header, it only looks at the PE timestamp and AddressOfEntryPoint. So even if you modify or add new sections, those won't change.

[mon]

Thanks for the clarifications. I think I can get away with a big refactor to add proper datecodes to all the existing patchers, but I definitely can't get copies of all the DLLs to make the hashed names, so it'll have to be a download button and trust the users.

That's acceptable to me, so I'll continue working on my rewrite.