PatchEngine: Allow patching of files on disk #7982

phire · 2019-04-11T12:05:58Z

Resident Evil 2/3 require patching code which live in .rel files (Relocatable Executable)

Our current memory patching system makes this difficult, we would have to patch the rel loader to detect which .rel was loaded and apply the correct patch.

So I've implemented file based patches instead.
Here is the patch for Resident Evil 2 (U)

[FilePatch_enabled]
$fix dcache issue
[FilePatch]
$fix dcache issue
/leon.rel
0x1903f0:dword:0x42800010
/claire.rel
0x190288:dword:0x42800010

Current limitations:

Can't patch main.dol
Can't patch files not on the main partition
Can't patch data outside of files
Not hooked up to editor UI

JosJuice

We talked a little on IRC about having this code in the Volume classes instead. I think that would make sense, as long as we make sure that the patches only are applied when the disc is being accessed by emulation and not the game list, maybe by having a function Volume::ApplyPatches that is called by the boot code or PatchEngine that stores the patches in the Volume object.

Source/Core/Core/HW/DVD/DVDThread.cpp

JosJuice · 2019-04-11T12:22:09Z

Source/Core/Core/PatchEngine.cpp

+    const auto& file = fs->FindFileInfo(patch.path);
+    if (file) {
+      for (const auto& entry : patch.entries) {
+        u64 disc_offset = disc.PartitionOffsetToRawOffset(file->GetOffset(), partition) + entry.address;


This is wrong for Wii discs, since entry.address doesn't account for the hashes at the beginning of each cluster. (Unless you expect the person writing the patch to correct for the hashes manually?) Patching a Wii disc using the current code is inconvenient anyway, since you would need to handle the encryption manually...

If the code isn't going to support Wii discs properly, you might as well skip the call to PartitionOffsetToRawOffset.

Disregard the previous comment. You can get Wii discs to work, but you would need to remove this call to PartitionOffsetToRawOffset, and PatchReadRequest would need to check which partition it's applying patches to.

Source/Core/Core/HW/DVD/DVDThread.cpp

Source/Core/Core/PatchEngine.cpp

Source/Core/Core/HW/DVD/DVDThread.cpp

Source/Core/Core/PatchEngine.cpp

Source/Core/Core/PatchEngine.h

JosJuice · 2019-04-23T06:03:38Z

Source/Core/Core/HW/DVD/DVDThread.cpp

@@ -386,15 +386,15 @@ static void PatchReadRequest(std::vector<u8>& buffer, u64 offset)
      u64 start = it->first - offset;
      std::memmove(buffer.data() + start, it->second.data(),
                   std::min(it->second.size(), buffer.size() - start));
-      INFO_LOG(DVDINTERFACE, "patch applied at %08lx", offset + start);
+      INFO_LOG(DVDINTERFACE, "patch applied at %08llux", offset + start);


You should use PRIx64 for u64.

why?

Current versions of Visual studio support %llux

%llu works for u64 without warnings on all compilers now? I guess you could use that then.

phire · 2019-04-24T04:16:58Z

Source/Core/Core/Boot/Boot.cpp

  DVDInterface::SetDisc(std::move(volume), auto_disc_change_paths);
-  return pointer;
+  return DVDThread::GetDiscVolume();


This is a really ugly situation here. It might actually be cleaner to return to the state before PR #4241

Right now I'm just making it clearer that we are grabbing a pointer internal state of DVDThread (and grabbing it after DVDThread has applyed the patch overlay).

I think the cleaner solution might be to split the EmulatedBS2 into two parts. Part one which sets up the memory/state based on the volume which runs before passing the volume to DVDThread. Part 2 will just load and run apploader, directly calling DVDThread to process read requests (adding a new DVDThread::HLE_Read() which is only intended to be used during boot)

But I'm not sure, I welcome feedback on this issue.

i30817 · 2020-04-10T23:35:31Z

Since this is stalled, do you mind if i create a romhacking.net patch for the discs using xdelta using your offsets? And is this for both games or just one? And the offset is inside the files, not the whole disc or partition right?

mbc07 · 2020-04-10T23:38:30Z

Just RE2...

phire · 2020-04-11T00:54:24Z

Coincidentally, I'm rebasing this PR right now.

Those offsets work only for the US version of RE2. Offset is within the file.

i30817 · 2020-04-11T01:26:32Z

If it's happening i'll wait since this only emulator for the gamecube worth a damn. I'd prefer to do/get/replay both at once anyway.

We also switch from keeping two lists in sync to a system with a single list.

i30817 · 2020-04-26T13:57:19Z

Do you happen to have already researched the offsets for Resident Evil 3 too?

JosJuice · 2020-12-15T14:21:53Z

I would like to go with PR #9308 instead of this PR. Since it seems like that PR's approach works for fixing RE2/3, this PR's more complicated solution seems like overkill for just fixing those two games. And I don't feel like this PR is useful for a lot more than fixing RE2/3, as large-scale game patching (like adding new levels to a game) requires features that can't be provided by this PR's approach, like changing the size of files.

JosJuice · 2020-12-15T14:31:43Z

The Unifiy AR/Gecko/Patch GUI dialogs commit seems like a good idea regardless, though. If you can split it out to a separate PR, I'd like to try to get it merged.

phire · 2020-12-16T00:14:21Z

like changing the size of files.

The plan was that the backend disc patching architecture here is generic. It simply takes lists of modified ranges in disc address space. A later PR could add an improved frontend that would understand more advanced patch types (such as deleting files, adding new files, replacing files with bigger versions, rearranging the filesystem, maybe even creating oversized discs) in more specialized formats and generate the "list of modified ranges" for the disc patching backend to consume.

…

On Wed, 16 Dec 2020, 3:32 AM JosJuice, ***@***.***> wrote: The Unifiy AR/Gecko/Patch GUI dialogs commit seems like a good idea regardless, though. If you can split it out to a separate PR, I'd like to try to get it merged. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7982 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABBZ5C3ZX53YNEJ5HLXZ4LSU5XN7ANCNFSM4HFFY4EQ> .

smurf3tte · 2020-12-16T00:35:00Z

I would like to go with PR #9308 instead of this PR.

I don't think this is an either-or situation. The RE 2&3 patches will function identically no matter which application mechanism is used. They can be trivially converted to use either.

That said, I would still like to go ahead with my change. It's just a dozen lines of code in the patch engine; the bulk of the source changes are in the Qt UI, and that code will be thrown away entirely if this PR is completed, which wouldn't bother me in the slightest.

phire · 2020-12-16T01:05:33Z

Yeah, it will function near identically. I chose this option because:

Checking to see if static data has changed every frame and then patching it seems like a massive hack to me.
Patching on load is a lot cleaner because it only happens once, and there is no edge-case between loading and vblank.
I couldn't actually find a guarantee in the REed code that the executable would always be loaded in the same place. The memory address to load the REL comes from a memory allocater and would change if there is a different order of allocations.
Though experimental testing has so-far proved my fears wrong.
Generic file seems like it will be very useful for a lot of other modding use cases. Especially once there is support for changing the length of files and adding new files.

Conditional patches are also useful in their own way. Ideally dolphin wants both.

smurf3tte · 2020-12-16T01:17:38Z

Agreed on all points.

JosJuice · 2020-12-16T09:15:22Z

The plan was that the backend disc patching architecture here is generic.
It simply takes lists of modified ranges in disc address space.

A later PR could add an improved frontend that would understand more
advanced patch types (such as deleting files, adding new files, replacing
files with bigger versions, rearranging the filesystem, maybe even creating
oversized discs) in more specialized formats and generate the "list of
modified ranges" for the disc patching backend to consume.

Wouldn't this make the list of modified ranges absolutely huge once you start moving files around on the disc?

phire · 2020-12-17T01:33:21Z

Wouldn't this make the list of modified ranges absolutely huge once you start moving files around on the disc?

Really depends on how easy it is to extend the disc beyond 1.2GB (something I haven't looked into). If the Nintendo SDK library routines simply fall apart, then we will be forced to pack everything into 1.2GB, deleting and packing files down. A HLE approach might be preferable.

If it's easy to extend discs, then it's super simple.
Leave all deleted/replaced data in-place. A few ranges to patch the filesystem data and one big range at the end of the disc to hold all the new data.

JosJuice · 2021-12-20T19:16:48Z

I'd like to close this now that we have PR #10127. Any objections?

phire added the WIP / do not merge Work in progress (do not merge) label Apr 11, 2019

phire requested review from delroth, lioncash and JosJuice April 11, 2019 12:05

JosJuice reviewed Apr 11, 2019

View reviewed changes

delroth reviewed Apr 11, 2019

View reviewed changes

lioncash reviewed Apr 11, 2019

View reviewed changes

Source/Core/Core/HW/DVD/DVDThread.cpp Outdated Show resolved Hide resolved

Source/Core/Core/PatchEngine.cpp Outdated Show resolved Hide resolved

Source/Core/Core/PatchEngine.h Outdated Show resolved Hide resolved

phire force-pushed the filePatch branch from 27abbb0 to 45600e2 Compare April 23, 2019 04:53

JosJuice reviewed Apr 23, 2019

View reviewed changes

phire force-pushed the filePatch branch from 2ec9b7b to 6f9c4f2 Compare April 24, 2019 03:48

phire commented Apr 24, 2019

View reviewed changes

iwubcode mentioned this pull request Oct 19, 2019

InputCommon: Dynamic Input Textures #8318

Merged

phire added 4 commits April 11, 2020 12:00

PatchEngine: Allow patching of files on disk

ccb5811

PatchEngine: Fix code review comments

f5de45a

Use llu format specifier for u64

277a375

Move FilePatch Functality to a volume wrapper

7acc4c5

phire force-pushed the filePatch branch from 6f9c4f2 to daa92e8 Compare April 11, 2020 04:44

OverlayVolume now inherrets VolumeDisc

53d8f5e

phire force-pushed the filePatch branch from daa92e8 to 61a6be7 Compare April 11, 2020 16:10

Unifiy AR/Gecko/Patch GUI dialogs

d50c337

We also switch from keeping two lists in sync to a system with a single list.

phire force-pushed the filePatch branch from 61a6be7 to d50c337 Compare April 11, 2020 17:30

JosJuice closed this Jan 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PatchEngine: Allow patching of files on disk #7982

PatchEngine: Allow patching of files on disk #7982

phire commented Apr 11, 2019

JosJuice left a comment

JosJuice Apr 11, 2019

JosJuice Apr 23, 2019

JosJuice Apr 23, 2019

phire Apr 23, 2019

JosJuice Apr 23, 2019

phire Apr 24, 2019

i30817 commented Apr 10, 2020 •

edited

Loading

mbc07 commented Apr 10, 2020

phire commented Apr 11, 2020

i30817 commented Apr 11, 2020

i30817 commented Apr 26, 2020

JosJuice commented Dec 15, 2020

JosJuice commented Dec 15, 2020

phire commented Dec 16, 2020 via email

smurf3tte commented Dec 16, 2020

phire commented Dec 16, 2020

smurf3tte commented Dec 16, 2020

JosJuice commented Dec 16, 2020

phire commented Dec 17, 2020

JosJuice commented Dec 20, 2021

PatchEngine: Allow patching of files on disk #7982

PatchEngine: Allow patching of files on disk #7982

Conversation

phire commented Apr 11, 2019

JosJuice left a comment

Choose a reason for hiding this comment

JosJuice Apr 11, 2019

Choose a reason for hiding this comment

JosJuice Apr 23, 2019

Choose a reason for hiding this comment

JosJuice Apr 23, 2019

Choose a reason for hiding this comment

phire Apr 23, 2019

Choose a reason for hiding this comment

JosJuice Apr 23, 2019

Choose a reason for hiding this comment

phire Apr 24, 2019

Choose a reason for hiding this comment

i30817 commented Apr 10, 2020 • edited Loading

mbc07 commented Apr 10, 2020

phire commented Apr 11, 2020

i30817 commented Apr 11, 2020

i30817 commented Apr 26, 2020

JosJuice commented Dec 15, 2020

JosJuice commented Dec 15, 2020

phire commented Dec 16, 2020 via email

smurf3tte commented Dec 16, 2020

phire commented Dec 16, 2020

smurf3tte commented Dec 16, 2020

JosJuice commented Dec 16, 2020

phire commented Dec 17, 2020

JosJuice commented Dec 20, 2021

i30817 commented Apr 10, 2020 •

edited

Loading