rsx: Asynchronous host memory management

[kd-11]

When requesting memory protection operations, enqueue the requests and handle them in batches. Applies to the texture cache only. A common issue is that when locking pages, we end up trampling on our own work (e.g upload texture 1, lock pages, upload texture 2, requires to unlock texture 1 then lock texture 1 and texture 2, etc). This can get really slow.
Instead, we don't actually care about the memory protection during command recording, so freely mark the slots for protection until we have gathered everything and are ready to render. Then apply the protections in bulk.
The result is a significant speedup in sensitive games though this is admittedly a rare scenario. Makes id tech 5 stop stuttering to single digit framerates.

[Darkhost1999]

Master VS PR

Master

RPCS3.log
PR

RPCS3.log

A significant performance increase in Need for Speed Most Wanted Note: I was unsuccessful in uncapping the framerate. The exact performance increase is an unknown value.

[MarioSonic2987]

Water no longer shakes in Saints Row: The Third, but breaks player's character textures.

In master or disabling Asynchronous Memory Manager in this PR fixes player's characters textures, but the water will shake again.

[kd-11]

@MarioSonic2987 Is the breakage in the screenshot? I am quite bad at finding differences in still images unless it's obvious or pointed out.

[A5362]

@MarioSonic2987 Is the breakage in the screenshot? I am quite bad at finding differences in still images unless it's obvious or pointed out.

We can see a glitch on his right hand (black texture)

[Pipi86]

@MarioSonic2987 Is the breakage in the screenshot? I am quite bad at finding differences in still images unless it's obvious or pointed out.

The poor dude in the screenshot got his eyebrows shaved and also his right palm has a pitch black spot.

[MarioSonic2987]

It's more noticeable when the camera is too close to the player. His left arm turns black in this screenshot.

Master:

[papagoghub]

However (Disable Asynchronous Memory Manager: False) this option will cause (Gran Turismo 5) the loading screen to freeze when returning to the main menu after a match or when exiting a replay, and must be changed to True before returning to the main menu.

[digant73]

Applied this PR on top of current rpcs3 release (today) and also on previous releases (3 days ago when this PR was opened) and in both cases I have some freezes in some games on specific sections. For example, attached the log on Resistance 3 when starting section Ambush on chapter ST. Louis, MO. When you skip the first cutscene the game freezes before the gameplay start.

RPCS3.log

[kd-11]

Applied this PR on top of current rpcs3 release (today) and also on previous releases (3 days ago when this PR was opened) and in both cases I have some freezes in some games on specific sections. For example, attached the log on Resistance 3 when starting section Ambush on chapter ST. Louis, MO. When you skip the first cutscene the game freezes before the gameplay start.

RPCS3.log

Any other games affected? I was hoping a smaller game or demo could reproduce. AAAs are hard to debug due to the fact that there is so much "noise" in the data.

[ItsLaguna]

In FIFA Street, Disable Async Memory Manager set to False breaks/doesnt show the appropiate colors for the crowd:

Setting it to True matches the current behaviour in master and the crowd is no longer looking partially blue or red/pink:

[digant73]

Applied this PR on top of current rpcs3 release (today) and also on previous releases (3 days ago when this PR was opened) and in both cases I have some freezes in some games on specific sections. For example, attached the log on Resistance 3 when starting section Ambush on chapter ST. Louis, MO. When you skip the first cutscene the game freezes before the gameplay start.
RPCS3.log

Any other games affected? I was hoping a smaller game or demo could reproduce. AAAs are hard to debug due to the fact that there is so much "noise" in the data.

yes, the freeze was present also on other games (I have to recheck which ones (I do not remember)) but they are AAA games (I do not have small games or demos). The freeze is present also with the new param Disable Asynchronous Memory Manager enabled. In the week end I will try your standalone PR just to be sure I didn't miss to integrate all your changes (by double checking it seems ok)

[digant73]

Applied this PR on top of current rpcs3 release (today) and also on previous releases (3 days ago when this PR was opened) and in both cases I have some freezes in some games on specific sections. For example, attached the log on Resistance 3 when starting section Ambush on chapter ST. Louis, MO. When you skip the first cutscene the game freezes before the gameplay start.
RPCS3.log

Any other games affected? I was hoping a smaller game or demo could reproduce. AAAs are hard to debug due to the fact that there is so much "noise" in the data.

yes, the freeze was present also on other games (I have to recheck which ones (I do not remember)) but they are AAA games (I do not have small games or demos). The freeze is present also with the new param Disable Asynchronous Memory Manager enabled. In the week end I will try your standalone PR just to be sure I didn't miss to integrate all your changes (by double checking it seems ok)

The issue is present also on your standalone PR (downloaded and compiled). I also verified that in Resistance 3 the freeze is due to WCB enabled

[Darkhost1999]

I also verified that in Resistance 3 the freeze is due to WCB enabled

#9133 (comment)

There's some running discussion of write color buffers accelerating freezes in games prone to freezing. I can't double confirm this PR as causation to the freezes because of this running discussion.
I just can't verify this PR creates any new issues.

[digant73]

I also verified that in Resistance 3 the freeze is due to WCB enabled

#9133 (comment)

There's some running discussion of write color buffers accelerating freezes in games prone to freezing. I can't double confirm this PR as causation to the freezes because of this running discussion. I just can't verify this PR creates any new issues.

I do not have any freeze on current master (or previous versions) with WCB (I always had WCB enabled). On my side, the only annoying issue on some Insomniac games (R1 and R3) is the Device Lost that is present with or without WCB.

EDIT: With this PR, the freeze is not random but it always happens when the game's section is loading (just when the screen is switched to black before the gameplay starts.

[digant73]

The freeze on my side seems to be related to the flag invalidation_cause::cause_uses_strict_data_bounds provided at line 2493:

invalidate_range_impl_base(cmd, tex_range, invalidation_cause::cause_is_read | invalidation_cause::cause_uses_strict_data_bounds, {}, std::forward<Args>(extras)...);

in texture_cache.h

If I remove that flag there is no more freeze.

[kd-11]

The freeze on my side seems to be related to the flag invalidation_cause::cause_uses_strict_data_bounds provided at line 2493:

invalidate_range_impl_base(cmd, tex_range, invalidation_cause::cause_is_read | invalidation_cause::cause_uses_strict_data_bounds, {}, std::forward<Args>(extras)...);

in texture_cache.h

If I remove that flag the is no more freeze.

Thanks for confirming. Skipping external bounds can work but requires some extra work to avoid replacement (RO vs NO flags). I've removed the change for now. Hopefully no more hangs.

[kd-11]

@ItsLaguna Retest when it builds as well. I need to make sure no regressions.

[MarioSonic2987]

With Asynchronous Memory Manager, water no longer shakes, but broken player character textures are still there:

Without that option, player character textures look normal, but water shakes again.

[ItsLaguna]

@ItsLaguna Retest when it builds as well. I need to make sure no regressions.

Game crashes when trying to load to main menu if Disable Asynchronous Memory Manager is set to False.
PPU[0x1000018] Thread (RenderThread) [0x0015a7ec]} VM: Access violation writing location 0x283c27 (read-only memory)

RPCS3.log

If it's set to True, the game reaches the main menu and can play just fine.

[digant73]

@ItsLaguna Retest when it builds as well. I need to make sure no regressions.

Game crashes when trying to load to main menu if Disable Asynchronous Memory Manager is set to False. PPU[0x1000018] Thread (RenderThread) [0x0015a7ec]} VM: Access violation writing location 0x283c27 (read-only memory)

RPCS3.log

If it's set to True, the game reaches the main menu and can play just fine.

@MarioSonic2987, @ItsLaguna
Eventually, try to clone again the project. With the git pull, the rollback on texture_cache.h was not present on my side while it was present cloning again the project

[ItsLaguna]

@MarioSonic2987, @ItsLaguna Eventually, try to clone again the project. With the git pull, the rollback on texture_cache.h was not present on my side while it was present cloning again the project

@digant73 Just saw this but I wasn´t cloning the project, I'm grabbing the artifacts produced from Azure.

I've been trying to get everything set up to build it myself but I'm struggling to do it, it would be faster if someone could provide an appimage with the rolledback file. Sorry.

[digant73]

@MarioSonic2987, @ItsLaguna Eventually, try to clone again the project. With the git pull, the rollback on texture_cache.h was not present on my side while it was present cloning again the project

@digant73 Just saw this but I wasn´t cloning the project, I'm grabbing the artifacts produced from Azure.

I've been trying to get everything set up to build it myself but I'm struggling to do it, it would be faster if someone could provide an appimage with the rolledback file. Sorry.

understood. Unfortunately, I can provide the Windows version only

[ItsLaguna]

@digant73 Just saw this but I wasn´t cloning the project, I'm grabbing the artifacts produced from Azure.
I've been trying to get everything set up to build it myself but I'm struggling to do it, it would be faster if someone could provide an appimage with the rolledback file. Sorry.

understood. Unfortunately, I can provide the Windows version only

Managed to build it but it crashes after opening any game... I tried 😅
I can boot into Windows and report back if you provide it.

[kd-11]

I'll try and make time for this over the weekend to address remaining issues.

[kd-11]

Time for some retests. If visuals are glitchy still I'll just invert the option to default to the old way and have users optionally enable this for the speedboost.

[MarioSonic2987]

Water no longer shakes and player character textures look fine:

Edit: Without Asynchronous Memory Manager, water still shakes.

[ItsLaguna]

FIFA Street now boots fine again and crowd colors aren't bugged anymore.

[MarioSonic2987]

Slowdowns that occur while Roxas attacks are greatly improved in this PR:

[Linear524]

Looks like everything works fine :)

[Apopoy]

This update has issue with WWE2K14 even I on "Write color buffers" the referee is still glowing red can you fix it? Thank you

[kd-11]

Open a proper bug report and link to this PR. This discussion is completed the moment the changes are committed.

[Asinin3]

Weird how this issue was originally in 2018: #4876 (comment) then broke again in 2020: #7338

Then came back and was fixed again here.