Vulkan:Multiple problems after #4908

[vsub]

I am using RPCS3 0.0.5-7657-6c10cb2e and this happens only while using Vulkan

Asuras Wraith
Ar noSurge
Atelier Escha & Logy
Atelier Shalie
The music keeps playing but the game pauses(Jump=>minimize=>you will stay there until you restore the window...controller inputs are ignored)

Persona 5
The emulation pauses and you can stat it again but only the audio start playing and nothing else is reacting
F {RSX [0x000711c]} RSX: class std::runtime_error thrown: Verification failed (e=0x7f):
(in file c:\projects\rpcs3\rpcs3\emu\rsx\vk\vkoverlays.h:229)

I can see how the vram usage goes up fast higher and higher until it reaches 3GB and then,the games below

Project Diva F
F {RSX [0x01e6e28]} RSX: class std::runtime_error thrown: Assertion Failed! Vulkan API call failed with unrecoverable error: Out of video memory (VRAM) (VK_ERROR_OUT_OF_DEVICE_MEMORY)
(in file c:\projects\rpcs3\rpcs3\emu\rsx\vk\vkhelpers.h:253)

Tales of Graces F
F {RSX [0x093ab68]} RSX: class std::runtime_error thrown: Assertion Failed! Vulkan API call failed with unrecoverable error: Out of video memory (VRAM) (VK_ERROR_OUT_OF_DEVICE_MEMORY)
(in file c:\projects\rpcs3\rpcs3\emu\rsx\vk\vkhelpers.h:253)

Tales of Vesperia
F {RSX [0x0085b24]} RSX: class std::runtime_error thrown: Assertion Failed! Vulkan API call failed with unrecoverable error: Out of video memory (VRAM) (VK_ERROR_OUT_OF_DEVICE_MEMORY)
(in file c:\projects\rpcs3\rpcs3\emu\rsx\vk\vkhelpers.h:253)

Unaffected games
BLEACH: Soul Resurreccion
Dragon's Crown
Enchanted Arms
ETERNAL SONATA
Hard Corps: Uprising
Ar Tonelico Qoga

Win10,i7-6700HQ,GTX1060 3GB and it doesn't matter what settings you are using other than using vulkan

[reaper185]

I think is a problem with intel already stated in #5424. Could be wrong.

[vsub]

Nope,I just found the one responsible #4908
I only test Project Diva F but
0.0.5-7594 - works fine
0.0.5-7596 - eats all vram and then it crashes

Edit:I just tested all other affected games...no crashes and the emulation don't pauses while minimized(gamepad inputs are accepted while minimized)

[psennermann]

I have experienced the same problem at least with the game Catherine

Win10, i5 7400, gtx1060 3gb

[kd-11]

Try with #5509 but I do not expect this to do anything. While minimized, nvidia cards do not run through the whole present loop which can possibly leak. This is because nvidia prohibits rendering whilst the main window is minimized (reports max dimensions of 0x0 which is impossible)

[vsub]

Sadly no change,all games are acting the same way
Also why some games work fine and other don't.

[kd-11]

While rendering is effectively paused, its does not stop completely. Operations like blit engine and framebuffer readbacks will still invoke compute operations which will consume resources and since there is no frame scope to work with, this can balloon. Its a tough situation here - we cannot stop the operations from happening as graphics will be corrupt when you restore the window and we cannot easily free/reset resources. I'll have to think of a workaround for this at a later time.

[vsub]

How about auto pausing the emulation(this is actually a nice feature to auto pause on minimize)as a workaround until a proper fix is done.
The way it is now on an affected game,accidental minimize(or minimize all to show the desktop)is instant game over

Or revert #4908

[vsub]

Well it seems that the 0x0 size it's not the problem
When I use the latest beta and minimize the window,rpcs3 freezes and I get this in Process Hacker for the game window

And when I try 0.0.5-7594,I also get the exact same results but rpcs3 don't freeze

[kd-11]

https://ci.appveyor.com/project/kd-11/rpcs3/builds/21420755/artifacts

[kd-11]

I think the PR is a red herring; the real culprit is likely FIFO rewrite. When emulator is paused, the main thread stalls because messages are not being processed (a lost commit is responsible for this part). Before the rewrite in the Qt related PR, Qt seems to have just returned some cached values of width/height and did not go through win32. GetClientRect however explicitly goes through win32 and the stalled msq thread suddenly becomes an issue. The build posted above does not stop the msq thread to avoid this and other similar issues.

[vsub]

Sadly no change,all of the mentioned games are acting the same way

[vsub]

Any update on this?
It still happens on 0.0.5-7758

[kd-11]

Retest

[vsub]

I didn't do much testing(just starting the game and then minimize it)but all games except Persona 5 are acting the same way.
Only Persona 5 seems to be fine now

[psennermann]

For me the problem is still there (0.0.6.7786)

[uaqlover]

problem is still there (0.0.6.7786)

[vsub]

I am not sure how helpful this will be but,if I disable the game window minimize button with Autohotkey

Loop
{
WinWaitActive,FPS ahk_exe rpcs3.exe
WinSet,Style,-0x20000,FPS ahk_exe rpcs3.exe
WinWaitClose,FPS ahk_exe rpcs3.exe
}

and try to minimize,then the problem is not triggered.
You don't really have many "normal" other ways do minimize the window except Win+D or "Show Desktop" button on the bottom right.
Those two don't apply WM_MINIMIZE to the window(if the button is disabled)so the problem is not triggered but if you actually send the minimize command to the window(from the task manager for example)even if the button is disabled,the WM_MINIMIZE style will be applied and rpcs3 will freeze

[vsub]

Still no change on the latest beta(0.0.6-8034)with the latest video drivers(430.39)
Minimizing the game window eats all of the vram and then rpcs3 freezes

[vsub]

Fixed since #6069