[Regression] Mercury Meltdown - black screen

[bslenul]

Game or games this happens in

Mercury Meltdown (USA) (En,Fr,Es) (v1.01) / ULUS10133 (said to be happening in the EU version ULES00466 as well, but I haven't tried)

What area of the game / PPSSPP

Currently it only shows a black screen when starting the game, no sound either. Tried GL/Vulkan/D3D11, no change. Happening on my Linux Mint VM as well, although a Linux user told me it was loading fine for him 🤔

What should happen

Game should start.

Logs

ppsspplog.zip

Platform

Windows 10 / Linux Mint

Mobile phone model or graphics card

GTX 970

PPSSPP version affected

Already bisected, started with 52c5f4b and still happening in current commit (63966cb).

Last working version

d066b39 (commit just before the one mentioned above).

Graphics backend (3D API)

Direct3D 11 / Vulkan / OpenGL

Checklist

• Test in the latest git build in case it's already fixed.
• Search for other reports of the same issue.
• Try resetting settings or older versions and include if the issue is related.
• Try without any cheats and without loading any save states.
• Include logs or screenshots of issue.

[ghost]

Not working since v1.9.4 according to reports
https://report.ppsspp.org/logs/game/ULES00466_1.01

[Sanaki]

I'm the one who confirmed it working on Linux. I do self-build both standalone and libretro, but it's working fine on both for me, both US and EU.

EDIT: Okay, bit of testing done. I checked the prebuilt core, still works fine. I moved that exact same core over to the steam version of RetroArch, which uses the soldier runtime and fixed library versions, and I experience the black screen there. That implies it's working on my system as a result of some library version I have. Since I'm on Mint 19.3 (Ubuntu 18.04), it's likely an outdated library that the game works on. (See below, this part was user error)

[NatParable]

I was the one who happened upon this issue in the first place and asked for help on the RetroAchievements Discord, which Sanaki was very kind to provide (hi!).

I'm on Windows 10 using the European version of Mercury Meltdown that I ripped myself (MD5: 7da9fb71400dd907dbc83aaf3cedf8ff), and have experienced this issue both in the latest standalone and the latest retroarch core.

The game ran perfectly fine in an older build of PPSSPP standalone before I updated to the latest version, though unfortunately I have no record of which older version that was.

[hrydgard]

Maybe it's affected by some setting, like I/O timing mode, or similar...

[ghost]

I cannot reproduce this issue in my phone redmi note 9.

Screenrecorder-2021-10-30-05-57-11-719.mp4

[Sanaki]

We tested identical settings via the libretro core. Standalone wasn't tested in the same fashion, but aligned with the core's behavior in every case.

[NatParable]

I cannot reproduce this issue in my phone redmi note 9.
Screenrecorder-2021-10-30-05-57-11-719.mp4

That's an android phone right? Since android is a modified version of linux, i'm not surprised that it works on there. It seems to be specifically a problem with Windows.

[Sanaki]

Android is a very different platform. The relation to linux is flimsy at best. Still good to know that it's unaffected though.

Looks like I see this log line only when it fails:
[libretro ERROR] [SCEUTIL] Savedata version with missing key on save: 2

[Sanaki]

Quick update... seems in the quick setup of the steam version I failed to set up my save location (thus the PSP directory) correctly. It works fine for me via steam as well. That said, the same is not true of the Windows version, where it fails. Apologies for the confusion that may have caused.

The line listed above -does- show on the Windows version, but does not appear when the game successfully boots.

[bslenul]

Funny, for me the problem exists in the Android version as well.

• With e128888 (latest working version from the buildbot, idk how to build for Android), the game boots:

20211030_002133.mp4

• With a9d7948 (current commit when typing this), black screen:

20211030_002300.mp4

My phone is pretty old tho (Samsung Galaxy A5 2016) and still runs under Android 7.

[Panderner]

I cannot reproduce this issue in my phone redmi note 9.

Screenrecorder-2021-10-30-05-57-11-719.mp4

@Gamemulatorer how this game is working in latest build? I can reproduce this issue for me.

[ghost]

It's weird my ancient phone android 5 with mali-450 gpu can reproduce this.

video.recording.online-video-cutter.com.mp4

[ghost]

I cannot reproduce this issue in my phone redmi note 9.
Screenrecorder-2021-10-30-05-57-11-719.mp4

@Gamemulatorer how this game is working in latest build? I can reproduce this issue for me.

I don't know why it's working properly this game in my Redmi Note 9 😂
maybe because of enhanced edition of MIUI?
😆

[Sanaki]

I can confirm the game works fine on my Razer Phone 2 running Android 9 on PPSSPP v1.12.3-102-ga9d7948f7 (a9d7948), grabbed from the orphis buildbot. Tested via vulkan.

That said, I just went back and linked my Linux PSP directory into my Windows libretro save directory and the game now launches. This seems to be caused by something missing from the PSP directory, though I'm not yet sure what. I'll try to narrow it down.

EDIT: The game boots when PSP/SAVEDATA/PPCD00001DLS001/DATA2.BIN is present. The PARAM.SFO has "Online system info" near the end, so presumably this may be something only people who dumped from their real PSP would have. Some searching seems to indicate it's a console ID or related to encryption somehow.

[Panderner]

@Sanaki where the PSP/SAVEDATA/PPCD00001DLS001/DATA2.BIN is created?

[ghost]

Yes the game boots if I use save data in my ancient phone.
Here's the save data link https://gamefaqs.gamespot.com/psp/931944-mercury-meltdown/saves

This is just like #14757

[Sanaki]

@Sanaki where the PSP/SAVEDATA/PPCD00001DLS001/DATA2.BIN is created?

Not entirely sure. Someone claimed starting up Wipeout Pure may generate the data in question, but I'm uncertain if that occurs when emulated, or only on real hardware. Given that I'm uncertain what all is contained in that file at the moment, I'm wary of sharing my own publicly. I found some limited info on it here:
http://lukasz.dk/mirror/forums.ps2dev.org/viewtopicad04.html?t=1795

If contributors need access to that file to attempt to develop a fix, please contact me privately. My keybase with contact info is linked in my github profile. I assume some of you will have it already though, if it's generated on real hardware.

@Gamemulatorer It doesn't seem at all similar to the issue you linked. The saves you linked also don't include the data I mentioned.

[unknownbrackets]

I think PPCD00001DLS001 is related to online/netplay and should be generated by savedata APIs. It's possible this issue only happens when that savedata doesn't yet exist, and happens when creating that savedata.

Does this only happen when netplay / wifi switch options are enabled?

-[Unknown]

[Sanaki]

Just verified reaching the data load screen on Wipeout Pure will generate the necessary data. The wifi setting does not affect the issue in this game.

[anr2me]

According to this old post https://forums.ps2dev.org/viewtopic.php?p=13291&sid=aef41c9a5ada05adce01b7b4ee322a5a#p13291
That PPCD00001DLS001 directory seems to be related to online update

[bslenul]

Just verified reaching the data load screen on Wipeout Pure will generate the necessary data. The wifi setting does not affect the issue in this game.

Can confirm, after booting Wipeout Pure at least once, the game now boots properly. Which is super weird because while I was bisecting the issue, I never had that "PPCD00001DLS001" folder.

[hrydgard]

Huh! Now that is quite surprising. Even more so that 52c5f4b would affect how the game reacts to the presence or not of that file.

Can we "manually" create a file like that which the game will accept? What are the contents?

[hrydgard]

Although that might not be the right question to ask, since the game obviously works on real hardware... Why isn't it creating the file itself, or why does it suddenly need it? Quite odd.

[bslenul]

I tried d066b39 again out of curiosity (last working commit), it looks like even when it was working without the "PPCD00001DLS001" folder it was still checking for that "DATA2.BIN" file:

sceeDLSThrea D[SCEIO]: HLE\sceIo.cpp:937 sceIoGetstat(ms0:/PSP/SAVEDATA/PPCD00001DLS001/DATA2.BIN, 09f72dc0) : FILE NOT FOUND

but it didn't prevent the game from booting.

There's also that line mentioned in previous replies, again only appearing when the "PPCD00001DLS001" folder is missing:

sceeDLSThrea E[SCEUTIL]: Dialog\SavedataParam.cpp:403 Savedata version with missing key on save: 2

not sure what that means but ingame save works perfectly fine, and again it doesn't prevent the game from booting with that commit.

[anr2me]

I tested this game (EUR version) on JPCSP and it also shows black screen the first time i boot this game (the log shows an infinite of repeated sceUtilitySavedataGetStatus returning 0x0)
However, i can see that it's creating the "PPCD00001DLS001" folder, thus the 2nd time i boot this game it was working fine.

Update: when i tested with JPCSP LLE(prx files) the game was able to creates the "PPCD00001DLS001" folder and boot without getting infinite black screen (i only see one sceUtilitySavedataGetStatus returning 0x0)

[unknownbrackets]

That's always welcome, but this issue isn't about that. You can check on Discord here https://discord.gg/5NJB6dD, or read the instructions here: https://github.com/hrydgard/ppsspp/tree/master/assets/lang#readme

-[Unknown]

[Sanaki]

Links were provided. Please use them. Conversation on this topic does not belong in this issue.

[anr2me]

Removing this error return will at least allows the game to create some files inside "PPCD00001DLS001" folder, but DATA2.BIN file size will be 0 bytes ( due to missing key? the data size provided on SavedataInitStart's param seems to be 0)

ppsspp/Core/Dialog/SavedataParam.cpp

Lines 407 to 408 in a9d7948

	ERROR_LOG_REPORT(SCEUTILITY, "Savedata version with missing key on save: %d", param->secureVersion);
	return SCE_UTILITY_SAVEDATA_ERROR_SAVE_PARAM;

However, the next time the game booted it won't get stuck in black screen anymore, and DATA2.BIN will have some data in it (276 bytes), which is similar behavior to JPCSP HLE

First boot:

Second boot:

[sum2012]

@unknownbrackets
It is real PSP log:
https://gist.github.com/sum2012/abb4390ffe6e95f45d3deeb7e3032962

[sum2012]

ppsspp v1.13.1-583-g1653dcdc1 bad log
https://gist.github.com/sum2012/1855a30cb5b45c8214a9f62da24adb20

[anr2me]

@sum2012 Could you test on real PSP to find out whether the PPCD00001DLS001 supposed to be created at boot time or not.

1. Delete or rename folder PSP/SAVEDATA/PPCD00001DLS001
2. Run the game until it shows the first video and then exit/quit the game (currently PPSSPP creates this folder right before the first video appeared, while JPCSP LLE didn't create it)
3. Check the PPCD00001DLS001 folder again whether it's existed or not, also check the files in it whether they have 0 bytes file size or not when they're created for the first time.

[sum2012]

@anr2me PPCD00001DLS001 supposed to be created at boot time
they have no 0 bytes file size when they're created for the first time.
PPCD00001DLS001.zip

[anr2me]

This is the content of the PPCD00001DLS001 created by PPSSPP for the first time/first boot and stuck with black screen, the DATA2.BIN size is 0 bytes, on the 2nd boot the file will be overwritten with the correct size/data and no longer stuck
PPCD00001DLS001.zip
PPCD00001DLS001-2ndBoot.zip

[sum2012]

There are 2 problem:
1: How to get correct DATA2.BIN of data in the first run?
2: Force firmware < 2.71 sceUtilitySavedataGetStatus return 4

[anr2me]

This is probably just timing issue, after commit 52c5f4b got merged the shutdown timing might slightly changed and the game didn't get the chance to get status = 4 anymore (like the last working version have), and goes directly to 0 after shutdown, thus showing a different behavior.

In order for the game to be able to create DATA2.BIN with the correct size and no longer stuck on the first boot, there are 2 ways:

1. By adding a minor delay (ChangeStatus(SCE_UTILITY_STATUS_NONE, 13); is good enough to give a chance for the game to get status = 4) as mentioned in [Regression] Mercury Meltdown - black screen #15068 (comment)
2. Or without changing the timing, but by making sure it didn't skip status = 4 like this:
   

The logs will be like this (similar to the last working version), where the highlighted line makes a difference to the game behavior

[sum2012]

I have reported to jpcsp.
gid15 fix it in jpcsp/jpcsp@ddf9061

@hrydgard @unknownbrackets Can we change to jpcsp 's timing ?

[sum2012]

Part of timing change
sum2012@e686050
Another part jpcsp use hleKernelExitDeleteThread()
https://github.com/jpcsp/jpcsp/blob/ddf9061d847d0bd63b987f3eac4a32578a670f55/src/jpcsp/HLE/modules/sceUtility.java#L613 for SCE_UTILITY_STATUS_NONE.
I cannot translate it,

[hrydgard]

I'm gonna wait with doing that change until after 1.14 is released (due to fear of other regressions), so tagging for 1.15.

[unknownbrackets]

I don't think that change makes sense to be clear, I haven't really messed with this but the finish shutdown should only be called after the delay (after it was already 4.) That happens in the MIPS code generated by UtilityDialogShutdown(). You can even see that it makes sure status has already been SCE_UTILITY_STATUS_SHUTDOWN before setting it immediately to SCE_UTILITY_STATUS_NONE.

Pretty sure this matches PSP behavior. But maybe there's something more subtle about the specific thread priority of the thread checking the status vs the accessThread priority it defined. We should already be using 2000us on errors, I think.

-[Unknown]

[anr2me]

It makes a different if this games relies on the status value before it can go to the correct route, although most games doesn't care about it.

Before PR 52c5f4b the logs are similar to the JPCSPTrace on PSP https://gist.github.com/sum2012/abb4390ffe6e95f45d3deeb7e3032962 where the game is able to retrieve status = 4, while after that PR the game's thread didn't had the chance to retrieve status = 4 because ChangeStatus(SCE_UTILITY_STATUS_NONE, 0); immediately overrides the status, thus resulting a different status sequences during GetStatus in the logs compared to a real PSP's logs.

Adding 13 usec delay was enough to give the game's thread a chance to retrieve status = 4, most-likely giving a chance for the thread to switch before the status changed to 0

PS: as i remembered the 4 status can gets overridden immediately with 0 by the fadeout code without giving the game a chance to retrieves the value first, so adding as small as 13 usec delay to ChangeStatus will let the thread context to switch and the game will have a chance to retrieve the status.
I kinda forgot what exactly happened, but i think it's between the status from the result and the status from fadeout can immediately overriding each other.

[sum2012]

@anr2me Simulate UMD delays work !!!

[anr2me]

You mean the one with slow reading, right? because the old Simulate UMD delays doesn't fix this issue as i remembered

[sum2012]

@anr2me sorry, I cannot solve again after re-test

[sum2012]

From full debug log from status 3 ,I cannot see it read any file.

49:18:747 sceeDLSThrea D[SCEUTIL]: HLE\sceUtility.cpp:461 3=sceUtilitySavedataGetStatus()
49:18:747 sceeDLSThrea D[HLE]: HLE\HLE.cpp:694 Compiling syscall to sceUtilitySavedataShutdownStart
49:18:747 sceeDLSThrea D[SCEUTIL]: HLE\sceUtility.cpp:447 00000000=sceUtilitySavedataShutdownStart()
49:18:747 sceeDLSThrea D[SCEKERNEL]: HLE\sceKernelThread.cpp:2521 0=sceKernelDelayThread(000003e8): delaying 1010 usecs
49:18:747 ScePafJob D[SCEKERNEL]: HLE\sceKernelThread.cpp:3087 Context switch: sceeDLSThread -> ScePafJob (287->289, pc: 08878584->08220e00, thread delayed) +157us
49:18:747 ScePafJob D[HLE]: HLE\HLE.cpp:694 Compiling syscall to __UtilityWorkUs
49:18:747 ScePafJob D[HLE]: HLE\HLE.cpp:694 Compiling syscall to __UtilityWorkUs
49:18:747 ScePafJob D[HLE]: HLE\HLE.cpp:694 Compiling syscall to __UtilityWorkUs
49:18:747 ScePafJob D[HLE]: HLE\HLE.cpp:694 Compiling syscall to __UtilityWorkUs
49:18:747 ScePafJob D[HLE]: HLE\HLE.cpp:694 Compiling syscall to __UtilityFinishDialog
49:18:747 ScePafJob D[SCEUTIL]: HLE\sceUtility.cpp:414 0=__UtilityFinishDialog(1)
49:18:747 ScePafJob D[HLE]: HLE\HLE.cpp:694 Compiling syscall to __KernelReturnFromThread
49:18:747 ScePafJob D[SCEKERNEL]: HLE\sceKernelThread.cpp:2134 __KernelReturnFromThread: 0
49:18:747 sceeDLSThrea D[SCEKERNEL]: HLE\sceKernelThread.cpp:3087 Context switch: ScePafJob -> sceeDLSThread (289->287, pc: 083fddc0->08878584, thread returned) +2012us
49:18:747 sceeDLSThrea D[SCEKERNEL]: HLE\sceKernelThread.cpp:442 Freeing thread stack ScePafJob
49:18:747 sceeDLSThrea D[SCEKERNEL]: Util\BlockAllocator.cpp:229 Merging Blocks
49:18:747 sceeDLSThrea D[SCEKERNEL]: Util\BlockAllocator.cpp:234 Block Alloc found adjacent free blocks - merging
49:18:747 sceeDLSThrea D[SCEKERNEL]: Util\BlockAllocator.cpp:229 Merging Blocks
49:18:747 sceeDLSThrea D[SCEKERNEL]: Util\BlockAllocator.cpp:254 Block Alloc found adjacent free blocks - merging
49:18:747 sceeDLSThrea D[SCEUTIL]: HLE\sceUtility.cpp:461 0=sceUtilitySavedataGetStatus()