Broken redering when use ARB ScummVM's "GRIM Fandango" shaders on X86/Linux and PPC/AmigaOS4

[kas1e]

To reproduce:

-- build scummvm
-- run it
-- choose in "Global Options" : Graphics/GraphicsMode: OpenGL (that at top of options in Graphics menu)
-- choose in "Game 3D Renderer": OpenGL (that option on the bottom in Graphics menu, need to scroll down window a little).
-- add "Grim Fandango" game, and be sure that in local settings for game it have the same OpenGL in graphics mode and "OpenGL" in Game 3D renderer.
-- run game

Now, when game start (you need to hit "esc" to skip intro). We have no rendering happens of the main person. And also, when we hit F1 , then we may or may not have rendering happens, but when we chooise to quite from the menu, then old menu items to not clears too.

Running the same without GL4ES, but with MESA on x86/linux provide correct results.

That how the first screen looks like with GL4ES (screenshot from amigaos4, but it's the same with Mesa on x86/linux with GL4ES now):

https://kas1e.mikendezign.com/aos4/scummvm/grim_bug/01_after_glad_grim_start.jpg

That how the first screen looks like with MESA (screenshot also from amigaos4, but also show the same issue):

https://kas1e.mikendezign.com/aos4/scummvm/grim_bug/01_before_glad_grim_start.jpg

See, main person rendering completely missed.

Now, to fix issue , i just enough to do "export LIBGL_NOARBPROGRAMM=1", which mean , that issue with the proper rendering of the main person is something with ARB shaders.

GRIM game use just 2 simple ARB shaders, and there they are:

https://github.com/scummvm/scummvm/blob/master/engines/grim/gfx_opengl.cpp#L54

Also as i see, when we on GL4ES, we don't have by some reason "FPS" values rendered too, but that probably another issue not related to the ARB shaders.

[kas1e]

And i can't say 100% sure that we have all problems caused by ARB shaders. Yeah, "LIBGL_NOARBPROGRAM=1" do fix rendering of the main person, but, it's like something wrong with whole "depth". For example, i can go over some parts on the screen lilke they on the floor (on mesa all fine).

I will made a video showing issue a little bit later.

It's like we may have for example and issue with ARB shaders and issue with Depth, or, it's all just issue with Depth, but just when we set "LIBGL_NOARBPROGRAM=1", its somehow change the visual output, but still not fix issue fully.

[kas1e]

Ok made a video showing those problems :

https://youtu.be/JGFvtaLFgsg

See right at the begining i run over MESA: FPS shown, main person shown, and you can see that on the first screen i can come close to us, and not overwrite that part of things which are near of us (i.e. person under them, and can't overwrite them by his own rendering).

Next, starting from 0:37, you can see how it when i run over GL4ES without doing anything : no FPS, no main person renders.

Next, starting from 0:50, you can see how i set LIBGL_NOARBPROGRAM and run the GRIM, and so : main person start to renders, but still no FPS, and, as you can see starting from 1:00 person can go over the things near of us, like they on the floor. And i feel whole "depth" broken, because i "stop" much early when i should when i trying to go to the table in the middle of screen. Probably whole Z thing is fucked up, and maybe issue is not with ARB Shaders per se, but with Depth, and it only side effect that we have fixed main person rendering by disabling ARB shaders.

[kas1e]

Just some update on that issue(s).

Lephilousople (one of ScummVms programmers), do some tests and find out, that the ARB program causing problem is the depth one. He disabled specifically the depth ARB and not the dim fragment ARB in the game's code, and hero is renders fine.

Through that didnt fix issue with FPS not being shown, and depth related problem when hero passing through objects still there too.

So maybe we are "lucky" to have 3 issues at once :) But at least 2 of them are "depth" related for sure.

[kas1e]

We go futher, and with ARB depth issue Lephilousople go futher, and found where issues coming from : the main problem is that pixel conversion function doesn't handle depth component at all. He made an some hack-test to make sure we use GL_UNSIGNED_SHORT if user as asked to use GL_UNSIGNED_SHORT (It's useless to upgrade to GL_UNSIGNED_INT (which is useful for DEPTH24 or DEPTH32)) , so his patch for gl4es looks like this:

diff --git a/src/gl/texture.c b/src/gl/texture.c
index 2f598e36..83391cb4 100644
--- a/src/gl/texture.c
+++ b/src/gl/texture.c
@@ -159,7 +159,9 @@ void internal2format_type(GLenum internalformat, GLenum *format, GLenum *type)
             break;
         case GL_DEPTH_COMPONENT:
             *format = GL_DEPTH_COMPONENT;
-            *type = (hardext.depth24)?GL_UNSIGNED_INT:GL_UNSIGNED_SHORT;
+            if (*type != GL_UNSIGNED_SHORT) {
+                *type = (hardext.depth24)?GL_UNSIGNED_INT:GL_UNSIGNED_SHORT;
+            }
             break;
         case GL_DEPTH_STENCIL:
         case GL_DEPTH24_STENCIL8:

Through, he not sure how correct that kind of patch is, but i think that knowing what the problem are, you can easyly add the fix in proper way ?

ps. And maybe second issue with Depth when main hero passing through objects will be something of this kind too ?

[kas1e]

Well, at least on AmigaOS4 this change fix nothing : ARB depth sahder still fail, and when i just do LIBGL_NOARBPROGRAM 1 and main hero renders, hero still the same passing through objects. I will check on my x86/Mesa just in case, but anyway it point out that patch is not good even if it will help on x86/mesa.

Anyway, Lephilousople also think he found something about ARB shader depth issue : he found that in gl4es_DrawPixel we have:

    if ((format == GL_STENCIL_INDEX) || (format == GL_DEPTH_COMPONENT)) {
        errorShim(GL_INVALID_ENUM);

While in GRIM's code we do:

glDrawPixels(w, h, GL_DEPTH_COMPONENT, GL_UNSIGNED_SHORT, data);

Hope that will help too.

[ptitSeb]

Problem is on gles hardware, we cannot do most of the things we can do on full opengl hardware with depth buffer (and stencil buffer).
I remember, back when residual was independant from scummvm, that I needed to hack the opengl code to get correct rendering of the caracters, because the operation done by on the engine on the depth buffer were just not supported on the gles2 hardware of the Pandora.

[kas1e]

Do you mean hack the opengl code of the game or of gl4es ?

[ptitSeb]

the code of the game. but I don't find that hack. Also, the code change seem to have changed a lot anyway. So maybe that is not relelvent anymore. Not sure...

[kas1e]

But isn't Lephilousople finding help any way ? Or he just miss the point and just mess in wrong way with all that "shorts" , etc ?

[ptitSeb]

I don't know. depth is tricky on gles2 hardware. But the glDrawPixel(...) on depth or stencil, well, I don't know how to do that with gles2 hardware :(

[kas1e]

Maybe you can give us advice about how properly rewrote the game's code so it will work everywhere ?

Strange that all through... because scummvm do support OGLES2 directly mean it works for them on OGLES2 hardware then ?

[AbduSharif]

For what it's worth, stencil buffer in OpenMW seems to work well:
https://gitlab.com/OpenMW/openmw/-/merge_requests/1635

An example of a mod with assets that make use of "stencil magic", they work on the Android port (which uses GL4ES):
https://www.nexusmods.com/morrowind/mods/50663

[AbduSharif]

Unfortunately though, the volunteer project for the Android port has stopped development, but you can update the build scripts for it to update to newer development changes (latest master branch commit of OpenMW has the stencil changes), it might be alot of work though.

(You could use this as a reference for the start, then there's more changes:)
uramer/openmw-android#1