Framebuffer Emulation is broken on Raspberry Pi (OpenGL Driver)

[AlessandroPorcelli91]

The game starts beautifully and runs great, though in about half an hour the textures corrupt in a very specific way: wherever the objects are (eg: Link, a tree, a NPC etc.) the terrain becomes transparent, and the transparency stays for as long as the emulator is running effectively making the game unplayable. I am currently running configuration version 12 according to mupen64plus.cfg. What can I do? Thanks.

[loganmc10]

Do you have Framebuffer Emulation enabled?

[AlessandroPorcelli91]

Thanks for responding. Yes I do, is that broken in the current build?

[loganmc10]

If you look inside https://github.com/gonetz/GLideN64/blob/master/src/FrameBuffer.cpp you'll see 3 instances of glDiscardFramebufferEXT(GL_FRAMEBUFFER, 1, discards); wrapped inside #ifdef VC (VC means Raspberry Pi)

Basically there is some kind of bug with the Raspberry Pi GL driver, and I've done my best to minimize the issue (everything was transparent all the time without those changes), but it still seems to pop up every now and then.

I haven't been able to figure out a better way to fix the issue. If you disable Framebuffer Emulation you won't run into this issue (but some things won't render properly, like the pause menu in Zelda: OOT)

[loganmc10]

Alternatively, you can just save your state when it happens, exit the game and reload it until the next time it happens...

[AlessandroPorcelli91]

I see, I guess the problem must be somewhere else...That is rather unfortunate, I liked the FBEmulation and this plugin runs smoother than the others too, in my experience. Adding more instances would be too cumbersome, wouldn it? :D

I could test whether gles2rice suffers from the same problem with FBEmu enabled...Do you happen to know which option enables it for OOT 1.0 (U)?

[loganmc10]

I only use this plugin, so I'm not sure about options for gles2rice. Even if you could get it to work in gles2rice it wouldn't bring us any closer to fixing it in this plugin.

[AlessandroPorcelli91]

I have no experience in coding, I just thought that if other plugins emulate framebuffer in a completely different way and the texture problem still does not vanish for them, then perhaps we would know that it's unfixable, and that is indeed something the driver is handling poorly. Anyway, thanks a lot for replying quickly :)

[AlessandroPorcelli91]

I'd like to check if further iterations of glDiscardFramebufferEXT(GL_FRAMEBUFFER, 1, discards); considerably lower the probability of the glitch happening. What are you using to compile GlideN64?

[loganmc10]

I follow the instructions here:
https://github.com/gonetz/GLideN64/tree/master/projects/cmake

For me it basically boils down to this:

cd src
./getRevision.sh
cd ../projects/cmake
cmake -DMUPENPLUSAPI=On -DNOHQ=On -DVEC4_OPT=On -DNEON_OPT=On ../../src/
make -j4
sudo cp plugin/release/mupen64plus-video-GLideN64.so /usr/local/lib/mupen64plus

[AlessandroPorcelli91]

Thanks a lot, I shall try to play through the game with a couple more iterations in the code and report if that's significantly better. I am aware this wouldn't be a definitive solution but if the average incidence becomes something like one failure every two hours that could be enough for a long session.

[AlessandroPorcelli91]

Sorry for double posting again but I feel like a dumb clueless person: in the three instances you clearly perform different operations, so I suppose that just adding another

#ifdef VC
const GLenum discards[] = {GL_DEPTH_ATTACHMENT};
glDiscardFramebufferEXT(GL_FRAMEBUFFER, 1, discards);
#endif

wouldn't actually accomplish anything, or would it? Sorry if I'm wasting your time.

[loganmc10]

No I don't think it would make much difference. glDiscardFramebufferEXT is a "hint" to the driver to tell it that it doesn't need to preserve the contents of GL_DEPTH_ATTACHMENT (the depth information) after the Framebuffer Object is unbound. It's unclear why this fixes the depth/transparency issues on the Raspberry Pi, but providing that hint twice in a row in the same place probably won't make much of a difference.

It's a hard thing to test because it only crops up after 30 minutes or so like you say. So to test it is basically: Take a shot in the dark as to where to put that hint, play for 30+ minutes to see if it fixes it, repeat....

[loganmc10]

By the way it's not a waste of my time, I enjoy working on this kind of thing. I probably spend more time trying to improve N64 emulation on my devices than I do actually playing N64 games lol

[AlessandroPorcelli91]

LOL, I know right? I have finished LOZ:OOT enough times in my life, I mostly hunt emulation bugs now. Except I can't actually fix them... Would it be useful if I found some locations in which the issue is more likely to happen than others? For example I could stand still and let it run for a while to see if it triggers anyway, or whatever.

[loganmc10]

If you could fine a situation where it was reproducible, and provide a save state right before it happens, yes that would be extremely useful. It would need to be a situation where I wouldn't have to wait long for it to happen, I can't wait for 30 minutes each time I try something different in the code.

[loganmc10]

Also for a shameless plug:

You can try https://github.com/loganmc10/GLupeN64

That is a libretro port that I am currently working on that uses the latest GLideN64 code.

Building instructions are here:
https://github.com/loganmc10/GLupeN64/blob/master/BUILDING.md

You could see if that gives you any better results than standalone mupen64plus

[gizmo98]

It would be interesting to see if this problem only occurs on a rpi3. Even without OC the SOC gets really hot after a while. After rpi3 was released i suffered temperature related hangs and crashs. The current firmware seems to be a little bit better (throttling?). If i have some time i will do a test with a rpi2.

[AlessandroPorcelli91]

I can give it a try later, I have to compile a custom version of libretro mupen64plus core because libretro does not allow to change the AnalogPeak value, and my controller is so sad it needs the default 32768 value to be lowered. I shall give any feedback ASAP.

@gizmo98 Though I have not installed an active cooling solution, according to vcgencmd the temperature was about 65 degrees Celsius after the glitch occured.

[AlessandroPorcelli91]

Ok, I played with a huge and noisy fan blowing on the RPi 3 keeping it at about 47 °C, and the issue happened anyway.

I am also trying to find some kind of pattern here, I have experienced three failures at very different times (yes, I am using a chronometer XD). One after barely 30 seconds of getting into the game, at Dodongo Cavern entrance, without moving.

Another one happened, again, in Dodongo Cavern after just two minutes: After standing still for the first minute I threw a bomb flower, whose explosion seemingly triggered the glitch.

The third instance was in Kakariko Village after 18 minutes of gameplay, in a spot that also triggered the glitch before.

If I have to take a wild guess, this seems to be correlated with the loading/unloading of some object in a map (the exploding bomb flower one was fishy), but I can't seem to consistently reproduce it. I had a savestate literally 5 seconds before the corruption happened but loading it back after restarting does not do anything, so I am afraid savestates won't help. Since the times ranged from less than 30 seconds to more than 50 minutes this can't even be correlated with lack of memory of some kind. Perhaps I have found something else which may or may not be correlated to this, I shall p
The Glitch Gremlin.zip
ost the screenshots here as soon as I can...A cutscene has been running on my Pi for the last hour (0.3 FPS) with some graphical madness going on.

UPDATE: I have uploaded a zip file with some pictures. The first four happened while running around, the huge chunk of creepypasta madness in the middle is what happened during the cutscene. All the textures are swapped around with the models (except link's, strangely). The last pictures are what was left after the cutscene, which took over an hour to finish due to the incredible slowdown. Notice that textures from the HUD are duplicated within the transparent frames. The persistence of the frames from the cutscene make me think that is the same issue. I have also included a savestate taken during the cutscene, but chances are nothing strange happens loading it.

[loganmc10]

@AlessandroPorcelli91 when you get a chance you should try out the latest code from GLupeN64 (https://github.com/loganmc10/GLupeN64)

I just added a commit that does a glDiscardFramebufferEXT before every glBindFramebuffer. Its supposed to be good for performance (https://community.arm.com/groups/arm-mali-graphics/blog/2014/04/28/mali-graphics-performance-2-how-to-correctly-handle-framebuffers) and may solve this problem as well.

[AlessandroPorcelli91]

Awesome, I shall try it ASAP and let you know :)

[AlessandroPorcelli91]

I have one question first though, due to GLupeN64 being based on libretro, access to mupen64plus.cfg and InputAutoCfg.ini is disabled, correct? If that's the case I'll have to change a couple values in emulate_game_controller_via_libretro.c in order to make it work with my half assed controller.

[loganmc10]

I guess, controller configuration is controlled though RetroArch, so yes there is no mupen64plus.cfg or anything

[AlessandroPorcelli91]

@loganmc10 Alright, I managed to get GLupeN64 up and running and I'm now going to run around for a while to check wether the bug is still there, in the meanwhile though I'd like to report that I'm not able to play at anything higher than 320x240, because the framerate drops dramatically if I change the GLupeN64 resolution option in the retroarch emulators configuration file. Even SM64 becomes pretty much unplayable, why could this happen? Is there some advanced trick enabled by default? By comparison, I get a similar performance at 800x600 in vanilla. Also, the pause screen takes quite a bit to come up, but I guess that's a drawback for the FBEmulation fix.

[loganmc10]

@AlessandroPorcelli91 thanks for testing it out. I don't think the performance difference is because of this potential fix for Framebuffer Emulation. glDiscardFramebufferEXT should never hurt performance, it can only help it.

RetroArch handles OpenGL frames differently than mupen64plus, so behind the scenes things are working a bit differently. To be honest I haven't really tested it higher than 320x240, although when I test higher resolutions on my phone with GLupeN64 it seems to work fine (this is with Framebuffer Emulation disabled however, it doesn't work on Android right now).

I'll test higher resolutions on my Raspberry Pi over the weekend to see if there are any improvements I could make.

[AlessandroPorcelli91]

@loganmc10 I should have tried fiddling around with the configuration files a bit more indeed, I shall compare the default settings with the ones I am running in vanilla for the plugin, though some are not accessible in libretro. I have been running the game for about one hour and nothing bad happened, therefore I am rather confident whatever you did most certainly fix the problem. Would it be messy to implement them in the main branch?

[loganmc10]

It would be a little tricky to implement directly in GLideN64 for technical reasons. glDiscardFramebufferEXT needs to be executed right before glBindFramebuffer. However, glDiscardFramebufferEXT cannot be used when the default framebuffer is bound (framebuffer "0"). The problem is that you don't really know what framebuffer is currently bound if you call glDiscardFramebufferEXT right before glBindFramebuffer.

RetroArch doesn't allow you to bind to framebuffer 0, and GLupeN64 intercepts the GL calls to make sure it doesn't happen, for instance, when GLideN64 does "glBindFramebuffer", this is the code that is actually executed in GLupeN64:
https://github.com/loganmc10/GLupeN64/blob/master/custom/glsm/glsm.c#L1580-L1596

Anyway, because in RetroArch/libretro I can be confident that framebuffer "0" is never bound, it is safe to always run glDiscardFramebufferEXT before glBindFramebuffer, that isn't the case with mupen64plus.

[AlessandroPorcelli91]

@loganmc10 Ok...I might say something stupid now, but keep in mind my knowledge about coding is extremely limited: according to the article on arm.com you posted this morning, Framebuffer "0" gets bounded for every even numbered call of glBindFramebuffer. If this is also the case for the actual application we need, couldn't we keep count of the number of calls and branch the code?

[loganmc10]

@AlessandroPorcelli91 That was just how their example did it. If you look inside
https://github.com/gonetz/GLideN64/blob/master/src/FrameBuffer.cpp
https://github.com/gonetz/GLideN64/blob/master/src/OpenGL.cpp

You'll see numerous instances of glBindFramebuffer all over the code, sometimes it binds to "0" and sometimes to an object.

There are even some instances like this:
https://github.com/gonetz/GLideN64/blob/master/src/OpenGL.cpp#L605
Where it decides between 0 and another value based on the value of a variable, so you can't be sure what the result will be.

There probably is some magical place you could put the glDiscardFramebufferEXT that would solve the problem, but it's almost impossible to tell given how randomly it occurs.

[loganmc10]

Actually you know, thinking about this, there is a way to do it, so I take that all back.

I'll see if I can create some code for you to try out in a bit.

[loganmc10]

Are you trying this on the Raspberry Pi or on your computer?

It sounds like you have some CLFAGS or some kind of compiler flag set wrong, have you been playing with the compiler flags?

[AlessandroPorcelli91]

I'm on my Raspberry Pi...Did I get it wrong?

[AlessandroPorcelli91]

It basically gets to 8% of the compilation and then it meets with the creator like this

log.txt

[loganmc10]

I'm not really sure why it's not working, you can remove -DNEON_OPT=On from the cmake options, but these issues aren't related to my code changes.

[gizmo98]

Remove -DVEC4_OPT=On or add -vfpu=neon CFLAGS.

[AlessandroPorcelli91]

Yes, I believe I messed with Neon because it was not allowing me to compile mupen64plus actually...How do I add that CFLAG back? (noob me)

[AlessandroPorcelli91]

Removing -DVEC4_OPT=On and/or -DNEON_OPT=On brings the compilation much further but still not quite to the end.
log2.txt

[loganmc10]

I would suggest rebooting the Pi (to clear any CLFAGS that might be in your environment), and then emptying the projects/cmake folder (like go into projects/cmake, do a rm -rf *) and then run cmake and make again

[AlessandroPorcelli91]

@loganmc10 That fixed it, I now have a compiled binary. I had to remove both the options above though, I don't know what am I missing.

[AlessandroPorcelli91]

@loganmc10 Alright, I have been testing it but sadly it looks like the problem is still there (I ran it through retropie putting the compiled mupen64plus-video-GLideN64.so into /opt/retropie/emulators/mupen64plus/lib/mupen64plus).

[loganmc10]

Really? thats fascinating, I'm really not sure why it works in GLupeN64 and not mupen64plus then. It's the same problem, just that things become transparent randomly over time?

[AlessandroPorcelli91]

Yup, I have uploaded a zip file some posts ago called "the glitch gremlin" in which I put some screenshots of the thing.

[AlessandroPorcelli91]

Actually, since I don't trust retropie's filesystem...let me check from inside raspbian through command line. I usually avoid using raspbian for mupen because fullscreen is completely broken in my personal experience.

[AlessandroPorcelli91]

Ok, maybe you actually fixed it...I'm saying this because I realized that it runs just as slow as it does in GLupeN64 🎯 I don't know what the hell retropie is loading as I explicitly gave it the correct plugin directory...640x480 is unviable but that's besides the point for now, I will try it windowed in a lower resolution and see if it goes full ISIS after a while. THEN I'll have to find out what's wrong with my retropie.

[AlessandroPorcelli91]

@loganmc10 I swear I'll stop posting this much...but I have bad news. The problem is still present in both GLupeN64 and the compiled GLideN64 plugin, apparently I just got lucky with the first run :( Sorry I made you port the code, I should have tested it better in the first place...

[loganmc10]

Oh that's ok, at least that makes sense (that they both act the same).

I'll try a few things over the next couple days, but this shows that adding glDiscardFramebufferEXT in additional places isn't going to do anything, there will need to be another solution.

[AlessandroPorcelli91]

Yeah, at least there's some logic behind it. It looks like it happens more often if I stand still or if I stay for a while in some locations (like the Market), I'm talking like less than 5 minutes for it to trigger. I don't yet know if it's just placebo but every longer session I had I actually ran around everywhere, so it could be that rendering the same thing over and over again makes it go crazy.

[gizmo98]

This problems happens in Zelda Majoras Mask and Conker BfD as well. Mario Tennis and Golf have glitches as well. I hope there will be a rpi4 with a better gpu.

[AlessandroPorcelli91]

Yeah...I can confirm Majora's Mask to have the same issue, too. So, do you think that's ultimately a hardware issue?

[loganmc10]

It's actually more of a driver thing than a problem with the GPU hardware. This problem doesn't happen with Eric Anholt's open source (mesa) GL driver (https://www.raspberrypi.org/blog/another-new-raspbian-release/)

However, the last time I tried mupen64plus with the opensource GL driver the performance was quite poor (this was a few months ago)

anholt/linux#57
anholt/mesa#32

[gizmo98]

I don't know if there is anything new for you but Eric Anholt has written down some VC4 performance tricks:
https://github.com/anholt/mesa/wiki/VC4-Performance-Tricks

[loganmc10]

I think I looked at that a while ago. Anyway I haven't tried it in a while maybe some improvements have been made. I'll try it out again sometime in the next few weeks.

[anholt]

Note that GL_EXT_discard_framebuffer does allow discards on the default framebuffer (0). This would be useful if you're using GLX, which normally doesn't discard depth on swaps (sigh).

I'm not sure which driver you're testing against here, but if it's the open source driver, we're unfortunately not doing anything with glDiscardFramebufferEXT() at all, but we need to .

[loganmc10]

@anholt all these issues are happening with the closed source driver, your driver doesn't suffer from these issues

[m-goetz]

Hi, I'm also facing the exact same issue when playing LOT:OOT on my RPi 3. Just as you said, the textures corrupt after an unpredictable amount of time, but I also tried Banjo Tooie, where I get the corrupted textures right at the beginning (the spinning "N" is missing parts for example). When I go through the menu to actually start the game, its imho the exact same effect as in OOT after some time. Maybe this helps you debugging...

[dankcushions]

thinking out-loud, does the recent change to allow non-0 framebuffer affect this issue? #1819

ie, could we force a non-0 framebuffer for raspberry pi? if someone could give me some hints on how i could create such a build, i could give it a try!

[loganmc10]

ie, could we force a non-0 framebuffer for raspberry pi? if someone could give me some hints on how i could create such a build, i could give it a try!

That change was just to support different rendering backends. The Raspberry Pi uses SDL, which uses "0" for the default framebuffer, so that change won't make any difference for the Pi.

[quicksilver7837]

This issue may have been solved inadvertently. Previously after running a game for about 20 mins using gliden64 on a raspberry pi the textures would corrupt and the Pi would freeze. (See https://retropie.org.uk/forum/topic/16739/n64-crashes-after-10-min/26). However after recently updating from source I no longer get the texture corruption/freeze issue and have been able to play using gliden64 on my pi for hours at a time.