fixes hang when alt+tab fullscreen direct3d game

[ericoporto]

Probable fix of #1287

My hypothesis: We are calling Reset on the direct3D device passing the d3dpp structure, and we are setting the Windowed property in it to false, but we are creating our window using SDL_WINDOW_FULLSCREEN_DESKTOP, which creates a "fake" fullscreen with a bordless window, so as far as Direct3D is aware, we have a Window, even in FullScreen.

https://docs.microsoft.com/en-us/windows/win32/api/d3d9/nf-d3d9-idirect3ddevice9-reset

[ivan-mogilko]

but we are creating our window using SDL_WINDOW_FULLSCREEN_DESKTOP, which creates a "fake" fullscreen with a bordless window

If this is what Direct3D renderer does, then it's a mistake, it should create proper fullscreen by default.
EDIT: ah, yes, it was a temporary hack to let every renderer work.

I think this should be normally fixed by expanding gfx mode to properly support both real fullscreen and borderless desktop window.

[ivan-mogilko]

Well, I can merge this as a temporary fix, if you confirm that it fixes the problem.

[ericoporto]

I think the bordless "fake" Fullscreen is better, the other one has the "crash changes desktop resolution" bug.

See #930 , which for now is entirely eliminated.

[ericoporto]

It fixes in my computer but the person that reported didn't have a "hang" like I had, they reported a proper crash, which I could not reproduce. :(

[ivan-mogilko]

I think the bordless "fake" Fullscreen is better

Engine needs to support both in the end.

Regarding this fix, I am not certain if SDL window type really conflicts with Direct3D, because Direct3D supposed to handle window modes itself. This fix simply forces Direct3D renderer to always be in a windowed mode, so there may be other reasons why it works.
Have you tried setting window flags to SDL_WINDOW_FULLSCREEN instead to see what happens?

[ericoporto]

Have you tried setting window flags to SDL_WINDOW_FULLSCREEN instead to see what happens?

I had not. But I just tried... And now I got a crash, when in Fullscreen, after a alt+tab! I changed ali3dd3d.cpp to use d3dpp.Windowed = mode.Windowed; back and changed sys_main.cpp to use SDL_WINDOW_FULLSCREEN

the error mentioned is as following:

DXERROR9(0x8876086c, "D3DERR_INVALIDCALL", "Invalid call")

Unfortunately is a super generic error. I need to sleep now before I mix something...

[ivan-mogilko]

Last question though, did fullscreen mode work correctly in before 3.6.0 (in allegro versions)?

[ericoporto]

did fullscreen mode work correctly in before 3.6.0 (in allegro versions)?

Yes, it works with 3.5.0.X versions of the engine in my Windows 10 computer, I can alt+tab from a fullscreen window and back without problem.

[ivan-mogilko]

Hm, after some thinking, I'd rather not apply such fix, but separate this into two tasks:

• implement proper choice between "exclusive fullscreen" and "desktop fullscreen" (full size borderless window) and support both modes; then at least there will be an alternative option if real fullscreen does not work.
• find out what is actually wrong with the exclusive fullscreen mode.

I'd begin with investigating how this flag is used in SDL2 source code, to make sure that it actually affects something (and what) in case when we do not create SDL_Renderer (and we only do that for the new software mode).
Besides sdl window, it may be also some mistake made in Direct3D class when porting to sdl2.
EDIT: btw, it could be something else: #1287 (comment)

[ericoporto]

Below is the function for the Direct3D renderer that is available (I think using the "Software" renderer), you can see in it's initialization it also needs to figure out if it's a "real" or "fake" Fullscreen there when setting the Windowed property.

https://github.com/libsdl-org/SDL/blob/157c3f8097403c08cc9747082de37276fb2ab28c/src/render/direct3d/SDL_render_d3d.c#L291

[ericoporto]

Yeah, I understand, I mean we can copy some of it's SDL Window handling.

[ericoporto]

I am closing this since I understand a different approach is desired.

[ericoporto]

I am leaving it reopened, but it's ok to close if a different decision is made - software is hard. :/

[ivan-mogilko]

I'll add it as a temporary fix.