SDLWindow: MSAA fallback retry + null-safe CreateWindow

[AxGord]

Summary

Three small defensive fixes in SDLWindow.cpp that we've been carrying
in our production fork (https://github.com/soccertutor/lime) for ~1 year+
on native OpenFL apps shipping to macOS, Windows, Android, and iPad:

1. MSAA fallback on window creation failure. When flags include
   WINDOW_FLAG_HW_AA / WINDOW_FLAG_HW_AA_HIRES and the initial
   SDL_CreateWindow fails, retry once with multisample buffers
   disabled. The Android Studio emulator (AVD) rejects window requests
   with multisampling, even though it supports the OpenGL context
   itself. Today the only signal a user gets is the generic
   "Could not create SDL window" message and a hard crash — the same
   app launches fine on physical Android hardware with MSAA support.
   The fallback turns "no window at all" into "a window without
   antialiasing."

2. int width, height; → int width = 0, height = 0; in
   GetWidth() / GetHeight(). If SDL_GetWindowSize returns
   without writing one of the values (early-out on a torn-down
   window), the previous version returned a stack-uninitialized int.
   Defensive only — observable as occasional one-frame size jumps
   during shutdown.

3. Null-safe CreateWindow. If the SDLWindow constructor
   finishes but sdlWindow == NULL (creation failed and we already
   logged), delete the half-constructed wrapper and return NULL
   instead of handing back an object that holds a null SDL handle.
   Without this, every subsequent caller has to defensively null-check
   sdlWindow on its own — and many don't, which manifests as a
   NULL deref deep inside event dispatch on the next frame.

Diff

project/src/backend/sdl/SDLWindow.cpp | 28 +++++++++++++++++++++++-----
1 file changed, 23 insertions(+), 5 deletions(-)

Test

The MSAA fallback path is the one with the most observable behaviour
change. Tested in production on an Android AVD that previously refused
to launch any Lime app with <window antialiasing=\"4\"/> in
project.xml — with this patch the app launches successfully with
antialiasing dropped to 0 on the same emulator image. No regression
on physical Android hardware that supports MSAA (the second
SDL_CreateWindow is only attempted when the first failed and MSAA
was requested).

The width/height defensive init has no observable behaviour change in
the happy path, only narrows the window during teardown races.

The null-safe CreateWindow is dead code on platforms where SDL
window creation never half-succeeds, but on the AVD failure path
described above the original code returned a window object pointing
to NULL — every SDL_* call against it then segfaulted. With this
patch the call site sees a clean NULL and can fall back to
Application::onException / display an error.

Risk

• Hardware/driver combos that previously failed window creation with
  MSAA flags now succeed with MSAA off. Apps that depend on receiving
  an MSAA-enabled context (e.g. expecting glIsEnabled(GL_MULTISAMPLE)
  to be true) and treat its absence as a fatal error would see
  different behaviour. We've never seen that in practice — apps that
  want MSAA enable it via <window antialiasing=\"N\"/> and treat
  "didn't get it" the same as "got it without".
• No platform code or public API touched outside SDLWindow.cpp.

[tobil4sk]

Maybe best to remove the emdash as it may cause issues in some editors

[AxGord]

Good catch — replaced with an ASCII hyphen in cbc833f.

[tobil4sk]

We have to make sure all callers of CreateWindow check for null with this change. We should also return nullptr rather than NULL since it is c++ code.

[AxGord]

Done — nullptr in cbc833f. On the caller side: both C++ callers (lime_window_create / hl_window_create in ExternalInterface.cpp) pass the result straight into CFFIPointer/HLCFFIPointer, which already null-check (CFFIPointer.cpp returns alloc_null() for a null pointer; HLCFFIPointer returns 0), so the Haxe side gets a real null. NativeWindow.create() did guard the width/height/x/y block with if (handle != null), but then called lime_window_get_scale(handle) and lime_window_get_context_type(handle) unconditionally — those deref a null handle. Fixed in the same commit by replacing the partial guard with an early return right after lime_window_create.

[tobil4sk]

Can SDL_GetWindowSize actually exit without writing these values? Sdl2 docs don't mention that case: https://wiki.libsdl.org/SDL2/SDL_GetWindowSize

[AxGord]

It can, via CHECK_WINDOW_MAGIC. In SDL2 (src/video/SDL_video.c) SDL_GetWindowSize starts with CHECK_WINDOW_MAGIC(window, );, which early-returns before writing *w/*h when the video subsystem is uninitialized, window is NULL, or window->magic doesn't match (a destroyed/stale window — the teardown race this guards). The wiki doesn't mention it but the source confirms it, so the = 0 init is defensive against exactly that path; the happy path is unaffected.

[tobil4sk]

Just to provide the source:
https://github.com/libsdl-org/SDL/blob/f0d1101920864995c0c265128cfd7b96eb959ac5/src/video/SDL_video.c#L2399

https://github.com/libsdl-org/SDL/blob/f0d1101920864995c0c265128cfd7b96eb959ac5/src/video/SDL_video.c#L161

The docs also say:

NULL can safely be passed as the w or h parameter if the width or height value is not desired.

Which can make the code cleaner and avoid setting the one we don't need to 0.

Also as an aside, it looks like in SDL3 we will be able to check the return value of SDL_GetWindowSize to see if it failed: https://wiki.libsdl.org/SDL3/SDL_GetWindowSize

[AxGord]

Good call — adopted in 16a34b6. GetWidth/GetHeight now pass NULL for the dimension they don't return and only init the one they do (still = 0 so the CHECK_WINDOW_MAGIC early-return path stays defensive). Thanks for the SDL source links and the SDL3 return-value note.

[tobil4sk]

Just to provide the source:
https://github.com/libsdl-org/SDL/blob/f0d1101920864995c0c265128cfd7b96eb959ac5/src/video/SDL_video.c#L2399

https://github.com/libsdl-org/SDL/blob/f0d1101920864995c0c265128cfd7b96eb959ac5/src/video/SDL_video.c#L161

The docs also say:

NULL can safely be passed as the w or h parameter if the width or height value is not desired.

Which can make the code cleaner and avoid setting the one we don't need to 0.

Also as an aside, it looks like in SDL3 we will be able to check the return value of SDL_GetWindowSize to see if it failed: https://wiki.libsdl.org/SDL3/SDL_GetWindowSize

[tobil4sk]

It seems this early return causes frameRate to not be set. Not sure what the expected state is for a window that failed to launch.

[AxGord]

Fixed in c668007 — frameRate = resolvedFrameRate; now runs before the early return, so a window that failed to launch still has frameRate populated for getFrameRate()'s field fallback. It's a plain field write (no handle deref) and resolvedFrameRate is already computed above, so it's safe on the failure tail. setTextInputEnabled(false) is still skipped, but that one only configures SDL state via the (now null) handle and is internally handle-guarded, so there's nothing meaningful to set for a window that never opened.