Skip to content

Fix signed integer overflow in GetCurrentMonitor distance calc#5842

Open
Vasilis-Narain wants to merge 1 commit intoraysan5:masterfrom
Vasilis-Narain:fix-getcurrentmonitor-overflow
Open

Fix signed integer overflow in GetCurrentMonitor distance calc#5842
Vasilis-Narain wants to merge 1 commit intoraysan5:masterfrom
Vasilis-Narain:fix-getcurrentmonitor-overflow

Conversation

@Vasilis-Narain
Copy link
Copy Markdown

Problem

GetCurrentMonitor() in rcore_desktop_glfw.c computes int dist = dx*dx + dy*dy where dx, dy are pixel offsets from window centre to monitor corner. On my setup (multi-monitor WSL2 through Windows 11) the squared sum exceeds INT_MAX, causing signed integer overflow UB. Today GCC wraps silently, but UBSan-instrumented builds (eg. Zig or clang with -fsanitize=undefined) fail.

Environment

  • OS: Ubuntu 24.04.1 on WSL2 (Windows 11 host), X11 via WSLg
  • Compiler: gcc 13.3.0
  • Display 1920x1200
  • Detected on: Zig 0.16 bundling raylib 6.0

Repro

Originally caught by Zig 0.16's UBSan trapping on the bundled raylib build:

thread panic: signed integer overflow: 1173062500 + 1050343281 cannot be represented in type 'int'
rcore_desktop_glfw.c:887

Reproduced standalone by adding a printf next to the overflow site and running examples/core/core_basic_window:

REPRO i=0 dx=-34250 dy=-32484 prod=2228272756 INT_MAX=2147483647 overflow=1
REPRO i=1 dx=-32330 dy=-32484 prod=2100439156 INT_MAX=2147483647 overflow=0

prod (computed as (long long)dx*dx + (long long)dy*dy) exceeds INT_MAX, confirming the truncation.

Fix

Promote dist and closestDist to long long. No behavior change for monitors within INT_MAX distance and correctly handles cases beyond it without UB.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant