SDL_EVDEV_is_console() uses type of wrong size when calling ioctl

[SDLBugzilla]

This bug report was migrated from our old Bugzilla tracker.

These attachments are available in the static archive:

• KDGKBTYPE type patch (kybrd.type.patch, text/plain, 2017-01-05 22:55:52 +0000, 463 bytes)

Reported in version: 2.0.5
Reported for operating system, platform: Linux, ARM

Comments on the original bug report:

On 2017-01-05 22:55:52 +0000, Rob wrote:

Created attachment 2659
KDGKBTYPE type patch

When calling ioctl(fd, KDGKBTYPE, &type) in SDL_EVDEV_is_console(), we declare type as an 'int'. This should be a 'char'. The subsequent syscall, and kernel code, only writes the lower byte of the word.

See: http://lxr.free-electrons.com/source/drivers/tty/vt/vt_ioctl.c?v=4.4#L399

ucval = KB_101;
ret = put_user(ucval, (char __user *)arg);

I've observed intermittent behavior related to this, and I can force an error condition by using an int initialized to 0xFFFFFFFF. The resulting ioctl will set type to 0XFFFFFF02, and the conditional return in SDL_EVDEV_is_console() will fail.

Recommend changing to char, or masking off unused bits.

See attached patch for suggested fix.

-Rob

On 2017-01-06 07:26:55 +0000, Sam Lantinga wrote:

This is in, thanks!
https://hg.libsdl.org/SDL/rev/6cfccf993c5d