GIF recorder uses wrong dimensions with SGB border

[dobyrch]

Version: 9630583
OS: Arch Linux x86_64

The GIF recorder always records Game Boy games at 160x144, resulting in cropped images when SGB borders are enabled.

[endrift]

Good catch. I'll see about fixing this soon.

[dobyrch]

@endrift Thanks for the fast fixes!

Unfortunately, I'm now getting a segfault, sometimes a second or two after clicking "start", and sometimes after clicking "stop." I managed to successfully record a gif once, but I haven't figured out how to reproduce it.

Also, mgba-qt aborts if SGB borders are turned off.

[dobyrch]

Scratch that, I did a fresh build today and everything is working great. Thanks again!

Edit: I spoke too soon... Recording works fine with borders disabled (and mgba-qt no longer aborts on start up), but I'm still getting segfaults when recording with borders on:

Stack trace of thread 7260:
#0  0x00007f33b2e1711e _int_malloc (libc.so.6)
#1  0x00007f33b2e1847a malloc (libc.so.6)
#2  0x00007f33b6613abb n/a (libMagickCore-7.Q16HDRI.so.5)
#3  0x00007f33b661a40f n/a (libMagickCore-7.Q16HDRI.so.5)
#4  0x00007f33b661a6cb QuantizeImage (libMagickCore-7.Q16HDRI.so.5)
#5  0x00007f33b6503946 SetImageType (libMagickCore-7.Q16HDRI.so.5)
#6  0x00007f335640babd n/a (gif.so)
#7  0x00007f33b6539c38 WriteImage (libMagickCore-7.Q16HDRI.so.5)
#8  0x00007f33b653a576 WriteImages (libMagickCore-7.Q16HDRI.so.5)
#9  0x00007f33b6a36884 MagickWriteImages (libMagickWand-7.Q16HDRI.so.5)
#10 0x000055ba4646bdb2 n/a (/home/dobyrch/Projects/mgba/build/qt/mgba-qt)
#11 0x00007f33b4a19b03 _ZN15QAbstractButton7clickedEb (libQt5Widgets.so.5)
#12 0x00007f33b4a19d1c n/a (libQt5Widgets.so.5)
#13 0x00007f33b4a1b15c n/a (libQt5Widgets.so.5)
#14 0x00007f33b4a1b35e _ZN15QAbstractButton17mouseReleaseEventEP11QMouseEvent (libQt5Widgets.so.5)
#15 0x00007f33b4964188 _ZN7QWidget5eventEP6QEvent (libQt5Widgets.so.5)
#16 0x00007f33b4923fec _ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent (libQt5Widgets.so.5)
#17 0x00007f33b492bf41 _ZN12QApplication6notifyEP7QObjectP6QEvent (libQt5Widgets.so.5)
#18 0x00007f33b3ba5cf0 _ZN16QCoreApplication15notifyInternal2EP7QObjectP6QEvent (libQt5Core.so.5)
#19 0x00007f33b492aeca _ZN19QApplicationPrivate14sendMouseEventEP7QWidgetP11QMouseEventS1_S1_PS1_R8QPointerIS0_Eb (libQt5Widgets.so.5)
#20 0x00007f33b497f949 n/a (libQt5Widgets.so.5)
#21 0x00007f33b4982060 n/a (libQt5Widgets.so.5)
#22 0x00007f33b4923fec _ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent (libQt5Widgets.so.5)
#23 0x00007f33b492b9c6 _ZN12QApplication6notifyEP7QObjectP6QEvent (libQt5Widgets.so.5)
#24 0x00007f33b3ba5cf0 _ZN16QCoreApplication15notifyInternal2EP7QObjectP6QEvent (libQt5Core.so.5)
#25 0x00007f33b4119434 _ZN22QGuiApplicationPrivate17processMouseEventEPN29QWindowSystemInterfacePrivate10MouseEventE (libQt5Gui.so.5)
#26 0x00007f33b411af86 _ZN22QGuiApplicationPrivate24processWindowSystemEventEPN29QWindowSystemInterfacePrivate17WindowSystemEventE (libQt5Gui.so.5)

[endrift]

Could be out of memory. How long are you trying to record for?

[dobyrch]

Not long at all, it usually locks up as soon as you hit start and crashes one or two seconds later.

I think I've figured out what's happening... ImageMagickGIFEncoderInit sets iwidth and iheight to the GBA dimensions, i.e. 240x160. Adding a print statement right before the malloc in ImageMagickGIFEncoderOpen confirms that the dimensions don't get updated. When SGB borders are enabled, the buffer isn't big enough, leading to an overflow that corrupts the heap—hence the segfault in a following call to malloc.

[endrift]

Good call, that seems entirely plausible.