Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Visual stimulus onset timing/timestamping/Retina workarounds for Apples macOS trainwrecks. #159
These are the long awaited workarounds to counteract the utter brokenness of Apples shoddy
They have been developed under contract with Denis Pelli. Denis donated about half of the cost for
These fixes should make the dreaded 'sync failures' and other timing errors go away or at least
The fixes have been so far successfully tested on single-display and dual-display setups under:
The fixes are targeted at machines running macOS 10.14.6 Mojave. macOS 10.15 Catalina may work as well, but is not yet officially supported.
What should work reasonably well, at least in fullscreen mode:
What will be as unfixably broken as before wrt. visual timing:
More bugs that were found, but are not related to this work and scope of the contract, sometimes on macOS 10.14.6 Mojave, sometimes on 10.15.1 Catalina during testing. Some of these are non-deterministic, sometimes it works, sometimes it doesn't, but most of the time it doesn't:
Please note that all bugs "fixed" in this patch series are not PTB bugs, but bugs in Apples operating systems. Also, the "fixes" are not fixes in the right sense of the word. They are - sometimes puke inducing ugly - hacks and workarounds to coerce Apples trainwrecks into more reasonable behaviour. These hacks may have yet unknown undesirable side effects. They also may break again in any future Apple macOS update.
Switching to a less insane and low quality operating system is still strongly recommended for any use case other than entertainment and education!
Technical high level overview of what is worked around:
With all those 7 (!) independent serious bugs in Apple macOS worked around, we seem to get reasonably reliable visual stimulus onset timing and timestamping again on the tested HW + OS combos.
OSX 10.14 Mojave doesn’t like it if one executes GUI operations like onscreen window creation on a thread other than the main application thread. This shows up as warnings in Matlab and crashes / abort() calls on Octave under 10.14. Try to use GCD’s dispatch_sync() function to call GUI related code in PsychCocoaGlue.c on the application main thread. This sort of seems to work, as in octave-gui no longer abort()’s with SIGABORT, and Matlab is still fine. And the onscreen window shows rendered content. However, now single-thread operation for octave-cli dies, as we must not use dispatch_sync() there! Also still sync failure on octave with apparently no vsync at all, whereas on Matlab we just get the regular sync failure due to Mojave’s miserable timing. -> Backup commit for further investigation.
…10.14 Apples latest disaster makes Octave crash on 10.14 at exit iff Screen() was used at least once in the session. Why, i don’t know, but it is related to some memory allocation/deallocation/reference counting whatever wrt. to the activity object we use as token to en/disable AppNap at Screen load/unload time. Hours of tinkering found a non-sensical solution that seems to work with Octave, while not hurting Matlab. We use an autorelease-pool for AppNap handling in PsychCocoaPreventAppNap() and then counteract the attempt of the autorelease pool to release our activity token NSObject by [activity retain] ing it! On AppNap disable, we [activity release] it, which according to the [activity retainCount] debug output has no effect - the retainCount stays 1 ! Go figure… Whatever, Matlab still works in GUI and CLI mode without crashes, and Octave-GUI in cli mode no longer crashes there, so there!
This is naturally limited to Linux and OSX(with kernel driver loaded and the dangerous override for Intel set), and the register mapping is chosen based on IVB and so far only tested for IvyBridge IGP. May or may not work on earlier or later gens, too lazy to look it up atm. -> This to debug the latest Apple provided macOS disasters for 10.14 Mojave.
All the testing with the new Intel gpu pageflip diagnostic on 10.13.6 and 10.14.0 with Intel Ironlake and Intel IvyBridge gfx, as well as testing on old Nvidia’s of the GT330M and GT120M series and with a brand-new Radeon Pro WX4100 showed that the root cause of most of the timing misery at least on 10.13 and 10.14 seems to be compositor interference. We are not in control of page flipping, but all our fullscreen windows get redirected to the compositor, which then screws up all our timing and timestamping, as not only PTB’s internal diagnostic shows, but also measurement with a Datapixx. Given that the idiots apparently totally screwed up their Cocoa-based fullscreen windowing support, which was supposed to be optimised, but apparently isn’t, i made a bold move: What about doing the exact opposite of everything Apple recommends wrt. performant graphics programming? Use the old CGLSetFullScreenOnDisplay() call again to switch the OpenGL rendering context into fullscreen mode on a display, completely bypassing the Cocoa API pile of shit. This in the faint hope that the way we did it over 4 years ago, before CGLSetFullScreenOnDisplay() got deprecated in 10.7 with the strong recommendation by Apple to avoid it, is now no longer the most broken way. Apparently the asshats managed to screw up their shiny new stuff so badly that the old buggy solution from years ago is now the less buggy solution. And voila! Going back to CGLSetFullScreenOnDisplay() for fullscreen windows immediately resolved all(!) the catastrophic visual timing and timestamping problems under 10.14 Mojave with Intel gfx. At least as tested on single display stimulation on single- and dual-display setups. Lets hope these results will hold up for other OS + GPUs + display-config combos as well. -> Promising backup commit.
Screen(‘Preference’,’Conservevram’, 8192) for simple method. Otherwise CGL backbuffer scaling is used — not very successfull.
Didn’t work at all on Retina panel with AMD Polaris. Made things worse on Ironlake and GeForce 330. No effect on Intel IvyBridge.
On AMD gpu’s, read the graphics format register during flips via MMIO on Linux or PTB kernel driver on macOS to report system framebuffer scanout format used by the AMD display engines and warn about troublesome mismatch with OpenGL fb format, if high verbosity levels of >= 16 are requested. -> This mostly to debug trouble with Apples latest trainwrecks.
As the previous commit showed, we have yet another new graphics and display system bug on top of the pile of shit that is current macOS display system. The geniuses at the iToys company now operate the system display framebuffer on some gpu's in color depth 30, ie. 10 bpc, ARGB2101010 display engine scanout mode. This without any rhyme or reason, because the OpenGL framebuffer exposed to fullscreen applications like PTB is still only 8 bpc ARGB8888, so there is not any net gain in precision over an 8 bpc system fb. Unless you request a floating point 16 bit framebuffer under OpenGL, supported on some macOS systems. In that case however, the system fb is still 10 bpc on those machines (8 bpc on others), not using a 16 bit half-float display scanout format. Up to 11 bits of grayscale precision are emulated with some Apple custom dithering even on 8 bpc panels, completely ignoring the gpu hw supported dithering, so even at 16 bit float precision the 10 bpc fb mostly doesn't make sense. A 10 bpc framebuffer is also run on pure 8 bpc display devices like old analog VGA monitors connected via USB-C -> HDMI -> DVI -> VGA adapters with only 8 bit DACs, or anything DVI 8 bit, or DP/HDMI 8 bit. The only meaningful explanation would be if some other applications using graphics API's other than OpenGL (Metal? CoreGraphics?) could use 10 bpc native and 10 bpc displays were connected - or 10 bpc dithered to 8 bpc via the gpu's *built-in* dithering hw, and the display shows the normal windowed desktop GUI. None of this is the case for a fullscreen exclusive OpenGL app, but then Apple already publicly declared that they don't give a shit about OpenGL anymore, so this may be just malicious negligence towards OpenGL compatibility. The net effect of this decision is that on such machines with 10 bpc system framebuffer, all supported OpenGL fb formats are incompatible with the display framebuffer format, and therefore the Quartz desktop compositor has to kick in to perform active format conversion at each flip, which means completely screwing up visual stimulus onset timing and timestamping, with no way around that! So what do we do? Whenever PTB operates in standard 8 bpc display mode (ARGB8888 OpenGL framebuffer) with a fullscreen exclusive onscreen window, we check if the system fb of the target display is in 10 bpc mode. If so, we request a switch of the system fb to 8 bpc ARGB8888 mode, so OpenGL fb and system fb format match and the compositor can go out of the way, restoring pageflipping controlled by PTB and thereby a chance of good timing. At window close time, we restore the format to whatever the system fb format was before. There ain't absolutely nothing we could do if the usercode requests the high precision 16 bit float framebuffer, as there is not hack in the world to make that compatible with compositor-bypass operation -- you can have either good timing or high precision, but not both. Now Apple's official display configuration api's have removed any ability to detect or select system fb bit depths -- Thanks Apple! In order to do the format switching anyway, we have to use Apple macOS system private interfaces (SPI's), belonging to the display server. These api's are not documented and strongly discouraged, found by reverse engineering. My testing shows they at least work on macOS 10.11, 10.12, 10.13 and 10.14. However, nobody can know or be sure that this sneaky use of SPI's may not have other awful side effects - Here be dragons! Also, Apple could break those SPI's at any time with any macOS version update for any reason and we would be back to square one with no known options left! I guess if one insists on using a shit operating system by a life-style toy company, one has to die one death :/ -> Tested and verified to work on 10.12.6 with a MacBookPro 2017 with AMD Polaris gpu with internal Retina display (10 bpc according to EDID) and external 8 bpc HDMI/DVI/VGA monitor. Also seems to work on 10.14.6 on same hw with the external standard DPI display, although with some occassional flakyness that could be caused by use of SPI's. On 10.14.6 the internal Retina display is broken again, but for what seems to be yet another display bug piled onto the macOS shit pile with 10.14.6. -> CGL backend still works on 10.14.6 with Intel HD 4000 Ivybridge (macMini 2012) under macOS 10.14.6, however Cocoa backend is broken again! Works CGL and Cocoa on Intel HD and old NVidia of MBP 2010 under macOS 10.13.6, iow. on systems that don't use a 10 bpc system fb. -> The bugs are all over the map between macOS 10.12, 10.13, 10.14 and different gpu x Retina/non-Retina combos. What a shit show of broken macOS graphics!
Adding braces, so proper dbug output shows under Matlab. Other calls, motivated by looking at SDL. None of which helped in any way.
Since pageflipping is completely broken under the Cocoa backend with macOS 10.14.6 Mojave and 10.15.0, we have to use CGL for both non-Retina and Retina displays. As CGL does not support Retina scaling in any way, and CGL backbuffer scaling usually kills pageflipping as well, we use the following hack: We try to find the native display mode of a builtin/Retina display and then modeswitch to it. Then use that to drive the Retina display directly at native resolution for page- flipping under CGL. Restore mode at end of session. This is dumb, but at least it works under 10.14.6 Mojave, as the only known way to fix Retina display timing. Also restructure matching logic, improve diagnostic debug output etc. So far tested on 10.14.6 with MBP 2017 AMD Polaris, with external standard DPI display - a VGA monitor - and on the internal laptop Retina panel.
Move the enable call for the framebuffer mode/format switching into PsychOSOpenOnscreenWindow(), where it belongs and where we actually know the windows OpenGL framebuffer color bit depth. We only skip enable for fullscree windows if skipsynctest level 2 is selected. Move disable call inside PsychReleaseScreen() after the CGDisplayRelease() call, as otherwise we observe random WindowServer hangs and crashes in about 1/3rd of all sessions.
Adapt to our new "switch to display native resolution" hack under CGL backend with fullscreen onscreen windows. ->Verified to work on normal display and on Retina display if resolution is set to "best for Retina" ie. exact half the native panel resolution. Also works with other scaled Retina resolution, with and without panel-fitter, but there work needs to be done wrt. beamposition endline detection. -> Tested on MBP 2017, AMD Polaris, macOS 10.14.6. -> Add a line of diagnostic code to LinesDemo.m to simplify Retina compat/native mode testing. Commented out by default.
Allow early abort of test via any key press. Test all color channels, not only red channel. Classify failure type and give more detailed/useful output for given failure type.
Need to requery vbl startline, to not confuse beampos query & timestamping on other than 2x scaling.
We used to call PsychAutoDetectScreenToHeadMappings() at Screen() init time, but now, just as on Linux, we call it at the end of PsychOSOpenOnscreenWindow(), after the OpenGL context for the window has been created. Why? Because at least on modern MacBookPro's, e.g., the 2017 15 inch MBP with AMD Polaris gpu, the discrete gpu from AMD may be powered down before the OpenGL context is created. Accessing the powered down gpu for register reads in PsychAutoDetectScreenToHeadMappings can cause a hard fault, leading to a unhandled machine check exception (MCA/MCE CATERR) and therefore to a hard crash -> emergency reboot --> Not what we want as user experience. This new way of calling after OpenGL context creation, we can be sure that the AMD gpu is actually powered up and online when we access it first. -> Fixes hard crashes of MBP 2017 if automatic graphics switching is active.
At least on the current PTB kernel driver with AMD gpu's that have a DCE-11 or later display engine, where we already know it does not work since at least macOS 10.12.
So NSWindowStyleMaskFullScreen and NSWindowCollectionBehaviorFullScreenPrimary did not help timing under Cocoa, but they do cause windows to be always drawn with decorations on Mojave + Matlab and always on Catalina. Remove them. Also disable the backbuffer scaling setup in CGL. Did not ever help, so just be safe.
Did not help at all on Mojave, did mostly help on Catalina in the sense that Catalina only gets one wrong pixel level.
If the user presses any key on any keyboard, then abort the tweaking. Very neccessary for runaway tweaks.
The Apple iMac's internal flat panels, despite having a native resolution, are not exposed by the trainwreck as having a native preferred optimal resolution. Instead they pretend to be like CRT monitors that can operate at different video resolutions with proper timing etc. Apparently the gpu's hw scaler is used to rescale from the PTB selected framebuffer size aka "video mode" to the true native panel size, iow. the gpu frontend and backend resolutions are mismatched. This doesn't seem to affect pageflipping and visual onset timing, but it does screw up our beamposition timestamping, as we need the true vbl startline for that, not the fake framebuffer height / video mode height. So where do we get it from? Various approaches were considered, e.g., direct vblank start position readout from the gpu, but all had drawbacks like high dependency on the PTB kernel driver, and/ or higher maintenance overhead for the driver, or ugly heuristics that may go wrong, given that we have to rely on macOS broken and lying video mode api's. In the end, the least awful solution seems to be a lookup table that looks up the true panel native pixel size by Mac modelId and panel width in mm and uses that for beamposition setup. This will require maintenance for all future Mac's that have builtin panels and this brain-damage implemented, but at least maintenance by editing a LUT and recompiling Screen() is easier than maintenance of the kernel driver. So far, only iMac's seem to suffer from this special type of misdesign, the Apple MacBook line laptops and external displays seem to be not affected by this stupidity. This keeps the effort low enough to only dealing with new iMac models. -> Survived basic test on a MBP 2010, test on actual iMac pending.
Beautify some debug messages, drop others, put yet others behind higher verbosity levels. Add some code formatting and comments. -> Update License.txt to acknowledge use of 3rd party code for SPI access for video mode switching via CGSxxx() calls.
We now nominally support and test macOS 10.13.6 “High Sierra” and macOS 10.14.6 “Mojave”. No official 10.15 Catalina support yet.
…sual fixes. We now do some of the mouse coordinate scaling neccessary for dealing with Retina displays already inside Screen('GetMouseHelper'). This fixes long- standing bugs in the original implementation, and new issues caused by interactions with the new collection of hacks to deal with Retina displays and CGL on Apples trainwrecks. Tested under CGL and Cocoa (for transparent windows etc.), with or without 'UseRetinaDisplay' aka native retina resolution vs. backwards compatibility mode, on Retina and non-Retina displays, single screen and dual screen.
-> Contains all the workarounds for many, many, oh so many awful and mind-blowing visual timing bugs in Apples parade of trainwrecks, or as they call it "The most advanced operating system in the world". -> Retested on MBP 2010 standard display, and MBP 2017 Retina and non-Retina display single/dual-display. -> We should be ready for merge and release.
kleinerm's pull for PTB BETA "Syncopation" General stuff: - help text and doc updates. - UpdatePsychtoolbox can now work even if the Psychtoolbox folder is not on the path. Contributed by Denis Pelli. - Improvements to diagnostics, e.g., DatapixxGPUDitherpatternTest, FlipTimingWithRtPhotoDiodeTest. Now also supports VideoSwitcher + RtBox for timing tests. Fixes and workarounds for the various horrifying Apple macOS operating system bugs: - KbCheck: Handle Apple MacBookPro TouchBar Gimmick. - KbCheck: Workaround for macOS Catalina's shoddy IOKit HID support which led to slowdowns of 50x - KbQueueCreate: Workaround for macOS Catalina's latest multi-threading bugs to avoid crashing. - Screen: Improve Retina display handling related to GetMouse and use of panel fitter. - Screen: Workaround the endless number of graphics/display driver bugs affecting visual stimulus onset timing and timestamping. This should bring improved or well working visual timing on a variety of hardware on macOS 10.11 to 10.15.1, but is targeted at specifically macOS 10.14.6 Mojave with the MacBookPro 2017 and the iMac 27 inch 5k Retina late 2014. Work done under contract with Denis Pelli. A summary of all the horrors in Apples trainwrecks, and how we work around them now can be found in this pull request: kleinerm#159 - Fix crashes of Octave 5.1 during operation and shutdown caused by macOS multi-threading bugs and other Apple screwups that just baffle my little brain to no end. - Fix crash of Matlab R2019b due to more horrendous Apple screwups. Improvements for Windows: - Make disable of display dithering for modern AMD gpu's on modern drivers actually work, for identity pixel passthrough needed by visual stimulators of VPixx and Cambridge Research Systems. Improvements for Linux: - Improve gpu detection for some new AMD gpu's.
PTB BETA Update "Syncopation" This release was aptly named by Denis Pelli: The main meat of it is substantial improvements to visual stimulation timing on Apple macOS, as well as other fixes for Apples trainwrecks, e.g., for keyboard queries and for running Octave and Matlab R2019b without crashing. The timing fixes were done as contract work, paid by Denis Pelli. For more details about the general state (=awfulness) of current macOS and about these fixes see: kleinerm#159 Iow. all Apple users should upgrade to this release asap. General stuff: - Help text and doc updates. - DownloadPsychtoolbox/UpdatePsychtoolbox: Improve some online help texts. Contributed by Ian Andolina. - UpdatePsychtoolbox can now work even if the Psychtoolbox folder is not on the path. Contributed by Denis Pelli. - Improvements to diagnostics, e.g., DatapixxGPUDitherpatternTest is even more careful now about detecting trouble. - FlipTimingWithRtPhotoDiodeTest. Now also supports VideoSwitcher + RtBox for timing tests. - Improved data sets: Update T_xyz1931 and T_xyz1964 with more decimal places and wavelength samples. Updated versions provided by Danny Garside. It’s possible this could break code that made assumptions about wavelength sampling, but code should not do that. Reviewed and merged by David Brainard. - New function contributed by Danny Garside: drawChromaticity() can plot various types of chromaticity diagrams with spectrum locus. Reviewed and merged by David Brainard. Fixes and workarounds for the various horrifying Apple macOS operating system bugs: - KbCheck: Handle Apple MacBookPro TouchBar Gimmick. - KbCheck: Workaround for macOS Catalina's shoddy IOKit HID support which led to slowdowns of 50x - KbQueueCreate: Workaround for macOS Catalina's latest multi-threading bugs to avoid crashing. - Screen: Improve Retina display handling related to GetMouse and use of panel fitter. - Screen: Workaround the endless number of graphics/display driver bugs affecting visual stimulus onset timing and timestamping. This should bring improved or well working visual timing on a variety of hardware on macOS 10.11 to 10.15.1, but is targeted at specifically macOS 10.14.6 Mojave with the MacBookPro 2017 and the iMac 27 inch 5k Retina late 2014. Work done under contract with Denis Pelli. A summary of all the horrors in Apples trainwrecks, and how we work around them now can be found in this pull request: kleinerm#159 - Fix crashes of Octave 5.1 during operation and shutdown caused by macOS multi-threading bugs and other Apple screwups that just baffle my little brain to no end. - Fix crash of Matlab R2019b due to more horrendous Apple screwups. Improvements for Windows: - Make disable of display dithering for modern AMD gpu's on modern drivers actually work, for identity pixel passthrough needed by visual stimulators of VPixx and Cambridge Research Systems. Improvements for Linux: - Improve gpu detection for some new AMD gpu's.