Improve performance with reusable buffers #424
Description
I propose adding optional support to reuse buffers for improved performance. This can be enabled with a keyword in the grab method, or in the MSS constructor. (The shot and save methods can unconditionally enable it when they call grab, if appropriate locking is used.)
The idea here is that, rather than creating new buffers based on the data we get back from the backend (like BitBlt or XGetImage), we return a memoryview buffer that points to the data we got back.
The good part of reusable buffers is that using them is faster, by a matter of a few milliseconds. You use the screenshot data that the system gives you in-place, rather than copying the 32 MB (4k resolution, BGRX) pixel buffer. In some cases, such as in XShmGetImage, the Python program can use the OS-supplied buffer as it stands, never having to copy it into its own memory, not even needing to read a socket. If you’re just using a subset of the image, the Python program doesn’t even have to touch the pixel memory it doesn’t need!
NumPy, PIL, PyTorch, and other frameworks all can reuse an existing buffer well (using their respective .frombuffer methods). You could read a screenshot, load it into PyTorch, run an AI model on it, and never once have to duplicate that 32 MB buffer.
Just for some vague and loose numbers, on my box, simply copying the data in a 4k RGBA buffer (3840 * 2160 * 4 bytes) takes 3.20 ms (for simply copying the contents, not allocating storage or anything). That would be the difference between 60 fps and 50 fps.
There is a downside, of course. With reused buffers, the returned ScreenShot object’s contents would only be valid until either (1) the next call to grab, or (2) the parent MSS object is destroyed, whichever comes first. The ScreenShot object itself would be valid for its lifetime — you could read attributes and hold references to it and such — but trying to read the pixel values after the next grab call would give you undefined data, or even possibly segfault.
This is not a big limitation in practice. In the vast majority of use cases, you only care about one screenshot at a time.
Some backends might not support a full zero-copy path, and that’s okay. This is an optimization, but not a guarantee.
For the backends that do support reusable buffers, making it optional is clearly easy: if the user didn’t request zero-copy, MSS just makes a copy before returning from grab, guaranteeing that the contents will be available for as long as the ScreenShot exists.
This would be an advanced use case, meant for users who need high performance. But for those users, milliseconds matter!