Crazy region based screenshot behavior on macOS

[svettwer]

Version
next

Short overview
Screenshots based on Regions on macOS are do not reflect screen content.

Issue occurs on

• Virtual machine
• Docker container
• Dev/Host system

Detailed error description
I'm using the following workaround from #154 to create a screenshot of a region

  const regionCapture = async (
    selectedRegion: NutRegion,
    outputFilename: string
  ) => {
    const regionImage = await (<any>screen).vision.grabScreenRegion(
      selectedRegion
    );
    await (<any>screen).vision.saveImage(regionImage, outputFilename);
  };

Based on the provided region, I receive various errors or incorrect images.
I'm working on a 1920x1080 build agent and configured it as described here.

with new NutRegion(0, 100, 650, 200) I get:

with new NutRegion(0, 50, 325, 100) I get:

Just thought a reduction of the resolution could help.

with new NutRegion(0, 0, 1900, 1000) I get:

[Wed Dec 09 2020 13:04:35 GMT+0000 (Coordinated Universal Time)] ERROR: Error in testcase getTextFromRegion: Error: Given width exceeds display dimensions
Error: Error: Given width exceeds display dimensions
    at Object.module.exports.screen.capture (/Users/runner/work/sakuli-ocr/sakuli-ocr/packages/sakuli-ocr/node_modules/@nut-tree/libnut-darwin/index.js:32:16)
    at /Users/runner/work/sakuli-ocr/sakuli-ocr/packages/sakuli-ocr/node_modules/@nut-tree/nut-js/dist/lib/provider/native/libnut-screen-action.class.js:30:46
    at new Promise (<anonymous>)
    at ScreenAction.grabScreenRegion (/Users/runner/work/sakuli-ocr/sakuli-ocr/packages/sakuli-ocr/node_modules/@nut-tree/nut-js/dist/lib/provider/native/libnut-screen-action.class.js:29:16)
    at VisionAdapter.grabScreenRegion (/Users/runner/work/sakuli-ocr/sakuli-ocr/packages/sakuli-ocr/node_modules/@nut-tree/nut-js/dist/lib/adapter/vision.adapter.class.js:43:28)

Whole screen
Funny enough, when I take a screenshot with grabScreen, it looks as expected.

Steps to reproduce error
As this happens on a build agent and because I've no mac myself, I cannot provide any information, unfortunately.

[svettwer]

We were able to reproduce the error during travis to GH Action migration of https://github.com/sakuli/sakuli as well. But this time, the issue also occurred with grabScreen. As nut.js is already on GH actions, it would be interesting to get some screenshots from within the build runner. 🤔

@s1hofmann Do you have an Idea what could cause this issue? We're currently a little confused about the root cause to be honest. 🙈

[s1hofmann]

@svettwer I'm currently debugging this issue to analyse the root cause.

[svettwer]

Awesome! Let me know if I can help. 👍

[s1hofmann]

Will continue later this evening. Might hit you up for a little session, depending on how late my later will be.

[s1hofmann]

@svettwer When capturing the whole screen via libnut.screen.capture(); and libnut.screen.capture(0, 0, libnut.screen.width, libnut.screen.height); it gives comparable data.
Once stored, there are offsets present within the region based data, resulting in an image tilt.

Overall, the issue seems not to be related to the capturing mechanism in libnut, will investigate further.

[s1hofmann]

@svettwer More details on why this happens can be found here: nut-tree/libnut-core#44

Looks like a platform problem.

[s1hofmann]

And it seems like it’s no longer a problem on Big Sur: https://github.com/nut-tree/libnut/actions/runs/500905237

[s1hofmann]

To sum things up: Broken screenshots are caused by and only happen on macOS Catalina. As it is no problem on Big Sur I guess we’ll live with it until explicitly required.