Face Boundaries shouldn't use Scale

[mrousavy]

Hey @nonam4, I'm creating this issue to follow up on the discussion about returned coordiantes/bounds being incorrect.

A Camera streams in it's native resolution, and sometimes in landscape orientation, since that's just how sensors work.
If I run any ML model on such a native raw frame, I expect the coordinates to be relative to exactly the frame dimensions and orientation, not the preview/screen size or orientation.

This is especially relevant when I want to draw back to the Frame (with Skia Frame Processors), because then the coordinate system will be the frame's .width and .height.

I am talking about these two lines: https://github.com/nonam4/react-native-vision-camera-face-detector/blob/22c87fe4af1fd44fdff7bf729dd9bebe41da176e/ios/VisionCameraFaceDetector.swift#L245-L246

Currently when drawing a blur mask over the face using Skia Frame Processors, it looks like this:

before.mp4

After I change scaleX and scaleY to 1.0, it will look correct:

RPReplay_Final1713799605.mp4

Can you remove scaleX/scaleY from your code, and maybe provide such converters on the JS side using the Dimensions.get('window') APIs from React Native? I think that makes more sense, especially now that people use Skia Frame Processors.

[luicfrr]

@mrousavy can you please check v1.5 branch? I added a condition auto scalling prop on it so you can handle scaling on js side.

I'm still testing this branch before merging but I'm having a problem on IOS that makes my frame processor freeze after a few detections.

I think this problem is related to this issue but I'm still working on it.

[mrousavy]

Did you try with latest V4? I am currently running it and everything's working great

[mrousavy]

I just created a demo app; check out https://github.com/mrousavy/FaceBlurApp

You can reproduce the issue there. With scaleX/scaleY set to 1.0, it all looks correct. With your values, it seems to be a bit off.

[luicfrr]

Did you try with latest V4? I am currently running it and everything's working great

it was just me messing with frame 😅

[luicfrr]

@mrousavy In the new version 1.6.0 you now have the option to control whether the plugin handles scaling or not 🚀