🐛 Android- frame processor alway return wrong width/height of frame

[vanhungoz]

What were you trying to do?

i try to use frame to recognize face

Reproduceable Code

const frameProcessor = useFrameProcessor(frame => {
    'worklet';
    console.log('frame.width', frame.width);
    console.log('frame.height', frame.height);
  }, []);

What happened instead?

frame data alway wrong.
camera's orientation is portrait
device's orientation is portrait.

Relevant log output

LOG  frame.width 1280
LOG  frame.height 720

Device

pixel 3L. Android 12

VisionCamera Version

2.12.0

Additional information

• I am using Expo
• I have read the Troubleshooting Guide
• I agree to follow this project's Code of Conduct
• I searched for similar issues in this repository and found none.

[vanhungoz]

frame preview is showing correctly. But frame data may be wrong
it is only happening on Android. iOS is working correctly

[vanhungoz]

it failed on both front and back camera.

[mrousavy]

Okay, maybe this commit broke something: 48da181

[vanhungoz]

@mrousavy thank you for your quick response.

i have checked "react-native-vision-camera": "2.11.1",
i still got the same error

[vanhungoz]

i checked release history, orientation is added from version 2.11.2

[mrousavy]

I mean technically it would be correct. The width is always the physical sensor width. But I agree that we can use the logical term width and height instead, just like on iOS

[mrousavy]

I just wonder where it goes wrong, here we set the target rotation:

react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView.kt

Lines 396 to 397 in 28fc6a6

	val imageAnalysisBuilder = ImageAnalysis.Builder()
	.setTargetRotation(outputRotation)

Could you maybe check here what image.width is?

react-native-vision-camera/android/src/main/java/com/mrousavy/camera/CameraView.kt

Lines 477 to 493 in 28fc6a6

	setAnalyzer(cameraExecutor, { image ->
	val now = System.currentTimeMillis()
	val intervalMs = (1.0 / actualFrameProcessorFps) * 1000.0
	if (now - lastFrameProcessorCall > intervalMs) {
	lastFrameProcessorCall = now
	val perfSample = frameProcessorPerformanceDataCollector.beginPerformanceSampleCollection()
	frameProcessorCallback(image)
	perfSample.endPerformanceSampleCollection()
	}
	image.close()
	if (isReadyForNewEvaluation) {
	// last evaluation was more than a second ago, evaluate again
	evaluateNewPerformanceSamples()
	}
	})

[vanhungoz]

ok, thank you, i'll check it and tell you

[vanhungoz]

@mrousavy it alway log 1280 as same as frame.width

[bkidai]

@vanhungoz @mrousavy isn't the wrong with/height It's because on Android the frame it's rotated by 90° degree (i'm having the same issue) even before the last commit #715

Btw, i have the same output when i use orientation="portrait"
console.log(frame.toString());
1280 x 720 Frame // output

[vanhungoz]

@bkidai have you tested with older version. Which version could we use to check frame processor with correct frame?

[vanhungoz]

@vanhungoz @mrousavy isn't the wrong with/height It's because on Android the frame it's rotated by 90° degree (i'm having the same issue) even before the last commit #715

Btw, i have the same output when i use orientation="portrait" console.log(frame.toString()); 1280 x 720 Frame // output

@bkidai have you found any workaround?

[bkidai]

@vanhungoz i rotate the image based on the ImageProxy.imageInfo.rotationDegrees natively in Android before using it

[vanhungoz]

@mrousavy do you have any suggest to find out this bug?

[vanhungoz]

matrix.setRotate((360- image.imageInfo.rotationDegrees).toFloat())
val rotatedBitmap = Bitmap.createBitmap(bitmap, 0, 0, bitmap.width, bitmap.height, matrix,true)

@bkidai could you share your code.
i'm rotating the image by this function

[bkidai]

@vanhungoz here's my code
Btw, it take a "long" time to rotate the images

matrix.postRotate(rotationDegrees.toFloat())
val rotatedBitmap = Bitmap.createBitmap(bitmap, 0, 0, image.width, image.height, matrix, true)

[vanhungoz]

thank you, the same as me

[xulihang]

If we rotated the image on the Android side, the frame.width is still wrong on the JavaScript side. Maybe frame should have a rotationDegrees prop.

[xulihang]

In order to show barcode overlays, I rotated the image on the Android side using the yuv2bitmap converting code from here: https://blog.minhazav.dev/how-to-convert-yuv-420-sp-android.media.Image-to-Bitmap-or-jpeg/#pure-java-approach

I then use Dimensions to infer whether I should switch frame width and frame height:

    if (frameWidth>frameHeight && Dimensions.get('window').width>Dimensions.get('window').height){
      viewBox = "0 0 "+frameWidth+" "+frameHeight;
    }else {
      console.log("Has rotation");
      viewBox = "0 0 "+frameHeight+" "+frameWidth;
    }    

Full example: https://github.com/xulihang/vision-camera-dynamsoft-barcode-reader/tree/main/example

[mrousavy]

Maybe this fixes the problem: #816

[xulihang]

The new setOutputImageRotationEnabled API used in the pr seems the correct way.

[yachaka]

I had the same issue
So I did

// If frame is inverted  
// This happens on Pixel 2, width and height are inverted
        if (screenWidth < screenHeight && frameWidth > frameHeight
            || ( screenWidth > screenHeight && frameWidth < frameHeight)) {
            const tmp = frameWidth;
            frameWidth = frameHeight;
            frameHeight = tmp;
        }

[mrousavy]

Hey! I've rewritten the entire Android codebase of VisionCamera from CameraX to Camera2 in the efforts of ✨ VisionCamera V3.

I just now completed the Camera2 rewrite and I believe the core structure is running, but there might be some edge cases to iron out. Can you try and test the PR #1674 for me to see if you can still reproduce this issue here?

Here's an instruction on how you can test that: #1674 (comment)

If the issue cannot be reproduced with that version/PR anymore, then hoorayy, I fixed it! 🎉
Otherwise please let me know and I'll keep this issue open to keep track of it.

Thank you!

[mrousavy]

Hey - I'm tracking Orientation in this feature request/issue now: #1891 ✨

Make sure to upvote or sponsor to support this feature, and leave a comment if you have any additional thoughts/ideas.