Unable to get camera frame on macbook

[bokutotu]

Thank you for creating such a great repository!
I have a issue when using Mabbook pro 14inc (M1 max).
Below are the steps to reproduce the error.

Ventura 13.1
machine MacBook Pro 14-inch 2021 Apple M1 Max
rust version 1.66.0
Cargo.toml

name = "show_image"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
show-image="0.13.1"
image = { version="0.24.5" }
nokhwa = { version = "0.10.3", features=["input-native", "output-threaded"] }

Code (main.rs)

use nokhwa::{
    nokhwa_initialize,
    pixel_format::{RgbAFormat, RgbFormat},
    query,
    utils::{ApiBackend, RequestedFormat, RequestedFormatType},
    threaded::CallbackCamera
};

fn main() {
    // only needs to be run on OSX
    nokhwa_initialize(|granted| {
        println!("User said {}", granted);
    });
    let cameras = query(ApiBackend::Auto).unwrap();
    cameras.iter().for_each(|cam| println!("{:?}", cam));

    let format = RequestedFormat::new::<RgbFormat>(RequestedFormatType::AbsoluteHighestFrameRate);

    let first_camera = cameras.first().unwrap();

    let mut threaded = CallbackCamera::new(first_camera.index().clone(), format, |buffer| {
        let image = buffer.decode_image::<RgbAFormat>().unwrap();
        println!("{}x{} {}", image.width(), image.height(), image.len());
    })
    .unwrap();
    threaded.open_stream().unwrap();
    #[allow(clippy::empty_loop)] // keep it running
    loop {
        let frame = threaded.poll_frame().unwrap();
        let image = frame.decode_image::<RgbAFormat>().unwrap();
        println!(
            "{}x{} {} naripoggers",
            image.width(),
            image.height(),
            image.len()
        );
    }
}

Error

User said true
CameraInfo { human_name: "FaceTime HD Camera", description: "Apple Inc.: FaceTime HD Camera - AVCaptureDeviceTypeBuiltInWideAngleCamera, Unspecified f0", misc: "47B4B64B-7067-4B9C-AD2B-AE273A71F4B5", index: Index(0) }
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ProcessFrameError { src: NV12, destination: "RGB", error: "bad input buffer size" }', src/main.rs:63:56
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: ProcessFrameError { src: NV12, destination: "RGB", error: "bad input buffer size" }', src/main.rs:55:57

If not already resolved, I would like to contribute.

[bookshiyi]

#72 (comment)
Source frame format NV12 may not be supported on macintosh untill now, I guess.

[l1npengtul]

It is likely a fault of the library improperly setting FourCC

[hallucinogen]

I found that the first problem is the NV12 checker in this line:

nokhwa/nokhwa-core/src/types.rs

Line 1445 in ad93e0a

if data.len() != ((resolution.width() * resolution.height() * 3) / 2) as usize {

if data.len() != ((resolution.width() * resolution.height() * 3) / 2) as usize {

I tried to change it into

if data.len() != (resolution.width() * resolution.height() * 2) as usize {

Now it passes the decoding round.

But the resulting decoded image is faulty. It looks like it's split in the middle, and then we put the left part on the right and the right part on the left. I feel I am close to a solution, but can't really point it out

[vaqxai]

Does not work on Macbook Air 2014 13". Same error.

[stolinski]

Anyone have a work around for this? All cams in macOS are coming in as NV12 and hitting the same bad input buffer size issue.

[anselanza]

Damn, I just hit this problem, too.

Until NV12 support comes in v0.11, it's possible to use a crate like DCV Color Primitives to do the necessary conversion.

[bluezheng]

Hi, is there any workaround for this issue?

[marcpabst]

Hi, I'm seeing this issue as well. I think the problem is that AVfoundation returns frames in UYVY format (https://wiki.videolan.org/YUV#UYVY), even if you tell nokhwa to request NV12 (or some other format).

Here is my current workaround to deal with this kind of data (note that this only works on Rust nightly). Alternativly, you could split the interleaved planes and use the aforementioned dcv-color-primitives crate for conversion.

#![feature(portable_simd)]
use std::simd::SimdFloat;
use std::simd::f32x4;
use rayon::prelude::*;

#[inline]
pub fn uyvy_to_rgb24(in_buf: &[u8], out_buf: &mut [u8]) {
    debug_assert!(out_buf.len() as f32 == in_buf.len() as f32 * 1.5);

    in_buf
        .par_chunks_exact(4) // FIXME: use par_array_chunks() when stabalized (https://github.com/rayon-rs/rayon/pull/789)
        .zip(out_buf.par_chunks_exact_mut(6))
        .for_each(|(ch, out)| {
            let y1 = ch[1];
            let y2 = ch[3];
            let cb = ch[0];
            let cr = ch[2];

            let (r, g, b) = ycbcr_to_rgb(y1, cb, cr);

            out[0] = r;
            out[1] = g;
            out[2] = b;

            let (r, g, b) = ycbcr_to_rgb(y2, cb, cr);

            out[3] = r;
            out[4] = g;
            out[5] = b;
        });
}

// COLOR CONVERSION: https://stackoverflow.com/questions/28079010/rgb-to-ycbcr-using-simd-vectors-lose-some-data

#[inline]
fn ycbcr_to_rgb(y: u8, cb: u8, cr: u8) -> (u8, u8, u8) {
    let ycbcr = f32x4::from_array([y as f32, cb as f32 - 128.0f32, cr as f32 - 128.0f32, 0.0]);

    // rec 709: https://mymusing.co/bt-709-yuv-to-rgb-conversion-color/
    let r = (ycbcr * f32x4::from_array([1.0, 0.00000, 1.5748, 0.0])).reduce_sum();
    let g = (ycbcr * f32x4::from_array([1.0, -0.187324, -0.468124, 0.0])).reduce_sum();
    let b = (ycbcr * f32x4::from_array([1.0, 1.8556, 0.00000, 0.0])).reduce_sum();

    (clamp(r), clamp(g), clamp(b))
}

#[inline]
fn clamp(val: f32) -> u8 {
    if val < 0.0 {
        0
    } else if val > 255.0 {
        255
    } else {
        val.round() as u8
    }
}

(adapted from here: https://gist.github.com/arifd/ea820ec97265a023e67a88b66955855d)

As a sidenote, it looks like recent MacOS versions do not support transparent raw access to the camera outputs anymore, i.e. you will always be offered uyvy422, yuyv422, nv12, 0rgb, and bgr0, regardless of what the camera actually supports. MJPEG streams will still work when requesting a resolution/fps mode only supported through MJPEG (as is common for higher resolutions on UVC cameras , but AVfoundation will handle the conversion to one of the listed pixel formats internally.

@l1npengtul, not sure what the current status is here (I'm a bit confused with the different branches) but I might be able to help you with debugging this.

Edit: Forgot to say that this is on a M2 MacBook Pro on Ventura 13.4.

[yamt]

Hi, I'm seeing this issue as well. I think the problem is that AVfoundation returns frames in UYVY format (https://wiki.videolan.org/YUV#UYVY), even if you tell nokhwa to request NV12 (or some other format).

see #151