CameraView frame processing support

[bijington]

Following on from a request from a community member (dotMorten) I have started a discussion on the idea of introducing barcode reading functionality to the toolkit.

I have performed some research to discover the possibility of using platform specific APIs to read the contents from an image. The ideal scenario will be that we could use this feature in combination with the CameraView but neither should depend upon each other.

Please note this is a work in progress while I build up my understanding of what is possible. If you have stumbled upon this and have experience with any part, please feel free to provide input.

This discussion/proposal will now take on multiple sections/increments. I have included them all into a single discussion for now but we will want to separate out into multiple PRs at the very least.

1. Expose platform specific access to captured frames

The aim will be to enhance the CameraManager implementations to expose the ability to make use of platform specific interaction with the platform supplied image data.

Android

The CameraX API provides the UseCase abstract class which developers can make use of in order to perform a camera preview, image capture or even image analysis.

I would like to propose that we introduce the following to the CameraManager.android implementation:

readonly IList<UseCase> additionalUseCases = [];
	
public void AddUseCase(UseCase useCase)
{
    additionalUseCases.Add(useCase);
}

public void RemoveUseCase(UseCase useCase)
{
    additionalUseCases.Remove(useCase);
}

// Current implementation
protected virtual async partial Task PlatformStartCameraPreview(CancellationToken token)
{
    camera = processCameraProvider.BindToLifecycle(
        owner, 
        cameraSelector, 
        [
            .. additionalUseCases,
            cameraPreview,
            imageCapture
        ]);
}

iOS/macOS

Apple provides us with the AVCaptureOutput abstract base class that developers can use to direct inputs from audio/video sources to.

I would like to propose that we introduce the following to the CameraManager.macios implementation:

readonly IList<AVCaptureOutput> additionalCaptureOutputs = [];

public void AddOutput(AVCaptureOutput captureOutput)
{
    additionalCaptureOutputs.Add(captureOutput);
}

public void RemoveOutput(AVCaptureOutput captureOutput)
{
    additionalCaptureOutputs.Remove(captureOutput);
}

// Current implemementation
protected virtual async partial Task PlatformStartCameraPreview(CancellationToken token)
{
    foreach (var output in additionalCaptureOutputs)
    {
        captureSession.AddOutput(output);
    }
}

2. Consume platform specific frames

In truth, once part 1 is complete developers could implement this section themselves in their own applications and I fully expect them to for more advanced scenarios. I would like to look at use providing some level of barcode scanning support which may need to be a separate package due to further dependencies - I am also happy to try and build a PoC outside of the toolkit to prove this works.

The aim will be that the following details will be encapsulated through a UseBarcodeScanning extension method or something similar. Happy for suggestions here as this is still a little unclear. Perhaps we have a BarcodeCameraView in the new package?

Android

https://developers.google.com/ml-kit/vision/barcode-scanning/android

This API requires Android API level 21 or above. Make sure that your app's build file uses a minSdkVersion value of 21 or higher.

Making use of the above details we could look to implement something like:

Note this is currently in Kotlin and lifted from the google docs

val imageAnalysis = ImageAnalysis.Builder()
    // enable the following line if RGBA output is needed.
    // .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
    .setTargetResolution(Size(1280, 720))
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .build()
imageAnalysis.setAnalyzer(executor, ImageAnalysis.Analyzer { imageProxy ->
    val rotationDegrees = imageProxy.imageInfo.rotationDegrees
    // insert your code here.
    ...
    // after done, release the ImageProxy object
    imageProxy.close()
})

cameraProvider.bindToLifecycle(this as LifecycleOwner, cameraSelector, imageAnalysis, preview)

Then we could make use of the CameraManager changes from section 1 and write something like:

var imageAnalysis = ...;

cameraManager.AddUseCase(imageAnalysis);

iOS/macOS

Apple provides the ability to analyze an image.

Analyze

https://developer.apple.com/documentation/visionkit/imageanalyzer/analyze(_:configuration:)

machineReadableCode

https://developer.apple.com/documentation/visionkit/imageanalyzer/analysistypes/machinereadablecode

This would then potentially allow developers to write their own consumer of an output as follows:

class OutputAnalyzingDelegate : AVCaptureVideoDataOutputSampleBufferDelegate
{
    readonly VisionKit.ImageAnalyzer imageAnalyzer = new();
		
    public override void DidOutputSampleBuffer(AVCaptureOutput captureOutput, CMSampleBuffer sampleBuffer, AVCaptureConnection connection)
    {
        base.DidOutputSampleBuffer(captureOutput, sampleBuffer, connection);

        var buffer = sampleBuffer.GetImageBuffer();

        if (buffer is not null)
        {
            imageAnalyzer.Analyze(buffer, CGImagePropertyOrientation.Up, null);
        }
    }
}

Note at the time of writing the ImageAnalyzer bindings are not available. At least not in a plain MAUI app.

Then we could make use of the CameraManager changes from section 1 and write something like:

var videoOutput = new AVCaptureVideoDataOutput();

videoOutput.AlwaysDiscardsLateVideoFrames = true;
videoOutput.SetSampleBufferDelegate(new OutputAnalyzingDelegate(), new DispatchQueue("abc"));

cameraManager.AddOutput(videoOutput);

Tizen

TBC

Windows

TBC

3. Platform agnostic frame access

This is for the scenarios when performance is less critical. We could look to expose a variety of options:

1. Make use of Vlads great work and rely on timed image capture
2. Expose a FrameReceived event/callback on the CameraManager that will return a byte[] or ArrayPool<byte> that the developer can work with.

Concerns

The Android implementation appears to rely on ML Kit and introduces the potential complexity around shipping with that dependency. The Android docs do provide the following details: https://developers.google.com/ml-kit/tips/installation-paths

Questions

Do we need to introduce this given that ZXing.NET already exists?

[dotMorten]

Here's the Windows version:

using Windows.Devices.PointOfService;
//..
  private async Task StartScanAsync()
  {
      BarcodeScanner scanner = await BarcodeScanner.GetDefaultAsync();
      ClaimedBarcodeScanner claimedScanner = await scanner.ClaimScannerAsync();
      claimedScanner.DataReceived += claimedScanner_DataReceived;
      claimedScanner.IsDecodeDataEnabled = true;
      await claimedScanner.EnableAsync();
      if (!string.IsNullOrEmpty(scanner.VideoDeviceId)) // Camera based
      {
          await claimedScanner.StartSoftwareTriggerAsync();
      }
  }
  private void claimedScanner_DataReceived(ClaimedBarcodeScanner sender, BarcodeScannerDataReceivedEventArgs args)
  {
      var data = Windows.Security.Cryptography.CryptographicBuffer.ConvertBinaryToString(Windows.Security.Cryptography.BinaryStringEncoding.Utf8, args.Report.ScanDataLabel);
  }

Concerns:

WinUI doesn't have CaptureElement implemented (see microsoft/microsoft-ui-xaml#8214), so it's not currently possible to create a live preview of what the camera sees if it is a camera-based scanner, albeit I've seen some hacks using the MediaPlayerElement. This makes it a little trickier to get the scan to work correctly, since you can't tell if you're holding the barcode too close or too far away.

[knocte]

Do we need to introduce this given that ZXing.NET already exists?

I guess you meant https://github.com/Redth/ZXing.Net.MAUI ?
But my answer is: yes please. I think ZXing.Net.{Mobile|Maui} have two major problems:

• A very unresponsive maintainer that takes ages to review PRs (let alone bugs/issues) and make new releases.
• An implementation that is kinda hit or miss, and I'm sure a truly native Android/iOS implementation (as in, calling the underlying platform APIs to scan) like the one proposed here would be more robust.

[knocte]

asked it in the past

Asked what? What do you mean with "it"?

[baaaaif]

To include the ZXing.Net.Maui code into the Community Toolkit

[knocte]

No one has suggested such a thing here. Please read the original issue, where it's proposed to have QR code scanning, but without the use of ZXing.Net; but rather with the help of APIs of each platform.

[shugaoye]

Another project https://github.com/hjam40/Camera.MAUI is better than ZXing.Net.MAUI.

[bijington]

Thanks. For the record, this discussion might be a little confusing based on the title. The aim of this will be to support a multitude of image analysis functions. I hope to expose an API surface that will allow us or developers to handle barcode detection but also make use of things like VisionKit and perform more advanced analysis and detection on it.

The description has been update to reflect some of this

[CliffAgius]

There is a version from @VladislavAntonyuk that could work Here

This works on all the platforms what I am not sure of yet as I have not tested is if it just works with QR codes or now. Now we have CameraView this could a be a fairly simple and great add to MCT.

[bijington]

I have come to realise that our CameraManager is currently internal and making it public and refactoring to perform my original proposal may be more work that we should commit to. That being said I think I have come up with an elegant design which I would love some feedback on.

Firstly we will expose a new concept called CameraScenario (open to a better name). This will be the base class for developer driven scenarios that will interact with the camera APIs. Think of this a little like Behavior:

public abstract class CameraScenario
{
	public bool IsInitialized { get; protected set; }

	public virtual void Initialize()
	{
		IsInitialized = true;
	}
}

We could then define something like a ZXingBarcodeDetectionScenario as follows:

public class ZXingBarcodeDetectionScenario : CameraScenario
{
	// Add in the magic here.
}

That all relies on a shared code layer which may not perform as well as the next part.

We implement a PlatformCameraScenario which we could think of like PlatformBehavior:

public abstract partial class PlatformCameraScenario : CameraScenario
{
}

Where the MacCatalyst/iOS base implementation currently looks like:

partial class PlatformCameraScenario : CameraScenario
{
	readonly Action<AVCaptureOutput> onInitialize;

	public PlatformCameraScenario(AVCaptureOutput output, Action<AVCaptureOutput> onInitialize)
	{
		Output = output;
		this.onInitialize = onInitialize;
	}

	public AVCaptureOutput Output { get; }

	public override void Initialize()
	{
		onInitialize.Invoke(Output);
		IsInitialized = true;
	}
}

Then we could implement a platform specific implementation such as:

/// <summary>
/// Apple based implementation
/// </summary>
public class PlatformBarcodeScanningScenario : PlatformCameraScenario
{
#if IOS || MACCATALYST
	public PlatformBarcodeScanningScenario() : base(
		new AVCaptureMetadataOutput(),
		o =>
		{
			((AVCaptureMetadataOutput)o).SetDelegate(new BarcodeDetectionDelegate(), DispatchQueue.MainQueue);
			((AVCaptureMetadataOutput)o).MetadataObjectTypes =
				AVMetadataObjectType.QRCode | AVMetadataObjectType.EAN13Code;
		})
	{
		
	}
#endif
}

sealed class BarcodeDetectionDelegate : AVCaptureMetadataOutputObjectsDelegate
{
	public override void DidOutputMetadataObjects(
		AVCaptureMetadataOutput captureOutput,
		AVMetadataObject[] metadataObjects,
		AVCaptureConnection connection)
	{
		foreach (var metadataObject in metadataObjects)
		{
			if (metadataObject is AVMetadataMachineReadableCodeObject readableObject)
			{
				var code = readableObject.StringValue;
					
				Console.WriteLine($"Metadata object {code} at {string.Join(",", readableObject.Corners?? [])}");
			}
		}
	}
}

And now this where I think it gets really elegant, developers can simply add scenarios to a CameraView in order to use them:

<toolkit:CameraView>
    <toolkit:CameraView.Scenarios>
        <toolkit:PlatformBarcodeScanningScenario />

        <toolkit:ZXingBarcodeDetectionScenario />
    </toolkit:CameraView.Scenarios>
</toolkit:CameraView>

What are peoples thoughts?

[bijington]

Thanks. Yes the idea is to enable support for more than just barcode scanning. I want to make a PoC for that to make sure this model will support it

[VladislavAntonyuk]

Does scenario work only with Output (AVCaptureOutput)?
Can it be applied for input (I don’t know if it’s even exists)? For example some filters?

[bijington]

I don't see why we couldn't expand CameraScenario to work for other use cases like applying filters. I think AVCaptureInput really only deals with where the data is coming from rather than being able to apply things to the incoming data. The closest example I have stumbled upon is still using AVCaptureOutput:

https://gist.github.com/bgayman/6b27428ea48750e8306975c735bd517e

I can look further into this though 👍

[CrustyJew]

I think this could work fine, but what's the concern / difficulty with making the CameraManager public instead? Why is it internal in the first place?

[bijington]

I think this could work fine, but what's the concern / difficulty with making the CameraManager public instead? Why is it internal in the first place?

At the moment the lifetime of the CameraManager is managed by handlers based on the life of the CameraView. We could expose the CameraManager on the CameraView but that would make the proposed usage more verbose. If we were to allow developers to create CameraManagers and interact with them differently then that is a whole different approach which would need considering. I'm keen to avoid changes like that if they aren't needed.

That being said all of the proposed changes here do not make it more difficult to open up CameraManager in the future

[MartinRobins]

Would the scenario you are suggesting allow a developer to access the preview frames from the various cameras (if I am reading it correctly, I think the answer will be yes)?

Additionally, could the "scenarios" have the ability to feed back some kind of overlay to the image on screen? I am looking at the feasibility of implementing document edge detection as such a scenario. This may also include some kind of processing of the captured image (non-preview) so that it can be de-skewed and resized appropriately based upon the edges detected.

It also looks to me that this suggestion would address discussions #1953, #1998, #2116 & #2160 so clearly such an enhancement would be popular.