❓ Apply filter to photos

[Angelk90]

Question

Hi @mrousavy , I had a look at the documentation, I was wondering if using the following library together with FrameProcessor, it would be possible to be able to do something similar as seen in the images.

In short, in real time you can apply filters to the photos/videos you make, doing everything by js, not touching the android/swift code.

What I tried

No response

VisionCamera Version

2.11.2

Additional information

• I am using Expo
• I have read the Troubleshooting Guide
• I agree to follow this project's Code of Conduct
• I searched for similar questions in the issues page as well as in the discussions page and found none.

[mrousavy]

Hey! This seems very similar as #526.

In theory; it is possible to build such an interface, but that requires a lot of my free time, which I currently don't have. I'm running a company.
If you, or anyone else reading this comment, is interested in this becoming reality, consider a collaboration with me and my app development/consulting agency "Margelo": Contact me at me@mrousavy.com

[mrousavy]

Actually a hacky solution right now would be to use Frame Processors to detect the faces, and then draw the face-filters ontop of the Camera using react-native-skia. This should be very performant and happen in realtime, and you can then just take a screenshot of the view (see this lib) to "capture" it.

A real solution would be to offer an intermediate layer to draw upon, whether it's 3D AR, or 2D overlays, which is also part of the capture pipeline (photo & video). Again, I don't have the free time to work on this, but I have some ideas how that can be implemented in C++ for a fast and easy way of drawing 3D/2D filters straight from JS.

[mrousavy]

cc @wcandillon & @chrfalch

[wcandillon]

let's do it 🙌🏼

[Angelk90]

What do you think about opening a discussion about a possible implementation?

@mrousavy:
So we can discuss any ideas that come to mind.

Actually a hacky solution right now would be to use Frame Processors to detect the faces, and then draw the face-filters ontop of the Camera using react-native-skia. This should be very performant and happen in realtime, and you can then just take a screenshot of the view (see this lib) to "capture" it.

Hacky Solution:
I too had thought of a solution like this.
But doing such a thing is not enough: https://mrousavy.com/react-native-vision-camera/docs/guides/frame-processors#interacting-with-frame-processors

Using frame-processors instead of drawing a rectangle, put an image on top.
What should react-native-skia do in this case?

The problem with this is how to take the photo with the filter together, using the screenshot method is not that convincing me.

We think of different photo sizes or different ratios.

It is an excellent solution only that it does not completely convince me.

A real solution would be to offer an intermediate layer to draw upon, whether it's 3D AR, or 2D overlays, which is also part of the capture pipeline (photo & video). Again, I don't have the free time to work on this, but I have some ideas how that can be implemented in C++ for a fast and easy way of drawing 3D/2D filters straight from JS.

I would be very curious to hear all your proposals, I cannot tell you that I will be able to contribute to the development, I do not know if I still have the skills.

@wcandillon : I found the following project I don't know if it can help you as inspiration, there are some masks you could also use: Github

[chrfalch]

Skia will support off-screen drawing very soon - and it should be possible to take the bytes from the current frame, create a new surface, draw on the surface and then return the bytes back to the frame processor. Would this be a solution we could work with? Maybe I can discuss this with @mrousavy and see if we can find a way to get this working?

[Angelk90]

@mrousavy : Frame Processor to detect faces, does it recognize even if there are multiple faces?

For example, if there are three or more people in a shot, do you recognize them all?

[mrousavy]

1. Frame Processors detect the face, and return the bounding box.
2. RN Skia draws a face filter at the bounding box coordinates, smoothly and at 60 FPS, and running on the Frame Processor Thread, ontop of the Camera View
3. You then take a "screenshot" of the view.

it does not completely convince me

@Angelk90 Well of course, I said it's a hacky solution right now.

Skia will support off-screen drawing very soon

@chrfalch that's exactly what's needed for this. Let's discuss this in DMs

does it recognize even if there are multiple faces?

@Angelk90 I am not sure if you understand Frame Processors correctly. It is up to you to implement the face detection, and there are tons of implementations for that out there. Frame Processors is an API that allows you to easily create those plugins and use them from JS, with VisionCamera. So if e.g. MLKit allows you to detect multiple faces, then use that.

[jsindos]

I'm also interesting in implementing a "Green Screen" effect, I think the method will be different in that we will output a mask from the ML model and subtract the rest of the image. I did a tiny bit of research into RN Skia but couldn't find a "subtract" feature, do you guys think this is possible with current RN packages?

Green screen effect:

[Angelk90]

@jsindos : What did you use to make this "Green screen effect"?
Java(android)?

[mrousavy]

Hey yes I think this is possible using RN Skia - there's a prop (called draw mode or something) which you can set to SRC_OUT or DST_OUT and it will only draw where the mask is transparent (cc @chrfalch)

[jsindos]

Ok cool thanks @mrousavy.

Basically my plan is to use the PyTorch implementation of https://github.com/PeterL1n/RobustVideoMatting (not 100% sure if this will be performant on mobile phones yet), and save a TorchScript copy of the model.

I'll then write the Objective-C equivalent code of:

bgr = torch.tensor([.47, 1, .6]).view(3, 1, 1).cuda()  # Green background.

for src in YOUR_VIDEO:  # src can be [B, C, H, W] (Batch, Channel, Height, Width) or [B, T, C, H, W]
    fgr, pha, *rec = model(src, *rec, downsample_ratio=0.25)
    writer.write(fgr * pha + bgr * (1 - pha)) # writes to the video, this will have to be moved to RN Skia?

where fgr, pha are the foreground and alpha predictions.

And package this up as a Frame Processor Plugin (not 100% sure how to do this).

[mrousavy]

Yep that should work! I'd recommend creating this in your private project first (just a single file per platform), then extracting it into a public package once it works for you. (See community plugins for reference)

This is maybe an interesting use-case @raedle? 😄

[raedle]

@mrousavy could be interesting, yes. It would require video support, which PyTorch Live currently doesn't have, but something we'd like to add in the future :)

[ridvanaltun]

It's not very relevant, but I have developed a library where you can use AR filters like beauty filters and face filters for React-Native, I hope it will be useful to someone: https://github.com/ridvanaltun/react-native-deepar

[bi885895]

Hi my friends by code,are there any news and someone know how to do it?

[thomas-coldwell]

Hey everyone! I'm working on bringing this to iOS in #1198 and have a plan for an intermediate drawing layer for Android using OpenGL too 🙌

[shyammanek]

i want to add filters like snapchat and instagram and reels on face filters like animals faces or etc...
anyonce can guide me @Angelk90 @raedle @mrousavy .

[mrousavy]

Hey everyone - this is a pretty old issue but I've actually finished the implementation for this using RN Skia, but I decided to not merge it into VisionCamera V3 since that overcomplicated the codebase by a lot and made VisionCamera heavier for everyone (including the people who don't use the Skia features).

So I'm preparing to release this as a separate package or fork soon.

[Angelk90]

@mrousavy : Wouldn't it be possible to make it possible to include it as an external lib?

<Camera
      style={StyleSheet.absoluteFill}
      device={device}
      isActive={true}
      filter={filter}  // Here
    />

[1337mus]

@mrousavy Any updates on the fork? Looking to try it out if there is a repo available. Thanks for everything you do.

[disleem1337]

@mrousavy Any updates on the fork? I also need that feature

[mrousavy]

no not yet

[kamilafsar]

@mrousavy It seems like a combination of rodgomesc/vision-camera-face-detector and something like Skia would be the way to go, right? What would be the added benefit of #1198? Do we still need it for such an application?

[mrousavy]

See #2727 👀

[Angelk90]

@mrousavy : You can create a usage example?

[mrousavy]

there is one in the PR.

[mrousavy]

VisionCamera V4 now supports Skia Frame Processors!!!

[mrousavy]

If you appreciate VisionCamera, please consider 💖 sponsoring me on GitHub 💖 to support the development of VisionCamera and thank me for my time spent on fixing bugs and building new features.

Skia Integration was insanely difficult to build.