Virtual Backgrounds

[ck-kc]

I would like webcamoid to have a Virtual Background feature (like in Zoom) that works without a green screen. Would it be difficult to implement in webcamoid?

[ackbar3]

I'm exactly searching for something like that.

[hipersayanX]

A lot of people ask for this feature.

Would it be difficult to implement in webcamoid?

Not really, I'll do it when had some time.

[egeland]

With a lot more people WFH due to COVID-19, I'm sure this feature would be very much appreciated! 😀

[doitux]

That would be really great. So i could stop using Zoom!

[fangfufu]

@hipersayanX, is there any ways for us to donate to you, for this specific plugin?

[hipersayanX]

@fangfufu #267

[beppe9000]

see also this for background manipulation without relying on chroma key
https://elder.dev/posts/open-source-virtual-background/

[utak3r]

I'm also going to implement the "multi-chromakey", the key built from many color samples, using ramps and so on. It can give really good results - not as great as using neural networks, but faster. I was already doing this for huge commercial projects. It's just... have to find time for this ;)

[beppe9000]

great!

[fangfufu]

@beppe9000, the blogpost you linked contains some mistakes. I fixed them, and created a turnkey solution. It should be fairly easy to follow.

https://github.com/fangfufu/Linux-Fake-Background-Webcam
Edit: updated repository link.

[beppe9000]

nice!

[hipersayanX]

@beppe9000 @fangfufu

Damn, is that much complex to do that? as complex as using Tensorflow for that?
I was thinking on something much simpler, adapting the code from ColorReplace, then ask the user to get out of the screen for a second, take a snapshot of the background, and use that reference to replace the background.
Anyway if there are more documentation and examples for this, post it.

[beppe9000]

@beppe9000 @fangfufu

Damn, is that much complex to do that? as complex as using Tensorflow for that?
I was thinking on something much simpler, adapting the code from ColorReplace, then ask the user to get out of the screen for a second, take a snapshot of the background, and use that reference to replace the background.
Anyway if there are more documentation and examples for this, post it.

i always wondered if such a method can react to changing light conditions or camera noise
I think an alternate approach could be using the openCV motion tracking with a very long memory level to account for when you stop moving

[hipersayanX]

@beppe9000 Camera noise can be removed with a light de-noise filter and light changes could be fixed normalizing the luma channel, but I'm talking based on my knowledge, I could be wrong.

[beppe9000]

i hope to try it soon or later then

[fangfufu]

@hipersayanX, you definitely don't need a convolutional neural network for that. The network in my example can segment out human without learning the background. Not learning the background is not a requirement.

I only tidied up his blog post because I can't be bothered to write my own code, and I want something working quickly.

[fangfufu]

@hipersayanX, you probably want to use a segmentation approach rather than colour replace, because the skin's RGB value can be quite close to the background RGB value, in some lighting condition. Certain parts of the skin maybe replaced.

[fangfufu]

@hipersayanX, finally don't even think about using the bodypix network, because the model is only available in nodejs format...

So in the code I gathered, it does this:
Webcam -> Python -> HTTP-> NodeJS->Tensorflow->Mask.

And yes, it spins up two Docker containers, and they communicate with each other via HTTP...

It is really stupid.

[hipersayanX]

@fangfufu

Webcam -> Python -> HTTP-> NodeJS->Tensorflow->Mask.

Damn, and they forgot to include Ruby on Rails, Angular, React and JQuery on that list 🤣

[beppe9000]

where is the obligatory mongo database ?!

[fangfufu]

@hipersayanX

Damn, and they forgot to include Ruby on Rails, Angular, React and JQuery on that list

It was not "they", it was a one-man job, done by a Google engineer... He even had time to add a hologram effect to the foreground, but he didn't have time to type up the correct command to spin up the Docker containers...

I think it says a LOT about Google as a company.

[fangfufu]

https://www.reddit.com/r/computervision/comments/g043z9/could_anyone_suggest_a_image_segmentation/

https://www.reddit.com/r/MachineLearning/comments/g04daq/d_could_anyone_suggest_a_image_segmentation/

People on Reddit are recommending CNNs... It does make me wonder if people in academia are out of their mind... CNNs are so impractical for a little webcam project...

[beppe9000]

they like their nets :P

[RazZziel]

Apparently it's possible to replicate Benjamin Elder's example in good old C++ and TensorFlow: https://github.com/floe/deepbacksub

[fangfufu]

I think it would be quite a nightmare if webcamnoid suddenly depend on tensorflow-lite to build. I have had difficulty with getting his code working,

Bodypix does not seem to be available in Python (tensorflow/tfjs#2043).

[beppe9000]

wouln't make sense to have that implementation as a separate plugin?

[fangfufu]

I reckon that's what needs to be done. It doesn't make sense to make webcamoid to depend on Tensorflow to build. Or at least set it as an optional plugin.

[spstarr]

I'm interested in this also, so far fake-chroma-key has been fastest, for my poor Haswell mobile CPU.

[cont1nuity]

@FC76 Here is the code for the background blur, it was added in the daily build, it works in all platforms.

Quick question: it appears that the blur is not done if no face is detected, making the whole background visible. Is this intended?
According to the comment I thought that the whole feed would be blurred in this case.

Edit: could this be because of the if (vecFaces.isEmpty()) akSend(packet) right before? I have not dug around much and my c/c++ is a bit rusty...

[hipersayanX]

@cont1nuity can you open a new bug report for that bug, please?

[cont1nuity]

@cont1nuity can you open a new bug report for that bug, please?

Thanks for the reply. Done in #416.

[hipersayanX]

@cont1nuity thanks 😃

[cont1nuity]

I had some time to mess around and created a PR. Reviews and tests would be very welcome!

[iBenMoore]

FYI, someone just started on an OBS plugin now too that's progressing nicely, might be a good opportunity for some cross knowledge/code sharing.

https://github.com/royshil/obs-backgroundremoval

[QuentinPi]

Maybe another source of inspiration :
https://jitsi.org/blog/march-update-new-toolbar-ui-virtual-backgrounds-and-more/
jitsi/jitsi-meet#8750

[fangfufu]

@hipersayanX , is there a guide on how to write plugins for Webcamnoid?

@cont1nuity , Mediapipe now supports selfie segmentation: https://github.com/google/mediapipe/releases/tag/v0.8.5, more information is available at https://google.github.io/mediapipe/solutions/selfie_segmentation, however it doesn't tell you how to do it in C++. I have done an implementation in Python, it works quite well: https://github.com/fangfufu/Linux-Fake-Background-Webcam. I don't know if you are interested in writing a new plugin for Webcamnoid.

[hipersayanX]

@fangfufu I've not published any official documentation for writing plugins because I'm not sure of keeping the interface as it, but there are some very simple plugins that can be used as references for writing other plugins.
Invert is the simplest one that just receives the frames, process it, and send it to the next processing step.
And Scale is the simplest ones if you want to add controls to your plugin.

[kno10]

https://git.jami.net/savoirfairelinux/jami-plugins/-/tree/master/GreenScreen
This is a plugin for the open-source Jami SIP/H.323 software. It apparently uses a pretrained neural network to segment foreground and background to allow blurring or replacing the background.
It did not work for me, probably because I do not have a Nvidia GPU.

[hipersayanX]

@kno10 I will check but, a solution that depends on a privative software or hardware is never a solution, if the algorithm requires hardware acceleration I would prefer it to depend on OpenCL instead.

[kno10]

I don't know if it can also use OpenCL. There is also some Android loader for the ONNX model. All I only know is that the plugin did not work for me.

Jitsi also has prebuilt tensorflow models for background removal:
https://github.com/jitsi/jitsi-meet/tree/bb8c30a6c9b41f92cec91a4779a1484c91420e7e/react/features/stream-effects/virtual-background/vendor/models
it works in firefox for me, but feels a bit slow.

[Hirnfiedler]

Looking for virtual backgrounds, I've just installed WebCamoid. Even if the effect 'virtual background' is still missing, I like this tool very much. Thanx a lot for your work.

Please, let me vote for this issue.

[Svetlana-T]

Hello, any implementations for this currently available please? I checked the linked obs plugin and it is rather rudimentary, it is oval shaped and it does not pick up the shape of the head. Interested in keeping my existing background and just blurring it.

[fangfufu]

@Svetlana-T this is what I have. It works for me. https://github.com/fangfufu/Linux-Fake-Background-Webcam

[AndKe]

just visiting to check for updates on this important topic.

[hipersayanX]

@AndKe nope, I have other priorities and other bugs to solve, and no one else beside me is working in the project.

[hipersayanX]

I've added the ColorKey plugin, not the exact same feature as wanted by op but it's at least least something.
It seems that automatic background removal wold require AI usage, but I don't like the idea of depending on a gigantic monster like TensorFlow just that feature, plus Android does not have (or at least I don't know if available) a binary repository for 3rd-party libraries (like brew.sh or MSYS), making Android development a lot more complicated.
I would prefer to study the segmentation algorithm in depth and implementing it my self.

[Svetlana-T]

Hi, thanks. Is the ColorKey plugin documented somewhere?

[hipersayanX]

@Svetlana-T No, just launch Webcamoid daily build and the plugin is there in the effect list.