Use iterative-gaussian blur for background #142
Conversation
| @@ -285,7 +285,7 @@ def compose_frame(self, frame): | |||
|
|
|||
| if self.postprocess: | |||
| mask = cv2.dilate(mask, np.ones((5,5), np.uint8) , iterations=1) | |||
| mask = cv2.blur(mask.astype(float), (10,10)) | |||
| mask = cv2.medianBlur(mask, 5) | |||
There was a problem hiding this comment.
I'm not sure how much it really helps, but i also replaced the blur on the mask to be median, since it generally handles edges better (and my wardrobe consists almost entirely of dark shirts) - but even i admit i'm not sure exactly how much it helped. There's probably a placebo effect going on.
There was a problem hiding this comment.
To be honest I didn't review the blurring code when it was merged in. Originally this repository was just a packaged-up version of that Google engineer's blog post. It did not come with blurring.
I have recently taken a slightly better (stricter) approach to code review - I basically learnt this concept at work. I was a PhD student - we don't do code review.
requirements.txt
Outdated
| @@ -1,5 +1,5 @@ | |||
| numpy>=1.19.3 | |||
| opencv-python>=4.4.0.46 | |||
| pyfakewebcam>=0.1.0 | |||
| mediapipe==0.8.5 | |||
| mediapipe==0.8.6.2 | |||
There was a problem hiding this comment.
I don't - I've only used it a couple days, but so far i've been able to leave it on continuously without issue. I'm on Mint 19.3, and i had to compile v4l2 manually (not using the dkms from apt) to get this project to work, so it's possible it's just different setups.
There was a problem hiding this comment.
It seems to work for me so far. I am going to leave it running for a bit before deciding what to do.
…ng, and used median blur to feather mask
|
Uh woops, looks like i'd used my work git to commit this. Rewrote the commit with my personal details (so it doesn't look like it's an official thing 😅). Same code changes just different author in the commit. Fixed now, shouldn't be a problem. |
|
Your version of blur is very blocky. I ran your code with the following command: In contrast with master branch and the following line: Not showing my face, because I look really fat after covid lockdown, not showing more of my room, because it is really messy. lol. I note the 2-4 FPS increase on my i7-4900MQ. I am okay with the results, if the blockiness is what you intend to achieve. If the new version of mediapipe is faulty, someone will eventually complain and open up a new issue. |
Yeh, make sure you don't do that. Some companies have policies against contributing to personal open source project using company resources. lol. |
|
Anyways, I am happy to merge this, if you confirm that the blockiness is what you intend to achieve. |
|
The blockiness wasn't intentional, and is probably due to the downsampling, the 25% size i picked is pretty severe. My camera is probably just not high enough resolution to show it much. Moving it to 50% or 75% ( |
| width = int(frameWidth * .25) | ||
| height = int(frameHeight * .25) |
There was a problem hiding this comment.
These two lines - give them a shot at .5 or .75, see if it looks more natural on your camera. I'll adjust those upward in the review if they look better on your end.
There was a problem hiding this comment.
Hmm it doesn't really matter. I don't really use blurring myself. The whole point is to make the fine features of the background unrecognisable. Having 0.25 as a parameter achieves that goal and performance.
|
Median blur isn't quite the same as a gaussian blur. OpenCV does have a gaussian blur function that I've been playing around with and achieved some nice results, but it can increase computational load significantly if the kernel size is too large. I can send some screenshots shortly for reference. The median filter is an improvement on the box filter blur that |
|
Just to clarify, the gauss blur is further down. The median is only on the mask, whereas the gauss is on the actual processing of |
|
Hmm I have had a look at the OpenCV's documentation. I am not so sure about the approach taken by this PR. I don't see the theoretical reason behind running Gaussian blur on a downsided image multiple times, when you can just run Gaussian blur on the big image using a bigger kernel. The default blurring kernel is a mean kernel, which is a good approach to blurring. Also @snlawrence, I will take your word for it when you say Gaussian blur increases computational load significantly when the kernel size is too big. I don't know how it is implemented underneath. If it is implemented using convolution, then perhaps the shape of kernel does not matter, but the size of the kernel does. My point is that a Gaussian kernel and a mean kernel with the same size should take equally long to convolve. |
Speed is the main reason. Kernel size increases cost exponentially, whereas iterations increase it only linearly. Ideally one would use two different one-dimensional blurs for best results (since gauss is separable), but ocv doesn't have a convenience method for that (afaik).
Yeah, box blurs suffer from being unweighted, leading to larger kernels having that "banding" or "doubling" problem in the "before" screenshot i posted.
That's true, gauss does require more work than a box blur at equal kernel sizes. However, that's part of why downsampling first is usually done - if you're after a blurry result, downsampling first, blurring, then upsampling provides the same results at lower cost. |
|
I will merge this in, thanks. |
|
P.S. I changed the default blurring iteration from 25 to 5. |
Apologies, I didn't see that. With a gaussian blur you can also vary the value of sigma to change the radius of the blur in the kernal. I found that a value of 1/6 of the kernel width gave nice results. @fangfufu I agree with you, I don't understand why it increases the load so much when it's just a convolution with the same shaped kernel, however, I can do more rigorous testing on that to get a better understanding.
@Knetic, Have you looked at sepFilter2d or filter2D? I believe these achieve what you're after. I did try creating a gaussian kernel with getGausssianKernel and then applying that in both dimensions separately with filter2D, however, it didn't appear to have the same result as using GaussianBlur, so I didn't continue playing with it. |
Oh I would love to know, but don't feel obliged. I feel the software is performing quite well as it is. If I ever have to use Gaussian blur with OpenCV intensively, I would definitely try and figure it out myself. :) |
|
|
| background_frame = cv2.GaussianBlur(background_frame, | ||
| (i,i), | ||
| cv2.BORDER_DEFAULT) | ||
| background_frame = cv2.GaussianBlur(background_frame, | ||
| (i,i), | ||
| cv2.BORDER_DEFAULT) |
There was a problem hiding this comment.
In these function calls cv2.BORDER_DEFAULT is being passed as the sigmaX parameter, not the borderType.
There was a problem hiding this comment.
Yes, you are right, well spotted.
|
What's the FPS on your implementation? I prefer the image on the right. Feel free to open a pull request. However I would note that it is probably easier to make out the details on the image on the right. |
|
30 FPS I partially agree, it's slightly easier, but I don't think it's significantly easier. I suppose it comes down to what the purpose of the blurring is. I'd argue that you're not trying to completely obscure you're background, but rather just remove the finer-scale details so that people aren't distracted by the particulars of your background and are focused on you instead. If you want to totally obscure it you'd just use a background image instead. I'll raise a PR shortly. |
|
@snlawrence , we can just increase the sigma if we want to blur more, without increasing the kernel size. At the limit it tends to box filter, I believe. Please note that I reverted the merge of this pull request, because of the implementation error you spotted. @Knetic , sorry for the unmerging. I know this code base is probably riddle with problems. I am just trying to do a better job at quality control when pull requests come in. Thanks for your contribution though. 🙂 |
|
My implementation definitely suffers from my preference on having a stronger blur. The downsampling (meant for speed) is very visible with fewer iterations (i use 13, in the 'after' i posted originally), too low and it absolutely starts to look blockier. Those who just want a slight effect would probably be fine with the slight speed hit for a more versatile result. |
|
Oh I only did the unmerging because implementation errors. |
|
@snlawrence , please do modify the default blurring factor, as you see fit. 🙂 |
I liked this project quite a lot, but wanted to heavily blur my background. The box blur that's done in master branch leads to "banding", which is pretty common with heavy box blurs (since it's unweighted). It's also a bit slow at higher kernel sizes.
This change downsamples first, then runs an iterative gauss blur (using the number of iterations from
background_blurthat users pass in), then upscales with a linear area filter. It went from ~20fps with master to >30fps (my camera doesnt go faster than that) with this branch.Here's a before/after of what it looks like for me;
It's most noticeable on the black lamp stem, which appears to be almost two different stems with master branch, and the edge of the chair, which similarly gets "pulled out".
Finally to get it to build i had to jump through hoops getting the right python (only 3.7+ works, which is not what you get from
python3in mint/ubuntu), so that's why the install and requirements are "updated" to use newer versions - it took some work to get to that point. I wouldn't protest if that part should get cut, i'm sure other people have other setups, but included it anyway.