WebGL implementation of masking

[tdurand]

Goal of this: Behind able to "mask" on the contours of the car rather than on the bounding box given by YOLO.

State of the art

A researcher wrote a paper comparing all the background subtraction algorithms

And the good thing is that he open-sourced the work in a library:

https://github.com/andrewssobral/bgslibrary

He also applied BG subtraction to tracking and counting vehicles in this project: https://github.com/andrewssobral/simple_vehicle_counting

I couldn't resist but to run it on our videos, and to my surprise it is pretty good (but yeah just counts car), but it's not realtime and because it just on the CPU . (and my CPU is not that good)

Regarding the bg-subtraction, I tried the project on our video too, and it even faster, it's almost realtime on my CPU for a 1280x720 video (it does't work on GPU)

It gives pretty amazing art:

And it doesn't work with an "average image", just by comparing the previous frame différence.

Pragmatic approach

Didn't investigate much the C++ code of the bgsubtraction library, but as it only runs on the CPU and implement pretty complex algorithms that are probably not designed to work on a GPU I've not spend much more time with it.. The effort to adapt it in Webgl seems pretty high, but it is a nice information base.

I've found some nice "little projects" doing screenshot "difference" , it's something pretty common to build a quality control for the UI implementation of software development.

The best one is from mapbox (the creator of leaflet, this guy is everywhere 😂) : https://github.com/mapbox/pixelmatch

The codebase is pretty simple, use some research work on antialiasing to diff two images (not just subtracting the pixels)

I did try with our average image + frame of the video:

They are some threehold params, but it seems it gives pretty good results.

But the algorithm is only for CPU, it is fast but not fast enough to run in the browser.

GPU / Webgl fragment shader implementation

I'm currently experimenting with our own Webgl implementation of that, just implemented a prototype of a simple substraction of the rgb channel of the average frame with a single frame and it gives that results

Shader code (simplified)

vec4 outputColor = vec4(frameText.rgb - bgText.rgb, 1.0);

Result:

Pretty nice also, so now on the todo list there is:

• See performance of the fragement shader in realtime feeding a video in entry ( I'm pretty sure it will be 60 FPS)
• Then do a prototype of masking a single contour of the car
• If we feels that it can work, improve algorithm using all the research on the top (anti-aliasing ..)

Also there is this amazing project to integrate GLSL shader to react easily: https://github.com/gre/gl-react

[tdurand]

forgot to ping you: @b-g @mmmmmmmmmmmmmmmm

[tdurand]

🎉🔥 update ! ping @b-g @mmmmmmmmmmmmmmmm

I successfully managed to do a webgl implem of the masking you can check it out here: https://traffic-cam-goapmhhqhp.now.sh/webgl

➡ You need to click play on the top right box to start the video playback

How does it work

I'm using a rather simple fragment shader that computes the sum of the red, green blue channel of the difference between the frame and the average image, and is that difference is superior to a certain threshold, it does render the pixel from the average image, else it render pixel from the frame.

void main () {
  float y = uv.y * 3.0;
  vec3 averageColor = texture2D(average, uv).rgb;
  vec3 frameColor = texture2D(children, uv).rgb;
  vec3 canvas2DColor = texture2D(canvas2d, uv).rgb;
  if(canvas2DColor.r == 1.0) {
    if(dot(abs(frameColor - averageColor), vec3(1)) >= 0.25) {
      gl_FragColor = vec4(averageColor, 1.0);
    } else {
      gl_FragColor = vec4(frameColor, 1.0);
    }
  } else {
    gl_FragColor = vec4(frameColor, 1.0);
  }
}

The results is not super good but we could use more stuff from the previous research to improve the masking and get the car totally invisible... ( I think get rid of the average image and do the pixels background subtraction magic without it, reasoning on previous frames )

Also I use this amazing react API for webgl, incredible work: https://gl-react-cookbook.surge.sh/ , you can check out the examples, it expose a super nice API to integrate a webgl canvas in a react app and compose shaders, inject variables from the react app to the shaders ... It is only for 2D stuff, we render two triangles for the vertex shader and only get to play with the fragment shader, that's why I did the trick explained in the schema to have the positions, that was itself inspired by this crazy example: https://gl-react-cookbook.surge.sh/behindasteroids

After having myself wrote a simple frame comparison with raw webgl, this library enabled me at least to save 300 lines of code and 2 days of work to get to the present result I think 🙏..

Darkmode

To see the power of having webgl, I've added a darkmode (just check the box in little settings panel), which is not masking the car, just doing the simple difference of average with each frame

gl_FragColor = vec4(frameColor - averageColor, 1.0);

This shows how can we transform the scene on dynamically, in this case there is a re-render flash but we could switch shader without re-render.

Perfs and cross-browser compat:

With the simple shader without condition if else (like the one in the demo in the gl-react library: https://gl-react-cookbook.surge.sh/video ), it works really well on Chrome and mobile chrome, almost a 60 FPS on my 2-3y old samsung device... But with the more complex one (like the one I wrote), it is slower on my android, 20 FPS, and feels slow.. But I have read that if-else are not good in shader code, I should avoid using then.. so they is surely room for improvement.... It's kind of my first shader code I write, I have not experience in that field.

Then we have CORS issue with safari , safari mobile and firefox ( https://forums.developer.apple.com/thread/36725 ) , seems they don't implement the crossorigin="anonymous" correctly on the video element, maybe firefox can be fixed but safari definitely no..... But this would work if we serve videos from our server and not from vimeo. (more devops complexity...)

Also I noticed that when serving the video from the same domain to make it work, with safari and firefox the framerate drops sometime, it doesn't feel really smooth (comparing to chrome)... If we go further with the webgl implem I'll need to investigate that, as it doesn't seems to happen with the official example of the gl-react library: https://gl-react-cookbook.surge.sh/video

Next steps:

I'll likely not work more on that this week because I have lots in the pipeline, but this Proof of concept is pretty promising 😉 , the only thing that we need to take in account is that with the days budgeted for TrafficCam game, it seems hard to do the full mobile / desktop compatibility with webgl + all the other stuff in the roadmap. I think It could be realistic to have a desktop version (maybe chrome only) working well with WebGL and falling back on the SVG masking implem on mobile. We can discuss this "real world" time constraint on our next call.

But anyway the research here can be super useful for future projects.

[markus-kre]

@tdurand Just to let you know, in safari the video playback for the webgl implementation doesn't work. In Chrome and Firefox it does perfectly.

[tdurand]

thanks @mmmmmmmmmmmmmmmm , yeah it is expected for safari, the only way is to host the video in the same domain and not on an external vimeo (or do a reverse proxy).

Firefox wasn't working correctly in local, but does work deployed, I suspect they allow the cross origin if you are on HTTPS ... Good news ! I just noticed some little FPS drops on firefox

[b-g]

@tdurand Great II!

IFs
No "ifs" in shaders, very slow and considered bad practice. The pattern I often used back in the day was something like this:

colorValue = (bool expression) * 1;
// bool expression is casted in shader to number and applied directly to pixel without an if in between

(Did you try whether the short form of if is faster? No idea ... but maybe worth a try)

gl_FragColor = (dot(abs(frameColor - averageColor), vec3(1)) >= 0.25) ? vec4(averageColor, 1.0) : vec4(frameColor, 1.0);

Color distance + threshold
Also if you are working with a color distance and a threshold ... I often had better result if I had squared the difference and then did a threshold gate thing. As then small changes become les important as big ones ... which often is exactly what you a need in the "color diff masking" realm.

CORS and Safari
A pitty ... not very keen for more dev ops. But maybe we can just implement it as an optional feature for all Chrome and Firefox users. Only if not too much effort. Lets discuss on Monday.

[tdurand]

Great ! And cool you have some exp on shader code/perf and image processing, nice ideas ! Will try them on monday as I'm working on other stuff, or if you have little time and want to play with them you can just edit the shader code here : https://github.com/moovel/lab-traffic-cam/blob/master/app/components/webgl/BackgroundSubtraction.js . To run the app localy just npm install and npm run dev 😉

Agree for no more dev ops, though a reverse proxy could be simple to implement using some node library but I have not much background with that so maybe it can blow off some much needed time budget: https://github.com/Rob--W/cors-anywhere . Let's keep that in mind and see in the end if we have some time left !

[b-g]

The CORS reverse proxy doesn't look super crazy. Lets discuss on Monday. But I guess then the entire video traffic of all videos on iphone would have to be pipe trough our server. Seems not like the best idea ever.

[tdurand]

Yep , and there is maybe more maintained library like this one: https://github.com/nodejitsu/node-http-proxy .. But indeed everything would pipe through our server, bandwith cost + also maybe would eat ressources from the other stuff. (or would need to have a separated instance on a subdomain , but more devops 😆 )

[tdurand]

Update! (more of a memo for myself)

Shader perfs

Coded the shader without branching (ifs), but no perf gain on mobile, 20 FPS on my medium end android, not good enough.

precision highp float;
varying vec2 uv;
uniform sampler2D children;
uniform sampler2D average;
uniform sampler2D canvas2d;

float when_eq(float x, float y) {
  return 1.0 - abs(sign(x - y));
}

float when_gt(float x, float y) {
  return max(sign(x - y), 0.0);
}

float when_neq(float x, float y) {
  return abs(sign(x - y));
}

void main () {
  float y = uv.y * 3.0;
  vec3 averageColor = texture2D(average, uv).rgb;
  vec3 frameColor = texture2D(children, uv).rgb;
  vec3 canvas2DColor = texture2D(canvas2d, 

- uv).rgb;

  float isMaskedPixel = 0.0;
  vec3 outputColor = vec3(0);

  isMaskedPixel = 1.0 * when_eq(canvas2DColor.r, 1.0) * when_gt(dot(sqrt(abs(frameColor - averageColor)), vec3(1)), 0.90);
  
  outputColor += frameColor * when_neq(isMaskedPixel, 1.0);
  outputColor += averageColor * when_eq(isMaskedPixel, 1.0);

  gl_FragColor = vec4(outputColor, 1.0);
}

Shader feel:

@b-g Tried the squared root trick, it seems to help to separate things, but I think to get better results I need to workout better the pixel substraction maths + maybe try to get rid of the average images and reason only on the previous frame.

Next steps:

1. Integrate it to the game and see challenges and complexity
2. If 1. is validated, improve shader results and see if we can have push webgl perf to mobile also.

[tdurand]

Kamino closed and cloned this issue to moovel/lab-beat-the-traffic