Real time / video processing?

[okim1227]

This low light feature enhancing application is super cool! I was wondering if this model is also capable of processing videos taken in low light to bring out their features in real time. If not, what kind of development would be needed to make this happen?

[Omkaragrawal]

Hey,
This model is converted from the Tensorflow (python) model and is used directly, which results in a larger as well as a not so efficient model. As a result, we had to run it on the server instead of transporting the model to the browser. Also apparently it requires some of TFJS APIs that are not available for TFJS for browsers.

If the model was prepared natively in TFJS using APIs that were available in TFJS for browsers so that model is <60mb then it would have been feasible for it to run on browsers.

We always recommend, encourage, and expect that you help us in creating such a model, and making a PR for the same. It will be appreciated.

[Rishit-dagli]

Hi @okim1227 ,
Thanks for bringing this up. I would also like to point out that execution performance in Node.js is almost the same as performance in Python and there should not be any performance differences in these environments. I was personally very excited to run this model on videos. However, the model is quite heavy and takes quite some time to run inferences, on a no-GPU machine, let's take GitHub codespaces for instance it usually takes 10-20 seconds on a single huge image.

However, if you see this notebook I show performing inferences on TensorFlow Python in Colab with a GPU and those are pretty fast. I believe working on real-time video outputs on a machine with GPU might be possible.

If not, what kind of development would be needed to make this happen?

As of now as suggested by @jasonmayes I will soon start working on an algorithm that could allow me to run this model on every single pixel in parallel and somehow try to put together the outputs. I strongly believe this should make it possible to run on videos. Please feel free to further any suggestions or questions you might have.