Skip to content

Latest commit

 

History

History
56 lines (47 loc) · 3.29 KB

README.md

File metadata and controls

56 lines (47 loc) · 3.29 KB
Raw

Streamix :- Enhancing Video Conferencing Platforms

DOI Build Status License: MIT YouTube Video Views

About Streamix

Streamix is a video conferencing feature that provides you seamless virtual background matting regardless of your surroundings and provides you eye gaze correction thus giving you a feeling of a natural face to face interaction. We have used python to develop virtual background mating model and we are using ReactJS and NodeJS to build a webapp where you can see the power of our algorithms.

Watch the video

Installation

  1. Clone Streamix Repo
  2. From the root directory run the following command
pip install -r requirements.txt

Usage

  1. Go to code directory and run the following command.
python server.py
  1. Run the docker file
docker build --tag my-python-app .
docker run --name python-app -p 5000:5000 my-python-app

How to Contribute?

Please take a look at our CONTRIBUTING.md where we provide instructions on contributing to the repo and help us in enhancing the current video conferencing platforms.

What things have been done for Phase1?

  • Implemented the segmentation algorithm discussed in this paper.
  • Trained the above-mentioned model for accurately locating the person in the video.
  • Did some experimentations with inference to speedup the algorithm (Current speed 25FPS).
  • Created a GUI using Tkinter for locally running the application.
  • Created a dummy fronted using NodeJS so that team in Phase2 can use this code as a reference.
  • Basic Fronted features such as selecting the Virtual BG.
  • Sending the images as response to the front end.
  • Rendering the images with Virtual BG on the front end.
  • Implemented the basic code for EyeGaze Correction but inference is very slow and not worth the deployment.

What is the plan for Phase2?

  • Improvise the front end for the application and add more features and functionalities to the same.
  • Create a pluging for Zoom using the above code as reference.
  • Try to render the eyegaze corrected images faster and if possible achieve the speed of around 10FPS.
  • Use some opensource video conferencing platforms and merge the above two features in that application.

Documentation

  1. Documentation for the Segmentation Algorithm
  2. Documentation for the Virtual Background
  3. Documentation for the LocalGUI implemetation details
  4. Documentation for the Flask implemetation details
  5. Documentation for the EyeGaze Correction details