Streamix is a video conferencing feature that provides you seamless virtual background matting regardless of your surroundings and provides you eye gaze correction thus giving you a feeling of a natural face to face interaction. We have used python to develop virtual background mating model and we are using ReactJS and NodeJS to build a webapp where you can see the power of our algorithms.
- Clone Streamix Repo
- From the root directory run the following command
pip install -r requirements.txt
- Go to code directory and run the following command.
python server.py
- Run the docker file
docker build --tag my-python-app .
docker run --name python-app -p 5000:5000 my-python-app
Please take a look at our CONTRIBUTING.md where we provide instructions on contributing to the repo and help us in enhancing the current video conferencing platforms.
- Implemented the segmentation algorithm discussed in this paper.
- Trained the above-mentioned model for accurately locating the person in the video.
- Did some experimentations with inference to speedup the algorithm (Current speed 25FPS).
- Created a GUI using Tkinter for locally running the application.
- Created a dummy fronted using NodeJS so that team in Phase2 can use this code as a reference.
- Basic Fronted features such as selecting the Virtual BG.
- Sending the images as response to the front end.
- Rendering the images with Virtual BG on the front end.
- Implemented the basic code for EyeGaze Correction but inference is very slow and not worth the deployment.
- Improvise the front end for the application and add more features and functionalities to the same.
- Create a pluging for Zoom using the above code as reference.
- Try to render the eyegaze corrected images faster and if possible achieve the speed of around 10FPS.
- Use some opensource video conferencing platforms and merge the above two features in that application.
- Documentation for the Segmentation Algorithm
- Documentation for the Virtual Background
- Documentation for the LocalGUI implemetation details
- Documentation for the Flask implemetation details
- Documentation for the EyeGaze Correction details