Seeing Through: Analyzing and Attacking Virtual Backgrounds in Video Calls

This is the artifact repository accompanying the publication

Felix Weissberg*, Jan Malte Hilgefort*, Steve Grogorick, Daniel Arp, Thorsten Eisenhofer, Martin Eisemann, Konrad Rieck. 2025. Seeing Through: Analyzing and Attacking Virtual Backgrounds in Video Calls. * Authors contributed equally.

Overview

This repository contains code for an attack that restores the real surroundings in video calls concealed by a virtual background -- a common feature of many video conferencing services. The attack leverages the compromise between quality and efficiency in real-time portrait segmentation used by these services, which must operate under constrained computational environments. This trade-off often results in imperfections in the caller's mask, causing some pixels to inadvertently reveal the real surroundings instead of the virtual background.

The attack works by contrasting a high-quality mask (DeepLabV3) of the caller with the mask generated by re-purposing the video conferencing service's portrait segmentation on the video containing the virtual background. By comparing the two masks, the attack identifies the pixels where the real surroundings leak through the virtual background in each frame. Over the course of the video call, this information is used to gradually reconstruct an image of the caller's environment.

In addition to the attack code, this repository includes the full evaluation framework and implementations of baseline methods used for comparison in the paper.

Dependencies

• apptainer
• curl

Quick start

Make sure all dependencies are installed on your machine before running the attack pipeline. Once they are satisfied, you can setup the container environments and download necessary data by running the scripts/build.sh script. After successful execution, the scripts/run.sh script can be run to execute the whole evaluation.

1. scripts/build.sh
2. scripts/run.sh

The run.sh script executes every step of the evaluation including the proposed reconstruction attack as well as the baselines. It first creates a dataset of videos with different real surroundings and virtual backgrounds based on a sample green screen video.The attacks are then conducted on this dataset and the quality of their reconstructed images is measured as described in the paper. In case of a successful execution, the run.sh should terminate with a prompt similar the following, showing the reconstruction performance of our attack (vader) as well as the two attacks by Sabra et al. (sabra) and Hilgefort et al. (hilgefort):

[*] Conduct evaluation
Attack:  vader
vader-mp-interview: mean 0.15463907988801348, median 0.15463907988801348
vader-zoom-interview: mean 0.16432017496528595, median 0.16432017496528595
Attack:  sabra
sabra-mp-interview: mean 0.05958382645539942, median 0.05958382645539942
sabra-zoom-interview: mean 0.06915551675188922, median 0.06915551675188922
Attack:  hilgefort
hilgefort-mp-interview: mean 0.009324126904120045, median 0.009324126904120045
hilgefort-zoom-interview: mean 0.0038870101331816543, median 0.0038870101331816543

Structure

The repository is organized into three primary components: The evaluation framework and attack methodology, the data used for running the attack, and the implementations of baseline attacks for comparison.

1. Evaluation Framework and Attack

Our evaluation framework generates multiple videos of individuals in video calls featuring diverse surroundings and virtual backgrounds. These videos are created using recordings in front of a green screen, which provides ground-truth masks of the caller. These masks enable the assessment of pixel leakage from the real surroundings in each frame. Based on these masks in combination with the masks created by the video conferencing services to insert the virtual background allow for a perfect reconstruction of the surroundings, serving as a reference for evaluating the attacks.

The process, starting from green screen recordings to the evaluation of an attack, is implemented in the following four steps:

1. Generation This step generates videos with real surroundings and extracts ground-truth portrait masks from the green screen recordings. Additionally, the real surroundings are replaced with virtual backgrounds using MediaPipe and Zoom’s virtual background replacement features. The code for this step is available in src/generate.

2. High-Quality Portrait Segmentation Since our attack and the approach by Sabra et al. both rely on high-quality portrait masks, these masks are generated separately in this step. The relevant code is located in src/deeplab.

3. Attack Our attack is conducted using the videos with virtual backgrounds generated in the first step and the high-quality masks from the second. The implementation of the attack is provided in src/attack. Note the limitations related to Zoom, as described in the Notes section below.

4. Evaluation Finally, the quality of the reconstructed surroundings from the attack is evaluated as outlined above. The evaluation code can be found in src/eval.

2. Data

We provide a sample recording in front of a green screen, along with images for the real surroundings and virtual background, in the following directories:

• data/raw
• data/real_backgrounds
• data/virtual_backgrounds

3. Baselines

For comparsion, we implemented two baseline attacks from Hilgefort et al. and Sabra et al.. Their implementations can be found in baselines/hilgefort and baselines/sabra, respectively.

Notes

Our proposed attack repurposes the virtual background feature of the video conferencing tool to replace real surroundings and extract the portrait mask used by each framework. This requires access to the respective model employed. While this is straightforward for the open-source framework MediaPipe (used in Google Meet, Jitsi, and BigBlueButton), it involves reverse-engineering the Zoom software. To comply with Zoom's terms and conditions, we are unable to share the tool we used to extract the portrait masks from the Zoom client. However, we provide the extracted masks for the sample video, enabling the execution of our attack without additional reverse-engineering.