Video Content Analysis

This repository contains the video content analysis pipeline from our CHI ’26 paper:

Counting How the Seconds Count: Understanding Algorithm–User Interplay in TikTok via ML-driven Analysis of Video Content — Paper | Slides | Talk

Overview

TikTok Videos
      ↓
Extract Video + Audio Embeddings
      ↓
Generate VCA Vectors
      ↓
Downstream Analysis

This pipeline uses Video-LLaMA to extract multimodal video and audio embeddings from TikTok videos.

Extract Video + Audio Embeddings

Setup

Clone Video-LLaMA and create output directories:

git clone https://github.com/DAMO-NLP-SG/Video-LLaMA.git
cd Video-LLaMA

mkdir -p embs
mkdir -p videodesc
mkdir -p vcavectors

Install Conda

MINICONDA3=Miniconda3-py37_4.9.2-Linux-x86_64.sh

wget -nc https://repo.continuum.io/miniconda/$MINICONDA3 -P ~/Downloads
chmod +x ~/Downloads/$MINICONDA3
~/Downloads/$MINICONDA3 -bf

source ~/miniconda3/bin/activate base

Install Dependencies

sudo apt update
sudo apt install ffmpeg
sudo apt install ubuntu-drivers-common
sudo apt install nvidia-cuda-toolkit
sudo apt install git-lfs

Create and activate the Video-LLaMA environment:

conda env create -f environment.yml
conda activate videollama

Install PyTorch:

conda install pytorch pytorch-cuda=12.1 -c pytorch -c nvidia

If needed, install the CUDA 11.8 PyTorch wheels explicitly:

pip install --no-cache-dir torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 \
  --index-url https://download.pytorch.org/whl/cu118

Download Video-LLaMA Checkpoints

git clone https://huggingface.co/DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned

Required Code Fixes

Depending on your environment, you may need to patch pytorchvideo:

vim /home/maleeha2/miniconda3/envs/videollama/lib/python3.9/site-packages/pytorchvideo/transforms/augmentations.py

Remove _tensor from the imports.

Replace Required Files

After cloning Video-LLaMA, replace the following files in the Video-LLaMA repository with the versions provided in this repository:

eval_configs/video_llama_eval_withaudio.yaml
video_llama/conversation/conversation_video.py

Then copy the embedding extraction script into the root of the Video-LLaMA repository:

cp video2embeddings.py /path/to/Video-LLaMA/

After this step, video2embeddings.py should be located at:

Video-LLaMA/video2embeddings.py

Run Embedding Extraction

Run the embedding extraction script with a TikTok video ID:

python video2embeddings.py \
  --cfg-path eval_configs/video_llama_eval_withaudio.yaml \
  --model_type llama_v2 \
  --gpu-id 0 \
  --videoname {video_id}

Replace {video_id} with a 19-digit TikTok video ID.

Example:

python video2embeddings.py \
  --cfg-path eval_configs/video_llama_eval_withaudio.yaml \
  --model_type llama_v2 \
  --gpu-id 0 \
  --videoname 7636001733549870369

The generated embeddings are saved under:

embs/

Video-LLaMA's response to "What is happening in the video?" is saved under:

videodesc/

Embedding Dimensions

For each TikTok video, the pipeline extracts both video and audio embeddings.

Video embedding shape: torch.Size([1, 32, 4096])
Audio embedding shape: torch.Size([1, 8, 4096])

Generate VCA Vectors

After generating the audio and video embeddings, concatenate them into a single VCA vector:

python make_vca_vector.py --videoname {video_id}

The output is saved as:

vcavectors/vca_{video_id}.npy

The generated Video Content Analysis (VCA) vectors can be used for:

• Content clustering
• Feed diversity analysis
• Temporal behavior analysis
• Similarity search
• User-interest modeling
• Recommendation system analysis

Citation

If you use this repository or build upon this pipeline, please cite:

@inproceedings{10.1145/3772318.3790311,
author = {Masood, Maleeha and Kannan, Shreya and Liu, Zikun and Vasisht, Deepak and Gupta, Indranil},
title = {Counting How the Seconds Count: Understanding TikTok Behavior via ML-driven Analysis of Video Content},
url = {https://doi.org/10.1145/3772318.3790311},
booktitle = {Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems},
series = {CHI '26}
}