EVIT: Object-Level Token Matching and KV Cache for Efficient Video Transformers

Author: Augusto Leogrande
Date: June 2026
Repository: https://github.com/LeograndeCode/EchoViT

Introduction

Starting from the ideas introduced in TBKV and its possible extensions, we propose a novel approach for efficient video transformer inference based on object-level token matching and KV caching.

The main idea is to perform token matching and caching at the prototype level rather than applying token reduction independently to each frame. The objective is to preserve important foreground information while aggressively reducing redundant tokens, such as stable background regions.

To achieve this, the original matching ratio hyperparameter ($r_{\text{match}}$) is divided into two separate mechanisms controlled by two different thresholds:

• Reuse threshold: determines whether a token/prototype is sufficiently stable to be reused from the KV cache without recomputation.
• Reduce threshold: determines whether a stable prototype can be merged and reduced.

The decision process is based on a stability metric defined through a simple hit-rate measurement. A hit occurs when the similarity between a new token and an existing prototype is higher than the reuse threshold.

Using this mechanism, frequently changing objects or rarely observed regions will have a lower stability score and will therefore be treated conservatively. Instead of performing aggressive reduction, these tokens will reuse their cached KV representation. Conversely, highly stable regions (e.g., static background areas) can be safely reduced through prototype merging.

Key Differences from Previous Approaches

• Merged tokens are renamed as prototypes, emphasizing their role as persistent representations of the scene.
• The single matching parameter is replaced with separate reuse and reduce thresholds.
• A prototype update mechanism is introduced to adapt stored representations to changing environmental conditions.
• An eviction mechanism based on prototype age is introduced to remove outdated representations.

This design aims to overcome limitations of previous approaches while providing a more elegant and adaptive formulation of token reduction for streaming video transformers.

Note: The prototype update mechanism is currently under development. If its implementation requires excessive additional time, we will temporarily revert to the previous formulation in order to prioritize experimental evaluation.

Quick Start

Prerequisites

• Linux/macOS system (Windows users should use WSL2)
• ~50GB free disk space (for datasets and results)
• GPU with CUDA support recommended (NVIDIA GPUs with compute capability 7.0+)

Environment Setup

1. Clone the Repository

git clone https://github.com/LeograndeCode/EchoViT.git
cd EchoViT

2. Install Miniconda

If you don't have Miniconda/Anaconda installed, download and install it:

# For Linux (x86_64)
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -p $HOME/miniconda3

# For macOS (Intel)
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
bash Miniconda3-latest-MacOSX-x86_64.sh -b -p $HOME/miniconda3

# For macOS (Apple Silicon)
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh
bash Miniconda3-latest-MacOSX-arm64.sh -b -p $HOME/miniconda3

# Activate conda
source $HOME/miniconda3/bin/activate

3. Create the Conda Environment

# Create environment from the provided environment.yml file
conda env create -f environment.yml

# Activate the environment
conda activate eventful-transformer

The environment.yml file includes:

• Python 3.10
• PyTorch 2.0 with CUDA 11.8 support
• Detectron2 (built from source)
• Required dependencies (OpenCV, FFmpeg, TensorBoard, etc.)

4. Verify Installation

python -c "import torch; print(f'PyTorch version: {torch.__version__}'); print(f'CUDA available: {torch.cuda.is_available()}')"

Folder Structure Setup

The evaluation framework requires the following folder structure:

EchoViT/
├── weights/           # Pre-trained model weights (create manually)
├── data/              # Datasets (auto-created during evaluation)
├── results/           # Evaluation outputs
└── configs/           # Configuration files (included in repo)

Create the weights folder:

mkdir -p weights

Repository Structure

The repository maintains the same overall structure regardless of whether the original or the proposed algorithm is used.

Main Algorithm Files

The src folder contains all source files required for the algorithm:

• src/core/: Original implementation of the base model

  • base.py - Base transformer module classes
  • blocks.py - Standard transformer block
  • backbones.py - Vision transformer backbone
  • utils.py - Utility layers (DropPath, PositionEncoding, etc.)

• src/evit/: EVIT-specific modifications and new modules

  • blocks.py - Modified transformer block with token matching
  • cache.py - Prototype KV cache implementation
  • modules.py - Token matching and caching algorithms
  • merge.py - Token merging strategies
  • backbones.py - EVIT backbone architecture

• src/models/: High-level model implementations

  • vivit.py - ViViT (Factorized Vision Video Transformer)
  • evit_vivit.py - EVIT ViViT with token optimization
  • vitdet.py - Vision Transformer for Detection

• src/datasets/: Dataset utilities

  • kinetics400.py - Kinetics-400 loader
  • epic_kitchens.py - Epic Kitchens loader
  • vid.py - Video utilities

• src/utils/: Utility modules

  • evaluate.py - Evaluation utilities
  • config.py - Configuration management
  • misc.py - Miscellaneous utilities

Evaluation Infrastructure

• configs/: Configuration files for models and datasets
• scripts/: Evaluation scripts for different tasks
  • scripts/evaluate/vivit_kinetics400.py - Main evaluation script
  • scripts/convert/ - Weight conversion scripts

Evaluation Setup

Initial Evaluation: ViViT-B on Kinetics-400

The initial experimental validation focuses on:

• Model: ViViT-B pretrained on Kinetics-400
• Dataset: Kinetics-400
• Evaluation script: scripts/evaluate/vivit_kinetics400.py

Running Evaluation

1. Download Pre-trained Weights

   Download the pre-trained ViViT-B checkpoint from the Eventful Transformer repository:

   # Place the downloaded weights in the weights folder
   mv vivit_b_kinetics400_final_24.pth weights/

2. Run Evaluation

   conda activate eventful-transformer
   python scripts/evaluate/vivit_kinetics400.py

   The script will:

   • Automatically download the Kinetics-400 dataset (5.2GB)
   • Perform inference on the dataset
   • Save results in results/evaluate/vivit_kinetics400/

   Note: First run will take time due to dataset download and preprocessing.

Output

Evaluation results are saved to:

results/evaluate/vivit_kinetics400/base/
├── config.yml         # Evaluation configuration
├── predictions.json   # Model predictions
└── metrics.json       # Accuracy and performance metrics

Comparison with Existing Methods

The evaluation compares EVIT against state-of-the-art methods:

• Token Merging (ToMe): Token reduction via similarity-based merging
• DynamicViT: Dynamic token pruning using learned importance
• Eventful Transformer: Original event-driven baseline

Evaluation metrics:

• Accuracy preservation after token reduction
• Computational efficiency (FLOPs reduction)
• Inference throughput
• GPU memory consumption

Extended Evaluation (Future)

• ViTDet for object detection
• EPIC-Kitchens for long-form egocentric video
• Ablation studies on:
  • Prototype KV caching
  • Stability metric
  • Reuse and reduction thresholds
  • Prototype update mechanism
  • Eviction strategies

Troubleshooting

Common Issues

1. CUDA out of memory

# Reduce batch size in config
python scripts/evaluate/vivit_kinetics400.py --batch_size 8

2. Dataset download fails

# Manually download Kinetics-400 and place in data/kinetics400/
# Dataset structure should be:
# data/kinetics400/
# ├── train_meta.csv
# ├── val_meta.csv
# └── videos/

3. ModuleNotFoundError

# Ensure conda environment is activated
conda activate eventful-transformer

# Verify imports
python -c "import src.core.base; print('✓ Imports successful')"

Check Environment

# List installed packages
conda list

# Check GPU
python -c "import torch; print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'No GPU')"

Development

Adding New Evaluation Scripts

1. Create a new script in scripts/evaluate/
2. Use configuration files from configs/evaluate/
3. Import from src modules following the established pattern

Converting Pre-trained Weights

# Convert ViViT weights
./scripts/convert/vivit.py input_weights.pth output_weights.pth ./configs/convert/vivit_b.txt

# Convert ViTDet weights
./scripts/convert/vitdet.py input_weights.pkl output_weights.pth ./configs/convert/vitdet_b.txt

References

• Original Eventful Transformer: GitHub Repository
• ViViT Paper: Factorized Video Transformers with Reuse and Decomposition
• Kinetics-400 Dataset: Official Website

License

[Specify your license here]

Citation

If you use this code, please cite:

@article{leogrande2026evit,
  title={EVIT: Object-Level Token Matching and KV Cache for Efficient Video Transformers},
  author={Leogrande, Augusto},
  year={2026}
}

Contact

For questions or issues, please open an issue on the GitHub repository.