Dynamic Vision Transformer (dViT)

This project implements a Dynamic Vision Transformer that uses dynamic inference to speed up image classification, instead of always processing images at high resolution (which wastes time and compute), it starts with low resolution and switches to high resolution when needed. This project resulted in achieving ~2x faster speedup than static high res vision transformer at inference.

Working

1. Training: The Vision Transformer is trained on CIFAR-10 at low resolution (32x32) for 2 epochs as loss variance was low.
2. Proxy Signal: During inference, the model measures how much the class token changes between an early layer and the final layer.
3. If the change is small (means the model is confident), it uses the low res (32x32) result. If the change is large (uncertain), it moves to high res (64×64).
4. Weight Sharing: The high resolution model shares weights from the low resolution model, resulting in no extra training.

Inference Metrics

Trained and Tested on 2000 CIFAR-10 Images

1. Average Latency: 7.90 ms/image
2. Peak VRAM: 88.01 MB
3. Escalation Rate: 9.0% (only 9% of images moved to high res.)

Results

1. Speedup: Dynamic inference is 2.27x faster than static high resolution, saves 10.01 ms per image compared to always using high resolution, which would result in saving hours of compute on larger image datasets.
2. Escalation: Only 9% of images require high-resolution processing, while 91% are handled at low resolution.
3. Memory Efficiency: Most images processed at low resolution (55 MB), with peak memory only when high-res is needed (88 MB).