Repository navigation
vit-capitoner v0.1.0
ViT Captioner v0.1.0
The initial release of ViT Captioner, a Python package that extracts keyframes from videos and generates captions using the Vision Transformer (ViT-GPT2) model.
Features
- Extracts keyframes from videos using Katna library or uniform sampling
- Generates captions for images using the ViT-GPT2 model
- Matches keyframes with precise timestamps in videos
- Converts videos to SRT subtitle files with AI-generated captions
- Visualizes keyframes and timeline data with corresponding metadata
- Provides both command-line interface and Python API
Installation
# Install from PyPI
pip install vit-captioner
# Or install from source
git clone https://github.com/lachlanchen/VideoCaptionerWithVit.git
cd VideoCaptionerWithVit
pip install -e .Usage Examples
Command Line
# Extract 10 keyframes from a video with visualization
vit-captioner extract -V /path/to/video.mp4 -N 10 -v
# Generate caption for an image
vit-captioner caption-image -I /path/to/image.jpg
# Convert video to captions (generates SRT and JSON)
vit-captioner caption-video -V /path/to/video.mp4 -N 10
# Find matching timestamps for keyframes
vit-captioner find-timestamps -V /path/to/video.mp4 -K /path/to/keyframes_folder -vPython API
from vit_captioner.keyframes.extractor import KeyFrameExtractor
from vit_captioner.captioning.image import ImageCaptioner
from vit_captioner.captioning.video import VideoToCaption
# Extract keyframes
extractor = KeyFrameExtractor("/path/to/video.mp4")
frames_dir = extractor.extract_key_frames("/path/to/video.mp4", 10)
# Generate caption for an image
captioner = ImageCaptioner()
caption = captioner.predict_caption("/path/to/image.jpg")
# Convert video to captions
converter = VideoToCaption("/path/to/video.mp4", num_frames=10)
converter.convert()Requirements
- Python 3.6+
- PyTorch
- Transformers
- OpenCV
- Katna (for keyframe extraction)
- Matplotlib
- tqdm (for progress bars)
All outputs include datetime suffixes for versioning, and visualization data is automatically saved alongside figures.