Skip to content

vit-capitoner v0.1.0

Choose a tag to compare

@lachlanchen lachlanchen released this 18 Apr 12:59
· 22 commits to main since this release

ViT Captioner v0.1.0

The initial release of ViT Captioner, a Python package that extracts keyframes from videos and generates captions using the Vision Transformer (ViT-GPT2) model.

Features

  • Extracts keyframes from videos using Katna library or uniform sampling
  • Generates captions for images using the ViT-GPT2 model
  • Matches keyframes with precise timestamps in videos
  • Converts videos to SRT subtitle files with AI-generated captions
  • Visualizes keyframes and timeline data with corresponding metadata
  • Provides both command-line interface and Python API

Installation

# Install from PyPI
pip install vit-captioner

# Or install from source
git clone https://github.com/lachlanchen/VideoCaptionerWithVit.git
cd VideoCaptionerWithVit
pip install -e .

Usage Examples

Command Line

# Extract 10 keyframes from a video with visualization
vit-captioner extract -V /path/to/video.mp4 -N 10 -v

# Generate caption for an image
vit-captioner caption-image -I /path/to/image.jpg

# Convert video to captions (generates SRT and JSON)
vit-captioner caption-video -V /path/to/video.mp4 -N 10

# Find matching timestamps for keyframes
vit-captioner find-timestamps -V /path/to/video.mp4 -K /path/to/keyframes_folder -v

Python API

from vit_captioner.keyframes.extractor import KeyFrameExtractor
from vit_captioner.captioning.image import ImageCaptioner
from vit_captioner.captioning.video import VideoToCaption

# Extract keyframes
extractor = KeyFrameExtractor("/path/to/video.mp4")
frames_dir = extractor.extract_key_frames("/path/to/video.mp4", 10)

# Generate caption for an image
captioner = ImageCaptioner()
caption = captioner.predict_caption("/path/to/image.jpg")

# Convert video to captions
converter = VideoToCaption("/path/to/video.mp4", num_frames=10)
converter.convert()

Requirements

  • Python 3.6+
  • PyTorch
  • Transformers
  • OpenCV
  • Katna (for keyframe extraction)
  • Matplotlib
  • tqdm (for progress bars)

All outputs include datetime suffixes for versioning, and visualization data is automatically saved alongside figures.