clipyard

A lightweight wrapper around yt-dlp that aims to make downloading large video datasets easier and faster through a simple CLI

Features

• Multiple Input Sources: Support for text files, CSV files, and HuggingFace datasets. For files, you can provide the actual file or a publicly accesible URL
• Input Types: Either provide a list of IDs and the platform (typically YouTube/Vimdeo, or a list of URLs)
• Parallel Downloads: Configurable parallel workers for fast batch downloads
• Metadata Tracking: Saves download summaries and failure reasons to quickly inspect download summaries, and easy re-launching of runs
• Good Defaults: Takes the pain out of configuring yt-dlp by putting good defaults in place. Videos are downloaded in h264, and m4a codecs (typically does not require re-encoding). This has two benefits: fast decoding, and ability to preview files quickly using the filesystem

PS: This library is in early development, and is an evolution of this script

Installation

pip install clipyard
pip install clipyard[datasets]  # For HuggingFace dataset support

clipyard -h
clipyard download -h

Or, if you use uv (recommended):

uv tool install clipyard[datasets]
uv tool run clipyard download -h

Examples (w/ Best Practices)

Refer to this doc to setup cookies for downloading. Note that some of these args presented here are not mandatory, but are the suggested way to use this tool for best quality of life.

SF20K Dataset (HuggingFace)

This cmd downloads test_expert split of the SF20K dataset, in 720p. Note

uv tool run clipyard download \
  --input-type huggingface \
  --hf-dataset "rghermi/sf20k" \
  --hf-split test_expert \
  --id-column video_id \
  --url-column video_url \
  --output-dir /mnt/DataSSD/datasets/sf20k/test_expert/720p/ \
  --metadata-dir /mnt/DataSSD/datasets/sf20k/test_expert/metadata/ \
  --cookies cookies.txt

VUE-TR-V2 Dataset (Text File)

uv tool run clipyard download \
  --input-type txt \
  --input "https://raw.githubusercontent.com/bytedance/vidi/refs/heads/main/VUE_TR_V2/video_id.txt" \
  --output-dir /mnt/DataSSD/datasets/vue-tr-v2/videos/ \
  --metadata-dir /mnt/DataSSD/datasets/vue-tr-v2/metadata/ \
  --cookies cookies.txt

---

Re-Launching Runs

If you ran a download with --metadata-dir, a download_config.json file is saved. You can relaunch the same download with:

clipyard download --config ./metadata/download_config.json

You can also override specific settings:

clipyard download --config ./metadata/download_config.json --workers 8  # More workers
clipyard download --config ./metadata/download_config.json --output-dir ./new-downloads  # Different output dir

Command-Line Arguments

Required Arguments

• --input-type: Type of input source (txt, csv, huggingface) — required unless --config is provided
• --output-dir: Directory to save downloaded videos — required unless --config is provided

Config File

• --config: Path to saved config JSON file (from previous run's --metadata-dir) — loads settings, CLI args can override

Input Arguments

• --input: Input file path or URL (for txt/csv) or dataset name (for huggingface)
• --platform: Platform for video IDs when not specified in URLs (eg: youtube, vimeo, default: None)
• --id-column: Column name containing video IDs (default: video_id) — at least one of --id-column or --url-column required
• --url-column: Column name containing video URLs (optional) — if both provided, URLs preferred; IDs extracted from URLs
• --hf-dataset: HuggingFace dataset identifier (e.g., "rghermi/sf20k")
• --hf-split: HuggingFace dataset split (default: train)

Output Arguments

• --metadata-dir: Directory to save metadata — creates download_summary.csv, failed_videos.csv, and download_config.json

Download Arguments

• --resolution: Video resolution (144, 240, 360, 480, 720, 1080, default: 720)
• --workers: Number of parallel workers (default: 4, recommended: 4-8)
• --threads: Threads per download, passed to yt-dlp --concurrent-fragments (default: 1, recommended: 2-4)
• --max-videos: Maximum number of videos to download (for testing)
• --cookies: Path to cookies file (.txt) for yt-dlp — required for restricted/private videos
• --replace-existing: Replace videos if already downloaded
• --silence-errors: Silence yt-dlp errors and warnings
• --sleep-interval: Seconds to sleep before each download (default: 5)
• --max-sleep-interval: Maximum seconds to sleep before each download (default: 10)

Input Formats

Text files: One video ID or URL per line. Empty lines and lines starting with # are ignored.

CSV files: At least one of the following must be provided:

• Video ID column (default: video_id): IDs are used to construct URLs
• Video URL column: IDs are extracted from URLs
• Both columns: URLs are preferred (IDs extracted from URLs)

HuggingFace datasets: Specify dataset name and split. These are then converted to CSV format via .to_pandas() and processed the same way as CSV files.

Python API

You can also use clipyard programmatically:

from clipyard import (
    parse_txt_file,
    download_videos,
    DownloadConfig,
    save_summary_csv,
)
from pathlib import Path

# Parse input
sources = parse_txt_file(Path("videos.txt"))

# Configure download
config = DownloadConfig(
    output_dir=Path("./downloads"),
    resolution=1080,
    workers=4,
    threads=2,
)

# Download
results = download_videos(sources, config)

# Save summary
save_summary_csv(results, Path("summary.csv"))

Context For LLMs

Use development.md for developer documentation, tailored for LLM use.

Future

• Break out DownloadConfig into yt-dlp specific args vs. metadata args
• Allow user-defined presets to allow different yt-dlp download presets. Use the currently hard-coded option as the default preset