v1.3.1 - Windows + Linux GPU / 双击启动

vid2dataset 1.3.1

小白怎么用（Windows）

方法 1：只下一个 exe（最快）

1. 在本页下方下载 vid2dataset.exe
2. 双击它
3. 选视频文件夹 → 选输出文件夹 → 点「提取」

如果弹出「Windows 已保护你的电脑」：点 更多信息 → 仍要运行。

方法 2：下载 zip 压缩包（推荐给不熟悉 exe 的人）

1. 在本页下方下载 vid2dataset-1.3.1-windows.zip
2. 右键 zip → 全部解压缩（Extract All），解到桌面或任意文件夹
3. 打开解压出来的文件夹，你会看到：
   • vid2dataset.exe
   • START.bat / 双击启动.bat
   • HOW_TO_START.txt
4. 双击 双击启动.bat（或 START.bat，或直接双击 exe）
5. 以后都进这个文件夹再双击。不要只把 exe 单独拖走。

打开软件后：选输入 → 选输出 → 点提取。
窗口右上角有 「检查更新」，以后有新版本可以一键更新。

第一次请先关掉「自动打标」，确认出图后再开。

Linux：源码里运行 chmod +x start.sh && ./start.sh

GPU（Windows 和 Linux 都能用）

• Windows .exe：勾选「GPU 加速」。第一次会下载 CUDA 版 PyTorch（约 2.5 GB），之后用缓存。
• Linux：先 ./venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cu126，再勾选 GPU。
• 没装 CUDA 也能提取，不会再报 No module named 'torch' 然后中止。

第一次请先关掉「自动打标」，确认出图后再开。

How to start

Method 1: download vid2dataset.exe → double-click.

Method 2: download vid2dataset-1.3.1-windows.zip → Extract All → open the folder → double-click START.bat or 双击启动.bat.

The app has Check for Updates in the top-right.

Linux: ./start.sh (install venv once if prompted).

GPU works on both OS (NVIDIA). Missing torch no longer blocks extract.

v1.3.0 - Double-click to start / 双击启动

vid2dataset 1.3.0

小白怎么用（不用会写代码）

1. 下载下面的 vid2dataset.exe
2. 双击它
3. 选视频文件夹 → 选输出文件夹 → 点「提取」

如果 Windows 弹出「已保护你的电脑」：点 更多信息 → 仍要运行。

第一次请先 关掉「GPU 加速」 和 「自动打标」，确认能出图后再开。

---

How to start (no Python, no install)

1. Download vid2dataset.exe below
2. Double-click it
3. Pick a video folder → pick an output folder → click Extract

SmartScreen? More info → Run anyway.

Leave GPU acceleration and Auto-tag off on the first run.

---

Also in this release

• Clear start files in the source tree: START.bat / 双击启动.bat / HOW_TO_START.txt
• GPU missing PyTorch no longer blocks extract (falls back to CPU)
• Optional new Tauri desktop UI for developers (scripts/run_tauri.py dev)

v1.2.0 - Advanced mode: cut segments, capture exact frames

Not just one click any more

Press Advanced… next to the Extract button:

• Segment cut / 片段选取: scrub each video on a timeline, mark [Set In] → [Set Out] ranges — the next Extract run samples only inside your segments. Skip intros, outros, loading screens; keep just the good choreography. Also scriptable: vid2dataset extract videos/ --segment "dance.mp4:30-95.5"
• Manual capture / 手动截取: step frame-by-frame (−10s/−1s/−1f/+1f/+1s/+10s) to the exact moment you want and press [Capture frame] — it's written instantly through the same letterbox-crop → bucket-resize pipeline, deliberately skipping the quality/diversity filters (you picked it yourself). Manual captures are auto-tagged along with everything else.

一键太粗糙？进阶模式让你拖时间轴标记片段（下次提取只取片段内），或逐帧找到想要的画面直接截取 — 手动截取跳过质量过滤（因为是你亲自选的），并会和其他图片一起自动打标。

Verified with a real video end-to-end: frames outside the marked segment never enter the pipeline. 114 unit tests.

v1.1.0 - Caption quality: trait pruning + tag-based curation

Your captions, under control

• Blacklist: tags like watermark, signature, blurry never enter captions.
• Trait pruning: set Prune ≥ 0.8 and tags present in ≥80% of images (blue hair, twintails…) are removed from every caption — the trigger word absorbs the character's constant traits. The pruned list shows up in _report.html.
• Tag-based curation: Require 1girl / Reject if multiple girls — frames that fail the rules are moved to output/_rejected/ automatically. The tagger now curates your dataset, not just captions it: no more group-shot pollution from gameplay/anime sources.
• Always-tags (tag_always config / --always): fixed tokens after the trigger, e.g. anime screencap.
• Everything also works standalone: vid2dataset tag FOLDER --require 1girl --exclude "multiple girls" --prune-threshold 0.8

屏蔽标签、特征修剪（触发词吸收固定特征）、按标签自动剔除多人/不合格帧到 _rejected/ —— 打标器现在不只写 caption，还会帮你筛数据集。

v1.0.0 - Auto-tagging: video in, kohya-ready dataset out

The last mile is connected

vid2dataset now produces complete, trainer-ready datasets. Tick Auto-tag images, type your trigger word, extract — every image gets a .txt caption sidecar (trigger, character tags, general tags) that kohya / OneTrainer read directly.

• WD tagger integration: wd-eva02-large-tagger-v3 (default, most accurate, ~1.2 GB) or wd-swinv2-tagger-v3 (~450 MB). Downloads once on first enable (huggingface.co, hf-mirror.com fallback for CN users).
• Any GPU works: inference runs on onnxruntime-DirectML (~25 MB, on-demand) — NVIDIA, AMD, and Intel GPUs are all accelerated on Windows, with automatic CPU fallback. The .exe itself stays 151 MB.
• kohya folder convention: set kohya_repeats = 10 (+ Flatten output) to get a 10_yourtrigger/ dataset folder.
• vid2dataset tag FOLDER: caption any existing image folder from the CLI — not just vid2dataset output.
• Spot dataset bias instantly: _report.html gains a top-30 tag frequency table; hover any image in _gallery.html to see its tags.
• Caption hygiene built in: single-line LF sidecars, kaomoji-safe underscore handling (^_^ stays ^_^), rating tags never enter captions, cancelled runs never write fake trigger-only captions.

勾选自动打标并填触发词，每张图自动生成可直接训练的 .txt 标签文件。首次启用下载一次模型（约 0.5–1.2 GB），任何显卡（NVIDIA/AMD/Intel）都可通过 DirectML 加速。

Verified end-to-end on real hardware: model + runtime downloads through the actual code paths, DirectML inference on RTX 3090, stdlib-only isolation test mirroring the .exe environment, 100 unit tests.

Scope note: vid2dataset prepares datasets — extraction, curation, captioning. Training, upscaling, image editing, and prompt tools are permanently out of scope.

v0.9.0 - torch 2.11 runtime: RTX 50-series support + safer GPU cache

What's new

RTX 50-series (Blackwell) GPU acceleration now works. The old runtime downloaded torch 2.5.1+cu124, which has no sm_120 kernels — RTX 5090/5080/5070 users got a 2.4 GB download that could never run. v0.9.0 detects Blackwell and fetches torch 2.11.0+cu128 instead; every other NVIDIA card gets cu126, which actually covers older cards than before (Maxwell through Hopper).

• GPU runtime: torch 2.5.1 → 2.11.0 (PyTorch no longer publishes cu121/cu124 wheels)
• GPU-swap detection: upgrading your graphics card re-prompts for the matching CUDA build instead of silently activating an incompatible cache
• Safer downloads: interrupted downloads can never be mistaken for complete wheels; upgrading never leaves orphaned files from the old torch; a failed download never destroys a still-working runtime
• Existing v0.8.x GPU caches are detected as outdated — the app offers a one-time re-download (~2.5 GB, 2.7 GB on RTX 50)

RTX 50 系列显卡现在可以正常使用 GPU 加速了。已有的 v0.8.x GPU 快取会被判定为过期，首次勾选 GPU 加速时会提示重新下载一次（约 2.5–2.7 GB）。

Verified end-to-end: real 2.5 GB download through the actual code path, stdlib-only activation (mirroring the .exe environment), and CUDA compute on an RTX 3090. 84 unit tests, ruff clean.

v0.8.6 - REAL fix: stdlib bundling + lazy gpu_filters (tested in .exe)

v0.8.6 — TESTED in the real .exe environment this time

What I did differently

Built a console-mode test .exe and added a hidden CLI command vid2dataset gpu-test that runs the full GPU activation sequence and prints what works / what fails. Iterated against the actual .exe environment until every step passed:

activate_runtime: ok=True
torch=2.5.1+cu121 from cache
cuda available: True
device: NVIDIA GeForce RTX 3090
matmul OK: sum=5037.38
device_summary: GPU pipeline: CUDA on NVIDIA GeForce RTX 3090
pipeline available: True
BatchSSIMFilter constructed OK
[OK] === GPU runtime fully working ===

Why v0.8.5 still failed

The error you got — ImportError: No module named 'timeit' — wasn't reproducible in my .venv tests because real Python has the entire stdlib available. The PyInstaller .exe is different: when --exclude-module torch removes torch from the import graph, PyInstaller stops tracing what torch imports. So 159 stdlib modules torch needs (timeit, unittest.mock, multiprocessing.connection, asyncio.events, ...) were missing from the .exe bundle.

之前的測試是在真 Python 跑，stdlib 完整所以看不到問題。.exe 用 PyInstaller bundle，因為 torch 被 exclude，PyInstaller 也把 torch 用到的 stdlib 一起丟掉，所以 timeit 等模組消失。這次直接在 .exe 裡面測。

Three fixes

1. build_exe.py bundles 159 stdlib modules as --hidden-import. List was AST-scanned from torch + numpy + sympy + the other cache deps to find every stdlib reference. Includes timeit, unittest.mock, multiprocessing.connection, concurrent.futures.process, etc.

2. gpu_filters lazy-imports torch. Previously _HAS_TORCH = False was baked at module load time (before activate_runtime()). After activation, the False stayed cached -> is_gpu_pipeline_available() returned False even with working CUDA. Now uses _ensure_torch() which re-checks on every call.

3. New diagnostic command vid2dataset gpu-test (also works in the .exe — just spawn a console). Prints exactly what loads, what fails, what CUDA reports. Use this if you hit any future activation issue.

Status

• v0.8.3 — discovered mirror race bug, fixed it, missed numpy / stdlib issues
• v0.8.4 — claimed numpy fix but the edit silently failed
• v0.8.5 — added cache-append + retry logic, but PyInstaller stdlib issue made it still fail in .exe
• v0.8.6 — actually verified end-to-end in the real .exe environment

Update

1. Check for Updates in the app, or download below
2. Tick GPU 加速
3. Cache will redownload (RUNTIME_VERSION bumped). After ~10min download, GPU should activate
4. If anything still fails: open a console and run vid2dataset.exe gpu-test, send me the output

Download

• vid2dataset.exe (151 MB)

v0.8.5 - Verified end-to-end with isolated cache

v0.8.5 — Verified working end-to-end

This time I tested it before shipping. Setup that mirrors the .exe environment (renamed venv torch + numpy to invisible names so only the downloaded cache could provide them):

activate_runtime: ok=True
torch from: %LOCALAPPDATA%/vid2dataset/gpu_runtime/torch
CUDA: True (NVIDIA GeForce RTX 3090)
numpy from: cache
gpu pipeline: True
Real extraction: ran full pipeline with gpu_accel=True
  -> scene detect -> ffmpeg keyframes -> auto-quality
  -> GPU SSIM filter -> bucket resize -> done

Sorry for the v0.8.3 / v0.8.4 churn. 對前兩個版本接連失敗道歉。

What was wrong with v0.8.4

1. numpy never actually got into the download list. My v0.8.4 edit relied on a string match that silently failed. Release notes claimed numpy was bundled — it wasn't. You hit the same activation failure plus a misleading patch note.

2. The fix that DID land made things worse. v0.8.4's sys.path.insert(0, cache) shadowed every PyInstaller-bundled dep. Even when numpy WOULD have been in cache, the cache's typing_extensions==4.12.2 overrode the .exe's bundled 4.13+. Result: ImportError: cannot import name 'Sentinel' from 'typing_extensions' when loading gpu_filters.

What v0.8.5 actually fixes

• numpy 2.1.3 is in _PYPI_DEPS (verified by re-reading the list after edit).
• activate_runtime() uses sys.path.append(cache) instead of insert. PyInstaller's bundled site-packages win for shared deps (typing_extensions, jinja2, etc.); torch is only in cache so it's found there.
• activate_runtime() returns the real error string. If anything still fails for you, the dialog will tell you the actual ImportError instead of "无法载入".
• RUNTIME_VERSION bumped, so old cache (without numpy) auto-refreshes.

Test before downloading

I ran the full pipeline (gpu_accel=True) in an environment where the only torch reachable was the cache copy. CUDA matmul, GPU SSIM filter, and the whole extractor ran clean.

Update

1. Click 检查更新 / Check for Updates in the app
2. Tick GPU 加速 — old cache will auto-refresh, this time with numpy
3. Activation dialog should say "GPU runtime ready"

Download

• vid2dataset.exe (151 MB)

v0.8.4 - Hotfix: numpy in runtime + show real activation errors

v0.8.4 — Hotfix: activation failed due to missing numpy + opaque error

What you saw

After v0.8.3 the download succeeded, but ticking GPU acceleration popped a dialog:

无法载入 GPU 运行时。请删除 %LOCALAPPDATA%/vid2dataset/gpu_runtime 后重试下载。

下载明明成功，但 GPU 加速勾不上，只看到「无法载入」却没有原因。

Root cause

1. numpy was missing from the cache. torch's C extension is built against a specific numpy ABI. When torch tried to import numpy, it found whatever numpy version PyInstaller bundled in the .exe — usually wrong version, or wrong location. Result: Failed to initialize NumPy then either crash or torch.cuda.is_available() == False.

2. The error was swallowed. activate_runtime() returned True/False. The dialog had no detail. v0.8.4 returns the actual error message.

What's fixed

• numpy 2.1.3 is now part of the GPU runtime download (~16 MB extra).
• activate_runtime() now returns the actual ImportError / RuntimeError text. The dialog will show it.
• It also explicitly verifies torch.cuda.is_available() before declaring success — so a torch-loads-but-driver-too-old situation is reported clearly instead of silently failing later.
• RUNTIME_VERSION bumped, so any v0.8.3 cache (which lacks numpy) is automatically invalidated and re-downloaded on the next GPU enable.

What you should do

1. Update via 检查更新 / Check for Updates.
2. Tick GPU 加速 again. The cache from v0.8.3 will be re-downloaded with numpy this time.
3. If activation still fails, the dialog will now show the actual error (e.g. ImportError: DLL load failed, cuda check failed, etc.) — much easier to debug.

Download

• vid2dataset.exe (151 MB)

v0.8.3 - Mirror race actually verified

v0.8.3 — Mirror race was lying. Now actually verified.

After v0.8.2 shipped, the question was "did you actually check all the URLs work?" — turns out no, two of three mirrors were misleading.

What was broken in v0.8.1 / v0.8.2

The mirror race tested directory index reachability, not whether the actual wheel file existed on each mirror. Tsinghua / Aliyun / USTC are PyPI mirrors — they host the regular torch (CPU only) but not the +cu121 / +cu124 builds. So a Chinese user could see "fastest mirror: tsinghua (0.05s)" — and then the real download would hit HTTP 404.

Worse: the URL picked by the race wasn't even passed to download_runtime — it just always used PyTorch official anyway. So the "race" was decorative.

之前的鏡像速度測試有兩個 bug：(1) 只測目錄索引能不能連到，沒測真正的 wheel 檔案在不在那邊。Tsinghua / Aliyun 是 PyPI 鏡像，沒有 CUDA 版本的 torch。(2) 速度賽選出的 URL 沒被實際使用。

What's fixed

• Real mirror probe: pick_fastest_mirror() now does a partial GET on the actual torch-X.Y.Z+cuXXX-cpYY-cpYY-win_amd64.whl filename. A mirror only wins if it's both fast AND actually has the file.
• Verified mirror list: only the 3 confirmed sources stay:
  • download.pytorch.org (official, redirects to R2)
  • download-r2.pytorch.org (Cloudflare R2, the actual host)
  • mirror.sjtu.edu.cn/pytorch-wheels/ (the real China mirror that hosts CUDA wheels)
• Race result actually used: download_runtime(torch_url=...) accepts the URL chosen by the race.

Verified all 11 wheels (cp312 / win_amd64)

Wheel	Size	Status
torch cu118	2700 MB	206 ✓
torch cu121	2449 MB	206 ✓
torch cu124	2510 MB	206 ✓
typing_extensions	37 KB	206 ✓
filelock	16 KB	206 ✓
fsspec	180 KB	206 ✓
sympy	6.2 MB	206 ✓
mpmath	536 KB	206 ✓
networkx	1.7 MB	206 ✓
jinja2	133 KB	206 ✓
MarkupSafe	17 KB	206 ✓

Download

• vid2dataset.exe (151 MB)

v0.8.x users: click 检查更新 / Check for Updates.