lorakit

Automated LoRA dataset creation from any folder of images, with first-class Ren'Py game support.

Drop a directory of art at lorakit and it figures out which images go with which character, builds clean per-character training sets in kohya-ss/sd-scripts layout, and (optionally) launches a LoRA training run for you.

my-images/             →   lora-output/
├── character_a_01.png        ├── alice/
├── character_a_02.png        │   ├── 60_ex_alice/
├── character_b_01.png        │   │   ├── ...image1.jpg
├── ...                       │   │   └── ...image1.txt   # caption
└── (1000s of files)          │   ├── masks/
                              │   └── dataset_config.toml  # kohya-ready
                              ├── bob/
                              ├── carol/
                              └── ...

Then:

lorakit train ./lora-output/alice --base-model sdxl --output ./alice.safetensors

---

Install

pip install lorakit

That's it. The default install pulls everything you need to run the full pipeline end-to-end: torch + ORT-GPU + transformers, the CCIP/DINOv3 embedding stack, vLLM with a pinned Qwen3-VL-30B-A3B-AWQ dep set, the captioner, etc.

Requirements:

• Python 3.11+
• Linux with an Nvidia GPU (tested on RTX 4090, 24 GB)
• CUDA 12.x runtime libraries available to ONNX Runtime. On boxes shipping a CUDA 13 driver, install matching cu12 wheels: pip install nvidia-cublas-cu12 nvidia-cudnn-cu12 nvidia-cufft-cu12 nvidia-curand-cu12 nvidia-cusolver-cu12 nvidia-cusparse-cu12 nvidia-cuda-runtime-cu12 nvidia-cuda-nvrtc-cu12
• For lorakit unpack-game only: xvfb (sudo apt install xvfb on Debian/Ubuntu)

Optional add-ons (genuinely orthogonal choices, not "do you have a GPU"):

Extra	What it adds
vlm-cloud	OpenAI / Anthropic VLM backends (use instead of local vLLM if GPU is tight)
segment	SAM-based instance segmentation in Stage 9
insightface	Real-photo face encoder (default is CCIP, anime-tuned)
all	All of the above

pip install "lorakit[all]"

Cloud / multi-machine deployment: out of scope for this project. Wrap lorakit with your own runner (vast.ai paste-SSH bootstrap, RunPod, etc.) — the pipeline is just a CLI you invoke.

---

Quick start

From a folder of images

# 1) Extract per-character training datasets
lorakit extract ./my-images --output ./datasets

# 2) (optional) Train a LoRA for one character
lorakit train ./datasets/alice --base-model sdxl --output ./alice.safetensors

From a Ren'Py game

# 1) Decompile + extract assets from the game itself
lorakit unpack-game ./MyGame   # writes to ./MyGame/.te-unpack/extracted/

# 2) Run the same pipeline against the extracted images
lorakit extract ./MyGame/.te-unpack/extracted/ --output ./datasets

# 3) Train as above
lorakit train ./datasets/alice --base-model sdxl

---

How it works

The extract pipeline runs in nine stages, each writing a parquet checkpoint so a kill-and-resume mid-pipeline is cheap:

Stage	What it does
0. inventory	Walk input dir, dedup by sha1, classify sprite vs. scene. Bootstrap an aliases.auto.json from filename tokens.
1. faces	Detect faces (anime YOLOv8 head + face union), embed with CCIP + DINOv3.
2. cluster	HDBSCAN on the L2-normalised embeddings. Filename-asserted images skip clustering and join assert_<canonical> directly.
3. prune	Cross-cluster centroid similarity → VLM pair-merge for ambiguous pairs. Asserted clusters keep their canonical names.
4. verify	Per-cluster identity check via VLM 3×3 grid. Tail-scan + in-place subcluster on contaminated clusters.
5. candidates	Build per-image candidate set, drop multi-character images (LORA_STRICT_PURITY).
6. dedup	dHash + DINOv3 full-image dedup.
7. budget	Aesthetic gate + max-min greedy diverse selection per cluster.
8. caption	PixAI Tagger v0.9 (or WD14 fallback) for booru-style tags.
9. emit	Write per-character folders with kohya-ready dataset_config.toml.

---

Configuration

For v0.1.0 the configuration surface is CLI flags + a few module-level constants in total_extraction/config.py. Run lorakit extract --help for the full list. Common knobs:

--vlm vllm-qwen3vl-30b     # default; or: vllm-qwen25vl-32b | vllm-qwen2vl-72b | openai | anthropic | off
--encoder ccip             # or: arcface | auto
--stage 0-4                # run only a subset
--seed-dir ./my-anchors    # optional: <name>/*.jpg seed crops to anchor identities
--alias-map ./aliases.json # optional: explicit canonical→aliases map
--force                    # ignore cached parquets

A future release will move the larger knob set into a lorakit.toml project file.

---

Troubleshooting

Symptom	Cause	Fix
ORT silently uses CPUExecutionProvider, Stage 1 ~2 it/s, GPU at 0%	CUDA 13 driver / cu12-only ORT wheel	Install the cu12 nvidia libs listed under Requirements above
vLLM EngineCore aborts with pthread_create … Resource temporarily unavailable	Container/cgroup limits the parent process's pthread count (vast.ai, Docker --pids-limit)	export OPENBLAS_NUM_THREADS=8 MKL_NUM_THREADS=8 OMP_NUM_THREADS=8 NUMEXPR_NUM_THREADS=8 before launch
vllm failed … Engine core initialization failed with CUDA OOM	24 GB card too tight for 32B-dense VLM	Use the default vllm-qwen3vl-30b, or lower VLLM_GPU_MEMORY_UTILIZATION
Model architectures … failed to be inspected on vLLM init	transformers>=5 (incompatible with vLLM 0.20)	pip install 'transformers<5,>=4.51' (now pinned in pyproject.toml)

---

Limitations

Honesty bullet points so you know what to expect:

• Best precision when filenames carry character signal (e.g. visual-novel asset filenames). Falls back to face-clustering otherwise.
• Tested on anime / cartoon / visual-novel art. Real photos work as best-effort but aren't a primary target — expect lower face-detection recall.
• GPU recommended. Tested on RTX 4090 (24 GB). Smaller cards may need --vlm openai or --vlm off to fit.
• Recall < precision in the current release. The pipeline favours clean per-character folders over completeness. A future GUI will make manual merge / rename of low-recall clusters fast.

---

Contributing

See CONTRIBUTING.md. Issues and pull requests welcome.

License

MIT.