loraprep

Turn a folder of photos into a clean, deduplicated LoRA training set — entirely in your browser.

Live: https://loraprep.benrichardson.dev

---

what it is

Half of a LoRA training set is usually the same photo four times. Burst frames off a phone, a shot re-saved at a lower quality, a thumbnail that came back from someone's email, three screenshots that got mixed in. Cleaning that up by hand is most of the work of preparing a dataset and none of the interesting part — and the alternative on offer is renting a cloud GPU to run a Python script you did not write, on photographs you would rather not upload.

Loraprep does it in a browser tab. Drop a folder (or a .zip, or drag a folder onto the page), and it decodes every image once, fingerprints it, groups the near-identical frames, picks the best copy from each group, and hands back a ZIP laid out for kohya_ss / sd-scripts and Hugging Face imagefolder — plus a report naming every image it set aside and the file it was judged against.

Nothing is uploaded. Nothing on your disk is touched: Loraprep only decides what goes into an export. Faces, client photography and unreleased product shots are exactly what goes into a LoRA set, and exactly what you cannot hand to a web service.

how it works

folder / zip / drag
        │
        ├─► decode workers ──► 256×256 grid ──► dHash-64, sharpness, flatness, dimensions
        │                            │
        │                            └──► inference worker ──► MobileCLIP-S0 ──► 512-d embedding
        │
        ├─► union-find over dHash edges  ─────────► DUPLICATE GROUPS (auto)
        └─► ranked cosine pairs          ─────────► LOOK-ALIKES (review only)
                                                          │
                                                    you decide
                                                          │
                                          ZIP + dedup-report.csv + embeddings.f32

the design decision that matters

Only the perceptual hash is allowed to set an image aside. The vision model never is.

The obvious build is: embed everything, threshold the cosine, cluster, keep one per cluster. Every "AI image deduplicator" works that way. It was measured against google/dreambooth — the canonical LoRA subject set, 158 hand-curated photographs with no duplicates in it — and it does not work:

	cosine
Different photos of the same subject (must keep)	median 0.80, p99 0.97, max 0.9916
Different subjects entirely	median 0.24, max 0.92
True duplicate, 2× upscale	0.981
True duplicate, re-encoded q40	0.956
True duplicate, mirror flip	0.878
True duplicate, greyscale	0.855
True duplicate, watermarked	0.726

The distributions overlap almost entirely, and in the wrong direction. At a 0.90 cutoff you miss flips, rotations, greyscale, crops and watermarks and falsely condemn 49 of 342 same-subject pairs. At 0.98 you catch one of thirteen duplicate families — the one the hash already finds at a distance of 1 bit. Clustering is worse: at 0.90, single-linkage collapsed six visually distinct photographs of one bowl into a single survivor, through a chain containing a pair scoring 0.809.

So the embedding ranks pairs for a human to look at, with nothing pre-selected, and the perceptual hash — measured at zero false matches across all 12,403 pairs of that same 158-image library at a Hamming distance of 5 — does the automatic part.

the second decision that matters

The keeper is chosen on two signals. Variance of the Laplacian measures high-frequency energy, and JPEG blocking is high-frequency energy: across three copies of one photograph at quality 92, 82 and 76, it scores 40, 53 and 73. A sharpest-wins rule keeps the most damaged copy every time. A pre-filter before the Laplacian was tried and does not fix it — blocking is structural, not noise.

So sharpness decides only what is genuinely in focus (a blurred or downscaled frame scores about a tenth of its sharp sibling), and among the in-focus frames the one with the most bytes per pixel wins. Two frames off the same camera at the same quality land in the same compression bracket and are decided on sharpness, which is what you want for a real burst.

the model checks itself before it is trusted

The 11.8 MB int8 export of MobileCLIP-S0's vision tower is numerically broken: cosine 0.09 against its own fp32 output, and four unrelated photographs come back with pairwise cosines of 0.73–0.88 where fp32 gives 0.08–0.17. Every image looks like every other image. Nothing throws; the shapes are right; the tool would confidently destroy a dataset.

Loraprep ships fp16 (22,876,479 B, cosine 0.99999 to fp32) and does not take its own word for it: at startup it embeds a fixed synthetic image, generated in code, and compares the result against a reference vector committed to this repository. Below 0.985 the backend is rejected. That is also what makes it safe to try WebGPU first and fall back to WebAssembly — a graphics backend that quietly returns wrong numbers is worse than a slow one.

browser APIs used

• Web Workers (module) — a decode pool and a separate inference worker. They must not share a thread: measured on two Chrome builds, alternating createImageBitmap with session.run in one loop degrades inference from 138 ms to ~1,050 ms.
• createImageBitmap + OffscreenCanvas — decode once to a fixed 256×256 grid, close() immediately. The final reduction to 9×8 is done in first-party JS, not by the browser's resizer, because resizeQuality: 'low' vs 'high' moves the hash by up to 21 bits on hard-edged content.
• WebGPU / WebAssembly via onnxruntime-web — WebGPU first, verified by the self-check, WASM otherwise. GitHub Pages cannot set COOP/COEP, so WASM is single-threaded; that is stated in the UI.
• Transferable ArrayBuffers — 256 KB pixel buffers move between workers without copying.
• File System Access API — showDirectoryPicker to read a folder, showSaveFilePicker + createWritable to stream the ZIP straight to disk, where available.
• DataTransferItem.webkitGetAsEntry — folder drag-and-drop, the route with the broadest support. <input webkitdirectory> and a dropped .zip are the fallbacks.
• Cache API — the model is stored after the first run, so the tool then works offline.
• Web Share API (level 2, files), Clipboard API, Service Worker (installable PWA).

security / privacy model

Protected

• Every image is decoded, measured, embedded and packaged in the browser tab. There is no endpoint in this application that receives a photograph.
• The vision model runs on your machine. No inference API, no account, no per-image cost.
• No cookies, no fingerprinting, no third-party fonts. One localStorage key, loraprep.theme.

Not protected

• The model is fetched from Hugging Face on first use — three files, 22,877,205 B in total, pinned to an immutable revision. Hugging Face sees your IP and that you asked for those files. It sees nothing about your images, because nothing about them is in that request.
• The exported ZIP contains your original image bytes unmodified, including whatever EXIF they carry — GPS, camera serial numbers, timestamps. Loraprep does not strip metadata.
• dedup-report.csv and metadata.jsonl contain your original file paths, which frequently name people, clients and projects.

Trust model

• The static bundle served by GitHub Pages and the TLS chain to it.
• The pinned model file, checked at startup against a reference embedding committed here.
• The ONNX Runtime WebAssembly binaries are served from this origin. transformers.js defaults to fetching them from cdn.jsdelivr.net; scripts/copy-ort.mjs and an explicit wasmPaths override stop that, which is why the CSP names no CDN and why the tool works with the network off.
• A Cloudflare Web Analytics beacon records anonymous page views — no cookies, no fingerprinting, no cross-site tracking; your files are never sent to it.

stack

• Vite 7 + vanilla TypeScript, Vitest, GitHub Pages via GitHub Actions
• @huggingface/transformers 4.2 (MobileCLIP-S0 vision tower only) and fflate for ZIP in and out

Everything else — the perceptual hash, the sharpness and flatness measures, the union-find grouping, the keeper rule, the JPEG/PNG/WebP/GIF/BMP header parser, the streaming ZIP writer, the dataset layout and the report writer — is first-party and unit-tested (171 tests).

local development

npm install
npm run dev      # vite dev server on :5173
npm test         # run vitest suite
npm run build    # produce dist/ for deploy
npm run preview  # serve dist/ locally

Two generators are dev-only and their output is committed, so CI never needs ImageMagick or a model download:

npm run sample                    # rebuild public/samples/lora-set.zip (needs ImageMagick)
node scripts/make-reference.mjs   # recompute the self-check reference vector

deploying

A push to main triggers .github/workflows/deploy.yml, which runs tests, builds, and deploys dist/ to GitHub Pages. The custom domain is set via public/CNAME — point a CNAME DNS record for loraprep.benrichardson.dev at ben-gy.github.io.

license

GNU Affero General Public License v3.0 or later, with an attribution requirement added under section 7(b) — see ADDITIONAL-TERMS.md.

In short: you may run, modify, redistribute and even sell this, but if you distribute it — or run a modified version where other people can reach it — you have to publish your source under the same licence and keep the attribution. A separate commercial licence without those obligations is available on request: hi@ben.gy.

The datasets Loraprep produces are yours. Nothing about the AGPL reaches your exported ZIP.

Third-party components keep their own licences — see THIRD-PARTY-NOTICES.md. MobileCLIP-S0 is downloaded at runtime and is not covered by this licence; read its model card before using it commercially.