SSCD copy detection in onnxruntime-web: weight-only int8, an fp32 head for WebGPU, and WASM threads

[Reblus (LaserYGD)]

I run a reverse video search (SauceMatch) and moved the per-frame embedding into the visitor's browser with onnxruntime-web. Sharing what worked and what didn't, in case it saves someone a few evenings.

Setup

• Model: SSCD (Meta's copy-detection ResNet-50, 512-d output after GeM pooling, fc and L2). Input 288x288, ImageNet mean/std.
• Runs in a Web Worker, WebGPU when available, WASM otherwise. Up to 48 frames per search.
• The browser sends only the fp16 vectors to the server for the nearest-neighbour lookup, then a C++/OpenCV keypoint matcher compiled to WebAssembly checks each candidate frame. Phones and weak devices fall back to doing it all server-side.

Quantization: weight-only int8, with a few layers left alone

• Full static int8 was not usable for this: cosine vs the fp32 server model dropped to 0.78 on a dark frame. Static QDQ was worse, 0.69.
• Weight-only int8 with conv1, layer1 and all biases kept in fp32 holds cosine >= 0.9977 against the server on our parity set. The file is 25.5 MB.
• Copy detection is sensitive to small drifts, so we re-run a parity script (browser output vs server output on the same frames) after every export change.

WebGPU: keep the head in fp32

• An fp16 graph gave NaN on WebGPU until GeM pooling, the fc layer and the L2 norm stayed fp32.

WASM threads need cross-origin isolation

• With COOP same-origin + COEP credentialless the WASM backend gets its threads: about 140 ms per frame vs about 420 ms single-threaded on the same machine. Our third-party captcha widget kept working under credentialless.

Static hosting

• The host caps files at 25 MiB, so the model and the WebGPU build of the ORT wasm (26.8 MB) ship in parts, listed in a small manifest with a SHA-256 per part. The worker joins them and keeps them in the Cache API, so a second visit downloads nothing.

How the whole search works: https://saucematch.com/how-it-works (the index is adult video, so the site itself is 18+). Longer write-up on the server side: https://dev.to/laserygd/how-i-built-a-reverse-video-search-engine-on-one-home-pc-o61

Happy to answer questions about the export or the parity checks.