v3.5.2 — bounded nvJPEG decoder pool

v3.5.2 — bounded nvJPEG decoder pool

Patch release, no API or model changes. Supersedes v3.5.1, which was tagged and published as images but never announced: its pooled decoders failed silently on threads that had not bound the CUDA context and the base64 routes fell back to CPU decoding. v3.5.2 fixes that and is the release to use.

The bug

The GPU server decoded JPEGs for the JSON image routes (/ocr with base64, /ocr/batch) on whichever work-pool thread received the request, using one nvJPEG decoder per thread. Each decoder holds ~190 MB of device memory and ~50 MB of host memory until the process exits, and the server starts max(pool*32, 128) work threads. A server fed JPEGs through those routes therefore grew by one decoder per thread until every thread had one: about 24 GB of VRAM and 6 GB of RSS at the defaults, reached slowly at low concurrency (reported as unbounded growth in #33) and within a minute under 16 concurrent clients on a 32 GB card, which exhausted the device and wedged the driver on teardown. The raw-bytes route (/ocr/raw) and PDF processing were never affected: they decode on the replica that runs inference.

The fix

Decoders now come from one bounded pool owned by the server, leased per decode and returned by RAII. The pool holds NVJPEG_DECODERS instances (default: one per pipeline replica, since decode throughput is bounded by inference), all created at startup so the footprint is visible in nvidia-smi from the first second. When every decoder is busy for 2 s the request decodes on the CPU instead of waiting. New knob: NVJPEG_DECODERS / --nvjpeg-decoders (0 = auto), shown in the effective-config dump and the startup line. A decoder handed across threads binds the calling thread to its CUDA device before decoding, and a failed GPU decode is now logged instead of silently falling back.

Measured (RTX 5090, tiny, pool 2, 1101 mixed 4–20 MP JPEGs, 16 concurrent clients, same corpus and engine cache for both binaries)

path	v3.5.0	v3.5.2
base64 /ocr, 32 work threads: decoders / VRAM after / RSS after	35 / 12.6 GB / 2.9 GB	4 / 7.5 GB / 2.0 GB
base64 /ocr, 32 work threads: req/s (p50 / p99)	47.2 (261 / 1799 ms)	48.9 (257 / 742 ms)
base64 /ocr, 128 work threads (default)	VRAM exhausted, driver wedged	4 decoders, 2202 requests at 48.9 req/s, flat
/ocr/raw, 128 work threads: req/s (p50)	145.7 (91 ms)	145.7 (90 ms)
speed matrix, text only / with layout, images/s	431 / 201	447 / 194
PDF, 80 pages / 40 pages at 23 MP	795 / 1849 ms	782 / 1854 ms
FUNSD-50 F1 / img/s (/ocr/raw, PNG)	85.40% / 431.8	85.40% / 431.3

Unit tests: 361 cases pass (7 new for the lease pool, 1 for the config knob). Integration suite: 308 pass, 40 skip, 2 fail; the two failures (test_batch_rejects_text0, test_invalid_mode_fallback) fail identically on v3.5.0 because their expectations predate the v3.5.0 fail-loud validation, and are not behaviour changes. Accuracy suite: identical results to v3.5.0 (same fixtures above and below their floors).

Upgrade notes

• No configuration change is required. Operators who set HTTP_THREADS=8 as a workaround for #33 can remove it.
• Each nvJPEG decoder costs ~190 MB of VRAM; raising NVJPEG_DECODERS above the replica count does not raise throughput.
• Unified-memory hosts (GB10 and similar) count the decoder footprint against the same pool as host RSS; the fix removes the slow climb there as well.

Images

• GPU (TensorRT, x86_64): ghcr.io/aiptimizer/turboocr:v3.5.2 (also :latest)
• CPU (ONNX Runtime, amd64+arm64): ghcr.io/aiptimizer/turboocr-cpu:v3.5.2 (also :latest, and ghcr.io/aiptimizer/turboocr:v3.5.2-cpu)

Models are unchanged from v3.5.0.