v3.5.3 — one JPEG path, bounded memory, no silent fallbacks

v3.5.3 — one JPEG path, bounded memory, no silent fallbacks

Supersedes v3.5.1 and v3.5.2. No API or model changes; two knobs retired with a warning.

What was wrong

• Host memory ratcheted on long-running servers (#33): request buffers freed on 128 worker threads sat in ~80 allocator arenas that never returned them; the server's own trimming only worked with glibc, so a preloaded jemalloc made it a no-op.
• The base64 route decoded on the CPU without saying so whenever the GPU decoder was busy or failed, with slightly different pixels than /ocr/raw; the decoder also let the GPU write straight into pageable heap memory on unified-memory systems (HMM), which stalled the driver under jemalloc.

What changed

• One JPEG path. Every route decodes JPEG on the replica that runs inference, with that replica's own decoder: /ocr/raw, /ocr (base64), /ocr/stream, /ocr/markdown, /infer, /ocr/batch and gRPC. The decoder uses the GPU's NVJPG engine when it has one and falls back to nvJPEG's hybrid GPU backend for bitstreams the engine refuses; the startup line now says which one is in use instead of claiming hardware offload everywhere. The work-thread decoder pool is gone; NVJPEG_DECODERS and --nvjpeg-decoders are still accepted, ignored with a warning, for one release. Layout-only requests take the device path too.
• Faults are errors, not detours. A GPU decoder fault returns 503 GPU_DECODE_FAILED (gRPC UNAVAILABLE, batch slot tag gpu_decode_failed). The host codec decodes only JPEG variants nvJPEG does not support at all (arithmetic, 12-bit, CMYK); when nvJPEG is present the host image decoder refuses JPEG outright, so a detour cannot pass unnoticed. Decoded pixels always land in device memory and are copied back through pinned buffers; the HMM shortcut and NVJPEG_DEVICE_COPY are removed.
• Memory is a budget. Page-sized host images live in a fixed pool of reusable pinned buffers (slots = work threads + 2 × replicas + PDF workers, each capped by MAX_IMAGE_PIXELS_MP, printed at startup); the device memory pool has a 256 MiB release threshold; nvJPEG scratch is allocated through the same pools; /ocr reads the base64 text straight from the request body. The work pool defaults to clamp(pool×4, 16, 64) threads, and the idle-memory reaper detects the allocator and releases freed memory on glibc and jemalloc alike.
• Two integration guards now fail on the class of bug above: /ocr and /ocr/batch must return the same text as /ocr/raw for the same JPEG, and a progressive JPEG must still succeed.

Measured (RTX 5090, tiny, pool 2, same corpus and engine cache)

Endpoint	v3.5.2	v3.5.3
/ocr/raw JPEG, 16 clients	144.9 req/s	144.0 req/s
/ocr base64 JPEG, 16 clients	48.8 req/s	145.0 req/s
/ocr/batch 8 × 100, 4 clients	35.2 img/s	92.8 img/s
/ocr/pdf 80-page scan / 40 × 23 MP pages	794 ms / 1844 ms	793 ms / 1839 ms
/ocr/markdown JPEG	14.9 ms	10.9 ms
/ocr/stream JPEG	14.6 ms	11.0 ms
/ocr/raw?text=0&layout=1	6.4 ms	3.3 ms
/ocr/pixels / PNG via /ocr/raw	11.7 / 6.2 ms	12.1 / 6.8 ms
FUNSD-50 F1 / img/s	85.40 / 431	85.40 / 426
RSS after 2202 base64 requests	2.06 GB	1.44 GB
jemalloc, six base64 bursts, idle floor	+180 MB per burst	+59 MB over all six
VRAM after base64 load	7.4 GB	7.2 GB

Response shapes of /ocr/raw, /ocr, /ocr/batch, /ocr/pdf, /ocr/markdown, /ocr/stream, /ocr/pixels, /health, /health/ready, /capabilities and /metrics are identical to v3.5.2. Unit tests 370, integration 311 pass (2 pre-existing stale expectations), accuracy suite unchanged.

Upgrade notes

• Remove NVJPEG_DECODERS / --nvjpeg-decoders and any HTTP_THREADS=8 workaround; all are unnecessary now (the two decoder knobs are ignored with a warning and go away next release).
• TURBO_OCR_DISABLE_HOST_IMAGE_POOL=1 restores OpenCV's allocator for troubleshooting.
• jemalloc users may additionally set MALLOC_CONF=background_thread:true,narenas:4; it is no longer required.

Images

• GPU (TensorRT, x86_64): ghcr.io/aiptimizer/turboocr:v3.5.3 (also :latest)
• CPU (ONNX Runtime, amd64+arm64): ghcr.io/aiptimizer/turboocr-cpu:v3.5.3 (also :latest, and ghcr.io/aiptimizer/turboocr:v3.5.3-cpu)

Models are unchanged from v3.5.0.