fix(cuda): preallocation memory leak + new vmaf_cuda_state_free API (Netflix#1300, ADR-0157)

[lusoris]

Summary

Addresses Netflix upstream issue #1300 ("CUDA-VMAF Memory Leak (preallocation method) in libvmaf", OPEN since 2024).

Root causes (four framework-side leaks confirmed via ASan):

1. VmafCudaState heap allocation had no public free. vmaf_cuda_state_init() mallocs the struct; vmaf_cuda_import_state() copies-by-value; vmaf_close() → vmaf_cuda_release() frees the internals + memsets, but never the heap allocation itself. No public API to free it.
2. CudaFunctions driver table never freed. vmaf_cuda_state_init dlopens libcuda.so via cuda_load_functions(); vmaf_cuda_release never called cuda_free_functions().
3. vmaf_ring_buffer_close() destroyed a locked pthread_mutex — POSIX UB.
4. Cold-start leak in init_with_primary_context() — retained primary context not released on cuStreamCreateWithPriority failure.

Fixes:

• New public API vmaf_cuda_state_free(VmafCudaState *cu_state) — NULL-safe free() wrapper. Must be called AFTER vmaf_close(). Mirrors the SYCL backend's vmaf_sycl_state_free() pattern.
• vmaf_cuda_release saves CudaFunctions* before the existing memset, then calls cuda_free_functions() via the saved local. Order is load-bearing.
• vmaf_ring_buffer_close now does unlock → destroy → free(pic) → free(rb).
• init_with_primary_context cold-start unwind releases the retained primary context; outer vmaf_cuda_state_init failure-unwind frees c + c->f.

Scope

• libvmaf/include/libvmaf/libvmaf_cuda.h — new vmaf_cuda_state_free() declaration.
• libvmaf/src/cuda/common.c — new implementation + cuda_free_functions() call in release + cold-start unwinds.
• libvmaf/src/cuda/ring_buffer.c — unlock + destroy before free.
• libvmaf/test/test_cuda_preallocation_leak.c — new 10-cycle GPU-gated reducer with full cleanup.
• libvmaf/test/test_cuda_pic_preallocation.c, test_cuda_buffer_alloc_oom.c — add missing vmaf_cuda_state_free() + vmaf_model_destroy() after vmaf_close().
• libvmaf/test/meson.build — register new reducer under enable_cuda.

Test

• meson test -C libvmaf/build-cuda → 40/40 pass (was 39; + new reducer).
• meson test -C build (CPU-only) → 35/35 pass.
• ASAN_OPTIONS='detect_leaks=1:leak_check_at_exit=1' build-asan-cuda/test/test_cuda_preallocation_leak → 183 bytes in 4 allocations, all inside libcuda.so.1 (cuInit's process-lifetime driver cache — does NOT grow per cycle; verified N=1 vs N=10 produces identical byte counts). Zero libvmaf/src/* frames in any leak trace.
• clang-tidy -p libvmaf/build-cuda --quiet <5 touched files> → exit 0.
• CI-equivalent clang-tidy -p build --quiet libvmaf/include/libvmaf/libvmaf_cuda.h (the only CI-visible file post-ADR-0156 exclusion) → exit 0.
• pre-commit run --files <touched> → all hooks pass.

Type

• fix — bug fix
• feat — new public API (vmaf_cuda_state_free)

Checklist

• Commits follow Conventional Commits.
• Pre-commit hooks green locally.
• Unit tests pass: meson test -C libvmaf/build-cuda → 40/40, meson test -C build → 35/35.
• ASan confirms zero framework-side leaked bytes across 10 cycles.
• Reducer verified fail-without-fix / pass-with-fix.
• Visible behaviour change flagged under ### Added + ### Fixed in CHANGELOG.
• ADR-0122 / ADR-0123 / ADR-0156 preserved verbatim.

Netflix golden-data gate (ADR-0024)

• I did not modify any assertAlmostEqual(...) score in the Netflix golden Python tests.
• CUDA ownership-API change only; CPU numerical paths untouched.

Cross-backend numerical results

N/A — memory-management refactor; no CUDA kernel arithmetic changed.
Netflix golden CPU pair bit-identical (35/35 CPU tests pass pre- and post-PR).

Deep-dive deliverables (ADR-0108)

• Research digest — no digest needed: upstream issue links directly from the ADR; direct fix.
• Decision matrix — ADR-0157 §Alternatives considered (4 options: full fix / surgical / implicit ownership / document).
• AGENTS.md invariant note — libvmaf/src/cuda/AGENTS.md under "Rebase-sensitive invariants".
• Reproducer / smoke-test command — pasted below.
• CHANGELOG.md "lusoris fork" entry — under ### Added + ### Fixed.
• Rebase note — entry 0050 in docs/rebase-notes.md.

Reproducer

Verify the fix is live:

cd libvmaf
meson setup build-cuda -Denable_cuda=true -Denable_sycl=false
ninja -C build-cuda
meson test -C build-cuda test_cuda_preallocation_leak -v
# Expect: test_cuda_preallocation_leak_reducer passes (10 cycles clean).

ASan verification:

cd libvmaf
meson setup build-asan-cuda -Db_sanitize=address -Denable_cuda=true -Denable_sycl=false --buildtype=debug
ninja -C build-asan-cuda
ASAN_OPTIONS='detect_leaks=1:leak_check_at_exit=1' build-asan-cuda/test/test_cuda_preallocation_leak
# Expect: "SUMMARY: AddressSanitizer: ~183 byte(s) leaked in 4 allocation(s)."
# ALL 4 allocations are inside libcuda.so.1 (cuInit process cache).
# ZERO entries with "in vmaf_" or "src/" frames.

Issue-matching real-world repro (requires actual video frames):

for (batch = 0; batch < N; batch++) {
    VmafContext *vmaf; VmafCudaState *cu_state; VmafModel *model;
    vmaf_init(&vmaf, cfg);
    vmaf_cuda_state_init(&cu_state, cu_cfg);
    vmaf_cuda_import_state(vmaf, cu_state);
    vmaf_model_load(&model, &mcfg, "vmaf_v0.6.1");
    vmaf_use_features_from_model(vmaf, model);
    vmaf_cuda_preallocate_pictures(vmaf, pic_cfg);
    for (i = 0; i < 30; i++) { /* process 30 frames */ }
    vmaf_close(vmaf);
    vmaf_cuda_state_free(cu_state);  /* NEW: required step */
    vmaf_model_destroy(model);
}
/* Pre-fix: GPU memory rises every iteration.
 * Post-fix: GPU memory flat. */

Migration guide for existing callers

Every caller that does:

vmaf_cuda_state_init(&cu_state, cfg);
vmaf_cuda_import_state(vmaf, cu_state);
... use ...
vmaf_close(vmaf);
/* cu_state pointer was leaked before */

must now add the free:

vmaf_cuda_state_init(&cu_state, cfg);
vmaf_cuda_import_state(vmaf, cu_state);
... use ...
vmaf_close(vmaf);
vmaf_cuda_state_free(cu_state);  // NEW: required AFTER vmaf_close()

Order is load-bearing: vmaf_close first (tears down the CUDA stream + ctx via the struct copy), then vmaf_cuda_state_free (releases the heap allocation). Reversing is a use-after-free. Callers who were using free(cu_state) directly must switch — mixing in-tree free() with the new API will double-free.

Known follow-ups

• ffmpeg filter libavfilter/vf_libvmaf.c should be audited for the same cleanup sequence during the next ffmpeg-patches refresh — noted as follow-up in ADR-0157.
• Self-hosted GPU runner (backlog T7-3) would give test_cuda_preallocation_leak real CI coverage instead of SKIP on the driver-probe step.

🤖 Generated with Claude Code