Skip to content

Clarify NVENC live view startup failures - #19

Merged
robertjamesprior merged 2 commits into
masterfrom
hypeship/kernel-2101-neko-nvenc-error-main
Sep 4, 2026
Merged

Clarify NVENC live view startup failures#19
robertjamesprior merged 2 commits into
masterfrom
hypeship/kernel-2101-neko-nvenc-error-main

Conversation

@robertjamesprior

@robertjamesprior robertjamesprior commented Sep 3, 2026

Copy link
Copy Markdown

Summary

CreatePipeline returns the raw GStreamer message when pipeline construction fails. On GPU builds, a failed NVENC/CUDA init surfaces as no element "nvh264enc", which reads like a missing plugin even though the encoder is installed and the real limit is GPU memory.

annotatePipelineError appends a short hint when all three hold:

  • the pipeline string asks for nvh264enc
  • the error message names nvh264enc
  • the error is a missing element/plugin error

Every other pipeline error passes through unchanged.

Test

go test ./pkg/gst -run TestAnnotatePipelineError -count=1

The new test covers both missing-element wordings and one unrelated encoder error. Building the full pkg/gst package needs the GStreamer dev headers, so run it where they are installed.


Note

Low Risk
Changes affect error messaging and lock ordering on failed pipeline creation only; successful capture paths are unchanged aside from linking -ldl for the CUDA probe.

Overview
When GPU live-view pipeline creation fails with a missing nvh264enc element/plugin error, CreatePipeline now appends a diagnostic hint instead of returning only the raw GStreamer message. A new gstreamer_cuda_context_probe (dynamic libcuda load, init, device count, create/destroy context) reports whether CUDA is missing, misconfigured, out of memory, or healthy—so operators can tell GPU/CUDA limits from a truly absent plugin.

annotatePipelineError only runs when the pipeline string and error both reference nvh264enc and match known “no element” wordings; all other errors are unchanged. On the failure path, pipelinesLock is released before the probe so a slow CUDA check cannot block other pipelines.

Tests cover NVENC error detection, lock release during probing, and the various nvencFailureDetail message branches.

Reviewed by Cursor Bugbot for commit 7ae62fc. Bugbot is set up for automated code reviews on this repo. Configure here.

@robertjamesprior
robertjamesprior marked this pull request as ready for review September 3, 2026 15:20

@sjmiller609 sjmiller609 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error returned could currently return for a few different reasons. It would be improved if we can surface unambiguous errors, so that we don't show the user an out-of-memory error for a potentially different problem.

Please merge this PR in, then adjust related PRs if needed to handle the more specific errors. Thank you #20

## summary

- probe the CUDA driver immediately when GStreamer cannot create
`nvh264enc`
- distinguish GPU memory exhaustion, missing GPUs, unavailable driver
symbols, and non-CUDA NVENC/plugin failures
- load `libcuda.so.1` at runtime so non-GPU images retain no CUDA build
dependency, while keeping initialized CUDA process state mapped safely
- release the global pipeline lock before running the diagnostic so a
slow CUDA driver cannot stall existing streams or unrelated pipelines

## testing

- `go test ./...`
- `go test -race ./pkg/gst -count=1`
- `go build -o /tmp/neko ./cmd/neko`
- validated dynamic probing against fake CUDA drivers returning success,
`CUDA_ERROR_OUT_OF_MEMORY`, and `CUDA_ERROR_NO_DEVICE`
- validated on a real GPU browser: healthy CUDA and NVENC pipeline
creation passed; forced GPU-memory exhaustion produced GStreamer's
missing-`nvh264enc` error and the detailed `CUDA_ERROR_OUT_OF_MEMORY
(2)` diagnosis; the slow-probe lock regression passed

---------

Co-authored-by: sjmiller609 <7516283+sjmiller609@users.noreply.github.com>
@robertjamesprior
robertjamesprior merged commit 00be612 into master Sep 4, 2026
4 checks passed
@robertjamesprior
robertjamesprior deleted the hypeship/kernel-2101-neko-nvenc-error-main branch September 4, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants