b10598

Nemotron-3-Omni fork, v2 release notes

Build: llama-b10598-bin-ubuntu-cuda-13.0-arm64.tar.gz (branch feat/video-on-master, commit 3effb43fc, build 10598).
Previous release: b10104.

The story in one sentence

Image, text and audio run on stock llama.cpp with these files; video and one-pass A/V need this fork.

What changed vs b10104

1. Rebased onto current upstream master

v1 sat on an older base with my own Parakeet FastConformer audio graph. Upstream has since merged its own Parakeet audio support, so v2 rebases the fork onto master at build 10598 and drops my audio graph in favor of upstream's. The fork surface shrinks to what upstream still lacks: the Nemotron video path (temporal patches + EVS token pruning) and one-pass A/V. To be precise about scope: upstream mtmd does carry a video ingestion path for Qwen-VL style models; what it lacks is the Nemotron video graph, which is what this fork provides.

2. Standard audio tensor layout in the converter

The v1 projector GGUF used my layout: batch norm folded into the conv weights and audio adapter tensors named mm.a.mlp.{0,1}. The upstream Parakeet loader expects a.mel_filters, a.window, unfolded conv_norm_mean / conv_norm_var, and mm.a.mlp.{1,2}. The v2 converter emits exactly that layout, which is what makes the audio path stock-compatible: the same projector file loads in upstream llama.cpp and in this fork.

3. Stock-compatible audio, image and text

With the v2 model GGUF and the v2 unified projector, a stock llama.cpp binary runs text, image and audio. The fork binary is only required for --video and for one-pass A/V (a soundtracked clip feeding frames to the vision tower and its audio track to the Parakeet tower in a single prompt).

Two accuracy fixes I carry that upstream nemotron-v2-vl still lacks: position-embedding resampling without align_corners=True, and pinning the exact-erf gelu. Both apply on top of the rebase (commit 10df159c6).

4. The three review fixes

• Feeder-thread SIGPIPE: when ffmpeg exits before consuming an in-memory media buffer, the blocked write used to take SIGPIPE and kill the whole process with exit 141. SIGPIPE is now blocked on the feeder thread only, so the short write is logged instead (f18c9b447).
• Non-streamable mp4: a moov-after-mdat file (the default ffmpeg mux layout) cannot be demuxed through a pipe. File inputs now demux by path; in-memory buffers are scanned and spooled to a temp file owned by the video context when needed, with the spool cleaned up on every exit path (dabc06b65).
• Video gather grouping: every non-clip bitmap keeps a group of its own in the Nemotron clip gather, an audio bitmap never anchors the clip, and the behavior is pinned in test-mtmd-impl (3effb43fc).

Validation

Video graph matches the PyTorch reference to 0.0019%, audio to 0.03%. EVS kept-index set matches the reference exactly with bit-identical gathered rows (test-nemotron-evs, test-nemotron-video). Gate tests on this box passed for image, audio, one-pass A/V, video and text against the known ground truths, and the stock-binary cross-check confirmed image, audio and text on an unpatched upstream build.

Remaining manual steps

1. Cut the GitHub release on VincentKaufmann/llama.cpp-omni from pkg-v2/llama-b10598-bin-ubuntu-cuda-13.0-arm64.tar.gz. The binaries were built at 3effb43fc (build 10598); the branch tip adds docs-only commits on top, so tagging the tip is fine.
2. Upload the v2 unified projector mmproj-nemotron3-omni-f16.gguf to the HF repo alongside the quants.
3. Edit the HF model card: the stock-compat sentence above, the b10598 prebuilt pointer, and the note that v1 projectors do not load in upstream llama.cpp while v2 projectors do.