Nemotron Diarization: who spoke when, on any hardware

Speaker diarization with nvidia/Nemotron-3-Diarization, served as a REST API with a web UI. Upload a video or audio file and get speaker segments, per-speaker statistics, overlaps and turn-taking, plus RTTM/CSV/SRT/JSON exports.

It runs on whatever hardware is available and picks the fastest option automatically:

If the preferred hardware fails at startup (no driver, wrong kernel, broken CUDA), the server logs why and falls back to the CPU instead of failing. CPU output matches the GPU reference exactly.

# any machine (x86_64 or arm64)
docker run -d --name nemotron-diarization -p 7860:7860 ghcr.io/anand34577/nemotron-diarization:cpu

# NVIDIA GPU (needs the NVIDIA Container Toolkit)
docker run -d --name nemotron-diarization -p 7860:7860 --gpus all ghcr.io/anand34577/nemotron-diarization:cuda

# Rockchip RK3566/68/76/88 board; RKNN_ACCEPT_LICENSE=yes accepts Rockchip's runtime license (see docs/DOCKER.md)
docker run -d --name nemotron-diarization -p 7860:7860 --privileged \
  -e RKNN_ACCEPT_LICENSE=yes -v rknn-runtime:/opt/rknn ghcr.io/anand34577/nemotron-diarization:rknn

Then open http://localhost:7860 for the UI, or http://localhost:7860/docs for the interactive API docs. docs/DOCKER.md covers compose, every environment variable, GPU and NPU host setup, and troubleshooting.

git clone https://github.com/anand34577/nemotron-diarization.git
cd nemotron-diarization

The installers detect your hardware, install the matching PyTorch build, download the model and run a self-check. On a Rockchip board, ./install.sh rknn also downloads the NPU model and, once you accept Rockchip's license, the NPU runtime. All configuration lives in .env (see .env.example); nothing needs to be set for a local run. Details: docs/DEPLOYMENT.md.

curl -F file=@meeting.mp4 http://127.0.0.1:7860/api/v1/diarize                         # JSON
curl -F file=@meeting.mp4 -F format=rttm -o meeting.rttm http://127.0.0.1:7860/api/v1/diarize

{
  "duration": 51.9, "num_speakers": 3, "accelerator": "NVIDIA GeForce GTX 1650 (CUDA)", "real_time_factor": 109.6,
  "speakers": [{"label": "Speaker 1", "talk_time": 13.7, "share_of_speech": 0.384, "segments": 7, ...}],
  "segments": [{"speaker": "Speaker 1", "start": 0.07, "end": 1.32, "duration": 1.25, "overlaps_with": []}, ...],
  "summary": {"speech_time": 35.7, "overlap_time": 0.0, "speaker_turns": 7, "interruptions": 0, ...},
  "transitions": {...}
}

Parameters: threshold, min_segment, merge_gap, include_probabilities, format. Optional API key auth (API_KEY), CORS, and upload/duration limits. Full reference: docs/API.md. Python client: examples/client.py.

• docs/DEPLOYMENT.md: every install method (Windows, Linux, macOS, Docker CPU/GPU/NPU, Rockchip boards), configuration, systemd, nginx/HTTPS, health checks, offline installs, troubleshooting
• docs/DOCKER.md: prebuilt images (cpu, cuda, rknn), docker run and compose, every environment variable, licenses inside the images
• docs/API.md: endpoints, parameters, response fields, errors, curl/Python/JavaScript examples
• .env.example: every setting with its default

• Audio is decoded to 16 kHz mono by ffmpeg (a system install, or the bundled imageio-ffmpeg binary). The model outputs per-speaker activity probabilities every 10 ms. Segments are derived by thresholding, gap merging and minimum-length filtering, with the same rules on the server (API) and in the browser (UI, where the sliders re-segment instantly).
• On the NPU, the heavy part (31-layer transformer encoder + classifier, 99M params) is exported as one fixed-shape graph (684 encoder frames = speaker cache 264 + FIFO 40 + chunk 340 + look-ahead 40), padded and masked, and compiled to FP16 RKNN, and to an experimental hybrid INT8 RKNN (attention scores stay FP16; not accurate enough yet on real recordings). Feature extraction, feature stacking and the Arrival-Order Speaker Cache stay the original transformers code on the CPU, so results match the GPU.
• RKNN validation (x86 simulator, FP16, full pipeline): identical segments to the GPU on the 52 s sample for rk3566/rk3576/rk3588; on the 186 s sample 72/72 segments match, and 0.02 % of 10 ms frames differ at a boundary.

• This project's code: Apache License 2.0. Copyright 2026 Anand. See NOTICE.
• The model nvidia/Nemotron-3-Diarization (NVIDIA), and the RKNN models converted from it: OpenMDW License 1.1. Commercial and non-commercial use are allowed; if you redistribute the model, include that license.
• Rockchip NPU runtime: Rockchip's proprietary RKNN SDK License. It is not distributed here; you download it from Rockchip after accepting that license.
• Docker images: they also contain Debian, FFmpeg, PyTorch, Python packages and, in the cuda image, NVIDIA CUDA libraries, each under its own license. See THIRD_PARTY_NOTICES.md and /licenses inside each image.

Responsible use. Speaker diarization processes recordings of people's voices, which is personal data in many jurisdictions (for example under the GDPR). Only process recordings you are allowed to process: get consent where the law requires it, and follow the recording and data-protection laws that apply to you. The model labels speakers by order of appearance ("Speaker 1", "Speaker 2"); it does not identify who a person is, and should not be used to try. Results can contain errors (for example with recordings of more than eight speakers, the model's maximum, noisy audio or very short turns), so review them before relying on them for decisions about people. The software and the model are provided "as is", without warranty of any kind (Apache-2.0 sections 7-8; OpenMDW-1.1).

NVIDIA, Nemotron and CUDA are trademarks of NVIDIA Corporation; Rockchip and RKNN are trademarks of Rockchip Electronics Co., Ltd. This project is independent and not endorsed by either company.

# Windows/Linux venv with torch + transformers
python conversion/export_onnx.py
# WSL / x86 Linux: venv with rknn-toolkit2==2.3.2, onnx==1.16.2, protobuf==4.25.4, numpy==1.26.4, torch>=2.7 (cpu),
#      transformers==5.19.0, librosa==0.11.0, onnxruntime
python conversion/convert_rknn.py --check
# INT8: calibration set from real audio, then hybrid quantization, then end-to-end scoring in the simulator
python conversion/calib_data.py samples/meeting.wav samples/long.wav
python conversion/convert_rknn.py --quant int8
python conversion/quant_eval.py --platform rk3566