-
Notifications
You must be signed in to change notification settings - Fork 0
local inference dgx
Nicolas Cravino edited this page Jul 22, 2026
·
1 revision
id: local-inference-dgx title: Local Inference & DGX Spark tags: [python, cuda, vllm, llm, local-inference, langgraph, chromadb, video, cli, qwen] created: 2026-07-22 updated: 2026-07-22 freshness: fresh
The CUDA counterpart to local-inference-mlx: repositories whose product is local inference on NVIDIA DGX Spark (GB10, Linux aarch64). Two live here today — a ternary-quantized 27B chat stack on a custom llama.cpp fork, and the DGX-only fork of the ScreenLens video-intelligence pipeline. Purpose-specific DGX ports (driftlab-dgx, tslit-dspy-dgx, ralph-dgx, nemotron-parse-spark) stay with their sibling categories and are tracked cross-cuttingly via the dgx stack in Sitemap-Stacks. The recurring host pattern: one shared local vLLM server on :8000 (Qwen3.6-27B-FP8 or Qwen3-Coder-Next-FP8) serving several projects, plus a shared HF model cache under ~/models/huggingface.
- Ternary Bonsai 27B runs the Prism ML ternary-quantized 27B chat model (~7.2 GB Q2_0 GGUF) on DGX Spark via the PrismML llama.cpp CUDA fork, fronted by
llama-server(OpenAI-compatible, :8080) and a Textual TUI with thinking-stream display. ^src:bonsai-ternary-27b-dgx - Model weights stay out of the repos: DGX projects share one HF cache (
~/models/huggingface), the same convention across the Desktop DGX fleet. ^src:bonsai-ternary-27b-dgx - ScreenLens-DGX is the DGX-only fork of dual-platform screen-lens: identical LangGraph pipeline (SSIM + pHash + HSV keyframing → captioning → OpenCLIP → ChromaDB → reconstruction), but captioning always targets vLLM :8000 with Qwen3.6-27B-FP8 at concurrency 2 — the macOS/oMLX path is removed entirely. ^src:screen-lens-dgx
- ScreenLens-DGX ships a Docker path (
compose.dgx-spark.yaml) alongside the bare-metal setup script — the first containerized deployment in the ScreenLens family. ^src:screen-lens-dgx
- Will the remaining MLX-only media wrappers (LTX, Sulphur, LongCat) grow DGX siblings — and if so, does a shared vLLM multimodal endpoint replace their per-app runtimes?
- Could the dual-platform split (mlx/dgx sibling repos) converge on a single repo with a provider-neutral inference seam instead of two trees?
- local-inference-mlx (the Apple Silicon half of every dual-platform pair)
- oscal-compliance (driftlab-dgx carries the runtime compliance observer to this host)
- ai-security-pentesting (tslit-dspy-dgx runs its detection brain on the shared vLLM stack)
- agentic-frameworks (ralph-dgx points DeepAgents Code CLI at the same server)
- developer-tools (nemotron-parse-spark was the org's first DGX Spark repo)