-
Notifications
You must be signed in to change notification settings - Fork 0
screenlens.stub
id: screenlens name: ScreenLens repo_path: ~/Documents/sw30labs/repos/screen-lens-mlx remote_url: https://github.com/sw30labs/screen-lens-mlx.git primary_language: Python framework: LangGraph + oMLX (OpenAI-compatible) + OpenCLIP + ChromaDB ingested: 2026-04-11 last_commit_date: 2026-07-13 category: local-inference-mlx stacks: [mlx, langgraph, rag]
Local video scene intelligence for Apple Silicon (inspired by NVIDIA VSS, rebuilt from scratch). Processes screen recordings through a LangGraph-orchestrated pipeline: smart keyframe extraction, Qwen3.5-VL captioning via a local oMLX server, CLIP embedding, ChromaDB vector search, and LLM summarization — no cloud dependencies. v0.2 adds a reconstruction pipeline that classifies recordings (code / docs / PDF / GUI demo) and rebuilds source, docs, and demo artifacts from what was on screen. Renamed on GitHub to screen-lens-mlx (July 2026) when the CUDA-only fork screen-lens-dgx split off; this stub keeps the stable id screenlens.
LangGraph StateGraph pipeline: Ingest → Caption → Embed → Classify → Plan → Reconstruct → QA (reflection, retry when incomplete) → Save. Search path: query → semantic search → summarize.
- Hybrid keyframe detection: SSIM (scene change) + pHash (visual similarity) + HSV histogram (color) — only distinct screens are captured
-
Vision captioning via oMLX: dedicated
omlx_clientspeaks the OpenAI-compatible VLM API (defaulthttp://127.0.0.1:8000/v1); auth viaMLX_API_KEY/OMLX_API_KEY, model selection viaMLX_MODEL/OMLX_MODEL/--omlx-model; dashboard-style URLs normalized to/v1. Replaced the original in-process mlx-vlm path (mlx-vlm and huggingface-hub dropped from deps — vision inference now sits behind the oMLX HTTP seam) - Fallback captioning: Ollama (llama3.2-vision) for cross-platform use
- Helper modules: transcribe (audio) and OCR alongside the captioner
-
Launcher: idempotent Conda setup + launcher
setup_and_run.sh(SCREENLENS_CONDA_ENVoverrides env name)
- Framework: LangGraph; oMLX server for all VLM/LLM inference (Apple Silicon native)
- Vision-Language: Qwen3.5-VL via oMLX OpenAI-compatible API
- Embedding: OpenCLIP ViT-B-32
- Vector DB: ChromaDB
- CLI: Typer + Rich
langgraph, langchain, langchain-ollama, langchain-community, chromadb, open-clip-torch, opencv-python-headless, scikit-image, Pillow, numpy, torch, tqdm, typer, rich, pydantic. Apache-2.0, v0.2.0.
python, langgraph, mlx, apple-silicon, local-inference, chromadb, video, ocr
- qwen3-vl-python-ocr-script-mlx — shared Qwen3 VL models
- tars-ai — sibling consumer of a local OMLX server over the OpenAI protocol
- screen-lens-dgx — DGX-only fork (vLLM + CUDA, concurrency 2), no oMLX path