videocortex.stub

---

id: videocortex name: videocortex repo_path: ~/Documents/sw30labs/repos/videocortex remote_url: https://github.com/sw30labs/videocortex.git primary_language: Python framework: TRIBE v2 + nilearn (renderer) / PyTorch (predict extra) category: miscellaneous-research stacks: [cli, neuroscience] ingested: 2026-08-29 last_commit_date: 2026-08-29

videocortex

Purpose

A local research instrument around Meta TRIBE v2: drop a clip (or audio, or text) and see which cortical regions the model's average subject would fire. Encoding only — stimulus in, predicted fMRI out. Not a decoder, not a mind-reader, not your cortex. Built so the model actually runs on a Mac instead of dying on a silent CPU fallback.

Honesty contract

• TRIBE v2 predicts an average subject, at fMRI temporal resolution (TR ≈ 1.49 s), haemodynamically lagged and smoothed. Upstream shifts predictions 5 s into the past; frame i is the response to around i − 5 s.
• Read output as "this clip drives these cortical regions", never as "this is what someone is thinking".
• Colour limits are computed once over the whole run. Per-frame restretch is treated as dishonest (a quiet TR would look as loud as a startling one) and is gated by a test.
• The wrapper is MIT. The model and weights are CC-BY-NC-4.0. meta-llama/Llama-3.2-3B is gated.

Architecture

Six CLI verbs (doctor, fetch, render, draw, overlay, serve) around a split install:

• pip install -e . — renderer only (nilearn / nibabel / matplotlib). draw and the synthetic sample work with no torch.
• pip install -e '.[predict]' — tribev2 + torch. That is the half that actually runs the model (~15 GB of frozen encoders on first use).
• src/videocortex/device.py — CUDA → MPS → CPU. Upstream device="auto" is CUDA-or-CPU and silently means CPU on a Mac.
• src/videocortex/model.py — rewrites the four hard-coded device: cuda fields in the published checkpoint (text / image / audio / video extractors), plus batch_size 8→1 and num_workers 20→0.
• src/videocortex/patches.py — WhisperX CPU compat (float16→int8, pin Python 3.11) and Llama 3.2 GQA on Metal (attn_implementation="eager", float32) so fused SDPA does not LLVM-abort.
• src/videocortex/render.py — nilearn plates, contact sheets, PIP cards. Shared colourbar; Destrieux names on the loudest regions.
• src/videocortex/overlay.py / spin.py — 2×2 PIP on the source video, optional 3D inflated-globe spin. Overlay does not touch the model.
• src/videocortex/web/ — stdlib loopback command deck on :8730 (Host pin + non-loopback refuse), same local-only posture as Cleanroom / Skillspector / STRIDE-Lite.
• videocortex doctor — five-second preflight (python, torch/accelerator, ffmpeg, uvx, fsaverage5, disk, HuggingFace access to the five repos) instead of discovering a miss 20 GB in.

Dependencies

Python ≥3.11 (upstream pin; 3.10 will not resolve). Renderer: numpy, nilearn, nibabel, matplotlib, scipy, pillow, huggingface_hub. Predict extra: tribev2 from facebookresearch + torch ≥2.5.1,<2.7. Dev: pytest. MIT wrapper; model is not.

CI/CD

No GitHub Actions. ./setup_and_run.sh (venv + tests + synthetic sample; --predict / --deck). make sample reproduces the seeded occipital→temporal contact sheet without the 20 GB download. pytest -m "not slow" skips surface rasterisation.

Entities and topics

• videocortex-spark — DGX Spark / GB10 sibling (CUDA 13, UMA, export + sonify; same honesty contract)
• screenlens — the org's other video-in instrument (captions / search, not cortex)
• cleanroom — same loopback command-deck posture
• Upstream: TRIBE v2 (d'Ascoli et al., Meta FAIR Brain & AI, 2026; arxiv:2605.04326)
• Tags: [python, apple-silicon, video, cli, neuroscience, fmri]