Language-grounded scene understanding for physical AI.
Ask an aerial scene a question in plain language and get back the exact pixels that answer it. A perception primitive for embodied agents (drones, robots, autonomous systems), demonstrated on drone-captured scenes.
Ask a scene a question and the bridge answers in about a second, drawn as terrain: contours mark equal confidence, you set the waterline. Held-out frames are scored against the annotation on the spot, and you can put the zero-shot baseline on the same query to see both answers together.
The Hugging Face Space runs both models side by side on one query, and is the API the app calls.
A perception model answers only the questions someone fixed at annotation time. A class list is decided months before deployment, and every new question costs another labeling round. Outside a warehouse, the questions are not enumerable in advance.
Voxae makes the question an input instead of a training decision.
Three query families:
- Referring: "the building with the red roof"
- Affordance: "where could a small drone land safely?"
- Metric: "is this gap wide enough for a 2.5 m vehicle?"
image + query
│
▼ vision-language model (Qwen2-VL-2B, LoRA)
hidden state at a <SEG> token
│
▼ 2-layer MLP projector
SAM 2.1 sparse prompt embedding
│
▼ mask decoder
binary mask
The language model never describes a region in words. Its hidden state at a special <SEG> token is projected directly into SAM 2.1's prompt space, so the gradient from a wrong pixel flows back through the projector and into the language model. Trained end to end with LoRA under cross-entropy + BCE + Dice.
A zero-shot compose baseline ships alongside it: a hosted VLM emits a bounding box and points as text, SAM 2.1 turns them into a mask, no training. It is the floor the trained bridge has to beat, and both run on the same query so the difference is visible rather than described.
The two run independently. The bridge is one forward pass and answers in about a second; the baseline waits on a third-party API and has been measured between 5 and 140 seconds, so the comparison is requested on its own rather than bundled into every query. That gap is one of the results.
Full details: docs/architecture.md.
Embodied AI / robotics is the use case today. A ground or aerial robot asking its own affordance questions at runtime ("where can I land," "what's blocking this path") instead of relying on a class list fixed before deployment is exactly what Voxae-Reason was built and evaluated on.
Looking forward, swapping verticals is a data problem, not an architecture problem: the same generate, train, evaluate pipeline reruns unchanged on a different source dataset and query set. Two adjacent examples:
- Construction and infrastructure inspection. Drone survey footage queried for site-specific risks (clearance, access, encroaching vegetation) without retraining a detector per site or per question.
- Disaster response and search. Fast, ad hoc queries over aerial imagery of an unfamiliar scene, where the questions that matter are not known in advance.
Held-out test split, 306 samples, both predictors on identical inputs.
| gIoU trained | gIoU zero-shot | cIoU trained | cIoU zero-shot | |
|---|---|---|---|---|
| all | 0.414 | 0.371 | 0.502 | 0.307 |
| referring | 0.399 | 0.538 | 0.393 | 0.494 |
| affordance | 0.421 | 0.290 | 0.511 | 0.290 |
The split is the result. A hosted VLM emitting a box is strong at naming an object and weak at reasoning about what a query implies: it wins referring by 26% and loses affordance by 45%. A single box cannot express a disjoint or proximity-constrained region; a projected embedding can.
| trained | zero-shot | |
|---|---|---|
| latency / query | 0.25 s | 20.8 s |
| unparseable responses | 0 | 1 / 306 |
Inference is one teacher-forced forward pass with no autoregressive generation. The baseline pays an API round trip plus a full SAM encode, and can emit output that fails to parse at all, a failure mode an embedding handoff does not have.
The gap survives deployment. Measured against the hosted Space over the public internet, a bridge query returns in 1.0 s warm and 3.8 s cold, including image upload and mask transport. The same request with the baseline attached has been measured between 36 s and 143 s, entirely dependent on how the third-party grounding API happens to be behaving.
Three affordance queries selected mechanically by scripts/make_figure.py: trained IoU at or above 0.5, zero-shot IoU at or below 0.3, ranked by margin. 44 test-split affordance samples meet that criterion; these are the top three. Aggregate figures for all 306 samples are in the table above.
3,062 queries over 614 drone images, split 2,464 / 292 / 306 by image.
The generation rule is the point: the language model never draws a pixel. It authors the query text and a symbolic target specification (class union, connected-component ids, proximity exclusion, area floor). The mask is then materialized from the source dataset's ground-truth annotations. Labels are therefore reproducible and traceable, never hallucinated.
A rule-based QC gate rejects degenerate samples: trivial targets, unconstrained spatial phrasing, over-reused templates, area out of bounds. A blind human audit of 100 samples scored 0.97 precision.
Provenance, licenses, and known limitations: docs/datasheet.md.
Stated plainly, because they affect what the numbers mean.
- No metric family in the current dataset. All three source datasets ship with EXIF stripped, so ground-sample distance cannot be recovered and no real-world-dimension queries could be generated. Metric grounding remains the design target; it is not in these results.
- In-distribution results. Results and demo examples use drone imagery matching the training distribution, which assumes a downward view at comparable altitude. Performance on other aerial photography is untested.
- Resolution ceiling. Training images are downscaled, so predicted mask boundaries are limited by the training resolution rather than by the model. This applies identically to the baseline.
- One backbone, one scale. Results are for a 2B model with LoRA. Nothing here establishes how it scales.
git clone https://github.com/nhipixel/voxae && cd voxae
uv sync --extra dev # core + tests (CPU, no weights)
uv run pytest # verify
cp .env.example .env # add VOXAE_VLM_API_KEY for live grounding
uv sync --extra ml --extra demo # full local demo
uv run python app.py # Gradio UISet VOXAE_CHECKPOINT_REPO to a Hub model repo (or VOXAE_CHECKPOINT_DIR to a local path) to enable the comparison view; without it the demo runs the baseline alone.
Trained weights: nhibuilds/voxae-checkpoint.
The app lives in web/ and is a client of the Space, not a second
copy of the model. A route handler holds the token server side and calls one JSON endpoint; the response
carries the probability field and masks as PNGs, around 60 KB, so every overlay and every number that
moves with the waterline is computed in the browser from a single request:
cd web && npm install
cp .env.example .env.local # HF_TOKEN and VOXAE_SPACE_URL
npm run devThe same endpoint is callable from anything:
from gradio_client import Client, handle_file
r = Client("nhibuilds/voxae").predict(
image=handle_file("frame.png"),
query="where could a small drone land safely?",
threshold=0.5,
include_baseline=False, # True also runs the zero-shot comparison
api_name="/predict",
)
print(r["trained"]["latency_s"], r["ground_truth"])Training and evaluation run from YAML configs:
python -m voxae.train.train --config voxae/train/configs/full_2b.yaml
python -m voxae.eval.run_eval --split test --predictor trained \
--checkpoint outputs/full_2b/latest --backbone Qwen/Qwen2-VL-2B-InstructReasoning segmentation exists. LISA introduced the <SEG> embedding-as-prompt idea, and it has been applied to satellite imagery (SegEarth-R1) and drone imagery (PixDLM, RIS-LAD). The bridge architecture here is not new and is not claimed as such.
What Voxae contributes:
- Symbolic target specification. Separating language generation from geometry means every label traces back to a ground-truth annotation. Most LLM-generated segmentation data cannot make that claim.
- A single-GPU-budget reproducible pipeline. Dataset generation, training, evaluation, and demo, end to end, on hardware a student can afford.
- Metric grounding as the next layer. Querying a reconstructed 3D scene so answers return in meters rather than pixels.
Code: Apache-2.0. Dataset annotations: CC BY-NC-SA 4.0 (inherits non-commercial seed-data licenses; see the datasheet). Demo gallery images: openly licensed CC0/CC BY, attribution in voxae/demo/assets/ATTRIBUTION.md.