FLORA

Flora turns robot and human manipulation videos into labels for robot learning: when each action happens, what it is, and how well it went.

• Flora-TempSeg splits a video into actions.
• Flora-Captioner writes an instruction for each action.
• Flora-Rewarder judges an episode: did it succeed, where did it slip, and why.

This repository holds the inference code. The weights are on Hugging Face; there is no training code here.

• 2026-10 Flora is released: weights, paper, project page and this inference code.

This repository provides sample vanilla inference code for the released weights, not the production inference system. The examples are intended as a straightforward starting point for running and integrating FLORA.

For large-scale workloads, throughput can be improved with optimized inference frameworks, dynamic batching, asynchronous video decoding and preprocessing, and queue-based scheduling across persistent GPU workers. These optimizations are left to downstream deployments; the sample scripts' runtime does not represent production throughput.

Both tempseg/ and captioner/ contain the complete segmentation-to-caption workflow. The annotation examples below use captioner/.

git clone https://github.com/NVIDIA/Flora.git && cd Flora

The annotation pipeline and the Rewarder pin different PyTorch builds, so each gets its own environment. Weights download from Hugging Face on first use. Validated on Linux (x86_64) with Python 3.12 and NVIDIA H100 80 GB GPUs.

python3.12 -m venv .venv-annotation && source .venv-annotation/bin/activate
pip install torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r captioner/batch_utils/requirements.txt

python captioner/infer.py --video my_video.mp4 --profile human --validate-only   # CPU check, no download
python captioner/infer.py --video my_video.mp4 --profile human --output my_video.json

The output is one JSON file per video, with times in seconds from the start of the video. For the guitar-repair video in the film (107 s, 20 actions), two of the actions, both matching the human annotation (abridged):

{"schema_version": "flora_demo_annotation_v1", "duration_s": 106.933333, "profile": "Human",
 "segments": [
  {"start_s": 56.5, "end_s": 58.1, "caption": "pick up guitar string with both hands", "status": "complete"},
  {"start_s": 58.1, "end_s": 63.6, "caption": "untangle guitar string with both hands", "status": "complete"}
 ]}

This sample code favors clarity over speed: it loads both models for every video and reads one window at a time. The guitar video took 23 minutes on one H100, including the first download.

Robot with three synchronized cameras:

python captioner/infer.py --video main.mp4 --left-wrist left_wrist.mp4 --right-wrist right_wrist.mp4 \
  --profile yam --output robot.json

• Profiles: human for egocentric human video (one camera), yam for parallel grippers, dex for dexterous hands (tianji is a variant for Tianji robots). Robot profiles take one camera or all three.
• Limits: at most 120 s, 256 MiB and 4096 px per side per video, at 4 FPS or more. Split longer episodes. Wrist videos must share the main video's frame timestamps.
• Many videos: python captioner/infer.py --input-jsonl jobs.jsonl --output-dir annotations/ (job format).
• Storage: about 130 GB of weights, in ~/.cache/flora/models and the Hugging Face cache; set FLORA_MODEL_CACHE and HF_HOME to move them.

python3.12 -m venv .venv-rewarder && source .venv-rewarder/bin/activate
pip install -r rewarder/batch_utils/requirements.txt

# Did it succeed? One score from 1 (failure) to 5 (success), with a one-line reason.
python -m rewarder.batch_utils.terminal_reward --model nvidia/Flora-Rewarder \
  --video episode.mp4 --task "Put the block in the silver bowl" --with-rationale --output outcome.jsonl

# Where did it slip? A score every 0.5 s (3 normal, 2 suboptimal, 1 failure) with reasons.
python -m rewarder.batch_utils.dense_reward --model nvidia/Flora-Rewarder \
  --video left_wrist.mp4 top.mp4 right_wrist.mp4 --task "Put the cup on the tray." --output dense.jsonl

# Score each step of your own rubric (2 GPUs).
python -m rewarder.batch_utils.rubric_reward --model nvidia/Flora-Rewarder \
  --jobs rubric_jobs.jsonl --output rubric.jsonl

Each command writes one JSON line per episode. The outcome for the example episode (abridged; the human label is also 5):

{"id": "episode", "task": "Put the block in the silver bowl", "score": 5, "expected_score": 5.0, "progress": 1.0,
 "probabilities": {"1": 0.0, "2": 0.0, "3": 0.0, "4": 0.0, "5": 1.0},
 "rationale": "The robot grasped the blue block, moved it over the silver bowl, and released it inside the bowl."}

• Cameras: pass all views of one episode in order; they are tiled into one frame. Two views: left, right. Three: left, center, right (for example left wrist, top, right wrist). Four: top, front, left, right. Rubric jobs list one description per camera instead (format, template).
• Many episodes: every command takes --jobs jobs.jsonl, one episode per line: {"id": "ep001", "task": "Put the cup on the tray.", "videos": ["left_wrist.mp4", "top.mp4", "right_wrist.mp4"]}.
• First run: downloads 56 GB and compiles GPU kernels for a few minutes. --dry-run prints the prompt without loading the model.
• Dense scores look a few seconds past each moment, so use them for offline labeling and filtering, not as an online signal.

Full options: --help on any command, the annotation guide and the Rewarder guide.

Excerpts with Flora's predictions. The outcome, dense and rubric scores agree with the human labels for these episodes.

Human video	Parallel grippers	Dexterous hands
15.85–37.25 s “hold bottle with left hand, scrape glass with utility knife in right hand”	8.80–12.80 s “Put the cucumber on the plate with the right hand.”	11.50–27.73 s “Peel the backing from the bandage with the left hand while holding the bandage with the right hand.”
Outcome reward	Dense reward	Rubric reward
Success: 5 / 5 “Put the block in the silver bowl” Reason: “The robot grasped the blue block, moved it over the silver bowl, and released it inside the bowl.”	Failure: 1 / 3, 188.2–199.9 s “… the toy drill is misaligned and contacts the gray screw head without engaging the screw”	Total: 1.00 / 1.00 Four placement steps, each scored 0.25 / 0.25.

From the paper (Tables 1, 4 and 5). Arrows mark the better direction.

Flora annotates video at an estimated $0.72 per hour of video (eight GB300 GPUs at $10 per GPU-hour), about 125× less than GPT-6 Astra at its highest reasoning effort.

AGENTS.md tells Claude Code, Codex, Cursor and other coding agents how to set up the two environments, check inputs without a GPU, run each model and read the JSON it writes. Try:

Read AGENTS.md, then add timestamped instructions to every video in ./videos, one JSON file per video.

• decoded fewer frames than its container reports: the MP4 hides frames with an edit list, which ffmpeg -ss … -c copy trims create. Re-encode it: ffmpeg -i in.mp4 -c:v libx264 -crf 18 -an out.mp4.
• Video exceeds 120 seconds: split the video; each part is annotated on its own clock.
• Arm64 machines (GH200, GB200): decord in the Rewarder requirements has no arm64 wheel; only x86_64 is validated.

Flora is for non-commercial research and evaluation only. Its outputs are annotations: do not use them to control a robot or as the only safety monitor of a physical system. Instructions are in English. Check the rewards on held-out episodes from your own robot before training on them.

@article{leng2026flora,
  title   = {Flora: An Open Model Suite for Scaling Robotics Data Engines},
  author  = {Sicong Leng and Yunze Man and Zhiqi Li and Jing Wang and
             Yuhe Zhang and Ryan Punamiya and Siddharth Patki and Kaiyuan Zheng and
             Zi-ang Cao and Zhiding Yu and Sifei Liu and Danfei Xu and Ben Levin and
             Jan Kautz and Yuke Zhu and Johan Bjorck and Linxi "Jim" Fan},
  year    = {2026}
}

This repository is released under the NVIDIA License, a research license that permits non-commercial research and evaluation use only. The model weights on Hugging Face come with the same license (LICENSE.pdf in each model repository). Third-party dependencies keep their own licenses; see the notices for annotation and the Rewarder.