SCOUT

Sim-to-Real Text-Based Person Retrieval by Embedding-Space Prediction over Frozen Video Features

ECCV 2026 Workshop paper (AI City Challenge Track 4: Text-Based Person Re-Identification, Sim2Real), accepted as a poster. Abdarahmane Traoré, Andy Couturier, Éric Hervet (Embia, Université de Moncton).

SCOUT casts cross-modal retrieval as prediction in embedding space: a trainable predictor (a Qwen3.5-0.8B decoder body) maps the patch tokens of a frozen V-JEPA video encoder into the embedding space of a frozen text encoder (EmbeddingGemma by default), trained with a bidirectional InfoNCE objective. Nothing in the base model is fine-tuned; retrieval at inference time is plain cosine similarity between two cached embeddings. A second, precision-targeted lever (ExPLoRA-adapts the video encoder) and a training-free VLM reranker push past what the frozen system alone can reach. On the competition leaderboard (final, scored on the full test set), a single frozen model scores about 61 mAP@10; the full retrieve-fuse-rerank system described in the paper's Section 6 scored 84.25 mAP@10 / 75.63 R@1, ranking 18th of 28 teams (the leader scored 99.30 / 98.74).

This repository reproduces every number the paper reports. It is organized in two tiers:

• Quickstart (below): train and evaluate a single SCOUT model on one GPU, reproducing any row of the ablation tables. This is the paper's scientific contribution.
• Advanced: the full board system: the fusion + VLM-rerank + cross-encoder-rerank pipeline behind the leaderboard result. It needs extra infrastructure (a hosted VLM, an external baseline model, the competition's private test set) most readers won't have, so it's documented separately rather than folded into the Quickstart.

Install

uv sync

Python >=3.12,<3.14. Model weights for the frozen backbones (V-JEPA2, EmbeddingGemma, CLIP, etc.) are downloaded automatically on first use from their own public sources, not from this repo. EmbeddingGemma is gated on Hugging Face; run hf auth login once before training/evaluating any config that uses it (the default). V-JEPA2 checkpoints cache to ~/.cache/scout/vjepa2_weights by default; override with the SCOUT_VJEPA2_CACHE_DIR environment variable.

Data setup

The paper trains and evaluates on PAB (People-in-Action Bench), the dataset released for AI City Challenge Track 4: 1,013,605 synthetic training images, a 36,773-image real gallery, and 1,978 real queries. PAB is not redistributed here.

1. Obtain the PAB archive from the dataset authors' release: OneDrive or Baidu Yun (extraction code ahgc), then:

   ./scripts/setup_pab.sh /path/to/downloaded/PAB_archive   # unpacks into $PAB_ROOT (default /tank2/PAB)

2. (Optional, recommended) Pre-resize to a JPEG cache for much faster data loading during training:

   uv run python scripts/build_pab_cache.py --pab-root /path/to/PAB --out-dir /path/to/PAB_cache --image-size 384

   Point PAB_CACHE_DIR at that directory when training/evaluating; leave it unset to read the original WebP images directly (slower).

data/pab.py defines a frozen 5,000-record held-out validation split (_VAL_PER_CLASS=2500, _VAL_SEED=42, _VAL_MAX_PER_SCENE=10) used for every number in this repo's tables. Do not modify these constants; doing so reshuffles the split and invalidates any comparison against the paper's reported numbers. The exact split is also materialized as data/val_split.csv (5,000 rows: image path, image id, category, scene, behavior), regenerable with python scripts/dump_val_split.py --pab-root /path/to/PAB; the script asserts the 2,500/2,500 class balance and the per-scene cap before writing.

Hard-negative-pair annotations (needed for scout_hard_neg.yaml, scout_x_explora_emb.yaml, and the val_hard/train_hard splits): the AI City Challenge Track 4 release of PAB strips the hard_i/hard_c fields from the dataset authors' original CMP annotations. We re-fetched the originals and publish them separately as a public HF dataset:

from huggingface_hub import snapshot_download
annotation_dir = snapshot_download(repo_id="Abdrah/scout-eccv-pab-annotations", repo_type="dataset")

Point a config's data.hard_neg_annotation_dir (or scripts/local_eval.py --hard-neg-annotation-dir) at {annotation_dir}/train.

Quickstart

Train the paper's best single-model recipe (x_explora, val mAP@10 = 0.9297) and evaluate it. x_explora builds on a trained y_clip_v17 checkpoint (LP-FT stage-1 initialization), so train that one first and point stage 2 at its final checkpoint via INIT_FROM:

# Stage 1: frozen-encoder base (CLIP-text Y target, batch-128 recipe)
CONFIG=train/configs/scout_y_clip_v17.yaml PAB_ROOT=/path/to/PAB ./scripts/train.sh

# Stage 2: ExPLoRA video-encoder adaptation, initialized from stage 1
CONFIG=train/configs/scout_x_explora.yaml PAB_ROOT=/path/to/PAB \
    INIT_FROM=runs/scout_y_clip_v17/step_00006000_final ./scripts/train.sh

uv run python scripts/local_eval.py \
    --checkpoint runs/scout_x_explora/step_00006000_final \
    --model scout --pab-root /path/to/PAB --query-source val \
    --scout-x-variant vit_large_384 --scout-K 64 --scout-predictor-layers 24 \
    --scout-readout concat --scout-y-model-id sentence-transformers/clip-ViT-L-14 \
    --scout-x-finetune

Skipping stage 1 and pointing INIT_FROM at nothing trains x_explora cold and will not reproduce the paper's number. Any other config below trains from scratch in a single stage. On the paper's reference setup of six GPUs, wall-clock scales with the per-GPU batch: roughly 40 minutes for the batch-16 recipes, 1.8 h at batch 32, 3.7 h at batch 64, 7.4 h for the batch-128 recipes, and 9 h for the hard-pairs recipe (measured tracker runtimes). Budget proportionally more on fewer GPUs.

Reproducing the paper's tables

Every run cited in the paper has a config under train/configs/. Train with CONFIG=<path> ./scripts/train.sh, then evaluate with scripts/local_eval.py --query-source val using --scout-* flags that mirror the config's model: section (each config's header comment documents its own matching eval invocation). Reported numbers are held-out val mAP@10 / Hit@10 at 384px, data.split=train.

Table 1, ablation arc (X-encoder + predictor + Y = EmbeddingGemma unless noted):

Config	Knob changed	mAP@10
(frozen CLIP-L, joint)	baseline, no training	0.5689
(frozen SigLIP2-L, joint)	baseline, no training	0.6895
scout_pab_v0.yaml	ViT-B, K=1, mean-pool	0.5935
scout_pab_v2.yaml	+ class-balanced sampling	0.6099
scout_pab_v4.yaml	+ predictor depth 8→24	0.6290
scout_pab_v12.yaml	+ K 1→64, concat readout	0.7719
scout_pab_v17.yaml	+ per-forward batch 16→128	0.8657
scout_hard_neg.yaml	+ pre-mined hard-negative pairs	0.8790
scout_pab_v18.yaml	+ cross-rank negative gathering	0.8679

Table 2, frozen Y-encoder study (all K=32, readout=concat, predictor_layers=24):

Config	Y-encoder	mAP@10
scout_y_clip.yaml	CLIP-ViT-L/14 text	0.7792
scout_pab_v11.yaml	EmbeddingGemma-300M	0.7481
scout_y_pecore.yaml	PE-Core-B-16 text	0.6757
scout_y_qwen.yaml	Qwen3-Embedding-0.6B	0.6137
scout_y_pecore_proj.yaml	PE-Core-B-16 + trainable projection	0.5887

Table 3, precision levers:

Config	Description	val mAP@10	board mAP@10 (R@1), full test
scout_y_clip_v17.yaml	v17 recipe + CLIP-text Y	0.8932	82.43 (73.05)
scout_x_explora.yaml	+ ExPLoRA-adapted V-JEPA encoder	0.9297	82.94 (73.71)
+ attribute-decomposed VLM rerank (see system/)	full system	n/a	84.18 (75.28)
+ fusion-member swap	best submission	n/a	84.25 (75.63)

Board scores are the full-test numbers: after the 2026-07-10 close, the organizers re-scored every submission on the 100% test set (in-challenge feedback was a 50% subset, which carried about a point of noise per submission). The best submission ranked 18th of 28 teams on the final leaderboard; the leader scored 99.30 / 98.74.

Negative results (for completeness, not levers that worked): scout_vitg_frozen.yaml (bigger frozen ViT-giant encoder, 0.8186, worse than the distilled ViT-L used everywhere else) and scout_x_explora_v2.yaml (a stronger ExPLoRA recipe, first-2+last-8 blocks unfrozen + rank-64 LoRA, 0.9282, no better than the standard recipe: the encoder-adaptation lever saturates quickly).

A note on eval resolution: the numbers above are at 384px, matching training and the only image cache this repo builds. Some very early internal runs were measured at a since-removed 256px eval path; if you see a v17 number of 0.8657 next to a v18 number computed at a different resolution elsewhere, that pairing is cross-resolution, not a discrepancy in this repo.

Pretrained checkpoints

Five checkpoints, covering the paper's best single model and the two precision levers (x_explora, itm_v3), are published on Hugging Face. Each is a weights-only export: just the parameters this repo actually trained, not the frozen backbones, which are reconstructed from their own public sources at load time (see scripts/export_checkpoint.py / scripts/load_from_hf.py). These five reproduce every number in this README's tables. The full leaderboard-champion fusion additionally used a handful of other SCOUT training runs (same configs, different recipe knobs) as CombSUM members alongside frozen CLIP/SigLIP2/CMP; their configs and the fusion recipe itself are documented in system/README.md, though not every one of their checkpoints is published here.

Checkpoint	Hugging Face repo	Val mAP@10
v17	Abdrah/scout-eccv-v17	0.8639
hard_neg	Abdrah/scout-eccv-hard-neg	0.8790
y_clip_v17	Abdrah/scout-eccv-y-clip-v17	0.8932
x_explora	Abdrah/scout-eccv-x-explora	0.9297
itm_v3	Abdrah/scout-eccv-itm-v3	0.8858 (blended)

from scripts.load_from_hf import load_from_hf

model = load_from_hf("Abdrah/scout-eccv-x-explora").eval()

or evaluate directly:

uv run python scripts/local_eval.py --hf-repo Abdrah/scout-eccv-x-explora --pab-root /path/to/PAB --query-source val

The itm_v3 cross-encoder evaluates through its own two-stage script instead of local_eval.py: download the checkpoint (hf download Abdrah/scout-eccv-itm-v3 --local-dir <dir>) and run uv run python scripts/eval_scout_itm_v3.py --checkpoint <dir> --pab-root /path/to/PAB (its --help lists the blend weights that produce the table's 0.8858).

Repository layout

scout/        core model: X-encoder (frozen V-JEPA2), Y-encoder, predictor, losses, ExPLoRA, ScoutITM
data/         PAB dataset loader (data/pab.py, do not edit the frozen split constants)
eval/         retrieval metric (mAP@K) and ranking utilities
baselines/    frozen CLIP / SigLIP2 joint-embedding baselines
train/        training loops (train/sft.py, train/itm.py) and every config (train/configs/)
scripts/      entrypoints: train, eval, checkpoint export/upload/load, data setup
system/       Advanced: the full retrieve-fuse-rerank board system (separate README)
analysis/     the Y-encoder cross-modal alignment study cited in Table 2's discussion

Citation

@inproceedings{traore2026scout,
  title     = {{SCOUT}: Sim-to-Real Text-Based Person Retrieval by Embedding-Space Prediction over
               Frozen Video Features},
  author    = {Traor{\'e}, Abdarahmane and Couturier, Andy and Hervet, {\'E}ric},
  booktitle = {ECCV 2026 Workshops},
  year      = {2026}
}

License

Apache License 2.0 (see LICENSE and NOTICE). This repository vendors model-definition code from facebookresearch/vjepa2 (also Apache 2.0); it does not vendor or redistribute any pretrained model weights.