Refining Distant People in 3D — Depth Anything 3 + Crop Super-Resolution

XAI506 mid-term project (Vision Foundation Models).

When you lift a single photo into a 3D point cloud with Depth Anything 3 (DA3), people standing far from the camera collapse into a shapeless blob. This project detects those people, super-resolves each one, re-estimates depth on the crop, and fuses it back so the distant person becomes a recognizable 3D shape again.

The problem. DA3 downsamples its input to a 504 px long edge. In a 4032×3024 photo, a person ~150×200 px tall shrinks to a ~19×25 px blob in the global depth map (504×378); after the confidence cut, that blob can drop to zero points.

The pipeline (one input image → interactive 3D HTML):

1. DA3 global depth — the whole scene, low resolution (distant people lost).
2. Grounding DINO + SAM — detect every person + a precise mask.
3. Crop the target person → Real-ESRGAN / HAT ×4 super-resolution.
4. DA3 re-inference on the SR crop — now the person fills the frame, so head, shoulders, and body geometry are recovered at high resolution.
5. Affine (scale + shift) alignment — fit the crop depth into the global scale.
6. Dense back-projection — the person is back-projected at original resolution and merged with the sparse global background.

Result: a 3D point cloud where the distant person goes from a ~0-point blob to a recognizable body of ~20k points.

The example photo and its per-person crops are not committed (they show real people); run demo.py on your own image to reproduce the stage images and HTML.

---

Installation

A CUDA GPU is required. Everything runs in one conda env.

conda create -y -n deep-learning python=3.10
conda activate deep-learning

# core deps
pip install "torch>=2" torchvision
pip install transformers pillow scipy imageio opencv-python matplotlib numpy \
            trimesh huggingface_hub plotly spandrel

# Depth Anything 3 — source install
cd depth_anything
git clone https://github.com/ByteDance-Seed/depth-anything-3.git repo
cd repo && pip install -e ".[app]"
cd ../..

# Super-resolution weights -> weights/  (any spandrel-loadable x4 SR .pth works)
#   RealESRGAN_x4plus.pth   (xinntao Real-ESRGAN v0.1.0 release)
#   HAT-L_SRx4_ImageNet-pretrain.pth  (optional, higher quality, larger)

Models pulled automatically from the Hugging Face Hub on first run: depth-anything/DA3-LARGE-1.1 (use DA3-BASE for less VRAM), IDEA-Research/grounding-dino-base, facebook/sam-vit-huge.

---

Usage

python demo.py path/to/your_image.jpg

This runs the full pipeline and writes everything to outputs/demo/:

• a numbered PNG for every stage (01_input … 11_pc_after_refined) plus a 00_contact_sheet.png montage, and
• demo_result.html — the final interactive 3D point cloud (open in a browser; drag = rotate, scroll = zoom). The distant person is drawn dense and opaque over the sparse scene background.

Options:

python demo.py path/to/image.jpg \
    --out outputs/myrun \
    --model-id depth-anything/DA3-BASE \      # less VRAM
    --sr-weights weights/RealESRGAN_x4plus.pth \
    --target-mode farthest \                  # 'sweet' (default) | 'farthest'
    --target-idx 7                            # refine a specific detected person

---

Code

File	What it does
demo.py	Main demonstration. Drives the pipeline end-to-end, saves a stage image per step, and writes the final interactive HTML.
refine_utils.py	All pipeline helpers.
subproc/distance_sweep.py	Evaluation: how far can Crop-SR recover a person? (synthetic distance sweep).
subproc/kitti_distance.py	Evaluation: the same, validated on real metric distance ground truth (KITTI).

refine_utils.py, grouped:

• Detection — detect_people (Grounding DINO + SAM), detect_boxes_multi (DINO boxes only), segment_boxes (box-prompted SAM); all run in-process.
• Super-resolution — load_sr_model, super_resolve (spandrel ×4, Lanczos fallback).
• DA3 on a crop — run_da3_on_crop.
• Geometry / alignment — expand_box, map_crop_depth_to_global, mask_to_global, fit_affine (IRLS), align_and_fuse, build_bbox_refined_depth, robust_da3_ab, scale_K, estimate_distance_m.
• Back-projection / rendering — backproject_to_3d, backproject_region, compute_alignment, apply_alignment, downsample, render_orbit_gif, render_interactive_html, render_interactive_html_split, depth_to_rgb_turbo.
• KITTI parsing — load_kitti_labels, parse_kitti_calib.

---

How far can Crop-SR recover a person? — subproc/distance_sweep.py

There is no clean "person walks toward/away with distance GT" dataset, so we simulate distance: take a clear NEAR person (where native DA3 is reliable = pseudo-GT) and progressively downscale the crop to mimic the person being f× farther. Distance in meters is a pinhole estimate from pixel height and DA3's (non-metric) intrinsics assuming ~1.7 m height (ru.estimate_distance_m) — a relative "how far" axis, not a calibration.

python subproc/distance_sweep.py
# -> outputs/depth_refine/distance_sweep/{sweep_error,sweep_structure,sweep_gallery,real_scatter}.png + csv

Findings (example image, near person simulated out to ~150 m).

• Point density: refined body points fall off smoothly with distance (vs the global blob, which is 0 after the confidence cut).
• Structure fidelity (gradient correlation vs pseudo-GT): held above 0.6 out to ~55 m, with a clear advantage over the no-SR baseline in the ~27–55 m band; collapses below 0.3 around ~73 m (pixel height < ~40 px).
• AbsRel stays low for both baseline and refined — an affine-aligned body is a smooth surface, so error doesn't discriminate; density and structure do.

Real distance GT validation (KITTI) — subproc/kitti_distance.py

The sweep simulated distance, so we validate on real metric distance: KITTI 3D Object Detection pedestrian labels carry a LiDAR-derived 3D location. KITTI gives GT 2D boxes, so no detector is needed — masks come from box-prompted SAM, then the same Crop-SR pipeline runs unchanged.

KITTI_ROOT=~/datasets/kitti bash subproc/fetch_kitti.sh         # image_2 + label_2 + calib
KITTI_ROOT=~/datasets/kitti/training python subproc/kitti_distance.py
# USE_LIDAR=1 also measures AbsRel against real LiDAR depth (needs velodyne/)

Findings (68 pedestrians, real GT 4–74 m).

• Pinhole distance estimate vs LiDAR GT: r = 0.97 (slope 0.59 → a constant ~1.7× under-estimate from DA3's focal bias, but an excellent relative axis).
• Recoverability: refined points ~20k (near) → ~110 (74 m), tracking the synthetic sweep; even at 74 m the refined cloud stays denser than the global blob.

---

Future work

The crop depth is currently DA3-only. A human-specific dense model (e.g. Sapiens pointmap/segmentation) could be fused on the body pixels to sharpen per-person geometry further — explored during the project and a natural next step.