R2P2V

This repository provides an implementation of R2P2V (Radar-to-Pose-to-Video) for video restoration under missing or failed cameras. The pipeline consists of two stages:

• Stage I (Radar → Pose): generate pose cues from mmWave radar.
• Stage II (Pose → Video): restore photorealistic frames conditioned on pose cues, radar features, and optional environment/appearance priors.

Dataset

Please request and download the raw HuPR radar dataset from:

• https://huggingface.co/datasets/nirajpkini/HuPR

Place the downloaded raw radar data under:

• preprocessing/raw_data/

Place the corresponding RGB frames under:

• Radar2Pose/frames/

Preparation

pip install -r requirements.txt

python setup.py

Data preprocessing

• Step 1: Preprocess the raw radar data collected by the radar sensor (IWR1843Boost). Convert the raw .bin files into npy files

  cd preprocessing
  python process_iwr1843.py

• Step 2: Extract environment and appearance priors (optional) This step prepares environment/appearance priors from RGB frames. It uses LaMa for inpainting/background completion. Please follow the LaMa repository to set up the environment. Then run:

  cd preprocessing
  python gerPersonImg.py
  python GetPeopleAppearence.py

Note: If you do not use appearance priors, you can skip this step and run Stage II in an environment-only (or no-prior) mode, depending on your config.

• Step 3: please note that this step3 must after finish train stage1 After Stage-I is ready (trained model or pretrained weights), export pose heatmaps and radar features required by Stage-II training/evaluation:

  cd preprocessing
  python gerPersonImg.py
  python GetPeopleAppearence.py

Configure

Edit configs/train.yaml and set:

• data.radar_feat_root
• data.visual_env_root
• data.visual_target_root
• data.video_root

Train

train stage1

cd Radar2Pose
python main.py

train stage2

cd scripts
python train.py --config configs/train.yaml

Eval

python scripts/eval.py --config configs/train.yaml --ckpt /path/to/best_model.pth

Notes:

• Validation metrics (IoU / Mask-L1) use Mask R-CNN from torchvision detection. If torchvision detection ops are not available in your environment, you may disable these metrics via config/flags.

Citation

If our work is useful for your research, please consider citing:

@article{TODO_R2P2V,
  title   = {TODO},
  author  = {TODO},
  journal = {TODO},
  year    = {TODO}
}

Aknowledgment

This code largely benefits from excellent work-HuPR repository, please also consider citing HuPR if you use this code.

@InProceedings{Lee_2023_WACV,
    author    = {Lee, Shih-Po and Kini, Niraj Prakash and Peng, Wen-Hsiao and Ma, Ching-Wen and Hwang, Jenq-Neng},
    title     = {HuPR: A Benchmark for Human Pose Estimation Using Millimeter Wave Radar},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {January},
    year      = {2023},
    pages     = {5715-5724}
}