Skip to content
BranchesTags

Repository files navigation

RIVER

Research codebase for reference-based image and video super-resolution.

Unofficial re-implementation. This repository reimplements LDIP (Long Distance Information Propagation for Video Super-Resolution, ICCV 2025) and ReBaIR (Reference-Based Image Restoration, ICCV 2025 workshop). Some architecture details may deviate from the methods described in the papers but core ideas are reflected accurately. This codebase is not an official Disney / DisneyResearch|Studios release and is in no way affiliated with DisneyResearch|Studios or The Walt Disney Company!

Research / non-commercial by default. First-party code is MIT, but default LDIP and ReBaIR+PDC graphs pull PDCNet+ / GOCor (CC BY-NC-SA 4.0) — research use only. C2-only ReBaIR avoids GOCor (Apache-2.0 matcher) but still has its own deps. See THIRD_PARTY_NOTICES.md.

Visual demos

LDIP drift-car — zoom & compare (IART vs LD_IART_L · scroll to zoom · drag to wipe)

Drift-car: IART vs LD_IART_L

ReBaIR LMR SRx4 — zoom & compare (MRefSR-GAN vs ReBaIR_DRCT-SR_C2+PDC_M · scroll to zoom · drag to wipe)

ReBaIR LMR SRx4

Core design

LDIP (video): backbone (BasicVSR++ or IART) → LRRF (PDC-aligned long-range references) → HRIP (RAFT flow + Swin fusion) → decode.

ReBaIR (still image): frozen classical ×4 backbone (DRCT-SR / DRCT-L-SR / SwinIR-M-SR) → ordered refinement pipeline (PDC, C2, C2+PDC, …) with frozen matcher + trainable multi-scale fusion → decode via the backbone head.

Device / offloading

Public LDIP / ReBaIR / video / matcher ops (BasicVSR++, IART, LRRF, HRIP/IP, RAFT, PDCNet+, ReBaIR backbones and refinements) keep weights on the module device and activations on the caller’s storage device:

  1. compute_device = self.device (parameters)
  2. offload_device = input.device (captured before any move)
  3. temporarily .to(compute_device) for work
  4. return .to(offload_device)

That lets LDIP / ReBaIR validation, inference_scripts/, and eval_scripts/ keep frames / intermediates on CPU (or in DiskTensorList) while the model stays on CUDA. Training still moves train batches to GPU for throughput; validation deliberately does not. Nested factory modules (warp / fusion / MSFE / …) are compute-device-only; public wrappers move first.

RIVER/                     # core installable library (method-agnostic building blocks)
presets/
  ldip/                    # LDIP factory + checkpoint
  rebair/                  # ReBaIR factory + checkpoint
  model_zoo.py             # named models
  hub.py                   # Hub transport
checkpoints/               # local inference packs (gitignored; Hub fills if missing)
training_runs/
  LDIP/                    # LDIP trainer + configs
  ReBaIR_SR/               # ReBaIR ×4 SR trainer + configs
  ReBaIR_denoise/          # ReBaIR denoising trainer (experimental; no Hub packs yet)
ext/                       # third-party installers (clone on demand)
eval_scripts/              # REDS (LDIP) + LMR (ReBaIR) metrics
inference_scripts/         # in-the-wild demos (LDIP frames, drift car, ReBaIR)
scripts/                   # dataset helpers + README wipe GIF tooling

Installation

python3 -m venv .venv
source .venv/bin/activate
pip install -e .
pip install -r requirements-training.txt   # tensorboard
# Optional Hub load/upload:
pip install -r requirements-hub.txt

Packaging uses setup.py (installs both RIVER and presets).

External components

Install only what you need (needs network; several scripts use gdown — pip install gdown if missing). DenseMatching also needs a CUDA-capable machine for cupy-cuda12x. Scripts are cwd-safe: bash ext/<Comp>/install.sh works from any directory.

bash ext/BasicVSRpp/install.sh      # LDIP BasicVSR++ backbone
bash ext/DenseMatching/install.sh   # PDCNet+ (LDIP + ReBaIR PDC)
bash ext/IART/install.sh            # LDIP IART backbone (+ in-repo basicsr/ stub)
bash ext/SwinIR/install.sh          # ReBaIR SwinIR-M
bash ext/DRCT/install.sh            # ReBaIR DRCT / DRCT-L (+ in-repo basicsr/ stub)
bash ext/C2Matching/install.sh      # ReBaIR C2 refinement (+ RefSR comparison)
# ReBaIR RefSR comparison only (not backbones):
bash ext/MRefSR/install.sh          # MRefSR (LMR baseline; Apache-2.0)
bash ext/DATSR/install.sh           # DATSR (CC BY-NC 4.0 — research only)

ext/DRCT/basicsr/ and ext/IART/basicsr/ are tracked slim BasicSR import stubs (not full BasicSR). Do not delete them; install.sh only clones the upstream tree next to them.

Use case Required ext/ installs
LDIP + BasicVSR++ BasicVSRpp, DenseMatching
LDIP + IART IART, DenseMatching
ReBaIR + DRCT + PDC DRCT, DenseMatching
ReBaIR + SwinIR-M + C2 SwinIR, C2Matching
ReBaIR RefSR comparisons C2Matching, MRefSR, DATSR
RAFT (LDIP HRIP) torchvision only

Video inference also needs system ffmpeg / ffprobe.

Datasets

REDS (LDIP)

Official sources (CC BY 4.0):

RIVER only needs the sharp GT packs (bicubic LR is generated on the fly; blur / JPEG / bicubic zips are unused):

Split Download Role
train_sharp train pack LDIP training
val_sharp val pack build REDS4 below

Expected layout after unpack:

REDS/
  train/train_sharp/000/00000000.png … 00000099.png   # 240 clips
  val/val_sharp/…                                      # 30 clips
  val/val_sharp4/000 001 006 017/…                     # REDS4 (see below)

REDS4 (val_sharp4) is not a separate official zip. After unpacking val_sharp, create it from clips 000, 001, 006, 017:

cd /path/to/REDS
mkdir -p val/val_sharp4
for c in 000 001 006 017; do cp -a "val/val_sharp/$c" "val/val_sharp4/$c"; done

Then point training / eval at those dirs:

  • TRAIN_ROOT → …/REDS/train/train_sharp
  • TEST_ROOT / --reds-root + --reds-subset val/val_sharp4 → …/REDS

LMR (ReBaIR)

LMR (Zhang et al., ICCV 2023; multi-reference RefSR from MegaDepth) is distributed via Academic Torrents — search “LMR” or open this torrent page. Code / paper context: wdmwhh/MRefSR.

From the release, extract at least:

  • MegaDepth_v3_5ref.tar (~55 GB) + meta_info_MegaDepth_v3_5ref.csv — training source
  • MegaDepth_v3-test.tar — test set (used as-is)

RIVER’s train loader expects cropped patches, not the raw MegaDepth folders. After extract:

python scripts/prepare_lmr_train.py --lmr-root /path/to/LMR

Expected layout:

LMR/
  MegaDepth_v3_5ref/          # from torrent (input to prepare script)
  meta_info_MegaDepth_v3_5ref.csv
  train/
    images/<id>.png           # written by prepare_lmr_train.py
    refs/<id>/0.png … 4.png
  MegaDepth_v3-test/
    <scene>_<img>/…           # from torrent; eval --lmr-root

Then:

  • TRAIN_ROOT → …/LMR/train
  • TEST_ROOT / --lmr-root → …/LMR/MegaDepth_v3-test

Training

LDIP (REDS)

cd training_runs/LDIP
cp paths.example.py paths.py   # TRAIN_ROOT / TEST_ROOT — see Datasets
python train.py --backbone BasicVSRpp --size S
torchrun --nproc_per_node=N train.py --backbone BasicVSRpp --size S

Published run recipes (auto run name {backbone}_{size}):

Run Flags
BasicVSRpp_L / IART_L --backbone … --size L
BasicVSRpp_S / IART_S --backbone … --size S

Size presets live in presets/ldip/factory.py and resolve into an explicit ModelConfig before training / checkpointing.

Train run dirs use {backbone}_{S|L} (e.g. BasicVSRpp_S). Inference packs / Hub paths use LDIP/LD_{backbone}_{S|L} (e.g. LDIP/LD_BasicVSRpp_S).

ReBaIR (LMR / MegaDepth)

cd training_runs/ReBaIR_SR
cp paths.example.py paths.py   # TRAIN_ROOT / TEST_ROOT — see Datasets
python train.py --backbone DRCT-SR --refinement PDC --size M
# Multi-stage (load one checkpoint per stage); quote '+' in bash:
python train.py --backbone DRCT-SR --refinement 'C2+PDC' --size M \
  --init-refinements runs/DRCT-SR_C2_M/last.pt,runs/DRCT-SR_PDC_M/last.pt

Size presets S / M / L live in presets/rebair/factory.py and are resolved into an explicit ModelConfig (channel pyramid + stages) before training / checkpointing.

Naming map (train run ↔ zoo ↔ Hub subfolder):

Train run dir Zoo name Hub path
DRCT-SR_C2_M ReBaIR_DRCT-SR_C2_M ReBaIR/DRCT-SR_C2_M
DRCT-SR_PDC_M ReBaIR_DRCT-SR_PDC_M ReBaIR/DRCT-SR_PDC_M
DRCT-SR_C2+PDC_M ReBaIR_DRCT-SR_C2+PDC_M ReBaIR/DRCT-SR_C2+PDC_M

Denoising experiments live under training_runs/ReBaIR_denoise/ (experimental; no zoo / Hub packs yet).

Each checkpoint stores model_config (including a method tag) alongside weights. Load via the matching family package — there is no shared method dispatcher.

Loading checkpoints

Inference packs live under checkpoints/ (gitignored):

checkpoints/LDIP/LD_BasicVSRpp_S/model.pt
checkpoints/LDIP/LD_BasicVSRpp_S/config.json

Named zoo resolution order:

  1. Explicit checkpoint= / --checkpoint path
  2. Local checkpoints/<hub_subfolder>/model.pt
  3. Download from Hugging Face into that local directory, then load
from presets.model_zoo import load_model

model = load_model("LD_BasicVSRpp_S", device="cuda")
# Or override:
model = load_model(
    "LD_BasicVSRpp_S",
    device="cuda",
    checkpoint="path/to/custom.pt",
)

# Family backbone adapters (pretrained via ext/*/install.sh; no Hub pack):
#   ldip_backbone:  BasicVSRpp, IART
#   rebair_backbone: SwinIR-M-SR, DRCT-SR, DRCT-L-SR
backbone = load_model("DRCT-SR", device="cuda")

Hugging Face Hub

Always export training runs to a local pack under checkpoints/ first. Upload that pack to the Hub later when you are ready to publish. Named zoo loading (load_model) uses the local pack if present; otherwise it downloads into the same checkpoints/<hub_subfolder>/ path.

pip install -r requirements-hub.txt
hf auth login   # write token if uploading
# 1) Export training run → local pack
python -m presets.hub export --family ldip \
  --run-dir training_runs/LDIP/runs/BasicVSRpp_S \
  --out-dir checkpoints/LDIP/LD_BasicVSRpp_S
# 2) Later: upload the local pack (path-in-repo defaults from checkpoints/)
python -m presets.hub upload \
  --pack-dir checkpoints/LDIP/LD_BasicVSRpp_S \
  --repo-id MichaelBernasconi/RIVER
from presets.model_zoo import load_model

# Local pack if present; else Hub → checkpoints/LDIP/LD_BasicVSRpp_S/
model = load_model("LD_BasicVSRpp_S", device="cuda")

Evaluation

LDIP on REDS4 (val/val_sharp4):

python eval_scripts/LDIP_REDS/evaluate.py \
  --reds-root /path/to/REDS \
  --reds-subset val/val_sharp4 \
  --models BasicVSRpp LD_BasicVSRpp_S LD_IART_L \
  --output-dir eval_scripts/LDIP_REDS/results

Published REDS4 scores: eval_scripts/LDIP_REDS/results/RESULTS.md.

ReBaIR on LMR MegaDepth test:

python eval_scripts/ReBaIR_LMR/evaluate.py \
  --lmr-root /path/to/LMR/MegaDepth_v3-test \
  --models DRCT-SR ReBaIR_DRCT-SR_C2_M C2Matching-MSE MRefSR-MSE DATSR-MSE \
  --output-dir eval_scripts/ReBaIR_LMR/results

Published LMR scores: eval_scripts/ReBaIR_LMR/results/RESULTS.md.

RefSR comparison zoo names (need matching ext/*/install.sh): C2Matching-MSE / C2Matching-GAN, DATSR-MSE / DATSR-GAN, MRefSR-MSE / MRefSR-GAN. Single-ref methods (C2, DATSR) run once per reference and keep the best score per metric. MRefSR uses all refs.

Metrics: PSNR_Y, LPIPS, SSIM_Y (4 px crop). LDIP supports optional --do-disk-cache and --keyframe-stride / --keyframe-start.

Inference

LDIP folder of frames (SELF key frames; no external refs):

python inference_scripts/LDIP_frames/inference.py \
  --input /path/to/frames \
  --model LD_BasicVSRpp_S \
  --output-dir outputs/my_clip \
  --keyframe-stride 10 \
  --keyframe-start 5

ReBaIR / RefSR comparison with references:

python inference_scripts/ReBaIR/inference.py \
  --input /path/to/lq.png \
  --refs /path/to/ref_a.png /path/to/ref_b.png \
  --model ReBaIR_DRCT-SR_C2_M \
  --output-dir outputs/rebair_demo
# or: --refs-dir /path/to/refs_folder
# comparisons: --model MRefSR-MSE | C2Matching-MSE | DATSR-MSE

Single-ref comparisons (C2, DATSR) use only the first reference and warn if more are given. MRefSR uses all refs.

LDIP drift-car demo (EXTERNAL close-up refs): inference_scripts/LDIP_drift_car/ (bash download_clip.sh then run its inference.py). See Visual demos above for the interactive comparison and wipe GIF.

Extending for research

  1. Copy an existing experiment (training_runs/LDIP/ or training_runs/ReBaIR_SR/) as a starting point.
  2. Implement ModelConfig + build_inference_model under presets/<name>/factory.py and family checkpoint helpers under presets/<name>/checkpoint.py. Core RIVER must not import presets.
  3. Keep training_runs/<name>/model.py as the training wrapper (freeze policy, losses).
  4. Register names in presets/model_zoo.py for eval / inference / Hub.
  5. Document any new ext/ dependency in THIRD_PARTY_NOTICES.md.

Citation

@InProceedings{Bernasconi_2025_ICCV_LDIP,
    author    = {Bernasconi, Michael and Djelouah, Abdelaziz and Zhang, Yang and Gross, Markus and Schroers, Christopher},
    title     = {LDIP: Long Distance Information Propagation for Video Super-Resolution},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2025},
    pages     = {11558-11567}
}

@InProceedings{Bernasconi_2025_ICCV_ReBaIR,
    author    = {Bernasconi, Michael and Djelouah, Abdelaziz and Zhang, Yang and Gross, Markus and Schroers, Christopher},
    title     = {ReBaIR: Reference-Based Image Restoration},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
    month     = {October},
    year      = {2025},
    pages     = {5548-5557}
}

Third-party code and licenses

See THIRD_PARTY_NOTICES.md for the full table.

Callout: PDCNet+ (DenseMatching) uses GOCor (CC BY-NC-SA 4.0). LDIP and ReBaIR PDC stages are research / non-commercial. C2-only ReBaIR graphs avoid GOCor but still depend on C2-Matching’s own license (Apache-2.0). The DATSR comparison models (DATSR-MSE / DATSR-GAN) are CC BY-NC 4.0 (research / non-commercial only).

About

LDIP (Long Distance Information Propagation) VSR and ReBaIR reference-based image SR — unofficial re-implementation

Topics

image-restoratioimage-srldiplong-distance-information-propagationrebairreference-basedreference-based-image-restorationrefsrsrsuper-resolutionvideo-super-resolutionvsr

Resources

Readme
MIT license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages