RecRecNet: Rectangling Rectified Wide-Angle Images by Thin-Plate Spline Model and DoF-based Curriculum Learning

Introduction

This is the official implementation for RecRecNet (ICCV2023).

Kang Liao^†, Lang Nie^†, Chunyu Lin, Zishuo Zheng, Yao Zhao

Problem

Given a rectified wide-angle image, RecRecNet aims to construct a win-win representation on both image content and boundary, with the perspective of the rectangling technique.

Features

• First win-win representation of the large field-of-view (FoV) vision
• A thin-plate spline (TPS) motion module is proposed to flexibly formulate the non-linear and non-rigid rectangling transformation
• A DoF-based curriculum learning is designed to grasp the progressive deformation rules and relieve the burden of complex structure approximation
• An in-depth analysis of why the deformed image boundary can significantly influence the vision perception models

Updates

Our recent work MOWA can solve multiple image warping tasks in a single and unified framework, including image rectangling, distortion rectification, and other practical tasks. Check out more details here!

Installation

Using the virtual environment (conda) to run the code is recommended.

conda create -n recrecnet python=3.6
conda activate recrecnet
pip install -r requirements.txt

Dataset

We constructed the first dataset for the rectified wide-angle rectangling task. The structure of the original rectified wide-angle image was first optimized by an energy function with line-preserving mesh deformation, as proposed in He et al.. And then we carefully filtered all results and repeated the selection process three times. The dataset can be downloaded here: train.zip, test.zip.

Pretrained Model

Download the pretrained model here and put it into the .\checkpoint folder.

Training

Curriculum Generation

Generate the curriculum to grasp the progressive deformation rules of rectangling. The source image can be collected from ImageNet or COCO. Please set the suitable $path1$, $path2$, and $dof$ (4 and 8) and run:

sh scripts/curriculum_gen.sh

TPS Model Training

Customize the paths of 4-dof dataset, 8-dof dataset, and wide-angle image rectangling dataset, and run:

sh scripts/train.sh

Testing

Customize the paths of checkpoint and test set, and run:

sh scripts/test.sh

The rectangling image and its corresponding warping mesh (formed by predicted TPS control points) can be found in the .\results folder.

Citation

If you feel RecRecNet is helpful in your research, please consider referring to it:

@article{liao2023recrecnet,
  title={RecRecNet: Rectangling rectified wide-angle images by thin-plate spline model and DoF-based curriculum learning},
  author={Liao, Kang and Nie, Lang and Lin, Chunyu and Zheng, Zishuo and Zhao, Yao},
  journal={arXiv preprint arXiv:2301.01661},
  year={2023}
}