Lester: Rotoscope Animation Through Video Object Segmentation and Tracking

1. Introduction

This repository contains data related to the Lester project. The project aims at developing a sytem capable of automatically synthetise retro-style 2D animations from videos. Another repository, rtous/lester-code contains the code.

paper

2 Acknowledgements

If you find this repository useful for your research, please cite the original publication:

@Article{a17080330,
	AUTHOR = {Tous, Ruben},
	TITLE = {Lester: Rotoscope Animation through Video Object Segmentation and Tracking},
	JOURNAL = {Algorithms},
	VOLUME = {17},
	YEAR = {2024},
	NUMBER = {8},
	ARTICLE-NUMBER = {330},
	URL = {https://www.mdpi.com/1999-4893/17/8/330},
	ISSN = {1999-4893},
	DOI = {10.3390/a17080330}
}

3. Results

YouTube videos:

• Video 1
  • Id: youtube1
  • Source: YouTube
  • input
  • result
  • result fine-tuned Stable Diffusion + ControlNet + EbSynth

• Video 2
  • Id: youtube2
  • Source: YouTube
  • input
  • result

• Video 3
  • Id: youtube3
  • Source: YouTube
  • input
  • result
  • result fine-tuned Stable Diffusion + ControlNet + EbSynth

• Video 4
  • Id: youtube4
  • Source: YouTube
  • input
  • result

• Video 5
  • Id: youtube5
  • Source: YouTube
  • input
  • result

• Video 6
  • Id: youtube6
  • Source: YouTube
  • input
  • result

• Video 7
  • Id: youtube7
  • Source: YouTube
  • input
  • result
  • result fine-tuned Stable Diffusion + ControlNet + EbSynth

• Video 8
  • Id: youtube8
  • Source: YouTube
  • input
  • result

• Video 9
  • Id: youtube9
  • Source: YouTube
  • input
  • result

• Video 10
  • Id: youtube10
  • Source: YouTube
  • input
  • result

Videos from the Fashion Video Dataset of the CV Lab of the University of British Columbia:

• Video 11
  • Id: fvd_1
  • Source: test/91oFLFG8UNS.mp4
  • input
  • result
  • result fine-tuned Stable Diffusion + ControlNet + EbSynth

• Video 12
  • Id: fvd_2
  • Source: test/91uY9usoa5S.mp4
  • input
  • result

• Video 13
  • Id: fvd_3
  • Source: test/A1gga724uoS.mp4
  • input
  • result

• Video 14
  • Id: fvd_4
  • Source: test/A1rJWpLlTwS.mp4
  • input
  • result

• Video 15
  • Id: fvd_5
  • Source: test/A1VF4yftLZS.mp4
  • input
  • result

Videos from the UCF101 Human Actions dataset:

• Video 16
  • Id: ucf101_1
  • Source: UCF101/v_WallPushups_g01_c03.avi
  • input
  • result

• Video 17
  • Id: ucf101_2
  • Source: UCF101/v_PizzaTossing_g05_c01.avi
  • input
  • result

• Video 18
  • Id: ucf101_3
  • Source: UCF101/v_BodyWeightSquats_g12_c01.avi
  • input
  • result

• Video 19
  • Id: ucf101_4
  • Source: UCF101/v_BodyWeightSquats_g20_c01.avi
  • input
  • result

• Video 20
  • Id: ucf101_5
  • Source: UCF101/v_BodyWeightSquats_g20_c01.avi
  • input
  • result

Videos from the Kinetics dataset:

• Video 21
  • Id: kinetics1
  • Source: U8LA_hHPISg_000102_000112.mp4
  • input
  • result
  • result fine-tuned Stable Diffusion + ControlNet + EbSynth

• Video 22
  • Id: kinetics2
  • Source: zxsSfO4cHRQ_000000_000010.mp4
  • input
  • result

• Video 23
  • Id: kinetics3
  • Source: yGA9OkTvvVM_000065_000075.mp4
  • input
  • result

• Video 24
  • Id: kinetics4
  • Source: xkp3fD8HL68_000000_000010.mp4
  • input
  • result
  • result fine-tuned Stable Diffusion + ControlNet + EbSynth

• Video 25
  • Id: kinetics5
  • Source: vg9Tg1gihmk_000015_000025.mp4
  • input
  • result