CurveFlow: Curvature-Guided Flow Matching for Image Generation

This project is built upon SimpleTuner.

Install Prerequisites

conda create --name CurveFlow python=3.11
pip install -r requirements.txt

Download Datasets

python scripts/download_coco17.py
python scripts/download_coco14.py

Data Pre-Processing

SimpleTuner requires each image and its corresponding caption to be stored in separate files that share the same identifier, such as [sample_ID].jpg and [sample_ID].txt.

Login to huggingface

huggingface-cli login

Fine-Tuning

CUDA_VISIBLE_DEVICES=0,1,2,3 CONFIG_ENV_FILE='config/config_base.env' CONFIG_JSON_FILE='config/config_coco17.json' CONFIG_BACKEND=json DISABLE_UPDATES=1 ./train.sh
CUDA_VISIBLE_DEVICES=0,1,2,3 CONFIG_ENV_FILE='config/config_base.env' CONFIG_JSON_FILE='config/config_coco17_lognorm.json' CONFIG_BACKEND=json DISABLE_UPDATES=1 ./train.sh
CUDA_VISIBLE_DEVICES=0,1,2,3 CONFIG_ENV_FILE='config/config_base.env' CONFIG_JSON_FILE='config/config_coco17_modesample.json' CONFIG_BACKEND=json DISABLE_UPDATES=1 ./train.sh
CUDA_VISIBLE_DEVICES=0,1,2,3 CONFIG_ENV_FILE='config/config_base.env' CONFIG_JSON_FILE='config/config_coco17_cosmap.json' CONFIG_BACKEND=json DISABLE_UPDATES=1 ./train.sh
CUDA_VISIBLE_DEVICES=0,1,2,3 CONFIG_ENV_FILE='config/config_proposed.env' CONFIG_JSON_FILE='config/config_coco17_proposed.json' CONFIG_BACKEND=json DISABLE_UPDATES=1 ./train_proposed.sh

Inference

CUDA_VISIBLE_DEVICES=0 bash ./inference_coco17.sh
CUDA_VISIBLE_DEVICES=0 bash ./inference_coco14.sh

Evaluation on Image Quality and Image-Text Alignment

bash ./evaluate_on_image_generation.sh

Evaluation on Image-To-Text Semantics Consistency with BLIP v2

# This would generate captions bsed on the generated images
bash ./evaluate_with_blip_caption.sh

Evaluation on CLAIR

bash ./evaluate_CLAIR_with_generated_captions.sh