Hyun Kang, Dohae Lee, Myungjin Shin, In-Kwon Lee
{
'stable_diffusion': StableDiffusionPipeline,
'composable_diffusion': ComposableStableDiffusionPipeline,
'structure_diffusion': StructureDiffusionPipeline,
'attend_and_excite': AttendAndExcitePipeline,
'syntax_guided_generation': SynGenDiffusionPipeline,
}
Dataset should be in '.csv' format in which "prompt" column contains target prompts
Baseline models' default configurations are in "configs/models" file
Configure experiment file (.yaml) to run multiple models all at once
python -m run configs/exp/example.yaml
clean-fid==0.1.35
- Clone
https://github.com/mjsh34/T2I-CompBench
, install requirements (in a separate environment).
Evaluate a single dataset (specified in ./configs/exp/example.yaml
) on fid
, blip-vqa
, clipscore
and unidet
metrics:
python eval.py \
--config ./configs/exp/example.yaml \
--methods fid blip-vqa clipscore unidet \
--t2i_compbench_path /path/to/T2I-CompBench/ \
--t2i_compbench_pyexe /path/to/T2I-CompBench/venv/bin/python \
--fid_ref_images_dir /path/to/coco/
Evaluate datasets (specified in configs inside ./configs/datasets/
) on fid
, blip-vqa
, clipscore
and unidet
metrics (if evaluation on some dataset results in an error the program will move on to the next one without aborting the whole process):
python eval.py \
--config ./configs/datasets/ \
--methods fid blip-vqa clipscore unidet \
--t2i_compbench_path /path/to/T2I-CompBench/ \
--t2i_compbench_pyexe /path/to/T2I-CompBench/venv/bin/python \
--fid_ref_images_dir /path/to/coco/
To see all options for evaluation run python eval.py --help
.