Official code for Improving the Evaluation of Generative Models with Fuzzy Logic by Julian Niedermeier and Gonçalo Mordido, presented at AAAI 2020 Meta-Eval.
Setup environment with Conda:
- For Tensorflow CPU:
conda env create -f environment.yml - For Tensorflow GPU:
conda env create -f environment-gpu.yml
Note: Tensorflow is only required by Improved Precision and Recall (IMPAR) and the Python files under data/. If you already have preprocessed data in the form generated by the files under data/ you do not require Tensorflow.
Note: Only supports Tensorflow v2.x
data: Contains code to generate raw data for the metrics.
figures: Contains figures from the paper.
metrics: Contains implementations for all the metrics.
misc: Contains helper code (e.g. used image distortions).
models: Contains code to use various auxiliary classifiers. See models/frozen_models/README.md for more details.
plot: Contains code to plot the figures as reported in the paper.
scripts: Contains shell scripts (containing a list of python commands) to prepare the data and run all experiments.
Files under root:
fid.py: Runs Fréchet Inception Distance.fti.py: Runs Fuzzy Topology Impact.impar.py: Runs Improved Precision and Recall.is.py: Runs Inception Score.prd.py: Run Precision and Recall for Distributions.
Scripts to prepare the data and run all experiments are located under scripts/.
Note: Scripts should be executed from the root directory of this repository.
scripts/raw_data: Contains scripts to generate data for the various experiments.
scripts/<metric>: Contains scripts to run experiments for a metric.
scripts/plot: Contains the commands to plot the experiment results.
If you would like to contribute your own experiments or metrics feel free to open up a pull request to make evaluation easier in the future.
Niedermeier, Julian, Gonçalo Mordido, and Christoph Meinel. "Improving the Evaluation of Generative Models with Fuzzy Logic." arXiv preprint arXiv:2002.03772 (2020).
License for all files except those noted otherwise: Apache License 2.0