RKE score

Paper: An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions

Mohammad Jalai ^1*, Cheuk Ting Li ², Farzan Farnia ²

¹ _{Isfahan University of Technology (IUT)}, ² _{The Chinese University of Hong Kong (CUHK)}

_{* Work done during an internship at CUHK}

1. Background

Abstract

The evaluation of generative models has received significant attention in the machine learning community. When applied to a multi-modal distribution which is common among image datasets, an intuitive evaluation criterion is the number of modes captured by the generative model. While several scores have been proposed to evaluate the quality and diversity of a model's generated data, the correspondence between existing scores and the number of modes in the distribution is unclear. In this work, we propose an information-theoretic diversity evaluation method for multi-modal underlying distributions. We define the R'enyi Kernel Entropy (RKE) as an evaluation score based on quantum information theory to measure the number of modes in generated samples. To interpret the proposed evaluation method, we show that the RKE score can output the number of modes of a mixture of sub-Gaussian components. We also prove estimation error bounds for estimating the RKE score from limited data, suggesting a fast convergence of the empirical RKE score to the score for the underlying data distribution. Utilizing the RKE score, we conduct an extensive evaluation of state-of-the-art generative models over standard image datasets. The numerical results indicate that while the recent algorithms for training generative models manage to improve the mode-based diversity over the earlier architectures, they remain incapable of capturing the full diversity of real data. Our empirical results provide a ranking of widely-used generative models based on the RKE score of their generated samples.

R'enyi Kernel Entropy Mode Count (RKE) and Relative R'enyi Kernel Entropy (RRKE)

Formulation

Toy example: Gaussian Distributions

Evaluation of Generative Models

2. Usage

Installation

Using PyPi

pip install rke-score

Manually

git clone https://github.com/mjalali/renyi-kernel-entropy-score
pip install -e .

Example

import numpy as np
from rke_score import RKE


num_real_samples = num_fake_samples = 10000
feature_dim = 1000

real_features = np.random.normal(loc=0.0, scale=1.0,
                                 size=[num_real_samples, feature_dim])

fake_features = np.random.normal(loc=0.0, scale=1.0,
                                 size=[num_fake_samples, feature_dim])

kernel = RKE(kernel_bandwidth=[0.2, 0.3, 0.4])


print(kernel.compute_rke_mc(fake_features))
print(kernel.compute_rrke(real_features, fake_features))

Guide to evaluate your model

You can evaluate your model using different feature extractors (mostly used: inceptionV3)

3. Cite our work

@inproceedings{
    jalali2023rke,
    title={An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions},
    author={Mohammad Jalali and Cheuk Ting Li and Farzan Farnia},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
    year={2023},
    url={https://openreview.net/forum?id=PdZhf6PiAb}
}