Memory leak in eval functions

[hannesdm]

System information

• OS Platform and Distribution: Windows 10 / Google Colab
• TensorFlow installed from binary: 2.1.0 - 2.2.0-rc3
• Tensorflow-gan version: 2.0.0 and master

Describe the current behavior
Calling tfg.eval.inception_score and tfg.eval.frechet_inception_distance with a large number of images or successively with a low number of images will cause a crash due to an out-of-memory exception. Calling both with a single batch, once without consecutive calls, works.
This likely occurs with other functions as well but has only been tested with inception_score and frechet_inception_distance.

Describe the expected behavior
As long as a single batch fits into memory, consecutive calls to the function should als work. The functions should also work with a large number of input image, but a fitting value for num_batches where you know that a single batch will fit in memory.

Standalone code to reproduce the issue

import tensorflow_gan as tfg
import numpy as np

a = np.random.uniform(size=(100,32,32,3)).astype('float32')
tfg.eval.inception_score(a, num_batches=100)

OR

a = np.random.uniform(size=(100,32,32,3)).astype('float32')
for i in range(a.shape[0]):
  tfg.eval.inception_score(np.expand_dims(a[i],0), num_batches=1)

Colab link: https://colab.research.google.com/drive/1ob6in02YjUB7iCbhKnRvD5FPbWpxc1ar