Updated readme for pretrained models and logs, smaller celeba, and ls…

…un generation script

README.md

@@ -17,16 +17,18 @@ pip install -r requirements.txt
 To setup (Open)MPI, check instructions on Horovod github [page](https://github.com/uber/horovod).
 
 ## Download datasets
-The datasets are in the Google Cloud locations `https://storage.googleapis.com/glow-demo/data/{dataset_name}-tfr.tar`. The dataset_names are below, we mention the exact preprocessing / downsampling method for a correct comparison of likelihood.
+For small scale experiments, use MNIST/CIFAR-10 (directly downloaded by `train.py` using keras)
+
+For larger scale experiments, the datasets used are in the Google Cloud locations `https://storage.googleapis.com/glow-demo/data/{dataset_name}-tfr.tar`. The dataset_names are below, we mention the exact preprocessing / downsampling method for a correct comparison of likelihood.
 
 Quantitative results
-- `imagenet-oord` - Unconditional ImageNet 32x32 and 64x64, as described in PixelRNN/RealNVP papers (we downloaded [this](http://image-net.org/small/download.php) processed version).
-- `lsun_realnvp` - LSUN 96x96. Random 64x64 crops taken at processing time, as described in RealNVP.
+- `imagenet-oord` - 20GB. Unconditional ImageNet 32x32 and 64x64, as described in PixelRNN/RealNVP papers (we downloaded [this](http://image-net.org/small/download.php) processed version).
+- `lsun_realnvp` - 140GB. LSUN 96x96. Random 64x64 crops taken at processing time, as described in RealNVP.
 
 Qualitative results
-- `celeba` - CelebA-HQ 1024x1024 dataset, as described in Progressive growing of GAN's.
-- `imagenet` - ImageNet 32x32 and 64x64 with class labels. Centre cropped, area downsampled.
-- `lsun` - LSUN 256x256. Centre cropped, area downsampled.
+- `celeba` - 4GB. CelebA-HQ 256x256 dataset, as described in Progressive growing of GAN's. For 1024x1024 version (120GB), use `celeba-full-tfr.tar` while downloading.
+- `imagenet` - 20GB. ImageNet 32x32 and 64x64 with class labels. Centre cropped, area downsampled.
+- `lsun` - 700GB. LSUN 256x256. Centre cropped, area downsampled.
 
 To download and extract celeb for example, run
 ```
@@ -35,6 +37,8 @@ tar -xvf celeb-tfr.tar
 ```
 Change `hps.data_dir` in train.py file to point to the above folder (or use the `--data_dir` flag when you run train.py)
 
+For `lsun`, since download can be quite big, you can instead follow the instructions in `data_loaders/generate_tfr/lsun.py` to generate the tfr file directly from LSUN images. `church_outdoor` will be the smallest category.
+
 ## Simple Train with 1 GPU
 
 Run wtih small depth to test
@@ -55,6 +59,11 @@ mpiexec -n 8 python train.py
 mpiexec -n 8 python train.py --problem cifar10 --image_size 32 --n_level 3 --depth 32 --flow_permutation [0/1/2] --flow_coupling [0/1] --seed [0/1/2] --learntop --lr 0.001
 ```
 
+Pretrained models, logs and samples
+```
+wget https://storage.googleapis.com/glow-demo/logs/abl-[reverse/shuffle/1x1]-[add/aff].tar
+```
+
 ##### CIFAR-10 Quantitative result
 
 ```
@@ -77,6 +86,11 @@ mpiexec -n 8 python train.py --problem imagenet-oord --image_size 64 --n_level 4
 mpiexec -n 8 python train.py --problem lsun_realnvp --category [bedroom/church_outdoor/tower] --image_size 64 --n_level 3 --depth 48 --flow_permutation 2 --flow_coupling 1 --seed 0 --learntop --lr 0.001 --n_bits_x 8
 ```
 
+Pretrained models, logs and samples
+```
+wget https://storage.googleapis.com/glow-demo/logs/lsun-rnvp-[bdr/crh/twr].tar
+```
+
 ##### CelebA-HQ 256x256 Qualitative result
 
 ```
@@ -88,6 +102,11 @@ mpiexec -n 40 python train.py --problem celeba --image_size 256 --n_level 6 --de
 mpiexec -n 40 python train.py --problem lsun --category [bedroom/church_outdoor/tower] --image_size [96/128] --n_level 5 --depth 64 --flow_permutation 2 --flow_coupling 0 --seed 0 --learntop --lr 0.001 --n_bits_x 5
 ```
 
+Logs and samples
+```
+wget https://storage.googleapis.com/glow-demo/logs/lsun-bdr-[96/128].tar
+```
+
 ##### Conditional CIFAR-10 Qualitative result
 ```
 mpiexec -n 8 python train.py --problem cifar10 --image_size 32 --n_level 3 --depth 32 --flow_permutation 2 --flow_coupling 0 --seed 0 --learntop --lr 0.001 --n_bits_x 5 --ycond --weight_y=0.01

data_loaders/generate_tfr/lsun.py

@@ -13,18 +13,23 @@
 # limitations under the License.
 # ==============================================================================
 
-r"""LSUN dataset formatting.
-Download and format the LSUN dataset as follow:
+""""
+LSUN dataset
+
+# Get image files
+Download the LSUN dataset as follows:
 git clone https://github.com/fyu/lsun.git
 cd lsun
 python2.7 download.py -c [CATEGORY]
-Then unzip the downloaded .zip files before executing:
+Unzip the downloaded .zip files and execute:
 python2.7 data.py export [IMAGE_DB_PATH] --out_dir [LSUN_FOLDER] --flat
-Then use the script as follow:
-python lsun_formatting.py \
-    --file_out [OUTPUT_FILE_PATH_PREFIX] \
-    --fn_root [LSUN_FOLDER]
+
+# Get tfr file from images
+Use this script to generate the tfr file.
+python lsun.py --category [CATEGORY] --lsun_dir [LSUN_FOLDER] --tfrecord_dir [OUTPUT_FOLDER] --write [--realnvp]
+Without realnvp flag you get 256x256 centre cropped area downsampled images, with flag you get 96x96 images with realnvp preprocessing.
 """
+
 from __future__ import print_function
 
 import os
@@ -133,7 +138,7 @@ def dump(fn_root, tfrecord_dir, max_res, expected_images, shards, write, realnvp
     parser = argparse.ArgumentParser()
     parser.add_argument("--category", type=str, help="LSUN category")
     parser.add_argument("--realnvp", action='store_true',
-                        help="Preprocess according to realnvp")
+                        help="Use this flag to do realnvp preprocessing instead of our centre-crops")
     parser.add_argument("--res", type=int, default=256, help="Image size")
     parser.add_argument("--lsun_dir", type=str,
                         required=True, help="place of lsun dir")