tf-auto-encoder.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Auto-Encoder\n",
    "\n",
    "Der Autoencoder versucht, eine Funktion $h_{W,b}(x) \\approx x$ zu lernen. Mit anderen Worten, er versucht, eine Annäherung an die Identitätsfunktion zu lernen, um $\\hat{x}$ auszugeben, die x ähnlich ist. Die Identitätsfunktion scheint eine besonders triviale Funktion zu sein, die zu lernen ist; aber indem wir dem Netzwerk Beschränkungen auferlegen, wie z.B. durch Begrenzung der Anzahl der Neuronen in Hidden Layers, können wir eine interessante Struktur über die Daten entdecken. \n",
    "\n",
    "Als konkretes Beispiel nehmen wir an, die Eingaben x sind die Pixel-Intensitätswerte aus einem 28×28-Bild (784 Pixel), also n=784, und es gibt $s_2$=64 Neuronen in dem Hidden Layer $L_2$. Beachten Sie, dass wir auch $y \\in \\mathbb{R}^{784}$ haben. \n",
    "\n",
    "Da es nur 64 Neuronen gibt, ist das Netzwerk gezwungen, eine \"komprimierte\" Darstellung der Eingabe zu lernen. D.h., wenn es nur den Vektor der Aktivierungen der Neuronen $a^{(2)} \\in \\mathbb{R}^{64}$ erhält, muss es versuchen, die 784-Pixel-Eingabe x zu \"rekonstruieren\". \n",
    "\n",
    "Wenn die Eingabe völlig zufällig wäre, wäre diese Komprimierungsaufgabe sehr schwierig. Wenn die Daten jedoch strukturiert sind, z.B. wenn einige der Eingabemerkmale korreliert sind, dann ist dieser Algorithmus in der Lage, einige dieser Korrelationen zu entdecken."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fashion MNIST Dataset\n",
    "\n",
    "`Fashion-MNIST` ist ein Datensatz von Zalandos Artikelbildern - bestehend aus einem Trainingssatz mit 60.000 Beispielen und einem Testsatz mit 10.000 Beispielen. \n",
    "\n",
    "Jedes Beispiel ist ein Graustufenbild im Format 28x28, das mit einem Label aus 10 Klassen verbunden ist. Fashion-MNIST ist ein direkter Drop-in-Ersatz für den ursprünglichen MNIST-Datensatz von handgeschriebenen Ziffern. Der Datensatz hat die gleiche Bildgröße und die gleiche Struktur von Trainings- und Test-Splits.\n",
    "\n",
    "Jedes Bild trägt einen der folgenden Label:\n",
    "\n",
    "Label | Description\n",
    "--- | ---\n",
    "0|T-shirt/top\n",
    "1|Trouser\n",
    "2|Pullover\n",
    "3|Dress\n",
    "4|Coat\n",
    "5|Sandal\n",
    "6|Shirt\n",
    "7|Sneaker\n",
    "8|Bag\n",
    "9|Ankle boot\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.keras.backend.set_floatx('float32')\n",
    "\n",
    "from tensorflow import keras\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# MNIST Dataset parameters.\n",
    "num_features = 784 # data features (img shape: 28*28).\n",
    "\n",
    "# Training parameters.\n",
    "learning_rate = 0.01\n",
    "training_steps = 5000\n",
    "batch_size = 64\n",
    "display_step = training_steps / 10\n",
    "\n",
    "# Network Parameters\n",
    "num_hidden_1 = 128 # 1st layer num features.\n",
    "num_hidden_2 = 64 # 2nd layer num features (the latent dim)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Daten laden"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fashion_mnist = keras.datasets.fashion_mnist\n",
    "(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Daten vorbereiten\n",
    "\n",
    "Jedes Bild wird konvertiert zu float32, normalisiert auf das Intervall [0, 1] and ausgestreckt auf ein 1-dimensionales Array von 784 Features (28*28)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to float32.\n",
    "x_train, x_test = x_train.astype(np.float32), x_test.astype(np.float32)\n",
    "# Flatten images to 1-D vector of 784 features (28*28).\n",
    "x_train, x_test = x_train.reshape([-1, num_features]), x_test.reshape([-1, num_features])\n",
    "# Normalize images value from [0, 255] to [0, 1].\n",
    "x_train, x_test = x_train / 255., x_test / 255."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wir erzeugen uns wieder einen iterierbaren **`tf.data.Dataset`** wie wir es schon beim Neuralen Netz aus den MNIST Ziffern gesehen hatten:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
    "train_data = train_data.repeat().shuffle(60000).batch(batch_size).prefetch(1)\n",
    "\n",
    "test_data = tf.data.Dataset.from_tensor_slices((x_test, y_test))\n",
    "test_data = test_data.repeat().batch(batch_size).prefetch(1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Autoencoder\n",
    "\n",
    "Als nächstes definieren wir unser Autoencoder Netzwerk, diesmal allerdings nicht mehr der High-Level API Keras, sondern mit den Tensorflow zu Fuß.\n",
    "\n",
    "Wir verwenden jeweils zwei Hidden Layer für das En- und Decoding, gehen dabei über 128 Neuronen auf 64 Neuronen.\n",
    "\n",
    "![](autoencode.png)\n",
    "\n",
    "### Variablen für Weight & Bias der Layer\n",
    "\n",
    "`tf.Variable` sind die Elemente, die der Algorithmus trainieren wird, also unsere Weights und Bias:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A random value generator to initialize weights.\n",
    "random_normal = tf.initializers.RandomNormal()\n",
    "\n",
    "weights = {\n",
    "    'encoder_h1': tf.Variable(random_normal([num_features, num_hidden_1])),\n",
    "    'encoder_h2': tf.Variable(random_normal([num_hidden_1, num_hidden_2])),\n",
    "    'decoder_h1': tf.Variable(random_normal([num_hidden_2, num_hidden_1])),\n",
    "    'decoder_h2': tf.Variable(random_normal([num_hidden_1, num_features])),\n",
    "}\n",
    "biases = {\n",
    "    'encoder_b1': tf.Variable(random_normal([num_hidden_1])),\n",
    "    'encoder_b2': tf.Variable(random_normal([num_hidden_2])),\n",
    "    'decoder_b1': tf.Variable(random_normal([num_hidden_1])),\n",
    "    'decoder_b2': tf.Variable(random_normal([num_features])),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building the encoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encoder(x):\n",
    "    # Encoder Hidden layer with sigmoid activation.\n",
    "    layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['encoder_h1']), biases['encoder_b1']))\n",
    "    \n",
    "    # Encoder Hidden layer with sigmoid activation.\n",
    "    layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['encoder_h2']), biases['encoder_b2']))\n",
    "    \n",
    "    return layer_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building the decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decoder(x):\n",
    "    # Decoder Hidden layer with sigmoid activation.\n",
    "    layer_1 = tf.nn.sigmoid(tf.add(tf.matmul(x, weights['decoder_h1']), biases['decoder_b1']))\n",
    "    \n",
    "    # Decoder Hidden layer with sigmoid activation.\n",
    "    layer_2 = tf.nn.sigmoid(tf.add(tf.matmul(layer_1, weights['decoder_h2']), biases['decoder_b2']))\n",
    "    \n",
    "    return layer_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mean square loss between original images and reconstructed ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mean_square(reconstructed, original):\n",
    "    return tf.reduce_mean(tf.pow(original - reconstructed, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adam optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.optimizers.Adam(learning_rate=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimization process\n",
    "\n",
    "TensorFlow bietet die `tf.GradientTape` API für die **automatische Differenzierung**, d.h. die Berechnung des Gradienten einer Berechnung in Bezug auf seine Eingangsvariablen. \n",
    "\n",
    "TensorFlow zeichnet alle Operationen, die im Kontext eines tf.GradientTape ausgeführt werden, auf einem \"Band\" auf. \n",
    "\n",
    "TensorFlow verwendet dann dieses Band und die Gradienten, die jeder aufgezeichneten Operation zugeordnet sind, um die Gradienten einer \"aufgezeichneten\" Berechnung zu berechnen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_optimization(x):\n",
    "    # Wrap computation inside a GradientTape for automatic differentiation.\n",
    "    with tf.GradientTape() as g:\n",
    "        reconstructed_image = decoder(encoder(x))\n",
    "        loss = mean_square(reconstructed_image, x)\n",
    "\n",
    "    # Variables to update, i.e. trainable variables.\n",
    "    trainable_variables = {**weights, **biases}.values()\n",
    "    \n",
    "    # Compute gradients.\n",
    "    gradients = g.gradient(loss, trainable_variables)\n",
    "    \n",
    "    # Update W and b following gradients.\n",
    "    optimizer.apply_gradients(zip(gradients, trainable_variables))\n",
    "    \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trainiere das Modell\n",
    "\n",
    "`take(count)` holt sich die nächsten `count` Batch-Elemente aus dem Dataset-Strom.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loss_results = []\n",
    "\n",
    "for step, (batch_x, _) in enumerate(train_data.take(training_steps + 1)):\n",
    "    \n",
    "    # Run the optimization.\n",
    "    loss = run_optimization(batch_x)\n",
    "    train_loss_results.append(loss)\n",
    "    \n",
    "    if step % display_step == 0:\n",
    "        print(\"step: %i, loss: %f\" % (step, loss))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, figsize=(8, 6))\n",
    "\n",
    "ax.set_ylabel(\"Loss\", fontsize=14)\n",
    "ax.set_xlabel(\"Epoch\", fontsize=14)\n",
    "ax.plot(train_loss_results);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing and Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Encode and decode images from test set and visualize their reconstruction.\n",
    "n = 16\n",
    "\n",
    "canvas_orig = np.empty((28, 28 * n))\n",
    "canvas_encod = np.ones((28, 28 * n))\n",
    "canvas_recon = np.empty((28, 28 * n))\n",
    "\n",
    "for i, (batch_x, _) in enumerate(test_data.take(n)):\n",
    "    # Encode and decode the digit image.\n",
    "    encoded_images = encoder(batch_x)\n",
    "    reconstructed_images = decoder(encoded_images)\n",
    "\n",
    "    part = slice(i * 28, (i + 1) * 28)\n",
    "    part_encod = slice(i * 28+10, i * 28 + 18)\n",
    "\n",
    "    # Display original images.\n",
    "    img = batch_x[i].numpy().reshape([28, 28])\n",
    "    canvas_orig[0:28, part] = img\n",
    "\n",
    "    # Display Encodings.\n",
    "    img = encoded_images[i].numpy().reshape([8, 8])\n",
    "    canvas_encod[10:18, part_encod] = img\n",
    "\n",
    "    # Display reconstructed images.\n",
    "    reconstr_img = reconstructed_images[i].numpy().reshape([28, 28])\n",
    "    canvas_recon[0:28, part] = reconstr_img\n",
    "\n",
    "print(\"Original Images\")\n",
    "plt.figure(figsize=(n, n))\n",
    "plt.axis('off')\n",
    "plt.imshow(canvas_orig, origin=\"upper\", cmap=\"gray\")\n",
    "plt.show()\n",
    "\n",
    "print(\"Encoded Images\")     \n",
    "plt.figure(figsize=(n, n))\n",
    "plt.axis('off')\n",
    "plt.imshow(canvas_encod, origin=\"upper\", cmap=\"gray\")\n",
    "plt.show()\n",
    "\n",
    "print(\"Reconstructed Images\")\n",
    "plt.figure(figsize=(n, n))\n",
    "plt.axis('off')\n",
    "plt.imshow(canvas_recon, origin=\"upper\", cmap=\"gray\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}