notebooks/Comet-Confusion-Matrix.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.comet.ml/images/logo_comet_light.png\" width=\"200px\"/>\n",
    "\n",
    "# Comet.ml Confusion Matrix\n",
    "\n",
    "*This page is available as an executable or viewable **Jupyter Notebook**:* \n",
    "\n",
    "<a href=\"https://mybinder.org/v2/gh/comet-ml/comet-examples/master?filepath=notebooks%2FComet-Confusion-Matrix.ipynb\" target=\"_parent\"><img src=\"https://mybinder.org/badge_logo.svg\"></a>\n",
    "\n",
    "<a href=\"https://nbviewer.jupyter.org/github/comet-ml/comet-examples/blob/master/notebooks/Comet-Confusion-Matrix.ipynb\" target=\"_parent\"><img  src=\"https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.png\" width=\"109\" height=\"20\"></a>\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/comet-ml/comet-examples/blob/master/notebooks/Comet-Confusion-Matrix.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\"></a>\n",
    "\n",
    "<hr/>\n",
    "Comet.ml can generate a variety of visualizations, including line charts, scatter charts, bar charts, and histograms. This notebook explores Comet's confusion matrix chart."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "The first thing we'll do in this notebook tutorial is install **comet_ml** and other items that we'll need for this demonstration. That will include **keras**, **tensorflow**, and **numpy**.\n",
    "\n",
    "First, install comet_ml (you may want to do this slightly differently on your computer):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade comet_ml>=3.10.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now tensorflow, keras, and numpy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install keras tensorflow numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the output may suggest, if anything got updated, it might be a good idea to restart the kernel and continue from here (should not be required on Google's colab)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import comet_ml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run the following experiments, you'll need to import comet_ml, and have your Comet API key configured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import comet_ml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET INFO: Comet API key is valid\n"
     ]
    }
   ],
   "source": [
    "comet_ml.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1: Simple Confusion Matrix\n",
    "\n",
    "First, we will create an experiment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET WARNING: As you are running in a Jupyter environment, you will need to call `experiment.end()` when finished to ensure all metrics and code are logged before exiting.\n",
      "COMET INFO: Experiment is live on comet.ml https://www.comet.ml/dsblank/confusion-matrix/ab5ffc64fc6a47b9bbd2e37d279703cb\n",
      "\n"
     ]
    }
   ],
   "source": [
    "experiment = comet_ml.Experiment(\n",
    "    project_name=\"confusion-matrix\", \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a simple example, let's consider that we have these six patterns that are our output targets (desired output):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "desired_output = [0, 1, 2, 0, 1, 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imagine that this is a classification task where each target (desired output) is composed of three output values, with one unit \"on\" (set to 1) and the others \"off\" (set to 0). This is sometimes called a \"one-hot\" representation and is a common way of representing categories. There are 6 patterns, where there are 2 each for category.\n",
    "\n",
    "In the above representation, we use \"labels\" to identify the correct classification, but we could have easily have used the one-hot representation as well:\n",
    "\n",
    "```python\n",
    "desired_output = [\n",
    "    [1, 0, 0],\n",
    "    [0, 1, 0],\n",
    "    [0, 0, 1],\n",
    "    [1, 0, 0],\n",
    "    [0, 1, 0],\n",
    "    [0, 0, 1],\n",
    " ]\n",
    "```\n",
    "\n",
    "Now, let's make up some sample data that an model might produce. Let's say initially that the output is pretty random and doesn't even add up to 1 for each row. This may be unrealistic as many such classification tasks might use an error/loss output metric that is based on [cross entropy](http://www.cse.unsw.edu.au/~billw/cs9444/crossentropy.html) which would make the sum of values closer to 1. That might be desirable, but is not required for our example here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "actual_output = [\n",
    "    [0.1, 0.5, 0.4],\n",
    "    [0.2, 0.2, 0.3],\n",
    "    [0.7, 0.4, 0.5],\n",
    "    [0.3, 0.8, 0.3],\n",
    "    [0.0, 0.5, 0.3],\n",
    "    [0.1, 0.5, 0.5],\n",
    " ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal now is to visualize how much the model mixes up the categories. That is, we'd like to see the Confusion Matrix comparing all categories against each other. We can do that easily by simply logging it with the experiment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment.log_confusion_matrix(desired_output, actual_output);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it! We can now end the experiment and take a look at the resulting matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET INFO: ---------------------------\n",
      "COMET INFO: Comet.ml Experiment Summary\n",
      "COMET INFO: ---------------------------\n",
      "COMET INFO:   Data:\n",
      "COMET INFO:     display_summary_level : 1\n",
      "COMET INFO:     url                   : https://www.comet.ml/dsblank/confusion-matrix/ab5ffc64fc6a47b9bbd2e37d279703cb\n",
      "COMET INFO:   Uploads:\n",
      "COMET INFO:     confusion-matrix         : 1\n",
      "COMET INFO:     environment details      : 1\n",
      "COMET INFO:     filename                 : 1\n",
      "COMET INFO:     git metadata             : 1\n",
      "COMET INFO:     git-patch (uncompressed) : 1 (71 KB)\n",
      "COMET INFO:     installed packages       : 1\n",
      "COMET INFO:     notebook                 : 1\n",
      "COMET INFO:     os packages              : 1\n",
      "COMET INFO:     source_code              : 1\n",
      "COMET INFO: ---------------------------\n",
      "COMET INFO: Uploading metrics, params, and assets to Comet before program termination (may take several seconds)\n",
      "COMET INFO: The Python SDK has 3600 seconds to finish before aborting...\n"
     ]
    }
   ],
   "source": [
    "experiment.end()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"800px\"\n",
       "            src=\"https://www.comet.ml/dsblank/confusion-matrix/ab5ffc64fc6a47b9bbd2e37d279703cb?experiment-tab=confusionMatrix\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f21c2803b80>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "experiment.display(tab=\"confusion-matrices\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more details on this tab, please see the details on the [Confusion Matrix user interface](https://www.comet.ml/docs/user-interface/#confusion-matrices-tab)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example #2: Log Confusion Matrices During Learning\n",
    "\n",
    "This example will create a series of confusion matrices showing how the model gets less confused as training proceeds.\n",
    "\n",
    "We will train the standard MNIST digit classification task.\n",
    "\n",
    "We import the items that we will need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.callbacks import Callback\n",
    "from tensorflow.keras.layers import Dense\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.optimizers import RMSprop\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "from keras.datasets import mnist\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_classes = 10\n",
    "\n",
    "# the data, shuffled and split between train and test sets\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "\n",
    "x_train = x_train.reshape(60000, 784)\n",
    "x_test = x_test.reshape(10000, 784)\n",
    "x_train = x_train.astype(\"float32\")\n",
    "x_test = x_test.astype(\"float32\")\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = to_categorical(y_train, num_classes)\n",
    "y_test = to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a function to create the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_model():\n",
    "    model = Sequential()\n",
    "    model.add(Dense(128, activation=\"sigmoid\", input_shape=(784,)))\n",
    "    model.add(Dense(128, activation=\"sigmoid\"))\n",
    "    model.add(Dense(128, activation=\"sigmoid\"))\n",
    "    model.add(Dense(10, activation=\"softmax\"))\n",
    "    model.compile(\n",
    "        loss=\"categorical_crossentropy\", optimizer=RMSprop(), metrics=[\"accuracy\"]\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define a Keras callback to log the confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ConfusionMatrixCallback(Callback):\n",
    "    def __init__(self, experiment, inputs, targets):\n",
    "        self.experiment = experiment\n",
    "        self.inputs = inputs\n",
    "        self.targets = targets\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs={}):\n",
    "        predicted = self.model.predict(self.inputs)\n",
    "        self.experiment.log_confusion_matrix(\n",
    "            self.targets,\n",
    "            predicted,\n",
    "            title=\"Confusion Matrix, Epoch #%d\" % (epoch + 1),\n",
    "            file_name=\"confusion-matrix-%03d.json\" % (epoch + 1),\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And create another Comet experiment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET WARNING: As you are running in a Jupyter environment, you will need to call `experiment.end()` when finished to ensure all metrics and code are logged before exiting.\n",
      "COMET INFO: Experiment is live on comet.ml https://www.comet.ml/dsblank/confusion-matrix/f1fb7618720c40e5be33b5100ad3b776\n",
      "\n"
     ]
    }
   ],
   "source": [
    "experiment = comet_ml.Experiment(\n",
    "    project_name=\"confusion-matrix\", \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before any training, we want to log the confusion so that we can see what it looks like before any adjusting of weights in the network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = create_model()\n",
    "\n",
    "y_predicted = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also supply the step (zero, before training), a title, and file_name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment.log_confusion_matrix(\n",
    "    y_test,\n",
    "    y_predicted,\n",
    "    step=0,\n",
    "    title=\"Confusion Matrix, Epoch #0\",\n",
    "    file_name=\"confusion-matrix-%03d.json\" % 0,\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now create the callback and train the data for 5 epochs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET INFO: Ignoring automatic log_parameter('verbose') because 'keras:verbose' is in COMET_LOGGING_PARAMETERS_IGNORE\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "500/500 [==============================] - 5s 8ms/step - loss: 1.2974 - accuracy: 0.6097 - val_loss: 0.3082 - val_accuracy: 0.9097\n",
      "Epoch 2/5\n",
      "500/500 [==============================] - 3s 7ms/step - loss: 0.2815 - accuracy: 0.9166 - val_loss: 0.2080 - val_accuracy: 0.9372\n",
      "Epoch 3/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.2041 - accuracy: 0.9396 - val_loss: 0.1647 - val_accuracy: 0.9520\n",
      "Epoch 4/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.1431 - accuracy: 0.9574 - val_loss: 0.1349 - val_accuracy: 0.9582\n",
      "Epoch 5/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.1198 - accuracy: 0.9647 - val_loss: 0.1178 - val_accuracy: 0.9638\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f211a138430>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "callback = ConfusionMatrixCallback(experiment, x_test, y_test)\n",
    "\n",
    "model.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    batch_size=120,\n",
    "    epochs=5,\n",
    "    callbacks=[callback],\n",
    "    validation_data=(x_test, y_test),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET INFO: ---------------------------\n",
      "COMET INFO: Comet.ml Experiment Summary\n",
      "COMET INFO: ---------------------------\n",
      "COMET INFO:   Data:\n",
      "COMET INFO:     display_summary_level : 1\n",
      "COMET INFO:     url                   : https://www.comet.ml/dsblank/confusion-matrix/f1fb7618720c40e5be33b5100ad3b776\n",
      "COMET INFO:   Metrics [count] (min, max):\n",
      "COMET INFO:     accuracy [5]                 : (0.785966694355011, 0.9656999707221985)\n",
      "COMET INFO:     batch_accuracy [250]         : (0.06666667014360428, 0.9666666388511658)\n",
      "COMET INFO:     batch_loss [250]             : (0.11659321188926697, 2.5504069328308105)\n",
      "COMET INFO:     epoch_duration [5]           : (3.6289265220984817, 5.298398307990283)\n",
      "COMET INFO:     loss [5]                     : (0.11707635223865509, 0.7549315094947815)\n",
      "COMET INFO:     val_accuracy [5]             : (0.9096999764442444, 0.9638000130653381)\n",
      "COMET INFO:     val_loss [5]                 : (0.11783557385206223, 0.3082138001918793)\n",
      "COMET INFO:     validate_batch_accuracy [45] : (0.8865079283714294, 1.0)\n",
      "COMET INFO:     validate_batch_loss [45]     : (0.02385331317782402, 0.3838881850242615)\n",
      "COMET INFO:   Others:\n",
      "COMET INFO:     trainable_params : 134794\n",
      "COMET INFO:   Parameters:\n",
      "COMET INFO:     Optimizer             : RMSprop\n",
      "COMET INFO:     RMSprop_centered      : 1\n",
      "COMET INFO:     RMSprop_decay         : 1\n",
      "COMET INFO:     RMSprop_epsilon       : 1e-07\n",
      "COMET INFO:     RMSprop_learning_rate : 0.001\n",
      "COMET INFO:     RMSprop_momentum      : 1\n",
      "COMET INFO:     RMSprop_name          : RMSprop\n",
      "COMET INFO:     RMSprop_rho           : 0.9\n",
      "COMET INFO:     epochs                : 5\n",
      "COMET INFO:     steps                 : 500\n",
      "COMET INFO:   Uploads [count]:\n",
      "COMET INFO:     confusion-matrix [6]     : 6\n",
      "COMET INFO:     environment details      : 1\n",
      "COMET INFO:     filename                 : 1\n",
      "COMET INFO:     git metadata             : 1\n",
      "COMET INFO:     git-patch (uncompressed) : 1 (71 KB)\n",
      "COMET INFO:     installed packages       : 1\n",
      "COMET INFO:     model graph              : 1\n",
      "COMET INFO:     notebook                 : 1\n",
      "COMET INFO:     os packages              : 1\n",
      "COMET INFO:     source_code              : 1\n",
      "COMET INFO: ---------------------------\n",
      "COMET INFO: Waiting for completion of the file uploads (may take several seconds)\n",
      "COMET INFO: The Python SDK has 10800 seconds to finish before aborting...\n",
      "COMET INFO: Still uploading 3 file(s)\n"
     ]
    }
   ],
   "source": [
    "experiment.end()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we take a look at the matrices created over the training. You can switch between confusion matrices by selecting the name in the upper, left-hand corner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"800px\"\n",
       "            src=\"https://www.comet.ml/dsblank/confusion-matrix/f1fb7618720c40e5be33b5100ad3b776?experiment-tab=confusionMatrix\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f211cbed5b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "experiment.display(tab=\"confusion-matrices\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 3: Create Images for Each Sample\n",
    "\n",
    "For this example, we will create images for each example, up to 25 examples per cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll do the same steps as above:\n",
    "\n",
    "1. create an experiment\n",
    "2. create the model\n",
    "3. log inital confusion\n",
    "4. create a callback\n",
    "5. train the model\n",
    "6. display the experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET WARNING: As you are running in a Jupyter environment, you will need to call `experiment.end()` when finished to ensure all metrics and code are logged before exiting.\n",
      "COMET INFO: Experiment is live on comet.ml https://www.comet.ml/dsblank/confusion-matrix/ddbe3072aad6417085c070e1ba414945\n",
      "\n"
     ]
    }
   ],
   "source": [
    "experiment = comet_ml.Experiment(\n",
    "    project_name=\"confusion-matrix\", \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = create_model()\n",
    "\n",
    "y_predicted = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment.log_confusion_matrix(\n",
    "    y_test,\n",
    "    y_predicted,\n",
    "    step=0,\n",
    "    title=\"Confusion Matrix, Epoch #0\",\n",
    "    file_name=\"confusion-matrix-%03d.json\" % 0,\n",
    "    images=x_test,\n",
    "    image_shape=(28, 28),\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ConfusionMatrixCallbackWithImages(Callback):\n",
    "    def __init__(self, experiment, inputs, targets):\n",
    "        self.experiment = experiment\n",
    "        self.inputs = inputs\n",
    "        self.targets = targets\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs={}):\n",
    "        predicted = self.model.predict(self.inputs)\n",
    "        self.experiment.log_confusion_matrix(\n",
    "            self.targets,\n",
    "            predicted,\n",
    "            title=\"Confusion Matrix, Epoch #%d\" % (epoch + 1),\n",
    "            file_name=\"confusion-matrix-%03d.json\" % (epoch + 1),\n",
    "            images=self.inputs,\n",
    "            image_shape=(28, 28),\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll train as before. \n",
    "\n",
    "**NOTE: this takes a lot longer than before, but we'll see how to speed this up in the next example.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "500/500 [==============================] - 4s 6ms/step - loss: 1.3291 - accuracy: 0.5844 - val_loss: 0.3127 - val_accuracy: 0.9076\n",
      "Epoch 2/5\n",
      "500/500 [==============================] - 3s 7ms/step - loss: 0.2766 - accuracy: 0.9190 - val_loss: 0.2006 - val_accuracy: 0.9396\n",
      "Epoch 3/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.1914 - accuracy: 0.9432 - val_loss: 0.1742 - val_accuracy: 0.9479\n",
      "Epoch 4/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.1497 - accuracy: 0.9542 - val_loss: 0.1320 - val_accuracy: 0.9595\n",
      "Epoch 5/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.1245 - accuracy: 0.9630 - val_loss: 0.1183 - val_accuracy: 0.9656\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f20d8569100>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "callback = ConfusionMatrixCallbackWithImages(experiment, x_test, y_test)\n",
    "\n",
    "model.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    batch_size=120,\n",
    "    epochs=5,\n",
    "    callbacks=[callback],\n",
    "    validation_data=(x_test, y_test),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET INFO: ---------------------------\n",
      "COMET INFO: Comet.ml Experiment Summary\n",
      "COMET INFO: ---------------------------\n",
      "COMET INFO:   Data:\n",
      "COMET INFO:     display_summary_level : 1\n",
      "COMET INFO:     url                   : https://www.comet.ml/dsblank/confusion-matrix/ddbe3072aad6417085c070e1ba414945\n",
      "COMET INFO:   Metrics [count] (min, max):\n",
      "COMET INFO:     accuracy [5]                 : (0.7693666815757751, 0.9639833569526672)\n",
      "COMET INFO:     batch_accuracy [250]         : (0.1037878766655922, 0.9833333492279053)\n",
      "COMET INFO:     batch_loss [250]             : (0.07256641238927841, 2.432967185974121)\n",
      "COMET INFO:     epoch_duration [5]           : (31.040708475047722, 39.406505106948316)\n",
      "COMET INFO:     loss [5]                     : (0.12004319578409195, 0.7886505126953125)\n",
      "COMET INFO:     val_accuracy [5]             : (0.9075999855995178, 0.9656000137329102)\n",
      "COMET INFO:     val_loss [5]                 : (0.1183130219578743, 0.31273534893989563)\n",
      "COMET INFO:     validate_batch_accuracy [45] : (0.8801587224006653, 0.9833333492279053)\n",
      "COMET INFO:     validate_batch_loss [45]     : (0.04573925957083702, 0.380536288022995)\n",
      "COMET INFO:   Others:\n",
      "COMET INFO:     trainable_params : 134794\n",
      "COMET INFO:   Parameters:\n",
      "COMET INFO:     Optimizer             : RMSprop\n",
      "COMET INFO:     RMSprop_centered      : 1\n",
      "COMET INFO:     RMSprop_decay         : 1\n",
      "COMET INFO:     RMSprop_epsilon       : 1e-07\n",
      "COMET INFO:     RMSprop_learning_rate : 0.001\n",
      "COMET INFO:     RMSprop_momentum      : 1\n",
      "COMET INFO:     RMSprop_name          : RMSprop\n",
      "COMET INFO:     RMSprop_rho           : 0.9\n",
      "COMET INFO:     epochs                : 5\n",
      "COMET INFO:     steps                 : 500\n",
      "COMET INFO:   Uploads [count]:\n",
      "COMET INFO:     confusion-matrix [6]     : 6\n",
      "COMET INFO:     environment details      : 1\n",
      "COMET INFO:     filename                 : 1\n",
      "COMET INFO:     git metadata             : 1\n",
      "COMET INFO:     git-patch (uncompressed) : 1 (71 KB)\n",
      "COMET INFO:     images [4026]            : 4026\n",
      "COMET INFO:     installed packages       : 1\n",
      "COMET INFO:     model graph              : 1\n",
      "COMET INFO:     notebook                 : 1\n",
      "COMET INFO:     os packages              : 1\n",
      "COMET INFO:     source_code              : 1\n",
      "COMET INFO: ---------------------------\n",
      "COMET INFO: Uploading 1 metrics, params and output messages\n"
     ]
    }
   ],
   "source": [
    "experiment.end()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"800px\"\n",
       "            src=\"https://www.comet.ml/dsblank/confusion-matrix/ddbe3072aad6417085c070e1ba414945?experiment-tab=confusionMatrix\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f20d8546af0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "experiment.display(tab=\"confusion-matrices\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is very nice about this is that if you click on a cell, you can see examples of the types of digits that fall into this group. For example if you click in the cell counting the confusion between 8's and 0's you'll see a sample of exactly which of those images.\n",
    "\n",
    "However, there is a large issue with this example. Looking at the summary above, you can see that many thousands of images were uploaded. In addition, if you explore the confusion matrices over the course of learning, you'll see different examples for every epoch. The next example fixes that issue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 4: Reuse ConfusionMatrix instance\n",
    "\n",
    "Now, we want to create example images for each of the cells in the matrix. In addition, we want to re-use the images if we can."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a callback, like, before; however, this time we will keep track of an instance of the `ConfusionMatrix`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ConfusionMatrixCallbackReuseImages(Callback):\n",
    "    def __init__(self, experiment, inputs, targets, confusion_matrix):\n",
    "        self.experiment = experiment\n",
    "        self.inputs = inputs\n",
    "        self.targets = targets\n",
    "        self.confusion_matrix = confusion_matrix\n",
    "\n",
    "    def on_epoch_end(self, epoch, logs={}):\n",
    "        predicted = self.model.predict(self.inputs)\n",
    "        self.confusion_matrix.compute_matrix(self.targets, predicted, images=self.inputs)\n",
    "        self.experiment.log_confusion_matrix(\n",
    "            matrix=self.confusion_matrix,\n",
    "            title=\"Confusion Matrix, Epoch #%d\" % (epoch + 1),\n",
    "            file_name=\"confusion-matrix-%03d.json\" % (epoch + 1),\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create another Comet experiment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET WARNING: As you are running in a Jupyter environment, you will need to call `experiment.end()` when finished to ensure all metrics and code are logged before exiting.\n",
      "COMET INFO: Experiment is live on comet.ml https://www.comet.ml/dsblank/confusion-matrix/df5910f0ed5b4a1bbf5929e72bdb1668\n",
      "\n"
     ]
    }
   ],
   "source": [
    "experiment = comet_ml.Experiment(\n",
    "    project_name=\"confusion-matrix\", \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And another model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = create_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, before training, we log the confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before any training:\n",
    "y_predicted = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we make an instance of the Confusion Matrix for later re-use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_matrix = experiment.create_confusion_matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we use the `comet_matrix` method of the `ConfusionMatrix` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_matrix.compute_matrix(y_test, y_predicted, images=x_test, image_shape=(28,28))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `ConfusionMatrix.display()` method to see a rough ASCII version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   A                Confusion Matrix            \n",
      "   c               Predicted Category           \n",
      "   t       0   1   2   3   4   5   6   7   8   9\n",
      "   u   0   0   0   0   0   0   0   0   0   0 980\n",
      "   a   1   0   0   0   0   0   0   0   0   0 113\n",
      "   l   2   0   0   0   0   0   0   0   0   0 103\n",
      "       3   0   0   0   0   0   0   0   0   0 101\n",
      "   C   4   0   0   0   0   0   0   0   0   0 982\n",
      "   a   5   0   0   0   0   0   0   0   0   0 892\n",
      "   t   6   0   0   0   0   0   0   0   0   0 958\n",
      "   e   7   0   0   0   0   0   0   0   0   0 102\n",
      "   g   8   0   0   0   0   0   0   0   0   0 974\n",
      "   o   9   0   0   0   0   0   0   0   0   0 100\n",
      "   r\n"
     ]
    }
   ],
   "source": [
    "confusion_matrix.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time, instead of logging the actual and predicted vectors, we instead pass in the entire ConfusionMatrix as the matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment.log_confusion_matrix(\n",
    "    matrix=confusion_matrix,\n",
    "    step=0,\n",
    "    title=\"Confusion Matrix, Epoch #0\",\n",
    "    file_name=\"confusion-matrix-%03d.json\" % 0,\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we create callbacks, and train the network (this will take just a little more time, as it is generating the assets on the fly):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "500/500 [==============================] - 4s 6ms/step - loss: 1.3040 - accuracy: 0.6069 - val_loss: 0.3161 - val_accuracy: 0.9086\n",
      "Epoch 2/5\n",
      "500/500 [==============================] - 3s 7ms/step - loss: 0.2903 - accuracy: 0.9162 - val_loss: 0.2125 - val_accuracy: 0.9367\n",
      "Epoch 3/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.1964 - accuracy: 0.9419 - val_loss: 0.1614 - val_accuracy: 0.9520\n",
      "Epoch 4/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.1500 - accuracy: 0.9562 - val_loss: 0.1390 - val_accuracy: 0.9590\n",
      "Epoch 5/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.1246 - accuracy: 0.9632 - val_loss: 0.1225 - val_accuracy: 0.9634\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f2098753940>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "callback = ConfusionMatrixCallbackReuseImages(experiment, x_test, y_test, confusion_matrix)\n",
    "\n",
    "model.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    batch_size=120,\n",
    "    epochs=5,\n",
    "    callbacks=[callback],\n",
    "    validation_data=(x_test, y_test),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We end the experiment (here you can see how many assets were uploaded):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET INFO: ---------------------------\n",
      "COMET INFO: Comet.ml Experiment Summary\n",
      "COMET INFO: ---------------------------\n",
      "COMET INFO:   Data:\n",
      "COMET INFO:     display_summary_level : 1\n",
      "COMET INFO:     url                   : https://www.comet.ml/dsblank/confusion-matrix/df5910f0ed5b4a1bbf5929e72bdb1668\n",
      "COMET INFO:   Metrics [count] (min, max):\n",
      "COMET INFO:     accuracy [5]                 : (0.7788000106811523, 0.9639666676521301)\n",
      "COMET INFO:     batch_accuracy [250]         : (0.11666666716337204, 0.9750000238418579)\n",
      "COMET INFO:     batch_loss [250]             : (0.10746046900749207, 2.4073240756988525)\n",
      "COMET INFO:     epoch_duration [5]           : (3.8023373920004815, 31.354876097058877)\n",
      "COMET INFO:     loss [5]                     : (0.12115586549043655, 0.773209810256958)\n",
      "COMET INFO:     val_accuracy [5]             : (0.9085999727249146, 0.9634000062942505)\n",
      "COMET INFO:     val_loss [5]                 : (0.1224675327539444, 0.31605249643325806)\n",
      "COMET INFO:     validate_batch_accuracy [45] : (0.8857723474502563, 1.0)\n",
      "COMET INFO:     validate_batch_loss [45]     : (0.042409297078847885, 0.38421693444252014)\n",
      "COMET INFO:   Others:\n",
      "COMET INFO:     trainable_params : 134794\n",
      "COMET INFO:   Parameters:\n",
      "COMET INFO:     Optimizer             : RMSprop\n",
      "COMET INFO:     RMSprop_centered      : 1\n",
      "COMET INFO:     RMSprop_decay         : 1\n",
      "COMET INFO:     RMSprop_epsilon       : 1e-07\n",
      "COMET INFO:     RMSprop_learning_rate : 0.001\n",
      "COMET INFO:     RMSprop_momentum      : 1\n",
      "COMET INFO:     RMSprop_name          : RMSprop\n",
      "COMET INFO:     RMSprop_rho           : 0.9\n",
      "COMET INFO:     epochs                : 5\n",
      "COMET INFO:     steps                 : 500\n",
      "COMET INFO:   Uploads [count]:\n",
      "COMET INFO:     confusion-matrix [6]     : 6\n",
      "COMET INFO:     environment details      : 1\n",
      "COMET INFO:     filename                 : 1\n",
      "COMET INFO:     git metadata             : 1\n",
      "COMET INFO:     git-patch (uncompressed) : 1 (45 KB)\n",
      "COMET INFO:     images [1180]            : 1180\n",
      "COMET INFO:     installed packages       : 1\n",
      "COMET INFO:     model graph              : 1\n",
      "COMET INFO:     notebook                 : 1\n",
      "COMET INFO:     os packages              : 1\n",
      "COMET INFO:     source_code              : 1\n",
      "COMET INFO: ---------------------------\n",
      "COMET INFO: Uploading 1 metrics, params and output messages\n"
     ]
    }
   ],
   "source": [
    "experiment.end()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, you'll notice that this trained much faster than the previous example, and the number of images was reduced by about 75%. That is becaused we reused the examples in each cell where we can."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the full confusion matrix, complete with sample images in each cell (click on a cell to see the examples):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"800px\"\n",
       "            src=\"https://www.comet.ml/dsblank/confusion-matrix/df5910f0ed5b4a1bbf5929e72bdb1668?experiment-tab=confusionMatrix\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f2098587d30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "experiment.display(tab=\"confusion-matrices\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `ConfusionMatrix` object allows many options, including:\n",
    "\n",
    "* automatically finding the \"most confused\" categories, if more than 25\n",
    "* limit the categories shown (use `ConfusionMatrix(selected=[...])`)\n",
    "* change the row and column labels\n",
    "* change the category labels\n",
    "* change the title\n",
    "* display text, URLs, or images in Example View"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 5: Using Sets of Examples\n",
    "\n",
    "Now, we'll use one image for a set of examples. Before we assumed that there was one image for each index. We change that assumption to use one image for a set.\n",
    "\n",
    "To do this, we'll subclass the ConfusionMatrix and override the method that caches the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "from comet_ml import ConfusionMatrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The heart of the solution is to change how we get the cache key. Since we want to map all of the instances of one class to a single example, when we encounter an new example, we map each category to a single instance.\n",
    "\n",
    "In the constructor, we grab all of the labels for all of the patterns. We then override the `_get_cache_key()` method to change the cache mapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyConfusionMatrix(ConfusionMatrix):\n",
    "    def __init__(self, y_test, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self.my_labels = self.winner_function(y_test)\n",
    "\n",
    "    def _get_cache_key(self, index):\n",
    "        key = self.my_labels[index]\n",
    "        return key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything else is the same:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET WARNING: As you are running in a Jupyter environment, you will need to call `experiment.end()` when finished to ensure all metrics and code are logged before exiting.\n",
      "COMET INFO: Experiment is live on comet.ml https://www.comet.ml/dsblank/confusion-matrix/2add2f772a264dfc895804eaa2119e0f\n",
      "\n"
     ]
    }
   ],
   "source": [
    "experiment = comet_ml.Experiment(\n",
    "    project_name=\"confusion-matrix\", \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And another model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = create_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, before training, we log the confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Before any training:\n",
    "y_predicted = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we make an instance, passing in the inputs and experiment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_matrix = MyConfusionMatrix(y_test, experiment=experiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we use the `comet_matrix` method of the `ConfusionMatrix` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_matrix.compute_matrix(y_test, y_predicted, images=x_test, image_shape=(28, 28))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `MyConfusionMatrix` instance to see a rough ASCII version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   A                Confusion Matrix            \n",
      "   c               Predicted Category           \n",
      "   t       0   1   2   3   4   5   6   7   8   9\n",
      "   u   0   0   0   0   0   0   0   0   0 980   0\n",
      "   a   1   0   0   0   0   0   0   0   0 113   0\n",
      "   l   2   0   0   0   0   0   0   0   0 103   0\n",
      "       3   0   0   0   0   0   0   0   0 101   0\n",
      "   C   4   0   0   0   0   0   0   0   0 982   0\n",
      "   a   5   0   0   0   0   0   0   0   0 892   0\n",
      "   t   6   0   0   0   0   0   0   0   0 958   0\n",
      "   e   7   0   0   0   0   0   0   0   0 102   0\n",
      "   g   8   0   0   0   0   0   0   0   0 974   0\n",
      "   o   9   0   0   0   0   0   0   0   0 100   0\n",
      "   r\n"
     ]
    }
   ],
   "source": [
    "confusion_matrix.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time, instead of logging the actual and predicted vectors, we instead pass in the entire `MyConfusionMatrix` as the matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment.log_confusion_matrix(\n",
    "    matrix=confusion_matrix,\n",
    "    step=0,\n",
    "    title=\"Confusion Matrix, Epoch #0\",\n",
    "    file_name=\"confusion-matrix-%03d.json\" % 0,\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we create callbacks, and train the network (this will take just a little more time, as it is generating the assets on the fly):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "500/500 [==============================] - 4s 6ms/step - loss: 1.3133 - accuracy: 0.6074 - val_loss: 0.2965 - val_accuracy: 0.9157\n",
      "Epoch 2/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.2788 - accuracy: 0.9174 - val_loss: 0.2104 - val_accuracy: 0.9383\n",
      "Epoch 3/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.1971 - accuracy: 0.9403 - val_loss: 0.1759 - val_accuracy: 0.9478\n",
      "Epoch 4/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.1539 - accuracy: 0.9539 - val_loss: 0.1387 - val_accuracy: 0.9594\n",
      "Epoch 5/5\n",
      "500/500 [==============================] - 3s 6ms/step - loss: 0.1281 - accuracy: 0.9619 - val_loss: 0.1298 - val_accuracy: 0.9614\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f210d801370>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "callback = ConfusionMatrixCallbackReuseImages(experiment, x_test, y_test, confusion_matrix)\n",
    "\n",
    "model.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    batch_size=120,\n",
    "    epochs=5,\n",
    "    callbacks=[callback],\n",
    "    validation_data=(x_test, y_test),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We end the experiment (here you can see how many assets were uploaded):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "COMET INFO: ---------------------------\n",
      "COMET INFO: Comet.ml Experiment Summary\n",
      "COMET INFO: ---------------------------\n",
      "COMET INFO:   Data:\n",
      "COMET INFO:     display_summary_level : 1\n",
      "COMET INFO:     url                   : https://www.comet.ml/dsblank/confusion-matrix/2add2f772a264dfc895804eaa2119e0f\n",
      "COMET INFO:   Metrics [count] (min, max):\n",
      "COMET INFO:     accuracy [5]                 : (0.7863666415214539, 0.9636499881744385)\n",
      "COMET INFO:     batch_accuracy [250]         : (0.10000000149011612, 0.9666666388511658)\n",
      "COMET INFO:     batch_loss [250]             : (0.12279674410820007, 2.3612191677093506)\n",
      "COMET INFO:     epoch_duration [5]           : (3.4647893810179085, 4.589158504968509)\n",
      "COMET INFO:     loss [5]                     : (0.12284887582063675, 0.7659122943878174)\n",
      "COMET INFO:     val_accuracy [5]             : (0.9157000184059143, 0.9613999724388123)\n",
      "COMET INFO:     val_loss [5]                 : (0.1298353374004364, 0.296523779630661)\n",
      "COMET INFO:     validate_batch_accuracy [45] : (0.8918699026107788, 1.0)\n",
      "COMET INFO:     validate_batch_loss [45]     : (0.03404945880174637, 0.3674503266811371)\n",
      "COMET INFO:   Others:\n",
      "COMET INFO:     trainable_params : 134794\n",
      "COMET INFO:   Parameters:\n",
      "COMET INFO:     Optimizer             : RMSprop\n",
      "COMET INFO:     RMSprop_centered      : 1\n",
      "COMET INFO:     RMSprop_decay         : 1\n",
      "COMET INFO:     RMSprop_epsilon       : 1e-07\n",
      "COMET INFO:     RMSprop_learning_rate : 0.001\n",
      "COMET INFO:     RMSprop_momentum      : 1\n",
      "COMET INFO:     RMSprop_name          : RMSprop\n",
      "COMET INFO:     RMSprop_rho           : 0.9\n",
      "COMET INFO:     epochs                : 5\n",
      "COMET INFO:     steps                 : 500\n",
      "COMET INFO:   Uploads [count]:\n",
      "COMET INFO:     confusion-matrix [6]     : 6\n",
      "COMET INFO:     environment details      : 1\n",
      "COMET INFO:     filename                 : 1\n",
      "COMET INFO:     git metadata             : 1\n",
      "COMET INFO:     git-patch (uncompressed) : 1 (45 KB)\n",
      "COMET INFO:     images [10]              : 10\n",
      "COMET INFO:     installed packages       : 1\n",
      "COMET INFO:     model graph              : 1\n",
      "COMET INFO:     notebook                 : 1\n",
      "COMET INFO:     os packages              : 1\n",
      "COMET INFO:     source_code              : 1\n",
      "COMET INFO: ---------------------------\n",
      "COMET INFO: Uploading 1 metrics, params and output messages\n"
     ]
    }
   ],
   "source": [
    "experiment.end()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, you'll notice that this trained much faster than the previous example, and the number of images was exactly 10. That is becaused we used one image for each image set, and we reused each of those between epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the full confusion matrix, complete with sample images in each cell (click on a cell to see the examples):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"800px\"\n",
       "            src=\"https://www.comet.ml/dsblank/confusion-matrix/2add2f772a264dfc895804eaa2119e0f?experiment-tab=confusionMatrix\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f210d7b0a60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "experiment.display(tab=\"confusion-matrices\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that each cell has at most 1 example and every cell in each row uses it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We hope that this gives you some ideas of how you can use the Comet Confusion Matrix! If you have questions or comments, feel free to visit the [Comet issue tracker](https://github.com/comet-ml/issue-tracking) and leave us a note."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}