examples/mnist_triton_e2e/mnist-triton.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "c56c0dc8-980b-4b62-962d-e6c60deccf82",
      "metadata": {},
      "source": [
        "# Tensorflow MNIST model and Triton (e2e example)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fad6fe68-dac2-4ded-b7d7-8637bdfaabf5",
      "metadata": {},
      "source": [
        "## Prerequisites\n",
        "\n",
        " * A kubernetes cluster with kubectl configured\n",
        " * rclone\n",
        " * curl\n",
        " * Poetry (optional)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a0047c34-c053-402c-82d8-7a0ec3a484a1",
      "metadata": {},
      "source": [
        "## Setup Seldon Core\n",
        "\n",
        "Use the setup notebook to [Setup Cluster](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html#Setup-Cluster) with [Ambassador Ingress](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html#Ambassador) and [Install Seldon Core](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html#Install-Seldon-Core). Instructions [also online](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html).\n",
        "\n",
        "We will assume that ambassador (or Istio) ingress is port-forwarded to `localhost:8003`\n",
        "\n",
        "## Setup MinIO\n",
        "\n",
        "Use the provided [notebook](https://docs.seldon.io/projects/seldon-core/en/latest/examples/minio_setup.html) to install Minio in your cluster.\n",
        "Instructions [also online](https://docs.seldon.io/projects/seldon-core/en/latest/examples/minio_setup.html).\n",
        "\n",
        "We will assume that MinIO service is port-forwarded to `localhost:8090`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "2836ab83-026c-4f78-bc5f-fccc0df611f0",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Overwriting rclone.conf\n"
          ]
        }
      ],
      "source": [
        "%%writefile rclone.conf\n",
        "[s3]\n",
        "type = s3\n",
        "provider = minio\n",
        "env_auth = false\n",
        "access_key_id = admin@seldon.io\n",
        "secret_access_key = 12341234\n",
        "endpoint = http://localhost:8090"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "b3334cd4-6a50-4e42-897c-ae0fa7a08a0b",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Overwriting secret.yaml\n"
          ]
        }
      ],
      "source": [
        "%%writefile secret.yaml\n",
        "apiVersion: v1\n",
        "kind: Secret\n",
        "metadata:\n",
        "  name: seldon-rclone-secret\n",
        "type: Opaque\n",
        "stringData:\n",
        "  RCLONE_CONFIG_S3_TYPE: s3\n",
        "  RCLONE_CONFIG_S3_PROVIDER: minio\n",
        "  RCLONE_CONFIG_S3_ENV_AUTH: \"false\"\n",
        "  RCLONE_CONFIG_S3_ACCESS_KEY_ID: \"admin@seldon.io\"\n",
        "  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: \"12341234\"\n",
        "  RCLONE_CONFIG_S3_ENDPOINT: http://minio.minio-system.svc.cluster.local:9000"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "4bcf531f-9afb-424e-9c23-201e199f5245",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "secret/seldon-rclone-secret configured\n"
          ]
        }
      ],
      "source": [
        "!kubectl apply -f secret.yaml"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "58e83e3f-c797-4f21-a545-c3be5aad3c9b",
      "metadata": {},
      "source": [
        "## Poetry (optional)\n",
        "\n",
        "We will use poetry.lock to fully define the training environment. \n",
        "Install poetry following official documentation. Usually this goes down to\n",
        "```\n",
        "curl -sSL https://install.python-poetry.org | python3 - --version 1.1.15\n",
        "```\n",
        "\n",
        "Note: as runtime environment will be defined by the Triton Server image this step is optional. However, keeping the training environment fully reproducible is usually a right thing to do."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fb4d5306-6962-4da8-93ab-ad4714081007",
      "metadata": {},
      "source": [
        "## Train MNIST Model"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c431941d-5cbb-4326-a97e-78589f97893f",
      "metadata": {},
      "source": [
        "### Prepare training environment"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "af345f34-e6d3-4974-9c55-f22ab2c3fa98",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting package metadata (current_repodata.json): done\n",
            "Solving environment: done\n",
            "\n",
            "## Package Plan ##\n",
            "\n",
            "  environment location: /home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/venv\n",
            "\n",
            "  added / updated specs:\n",
            "    - conda-ecosystem-user-package-isolation\n",
            "    - python=3.8\n",
            "\n",
            "\n",
            "The following NEW packages will be INSTALLED:\n",
            "\n",
            "  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge\n",
            "  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-1_gnu\n",
            "  ca-certificates    conda-forge/linux-64::ca-certificates-2021.10.8-ha878542_0\n",
            "  conda-ecosystem-u~ conda-forge/linux-64::conda-ecosystem-user-package-isolation-1.0-0\n",
            "  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.36.1-hea4e1c9_2\n",
            "  libffi             conda-forge/linux-64::libffi-3.4.2-h9c3ff4c_4\n",
            "  libgcc-ng          conda-forge/linux-64::libgcc-ng-11.2.0-h1d223b6_11\n",
            "  libgomp            conda-forge/linux-64::libgomp-11.2.0-h1d223b6_11\n",
            "  libnsl             conda-forge/linux-64::libnsl-2.0.0-h7f98852_0\n",
            "  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-11.2.0-he4da1e4_11\n",
            "  libzlib            conda-forge/linux-64::libzlib-1.2.11-h36c2ea0_1013\n",
            "  ncurses            conda-forge/linux-64::ncurses-6.2-h58526e2_4\n",
            "  openssl            conda-forge/linux-64::openssl-3.0.0-h7f98852_2\n",
            "  pip                conda-forge/noarch::pip-21.3.1-pyhd8ed1ab_0\n",
            "  python             conda-forge/linux-64::python-3.8.12-hf930737_2_cpython\n",
            "  python_abi         conda-forge/linux-64::python_abi-3.8-2_cp38\n",
            "  readline           conda-forge/linux-64::readline-8.1-h46c0cb4_0\n",
            "  setuptools         conda-forge/linux-64::setuptools-58.5.3-py38h578d9bd_0\n",
            "  sqlite             conda-forge/linux-64::sqlite-3.36.0-h9cd32fc_2\n",
            "  tk                 conda-forge/linux-64::tk-8.6.11-h27826a3_1\n",
            "  wheel              conda-forge/noarch::wheel-0.37.0-pyhd8ed1ab_1\n",
            "  xz                 conda-forge/linux-64::xz-5.2.5-h516909a_1\n",
            "  zlib               conda-forge/linux-64::zlib-1.2.11-h36c2ea0_1013\n",
            "\n",
            "\n",
            "Preparing transaction: done\n",
            "Verifying transaction: done\n",
            "Executing transaction: done\n",
            "#\n",
            "# To activate this environment, use\n",
            "#\n",
            "#     $ conda activate /home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/venv\n",
            "#\n",
            "# To deactivate an active environment, use\n",
            "#\n",
            "#     $ conda deactivate\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!conda create --yes --prefix ./venv python=3.8"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "ee6b7cf9-15a9-4f0d-acb3-bdcad2560413",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Overwriting pyproject.toml\n"
          ]
        }
      ],
      "source": [
        "%%writefile pyproject.toml\n",
        "[tool.poetry]\n",
        "name = \"mnist_triton_e2e\"\n",
        "version = \"0.1.0\"\n",
        "description = \"\"\n",
        "authors = [\"Seldon Technologies Ltd. <hello@seldon.io>\"]\n",
        "license = \"Business Source License 1.1\"\n",
        "\n",
        "[tool.poetry.dependencies]\n",
        "python = \"~3.8\"\n",
        "tensorflow = \"2.6.2\"\n",
        "\n",
        "[tool.poetry.dev-dependencies]\n",
        "\n",
        "[build-system]\n",
        "requires = [\"poetry-core>=1.0.0\"]\n",
        "build-backend = \"poetry.core.masonry.api\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "31dae9b0-2cf8-4caa-a0b1-be4725da4bf5",
      "metadata": {},
      "source": [
        "On first run of `poetry install` command it will create a `poetry.lock` file. Every next run of the installation command will use the package version from the lock file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "ff28cbca-1311-4091-a875-545783dfc0d5",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Installing dependencies from lock file\n",
            "\n",
            "Package operations: 37 installs, 0 updates, 0 removals\n",
            "\n",
            "  • Installing certifi (2021.10.8)\n",
            "  • Installing charset-normalizer (2.0.7)\n",
            "  • Installing idna (3.3)\n",
            "  • Installing pyasn1 (0.4.8)\n",
            "  • Installing urllib3 (1.26.7)\n",
            "  • Installing cachetools (4.2.4)\n",
            "  • Installing oauthlib (3.1.1)\n",
            "  • Installing pyasn1-modules (0.2.8)\n",
            "  • Installing rsa (4.7.2)\n",
            "  • Installing requests (2.26.0)\n",
            "  • Installing six (1.15.0)\n",
            "  • Installing google-auth (1.35.0)\n",
            "  • Installing requests-oauthlib (1.3.0)\n",
            "  • Installing absl-py (0.15.0)\n",
            "  • Installing google-auth-oauthlib (0.4.6)\n",
            "  • Installing grpcio (1.41.1)\n",
            "  • Installing numpy (1.19.5)\n",
            "  • Installing tensorboard-data-server (0.6.1)\n",
            "  • Installing tensorboard-plugin-wit (1.8.0)\n",
            "  • Installing protobuf (3.19.1)\n",
            "  • Installing werkzeug (2.0.2)\n",
            "  • Installing markdown (3.3.4)\n",
            "  • Installing astunparse (1.6.3)\n",
            "  • Installing clang (5.0)\n",
            "  • Installing gast (0.4.0)\n",
            "  • Installing google-pasta (0.2.0)\n",
            "  • Installing keras (2.6.0)\n",
            "  • Installing keras-preprocessing (1.1.2)\n",
            "  • Installing opt-einsum (3.3.0)\n",
            "  • Installing tensorboard (2.6.0)\n",
            "  • Installing termcolor (1.1.0)\n",
            "  • Installing h5py (3.1.0)\n",
            "  • Installing flatbuffers (1.12)\n",
            "  • Installing typing-extensions (3.7.4.3)\n",
            "  • Installing tensorflow-estimator (2.6.0)\n",
            "  • Installing wrapt (1.12.1)\n",
            "  • Installing tensorflow (2.6.2)\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "source ~/miniconda3/etc/profile.d/conda.sh\n",
        "conda activate ./venv\n",
        "poetry install"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "feb34919-1d86-4249-b352-32b64c04205f",
      "metadata": {},
      "source": [
        "### Prepare Training Script"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "0545621c-d9f5-4b8f-8f8f-16e16377c4e1",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Overwriting train.py\n"
          ]
        }
      ],
      "source": [
        "%%writefile train.py\n",
        "\n",
        "import numpy as np\n",
        "np.random.seed(123)   # for reproducibility\n",
        "\n",
        "from tensorflow.keras.layers import (Activation, Convolution2D, Dense, Dropout,\n",
        "                                     Flatten, MaxPooling2D)\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "\n",
        "from tensorflow.keras.datasets import mnist\n",
        "\n",
        "\n",
        "# Load pre-shuffled MNIST data into train and test sets\n",
        "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
        "\n",
        "# Reshape data for Tensorflow\n",
        "X_train = X_train.reshape(-1, 28, 28, 1)\n",
        "X_test = X_test.reshape(-1, 28, 28, 1)\n",
        "\n",
        "X_train = X_train.astype(\"float32\")\n",
        "X_test = X_test.astype(\"float32\")\n",
        "X_train /= 255\n",
        "X_test /= 255\n",
        "\n",
        "# Convert 1-dimensional class arrays to 10-dimensional class matrices\n",
        "Y_train = to_categorical(y_train, 10)\n",
        "Y_test = to_categorical(y_test, 10)\n",
        "\n",
        "# define model\n",
        "model = Sequential()\n",
        "\n",
        "# declare input layer\n",
        "model.add(Convolution2D(32, (3, 3), activation=\"relu\", input_shape=(28, 28, 1)))\n",
        "\n",
        "# add more layers\n",
        "model.add(Convolution2D(32, (3, 3), activation=\"relu\"))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "model.add(Dropout(0.25))\n",
        "\n",
        "# and even more layers\n",
        "model.add(Flatten())\n",
        "model.add(Dense(128, activation=\"relu\"))\n",
        "model.add(Dropout(0.5))\n",
        "model.add(Dense(10, activation=\"softmax\"))\n",
        "\n",
        "model.summary()\n",
        "\n",
        "# compile model and train\n",
        "model.compile(loss=\"categorical_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n",
        "model.fit(X_train, Y_train, epochs=4, batch_size=32, verbose=1)\n",
        "\n",
        "score = model.evaluate(X_test, Y_test, verbose=0)\n",
        "print(\"\\n%s: %.2f%%\" % (model.metrics_names[1], score[1] * 100))\n",
        "\n",
        "# save model\n",
        "model.save(\"models-repository/mnist/1/model.savedmodel\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "adaaef1b-2fe2-4646-aab1-61c3961d39e9",
      "metadata": {},
      "source": [
        "### Train Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "c6be3232-40f2-403f-89bc-dea468065f5c",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2021-11-09 00:16:11.767552: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-11-09 00:16:11.774042: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-11-09 00:16:11.774341: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-11-09 00:16:11.774801: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
            "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
            "2021-11-09 00:16:11.775503: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-11-09 00:16:11.775756: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-11-09 00:16:11.775996: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-11-09 00:16:12.252515: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-11-09 00:16:12.252816: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-11-09 00:16:12.253061: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
            "2021-11-09 00:16:12.253297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1356 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1650, pci bus id: 0000:01:00.0, compute capability: 7.5\n",
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "conv2d (Conv2D)              (None, 26, 26, 32)        320       \n",
            "_________________________________________________________________\n",
            "conv2d_1 (Conv2D)            (None, 24, 24, 32)        9248      \n",
            "_________________________________________________________________\n",
            "max_pooling2d (MaxPooling2D) (None, 12, 12, 32)        0         \n",
            "_________________________________________________________________\n",
            "dropout (Dropout)            (None, 12, 12, 32)        0         \n",
            "_________________________________________________________________\n",
            "flatten (Flatten)            (None, 4608)              0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 128)               589952    \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, 128)               0         \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 10)                1290      \n",
            "=================================================================\n",
            "Total params: 600,810\n",
            "Trainable params: 600,810\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "2021-11-09 00:16:12.553857: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n",
            "Epoch 1/4\n",
            "2021-11-09 00:16:13.375537: I tensorflow/stream_executor/cuda/cuda_dnn.cc:369] Loaded cuDNN version 8101\n",
            "1875/1875 [==============================] - 8s 3ms/step - loss: 0.2046 - accuracy: 0.9378\n",
            "Epoch 2/4\n",
            "1875/1875 [==============================] - 5s 3ms/step - loss: 0.0872 - accuracy: 0.9742\n",
            "Epoch 3/4\n",
            "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0676 - accuracy: 0.9791\n",
            "Epoch 4/4\n",
            "1875/1875 [==============================] - 6s 3ms/step - loss: 0.0546 - accuracy: 0.9833\n",
            "\n",
            "accuracy: 98.98%\n",
            "2021-11-09 00:16:38.051127: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.\n"
          ]
        }
      ],
      "source": [
        "!./venv/bin/python3 train.py"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "aa412155-b837-4ad0-8d1d-6f0b43b10cdc",
      "metadata": {},
      "source": [
        "## Deploy and Test"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "642c0861-bc3e-4ed6-9dbb-043972b362d2",
      "metadata": {},
      "source": [
        "### Configure Triton Server"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "9352ad51-acc2-4124-8b91-e1ad82abb4bd",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Overwriting models-repository/mnist/config.pbtxt\n"
          ]
        }
      ],
      "source": [
        "%%writefile models-repository/mnist/config.pbtxt\n",
        "name: \"mnist\"\n",
        "platform: \"tensorflow_savedmodel\"\n",
        "max_batch_size: 100\n",
        "dynamic_batching { preferred_batch_size: [ 50 ]}\n",
        "instance_group [ { count: 2 }]\n",
        "input [\n",
        "  {\n",
        "    name: \"conv2d_input\"\n",
        "    data_type: TYPE_FP32\n",
        "    dims: [ 28, 28, 1 ]\n",
        "  }\n",
        "]\n",
        "output [\n",
        "  {\n",
        "    name: \"dense_1\"\n",
        "    data_type: TYPE_FP32\n",
        "    dims: [ 10 ]\n",
        "  }\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "41d41886-5d40-4921-bc30-1830c62765e8",
      "metadata": {},
      "source": [
        "### Copy model into MinIO instance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "c55a35c8-cfa6-4003-84e8-2e5ac31a083d",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[01;34mmodels-repository/\u001b[00m\n",
            "└── \u001b[01;34mmnist\u001b[00m\n",
            "    ├── \u001b[01;34m1\u001b[00m\n",
            "    │   └── \u001b[01;34mmodel.savedmodel\u001b[00m\n",
            "    │       ├── \u001b[01;34massets\u001b[00m\n",
            "    │       ├── keras_metadata.pb\n",
            "    │       ├── saved_model.pb\n",
            "    │       └── \u001b[01;34mvariables\u001b[00m\n",
            "    │           ├── variables.data-00000-of-00001\n",
            "    │           └── variables.index\n",
            "    └── config.pbtxt\n",
            "\n",
            "5 directories, 5 files\n"
          ]
        }
      ],
      "source": [
        "!tree models-repository/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "8bf79c13-4f9f-4d19-be4e-ec171ce3de43",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2021/11/09 00:16:39 DEBUG : rclone: Version \"v1.56.2\" starting with parameters [\"/home/rskolasinski/.asdf/installs/rclone/1.56.2/bin/rclone\" \"-vvv\" \"--config=rclone.conf\" \"copy\" \"models-repository\" \"s3://triton-models/mnist-model\"]\n",
            "2021/11/09 00:16:39 DEBUG : Creating backend with remote \"models-repository\"\n",
            "2021/11/09 00:16:39 DEBUG : Using config file from \"/home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/rclone.conf\"\n",
            "2021/11/09 00:16:39 DEBUG : fs cache: renaming cache item \"models-repository\" to be canonical \"/home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/models-repository\"\n",
            "2021/11/09 00:16:39 DEBUG : Creating backend with remote \"s3://triton-models/mnist-model\"\n",
            "2021/11/09 00:16:39 DEBUG : fs cache: renaming cache item \"s3://triton-models/mnist-model\" to be canonical \"s3:triton-models/mnist-model\"\n",
            "2021/11/09 00:16:39 DEBUG : mnist/config.pbtxt: Modification times differ by -8m54.901977129s: 2021-11-09 00:16:39.278686721 +0000 GMT, 2021-11-09 00:07:44.376709592 +0000 GMT\n",
            "2021/11/09 00:16:39 DEBUG : mnist/config.pbtxt: md5 = 8bf5c24a18dff3603cf2cdcbd5d5e929 OK\n",
            "2021/11/09 00:16:39 DEBUG : S3 bucket triton-models path mnist-model: Waiting for checks to finish\n",
            "2021/11/09 00:16:39 INFO  : mnist/config.pbtxt: Updated modification time in destination\n",
            "2021/11/09 00:16:39 DEBUG : mnist/config.pbtxt: Unchanged skipping\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/keras_metadata.pb: Modification times differ by -54m41.948759938s: 2021-11-09 00:16:38.51069757 +0000 GMT, 2021-11-08 23:21:56.561937632 +0000 GMT\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/saved_model.pb: Modification times differ by -54m41.948760136s: 2021-11-09 00:16:38.49869774 +0000 GMT, 2021-11-08 23:21:56.549937604 +0000 GMT\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/keras_metadata.pb: md5 = a4c5251702f32808e94e0d2b89a85aec OK\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.index: Modification times differ by -54m41.956760288s: 2021-11-09 00:16:38.490697853 +0000 GMT, 2021-11-08 23:21:56.533937565 +0000 GMT\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.index: md5 = 6e5f3d005dbe11b90507f2865a1a0713 (Local file system at /home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/models-repository)\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.index: md5 = be8df29950bcde41116269ed7b7ac6a2 (S3 bucket triton-models path mnist-model)\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.index: md5 differ\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/saved_model.pb: md5 = 936afcc716e6828695b1a11ad5a0d691 (Local file system at /home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/models-repository)\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/saved_model.pb: md5 = dcc8acd1b44e285751f93d4f3499decd (S3 bucket triton-models path mnist-model)\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/saved_model.pb: md5 differ\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.data-00000-of-00001: Modification times differ by -54m41.960760297s: 2021-11-09 00:16:38.490697853 +0000 GMT, 2021-11-08 23:21:56.529937556 +0000 GMT\n",
            "2021/11/09 00:16:39 INFO  : mnist/1/model.savedmodel/keras_metadata.pb: Updated modification time in destination\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/keras_metadata.pb: Unchanged skipping\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/saved_model.pb: md5 = 936afcc716e6828695b1a11ad5a0d691 OK\n",
            "2021/11/09 00:16:39 INFO  : mnist/1/model.savedmodel/saved_model.pb: Copied (replaced existing)\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.index: md5 = 6e5f3d005dbe11b90507f2865a1a0713 OK\n",
            "2021/11/09 00:16:39 INFO  : mnist/1/model.savedmodel/variables/variables.index: Copied (replaced existing)\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.data-00000-of-00001: md5 = 254dd0a480879c42a9ec648b6f1fabf4 (Local file system at /home/rskolasinski/work/seldon-core-mnist-triton/examples/mnist_triton_e2e/models-repository)\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.data-00000-of-00001: md5 = 176268afe35d0445761ce3c0698f4d16 (S3 bucket triton-models path mnist-model)\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.data-00000-of-00001: md5 differ\n",
            "2021/11/09 00:16:39 DEBUG : S3 bucket triton-models path mnist-model: Waiting for transfers to finish\n",
            "2021/11/09 00:16:39 DEBUG : mnist/1/model.savedmodel/variables/variables.data-00000-of-00001: md5 = 254dd0a480879c42a9ec648b6f1fabf4 OK\n",
            "2021/11/09 00:16:39 INFO  : mnist/1/model.savedmodel/variables/variables.data-00000-of-00001: Copied (replaced existing)\n",
            "2021/11/09 00:16:39 INFO  : \n",
            "Transferred:   \t    7.026Mi / 7.026 MiByte, 100%, 0 Byte/s, ETA -\n",
            "Checks:                 5 / 5, 100%\n",
            "Transferred:            3 / 3, 100%\n",
            "Elapsed time:         0.0s\n",
            "\n",
            "2021/11/09 00:16:39 DEBUG : 15 go routines active\n"
          ]
        }
      ],
      "source": [
        "!rclone -vvv --config=\"rclone.conf\" copy models-repository s3://triton-models/mnist-model"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "91035542-7ff7-409c-be2a-42a5724958d9",
      "metadata": {},
      "source": [
        "### Deploy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "bddca43b-4bc6-45af-a868-8a29d24d2474",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Overwriting deployment.yaml\n"
          ]
        }
      ],
      "source": [
        "%%writefile deployment.yaml\n",
        "apiVersion: machinelearning.seldon.io/v1\n",
        "kind: SeldonDeployment\n",
        "metadata:\n",
        "  name: mnist\n",
        "  namespace: seldon\n",
        "spec:\n",
        "  name: default\n",
        "  predictors:\n",
        "  - graph:\n",
        "      implementation: TRITON_SERVER\n",
        "      logger:\n",
        "        mode: all\n",
        "      modelUri: s3:triton-models/mnist-model\n",
        "      envSecretRefName: seldon-rclone-secret\n",
        "      name: mnist\n",
        "      type: MODEL\n",
        "    name: default\n",
        "    replicas: 1\n",
        "  protocol: kfserving"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "7e5d19ab-46f1-4e97-8f52-cc08c099a5d1",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "seldondeployment.machinelearning.seldon.io/mnist unchanged\n"
          ]
        }
      ],
      "source": [
        "!kubectl apply -f deployment.yaml"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "5d5db663-d365-4ebc-9415-9952fd1eb174",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "deployment \"mnist-default-0-mnist\" successfully rolled out\n"
          ]
        }
      ],
      "source": [
        "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=mnist -o jsonpath='{.items[0].metadata.name}')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "15d055dd-45ea-449f-b0dc-221d5fe74c30",
      "metadata": {},
      "source": [
        "### Test (dummy data)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "3599b756-8f48-4994-8199-6cbe787e2911",
      "metadata": {},
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import requests"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "id": "a34e71ed-c055-4c20-a14b-bf95b1f7953d",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[1;39m{\n",
            "  \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"mnist\"\u001b[0m\u001b[1;39m,\n",
            "  \u001b[0m\u001b[34;1m\"versions\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m[\n",
            "    \u001b[0;32m\"1\"\u001b[0m\u001b[1;39m\n",
            "  \u001b[1;39m]\u001b[0m\u001b[1;39m,\n",
            "  \u001b[0m\u001b[34;1m\"platform\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"tensorflow_savedmodel\"\u001b[0m\u001b[1;39m,\n",
            "  \u001b[0m\u001b[34;1m\"inputs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m[\n",
            "    \u001b[1;39m{\n",
            "      \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"conv2d_input\"\u001b[0m\u001b[1;39m,\n",
            "      \u001b[0m\u001b[34;1m\"datatype\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"FP32\"\u001b[0m\u001b[1;39m,\n",
            "      \u001b[0m\u001b[34;1m\"shape\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m[\n",
            "        \u001b[0;39m-1\u001b[0m\u001b[1;39m,\n",
            "        \u001b[0;39m28\u001b[0m\u001b[1;39m,\n",
            "        \u001b[0;39m28\u001b[0m\u001b[1;39m,\n",
            "        \u001b[0;39m1\u001b[0m\u001b[1;39m\n",
            "      \u001b[1;39m]\u001b[0m\u001b[1;39m\n",
            "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
            "  \u001b[1;39m]\u001b[0m\u001b[1;39m,\n",
            "  \u001b[0m\u001b[34;1m\"outputs\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m[\n",
            "    \u001b[1;39m{\n",
            "      \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"dense_1\"\u001b[0m\u001b[1;39m,\n",
            "      \u001b[0m\u001b[34;1m\"datatype\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"FP32\"\u001b[0m\u001b[1;39m,\n",
            "      \u001b[0m\u001b[34;1m\"shape\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m[\n",
            "        \u001b[0;39m-1\u001b[0m\u001b[1;39m,\n",
            "        \u001b[0;39m10\u001b[0m\u001b[1;39m\n",
            "      \u001b[1;39m]\u001b[0m\u001b[1;39m\n",
            "    \u001b[1;39m}\u001b[0m\u001b[1;39m\n",
            "  \u001b[1;39m]\u001b[0m\u001b[1;39m\n",
            "\u001b[1;39m}\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "!curl -s http://localhost:8003/seldon/seldon/mnist/v2/models/mnist | jq ."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "id": "38ccab84-7dcf-490d-a783-cd60b473322a",
      "metadata": {},
      "outputs": [],
      "source": [
        "URL = \"http://localhost:8003/seldon/seldon/mnist\"\n",
        "\n",
        "\n",
        "def predict(data):\n",
        "    data = {\n",
        "        \"inputs\": [\n",
        "            {\n",
        "                \"name\": \"conv2d_input\",\n",
        "                \"data\": data.tolist(),\n",
        "                \"datatype\": \"FP32\",\n",
        "                \"shape\": data.shape,\n",
        "            }\n",
        "        ]\n",
        "    }\n",
        "\n",
        "    r = requests.post(f\"{URL}/v2/models/mnist/infer\", json=data)\n",
        "    predictions = np.array(r.json()[\"outputs\"][0][\"data\"]).reshape(\n",
        "        r.json()[\"outputs\"][0][\"shape\"]\n",
        "    )\n",
        "    output = [np.argmax(x) for x in predictions]\n",
        "    return output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "id": "211d032d-5f0b-40ed-a1d9-0e0cdc57cdcb",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[8, 8, 8, 8, 2, 8, 8, 8, 8, 3]"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "output = predict(np.random.rand(10, 28, 28, 1))\n",
        "output"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "521ac271-492a-4950-93fe-bbfb7ba60cea",
      "metadata": {},
      "source": [
        "### Test (real data - optional)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "id": "995e4698-2293-49ca-9c64-50af1680c306",
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "2021-11-09 00:16:41.193198: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0\n"
          ]
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "from tensorflow.keras.datasets import mnist\n",
        "from tensorflow.keras.utils import to_categorical"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "id": "1a589239-24a0-49ed-b69c-bf51c9a49ac0",
      "metadata": {},
      "outputs": [],
      "source": [
        "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
        "\n",
        "X_test = X_test.reshape(-1, 28, 28, 1)\n",
        "X_test = X_test.astype(\"float32\")\n",
        "X_test /= 255\n",
        "\n",
        "Y_test = to_categorical(y_test, 10)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "id": "c0352c7c-8ea6-4e6b-ba0c-37816cbb5110",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2cAAAC/CAYAAAB6zqS6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAliUlEQVR4nO3de7xWdZn38e8l4qmtJsEgOhwacFQyg2mTmSWOlg/iMdTUcYwaJ5WGKUcfzcjUNHvMAzWmj4qpWGiIiAoOpmUnHI/bI4qHRKFRUU6mhgcEr+ePtXja8lube+37uH5rf96v137BvvY6XOveX27W7173+t3m7gIAAAAAtNZGrW4AAAAAAMDgDAAAAAAKgcEZAAAAABQAgzMAAAAAKAAGZwAAAABQAAzOAAAAAKAAGJzVgZn9zsz+tQHb3cvMXtzAz6ea2fe7+NlXzOzueveEciCziA2ZRYzMbJGZfb4B291g9jb078XMzjKzafXuCeVAZluv1IMzM/vLel9rzewnre4rNmZ2+XqP47tm9mar+yobM9vUzK4ys8Vm9qaZPWpm+7W6rxiZ2Xgze8jM3jCzF83sfDPbuNV9lZGZTTSzjvR5YWqr+4mVme1iZneY2XIz4wNIG8zM+pjZzWa2Kn3O/adW9xQjS/zQzFakXz80M2t1X2VmZjuY2Ts9abDSCGa2iZk9taEX51ql1IMzd29b9yVpW0lvS7qxxW1Fx91PWO+x/IV4HBthY0n/I2m0pK0lnS5phpkNaWVTkdpC0omS+kraTdI+kv53KxsqsZclfV/S1a1uJHLvSZoh6dhWN9JDXCpptaT+ko6WdJmZfay1LUXpOEmHSPqEpF0lHSjp+FY21ANcKunBVjdRAqdIWtbqJrKUenC2nkMlLZU0r5qVzWwzM5uWvjL0ZzN70Mz6d1pksJn9d3rF404z69tp3YPM7Ml0vd+Z2c6dfuZmNqzT9xt6C81IM3s43ccNkjar3LZdYmavm9nTZrZPpx9snV6lWWJmL5nZ982sV47H4UNKHstrKy2L7nH3Ve5+lrsvcvf33f02SS9I+mQ12+vJmXX3y9x9nruvdveXJF0naY8K+0YV3H2Wu98iaUWt2+rhmX3G3a+S9GSF/aFGnf4f+667/8Xd75Y0W9IxVW6vr5ndlmZvpZnNM7PO51cjzOzxNCM3mNlmndb9mpk9l64328y2S+tD0txu3GnZDb3t6wtp/l43s0skVbp6tVnay5tp3j/RaVvbmdlNZrbMzF4ws29sYDvjJV3k7i+mz7UXSfpKhX2jSmZ2pKQ/S7qrxu305MzKzD4q6Z8l/Z8K+2yJnjQ4Gy/pZ+5e7dtFxiu5mjFQ0kcknaDkStw6/yTpq5L+RtImSl+lN7O/V3Kl6URJ/STNlTTHzDbpzs7T5W+R9HNJfZRcuTq0wmq7SVqo5OrBmZJmmVmf9GdTJa2RNEzSSEn7SspzP8ehSl5p+EN3+kf3pSelf6/qT9bI7F/tKU56Y0Bm0Qx/L2mNuz/bqfaYpGqvnJ0s6UUl2esvaZKkzucaX5I0RtJHlVxd+ookmdneSk4OvyRpgKTFkqZ3d+fpixSzlLzboq+SPFZ6MepgJfnuI+l6SbeYWe/0BH2OksdjeyXvOjjRzP5XF9v5WLrsOrU8jtgAM9tK0tmSTqrD5npyZiXpJ0qO+e0NLNMyPWJwZmaDlbxVrJarPe8pOVkY5u5r3f0hd3+j08+vcfdn3f1tJW9NGZHWj5D0X+7+K3d/T9KFkjaX9Jlu7v/TknpL+rG7v+fuM1X5svbSTsvfIOkZSfunJ/1jJZ2YXq1ZKulHko7M0Uetg1zkYGa9lVztudbdn65yM2RWkpn9i6T29DhQbGQWzdAm6Y31aq9L2rLK7b2n5ER1cJqDeev9H3mxu7/s7iuVnESOSOtHS7ra3R9293clfVvS7tb9t7KPlfSku89M8/9jSa9UWOehTstPVnKF+NOSRknq5+5np+88eF7Sleo6t21KHrt1XpfUZsZ9Zw1wjqSr3L0e90j12Mya2Rcl9XL3m7vZc9P0iMGZkrcq3O3uL3S1gCVvh1k34cXnMhb5uaQ7JE03s5ctmWCgd6efdw7VW0qesCRpOyWvLEiS3P19JfcVbd/NY9hO0kvr/eNZ3NXCqazlt5M0WMkJyJL0kvafJV2h5NXoLpnZIEl7SfpZ91pHd6SvAv1cyf0QEzewHJmtnNlDlLzKt5+7L+9e+6g3Mls5s2iKv0jaar3aVpIyJ7qyD06INShjkQskPSfpTjN73sxOW+/neXP7FyVvD64mt//TaTve+fsudF7+fSVXUdbldrt1mU1zO0nJ1ZUs6z+WW0n6Cy/g1peZjZD0eSUv8ORZnsxmZNaStzSfL2mDb3tstZ4ye9mXJZ23oQXcfYOX4dOR+vckfS99hWCukldIr6qw75clfXzdN+mrSQMlvZSW3lIyecE62yoJ3PqWSNrezKzTk94gJZeCu5K1/GwlAX9XUl93X1Oh/86OkfTf6asSaIA0H1cpeVIZm+YuE5ndMDMbo+TVs/3dfX6eddBYZBYF8aykjc1sB3f/Y1r7hLp467MnE2F1yd3fVPI2sZPNbBdJvzGzB9290n1BLys5sZT0/08cP6Ikt6vS8hb661W+bbvYzhIleV+3Hev8fRc6L7+RpL9N+1kj6QV336HC+us8qeSxeyD9vsvHETXZS9IQSX9KL0q2SeplZsPd/R/WX5jMdmkHJY/jvPRx3ETS1mb2iqRPu/uiHNtouNJfOTOzzygZ0dc0u6CZ/aOZfdySm7nfUHJJ+P0cq85Q8haXfdJXgE9W8h/2PenPH5X0T2bWKz2ZHN3Fdu5VEsBvpO+xHSfpUxX2/Tedlj9c0s6S5rr7Ekl3SrrIzLYys43MbKiZdbXvdb6s5B4KNM5lSn5PB6Zv3apaT86sJe+Lv07Soe7+QNYyqA8z29iSm8V7KTlZ2Myq/NiCHp5ZSx/HTdLvNzOzTXMcO7rJ3Vcpud/lbDP7kJntoeR+lp9Xsz0zO8DMhqUnmK9LWqt8uf2FpK+a2Yj0d/0DSfd7MinUMiUnvP+c5vZfJA3tYjv/JeljZjYu/bf3DXV9UrzOJzstf6KSfy/3KRlkvWlm3zKzzdN972Jmo7rYzs8knWRm21syMcTJ4jyhEaYo+f2PSL8uV/J739B9VV3qwZl9Qskgb0T69a+SXk3/XunKXdOUfnCm5B6pWemrBLXYVtJMJScMT0n6vXI8kbv7M0pmhPmJpOVKppk90N1Xp4t8M639Wcl7eW/pYjurJY1TclPmSiX3WMyqsPv7lbxKsFzSuZIOc/d1M6p9WclJwAJJr6XHNqCrDZnZ7kpepWAK/Qax5N7I45U8Sbxif31LwtFVbrInZ/a7SiaWmNvpcby9wr5RndOV3FR9mpLcvJ3WqtGTMztYyWO37qrD20quGqIxvq7kvsSlSk44J7h7tVd8dpD0ayVv8btX0v91999WWsndf63kueomJVcShuqD98l8Tcl03yuUTLJxz/rbSLezXNLhSt4htCLt578r7P5WJfl+Tcm7Ysal9x6tlXSAkv+HXlCS658qeT7NcoWSe5LmKznx/a+0hjpy97fc/ZV1X0qy9k46IKpGj8ysu69Z73FcKen99Pu1lY6/WYy3BQMAAABA6/WEK2cAAAAAUHgMzgAAAACgABicAQAAAEABMDgDAAAAgAKo6XPO0imJ/1PJFMo/dfcNfpZY3759fciQIbXsEj3YokWLtHz5cqtlG2QWzVSPzErdyy2ZRa0eeuih5e7er5ZtkFk0U7MzK5Fb1GZD5wdVD87Sz6G5VNIXlHyY54NmNtvdF3S1zpAhQ9TR0VHtLtHDtbe317Q+mUWz1ZpZqfu5JbOolZktrnF9MoumanZmJXKL2mzo/KCWtzV+StJz7v58+tkw05V8iCNQVGQWMSK3iA2ZRWzILAqjlsHZ9vrgp2m/mNY+wMyOM7MOM+tYtqzaz8oD6oLMIkYVc0tmUTBkFrHh/ACF0fAJQdx9iru3u3t7v341vR0YaAoyi9iQWcSGzCJG5BbNUMvg7CVJAzt9/7dpDSgqMosYkVvEhswiNmQWhVHL4OxBSTuY2UfNbBNJR0qaXZ+2gIYgs4gRuUVsyCxiQ2ZRGFXP1ujua8xsoqQ7lEw7erW7P1m3zoA6I7OIEblFbMgsYkNmUSQ1fc6Zu8+VNLdOvQANR2YRI3KL2JBZxIbMoigaPiEIAAAAAKAyBmcAAAAAUAAMzgAAAACgABicAQAAAEABMDgDAAAAgAJgcAYAAAAABcDgDAAAAAAKoKbPOQMQlwsvvDCovf3220Ht8ccfD2ozZ87MvZ8JEyYEtd133z2oHXPMMbm3CQAAUHZcOQMAAACAAmBwBgAAAAAFwOAMAAAAAAqAwRkAAAAAFAATggAldcQRRwS1G2+8sertmVnuZS+//PKg9utf/zqojR49OqgNGjSoe40BDfLss88GtR133DGoXXzxxUHt3//93xvSE8pj1apVQe2UU04JalnPp5LU3t4e1LKe4wcPHlxFdwBahStnAAAAAFAADM4AAAAAoAAYnAEAAABAATA4AwAAAIACqGlCEDNbJOlNSWslrXH38O5UoGDKltusiT+k2ib/2GmnnYLamDFjgtrzzz+fuf7s2bOD2nPPPRfUpk2bFtQmTZqUp8UepWyZjcUjjzwS1DbaKHxNc/vtt29GO1Ehs5W9/PLLQe3KK68Mar169cpcv6OjI6jNmTMnqE2cOLGK7nqenp7Zhx9+OKiNGzcuqC1atKgJ3eR35513ZtZ33nnnoDZw4MBGt1MX9Zit8R/dfXkdtgM0E7lFbMgsYkNmERsyi5bjbY0AAAAAUAC1Ds5c0p1m9pCZHZe1gJkdZ2YdZtaxbNmyGncH1MUGc0tmUUBkFrEhs4gN57QohFoHZ59193+QtJ+kfzOzPddfwN2nuHu7u7f369evxt0BdbHB3JJZFBCZRWzILGLDOS0KoaZ7ztz9pfTPpWZ2s6RPSfpDPRoDGiXm3GbdAH7zzTfnXn+XXXYJalmTd/Tt2zeotbW1BbXVq1dn7me33XYLao899lhQW7FiReb6+KCYMxuzRx99NKhl/TvIumm+pyOzH5R1lWX8+PEt6ARd6emZveOOO4Lau+++24JOuifrHEaSrr766qA2ffr0RrdTF1VfOTOzD5nZluv+LmlfSU/UqzGgEcgtYkNmERsyi9iQWRRJLVfO+ku62czWbed6d/9lXboCGofcIjZkFrEhs4gNmUVhVD04c/fnJX2ijr0ADUduERsyi9iQWcSGzKJImEofAAAAAAqgHh9CHYWZM2cGtSuvvDKobbfddpnrb7bZZkHt6KOPDmrbbrttUBs2bFieFoGKlixZEtTcPXPZrMk/sm74HTBgQNX9XHjhhZn1p556Ktf6BxxwQNX7Bupp/vz5Qe0nP/lJUPvyl7/cjHYQsYsvvjio3XLLLUHtwQcfrPu+582bF9Sy/o/4xCfCi0R77hlMToiSWrNmTVCbO3duCzqpXXt7e2Z98uTJQW3VqlVB7UMf+lDde6oVV84AAAAAoAAYnAEAAABAATA4AwAAAIACYHAGAAAAAAXQYyYEOeWUU4LaokWLatrm5ZdfHtS22mqroDZ8+PCa9tMsAwcODGqnnnpqUOvq5ks03oEHHhjUnnvuucxlt9xyy6DWp0+fuvZzww03ZNZXr15d1/0AjfbMM88Etaybx4844ohmtIOInXjiiUGtV69eTdn3rFmzctUGDRoU1GbMmJG5zU9+8pO1N4ZC+e1vfxvU7rnnnqD2rW99qxnt1GTlypWZ9SeffDKovfXWW0GNCUEAAAAAAJkYnAEAAABAATA4AwAAAIACYHAGAAAAAAXA4AwAAAAACqDHzNb405/+NKg99thjQa2rmRUXLFgQ1B555JGg9rvf/S6o3XfffUEta6akP/3pT5n7zqt3795BrW/fvkFtyZIlmetn9Zk1gyOzNRbL4MGDm7KfCy64IKg9++yzudffbbfdctWAVjj//POD2pAhQ4Iaz3/obOzYsUHN3YPa2rVr677vrP/fs2aeW7x4cVB74YUXgtqoUaMy9/P+++9X0R2KYP78+Zn1I488MqgNGzYsqE2aNKnuPdXb7NmzW91C3XHlDAAAAAAKgMEZAAAAABQAgzMAAAAAKICKgzMzu9rMlprZE51qfczsV2b2x/TPbRrbJtA95BaxIbOIDZlFbMgsYpBnQpCpki6R9LNOtdMk3eXu55nZaen336p/e/Wzzz775Kp1ZcyYMbmWe+2114Ja1sQhWTeVP/jgg7n7ybLpppsGtR133DGo7bTTTpnrr1y5MqgNHTq0pp5aaKpKkNtWue2224LaGWecEdTefffdzPX79+8f1M4777ygtsUWW1TRXWlNFZltuEWLFmXWs55/s54/syZc6MGmqodk9ve//31m/emnnw5qZhbUevXqVfW+TzjhhMz6vvvuG9S23nrroPab3/wmqJ177rm593/ZZZcFtQkTJuRev2CmqodkVur69/zWW28FtWnTpgW1tra2uvdUi6zz1K7+bWb9O4xFxStn7v4HSes/GgdLujb9+7WSDqlvW0BtyC1iQ2YRGzKL2JBZxKDae876u/u6+dhfkRS+TA4UD7lFbMgsYkNmERsyi0KpeUIQTz7QI/xQj5SZHWdmHWbWsWzZslp3B9TFhnJLZlFEZBaxIbOIDee0KIJqB2evmtkASUr/XNrVgu4+xd3b3b29X79+Ve4OqItcuSWzKBAyi9iQWcSGc1oUSp4JQbLMljRe0nnpn7fWraPIbbNNOMnP3nvvnWvd7kxQktdNN90U1LImLZGkXXfdNahlfYp8xMhtTh0dHUGtq8k/shxxxBFBbfTo0TX11EOR2Trr6ubxLJx8VSX6zGZNGtPV/4XLly+vej+DBg0KaocddlhQO/PMMzPXzzuh0uDBg4PaFVdcEdS6OpZTTz01qL3zzjtBbeLEiUGtd+/eeVpstegzK0kzZ84ManPnzs1cdtiwYUFt1KhRde+p3r7//e8Hta4m/thrr72C2oc//OE6d9QYeabS/4WkeyXtaGYvmtmxSgL8BTP7o6TPp98DhUFuERsyi9iQWcSGzCIGFa+cuftRXfyo/pd5gDoht4gNmUVsyCxiQ2YRg5onBAEAAAAA1I7BGQAAAAAUQLUTgqCAli4NJxj6+te/HtSSmWJDZ5xxRlDr06dP7Y2h0A455JCgdscdd+Rad/z48Zn1rJt2gSJ4/PHHcy+bNRECyu+9994LarVM/CFJe+65Z1C74YYbglrfvn1r2k+WrAlBJk2aFNROOumkzPVXrVoV1LL+bRx00EFBbejQoXlaRB3ceOONQS3rdydJEyZMaHQ7NcuamOf6668PahtvnD2UOf3004NaJBPUcOUMAAAAAIqAwRkAAAAAFACDMwAAAAAoAAZnAAAAAFAATAhSIpdeemlQy5okpKtPSN9xxx3r3RIKZsmSJUHtnnvuCWrvvvtuUOvXr19Qy7rhVpLa2tqq6A6or3vvvTeoXXPNNZnLjhw5Mqh94QtfqHtPKL9Ro0YFtazcNWLyj7yyJu+47rrrMpd94IEHGt0Ouun1118Pavfdd1/u9bMmiyuaKVOmBLVly5YFteHDh2euv/fee9e9p2bhyhkAAAAAFACDMwAAAAAoAAZnAAAAAFAADM4AAAAAoACYECRSd999d1A777zzcq176623ZtZ32WWXmnpC8Y0bNy6oLV++PNe6Rx99dFAbOnRozT0BjXLXXXcFtddeey1z2TFjxgS1zTbbrO49IU5r167Nvez999/fwE7qw92D2vvvv5972azH48wzzwxq06ZNq6I7VJI1adeLL74Y1I466qhmtNMQCxcuzLVcGc9duXIGAAAAAAXA4AwAAAAACoDBGQAAAAAUAIMzAAAAACiAioMzM7vazJaa2ROdameZ2Utm9mj6NbaxbQL5kVnEiNwiNmQWsSGziEGe2RqnSrpE0s/Wq//I3S+se0fIZe7cuUFt9erVQe3zn/98UNt9990b0lOBTBWZ1ezZs4PaI488kmvdvfbaK6idffbZtbaEDZsqcltXjz32WO5lDz/88AZ2UlpTVcLMXn755UGtV69eLeikcebMmRPUuvr/wcyCWtbj8b3vfa/2xhpvqkqQ2S233DKojRgxIqjNnz8/c/2VK1cGtT59+tTcV7WWLl0a1G688cZc6+6xxx71bqflKl45c/c/SAp/i0BBkVnEiNwiNmQWsSGziEEt95xNNLPH00vE23S1kJkdZ2YdZtaxbNmyGnYH1IzMIkYVc0tmUTBkFrHh/ACFUe3g7DJJQyWNkLRE0kVdLejuU9y93d3b+/XrV+XugJqRWcQoV27JLAqEzCI2nB+gUKoanLn7q+6+1t3fl3SlpE/Vty2gvsgsYkRuERsyi9iQWRRNnglBAmY2wN2XpN9+UdITG1oetXn77beD2i9/+cugtummmwa1rBt0e/fuXZ/GIlLmzK5YsSKz/oMf/CCoZU0akyXrxuK2trZu9YXalTm39fbKK68EtXnz5gW1nXbaKXP9L37xi3XvqScqQ2Zvu+22VrdQtay32i1YsCCoZf3/0B19+/YNarGeW8SY2c033zyoDRs2LKjNnDkzc/39998/qJ100km1N9bJE09kP4wLFy4MaosXLw5qWRPRZNloo/J9KljFwZmZ/ULSXpL6mtmLks6UtJeZjZDkkhZJOr5xLQLdQ2YRI3KL2JBZxIbMIgYVB2fuflRG+aoG9ALUBZlFjMgtYkNmERsyixiU71ogAAAAAESIwRkAAAAAFEBVE4KguS644IKg9sgjjwS1/fbbL6h95jOfaUhPKI6LLsqe9feBBx7Itf4hhxwS1M4+++xaWgKaburUqUHt1VdfDWpZz5NAWZx77rlB7dJLL61pm0OGDAlq1157bVAbNGhQTftBbc4666yg5u6Zy2ZNenPkkUfWtZ+uPmoga6KP5cuXV72fr371q1WvW1RcOQMAAACAAmBwBgAAAAAFwOAMAAAAAAqAwRkAAAAAFAATghRI1g2aknTOOecEta233jqoffe73617Tyi+yZMn17R+1s3ibW1tNW0TaLbFixfnWm6bbbZpcCdAc4wdOzaoPf3003Xfz/Dhw4Pa5z73ubrvB7XZeeedg9qMGTMyl82aVG7hwoV17eewww7Lvez48eOD2rRp03Ktu/nmm+feTyy4cgYAAAAABcDgDAAAAAAKgMEZAAAAABQAgzMAAAAAKAAmBGmRFStWBLVvfOMbmcuuWbMmqGXdCLz77rvX3hh6nKws9u7du+77yZrEJms/7733XlB7/fXXc+/ntddeC2o/+tGPcq+/vl69emXWf/jDHwa1LbbYour9oDZz5szJtdwBBxzQ4E4QO3cPamvXrs29/u23355rua997WtB7eWXX869n6w+zSz3+nl1NVkZ4jVy5MhctWb5u7/7u6rXnT9/fmb94x//eNXbbDWunAEAAABAATA4AwAAAIACYHAGAAAAAAXA4AwAAAAACqDi4MzMBprZb81sgZk9aWbfTOt9zOxXZvbH9M9tGt8uUBmZRWzILGJEbhEbMosY5JmtcY2kk939YTPbUtJDZvYrSV+RdJe7n2dmp0k6TdK3GtdqvLJmeRozZkxQe+GFFzLXHzZsWFA755xzam+svMhsN+y6665N2c+XvvSloDZgwICg9uqrrwa16dOnN6SnWvTv3z+onX766dVujsx2w7x584JaVm7QcKXM7YQJE4Laqaeemnv9/fffP6h1NetrtctJ2ecW3Vl/fSeccELV60aklJmNXdbMo1m1LDHPytiVilfO3H2Juz+c/v1NSU9J2l7SwZKuTRe7VtIhDeoR6BYyi9iQWcSI3CI2ZBYx6NY9Z2Y2RNJISfdL6u/uS9IfvSIpfBk5Wec4M+sws45ly5bV0ivQbWQWsSGziFF3c0tm0Wo816Kocg/OzKxN0k2STnT3Nzr/zJNrj5nXH919iru3u3t7v379amoW6A4yi9iQWcSomtySWbQSz7UoslyDMzPrrSTE17n7rLT8qpkNSH8+QNLSxrQIdB+ZRWzILGJEbhEbMouiqzghiJmZpKskPeXukzv9aLak8ZLOS/+8tSEdlsDChQuDWkdHR+71J0+eHNSGDh1aU09l1tMyO3bs2Mz6Lbfc0txGKpgxY0bdt9m7d++gttFG+d4QcNBBBwW19vb23Pv+7Gc/m3vZSnpaZmt18803B7U1a9YEtZEjRwa10aNHN6SnnqisuR03blxQO//88zOXXb58eaPb6Za+ffsGtZ133jmoXXnllUEta4KmsilrZmOX/Foq13qKPLM17iHpGEnzzezRtDZJSYBnmNmxkhZLCqdiA1qDzCI2ZBYxIreIDZlF4VUcnLn73ZK6Gr7uU992gNqRWcSGzCJG5BaxIbOIQbdmawQAAAAANAaDMwAAAAAogDz3nKEbFi9eHNT23XffXOteeOGFmfUDDjigpp5QbrNmzcqsZ93Avnr16qr3s2DBgqA2ffr0qrcnSccee2xQGzx4cO71Dz300KCWdfM74vXWW28Ftdtvvz3XuocffnhQ69WrV809odyynoNuuOGGzGWzJl768Y9/XOeO8vvOd74T1CZOnNiCToD83nnnnVzLbb755g3upBi4cgYAAAAABcDgDAAAAAAKgMEZAAAAABQAgzMAAAAAKAAmBKmzK664IqhlTRKSZfTo0Zn1nvwp6ajeqaee2vB9XH/99Q3fB3q23r17B7UPf/jDQe3ggw8Oat/85jcb0RJ6oD333DN3PWsSsClTpgS1OXPmBLUDDzwwqB1//PGZ+3b3oDZ8+PDMZYEiu+aaa4Ja1vP8GWec0YRuWo8rZwAAAABQAAzOAAAAAKAAGJwBAAAAQAEwOAMAAACAAmBCkBrMmzcvqF1yySUt6AQAyilrQpB77723BZ0A+YwZMyZXDUBi1KhRQe0//uM/gtree+/djHZajitnAAAAAFAADM4AAAAAoAAYnAEAAABAATA4AwAAAIACqDghiJkNlPQzSf0luaQp7v6fZnaWpK9JWpYuOsnd5zaq0SK6++67g9qbb76Za91hw4YFtba2tpp7AplFfMgsYkNmESNyW0xz5sxpdQuFkme2xjWSTnb3h81sS0kPmdmv0p/9yN0vbFx7QFXILGJDZhEbMosYkVsUXsXBmbsvkbQk/fubZvaUpO0b3RhQLTKL2JBZxIbMIkbkFjHo1j1nZjZE0khJ96eliWb2uJldbWbbdLHOcWbWYWYdy5Yty1oEaBgyi9iQWcSGzCJG5BZFlXtwZmZtkm6SdKK7vyHpMklDJY1Q8irERVnrufsUd2939/Z+/frV3jGQE5lFbMgsYkNmESNyiyLLNTgzs95KQnydu8+SJHd/1d3Xuvv7kq6U9KnGtQl0D5lFbMgsYkNmESNyi6LLM1ujSbpK0lPuPrlTfUD63l1J+qKkJxrTYvxGjBgR1O66666g1qdPnyZ0U35kFrEhs4gNmUWMyC1ikGe2xj0kHSNpvpk9mtYmSTrKzEYomYp0kaTjG9AfUA0yi9iQWcSGzCJG5BaFl2e2xrslWcaP+PwHFBKZRWzILGJDZhEjcosYdGu2RgAAAABAYzA4AwAAAIACyHPPGbrw7W9/O1cNAAAAACrhyhkAAAAAFACDMwAAAAAoAAZnAAAAAFAADM4AAAAAoADM3Zu3M7Nlkhan3/aVtLxpO2+sMh2LVNzjGezu/Zq5QzIbjaIeD5mtnzIdi1Ts42lqbkucWalcx1PkY2nlc22RH5dqlOl4inwsXWa2qYOzD+zYrMPd21uy8zor07FI5TueeinT41KmY5HKdzz1UqbHpUzHIpXveOqlbI9LmY6nTMdST2V7XMp0PLEeC29rBAAAAIACYHAGAAAAAAXQysHZlBbuu97KdCxS+Y6nXsr0uJTpWKTyHU+9lOlxKdOxSOU7nnop2+NSpuMp07HUU9kelzIdT5TH0rJ7zgAAAAAAf8XbGgEAAACgABicAQAAAEABNH1wZmZjzOwZM3vOzE5r9v5rZWZXm9lSM3uiU62Pmf3KzP6Y/rlNK3vMy8wGmtlvzWyBmT1pZt9M61EeT6OQ2eIgs/mQ2eIgs/nFnNsyZVYit3nFnFmpXLktU2abOjgzs16SLpW0n6Thko4ys+HN7KEOpkoas17tNEl3ufsOku5Kv4/BGkknu/twSZ+W9G/p7yPW46k7Mls4ZLYCMls4ZDaHEuR2qsqTWYncVlSCzErlym1pMtvsK2efkvScuz/v7qslTZd0cJN7qIm7/0HSyvXKB0u6Nv37tZIOaWZP1XL3Je7+cPr3NyU9JWl7RXo8DUJmC4TM5kJmC4TM5hZ1bsuUWYnc5hR1ZqVy5bZMmW324Gx7Sf/T6fsX01rs+rv7kvTvr0jq38pmqmFmQySNlHS/SnA8dURmC4rMdonMFhSZ3aAy5rYUv2Ny26UyZlYqwe849swyIUidefLZBFF9PoGZtUm6SdKJ7v5G55/FeDzonhh/x2S2Z4vxd0xme7ZYf8fktmeL8Xdchsw2e3D2kqSBnb7/27QWu1fNbIAkpX8ubXE/uZlZbyUhvs7dZ6XlaI+nAchswZDZishswZDZXMqY26h/x+S2ojJmVor4d1yWzDZ7cPagpB3M7KNmtomkIyXNbnIPjTBb0vj07+Ml3drCXnIzM5N0laSn3H1ypx9FeTwNQmYLhMzmQmYLhMzmVsbcRvs7Jre5lDGzUqS/41Jl1t2b+iVprKRnJS2U9J1m778O/f9C0hJJ7yl5f/Gxkj6iZAaYP0r6taQ+re4z57F8Vsnl3cclPZp+jY31eBr4OJHZgnyR2dyPE5ktyBeZ7dZjFW1uy5TZ9HjIbb7HKdrMpv2XJrdlyqylBwQAAAAAaCEmBAEAAACAAmBwBgAAAAAFwOAMAAAAAAqAwRkAAAAAFACDMwAAAAAoAAZnAAAAAFAADM4AAAAAoAD+H3Eh6H63npkyAAAAAElFTkSuQmCC",
            "text/plain": [
              "<Figure size 1080x216 with 5 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "predictions = predict(X_test[:5])\n",
        "fig, axs = plt.subplots(1, 5, figsize=(15, 3))\n",
        "\n",
        "for i in range(5):\n",
        "    axs[i].imshow(X_test[i, :, :, 0], cmap=\"binary\")\n",
        "    axs[i].set_title(\"{} - should be {}\".format(predictions[i], np.argmax(Y_test[i])))"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.11"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}