docs/notebooks/ml/symbolic_ml.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WIV-wwE8RzcF"
      },
      "source": [
        "# Symbolic Machine Learning\n",
        "\n",
        "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google/pyglove/blob/main/docs/notebooks/ml/symbolic_ml.ipynb)\n",
        "\n",
        "Machine learning (ML) is sensitive to neural architectures and hyperparameters, making their representation critical to the design of an ML system. ML practitioners rely on these representations to conduct experiments and deploy models in production.\n",
        "\n",
        "In a traditional ML system, ML components are built to serve the system's purpose, with a separate configuration system used to adjust their hyperparameters. This requires developers and users to maintain a mapping between the components and their configurations, ensuring that they remain consistent. PyGlove offers an alternative approach by using symbolic classes to develop ML components. Symbolic objects are a natural and mutable representation of these components, eliminating the need for a separate configuration system. As a result, symbolic ML systems are simpler and more user-friendly. In this Colab, we provide an example of a symbolic ML pipeline that can support the entire experimentation life-cycle with ease and power."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_BbNM22VCOOm"
      },
      "outputs": [],
      "source": [
        "!pip install pyglove"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NEpx8dzySCla"
      },
      "source": [
        "\u003ca name=\"overview\"\u003e\u003c/a\u003e\n",
        "## An Overview of a Symbolic ML Experiment\n",
        "\n",
        "In PyGlove, an ML pipeline (or experiment) can be represented as a symbolic object - an object that can be manipulated safely after their creation based on their construction signatures. The root object is the whole experiment, whose sub-nodes are symbolic objects of ML components that specify the details of the experiment. \n",
        "\n",
        "All ML components (including the `Experiment` class) are symbolic, meaning that they can be safely manipulated after creation. With PyGlove, users can create a symbolic class by extending `pg.Object` or symbolizing a regular class via `pg.symbolize`. The code below shows the definition of `Experiment` class, and lower-level ML components symbolized from functions or Keras classes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4USYYRWxgN0E"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import pyglove as pg\n",
        "\n",
        "@pg.members([\n",
        "  ('model', pg.typing.Object(tf.keras.Model),\n",
        "   'Model used for both training and evaluation.'),\n",
        "  ('training', pg.typing.Dict([\n",
        "    ('input', pg.typing.Callable(), \n",
        "     'A callable that returns a tuple of (features, labels) for training.'),\n",
        "    ('batch_size', pg.typing.Int(min_value=1),\n",
        "     'Training batch size.'),\n",
        "    ('num_epochs', pg.typing.Int(min_value=1),\n",
        "     'Number of epochs for training.'),\n",
        "    ('loss', pg.typing.Object(tf.keras.losses.Loss),\n",
        "     'Loss used for optimization.'),\n",
        "    ('optimizer', pg.typing.Object(tf.keras.optimizers.Optimizer),\n",
        "     'Optimizer for minimizing the loss.'),\n",
        "  ])),\n",
        "  ('evaluation', pg.typing.Dict([\n",
        "    ('input', pg.typing.Callable(),\n",
        "     'A callable that returns a tuple of (feature, labels) for evaluation.'),\n",
        "    ('batch_size', pg.typing.Int(min_value=1),\n",
        "     'Evaluation batch size.'),\n",
        "    ('metric', pg.typing.Object(tf.keras.metrics.Metric),\n",
        "     'Evaluation metrics.')\n",
        "  ]))\n",
        "])\n",
        "class Experiment(pg.Object):\n",
        "\n",
        "  def run(self):\n",
        "    self.model.compile(\n",
        "        optimizer=self.training.optimizer,\n",
        "        loss=self.training.loss,\n",
        "        metrics=[self.evaluation.metric])\n",
        "    x_train, y_train = self.training.input()\n",
        "    self.model.fit(x_train, y_train, epochs=self.training.num_epochs)\n",
        "\n",
        "    x_test, y_test = self.evaluation.input()\n",
        "    _, test_metric = self.model.evaluate(x_test, y_test)\n",
        "    return test_metric\n",
        "\n",
        "\n",
        "# Create an experiment instance.\n",
        "@pg.symbolize\n",
        "def mnist(training):\n",
        "  (x_train, y_train),(x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
        "  x_train, x_test = x_train / 255.0, x_test / 255.0\n",
        "  if training:\n",
        "    x_train /= 255.0\n",
        "    return x_train, y_train\n",
        "  else:\n",
        "    x_test /= 255.0\n",
        "    return x_test, y_test\n",
        "\n",
        "# Symbolize regular Keras classes to make them serializable.\n",
        "Sequential = pg.symbolize(tf.keras.Sequential)\n",
        "Flatten = pg.symbolize(tf.keras.layers.Flatten)\n",
        "Dense = pg.symbolize(tf.keras.layers.Dense)\n",
        "Dropout = pg.symbolize(tf.keras.layers.Dropout)\n",
        "Adam = pg.symbolize(tf.keras.optimizers.Adam)\n",
        "SparseCategoricalCrossentropy = pg.symbolize(\n",
        "    tf.keras.losses.SparseCategoricalCrossentropy)\n",
        "SparseCategoricalAccuracy = pg.symbolize(\n",
        "    tf.keras.metrics.SparseCategoricalAccuracy)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u3AmROgVENPR"
      },
      "source": [
        "With symbolic classes, an experiment can be expressed as an `Experiment`. And we can run the experiment by calling its `run` method, and save it for reproduction later."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 17966,
          "status": "ok",
          "timestamp": 1657147628587,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "ITkW5SZHENdB",
        "outputId": "dbf09117-c80d-4046-bf15-6b7ffe422b75"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11490434/11490434 [==============================] - 0s 0us/step\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 9s 4ms/step - loss: 0.7588 - sparse_categorical_accuracy: 0.8040\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 6s 3ms/step - loss: 0.3478 - sparse_categorical_accuracy: 0.9005\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.3042 - sparse_categorical_accuracy: 0.9105\n"
          ]
        }
      ],
      "source": [
        "# Create an instance of Experiment via composition.\n",
        "exp1 = Experiment(\n",
        "  model=Sequential(layers=[\n",
        "     Flatten(input_shape=(28, 28)),\n",
        "     Dense(1024, activation='relu'),\n",
        "     Dropout(0.2),\n",
        "     Dense(10, activation='softmax')\n",
        "  ]),\n",
        "  training=pg.Dict(\n",
        "      input=mnist(training=True),\n",
        "      batch_size=32,\n",
        "      num_epochs=2,\n",
        "      loss=SparseCategoricalCrossentropy(),\n",
        "      optimizer=Adam(),\n",
        "  ),\n",
        "  evaluation=pg.Dict(\n",
        "      input=mnist(training=False),\n",
        "      batch_size=32,\n",
        "      metric=SparseCategoricalAccuracy()\n",
        "  ))\n",
        "\n",
        "# Run and save experiment.\n",
        "exp1.run()\n",
        "exp1.save('exp1.json')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2NK2pbybbI2O"
      },
      "source": [
        "\u003ca name=\"lifecycle\"\u003e\u003c/a\u003e\n",
        "## Life-cycle of a Symbolic ML Experiment\n",
        "\n",
        "Most advancements in machine learning are based on iterating existing experiments. The life-cycle for experimenting a new idea can be described in 4 phases:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c0L2hJhylSSp"
      },
      "source": [
        "![ml_lifecycle.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAj4AAACtCAYAAACwX3VoAAAK5GlDQ1BJQ0MgUHJvZmlsZQAASImVlwdUU1kagO976SEhQEIEpITeBOkEkBJ6AKVXUQlJIKGEmBAUbKgMjuCIICLFMpRREQVHR0DGgliwothQUQdkUFDWwYINlX3AEmZmz+6e/d+5737nz3//cs+9Of8DgBzEEYvTYCUA0kWZkjA/T0ZMbBwD9wxggTaAgCEw4HClYlZISBBAZGb+q7y/h1gictti0te///5fRYXHl3IBgOIRTuRJuekItyPjFVcsyQQAdQTR66/IFE/yHYRpEiRBhIcmOXmav0xy4hSjlaZsIsK8EDYAAE/icCTJAJCsED0ji5uM+CGFIGwl4glFCOci7MYVcHgII3HBvPT0jEkeQdgEsRcDQKYhzEz8k8/kv/hPlPvncJLlPF3XlOC9hVJxGif7/9ya/y3pabKZGEbIIAkk/mHITEf2735qRqCcRYmLgmdYyJuyn2KBzD9yhrlSr7gZ5nG8A+Vr0xYFzXCS0Jct95PJjphhvtQnfIYlGWHyWEkSL9YMcySzcWWpkXK9gM+W+88RRETPcJYwatEMS1PDA2dtvOR6iSxMnj9f5Oc5G9dXXnu69E/1CtnytZmCCH957ZzZ/Pki1qxPaYw8Nx7f22fWJlJuL870lMcSp4XI7flpfnK9NCtcvjYTOZyza0Pke5jCCQiZYeANfEAQ8jBAJLAFdsAaOAFkZzL5KzMni/HKEGdLhMmCTAYLuXF8BlvEtZzHsLGysQFg8v5OH4m3YVP3EqKfntVl1CFH+T1yZ4pndYmlALTkA6D2cFZnsAcASh4AzR1cmSRrWoeefGEAEVAADagj/w/6wARYABvgAFyAB5JxAAgGESAWLAVcIADpQAJWgNVgPcgHhWAb2AEqwV5QCw6Aw+AoaAEnwVlwEVwFN8Fd0Av6wCB4CUbBezAOQRAOIkNUSB3SgQwhc8gGYkJukA8UBIVBsVAClAyJIBm0GtoIFUIlUCVUDdVDP0MnoLPQZagbegD1Q8PQG+gzjIJJMA3Wgo3g+TATZsGBcAS8BE6Gl8M5cB68FS6Ha+BDcDN8Fr4K34X74JfwGAqgFFB0lC7KAsVEeaGCUXGoJJQEtRZVgCpD1aAaUW2oTtRtVB9qBPUJjUVT0Qy0BdoF7Y+ORHPRy9Fr0VvQlegD6Gb0efRtdD96FP0NQ8ZoYswxzhg2JgaTjFmByceUYfZhjmMuYO5iBjHvsVgsHWuMdcT6Y2OxKdhV2C3Y3dgmbDu2GzuAHcPhcOo4c5wrLhjHwWXi8nEVuEO4M7hbuEHcR7wCXgdvg/fFx+FF+A34MvxB/Gn8Lfxz/DhBiWBIcCYEE3iEbEIRoY7QRrhBGCSME5WJxkRXYgQxhbieWE5sJF4gPiK+VVBQ0FNwUghVECrkKpQrHFG4pNCv8ImkQjIjeZHiSTLSVtJ+UjvpAektmUw2InuQ48iZ5K3kevI58hPyR0WqoqUiW5GnuE6xSrFZ8ZbiKwqBYkhhUZZScihllGOUG5QRJYKSkZKXEkdprVKV0gmlHqUxZaqytXKwcrryFuWDypeVh1RwKkYqPio8lTyVWpVzKgNUFFWf6kXlUjdS66gXqIM0LM2Yxqal0Apph2ldtFFVFVU71SjVlapVqqdU++gouhGdTU+jF9GP0u/RP8/RmsOaw5+zeU7jnFtzPqjNVfNQ46sVqDWp3VX7rM5Q91FPVS9Wb1F/rIHWMNMI1VihsUfjgsbIXNpcl7ncuQVzj859qAlrmmmGaa7SrNW8pjmmpa3lpyXWqtA6pzWiTdf20E7RLtU+rT2sQ9Vx0xHqlOqc0XnBUGWwGGmMcsZ5xqiupq6/rky3WrdLd1zPWC9Sb4Nek95jfaI+Uz9Jv1S/Q3/UQMdgocFqgwaDh4YEQ6ahwHCnYafhByNjo2ijTUYtRkPGasZs4xzjBuNHJmQTd5PlJjUmd0yxpkzTVNPdpjfNYDN7M4FZldkNc9jcwVxovtu8ex5mntM80byaeT0WJAuWRZZFg0W/Jd0yyHKDZYvlq/kG8+PmF8/vnP/Nyt4qzarOqtdaxTrAeoN1m/UbGzMbrk2VzR1bsq2v7TrbVtvXduZ2fLs9dvftqfYL7TfZd9h/dXB0kDg0Ogw7GjgmOO5y7GHSmCHMLcxLThgnT6d1TiedPjk7OGc6H3X+w8XCJdXloMvQAuMF/AV1CwZc9Vw5rtWufW4MtwS3H9363HXdOe417k899D14Hvs8nrNMWSmsQ6xXnlaeEs/jnh+8nL3WeLV7o7z9vAu8u3xUfCJ9Kn2e+Or5Jvs2+I762fut8mv3x/gH+hf797C12Fx2PXs0wDFgTcD5QFJgeGBl4NMgsyBJUNtCeGHAwu0LHy0yXCRa1BIMgtnB24MfhxiHLA/5NRQbGhJaFfoszDpsdVhnODV8WfjB8PcRnhFFEb2RJpGyyI4oSlR8VH3Uh2jv6JLovpj5MWtirsZqxApjW+NwcVFx++LGFvss3rF4MN4+Pj/+3hLjJSuXXF6qsTRt6alllGWcZccSMAnRCQcTvnCCOTWcsUR24q7EUa4Xdyf3Jc+DV8ob5rvyS/jPk1yTSpKGkl2TtycPC9wFZYIRoZewUvg6xT9lb8qH1ODU/akTadFpTen49IT0EyIVUarofIZ2xsqMbrG5OF/ct9x5+Y7lo5JAyT4pJF0ibc2kIY3SNZmJ7DtZf5ZbVlXWxxVRK46tVF4pWnkt2yx7c/bzHN+cn1ahV3FXdazWXb1+df8a1prqtdDaxLUd6/TX5a0bzPXLPbCeuD51/fUNVhtKNrzbGL2xLU8rLzdv4Du/7xryFfMl+T2bXDbt/R79vfD7rs22mys2fyvgFVwptCosK/yyhbvlyg/WP5T/MLE1aWtXkUPRnm3YbaJt94rdiw+UKJfklAxsX7i9uZRRWlD6bseyHZfL7Mr27iTulO3sKw8qb60wqNhW8aVSUHm3yrOqaZfmrs27Puzm7b61x2NP416tvYV7P/8o/PF+tV91c41RTVkttjar9lldVF3nT8yf6vdp7Cvc93W/aH/fgbAD5+sd6+sPah4saoAbZA3Dh+IP3Tzsfbi10aKxuoneVHgEHJEdefFzws/3jgYe7TjGPNb4i+Evu45Tjxc0Q83ZzaMtgpa+1tjW7hMBJzraXNqO/2r56/6TuierTqmeKjpNPJ13euJMzpmxdnH7yNnkswMdyzp6z8Wcu3M+9HzXhcALly76XjzXyeo8c8n10snLzpdPXGFeabnqcLX5mv2149ftrx/vcuhqvuF4o/Wm08227gXdp2+53zp72/v2xTvsO1fvLrrbfS/y3v2e+J6++7z7Qw/SHrx+mPVwvDf3EeZRwWOlx2VPNJ/U/Gb6W1OfQ9+pfu/+a0/Dn/YOcAde/i79/ctg3jPys7LnOs/rh2yGTg77Dt98sfjF4Evxy/GR/H8o/2PXK5NXv/zh8ce10ZjRwdeS1xNvtrxVf7v/nd27jrGQsSfv09+Pfyj4qP7xwCfmp87P0Z+fj6/4gvtS/tX0a9u3wG+PJtInJsQcCWeqFUAhA05KAuDNfqQ/jgWAehMA4uLp/npKoOlvgikC/4mne/ApcQCgtgeAiFUABF0HoKISaWkR/xTkuyCEguhdAGxrKx//EmmSrc20L5I70po8nph4awIArhiAr8UTE+O1ExNfa5FkewFoz57u6ydF6RAA1TZWPr5BvWuu5oK/yXTP/6ca/z6DyQzswN/nfwLs6RzQjnZCWQAAAFZlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA5KGAAcAAAASAAAARKACAAQAAAABAAACPqADAAQAAAABAAAArQAAAABBU0NJSQAAAFNjcmVlbnNob3TMahy/AAAB1mlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4xNzM8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+NTc0PC9leGlmOlBpeGVsWERpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6VXNlckNvbW1lbnQ+U2NyZWVuc2hvdDwvZXhpZjpVc2VyQ29tbWVudD4KICAgICAgPC9yZGY6RGVzY3JpcHRpb24+CiAgIDwvcmRmOlJERj4KPC94OnhtcG1ldGE+Cqt3PhYAAC5jSURBVHgB7Z0FtBxFFoYrEGBxDS7BSbDl4BbcPUGDBc/itmjgBAiLSxYWdwma4K4bYFncFoK7J7jrbH/FqaZev56Znjc9LW/+Omfe9PRUV1V/dbvqr1tV83pUgmAUREAEREAEREAERKANCIzXBveoWxQBERABERABERABS0DCR4YgAiIgAiIgAiLQNgQkfNqmqnWjIiACIiACIiACEj6yAREQAREQAREQgbYhIOHTNlWtGxUBERABERABEZDwkQ2IgAiIgAiIgAi0DQEJn7apat2oCIiACIiACIiAhI9sQAREQAREQAREoG0ISPi0TVXrRkVABERABERABCR8ZAMiIAIiIAIiIAJtQ0DCp22qWjcqAiIgAiIgAiIg4SMbEAEREAEREAERaBsCEj5tU9W6UREQAREQAREQAQkf2YAIiIAIiIAIiEDbEJDwaZuq1o2KgAiIgAiIgAhI+MgGREAEREAEREAE2oaAhE/bVLVuVAREQAREQAREQMJHNiACIiACIiACItA2BCR82qaqdaMiIAIiIAIiIAISPrIBERABERABERCBtiHQs4h3+tJLL9lijRkzpojF63Zl6tOnj+nbt2/p7kt2kl2VFd1GZAuyBQhgB+o3srGForcJtSgUSviMHDnS8FLIh8CAAQMMryIHGrZRo0bZBq7I5eyuZUMg0+AVwU5kC/lbWRHaDPoMxI4Tv/lTaa8SYANlE0E9KkHIu5rUgOVdAx3zL0Jj1rFEf3waNmyYGrc4MDmcy9tGNEjKodKrZIkY7t+/f+ZeY/UbVSokp9N5twmN3HbuwkcNWCPVlV3cIhkxDRyiR6F4BIYMGZJ5hycBXDw7oERZthnqN4ppA4hg2oSih1yFTzXj5QEqm+us6BVdrXyICtzEca7iLBuyauXj/MCBAzt97UaZfMGxQusIOBshB55ZP2Td0MW1GbIFv0Zaf+zsIWoL5JyFEK42EJIdtL7uXQ7OBvgctYOs2wRXpkbecxU+0Q6tDMAagVu2uHGdShYNWS1O0TK5xo13hewJROuDEmQlkPPMO3vS5cgxWidZtOFRj5/ahHxtJWoDlCbvfqMekfGHBqFepFZ8Dyx/9T2N5+DBg1uRldJMSMCJCb9exo0bZ/r165cwhXSjxT1Qw4cPN7169Uo3I6WWmAA2gjd29OjR4TXYC+daXS/R6c6sBFd4ozroRAB7oP7Hjh1rv3Pvri3pdEGTJ2gTfNtz/Uarba/JYnfry+PaBOqIuilqyO13fDBgPxQZkl/O7n5MPfiNFi5NXnkEX4CRv2wkj1ronCf2EZ3HZ6ddK0NceyF7aCXx5GlHbSFaV8lTqh8zmrZsoD6zLGLQJkTrIq9+I8n95iJ8okCiwJIUXHFaR4AdGn6IChD/u1Ye+3aCjchOWkm7sbRp6HyB3NjVzceWLTTPMM0UouLHf3bTzMdPSzbg08j/OFofrR4MNXPHuQifaEcaBdbMDena5glEO7RofTWfg1LobgRa3dH5Nhi1z+7GUvcTTyBqY0yvKhSLQFmezVyET7GqSqWJI5C3AauRi6uVYp2LegajdZZmaf201eGlSTadtKLthS9U08nBdFgTSprRPNPKR+mkQ8B/ZtNJMb1UchE+/kMh402vMluVUpENuFX3rHRFQAREQAQaI1CWQUkuwqcxlIotAiIgAiIgAiIgAukQkPBJh6NSEQEREAEREAERKAEBCZ8SVJKKKAIiIAIiIAIikA4BCZ90OCoVERABERABERCBEhCQ8ClBJamIIiACIiACIiAC6RCQ8EmHo1IRAREQAREQAREoAQEJnxJUkoooAiIgAiIgAiKQDgEJn3Q4KhUREAEREAEREIESEJDwKUElqYgiIAIiIAIiIALpEJDwSYejUhEBERABERABESgBgS4JH/6FwciRI0tweypingTOOOMM88QTT+RZhELl/dlnn5kff/yxUGXKqjBnnXWWueOOO8yXX36ZVZbKp2AE3n33XXPRRRcZ/QucglVMGxanS8KH/6/F/+QYOHCgFUBZiaBzzjnHTDfddB1evXr1MgsuuKBZf/31zSWXXFKaKlx44YXNkCFDSlPerhT0scceM6eddpoZPHiwufTSS82rr77alWQaumb06NGhfXAcF1555ZUwzocffhgXpaFz3N8qq6wSXkPdHnbYYeHnF1980cwzzzw2z3b+33SXX3652X333c0pp5xi/vOf/5hKpRIyasXBfvvtF9ZztN1wn5dbbrlWZK00qxC49957zbBhw8wBBxxgrr/+evPBBx9UiZne6Vb0GzzT2NADDzyQXkGVUmYEenY1JxpwOm6MmID4GTBggD127/ZDin8YLTNqPvbYYw2CxwVGEIwmd9hhBzN27Fjz97//3X1V2PfPP//cfPfdd4UtX5oF+/rrr81dd91lX7PNNptZZpllzLLLLmtmnHHGNLOxaf3yyy/WRnr06GGuvvpq069fv055cB47Ivz222+dvm/0xDfffNPBkzHvvPN2uLcjjjjCUN/XXnutmWuuuRpNvtvFf+qppwyvCy+80NoB9rDQQgulfp+rr766mXbaacN0r7rqKvPee++Zgw46KDxH56WQPYGPPvrIjBo1yr4WWGCBsE2YfPLJUy9MK/qNX3/91bYhP//8c+rlVYKtJ9Bl4UPR4sQP550IwivUihHuFltsYeaee26yCsNJJ51kO5Vzzz23FMInLHibHdDx8Lruuuuspw4BRMc3ySSTpEpi6aWXto0q023jjz9+h7QRPthlq1zuNOh+eP/9982KK65oNttsM/902x//8MMP5v7777evGWaYwYog7AFxnEZYb731DC8Xnn76aStQu7un1d1vWd5ffvllwwuP/VJLLWXtgOc37aB+I22i5U2vS1Nd/u068eOf4xjxgzfITYdFv0/7c8+ePc2qq65q3njjDUOD6gIeINyqK620kllnnXUMAgm17gKuyr/97W/2uh133NF2UIcffniHaRncsdtuu63tKDfffHPTv3//cO0K89Z77LGHWX755W36J5xwgomOAj755BNz3HHH2emQrbfe2jz55JMue/tOGqT/wgsvdDh/9tlnG8riB6aP9t9/f+vJOPDAA80jjzzif23weHCP6667ro1D3HHjxnWIU5QPuIsvuOACs8suu5gzzzyzE5dmykkjB/fodNdzzz1nXnvtNUM9RgMeuGOOOcasscYalh1TJaThB2wHcb3RRhvZ6dVrrrnG/9oe77PPPuaKK64wX331la1XbPL555+3x9wvdf3mm292uO7jjz+25+mc2zHA+cYbbzQHH3ywGTp0qLnzzjstvyxY3H333Za93y6Q7957723rkWNXP++88475xz/+YfAm0Z5gC9Hw7LPPGtoSBD1iN6ulANFylPHz448/boYPH2523XVXc/HFF5sxY8a09Daq9RtdaUfr9TXcCHGwcWwHcUf/iDfcD3jDaHvoU1ZbbTXb3nOdH5Lk5cfXcUcCdT0+biqr42WdP80xxxyGRiEu8OA7L1Dc92mcY90GjSUu84knntgm+fbbb9vGh7UEgwYNsl6FE0880dx22212lDneeOPZTpCRxs0332wbKabLcMFzDnc80zEsyKQjo0EjLRpIplLIc/HFF7eepu2228588cUXtlHkXh9++GEz4YQT2sWsG2ywgaGTp4OfaKKJzFprrWW+//778La5jvS32WYbw/oQF0jjf//7n53a4xwihwZ35plnttN6dJIrrLCC7dzxKDBtw1oTyk2Dy0PDugpc/IitWWaZxSXdpXcEZCPh999/TxQdpqz54DXllFOmMg226KKLGlzoeJb89Td4e/iMh8EPuMPpqGh0ELKTTjqpXYhJvVAupq8ICElE2iabbGLLybTJt99+a2afffYwOdfRbbzxxtYzSZ1PNtlk9hgvKOkz5XXUUUeF11x55ZXmhhtuMIjdsgbKjs3HhUYWNbMWjNdll11mllhiCcs5Ls20ztG5Us+IUjpCF6hHnnOeS6ZricPziK1Q/zxnrO9CMGMXBNeOYC9ch7cPEX7ooYdaUe3S7u7viEmeGz8gJpIGnql77rnHvmjv8AQypZx2iOs3utKOJulrEM+0S9NPP70dPJMP3mHaZwbgK6+8sm3DaeNpO7fcckvb37A+knaMARTPV5K80ubU1fSS6odG0qcNbXY5zZ9PeUzOPPhpTge4DiEmq4ZOMRKjIyFgPEyd4C1hfvj8888P06JT4mGh8Zl66qnteRZB05hiTIgcAg0ZIE8//XT7Ga/M/PPPb/bdd1+7TsSeDP4ggngYXcCLNNNMMxlGKX/5y1/saYx3zTXXtKMW1hrhfWG0jyu3d+/eNg75x3kcXLrV3jfddFMrmnhYEG0E8jrkkEOsKGL0iThiBMF5AotJEaUstuWemw0Ig1YGvCQI2DQCHQ6dsT/dhfBhzU3UK4dHjjrCjhBMBDo16hw7QCzjPSKtESNGmK222srGwVacKLInvD/YKN6LW265xcw333z2mK8RwggdX/ggULFBZ9deMqU5dOum0iwwgj3qIU0z/UbTwh5YT0igc2Lgw9othA/tyJ577mnP+Z3+nHPOaZ9RPH3YQTsE2t20hAobENLoO5L2G11pR5P0NbTbP/30k21L3ECJQdCss85qbr/9dit8EOFuzeraa69tTYWBMgLirbfesv1SkryKYmNp6gd3T6TZUuHjMkrrnWkxpolo5Kt5h5LkxVQWIzG8Jo8++qi9BIGB23CaaaYJk8CYmLZgqsdN90wxxRTW0BiROOHDBXRGLjBCp9NEifsB0eQCHSeNGwbpRA/f4U1AZJE+wofyMQXnRA9xEDB+OTlXLzAdxohh5513DkUP19x0003hKJvOmdERQofpHBdY4Et5mg2MhhtZf4EgbTTAhdEdHhm2vjYTEJeICwQL9fLf//7X0IhigwggPzDiYqTlRA/fYSt4zRC7dHJcjyeIkZgL8MBtzSgsacDusCVsg3tFGDMFd+qppyZNopDx2HDgPwt+IRFFvpfT/67WMdMBMHKDklpxs/iOKU4XGHwsssgiYRuEaMbmmTr3nz8GOniJsbF2ET48O3hv/YDHhzas0cAzhg0wKIpOCzWSVtJ+oyvtaJK+hkEoLwLtCW06XkMG7Hi4CHiCaWP22msvK6IRP9w7ZXIhSV4ubt7vaa7xTVNE1fT4+NAY5dYKCIBqBXOCx0GgYpsJeHXc4mbWSqwceFloGBErLvCA4YJGGPCKBn+NBe7D6O4fpoVoxPydV754QX3TmPnnyANxQCfr1uvABPHlB0RbIwKCa1160evctB5xcIXSsfudN+ddoONpZhEx65caCbj6ecDrBZjxcDPVtNhii9no1WypXlr+99gbU4eMyKkTxA4iZaqppvKj2WM6KqamogHP2a233mobbMqEh4/68wNCsxHhQ2NGOkydcN9M6ZAGZSxz2G233apuZuB3fJi6TRJwZWMLsCmaBwyPjx94/vA6E3j+CHhX/Z8zsCeDP36b485113favOionI4e73SS4Ka8sQMnFpv1+iTpNyhbknbUv4ekfQ1T+niM8bzTliAEEc4MovmOQPvMln+8zXiaCfwUBusG8SYmzcteWIA/aW4kqKUxGr3VxMKnVsJxBYqKnVrXN/MdCpn5TxaC8aCxS4SOlAeHDgpjoXzR4O/0QcC4qSMXj+3HbHX1RRrpuuC2ybI+JxoQGE6Y4caMW9/gCyrXkUbnwP2pA0ZQhGhapMN1dOZ4mhAODz74oI0b/dOM6ImmlcZnHno6N17V1oY0mw9eHxqbf/7zn9ZOqnlVqM9qdYkN0OElqcsk5cX2mPbAo4VgZ1CBJ8/ZQZI0ulsct44DW+A4y+C48xzh7SXQEcXZg4sbVz43nc6CdzdN4cdrlY37eZT5GLZO8OIla2Wo1m+QZ6PtaNK+hn5o6NChVsQwiMSbibeHttsJH/KHAetJGVwz7c+ADQ8QtslgEk5J+jXSUogn8MdCkfjvEp31RQ9iB4VHQ8678/AkSqiJSBgQiwcfeughc/LJJ9uUGImx9oJRAqNGhAMvjAbjwQvgAl6J6GgU1c38fbWAKGLUHnW94o5l55XzXPAAs+6G+X8X8CT5HgI3qv30009dFOtN8r0eeHHoMLlHP7DriwXdzB0jJJ555hnr9XH3yzt1wQNXhNA7WOfEVBE7Nxj9sU6qlR0CXkC40tBQN/6Ups8DdkyJ+TsC+Z765TuEMXXJlCnTUi5gO9Gdde67Wu9Md5EWP/DIuqntt9++VvRu+R12j2fgyCOPtM8tA5esRQ9g454/PIBRW6hXCfyQKnbCInX/+Xv99dftmj6mNBU6E4Abwh+PDB18q0WPK0Fcv8F3jbajSfsa+iI2ozD4Ylod0YNt0C45ryG/dM8zgXeQtWFMm9IXIa7oV5Lm5e5R7/EEmhI+zpNCg0XnmqXYid4OeSMOWNOBMRE4ZuqHBagsQEZIsM2UuXbWbviB86w9opMkLRZU0iDXCkcffbTdggsHtheyE4VOHc+Q68h4kHFl0tGxLZ6FkewIcYZO+ix0Q0Qdf/zxthOlY8Uj4Ht8WD+B+5NdaWy5RjwhMNmBRnkZDSD+WGNB3mwNZlE3cfF4sOsrr8AIiu31lIWtwBtuuGGHH6BsZbkQv4hQ8ibfal4vFjxTh4hiGh22VzNdgRhiaymBUTyjMXbwMSJj+pH6ZlF0owFbJS1slMbQeQgbTaeM8fmtFlz35513nn0uqk3NZnVvdIAEFiiz+xKPKZ5COplGAh5Bdm4ifLAdRuzYD/aCbdUaSDWST3eICyt2niICGLyxDrLas9nK+43rN7rSjibpa1iSQb/CQImBKmv8aJMYjLsZAEQX/RdrgRA69F+034gjGBGS5NVKZt0h7T/nbhq8G9Qri0Sz8urUKx4dPw0pHgR+A4IpLzolpp343QTchRNMMIHtZNhR41bVu3RZGE2jhUEyp0ycej9nv9NOO1lPDg8PHSeCZ8kllzT33XefXX1P2nRoiBAEEA87XhvclL5rk+tY50EDSSfIZ8QaDTHbY11gATcjSjpxRBkPDCKHchDwpnDf3AfiisCDhLfDfbYnM/oDP6YunPcro2w7ZYPXB++S24nVKUJwgnUleHdYp+JECB0yYtj98CCikrqknui8mRphXQ78u7LzCDFMfoMGDYorUrc7x/1iD87DUpQbxOOAp5jnmPWAjMQZzMStDaxXZqYuuT86dHYK4vlhnRidlT+9Xi+d7vo9O5SwAbduJ+/7jOs3utKOJulrGECxfohBN20Hswb8FwIEEO02fQLl4V9s4AFjYETA2+M27/A5SV7EU6hOoEcAu+o/zEHc8CLUW9xcPYvO38RNj3WOle4ZvB80aBiRHxBLdD6snaFjw+MTXUDsx487BiG71DDkWo06Uxo0hP66oWh6TIHxOw+1Rj88NIwmmRaolh9rlJhe6+rUgV9HlDHN+o/ec9xnvHOUwQU6paxENjbAFFZ0IasrC+/8rgtTIdHfA/Lj1DvGW4fnA7vANssWsqwjBiYu4GHmlXZgrR/PMR1fsyIFjy7PMgMsBlztEFpdR35/BM8s2qSutKPV+hpnA+zgYv1YrX6G9gdvD2099sggNy7Uyyvumlaea2Ud+X1Ss3XfZY9PK+G1Im28LfUCaruWMVa7HqPsHRhnvcB0Vr2QJB0a0nojpka3y9crVzt9j/CsF9wajnrx4r5HKCO08Qrg2Suj6Im7r7Kfw9PqvH3N3gvCKa20mi2Lru86ga60o/X6Ggar1QasrqR49uulQ9wkcVyaev+TQFNrfP5MRkciIAJJCbCmhDVbzNsPLcii86RlVzwREAERKDuBtvH4VKso5pxZt1Fr+qnatTovAl0hwHoPpipZ2JjEC9iVPHSNCIiACIhAPIG2Fz78cBwvBRHIigA/oqggAiIgAiKQDwFNdeXDXbmKgAiIgAiIgAjkQEDCJwfoylIEREAEREAERCAfAhI++XBXriIgAiIgAiIgAjkQkPDJAbqyFAEREAEREAERyIeAhE8+3JWrCIiACIiACIhADgQkfHKArixFQAREQAREQATyIZC78PH/A3k+CJRrPQJZ/auIeuXQ98UiMGbMmFwKlFe+udysMq1KQH1HVTT6og6BXIQP/9zUDzJgn0Yxjv064R94Zh2iYkudXdY10Hh+0TprPIXqV7Qy7eq56pukBPz2Iuk1jcbLox1qtIztHt9vp4v8zOYifNrdOHT/yQj4D47/QCW7WrFaTcCvE7+uWpGv3+ll0cm24h66c5q+LXCfrfgnslEbGzVqVHdGWsp7859N/5kt2s3kInxkwEUzg47l4T/s+qEIBswD5T9Ufvl0nD0BbMSvj1bbSDT9qI1mT0A5+gSyqg+/71Cb4NdA/sdZ2UAad5qL8KHgUQMuE7Q0wBc1DerBrwvqya+rLMsdnRLVCC9L+rXz8m2EmFFhUvvqxr+N2iH5+8Kr8RR1RVoEorbQCm+PK2vUztQmODL5vmMDWdpBs3ebm/AZMmRIhw41DlyzN6frGyMQVwdR8dFYis3FprPzG1E6uoEDB3Z6wJrLRVc3QsDVgX8NdZSFOI7a4rBhw2QLfkVkfIwtROsg+symXaSorVGGaIebdp5KrzaBuH7Db7drX53Pt7n+k1IaMgzXBQcQaCj7LBpTl3e7vjv+jJzcsWMRbWTc+SzfKQPrB/yy+Q2d7KT1teHYx9lIqzs6/+5cXn79+8eyBZ9Wa45r2QI5RsVpK0pRq98gv6J3uq1gknWatewA/kWvg1yFT1xDRgX6jVnWFar8/iDg6qYIPKINHWWSjeRfM9hIFh2df6euQfXr3z/24+o4WwLUDTbR6uDapmi9u8/uvdXlUPqdCbi66fxNsc7kKnxAEdeQFQtR+5UmOg2ZNwEephEjRlixo0Yt79r4I3+eW/fsZl0il69sIWvy8fnxfCKAec8qOPvDBmQHWVGvnY+rk9qxivFt7sIHDA6YjDhfo6DhQvQUNbgOLzr1VdTydsdy5dHJxXFUmxFHJdtz2ALTi+65zDb3P3JzeUv85EH/jzyL0iY0QqBHJQjVLvCFCCPuLANziNHfhsgy/3bJi4aLgPGWLchGsqmxMtiIbCEbWyCXIq+lkh1kYwd5tAkspKd+Cc3qkUJ4fOKqio64jJ1x3L3oXGsIyEZaw7WMqcoWylhr6ZdZdpA+0+6YYm7b2bsjTN2TCIiACIiACIhAsQlI+BS7flQ6ERABERABERCBFAlI+KQIU0mJgAiIgAiIgAgUm4CET7HrR6UTAREQAREQARFIkYCET4owlZQIiIAIiIAIiECxCUj4FLt+VDoREAEREAEREIEUCUj4pAhTSYmACIiACIiACBSbgIRPsetHpRMBERABERABEUiRgIRPijCVlAiIgAiIgAiIQLEJSPgUu35UOhEQAREQAREQgRQJSPikCFNJiYAIiIAIiIAIFJuAhE+x60elEwEREAEREAERSJGAhE+KMJWUCIiACIiACIhAsQlI+BS7flQ6ERABERABERCBFAlI+KQIU0mJgAiIgAiIgAgUm4CET7HrR6UTAREQAREQARFIkYCET4owlZQIiIAIiIAIiECxCUj4FLt+VDoREAEREAEREIEUCUj4pAhTSYmACIiACIiACBSbgIRPsetHpRMBERABERABEUiRgIRPijCVlAiIgAiIgAiIQLEJSPgUu35UOhEQAREQAREQgRQJSPikCFNJiYAIiIAIiIAIFJuAhE+x60elEwEREAEREAERSJGAhE+KMJWUCIiACIiACIhAsQlI+BS7flQ6ERABERABERCBFAlI+KQIU0mJgAiIgAiIgAgUm4CET7HrR6UTAREQAREQARFIkYCET4owlZQIiIAIiIAIiECxCUj4FLt+VDoREAEREAEREIEUCUj4pAhTSYmACIiACIiACBSbgIRPsetHpRMBERABERABEUiRgIRPijCVlAiIgAiIgAiIQLEJ9ExavIEDByaNqngiIAIiIAIiIAIiUEgCNT0+ffr0KWShVSgREAEREAEREAER6AqBmh6fvn37mgEDBpgxY8Z0JW1dIwIiIAIiIAIiIAKpEejfv3/TafWoBKHpVJSACIiACIiACIiACJSAQM2prhKUX0UUAREQAREQAREQgcQEJHwSo1JEERABERABERCBshOQ8Cl7Dar8IiACIiACIiACiQlI+CRGpYgiIAIiIAIiIAJlJyDhU/YaVPlFQAREQAREQAQSE5DwSYxKEUVABERABERABMpOQMKn7DWo8ouACIiACIiACCQmIOGTGJUiioAIiIAIiIAIlJ2AhE/Za1DlFwEREAEREAERSExAwicxKkUUAREQAREQAREoOwEJn7LXoMovAiIgAiIgAiKQmICET2JUiigCIiACIiACIlB2AhI+Za9BlV8EREAEREAERCAxAQmfxKgUUQREQAREQAREoOwEJHzKXoMqvwiIgAiIgAiIQGICEj6JUSmiCIiACIiACIhA2QlI+JS9BlV+ERABERABERCBxAQkfBKjUkQREAEREAEREIGyE5DwKXsNqvwiIAIiIAIiIAKJCUj4JEaliCIgAiIgAiIgAmUnIOFT9hpU+UVABERABERABBITkPBJjEoRReAPAr/88osZN25c2+P45ptvzNdff932HARABESgXAQkfMpVX92ytB999JGZbrrp6r5uvPFGc9lll9l4rehwX3zxRZv2Aw88YDlffvnl9vOXX34Zct97773NlFNOaXr16mXLEn6R88E888xjhg0blkkpvv32W7PKKquYqaaayrJ49913M8nXZXLcccfZepl11lnNd9995053eL/oootsnCWXXDI8f8wxx9hz1a4JI5bw4M0337T3dttttxWm9AwQrrvuusKURwURAUegpzvQuwjkRWCyySYz++67b5j9J598Ys4880yz0UYbmSWWWCI8P99885mHH37YfPbZZ6ZSqYTn0zr49ddfbdo///yzTXLqqac2ffv2NeOPP779/Pzzz5szzjjDbLvttmaHHXYwiyyySFpZN50OTLLq0BEVDz74oDnppJPM4osvbmabbbamy99IAt9//72tpx49ehg6+s0337zT5YhWmCBSXXDXtcJ2XB55vU844YTWVieffPK8itApXwYJjzzyiNlss806facTIpAnAQmfPOkrb0uAxnrIkCEhDQQGwme99dYzu+yyS3ieA4RPVmH99dc3vFx4//337eF+++1nFltsMXe67d7hgKA44IADDOIjr7D00ktbj0JU+Hz44Ydm9OjRVgj8+OOPeRUv03zxfnHPRQpZCfEi3bPKUg4CmuoqRz2plBECeIX23HNPs9xyy1kPjJue8qM9++yzZscddzTLLLOMHXWOHDnS/7ruMV4NvDs04BdffLH1cHDR0KFDzSGHHBJeT7p0vuSDJ4h8o+Gxxx4z+++/v+nXr5858MAD7UiYOJ9//rnNI26KghHzlVdeGSbFtAH3wzTT8ccfb15//fXwu7iDJOWKXoc35NJLL7XeNu5n0KBB5qmnngqjHXzwwebWW281eMW22247Gzf8MnLw3HPPmd12282stNJK9r533XVX8+qrr4axqLM99tjDfPrpp2afffYxyy+/vK2n+++/P4xT62CLLbYwt99+eydPF5z69OljFl544VqXN/RdLVv67bffDHWFIP7999/DdG+44QZbt6+88or5+OOPw+ODDjrIrLjiimbw4MGmK3bLoODss882eLVWXnllc+SRRxqeB2z1ySeftPmT7l577WUQqZRthRVWsKyZlmRtFvVIvRx++OGd7Gjs2LFW1PL9OuusY+0eb6gLSertlFNOsTb+3nvv2XJhCwoiUBQCEj5FqQmVoyECdBw09jT8jz76qG2g/Q765ptvNngEHn/8cbPxxhubGWaYwdBRHnHEEYnzQVhcccUVtpNnTc9MM81kr5199tnD6R0EEK78L774wqbP9Ar53nHHHWE+uPsp50033WTWXHNNw3oMOqKHHnrITDPNNOall14yJ554YhifA+6FabWZZ57Znj/11FOtuEI4rLXWWgZRgzDx1x/5CSQplx/fHSNOdtppJzPLLLOYrbbaynzwwQc2H7dWg2kt1vYw/Tf33HPbdSXuWv/9zjvvNKyvefvtt63XbMEFFzTUyVJLLWW++uorG/W1116z66SoSzrGVVdd1YwZM8asscYa5oknnvCTiz2GOx6dqGi8+uqrbdljL+rCyXq2BAtE2+mnn249lWRBHW+//fZ2Snb++ee3i8CxpbXXXtveI4yZeuNefaFXLy/S/ve//22GDx9uhQ1CC2GFmCF9hAYBtpdccokVN9gmHkq+79+/v7VBnhlskDVzvseM+kIwEhf74oVtrr766qGoS1JvPCtMYTMFh51MMskktlz6IwKFIBCM8BREoFAEgk6QBTyV8847r1O5zj33XPtdMMoNvwsEij0XLHq153744YdK0EFXll122TAOByeffHKlZ8+elaCj6HDefQhG9TadoNO2p84//3z7OfDK2M9Bp2Q/v/zyy/ZzMCVXGW+88SqHHnqoS8K+B9NjlbnmmqsSLO60n2ecccZKsF6pEngGwnhBh1cJvFX2cyBwKsGUUeWdd94Jvw88IJU55pijEnRslUAY2e8pvwuB4KkEC8Irhx12mD0ViJFKIHbscdJyubTce9AB2/u74IIL3Cmbf9CpV6affvpK0Lna84HnqsI91QoDBw6s/PWvf60EnoIw2jXXXGPTD4SKPefqMvBehXECb0MlEBKVwCMRnoseBNOiNh3YBF6JyoABA8Iob731lmWFTQRC19aD+xI+2JW7D3e+1nsjtrTNNttUJp100gp5Y3vBgvNKsAjfJo/Nkfeiiy5qmbo8t9xyS8uSe0ma16abbmrTCoSzS6YSiBF7btSoUfacY3vUUUeFcYLF3TYOeboQiHF7LhA89lQgJiuBSKk4m+dk4EWycYK1XTaOS7tevQUeqEogouw1+iMCRSIgj08h5KcK0SiBoLMLL2FE2bt379Bl/8ILL9iR7wYbbGBHvoxQebFQGpd93PRCmFgDB3g1GHGvttpqHfLBg8GI/4033jBMLTDNsfPOO5tAJIWp4/1xazICkWBHxm5aizKOGDHCTiWxhgbvR9Bo2GkjlwBrbIJO3hx77LHuVPiepFxhZO8ALkHHbae33Gny32STTex0VCPTFdzLM888Yz1DTIvh1QqEnU2WXWF+YBG7C+zuw8vFFGCSEJ3uwtuDpwmbSCM0Ykv/+te/rAcsED3WY0dZoouNWbfmr4tiuhD7oC4byQuvGx6beoFnwAUWohPwgLoQCHR7SN4Epg7xQvFzDe65mWKKKQxriO6++24bx/1ppt5cGnoXgTwIaHFzHtSVZ9MEojuJ6LDZPktAcBACb4h92Q/eH0RJGsHlwzRAXCAfhBEhWt6JJ544vITprg033NCu5wm8RwbhwjoLpkoIdEp0dEwd+CH62X2XpFxMv0QDHR3ldLvY3PdMzxEQLkzpJAlMwbF9nPVAlGeiiSay6264FhHnh8B75H+00yKsm0kSEMCsZSEfRBBiY1CwLimt4FgmsSUEAvFY10S5nNDwy4Ko8ANTigSm+JiuIiTJC6GfJPh2F3g77SXsjnRhggkmcIdWgLGeDVHOKxqiz00z9RZNW59FIEsCEj5Z0lZeqRHwvSfRRNmGTgimVuyaiuj3rDtII5APHQeCweXpp8u6BtZSEKJrcehgEGoIGgKLotddd13Djja8JXiNnNeCDpVOETHgixK8IggoX0SRVpJyES8app12WrtWKXqetSgEV57o93Gf6fiffvppu1uPe2GNCb+TxHtU+PgekLi0ap0LpuDs+inWIAVTa1Yk+mtWal2b5DtXr0lsiTo64YQTrHeEhc140FiI7oeooHOeLYSM2zWYJC8nYvy04459YRP3vX8OLyJ1waaBuN+E8m2P65qpNz9fHYtA1gT+9L1nnbPyE4EWEWAhLcKIzgfR4F4sVqZTRFykEfgdH8QLHhqXB+90wngdmNJZYIEFrFhhIbMf2E2z0EILmZ9++smexqvCFA+dHtMNvteCeHSYTkS5dFgMzE6eaEhSrug1fOY6FoxHd6Xddddd9h6S/m4Ri2lZsMtuIra8s6CZDtjtOIp2/nFlaeQcnh4Wk18SLOZl15xbEN5IGtXiNmJL7ExjisgtZsdjFxW8bnrT5XfvvfdaDxd20khe7vo03xHQ8847r104jzfR2TQCJ1i/ZK699tqGsuMZdB7Phi5UZBFoMQEJnxYDVvLZE2A9Ar//g/Bh2oD1E3Q4rKdgCiluCqIrpWRXEaKEKR3Wd/AL1IgeOnymGPDmsBuMbcvsjCEeu25Yv3PhhRdabwhTQARG05SPnUGs8SFtF5gGQ3QghujcWTcULFq13o2jjz7aRQvfk5QrjOwd0FGz5oNOju33dNpsmabsbDuPepa8Szsc4iUJFvHa6Se8YXhCrr/+erudn4hp/74LO5VYR8TON3ai1QuwP+ecczq82MIfF5LaErvs+MkDtnGz6y9YIG7XKe2+++4dkuV7vCms67nqqqvMaaedZu2C+k+aV4cEU/6AXfE7SFtvvbXdEcnaLH5CAe8Va9kaCaxvYnqUX1znfhVEoDAEirTSWmURAQgk2dUVdModYAWj5UogHMJz7JAJvA2VQFjYHSnB6LXCbphguiWMEz1odFcX1wfTE5Vgward3RU81JWg86oEazw67BwKpooqwTqUSuD1sGUJRtCVQMRUgs66QxHYLUYa7A6KhkDsVNgJxrXEYfdQIJLCaIHICnd1cTJJucKLvYNgHUcl8CSF+bB7i91Q7DpyIcmurnvuuacSbIW26QQjf7u7JxCflWC6rBIILJuU2x0Urctg/VEl6Gxddp3e/V1d7stgm7jlG/ycgDtVdVcX/KIvdshVC/VsKdjyXwmmCSuB165DEuzWI59g6tLuJOQ4mEayO+Q4Zqdc8PMKHa6plxeRseNgoX6H6wKBafOK7ury2QaLk22cYAoyvNbZ3C233BKeC4Sh3TFIGbHZYLquwu4vF5LWW7AovxIIf5snLBREoCgEelCQwMAVRKBbEmBaJdiqa0fhjax3aBRG0GFZj4/bJRN3PdNieJ+YiolbmMxUHFMNTBNF14a49Jg+Y0TOmpAka5WSlMul7b+TD96xOeec0z/d8DEjfcrJAu6yh2Zsid9fYkE53g88eNhk8HMFHXb6+XyayctPp5lj1hzhtfH/7UdX0uO3rfAC1lqX15V0dY0IdJWAhE9Xyek6EUiJAGMPOjqmFPjBRXb4aOFoSnALkowvfPxt4AUpnoohAm1FQLu62qq6dbNFJMAC4LPOOsuKHxYSS/QUsZZUJhEQge5CQB6f7lKTuo/SEuB3etjdw/QWW7IVuh8Bpg7vu+8+++9Mor9/0/3uVnckAsUmIOFT7PpR6URABERABERABFIkoO3sKcJUUiIgAiIgAiIgAsUmIOFT7PpR6URABERABERABFIkIOGTIkwlJQIiIAIiIAIiUGwCEj7Frh+VTgREQAREQAREIEUCEj4pwlRSIiACIiACIiACxSYg4VPs+lHpREAEREAEREAEUiQg4ZMiTCUlAiIgAiIgAiJQbAISPsWuH5VOBERABERABEQgRQISPinCVFIiIAIiIAIiIALFJiDhU+z6UelEQAREQAREQARSJCDhkyJMJSUCIiACIiACIlBsAhI+xa4flU4EREAEREAERCBFAhI+KcJUUiIgAiIgAiIgAsUmIOFT7PpR6URABERABERABFIkIOGTIkwlJQIiIAIiIAIiUGwCEj7Frh+VTgREQAREQAREIEUCEj4pwlRSIiACIiACIiACxSYg4VPs+lHpREAEREAEREAEUiTwfxtMV2LeGa6aAAAAAElFTkSuQmCC)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "77SAfVDdbP1h"
      },
      "source": [
        "\u003ca name=\"reproduction\"\u003e\u003c/a\u003e\n",
        "### Reproduction\n",
        "\n",
        "Reproducing an existing experiment is simply clone the existing experiment object, or load if from a saved JSON file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 16318,
          "status": "ok",
          "timestamp": 1657147645035,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "So8dHXxoQqPw",
        "outputId": "30c8fbf9-597a-49b0-dd57-6cc53cae969f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 6s 3ms/step - loss: 0.7663 - sparse_categorical_accuracy: 0.8054\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 7s 4ms/step - loss: 0.3503 - sparse_categorical_accuracy: 0.9003\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.3026 - sparse_categorical_accuracy: 0.9126\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "0.9125999808311462"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "experiment = exp1.clone(deep=True)\n",
        "assert pg.eq(experiment, exp1)\n",
        "\n",
        "experiment = pg.load('exp1.json')\n",
        "assert pg.eq(experiment, exp1)\n",
        "\n",
        "experiment.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2b9Y3gtcbwlC"
      },
      "source": [
        "\u003ca name=\"modification\"\u003e\u003c/a\u003e\n",
        "### Modification\n",
        "\n",
        "To iterate an idea, an existing ML experiment needs to be modified into new ones. With PyGlove, a symbolic object can be manipluated into another object without modifying existing code. This gives the maximum flexibility to accomondate unanticpated changes when new ideas pop up.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 20945,
          "status": "ok",
          "timestamp": 1657147666116,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "DgvRcf7_cNNH",
        "outputId": "d1c5087c-dece-4c4c-c1a6-aa070cc8e254"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Experiment(\n",
            "  model = NewModel(\n",
            "    activation = ReLU(\n",
            "      max_value = None,\n",
            "      negative_slope = 0.0,\n",
            "      threshold = 0.0\n",
            "    )\n",
            "  ),\n",
            "  training = {\n",
            "    input = mnist(\n",
            "      training = True\n",
            "    ),\n",
            "    batch_size = 256,\n",
            "    num_epochs = 2,\n",
            "    loss = SparseCategoricalCrossentropy(\n",
            "      from_logits = False,\n",
            "      reduction = 'auto',\n",
            "      name = 'sparse_categorical_crossentropy'\n",
            "    ),\n",
            "    optimizer = RMSprop(\n",
            "      learning_rate = 0.01,\n",
            "      rho = 0.9,\n",
            "      momentum = 0.0,\n",
            "      epsilon = 1e-07,\n",
            "      centered = False,\n",
            "      name = 'RMSprop'\n",
            "    )\n",
            "  },\n",
            "  evaluation = {\n",
            "    input = mnist(\n",
            "      training = False\n",
            "    ),\n",
            "    batch_size = 32,\n",
            "    metric = SparseCategoricalAccuracy(\n",
            "      name = 'sparse_categorical_accuracy',\n",
            "      dtype = None\n",
            "    )\n",
            "  }\n",
            ")\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 10s 5ms/step - loss: 0.3879 - sparse_categorical_accuracy: 0.8869\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 9s 5ms/step - loss: 0.1961 - sparse_categorical_accuracy: 0.9427\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.1477 - sparse_categorical_accuracy: 0.9580\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "0.9580000042915344"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "@pg.symbolize\n",
        "class NewModel(tf.keras.Model):\n",
        "\n",
        "  def __init__(self, activation):\n",
        "    super().__init__()\n",
        "    self._dense1 = Dense(1024, activation=activation)\n",
        "    self._dense2 = Dense(10, activation='softmax')\n",
        "  \n",
        "  def call(self, inputs):\n",
        "    x = Flatten()(inputs)\n",
        "    x = self._dense1(x)\n",
        "    x = Dropout(0.2)(x)\n",
        "    x = self._dense2(x)\n",
        "    return x\n",
        "\n",
        "ReLU = pg.symbolize(tf.keras.layers.ReLU)\n",
        "RMSProp = pg.symbolize(tf.keras.optimizers.RMSprop)\n",
        "\n",
        "experiment.rebind({\n",
        "  'model': NewModel(activation=ReLU()),\n",
        "  'training.batch_size': 256,\n",
        "  'training.optimizer': RMSProp(0.01)\n",
        "})\n",
        "print(experiment)\n",
        "experiment.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Qs2Bw69fFa7"
      },
      "source": [
        "\u003ca name=\"tuning\"\u003e\u003c/a\u003e\n",
        "### Tuning\n",
        "\n",
        "Once an idea works, ML practitioners can squeeze out the performance by tuning its hyperparameters. With PyGlove, ML practitioners can tune any part of the experiment with ease. Here we do a hyperparameter sweep on both learning rate for the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 47911,
          "status": "ok",
          "timestamp": 1657147714150,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "0z9K1NThfUSA",
        "outputId": "76ea004a-074d-4e2d-fe9b-0b53b18bd620"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Trial 1: DNA(0)\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 11s 5ms/step - loss: 0.3913 - sparse_categorical_accuracy: 0.8812\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 11s 6ms/step - loss: 0.1976 - sparse_categorical_accuracy: 0.9421\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.1606 - sparse_categorical_accuracy: 0.9527\n",
            "Trial 2: DNA(1)\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 11s 5ms/step - loss: 0.5973 - sparse_categorical_accuracy: 0.8466\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 11s 6ms/step - loss: 0.4303 - sparse_categorical_accuracy: 0.8979\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.3198 - sparse_categorical_accuracy: 0.9268\n"
          ]
        }
      ],
      "source": [
        "ssd = experiment.clone(deep=True).rebind({\n",
        "    'training.optimizer.learning_rate': \n",
        "pg.oneof([1e-2, 1e-1]),\n",
        "})\n",
        "\n",
        "for exp, feedback in pg.sample(ssd, pg.geno.Sweeping()):\n",
        "  print(f'Trial {feedback.id}: {feedback.dna}')\n",
        "  accuracy = exp.run()\n",
        "  feedback(accuracy)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3xfoI6JAoRxv"
      },
      "source": [
        "\u003ca name=\"release\"\u003e\u003c/a\u003e\n",
        "### Release\n",
        "\n",
        "Release is simply to save the experiment for future reproduction, with making all new components available to others."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-U98fTjTqPiI"
      },
      "outputs": [],
      "source": [
        "experiment.save('exp2.json')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rgx-ZtuMq2mU"
      },
      "source": [
        "\u003ca name=\"symbolic-features\"\u003e\u003c/a\u003e\n",
        "## The Power of Symbolic Machine Learning\n",
        "\n",
        "The power of symbolic ML is revealed during the whole process of experimentation, from boosting the productivity of developement and iterations, to implementing ideas with new ways of programming, and applying AutoML."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FTDte46LtNgv"
      },
      "source": [
        "\u003ca name=\"productivity-features\"\u003e\u003c/a\u003e\n",
        "### Experimenting with More Productivity\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FNHT22vpq8db"
      },
      "source": [
        "#### Rich formatting in human-readable form"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 4,
          "status": "ok",
          "timestamp": 1657147714509,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "lqwIVkprrO8p",
        "outputId": "ae1ad16e-6fc4-4b4d-9c76-55e7d29f3a84"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Experiment(\n",
            "  model = NewModel(\n",
            "    activation = ReLU(\n",
            "      max_value = None,\n",
            "      negative_slope = 0.0,\n",
            "      threshold = 0.0\n",
            "    )\n",
            "  ),\n",
            "  training = {\n",
            "    input = mnist(\n",
            "      training = True\n",
            "    ),\n",
            "    batch_size = 256,\n",
            "    num_epochs = 2,\n",
            "    loss = SparseCategoricalCrossentropy(\n",
            "      from_logits = False,\n",
            "      reduction = 'auto',\n",
            "      name = 'sparse_categorical_crossentropy'\n",
            "    ),\n",
            "    optimizer = RMSprop(\n",
            "      learning_rate = 0.01,\n",
            "      rho = 0.9,\n",
            "      momentum = 0.0,\n",
            "      epsilon = 1e-07,\n",
            "      centered = False,\n",
            "      name = 'RMSprop'\n",
            "    )\n",
            "  },\n",
            "  evaluation = {\n",
            "    input = mnist(\n",
            "      training = False\n",
            "    ),\n",
            "    batch_size = 32,\n",
            "    metric = SparseCategoricalAccuracy(\n",
            "      name = 'sparse_categorical_accuracy',\n",
            "      dtype = None\n",
            "    )\n",
            "  }\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "print(experiment)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 61,
          "status": "ok",
          "timestamp": 1657147714720,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "fzf5fFL2rSzZ",
        "outputId": "6982dcea-2c5c-43c2-ba5a-6f8bf826e5b8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Experiment(model=NewModel(activation=ReLU(max_value=None, negative_slope=0.0, threshold=0.0)), training={input=mnist(training=True), batch_size=256, num_epochs=2, loss=SparseCategoricalCrossentropy(from_logits=False, reduction='auto', name='sparse_categorical_crossentropy'), optimizer=RMSprop(learning_rate=0.01, rho=0.9, momentum=0.0, epsilon=1e-07, centered=False, name='RMSprop')}, evaluation={input=mnist(training=False), batch_size=32, metric=SparseCategoricalAccuracy(name='sparse_categorical_accuracy', dtype=None)})\n"
          ]
        }
      ],
      "source": [
        "# A single-line print.\n",
        "print(repr(experiment))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 61,
          "status": "ok",
          "timestamp": 1657147714919,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "5hMsWNtPrZgE",
        "outputId": "3e627f53-29df-4aed-9b82-dded8bee8816"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Experiment(\n",
            "  # Model used for both training and evaluation.\n",
            "  model = NewModel(\n",
            "    # Argument 'activation'.\n",
            "    activation = ReLU()\n",
            "  ),\n",
            "  training = {\n",
            "    # A callable that returns a tuple of (features, labels) for training.\n",
            "    input = mnist(\n",
            "      # Argument 'training'.\n",
            "      training = True\n",
            "    ),\n",
            "\n",
            "    # Training batch size.\n",
            "    batch_size = 256,\n",
            "\n",
            "    # Number of epochs for training.\n",
            "    num_epochs = 2,\n",
            "\n",
            "    # Loss used for optimization.\n",
            "    loss = SparseCategoricalCrossentropy(),\n",
            "\n",
            "    # Optimizer for minimizing the loss.\n",
            "    optimizer = RMSprop(\n",
            "      # Argument 'learning_rate'.\n",
            "      learning_rate = 0.01\n",
            "    )\n",
            "  },\n",
            "  evaluation = {\n",
            "    # A callable that returns a tuple of (feature, labels) for evaluation.\n",
            "    input = mnist(\n",
            "      # Argument 'training'.\n",
            "      training = False\n",
            "    ),\n",
            "\n",
            "    # Evaluation batch size.\n",
            "    batch_size = 32,\n",
            "\n",
            "    # Evaluation metrics.\n",
            "    metric = SparseCategoricalAccuracy()\n",
            "  }\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "# Print with docstr, and hide the default values.\n",
        "print(pg.format(experiment, verbose=True, hide_default_values=True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g24ROBH8sdrp"
      },
      "source": [
        "#### Retrieving the schema of hyper-parameters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 57,
          "status": "ok",
          "timestamp": 1657147715098,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "pkKbFjA5s9QV",
        "outputId": "14545c37-29a3-4a79-ac2e-815e0a72da3c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Schema(\n",
            "  # Model used for both training and evaluation.\n",
            "  model = Object(Model),\n",
            "\n",
            "  training = Dict({\n",
            "    # A callable that returns a tuple of (features, labels) for training.\n",
            "    input = Callable(),\n",
            "\n",
            "    # Training batch size.\n",
            "    batch_size = Int(min=1),\n",
            "\n",
            "    # Number of epochs for training.\n",
            "    num_epochs = Int(min=1),\n",
            "\n",
            "    # Loss used for optimization.\n",
            "    loss = Object(Loss),\n",
            "\n",
            "    # Optimizer for minimizing the loss.\n",
            "    optimizer = Object(OptimizerV2)\n",
            "  }),\n",
            "  evaluation = Dict({\n",
            "    # A callable that returns a tuple of (feature, labels) for evaluation.\n",
            "    input = Callable(),\n",
            "\n",
            "    # Evaluation batch size.\n",
            "    batch_size = Int(min=1),\n",
            "\n",
            "    # Evaluation metrics.\n",
            "    metric = Object(Metric)\n",
            "  })\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "print(Experiment.__schema__)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xMkJuOw4y86r"
      },
      "source": [
        "#### Catching bad experiment specifications"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 62,
          "status": "ok",
          "timestamp": 1657147715325,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "9BVuk3RZzA9n",
        "outputId": "56c67233-5f44-4904-b843-0e6189b3fb6f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Value 0 is out of range (min=1, max=None). (path=training.batch_size)\n"
          ]
        }
      ],
      "source": [
        "try:\n",
        "  experiment.rebind({\n",
        "      # Misput batch size.\n",
        "      'training.batch_size': 0\n",
        "  })\n",
        "except ValueError as e:\n",
        "  print(e)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vacGyGiIv5w6"
      },
      "source": [
        "#### Showing the differences between two experiments"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 61,
          "status": "ok",
          "timestamp": 1657147715506,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "_7lCGBZUwBmX",
        "outputId": "3ebf2c3f-f5aa-4cc0-bf99-61da1d653684"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Experiment(\n",
            "  training = Dict(\n",
            "    batch_size = Diff(\n",
            "      left = 256,\n",
            "      right = 128\n",
            "    ),\n",
            "    optimizer = RMSprop(\n",
            "      learning_rate = Diff(\n",
            "        left = 0.01,\n",
            "        right = 0.2\n",
            "      )\n",
            "    )\n",
            "  )\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "print(pg.diff(experiment, experiment.clone(override={\n",
        "    'training.batch_size': 128,\n",
        "    'training.optimizer.learning_rate': 0.2\n",
        "})))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xCTA3jXsryS6"
      },
      "source": [
        "#### Querying parts of an experiment\n",
        "\n",
        "The ML experiment is now a symbolic tree, which can be queried by nodes' locations or values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 62,
          "status": "ok",
          "timestamp": 1657147715687,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "kwwV58WOr4qL",
        "outputId": "6c9077f5-048a-4ac1-dddf-6945abd19b66"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'evaluation.input': mnist(training=False),\n",
              " 'training.input': mnist(training=True)}"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Query by key path.\n",
        "pg.query(experiment, '.*input')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 58,
          "status": "ok",
          "timestamp": 1657147715905,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "xx8PYWWasFpe",
        "outputId": "8a957e65-1bbd-45a8-bf35-4b57af44385f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'evaluation.input.training': False,\n",
              " 'training.input.training': True,\n",
              " 'training.loss.from_logits': False,\n",
              " 'training.optimizer.centered': False}"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Query by values.\n",
        "pg.query(experiment, where=lambda v: isinstance(v, bool))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rzV-4AsduTMG"
      },
      "source": [
        "#### Meta-programming by traversing the experiment"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 60,
          "status": "ok",
          "timestamp": 1657147716092,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "yMiqLuLMuMrB",
        "outputId": "aec97107-2f3d-41a3-addf-154eea9ab2c2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "model\n",
            "model.activation\n",
            "model.activation.max_value\n",
            "model.activation.negative_slope\n",
            "model.activation.threshold\n",
            "training\n",
            "training.input\n",
            "training.input.training\n",
            "training.batch_size\n",
            "training.num_epochs\n",
            "training.loss\n",
            "training.loss.from_logits\n",
            "training.loss.reduction\n",
            "training.loss.name\n",
            "training.optimizer\n",
            "training.optimizer.learning_rate\n",
            "training.optimizer.rho\n",
            "training.optimizer.momentum\n",
            "training.optimizer.epsilon\n",
            "training.optimizer.centered\n",
            "training.optimizer.name\n",
            "evaluation\n",
            "evaluation.input\n",
            "evaluation.input.training\n",
            "evaluation.batch_size\n",
            "evaluation.metric\n",
            "evaluation.metric.name\n",
            "evaluation.metric.dtype\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "def list_keys(key, value, parent):\n",
        "  print(key)\n",
        "\n",
        "pg.traverse(experiment, list_keys)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1VH4WMMgtE_u"
      },
      "source": [
        "#### Comparing concepts\n",
        "\n",
        "When the user want to check if two components have the same symbolic representation, symbolic comparison can be applied.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 60,
          "status": "ok",
          "timestamp": 1657147716271,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "E-kXDFx0tJHr",
        "outputId": "7f492721-2865-4b91-8f65-637e374eea48"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "False\n",
            "True\n"
          ]
        }
      ],
      "source": [
        "print(pg.eq(experiment, exp1))\n",
        "print(pg.eq(experiment, experiment.clone(deep=True)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5YXm3xfwt3cr"
      },
      "source": [
        "#### Saving/Loading concepts\n",
        "\n",
        "Objects can be serialized solely based on their symbolic representation, regardless their internal states."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 57,
          "status": "ok",
          "timestamp": 1657147716503,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "Y8zcnagBt2KT",
        "outputId": "b8552fda-a48c-4ea8-89eb-91eadbdb87aa"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pg.eq(experiment, pg.from_json_str(experiment.to_json_str()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RiTYvEmRuBLQ"
      },
      "source": [
        "#### Preventing an object from further modification"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 58,
          "status": "ok",
          "timestamp": 1657147716681,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "9-E8YWenuAOV",
        "outputId": "291f8349-e8b3-4496-991f-7eee3343dd01"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Cannot rebind key 'batch_size' of sealed Dict: {input=mnist(training=True), batch_size=256, num_epochs=2, loss=SparseCategoricalCrossentropy(from_logits=False, reduction='auto', name='sparse_categorical_crossentropy'), optimizer=RMSprop(learning_rate=0.01, rho=0.9, momentum=0.0, epsilon=1e-07, centered=False, name='RMSprop')}. (path='training')\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "Experiment(model=NewModel(activation=ReLU(max_value=None, negative_slope=0.0, threshold=0.0)), training={input=mnist(training=True), batch_size=256, num_epochs=2, loss=SparseCategoricalCrossentropy(from_logits=False, reduction='auto', name='sparse_categorical_crossentropy'), optimizer=RMSprop(learning_rate=0.01, rho=0.9, momentum=0.0, epsilon=1e-07, centered=False, name='RMSprop')}, evaluation={input=mnist(training=False), batch_size=32, metric=SparseCategoricalAccuracy(name='sparse_categorical_accuracy', dtype=None)})"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "experiment.seal()\n",
        "try:\n",
        "  experiment.rebind({\n",
        "      'training.batch_size': 5\n",
        "  })\n",
        "except pg.WritePermissionError as e:\n",
        "  print(e)\n",
        "experiment.seal(False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aOSQFwfevN4N"
      },
      "source": [
        "\u003ca name=\"manipulation\"\u003e\u003c/a\u003e\n",
        "### Riding the Power of Symbolic Manipulation\n",
        "\n",
        "Symbolic manipulation allows modification of the parts of an experiment after its creation. Moreover, it provides programming interfaces to meta-program the experiment by rules or algorithms. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ikzX19tYqYM-"
      },
      "source": [
        "#### Encapsulating ideas and making them reusable\n",
        "\n",
        "For common software systems, code reuse happens at type definition level; for traditional ML systems, code reuse happens at instance level (e.g. reusing an experiment). For symbolic ML systems, code reuse can takes place at an even higher level - the procedure (meta-program) that transforms object A to object B. Essentially, such meta-programs encapsulates ideas that make ML effective, such as linear scaling of learning rate when batch size increaes, replacing ReLU layers with ELu layers, and etc."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 111,
          "status": "ok",
          "timestamp": 1657147716910,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "DJIo62usu9k3",
        "outputId": "97209002-3838-4273-eed6-bbad82937692"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Experiment(\n",
            "  training = Dict(\n",
            "    batch_size = Diff(\n",
            "      left = 256,\n",
            "      right = 1024\n",
            "    ),\n",
            "    optimizer = RMSprop(\n",
            "      learning_rate = Diff(\n",
            "        left = 0.01,\n",
            "        right = 0.04\n",
            "      )\n",
            "    )\n",
            "  )\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "@pg.patcher([\n",
        "  ('batch_size', pg.typing.Int(min_value=1))\n",
        "])\n",
        "def scale_lr_with_batch_size(exp, batch_size):\n",
        "  # A patcher returns a dict of location to updated values\n",
        "  # within the patching target.\n",
        "  return {\n",
        "      'training.batch_size': batch_size,\n",
        "      'training.optimizer.learning_rate': (\n",
        "          exp.training.optimizer.sym_init_args.learning_rate * (\n",
        "              batch_size / exp.training.batch_size))\n",
        "  }\n",
        "\n",
        "new_experiment = experiment.clone(deep=True)\n",
        "pg.patch(new_experiment, scale_lr_with_batch_size(batch_size=1024))\n",
        "print(pg.diff(experiment, new_experiment))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B-25HIGP0knV"
      },
      "source": [
        "Patcher can also by defined as transforms that is based on patterns in the symbolic tree. For example, swapping all ReLU activations to ELU can be expressed as"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 134,
          "status": "ok",
          "timestamp": 1657147717165,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "SBHTvoaO1E_6",
        "outputId": "403e8e4f-8252-4975-abd5-e225f58a2a1a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Experiment(\n",
            "  model = NewModel(\n",
            "    activation = Diff(\n",
            "      left = ReLU(\n",
            "        max_value = None,\n",
            "        negative_slope = 0.0,\n",
            "        threshold = 0.0\n",
            "      ),\n",
            "      right = ELU(\n",
            "        alpha = 1.0\n",
            "      )\n",
            "    )\n",
            "  )\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "ELU = pg.symbolize(tf.keras.layers.ELU)\n",
        "\n",
        "@pg.patcher()\n",
        "def relu_to_elu(unused_exp):\n",
        "  def _swap_activation(k, v, p):\n",
        "    \"\"\"Transform fn for each node in the symbolic tree in bottom-up a manner.\n",
        "\n",
        "    Args:\n",
        "      k: a `pg.KeyPath` object representing the location of current node.\n",
        "      v: the value of current node.\n",
        "      p: the parent node.\n",
        "\n",
        "    Returns:\n",
        "      Transformed value.\n",
        "    \"\"\"\n",
        "    if isinstance(v, ReLU):\n",
        "      return ELU()\n",
        "    return v\n",
        "  return _swap_activation\n",
        "\n",
        "new_experiment = experiment.clone(deep=True)\n",
        "pg.patch(new_experiment, relu_to_elu())\n",
        "print(pg.diff(experiment, new_experiment))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xHeC3T2TrNqC"
      },
      "source": [
        "#### Combining ideas\n",
        "\n",
        "Ideas can be easily combined by chaining the patchers together. For example, we can apply both techiniques introduced above to a new experiment by:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 79,
          "status": "ok",
          "timestamp": 1657147717401,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "Ws9zvea1zqCI",
        "outputId": "71d56a83-3006-416c-93b2-868dfcfe1747"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Experiment(\n",
            "  model = NewModel(\n",
            "    activation = Diff(\n",
            "      left = ReLU(\n",
            "        max_value = None,\n",
            "        negative_slope = 0.0,\n",
            "        threshold = 0.0\n",
            "      ),\n",
            "      right = ELU(\n",
            "        alpha = 1.0\n",
            "      )\n",
            "    )\n",
            "  ),\n",
            "  training = Dict(\n",
            "    batch_size = Diff(\n",
            "      left = 256,\n",
            "      right = 1024\n",
            "    ),\n",
            "    optimizer = RMSprop(\n",
            "      learning_rate = Diff(\n",
            "        left = 0.01,\n",
            "        right = 0.04\n",
            "      )\n",
            "    )\n",
            "  )\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "new_experiment = experiment.clone(deep=True)\n",
        "# Combining ideas by chaining the patchers.\n",
        "pg.patch(new_experiment, [\n",
        "    scale_lr_with_batch_size(batch_size=1024),\n",
        "    relu_to_elu(),\n",
        "])\n",
        "print(pg.diff(experiment, new_experiment))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NcoXp9jU3kOc"
      },
      "source": [
        "#### Patching experiment from the command line\n",
        "\n",
        "Though it's easy to patch experiments via Python code. It would be better if ML practitioners can patch experiments from the command line, which requires no rebuild/repackage of the source code.\n",
        "\n",
        "To address this requirement, patcher can also be invoked as a URL-like string. Therefore, the strings can be passed in as command-line arguments."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 115,
          "status": "ok",
          "timestamp": 1657147717665,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "IHzc-iuu4IYP",
        "outputId": "a4801a12-5f1d-4a36-fc7d-0f3ba2080d82"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Experiment(\n",
            "  model = NewModel(\n",
            "    activation = Diff(\n",
            "      left = ReLU(\n",
            "        max_value = None,\n",
            "        negative_slope = 0.0,\n",
            "        threshold = 0.0\n",
            "      ),\n",
            "      right = ELU(\n",
            "        alpha = 1.0\n",
            "      )\n",
            "    )\n",
            "  ),\n",
            "  training = Dict(\n",
            "    batch_size = Diff(\n",
            "      left = 256,\n",
            "      right = 1024\n",
            "    ),\n",
            "    optimizer = RMSprop(\n",
            "      learning_rate = Diff(\n",
            "        left = 0.01,\n",
            "        right = 0.04\n",
            "      )\n",
            "    )\n",
            "  )\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "print(pg.diff(experiment, pg.patch(experiment.clone(deep=True), [\n",
        "    'scale_lr_with_batch_size?batch_size=1024',\n",
        "    'relu_to_elu'\n",
        "])))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pVfgRRRUqmhW"
      },
      "source": [
        "#### Summary: A better expression of ML experiment\n",
        "\n",
        "Putting things together, a new experiment can be expressed as an existing experiment applying new ML ideas, with each patcher represents an ML idea. Such expression is more high-level than hyperparameter-level differences, and better reasons their differencs. Moreover, the ML ideas can be applied to other experiments too.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wihLBNPlqWzf"
      },
      "source": [
        "See also:\n",
        "\n",
        "* [Symbolic Neural Modeling](https://github.com/google/pyglove/blob/main/docs/notebooks/ml/neural_modeling.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kzAjT-TPvY8p"
      },
      "source": [
        "\u003ca name=\"automl\"\u003e\u003c/a\u003e\n",
        "### Embracing AutoML as a Part of the ML Experimentation Process\n",
        "\n",
        "Machine learning is a process of trials and errors. Today, most of such trials are manually done with human-conducted evolution, a repeated and laboring process. On the other hand, automatic hyperparameter tuning is not a stranger, it's often supported for large scale ML systems. More complex automatic explorations, like neural architecture search, however, are still considered advanced technologies that most ML practioners do not have access to. \n",
        "\n",
        "AutoML should be an integral part of ML experimentation process, and it should be straightforward for every ML practioner. PyGlove enables this with a few lines of code changes. This means, an arbitrary part of an ML experiment can be added to the search space easily, and to be optimized by state-of-the-art search algorithms.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6kdUUWq0UCHO"
      },
      "source": [
        "#### Defining what to explore\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 57,
          "status": "ok",
          "timestamp": 1657147717849,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "zrcQe0DxVQLB",
        "outputId": "411ede09-8ce9-4e64-fc15-0511c0535c78"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Experiment(\n",
            "  model = NewModel(\n",
            "    activation = ReLU(\n",
            "      max_value = None,\n",
            "      negative_slope = 0.0,\n",
            "      threshold = 0.0\n",
            "    )\n",
            "  ),\n",
            "  training = {\n",
            "    input = mnist(\n",
            "      training = True\n",
            "    ),\n",
            "    batch_size = 256,\n",
            "    num_epochs = 2,\n",
            "    loss = SparseCategoricalCrossentropy(\n",
            "      from_logits = False,\n",
            "      reduction = 'auto',\n",
            "      name = 'sparse_categorical_crossentropy'\n",
            "    ),\n",
            "    optimizer = RMSprop(\n",
            "      learning_rate = 0.01,\n",
            "      rho = 0.9,\n",
            "      momentum = 0.0,\n",
            "      epsilon = 1e-07,\n",
            "      centered = False,\n",
            "      name = 'RMSprop'\n",
            "    )\n",
            "  },\n",
            "  evaluation = {\n",
            "    input = mnist(\n",
            "      training = False\n",
            "    ),\n",
            "    batch_size = 32,\n",
            "    metric = SparseCategoricalAccuracy(\n",
            "      name = 'sparse_categorical_accuracy',\n",
            "      dtype = None\n",
            "    )\n",
            "  }\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "# Let's take another look of the experiment.\n",
        "print(experiment)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "19iWbtSuV3EN"
      },
      "source": [
        "Now assume that we have three ideas:\n",
        "* Try different activations.\n",
        "* Try different optimizers.\n",
        "* Try different batch size.\n",
        "\n",
        "We can make a search space with jointly optimizing these three apsects:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 117,
          "status": "ok",
          "timestamp": 1657147718106,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "JRv-sjNCWD9V",
        "outputId": "f2a1af0d-0d08-4b70-d79b-ce7e776b0391"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Space({\n",
            "  0 = 'model.activation': Choices(num_choices=1, [\n",
            "    (0): ReLU()\n",
            "    (1): ELU()\n",
            "    (2): PReLU()\n",
            "  ])\n",
            "  1 = 'training.batch_size': Choices(num_choices=1, [\n",
            "    (0): 32\n",
            "    (1): 64\n",
            "    (2): 128\n",
            "  ])\n",
            "  2 = 'training.optimizer': Choices(num_choices=1, [\n",
            "    (0): RMSprop()\n",
            "    (1): SGD(learning_rate=0.001)\n",
            "    (2): Adam()\n",
            "  ])\n",
            "})\n"
          ]
        }
      ],
      "source": [
        "SGD = pg.symbolize(tf.keras.optimizers.SGD)\n",
        "Adam = pg.symbolize(tf.keras.optimizers.Adam)\n",
        "PReLU = pg.symbolize(tf.keras.layers.PReLU)\n",
        "\n",
        "learning_rate = 0.001\n",
        "\n",
        "experiment_space = experiment.clone(override={\n",
        "    'model.activation': pg.oneof([ReLU(), ELU(), PReLU()]),\n",
        "    'training.optimizer': pg.oneof([\n",
        "        RMSProp(learning_rate=learning_rate),\n",
        "        SGD(learning_rate=learning_rate),\n",
        "        Adam(learning_rate=learning_rate),\n",
        "    ]),\n",
        "    'training.batch_size': pg.oneof([32, 64, 128])\n",
        "})\n",
        "\n",
        "# Let's inspect the decision points from this space:\n",
        "print(pg.dna_spec(experiment_space))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 106,
          "status": "ok",
          "timestamp": 1657147718330,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "VF-rRNOOZnVo",
        "outputId": "961e281b-2849-4867-df48-cb662c27bef3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "27\n"
          ]
        }
      ],
      "source": [
        "# Check the size of the space.\n",
        "print(pg.dna_spec(experiment_space).space_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EDly0Z64UVAz"
      },
      "source": [
        "#### Automatic exploration with regularized evolution\n",
        "\n",
        "With an experiment space, we can specify how to optimize based on:\n",
        "\n",
        "* Search algorithm: we use regularized evolution here.\n",
        "* Reward function: we use accuracy as the reward."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "executionInfo": {
          "elapsed": 430113,
          "status": "ok",
          "timestamp": 1657148148567,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "uY99AI_XXADH",
        "outputId": "24eeb28f-5989-45af-e7a3-c909f46bd889"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Trial 1: DNA([2, 0, 0])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 11s 6ms/step - loss: 0.7484 - sparse_categorical_accuracy: 0.8067\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 12s 6ms/step - loss: 0.3284 - sparse_categorical_accuracy: 0.9041\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2791 - sparse_categorical_accuracy: 0.9189\n",
            "Trial 2: DNA([0, 2, 1])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 7s 4ms/step - loss: 2.3005 - sparse_categorical_accuracy: 0.1387\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 7s 4ms/step - loss: 2.2995 - sparse_categorical_accuracy: 0.1124\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 2.2991 - sparse_categorical_accuracy: 0.1135\n",
            "Trial 3: DNA([2, 1, 0])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 12s 6ms/step - loss: 0.7542 - sparse_categorical_accuracy: 0.8035\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 11s 6ms/step - loss: 0.3277 - sparse_categorical_accuracy: 0.9045\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2805 - sparse_categorical_accuracy: 0.9177\n",
            "Trial 4: DNA([0, 2, 2])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 8s 4ms/step - loss: 0.7650 - sparse_categorical_accuracy: 0.8047\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 7s 4ms/step - loss: 0.3493 - sparse_categorical_accuracy: 0.9004\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.3022 - sparse_categorical_accuracy: 0.9116\n",
            "Trial 5: DNA([1, 0, 2])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 8s 4ms/step - loss: 0.6191 - sparse_categorical_accuracy: 0.8357\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 8s 4ms/step - loss: 0.3196 - sparse_categorical_accuracy: 0.9076\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2874 - sparse_categorical_accuracy: 0.9167\n",
            "Trial 6: DNA([2, 0, 0])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 13s 6ms/step - loss: 0.7411 - sparse_categorical_accuracy: 0.8104\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 11s 6ms/step - loss: 0.3298 - sparse_categorical_accuracy: 0.9043\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2834 - sparse_categorical_accuracy: 0.9178\n",
            "Trial 7: DNA([2, 2, 0])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 11s 6ms/step - loss: 0.7477 - sparse_categorical_accuracy: 0.8055\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 11s 6ms/step - loss: 0.3297 - sparse_categorical_accuracy: 0.9037\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2842 - sparse_categorical_accuracy: 0.9167\n",
            "Trial 8: DNA([2, 1, 0])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 12s 6ms/step - loss: 0.7410 - sparse_categorical_accuracy: 0.8085\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 11s 6ms/step - loss: 0.3315 - sparse_categorical_accuracy: 0.9035\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2880 - sparse_categorical_accuracy: 0.9163\n",
            "Trial 9: DNA([2, 2, 0])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 12s 6ms/step - loss: 0.7410 - sparse_categorical_accuracy: 0.8096\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 11s 6ms/step - loss: 0.3306 - sparse_categorical_accuracy: 0.9041\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2832 - sparse_categorical_accuracy: 0.9203\n",
            "Trial 10: DNA([1, 0, 2])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 9s 4ms/step - loss: 0.6182 - sparse_categorical_accuracy: 0.8333\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 8s 4ms/step - loss: 0.3186 - sparse_categorical_accuracy: 0.9082\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2917 - sparse_categorical_accuracy: 0.9180\n",
            "Trial 11: DNA([2, 2, 0])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 12s 6ms/step - loss: 0.7430 - sparse_categorical_accuracy: 0.8098\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 11s 6ms/step - loss: 0.3287 - sparse_categorical_accuracy: 0.9062\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2835 - sparse_categorical_accuracy: 0.9206\n",
            "Trial 12: DNA([0, 0, 2])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 8s 4ms/step - loss: 0.7525 - sparse_categorical_accuracy: 0.8066\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 7s 4ms/step - loss: 0.3468 - sparse_categorical_accuracy: 0.9012\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.3011 - sparse_categorical_accuracy: 0.9113\n",
            "Trial 13: DNA([2, 2, 2])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 9s 5ms/step - loss: 0.6936 - sparse_categorical_accuracy: 0.8222\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 9s 5ms/step - loss: 0.3054 - sparse_categorical_accuracy: 0.9114\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2553 - sparse_categorical_accuracy: 0.9276\n",
            "Trial 14: DNA([1, 2, 0])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 11s 6ms/step - loss: 0.6475 - sparse_categorical_accuracy: 0.8315\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 10s 5ms/step - loss: 0.3251 - sparse_categorical_accuracy: 0.9057\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2974 - sparse_categorical_accuracy: 0.9146\n",
            "Trial 15: DNA([1, 2, 2])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 8s 4ms/step - loss: 0.6198 - sparse_categorical_accuracy: 0.8307\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 8s 4ms/step - loss: 0.3198 - sparse_categorical_accuracy: 0.9072\n",
            "313/313 [==============================] - 2s 3ms/step - loss: 0.2983 - sparse_categorical_accuracy: 0.9145\n",
            "Trial 16: DNA([2, 2, 2])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 8s 4ms/step - loss: 0.6880 - sparse_categorical_accuracy: 0.8197\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 8s 4ms/step - loss: 0.3033 - sparse_categorical_accuracy: 0.9119\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2539 - sparse_categorical_accuracy: 0.9267\n",
            "Trial 17: DNA([1, 2, 2])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 8s 4ms/step - loss: 0.6214 - sparse_categorical_accuracy: 0.8320\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 7s 4ms/step - loss: 0.3190 - sparse_categorical_accuracy: 0.9081\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.2949 - sparse_categorical_accuracy: 0.9149\n",
            "Trial 18: DNA([1, 0, 2])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 7s 4ms/step - loss: 0.6205 - sparse_categorical_accuracy: 0.8348\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 7s 4ms/step - loss: 0.3186 - sparse_categorical_accuracy: 0.9072\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2971 - sparse_categorical_accuracy: 0.9128\n",
            "Trial 19: DNA([2, 1, 2])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 9s 4ms/step - loss: 0.6899 - sparse_categorical_accuracy: 0.8204\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 8s 4ms/step - loss: 0.3035 - sparse_categorical_accuracy: 0.9126\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2543 - sparse_categorical_accuracy: 0.9272\n",
            "Trial 20: DNA([2, 1, 2])\n",
            "Epoch 1/2\n",
            "1875/1875 [==============================] - 9s 5ms/step - loss: 0.6906 - sparse_categorical_accuracy: 0.8208\n",
            "Epoch 2/2\n",
            "1875/1875 [==============================] - 9s 5ms/step - loss: 0.3051 - sparse_categorical_accuracy: 0.9125\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.2538 - sparse_categorical_accuracy: 0.9269\n",
            "Best acurracy: 0.9276000261306763\n",
            "Best experiment: Experiment(\n",
            "  model = NewModel(\n",
            "    activation = PReLU(\n",
            "      alpha_initializer = 'zeros',\n",
            "      alpha_regularizer = None,\n",
            "      alpha_constraint = None,\n",
            "      shared_axes = None\n",
            "    )\n",
            "  ),\n",
            "  training = {\n",
            "    input = mnist(\n",
            "      training = True\n",
            "    ),\n",
            "    batch_size = 128,\n",
            "    num_epochs = 2,\n",
            "    loss = SparseCategoricalCrossentropy(\n",
            "      from_logits = False,\n",
            "      reduction = 'auto',\n",
            "      name = 'sparse_categorical_crossentropy'\n",
            "    ),\n",
            "    optimizer = Adam(\n",
            "      learning_rate = 0.001,\n",
            "      beta_1 = 0.9,\n",
            "      beta_2 = 0.999,\n",
            "      epsilon = 1e-07,\n",
            "      amsgrad = False,\n",
            "      name = 'Adam'\n",
            "    )\n",
            "  },\n",
            "  evaluation = {\n",
            "    input = mnist(\n",
            "      training = False\n",
            "    ),\n",
            "    batch_size = 32,\n",
            "    metric = SparseCategoricalAccuracy(\n",
            "      name = 'sparse_categorical_accuracy',\n",
            "      dtype = None\n",
            "    )\n",
            "  }\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "best_exp, best_accuracy = None, None\n",
        "history = []\n",
        "for exp, feedback in pg.sample(experiment_space,\n",
        "                               pg.evolution.regularized_evolution(\n",
        "                                   population_size=5, tournament_size=3),\n",
        "                               num_examples=20):\n",
        "  print(f'Trial {feedback.id}: {feedback.dna}')\n",
        "  accuracy = exp.run()\n",
        "  if best_accuracy is None or best_accuracy \u003c accuracy:\n",
        "    best_accuracy, best_exp = accuracy, exp\n",
        "  history.append(accuracy)\n",
        "  feedback(accuracy)\n",
        "\n",
        "print(f'Best acurracy: {best_accuracy}')\n",
        "print(f'Best experiment: {best_exp}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "height": 265
        },
        "executionInfo": {
          "elapsed": 321,
          "status": "ok",
          "timestamp": 1657148149016,
          "user": {
            "displayName": "",
            "userId": ""
          },
          "user_tz": 420
        },
        "id": "Ga8u16b2aKAn",
        "outputId": "2aa0a6c2-85c0-4c9d-9bec-029a05f729ea"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90\nbGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsT\nAAALEwEAmpwYAAAiI0lEQVR4nO3da4xc533f8e9/bnuZ3SV3l7elJJpSzOhiGZbUja3ETpBGoiEz\nqan2hSGjadg0gGAgaeOiRatAgJG+KZS2CZIWQQI2ccM0hmMksSMhZVJLjFMjaCWbulimTMmUZV25\nIlezS+7uLHdnZ+bfF3Nmdjia2duZ25n5fYDBnDPn4ZyHh8PfPPPMOf8xd0dERHpfrNMdEBGR9lDg\ni4j0CQW+iEifUOCLiPQJBb6ISJ9IdLoDG9mzZ48fPny4090QEYmMZ5999j1331tvW1cH/uHDhzl7\n9mynuyEiEhlm9kajbZrSERHpEwp8EZE+ocAXEekTCnwRkT6hwBcR6RMKfBGRPqHAFxHpE119Hr6I\nbJ2784PZLN/64Ry5fIE7b9jF7VNjpAei8998NV/gwqUlVvNFfnT/CKODyU53aVvmszleeOsKb1+5\nhgFmYFhwD7FgobTNSo/F1tsQPD6YiPHJDx1oev+i80oQkeu4OxcuL/HMaxmefm2OZ344x3tLq9e1\nMYNb9qT58A27uDO43XFwjLEuCNL3llY5P7PA9y4ucH5mgfMzi7w6u0ShuP4bHTeOD3HbgTFunxrl\ntgNj3HpglJv3pInHrIM9L1krFHl5ZpEX3prn+Tev8PxbV/jhe9mmPPeekQEF/la4O3/27Nt8cN8I\n9xwa70gfVvMF8gXv6pFVsejki06h6OSLxeDeyRdK62ZGImbEY0YyFiMeX19PxAyz9v2Hc3eurRVY\nuJZncWWNhZU1FlbyLFwr3S+urFVtK90XHQYTMQaTcQaTpfuByvr1y4PJGIOJ+HVt3eHaWqF0yxVY\nqVq+tlZaXy4vB/fVbYdScQ5PpvnAZJqb9wzzgck0H5gcZji189dEsei8cmmRp1/L8Mxrc3zr9Tnm\nsjkApnYN8okPTnLvLZN87JZJhpJxXrp4le++c5Vz7yzw9Gtz/OULFyvPdXhyuPIG8OEbdvGhg2Ps\nHk6F/reqJ18o8nomy0sXS6F+fmaB780sMLu4/uZ0cNcgt0+NcfSO/dxxcIxUPMYrlxZ5+d1FXp5Z\n4BuvXK68EQwkYvzo/lFuOzDKrQdGuX1qjNsOjDI5MtCS/pe9e3WF59+c5/m3rvD8m/O8+PZVVvNF\nAPaODnDPod18Zvom7j60m1v2pMHAPbjhwX3p3xGuf7zojgePgZc+CbSAdfMvXk1PT/tOSivc8YW/\n4aEfO8QX/tEdLejV5o7+1v/hwuUlhlNx9o4OsGdkgL0jA+vLo+XlVOWxwWR8w+d0dxZW8sxnc8wt\n55jP5phfXqtZzzGfXWNuOcfKWiEIb6dQLFbCvBzwxZD/7PGq8E/EjEQ8tr4eN2JWulU+1poRMyov\n5JiVPsKW78vbyx97V/KF60K8sEmHk3FjbDDJ2FCS0cEEZsZqEMwra0VW8gVWg/tmvORjBsOpBIPJ\nOEOpGEPJOEOVN5A4iytrvJFZJhMEctm+0YHgjWCYw3uC+2C9dvqiUHTOzyyUAv6Hc3z79TmuLK8B\ncMPuoSDcJ7j35klumhja9E14dnGVcxev8tI7628E71y5Vtl+08QQdx4svQncsHuI2qcrP79d91hw\nHzxaXp9dXK0E+yvvLlaCMRk3juwrhfQdB0sj99sPjDGe3vjNZmWtwKuXlypvAK9cWuT8zOJ1n2j2\njg5w24HSG8FNE8MMJuIMJGMMVO6D5cT6AKCyPVHaXv47rqwV+O47V3n+zXleeOsKz795hZmrKwCk\n4jHuvGGMuw+Nc/eh3dx9aJyDuwbbOgjaiJk96+7Tdbf1YuB/4jf+lukPjPPbD93dgl5trFh0Pvjo\naT528yR3HBxjdnGV95ZWmV1cZXZptfIfttboYOK6N4dC0a8P8uW1hqGXiBnj6RQTwynG00nGh1MM\npeLBiDxGMr4ezltZTwQfl/PBm8P6G4WTLxSrPhk0Xi967chlfb1YHvUE26rbQinoBpNxRgcTQYgn\nGB1MMjZYCvNyqI8NJhkL1qv/s27E3ckViqysFYM3hNKbQOWNIXiTiJkxlCqF91AyzlAqvh7qqRip\n+Nb2t7CyxpuZZV7PZHkjs8zr7wX3mSyXF6+fftkzkqp8Eri6vMa3Xp9jcSUPwKGJYT5280Ql5G8c\nH95031sxn81x7mIp/M+9c5VzF6/yRma5Kc89kU5x+9Qod0yNcXtw+5G9I6QSzTtXZHZxlVfeXeTl\nd0ufHl65tMD3Ly2RC95gtqsc/Mu5Avng/9uhiWHuPrSbu24qhfvtU6MMJDYeoHXSRoHflDkHM3sA\n+B0gDvyBuz9Ws92C7ceAZeCfu/tzzdh3PZPp1PtGVu1y5VppOuGTH9rPL3785vdtz+WLZLKrvLeY\nY3ZpJXhDyJXeEII3hfMzCyTixvhwig/uG2H3cIqJIMgn0inGh1PXBfzIQKJrRhfdzsyCEV0chlo/\njz02mKxMndTKruZ5I7PMG5ksr1fus/zfVzMMp+L87IenKgE/tWuoJf0bT6f4ySN7+ckj68UVr15b\nIxOMnMtDjPVx4fqgo/zY+9vA7uEk+0YHWv66LH9a/sSRPZXH8oUic8s5cvkiq/kiq2tFVvOlN/TV\nfKH0WL70hl9Zrt6+VmRkIMFdN+3mrkO72dPiqaJ2Ch34ZhYHfhc4CrwNfNvMnnD371U1+xRwJLh9\nDPi94L4lJtKp942e2mUuu1rpQz2pRIypXUPBf+D3h4D0j/RAgjsOlqY2usmuoSS72vBm2CqJeIx9\no4Od7kZXasZnq48Cr7r7a+6eA/4UOF7T5jjwx17yNLDbzKaasO+6JtIDlS+z2i2zVNrvZLp3RgUi\n0huaEfg3AG9Vrb8dPLbdNgCY2cNmdtbMzs7Ozu6oQ5MjpSmdTnw/UX6jaTTCFxHplGYEfr1Jutqk\n3Uqb0oPuJ9192t2n9+6t+6Mtm5pIp8jli2RzhR39+TDK3x1MjijwRaS7NCPw3wZuqlq/Ebi4gzZN\nUx5dzy21f1qnPMIfb9E5zSIiO9WMwP82cMTMbjazFPAQ8ERNmyeAX7CSe4Gr7j7ThH3XNRkEfibb\n/i9u57I5RgcTTT31TESkGUKfpePueTP7FeB/Uzot84vu/pKZfS7Y/vvAaUqnZL5K6bTMXwy7341U\nRvgd+OI2k81V3nBERLpJU87Dd/fTlEK9+rHfr1p24Jebsa+tKJ8h04lz8eeyq/rCVkS6Uk/OO0yM\ndHCEv5RjQqdkikgX6snAT6fipBKxjgT+nKZ0RKRL9WTgm1mpvEKbz9Jxd+aXc5VPGCIi3aQnAx9K\nX9zOtfksnYWVPGsF1whfRLpSbwd+g8qUraKrbEWkm/Vs4E92YIS/WeE0EZFO6tnAn0gPtP1KWxVO\nE5Fu1rOBPzmSIhv83Fy7VKZ09KWtiHShng38TlxtWymcpikdEelCCvwmmsvmGA5+Fk9EpNv0bOCv\nF1Brb+DrC1sR6VY9G/jrI/z2namjwmki0s16NvArBdTaeKaOCqeJSDfr2cAfG0qQiFl75/BVOE1E\nuljPBr6ZMZ5OtS3w3b00paNTMkWkS4UKfDObMLMnzexCcD/eoN0XzeyymZ0Ls7/tmkyn2val7XKu\nwGq+qCkdEelaYUf4jwBn3P0IcCZYr+ePgAdC7mvbJto4wlcdHRHpdmED/zhwKlg+BTxYr5G7fxOY\nC7mvbWtn4OuiKxHpdmEDf3/5x8iD+31hO2RmD5vZWTM7Ozs7G+q5SjXx23NapgqniUi32/Q3bc3s\nKeBAnU2PNr874O4ngZMA09PTHua5JtIDQY36Isl4a7+fVuE0Eel2mwa+u9/faJuZXTKzKXefMbMp\n4HJTexdSuYjZfDbHvrHBlu5LhdNEpNuFHfY+AZwIlk8Aj4d8vqZqZ3mFuWyOVCJGOqU6OiLSncIG\n/mPAUTO7ABwN1jGzg2Z2utzIzL4M/D/gVjN728x+KeR+t6SdBdTKZRXMrOX7EhHZiU2ndDbi7hng\nvjqPXwSOVa1/Nsx+dqrdI3x9YSsi3axnr7QFGC+P8Ntwpk5GgS8iXa63A384hVl7pnTmsqs6B19E\nulpPB348ZuweSrZnSkeF00Sky/V04EN7rrZdWSuQzRVUOE1EulrPB/5keqDlI3zV0RGRKOj5wG/H\nCF+BLyJR0PuBP9L6wFfhNBGJgp4P/Ml0ivnlHIViqLI8G1LhNBGJgp4P/Il0Cne4sty6Ub4Kp4lI\nFPRF4ENrz8Wfy+ZIxIyxoVAXLouItFTPB3551N3KM3XmsjnGVUdHRLpczwd+eYQ/38LALxdOExHp\nZj0f+OWLoVo9wtcXtiLS7Xo+8MeH2zOHr8AXkW7X84GfSsQYHUy0NPAzSyqcJiLdL1Tgm9mEmT1p\nZheC+/E6bW4ys2+Y2Xkze8nMfjXMPndiMp1q2ZTOWqHIwkpehdNEpOuFHeE/Apxx9yPAmWC9Vh74\nN+5+O3Av8MtmdkfI/W5LqbxCa2riz+u3bEUkIsIG/nHgVLB8CniwtoG7z7j7c8HyInAeuCHkfrdl\nIj1QuTiq2VRWQUSiImzg73f3GSgFO7Bvo8Zmdhi4G3gm5H63ZbKFBdRUOE1EomLTS0PN7CngQJ1N\nj25nR2Y2AvwF8Hl3X9ig3cPAwwCHDh3azi4amhgp1dNx96ZfHKURvohExaaB7+73N9pmZpfMbMrd\nZ8xsCrjcoF2SUth/yd2/usn+TgInAaanp5tS8WwynWKt4Cys5Nk1lGzGU1aUfy9XI3wR6XZhp3Se\nAE4EyyeAx2sbWGlI/YfAeXf/rZD725FW1tOZy+Ywg93DCnwR6W5hA/8x4KiZXQCOBuuY2UEzOx20\n+Tjwz4CfMbMXgtuxkPvdlvXAb/6ZOplsjvHhFPGY6uiISHcLVd7R3TPAfXUevwgcC5b/HuhoGlYK\nqLXgTB1dZSsiUdHzV9rC+jnyrZjSySjwRSQi+iLwy2fQtOJq2zlVyhSRiOiLwB9MxhlOxVv2pa1G\n+CISBX0R+FAur9DcwC8UnflljfBFJBr6JvBbUUDtynIOd52DLyLR0DeB34oCapWyCiOqlCki3a+P\nAn+AuSaflqmyCiISJX0T+JMjpSkd96ZUawBUOE1EoqVvAn8inWI1X2Q5V2jac2qELyJR0leBD829\n+Ko8RTSuwBeRCOibwG/FxVdz2VXGBhMk431zGEUkwvomqVpRQC2TzTGpM3REJCL6LvCbWUBNV9mK\nSJT0XeA3dQ5fgS8iEdI3gT8ykCAVjzU18DMqnCYiEdI3gW9mTDSxvIK7M68RvohESKjAN7MJM3vS\nzC4E9+N12gya2bfM7Dtm9pKZ/Ycw+wyjmQXUFq7lyRddgS8ikRF2hP8IcMbdjwBngvVaq8DPuPtH\ngLuAB8zs3pD73ZHy1bbNkAnO9pkcUeCLSDSEDfzjwKlg+RTwYG0DL1kKVpPBrXn1DbZhIp1ivkmB\nv15WQadlikg0hA38/e4+AxDc76vXyMziZvYCcBl40t2fafSEZvawmZ01s7Ozs7Mhu3e9Zk7pqKyC\niETNpj9ibmZPAQfqbHp0qztx9wJwl5ntBr5mZne6+7kGbU8CJwGmp6eb+klgMp1iaTXPar7AQCIe\n6rlUOE1EombTwHf3+xttM7NLZjbl7jNmNkVpBL/Rc10xs78DHgDqBn4rladf5rI5pnYNhXouBb6I\nRE3YKZ0ngBPB8gng8doGZrY3GNljZkPA/cDLIfe7I8282jazlCOdijOYDPdJQUSkXcIG/mPAUTO7\nABwN1jGzg2Z2OmgzBXzDzF4Evk1pDv+vQu53R8pn1DRjHn8uu8qEztARkQjZdEpnI+6eAe6r8/hF\n4Fiw/CJwd5j9NEszyytksjmdoSMikdI3V9pCc0skz6msgohETF8F/thgknjMmlIiWYXTRCRq+irw\nYzFjfDj8ufjursJpIhI5fRX4UJrWCXuWTjZXIJcvaoQvIpHSd4HfjKtty79lq8AXkSjpv8AfCR/4\nKpwmIlHUd4E/2YSa+CqcJiJR1HeBP5FOcfXaGmuF4o6fQ4XTRCSK+i7wyyE9v7zzUb7q6IhIFPVd\n4FcXUNupuWyOgUSM4ZTq6IhIdPRh4AflFUKcmplZKp2Db2bN6paISMv1XeCXz6wJ88WtCqeJSBT1\nXeA3o4DanAqniUgE9V3gjw+nMAs3wldZBRGJor4L/HjM2D2UDFVATYXTRCSKQgW+mU2Y2ZNmdiG4\nH9+gbdzMnjezjvz4SbUw5RVW1gos5woKfBGJnLAj/EeAM+5+BDgTrDfyq8D5kPtrisn0wI4LqOmi\nKxGJqrCBfxw4FSyfAh6s18jMbgR+FviDkPtrijAjfBVOE5GoChv4+919BiC439eg3W8D/w7YtJ6B\nmT1sZmfN7Ozs7GzI7tUXpoCaCqeJSFRt+pu2ZvYUcKDOpke3sgMz+zngsrs/a2Y/vVl7dz8JnASY\nnp72rexjuybTKeaXcxSLTiy2vYunVDhNRKJq08B39/sbbTOzS2Y25e4zZjYFXK7T7OPAp83sGDAI\njJnZn7j7z++41yFNpFMUHa5cW9v21Izq6IhIVIWd0nkCOBEsnwAer23g7r/m7je6+2HgIeBvOxn2\nUH3x1fZPzcxkcyTjxtjgpu+VIiJdJWzgPwYcNbMLwNFgHTM7aGanw3auVSaD6ZidnKkzt5QLLt5S\nHR0RiZZQw1R3zwD31Xn8InCszuN/B/xdmH02w3g6CeysvEJGF12JSET13ZW2sD7Cn9tBTfy57KrO\n0BGRSOrLwK+M8HcypaPCaSISUX0Z+AOJOKMDiR0VUFPhNBGJqr4MfNjZxVe5fJHFlbzm8EUkkvo3\n8HdQXqH8O7gKfBGJor4N/Ml0attTOuXTODWlIyJR1LeBXxrhb+/CK11lKyJR1seBP8BcNof71sv1\nqHCaiERZ3wb+ZDrFWsFZXM1v+c+ocJqIRFnfBn6lns42zsWfy+aIGeweSraqWyIiLdO/gR9My2zn\ni9tMtlRHZ7sllUVEukHfBv5kpWLmNkb4S6qjIyLR1beBv5MSyXMqnCYiEda3gV8pkbytKR0VThOR\n6OrbwB9KxRlKxrf9pa1G+CISVaHq4ZvZBPAV4DDwOvAZd5+v0+51YBEoAHl3nw6z32bZTnmFQtGD\nn0TUKZkiEk1hR/iPAGfc/QhwJlhv5B+6+13dEvZQuoBqq1M688s53FVWQUSiK2zgHwdOBcungAdD\nPl9bbWeEr7IKIhJ1YQN/v7vPAAT3+xq0c+DrZvasmT280ROa2cNmdtbMzs7Ozobs3sa2E/gqnCYi\nUbfpHL6ZPQUcqLPp0W3s5+PuftHM9gFPmtnL7v7Neg3d/SRwEmB6enrrhW52oFQxc2unZVZG+DpL\nR0QiatPAd/f7G20zs0tmNuXuM2Y2BVxu8BwXg/vLZvY14KNA3cBvp4n0ACtrRZZzeYZTGx+K8vn6\nmtIRkagKO6XzBHAiWD4BPF7bwMzSZjZaXgY+CZwLud+mKE/PZLZwamb5y93xYQW+iERT2MB/DDhq\nZheAo8E6ZnbQzE4HbfYDf29m3wG+Bfwvd/+bkPttioltlFeYy+bYNZQkGe/bSxdEJOJCnYfv7hng\nvjqPXwSOBcuvAR8Js59WKc/HbyXw9ePlIhJ1fT1crUzpbGWEr8JpIhJxfR342ymgprIKIhJ1fR34\nIwMJUvHYlkb4mWxOhdNEJNL6OvDNrHTx1SZn6RSLzvyyRvgiEm19HfiwtattF1bWKBRdhdNEJNL6\nPvC3UkCtvF1n6YhIlPV94G9lhK/CaSLSCxT46RTzm43wlxT4IhJ9fR/4k+kUi6t5VvOFhm3KI3yd\npSMiUdb3gV/+InY+u9awjQqniUgvUOCnkwAblknOZHOMDCQYSMTb1S0RkaZT4Acj/I2+uNVVtiLS\nCxT4W6iYqcAXkV7Q94G/lZr4mSVVyhSR6Ov7wN81lCQeM43wRaTnhQp8M5swsyfN7EJwP96g3W4z\n+3Mze9nMzpvZj4fZbzPFYsb4cLLh1bbuXgp8nZIpIhEXdoT/CHDG3Y8AZ4L1en4H+Bt3v43Sj6Gc\nD7nfpipdbVv/LJ2l1Ty5QlFTOiISeWED/zhwKlg+BTxY28DMxoCfAv4QwN1z7n4l5H6baqPyCutl\nFVQ4TUSiLWzg73f3GYDgfl+dNrcAs8D/MLPnzewPgh8z7xqT6YGGUzoqnCYivWLTwDezp8zsXJ3b\n8S3uIwHcA/yeu98NZGk89YOZPWxmZ83s7Ozs7BZ3Ec6GI3zV0RGRHrHpj5i7+/2NtpnZJTObcvcZ\nM5sCLtdp9jbwtrs/E6z/ORsEvrufBE4CTE9P+2b9a4aJdIory2vkC0US8evfA1UpU0R6RdgpnSeA\nE8HyCeDx2gbu/i7wlpndGjx0H/C9kPttqnJRtPnl99fTyahwmoj0iLCB/xhw1MwuAEeDdczsoJmd\nrmr3L4EvmdmLwF3Afwy536ba6Grbuewqg8kYw6lNPwyJiHS1UCnm7hlKI/baxy8Cx6rWXwCmw+yr\nlcqBXyqgNnrdtkw2x6TO0BGRHtD3V9oClUCvP8LXVbYi0hsU+Gw2paPAF5HeoMAHxoeDmvh1Cqip\ncJqI9AoFPpCIx9g9nNQIX0R6mgI/UO/iq2u5AtfWCiqcJiI9QYEfmEyn3vczh+V1TemISC9Q4Afq\njfBVOE1EeokCPzCRHnhf4GdUVkFEeogCPzCZTjG/vEaxuF6+p1w4TVM6ItILFPiBiXSKQtG5em29\nnk5lSkdf2opID1DgB8rF0arr4meyOZJxY3RAdXREJPoU+IF6V9vOZVeZSKcws051S0SkaRT4gfXA\nXz81s3TRlc7QEZHeoMAPlAuo1U7p6AtbEekVCvzAeLpUT2duqXpKR2UVRKR3KPADA4k4owMJ5par\nAn9JgS8ivSNU4JvZhJk9aWYXgvvxOm1uNbMXqm4LZvb5MPttlYmR9attV/MFFlfzmtIRkZ4RdoT/\nCHDG3Y8AZ6jz4+Tu/oq73+XudwH/AFgGvhZyvy1RXV5hPls6H1/n4ItIrwgb+MeBU8HyKeDBTdrf\nB/zA3d8Iud+WmEynKjXxVThNRHpN2MDf7+4zAMH9vk3aPwR8eaMGZvawmZ01s7Ozs7Mhu7c91SN8\nFU4TkV6z6SWkZvYUcKDOpke3syMzSwGfBn5to3bufhI4CTA9Pe0btW22cgE1d68KfI3wRaQ3bBr4\n7n5/o21mdsnMptx9xsymgMsbPNWngOfc/dIO+tkWk+kUuUKRpdV8ZWpHUzoi0ivCTuk8AZwIlk8A\nj2/Q9rNsMp3TaeNV5RXmsjniMWPXULLDvRIRaY6wgf8YcNTMLgBHg3XM7KCZnS43MrPhYPtXQ+6v\npcqj+Uw2RyabY3w4SSymOjoi0htClYF09wylM29qH78IHKtaXwYmw+yrHSr1dJZylcJpIiK9Qlfa\nVpmomdJR4ItIL1HgV6muiV8qnKZTMkWkdyjwqwynEgwmY8xlVzXCF5Geo8CvMZkeYHZxlSvLawp8\nEekpCvwaE+kUP5jNAutTPCIivUCBX2MineLVy0uVZRGRXqHArzGZTnFtrQAo8EWktyjwa1SHvM7S\nEZFeosCvUV3/XiN8EeklCvwa1cXSxodVR0dEeocCv0a5/v3u4SSJuA6PiPQOJVqN8jSOpnNEpNco\n8GuUp3RUB19Eeo0Cv0b5S1uN8EWk1yjwa4wOJEjFY/otWxHpOaEC38wmzOxJM7sQ3I83aPevzewl\nMztnZl82s8Ew+20lM+PRn72dh37spk53RUSkqcKO8B8Bzrj7EeBMsH4dM7sB+FfAtLvfCcSBh0Lu\nt6VO/MRhPnLT7k53Q0SkqcIG/nHgVLB8CniwQbsEMGRmCWAYuBhyvyIisk1hA3+/u88ABPf7ahu4\n+zvAfwHeBGaAq+7+9UZPaGYPm9lZMzs7OzsbsnsiIlK2aeCb2VPB3Hvt7fhWdhDM6x8HbgYOAmkz\n+/lG7d39pLtPu/v03r17t/r3EBGRTWz6I+bufn+jbWZ2ycym3H3GzKaAy3Wa3Q/80N1ngz/zVeAn\ngD/ZYZ9FRGQHwk7pPAGcCJZPAI/XafMmcK+ZDZuZAfcB50PuV0REtils4D8GHDWzC8DRYB0zO2hm\npwHc/Rngz4HngO8G+zwZcr8iIrJN5u6d7kND09PTfvbs2U53Q0QkMszsWXefrrdNV9qKiPSJrh7h\nm9ks8MYO//ge4L0mdqfZ1L9w1L9w1L9wurl/H3D3uqc4dnXgh2FmZxt9rOkG6l846l846l843d6/\nRjSlIyLSJxT4IiJ9opcDv9tP/VT/wlH/wlH/wun2/tXVs3P4IiJyvV4e4YuISBUFvohIn4h04JvZ\nA2b2ipm9amb1fnzFzOy/BttfNLN72ty/m8zsG2Z2PvjFr1+t0+anzeyqmb0Q3L7Q5j6+bmbfDfb9\nvsuaO3kMzezWquPygpktmNnna9q09fiZ2RfN7LKZnat6bKu//Lbh67WF/fvPZvZy8O/3NTPb3eDP\nbvhaaGH/ft3M3qn6NzzW4M926vh9papvr5vZCw3+bMuPX2juHskbpV/O+gFwC5ACvgPcUdPmGPDX\ngAH3As+0uY9TwD3B8ijw/Tp9/Gngrzp4HF8H9mywvaPHsObf+11KF5V07PgBPwXcA5yreuw/AY8E\ny48Av9Gg/xu+XlvYv08CiWD5N+r1byuvhRb279eBf7uFf/+OHL+a7b8JfKFTxy/sLcoj/I8Cr7r7\na+6eA/6UUt39aseBP/aSp4HdQRnntnD3GXd/LlhepFQl9IZ27b9JOnoMq9wH/MDdd3rldVO4+zeB\nuZqHt/LLb1t5vbakf+7+dXfPB6tPAzc2e79b1eD4bUXHjl9ZUO33M8CXm73fdoly4N8AvFW1/jbv\nD9OttGkLMzsM3A08U2fzj5vZd8zsr83sQ+3tGQ583cyeNbOH62zvlmP4EI3/o3Xy+MEWfvmN7jmO\n/4LSJ7Z6NnsttNKvBFNOX2wwJdYNx+8ngUvufqHB9k4evy2JcuBbncdqzzHdSpuWM7MR4C+Az7v7\nQs3m5yhNU3wE+G/AX7a5ex9393uATwG/bGY/VbO948fQzFLAp4E/q7O508dvq7rhOD4K5IEvNWiy\n2WuhVX4P+BHgLko/g/qbddp0/PgBn2Xj0X2njt+WRTnw3wZuqlq/kff/OPpW2rSUmSUphf2X3P2r\ntdvdfcHdl4Ll00DSzPa0q3/ufjG4vwx8jdJH52odP4aU/gM95+6Xajd0+vgFLpWnuazxL7919Dia\n2Qng54B/6sGEc60tvBZawt0vuXvB3YvAf2+w304fvwTwT4CvNGrTqeO3HVEO/G8DR8zs5mAE+BCl\nX+Cq9gTwC8GZJvdS+gH1mXZ1MJjz+0PgvLv/VoM2B4J2mNlHKf2bZNrUv7SZjZaXKX25d66mWUeP\nYaDhyKqTx6/KVn75bSuv15YwsweAfw982t2XG7TZymuhVf2r/k7oHzfYb8eOX+B+4GV3f7vexk4e\nv23p9LfGYW6UziD5PqVv7x8NHvsc8Llg2YDfDbZ/F5huc/8+Qelj54vAC8HtWE0ffwV4idJZB08D\nP9HG/t0S7Pc7QR+68RgOUwrwXVWPdez4UXrjmQHWKI06fwmYBM4AF4L7iaDtQeD0Rq/XNvXvVUrz\n3+XX4O/X9q/Ra6FN/fufwWvrRUohPtVNxy94/I/Kr7mqtm0/fmFvKq0gItInojylIyIi26DAFxHp\nEwp8EZE+ocAXEekTCnwRkT6hwBcR6RMKfBGRPvH/AdnylZFf4iihAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "\u003cFigure size 600x400 with 1 Axes\u003e"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.plot(list(range(len(history))), [a - 0.9 for a in history])\n",
        "plt.show()\n",
        "\n",
        "# It turns out that SGD with the given learning rate doesn't work well."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xFh1gIliSjCX"
      },
      "source": [
        "\n",
        "\u003c!-- Hyperparameter tuning is already an integral part of ML experimentation. But it is not so for experimenting different neural architectures, losses, optimization algorithms and etc. Nerual architecture search is still considered an advanced/complex ML techinique. W --\u003e\n",
        "\n",
        "See also:\n",
        "* [NAS with MNIST](https://github.com/google/pyglove/tree/main/examples/automl/mnist)\n",
        "* [NAS-Bench-101 with PyGlove](https://github.com/google/pyglove/tree/main/examples/automl/nasbench)\n",
        "* [Evolving Reinforcement Learning Algorithms](https://github.com/google/brain_autorl/blob/main/evolving_rl/EvolvingRL_Demo.ipynb)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "last_runtime": {
        "build_target": "//learning/deepmind/dm_python:dm_notebook3",
        "kind": "private"
      },
      "name": "Symbolic Machine Learning",
      "provenance": [
        {
          "file_id": "1u_0OuteakM9kLtHzpgvkzMFlZIDv-SKf",
          "timestamp": 1656530243208
        }
      ],
      "toc_visible": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}