nngp_nas/NNGP_on_NASBench101.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vf0LrbTEGGid"
      },
      "source": [
        "\n",
        "\u003ca href=\"https://colab.research.google.com/github/google-research/google-research/blob/master/nngp_nas/NNGP_on_NASBench101.ipynb\" target=\"_parent\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\u003c/a\u003e\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XWxIm3QxGBLH"
      },
      "source": [
        "Copyright 2020 Google LLC\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "you may not use this file except in compliance with the License.\n",
        "You may obtain a copy of the License at\n",
        "\n",
        "     https://www.apache.org/licenses/LICENSE-2.0\n",
        "\n",
        "Unless required by applicable law or agreed to in writing, software\n",
        "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "See the License for the specific language governing permissions and\n",
        "limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-rMTbJ6Epz3U"
      },
      "source": [
        "# Colab demonstration for NNGP evaluation.\n",
        "\n",
        "This colab accompanies the paper [**Towards NNGP-guided Neural Architecture Search**](https://arxiv.org/abs/2005.09629). We construct an NNGP evaluator of networks in the [**NAS-Bench-101**](http://proceedings.mlr.press/v97/ying19a.html) dataset, and compute some metrics on how well it predicts the ground truth performance of the networks.\n",
        "\n",
        "The publicly available and free hosted colab instances can be used to run this colab. This colab uses the NAS-Bench-101 package, which can be found at https://github.com/google-research/nasbench."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mEeWh4-51EFA"
      },
      "outputs": [],
      "source": [
        "# NAS-Bench-101 uses TF 1.x\n",
        "%tensorflow_version 1.x\n",
        "\n",
        "# Download the raw data (only 108 epoch data points).\n",
        "!curl -O https://storage.googleapis.com/nasbench/nasbench_only108.tfrecord\n",
        "\n",
        "# Clone and install the code and dependencies.\n",
        "!git clone https://github.com/google-research/nasbench\n",
        "!pip install ./nasbench\n",
        "\n",
        "# Initialize the NASBench object which parses the raw data into memory (this\n",
        "# should only be run once as it takes up to a few minutes).\n",
        "from nasbench import api\n",
        "\n",
        "# Load NAS-Bench-101 data.\n",
        "# This will take a while, and should only be run once.\n",
        "nasbench_only108 = api.NASBench('nasbench_only108.tfrecord')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M_xlxmB3LUBx"
      },
      "source": [
        "# Imports and Definitions\n",
        "\n",
        "We define some useful constants and the `NNGP_CIFAR10_Evaluator` to be used for NNGP evaluation of networks."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "3HXZu_TlpSHJ"
      },
      "outputs": [],
      "source": [
        "#@title Imports\n",
        "\n",
        "import numpy as np\n",
        "import scipy\n",
        "import scipy.stats as stats\n",
        "import sklearn\n",
        "import tensorflow as tf\n",
        "import tensorflow_datasets as tfds\n",
        "from tqdm.notebook import tqdm\n",
        "\n",
        "from nasbench import api\n",
        "from nasbench.lib import cifar\n",
        "from nasbench.lib import base_ops\n",
        "from nasbench.lib import model_builder\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "tf.get_logger().setLevel('ERROR')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "bQ0Hre7XA20M"
      },
      "outputs": [],
      "source": [
        "#@title Useful Constants\n",
        "INPUT = 'input'\n",
        "OUTPUT = 'output'\n",
        "CONV3X3 = 'conv3x3-bn-relu'\n",
        "CONV1X1 = 'conv1x1-bn-relu'\n",
        "MAXPOOL3X3 = 'maxpool3x3'\n",
        "NUM_VERTICES = 7\n",
        "MAX_EDGES = 9\n",
        "EDGE_SPOTS = NUM_VERTICES * (NUM_VERTICES - 1) / 2   # Upper triangular matrix\n",
        "OP_SPOTS = NUM_VERTICES - 2   # Input/output vertices are fixed\n",
        "ALLOWED_OPS = [CONV3X3, CONV1X1, MAXPOOL3X3]\n",
        "ALLOWED_EDGES = [0, 1]   # Binary adjacency matrix\n",
        "\n",
        "# Default config for building architecture\n",
        "_CIFAR_DEFAULT_CONFIG = {\n",
        "    'nasbench_stem_filter_size': 128,\n",
        "    'nasbench_num_modules_per_stack': 3,\n",
        "    'nasbench_num_stacks': 3,\n",
        "    'nasbench_data_format': 'channels_last',\n",
        "    'nasbench_num_labels': 10,\n",
        "}\n",
        "\n",
        "# Pixel statistics for CIFAR10.\n",
        "CIFAR_MEAN = np.array([125.3, 123.0, 113.9])\n",
        "CIFAR_STDDEV = np.array([63.0, 62.1, 66.7])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "qciVMd3V_a1w"
      },
      "outputs": [],
      "source": [
        "#@title Helper functions\n",
        "\n",
        "def one_hot_cifar(label_batch):\n",
        "  \"\"\"One-hot function for CIFAR10.\n",
        "  \n",
        "  Args:\n",
        "    Batch of labels.\n",
        "\n",
        "  Returns:\n",
        "    Batch of adjusted one-hot labels with mean-zero.\n",
        "  \"\"\"\n",
        "  return np.array([[float(x==i) - 0.1 for i in range(10)] for x in label_batch])\n",
        "\n",
        "def build_model_fn(spec, config):\n",
        "  \"\"\"Returns a model function returning a feature vector.\"\"\"\n",
        "  if config['nasbench_data_format'] == 'channels_last':\n",
        "    channel_axis = 3\n",
        "  elif config['nasbench_data_format'] == 'channels_first':\n",
        "    # Currently this is not well supported\n",
        "    channel_axis = 1\n",
        "  else:\n",
        "    raise ValueError('invalid nasbench_data_format')\n",
        "\n",
        "  def model_fn(features, is_training):\n",
        "    \"\"\"Returns feature vector and training op from the input features.\n",
        "    \n",
        "    Args:\n",
        "      features: Input features.\n",
        "      is_training: Bool indicating whether to update batch-norm parameters.\n",
        "        Batch-norm parameters are updated when set to `true`.\n",
        "\n",
        "    Returns:\n",
        "      (net, train_op)\n",
        "      net: Feature vectors from penultimate layer of network.\n",
        "      train_op: Batch-normalization op.\n",
        "    \"\"\"\n",
        "    # Initial stem convolution\n",
        "    with tf.variable_scope('stem'):\n",
        "      net = base_ops.conv_bn_relu(\n",
        "          features, 3, config['nasbench_stem_filter_size'],\n",
        "          is_training, config['nasbench_data_format'])\n",
        "    for stack_num in range(config['nasbench_num_stacks']):\n",
        "      channels = net.get_shape()[channel_axis].value\n",
        "      # Downsample at start (except first)\n",
        "      if stack_num \u003e 0:\n",
        "        net = tf.layers.max_pooling2d(\n",
        "            inputs=net,\n",
        "            pool_size=(2, 2),\n",
        "            strides=(2, 2),\n",
        "            padding='same',\n",
        "            data_format=config['nasbench_data_format'])\n",
        "        # Double output channels each time we downsample\n",
        "        channels *= 2\n",
        "      with tf.variable_scope('stack{}'.format(stack_num)):\n",
        "        for module_num in range(config['nasbench_num_modules_per_stack']):\n",
        "          with tf.variable_scope('module{}'.format(module_num)):\n",
        "            net = model_builder.build_module(\n",
        "                spec,\n",
        "                inputs=net,\n",
        "                channels=channels,\n",
        "                is_training=is_training)\n",
        "    # Global average pool\n",
        "    if config['nasbench_data_format'] == 'channels_last':\n",
        "      net = tf.reduce_mean(net, [1, 2])\n",
        "    elif config['nasbench_data_format'] == 'channels_first':\n",
        "      net = tf.reduce_mean(net, [2, 3])\n",
        "    else:\n",
        "      raise ValueError('invalid nasbench_data_format')\n",
        "\n",
        "    # Define the training op as the update ops in the graph.\n",
        "    # The results in the training op being set to batch-norm updates.\n",
        "    train_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n",
        "    train_op_w = tf.group(*train_ops)\n",
        "\n",
        "    return net, train_op_w\n",
        "\n",
        "  return model_fn"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "Dq9K8xNBLUKb"
      },
      "outputs": [],
      "source": [
        "#@title NNGP_CIFAR10_Evaluator\n",
        "\n",
        "class NNGP_CIFAR10_Evaluator:\n",
        "  \"\"\"NNGP evaluator of models on CIFAR10.\"\"\"\n",
        "\n",
        "  def __init__(self,\n",
        "               train_size,\n",
        "               val_size,\n",
        "               random_seed=0,\n",
        "               config=_CIFAR_DEFAULT_CONFIG):\n",
        "    \"\"\"Initialize evaluator object.\n",
        "\n",
        "    Args:\n",
        "      train_size: Size of NNGP training set selected from CIFAR10 training set.\n",
        "      val_size: Size of NNGP validation set selected from CIFAR10 training set.\n",
        "      random_seed: Numpy random seed used for shuffing CIFAR10 training set.\n",
        "      config: Configurations required to build network.\n",
        "    \"\"\"\n",
        "    self.config = config\n",
        "    self._random_seed = random_seed\n",
        "    self._train_size = train_size\n",
        "    self._val_size = val_size\n",
        "    self._construct_dataset()\n",
        "\n",
        "  def get_train_size(self):\n",
        "    \"\"\"Return NNGP training set size.\"\"\"\n",
        "    return self._train_size\n",
        "\n",
        "  def get_val_size(self):\n",
        "    \"\"\"Return NNGP validation set size.\"\"\"\n",
        "    return self._val_size\n",
        "\n",
        "  def _construct_dataset(self):\n",
        "    \"\"\"Construct dataset for NNGP training and evaluation from CIFAR10.\n",
        "\n",
        "    Args:\n",
        "      train_size: Size of NNGP training set selected from CIFAR10 training set.\n",
        "      val_size: Size of NNGP validation set selected from CIFAR10 training set.\n",
        "    \"\"\"\n",
        "    train_size = self._train_size\n",
        "    val_size = self._val_size\n",
        "    random_seed = self._random_seed\n",
        "\n",
        "    assert train_size + val_size \u003c= 50000\n",
        "\n",
        "    # Load CIFAR10 training set\n",
        "    ds, _ = tfds.load('cifar10', split='train', batch_size=-1, with_info=True)\n",
        "    ds = tfds.as_numpy(ds)\n",
        "\n",
        "    # Shuffle index\n",
        "    np.random.seed(random_seed)\n",
        "    shuffled = np.arange(50000)\n",
        "    np.random.shuffle(shuffled)\n",
        "    train_index = shuffled[:train_size]\n",
        "    val_index = shuffled[-val_size:]\n",
        "\n",
        "    # Select training subset and validation subset.\n",
        "    train_images = ds['image'][train_index]\n",
        "    train_images = (train_images - CIFAR_MEAN) / CIFAR_STDDEV\n",
        "    train_labels = ds['label'][train_index]\n",
        "\n",
        "    val_images = ds['image'][val_index]\n",
        "    val_images = (val_images - CIFAR_MEAN) / CIFAR_STDDEV\n",
        "    val_labels = ds['label'][val_index]\n",
        "\n",
        "    self.data_dict = dict(train_images=train_images,\n",
        "                          train_labels=train_labels,\n",
        "                          val_images=val_images,\n",
        "                          val_labels=val_labels)\n",
        "    \n",
        "    del ds\n",
        "\n",
        "  def nngp_eval(self,\n",
        "                spec,\n",
        "                batch_size=100,\n",
        "                ensemble_number=4,\n",
        "                verbose=False):\n",
        "    \"\"\"Evaluate NAS-Bench-101 model using NNGP evaluation.\n",
        "\n",
        "    Args:\n",
        "      spec: ModelSpec object.\n",
        "      batch_size: Batch size to use for NNGP training and evaluation.\n",
        "        The batch size must divide training and val set size.\n",
        "      ensemble_number: Ensemble number for monte-carlo NNGP kernel estimation.\n",
        "\n",
        "    Returns:\n",
        "      NNGP validation accuracy.\n",
        "    \"\"\"\n",
        "\n",
        "    # Get data\n",
        "    data_dict = self.data_dict\n",
        "    train_images = data_dict['train_images']\n",
        "    train_labels = data_dict['train_labels']\n",
        "    val_images = data_dict['val_images']\n",
        "    val_labels = data_dict['val_labels']\n",
        "\n",
        "    n_t = len(train_labels)\n",
        "    n_v = len(val_labels)\n",
        "\n",
        "    # Initialize Kernel matrices.\n",
        "    K_tt = np.zeros((n_t,n_t),dtype=np.float32)\n",
        "    K_vt = np.zeros((n_v,n_t),dtype=np.float32)\n",
        "    assert n_t % batch_size == 0, (\n",
        "        \"Batch size does not divide training set size.\")\n",
        "    assert n_v % batch_size == 0, (\n",
        "        \"Batch size does not divide validation set size.\")\n",
        "    \n",
        "    model_fn = build_model_fn(spec, self.config)\n",
        "\n",
        "    with tf.Graph().as_default():\n",
        "      features = tf.placeholder(dtype=tf.float32, shape=[batch_size, 32, 32, 3])\n",
        "      is_training = tf.placeholder(dtype=tf.bool, name='is_training')\n",
        "      vecs, ops = model_fn(features, is_training)\n",
        "      tf.logging.info(\"Model constructed.\")\n",
        "      for curr_m in range(ensemble_number):\n",
        "        with tf.Session() as sess:\n",
        "          sess.run(tf.global_variables_initializer())\n",
        "          # Set batch norm parameters using random batch of training set.\n",
        "          init_ind = np.random.choice(\n",
        "              n_t, batch_size, replace=False)\n",
        "          init_sample = train_images[init_ind]\n",
        "          _ = sess.run(ops,\n",
        "                      feed_dict={features: init_sample, is_training: True})\n",
        "          # Get train feature vectors.\n",
        "          fvecs = []\n",
        "          iterations = n_t // batch_size\n",
        "          for i in range(iterations):\n",
        "            feed = train_images[batch_size * i:batch_size * (i+1)]\n",
        "            vres = sess.run([vecs],\n",
        "                            feed_dict={features: feed, is_training: False})\n",
        "            fvecs += vres\n",
        "          fvecs_t = np.concatenate(fvecs, axis=0)\n",
        "          # Update K_tt using train feature vectors.\n",
        "          K_tt += np.matmul(fvecs_t, fvecs_t.T) * (1.0/ensemble_number)\n",
        "          # Get validation feature vectors.\n",
        "          fvecs = []\n",
        "          iterations = n_v // batch_size\n",
        "          for i in range(iterations):\n",
        "            feed = val_images[batch_size * i:batch_size * (i+1)]\n",
        "            vres = sess.run([vecs],\n",
        "                            feed_dict={features: feed, is_training: False})\n",
        "            fvecs += vres\n",
        "          fvecs_v = np.concatenate(fvecs, axis=0)\n",
        "          # Update K_vt using val feature vetors.\n",
        "          K_vt += np.matmul(fvecs_v, fvecs_t.T) * (1.0/ensemble_number)\n",
        "          \n",
        "      performance = 0.0\n",
        "      # Range of regularizer set manually.\n",
        "      diag_reg_values = np.logspace(-7, 2, num=20)\n",
        "      one_hot_labels_t = one_hot_cifar(train_labels)\n",
        "      for epsilon in diag_reg_values:\n",
        "        # Regularize K_tt.\n",
        "        K_tt_reg = K_tt + epsilon * np.trace(K_tt) / n_t * np.eye(n_t)\n",
        "        # 'try' statement, since scipty.linalg.solve can fail.\n",
        "        try:\n",
        "          # Perform NNGP inference to obtain validation accuracy.\n",
        "          inv_labels = scipy.linalg.solve(\n",
        "              K_tt_reg, one_hot_labels_t, sym_pos=True)\n",
        "          prediction = np.matmul(K_vt, inv_labels)\n",
        "          correct_predictions = (\n",
        "              np.argmax(prediction, axis=1) == val_labels).astype(np.float32)\n",
        "          performance = max(performance, np.mean(correct_predictions))\n",
        "        except:\n",
        "          if verbose:\n",
        "            print(\"Matrix inversion error for epsilon = {}\".format(epsilon))\n",
        "          continue\n",
        "    \n",
        "    return performance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tx8Bv3BrBEx_"
      },
      "source": [
        "# Single example\n",
        "\n",
        "Here we construct a single network cell spec and compute its NNGP validation accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hasG33UbM59Z"
      },
      "outputs": [],
      "source": [
        "# Construct evaluator.\n",
        "evaluator = NNGP_CIFAR10_Evaluator(train_size=500, val_size=200)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "executionInfo": {
          "elapsed": 164845,
          "status": "ok",
          "timestamp": 1602028098579,
          "user": {
            "displayName": "Jaehoon Lee",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhxHqEwdpweGC4ZrtdwaxYD0GM-tam3aLuSS-I4aQ=s64",
            "userId": "07296832508386080109"
          },
          "user_tz": 420
        },
        "id": "fD1iu3nUOPOR",
        "outputId": "d4bdd189-d744-488f-c4e8-173b2b079e4e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "NNGP validation accuracy:\n",
            "0.38\n"
          ]
        }
      ],
      "source": [
        "# Build ModelSpec of Inception-like cell.\n",
        "cell = api.ModelSpec(\n",
        "  matrix=[[0, 1, 1, 1, 0, 1, 0],    # input layer\n",
        "          [0, 0, 0, 0, 0, 0, 1],    # 1x1 conv\n",
        "          [0, 0, 0, 0, 0, 0, 1],    # 3x3 conv\n",
        "          [0, 0, 0, 0, 1, 0, 0],    # 5x5 conv (replaced by two 3x3's)\n",
        "          [0, 0, 0, 0, 0, 0, 1],    # 5x5 conv (replaced by two 3x3's)\n",
        "          [0, 0, 0, 0, 0, 0, 1],    # 3x3 max-pool\n",
        "          [0, 0, 0, 0, 0, 0, 0]],   # output layer\n",
        "  # Operations at the vertices of the module, matches order of matrix.\n",
        "  ops=[INPUT, CONV1X1, CONV3X3, CONV3X3, CONV3X3, MAXPOOL3X3, OUTPUT])\n",
        "\n",
        "# Perform NNGP inference using evaluator to obtain NNGP validation accuracy.\n",
        "nngp_validation_accuracy = evaluator.nngp_eval(cell, batch_size=100, ensemble_number=4)\n",
        "\n",
        "print(\"NNGP validation accuracy:\")\n",
        "print(nngp_validation_accuracy)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 408
        },
        "executionInfo": {
          "elapsed": 164838,
          "status": "ok",
          "timestamp": 1602028098580,
          "user": {
            "displayName": "Jaehoon Lee",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhxHqEwdpweGC4ZrtdwaxYD0GM-tam3aLuSS-I4aQ=s64",
            "userId": "07296832508386080109"
          },
          "user_tz": 420
        },
        "id": "mQBJfENPRnt1",
        "outputId": "a14d814e-98c8-4fd0-d6ef-32a7e0fe23cf"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([8, 9, 3, 7, 6, 3, 3, 9, 8, 7, 6, 4, 2, 1, 9, 1, 3, 9, 3, 7, 9, 9,\n",
              "       5, 8, 1, 7, 2, 8, 0, 3, 3, 7, 8, 9, 1, 3, 5, 4, 8, 3, 8, 3, 7, 7,\n",
              "       6, 9, 3, 5, 4, 7, 9, 0, 9, 9, 7, 2, 4, 7, 8, 1, 9, 4, 8, 7, 2, 0,\n",
              "       1, 1, 6, 0, 3, 4, 5, 6, 4, 6, 9, 5, 1, 7, 7, 6, 2, 8, 2, 8, 4, 0,\n",
              "       5, 5, 3, 0, 4, 4, 0, 6, 2, 8, 9, 9, 8, 2, 6, 2, 9, 7, 0, 9, 8, 9,\n",
              "       6, 6, 5, 7, 3, 6, 0, 4, 9, 3, 0, 4, 0, 2, 7, 4, 1, 1, 6, 0, 5, 0,\n",
              "       0, 8, 3, 1, 4, 6, 4, 5, 6, 1, 6, 1, 1, 7, 1, 2, 3, 9, 3, 5, 8, 2,\n",
              "       0, 0, 4, 9, 0, 0, 8, 9, 5, 7, 8, 5, 3, 6, 7, 6, 7, 9, 2, 6, 3, 7,\n",
              "       8, 1, 8, 3, 9, 4, 0, 3, 0, 0, 3, 9, 3, 6, 4, 2, 3, 5, 9, 6, 2, 5,\n",
              "       7, 4, 4, 4, 1, 2, 2, 4, 2, 9, 7, 8, 7, 1, 7, 9, 1, 3, 1, 4, 3, 6,\n",
              "       9, 4, 9, 2, 1, 5, 8, 0, 2, 3, 9, 3, 9, 9, 0, 2, 9, 7, 9, 7, 9, 9,\n",
              "       6, 1, 6, 5, 1, 8, 9, 4, 5, 5, 6, 9, 9, 8, 2, 8, 2, 8, 2, 8, 1, 8,\n",
              "       3, 5, 1, 6, 2, 3, 2, 2, 0, 8, 3, 5, 1, 8, 9, 8, 0, 1, 7, 4, 1, 8,\n",
              "       8, 8, 2, 6, 9, 0, 4, 1, 4, 3, 5, 1, 0, 2, 0, 3, 4, 8, 2, 9, 1, 5,\n",
              "       5, 6, 4, 5, 8, 5, 3, 5, 8, 7, 0, 1, 5, 7, 2, 4, 3, 4, 9, 5, 3, 6,\n",
              "       8, 0, 2, 1, 6, 0, 4, 0, 7, 2, 3, 1, 6, 0, 5, 7, 9, 7, 9, 0, 5, 8,\n",
              "       0, 9, 0, 0, 4, 4, 8, 2, 4, 9, 6, 0, 5, 2, 9, 8, 0, 8, 1, 1, 1, 3,\n",
              "       7, 2, 1, 9, 1, 3, 9, 2, 0, 5, 2, 5, 3, 7, 7, 5, 9, 0, 0, 3, 1, 5,\n",
              "       9, 7, 9, 4, 5, 5, 1, 9, 2, 8, 4, 5, 3, 3, 4, 4, 1, 1, 1, 7, 4, 8,\n",
              "       1, 2, 7, 7, 5, 5, 9, 4, 8, 2, 0, 6, 9, 2, 2, 1, 1, 0, 4, 3, 0, 9,\n",
              "       6, 2, 7, 0, 5, 2, 4, 3, 4, 2, 2, 4, 3, 7, 7, 0, 0, 3, 5, 7, 1, 0,\n",
              "       7, 7, 0, 5, 2, 5, 8, 9, 5, 5, 8, 3, 1, 8, 5, 2, 9, 3, 2, 1, 9, 9,\n",
              "       1, 3, 9, 3, 5, 1, 9, 1, 6, 3, 2, 8, 7, 8, 5, 6])"
            ]
          },
          "execution_count": 11,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "evaluator.data_dict['train_labels']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "executionInfo": {
          "elapsed": 171360,
          "status": "ok",
          "timestamp": 1602028105113,
          "user": {
            "displayName": "Jaehoon Lee",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhxHqEwdpweGC4ZrtdwaxYD0GM-tam3aLuSS-I4aQ=s64",
            "userId": "07296832508386080109"
          },
          "user_tz": 420
        },
        "id": "mnr55ZMUQ8Xv",
        "outputId": "aea782c2-a3cd-44f2-fd05-8b710a5709aa"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([7, 8, 4, 4, 6, 5, 2, 9, 6, 6, 9, 9, 3, 0, 8, 7, 9, 0, 4, 9])"
            ]
          },
          "execution_count": 12,
          "metadata": {
            "tags": []
          },
          "output_type": "execute_result"
        }
      ],
      "source": [
        "ds, _ = tfds.load('cifar10', split='train', batch_size=-1, with_info=True)\n",
        "ds = tfds.as_numpy(ds)\n",
        "ds['label'][:20]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A7N9-9EiCwL1"
      },
      "source": [
        "# NNGP inference on subsets of NAS-Bench-101\n",
        "\n",
        "Let us now select 50 random networks from the NAS-Bench-101 dataset and plot the ground-truth performance (obtained by taking the average of the final test set accuracy after 108-epoch training) against NNGP validation accuracy.\n",
        "\n",
        "While we set the NNGP training and validation set size to 100, and the ensemble number to 2 here for rapid evaluation, these parameters may be adjusted.\n",
        "\n",
        "We also compute some basic metrics on how well the NNGP validation accuracy predicts the ground truth performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "Yhf9zumrOXO9"
      },
      "outputs": [],
      "source": [
        "#@title Function defs\n",
        "\n",
        "def random_specs(oracle, num, random_seed=12345):\n",
        "  \"\"\"Returns a list of valid specs in oracle.\n",
        "  \n",
        "  Args:\n",
        "    oracle: An api.NASBench object.\n",
        "    num: Number of specs to return.\n",
        "\n",
        "  Returns:\n",
        "    A list of random valid specs from oracle.\n",
        "  \"\"\"\n",
        "  specs = []\n",
        "  hashes = set([])\n",
        "  \n",
        "  np.random.seed(random_seed)\n",
        "  while len(specs) \u003c num:\n",
        "    # Find a valid random spec.\n",
        "    while True:\n",
        "      matrix = np.random.choice(ALLOWED_EDGES,\n",
        "                                size=(NUM_VERTICES, NUM_VERTICES))\n",
        "      matrix = np.triu(matrix, 1)\n",
        "      ops = np.random.choice(ALLOWED_OPS, size=(NUM_VERTICES)).tolist()\n",
        "      ops[0] = INPUT\n",
        "      ops[-1] = OUTPUT\n",
        "      spec = api.ModelSpec(matrix=matrix, ops=ops)\n",
        "      if oracle.is_valid(spec):\n",
        "        break\n",
        "    # Check if spec is already in collected set.\n",
        "    h = spec.hash_spec(ALLOWED_OPS)\n",
        "    if h in hashes:\n",
        "      continue\n",
        "    specs += [spec]\n",
        "    hashes.add(h)\n",
        "\n",
        "  return specs\n",
        "\n",
        "\n",
        "def mean_108epoch_performance(spec, oracle):\n",
        "  \"\"\"Returns the mean performance of model after 108-epoch training.\n",
        "\n",
        "  Args:\n",
        "    spec: ModelSpec.\n",
        "    oracle: An api.NASBench object containing final_test_accuracy for\n",
        "      108-epoch training.\n",
        "\n",
        "  Returns:\n",
        "    Mean final_test_accuracy of model built from spec upon 108-epoch training.\n",
        "  \"\"\"\n",
        "  result = np.mean([d['final_test_accuracy']\n",
        "                    for d in oracle.get_metrics_from_spec(spec)[1][108]])\n",
        "\n",
        "  return result"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XrOJVsQ2pk8h"
      },
      "source": [
        "## Experiment\n",
        "\n",
        "Let us run the experiment and present performance metrics.\n",
        "\n",
        "This make take up to ~30 mins on a GPU (V100) instance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tLFLRbgQ9xfQ"
      },
      "outputs": [],
      "source": [
        "# Experiment configurations\n",
        "NUM_SPECS = 250\n",
        "RANDOM_SEED = 123\n",
        "TRAIN_SIZE = 100\n",
        "VAL_SIZE = 200\n",
        "BATCH_SIZE = 100\n",
        "ENSEMBLE_NUMBER = 2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qD8mErilX0Gc"
      },
      "outputs": [],
      "source": [
        "# Get 250 random specs from NAS-Bench-101 dataset.\n",
        "nasbench_specs = random_specs(\n",
        "    oracle=nasbench_only108, num=NUM_SPECS, random_seed=RANDOM_SEED)\n",
        "\n",
        "# Construct NNGP evaluator.\n",
        "evaluator = NNGP_CIFAR10_Evaluator(train_size=TRAIN_SIZE, val_size=VAL_SIZE)\n",
        "\n",
        "# Retrieve mean 108-epoch trained test-accuracy for the 250 specs.\n",
        "ground_truth = [\n",
        "                mean_108epoch_performance(spec=spec, oracle=nasbench_only108) \n",
        "                for spec in nasbench_specs\n",
        "                ]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 66,
          "referenced_widgets": [
            "6e6ccad3e71b4fc593be0143edf845c7",
            "b619a80c5d794002b34faba9e210b393",
            "ec14e1007b04450eb6dc7147a52508d6",
            "8b475a3beaca4ab39771ff5ef2afe5f4",
            "cb151c02cf624186b3489757a8e1e0eb",
            "e20930b90d27465fb7464a1f77b46620",
            "c7430782e31946dd9b2b60510840d8b4",
            "8cc0d1a4e21046f99d657b015a1315a4"
          ]
        },
        "executionInfo": {
          "elapsed": 1367509,
          "status": "ok",
          "timestamp": 1602036547206,
          "user": {
            "displayName": "Jaehoon Lee",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhxHqEwdpweGC4ZrtdwaxYD0GM-tam3aLuSS-I4aQ=s64",
            "userId": "07296832508386080109"
          },
          "user_tz": 420
        },
        "id": "t-uz_5q9Z_jt",
        "outputId": "4ed03f9f-56a7-48c0-b219-3cf583c0904c"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6e6ccad3e71b4fc593be0143edf845c7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, max=250.0), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [
        "# Compute NNGP validation accuracy for the 250 specs.\n",
        "nngp_val = [\n",
        "            evaluator.nngp_eval(spec, batch_size=BATCH_SIZE, \n",
        "                                ensemble_number=ENSEMBLE_NUMBER) \n",
        "            for spec in tqdm(nasbench_specs)\n",
        "            ]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bdpCK9NxyOTP"
      },
      "source": [
        "## Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G5onNrslr-Kq"
      },
      "source": [
        "### Ground Truth vs. NNGP Validation Accuracy Plot\n",
        "\n",
        "We plot the groud truth accuracy against NNGP validation accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "executionInfo": {
          "elapsed": 1366098,
          "status": "ok",
          "timestamp": 1602036547413,
          "user": {
            "displayName": "Jaehoon Lee",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhxHqEwdpweGC4ZrtdwaxYD0GM-tam3aLuSS-I4aQ=s64",
            "userId": "07296832508386080109"
          },
          "user_tz": 420
        },
        "id": "Gqc-3e3Fa2wm",
        "outputId": "7b8ceead-4787-48d2-d04e-86642b7c82be"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO29ebhcRbW//34yDwRCSETJyCzDFZADKArBgckBRPQKXkDQK0ZBEPEqehURr1+5VxFRUAR/iKCIzMQBAUEQZUogBAgYSAJkwhhIAmQgyUnW74+qDvvs3n26+qT7dJ+T9T5PP7332rVrrz3V2rWqapXMDMdxHMfJ06fZCjiO4zitiRsIx3EcpxA3EI7jOE4hbiAcx3GcQtxAOI7jOIW4gXAcx3EKcQPRgkiaIMkk9evm494t6T+785hO70DSFZL+Jy4fIGlmStouHmu5pO26ur+TziZrICQdI+lBSSsk/Ssuf06Smq1bNSQ9J+m9G5nHOZJ+VS+dnHQkHRQ/AH6Sk/9N0olx+cSY5su5NPMlHZRZ31HSNZIWS3pF0jOSfixpTOZY62Oh+qqkmZJOauT5mdm9ZrZzPfIq+mgxs83MbE498nc6Z5M0EJLOBC4Evge8EdgamAS8AxhQYZ++3abgRtLdNY9m0AvOcQVwvKQJnaRZAnxZ0rCijZJ2AB4EFgJ7mdnmhGd4NvDOTNKFZrYZsDnwFeAySbtu9Bk4dUOB1iuPzWyT+gFbEF7Oo6ukuwL4KfDHmP69wC7A3cAyYAZwRCb93cB/ZtZPBP6WWTeCEXom7n8xoLitL/B94EVgDnBKTN+vQK+rgPXAKmA58GVgQkz/KWAu8FfgIGB+bt/n4nkcBqwB1sY8pmfO4dvA34FXgduBkRWuz5bA74HFwNK4PCazfQTwC0LhtRS4ObPtSOBR4BVCYXZYVr9MunOAX8XlsnOM8k8DT0V9nwTeGuVnxbxL8qOifACh4P23zHHeAKwERuXOcWC8V7tnZKPitX8DMDKe97KY571An4Rn8CBgPvBj4BcZ+d+AE7PPD/A74JuZNPOBg+Lyr4DfpRwrJ1sMfKQg7VPABzLr/WLa0jW9Dvgn8HJ8xnbLvS//U3RMYC/gkXgvfgtck0lb8TkCvgOsA14jPKcXZd6lHTLv85Vx/+eBr5fuQeYafj/m/SxweCfXqvCZyWyv9KyNBW6MOryU0fMc4vObe4b7Zd637xDet1XADsBJmWPMAT6T06Hs3QE+CjycS/dF4JaNLi83NoOe9osXtJ2CwjeX7or4IryDUNMaBswCvkYoZN4db+LOmZtdzUD8HhgOjIsPU6lgnAT8Iz5oI4C/UMFAxPTP0bEgLT14VwJDgcF0YiCKHt7MOcwGdop53A2cV0GHrYCjgSHx2lxHRyPwB0JhsCXQH5gY5fvG63pwvK6jgTdXOK8NOlY4x48CC4B9AMUXbHxM/1Fgm3iMjxGM/Jvitp8A/5s5zulUKGiBy4HvZNZPAf4Ul78LXBLPrz9wANHoV3m2DiIU9G8kvOilZ6jIQOxJKNxGRHnWQPyzlL7aseJyH+AowofBzgVpzwZ+nVl/P/BUZv2T8V4PBH4IPJp7X8oMBOFdeR44I16jj8Tjl9JWe47uJvNeZd6lkoG4Ergl7jsBeBr4VOYariUU7H2BzxI+WArvUZVnpvBZi/lOBy4gPJeDgHcWvWMUG4i5wG4EY9w/XvPt4zEmEj5cSoao8N2J92MJsEvmWNOo8hGcVF5ubAY97QccB/wzJ7uP8BW4Cjgw88BfmUlzAOGF7JOR/QY4p+hBpthAvDOzfi1wVly+C5iU2XYIXTMQ22VkB9E1A/H1zPrniIVhwnXdE1gal99EqOVsWZDuZ8AFiee1QccK53gbcHqifo8CR8bl/eKLWarBTQX+vcJ+7wVmZ9b/DpwQl88lFE471PgMbrg3wP8Bv43LZQYi86z8b1zOGoh24kdGXD+V8BwvBy7LHGs9r9dyHgWOqaDXDoSPniFx/dfA2RXSDo/3Y4vM+1JkIA4kVygT3rf/qfYcFb1XmXdpB0LhvAbYNbPtM8DdmWs4K7NtSNz3jV14ZgqfNeDthI+9otr+huc39wxnDcS5VXS4uXRcOn93fkr8kCEYnKXAwFqey6Jf6/m8Gs9LwMisD9vM9jez4XFb9prMyyxvA8wzs/UZ2fMEK57KPzPLK4HNsnnn8u0K86onqUolHTsgaYikn0l6XtIrBJfD8NhWMxZYYmZLC3YdS6ildJXsOVbMS9IJkh6VtEzSMmB3gksIM3uQcG4HSXozobCZXOF4fwGGSNovthfsCdwUt32PUKu8XdIcSWd14Xz+FzhU0h6dpDkb+KykrXPylwjGGAAzuyg+xz8kfI2WWGhmw81shJntaWbXFB3EzGYR3BsflDQEOAK4GkIbnKTzJM2O9/u5uNvIKue3DbDAYskV2fB8V3mOqjEynmf2fcm/kxueZzNbGRcrPdMVnxkqP2tjgefNrD1B3yI6vLOSDpf0gKQlUYf3JegA8Evg47GTzfHAtWa2uos6bWBTNBD3A6sJvrxqZB/qhcDYXEPSOEK1E0J1dEhm2xtr0OkFws3P5puqVyV5B33iCzcqIY9UzgR2Bvaz0Dh6YOlQhId+hKThBfvNI1Shi0i5hlm9C/OSNB64jPBFvVUsNJ+IupX4JaE2eTxwvZm9VqSQma0jfMEfG3+/N7NX47ZXzexMM9uOUJh+UdJ7KpxbIWb2EqFA/3Ynaf5B8HH/d27TncCHazleAr8hnOeRwJPRaAB8PMreS/D7T4jyar3+XgBG53oHZp/vzp4j6Pw5fZHgQhqfy3tBcfLKJDwzlZ7becC4Cp0manqeJQ0EbiC0mWwddfhjgg6Y2QOE2tQBhHt1VVG6WtnkDISZLQO+BfxE0kckDZPUR9KeBB9iJUpfnV+W1D92NfwgocENQnX0w/GLaAdCY2oq1wKnSRojaUtCY1lnLAKq9QN/Ghgk6f2S+hMa7wbm8piwET0nhhFccsskjQC+WdpgZi8AtxKu8ZbxepVe/P8POEnSe+J1Hx2/4iG6P2L6NoK/ujN+DnxJ0t6xF8gO8UUfSnjxFgPEbp275/b9FcEffxzBj90ZVxN80v8Rl4n5fiAeUwTf8DqCO6dWfgDsT+gEUYlvERows0b3HOAAST+QNDrqNLJKPtW4huDi/CyZcyXc79WEWssQ4P8l5nc/wRV2WryvHyb40rP5Fj5HkYrPesZ4fye+x+MJjbNd6b5d7Zmp9Kw9RDCC50kaKmmQpHfEfR4FDpQ0TtIWwFer6DCA8I4uBtolHU64FyU6e3cgPMcXAWvN7G9duAZlbHIGAsDM/o/wIH2Z8AAuIvj3vkLwjxbts4ZgEA4nfLn8hOCL/kdMcgHBgi8ifJ3+ugaVLiP4OKcTenvcWCX9d4Gvx6rwlyro+zKhDeHnhC+qFQT/dYnr4v9Lkh6pQdcSPyQ0FL8IPAD8Kbf9eMLX3T+AfwFfiHo9RCjoLiAUqvfw+hfgNwhfSEsJBeLVdIKZXUfoBXI1wXd+M6Ex90ngfELhtAj4N0LbQXbfeYRrbYTeR50d50HC9duGYPhK7Aj8meDzvx/4iZn9BUDSrZK+1lm+mfxfIbRFjOgkzbOEr8KhGdnThPaUMcB0Sa/G81xIuJY1E437/QSD9dvMpisJ7psFhB48DyTmt4ZQyzmR0AbyMTo+39WeowuBj0haKulHBYf4POHezCG04VxN6FhQE9WemU6etXWEcmEHQrvW/HiOmNkdhGv4GPAwoZNKZzq8CpxGMHpLCTWByZntnb07EJ6P3emagSyk1EjnOJscki4n+Oe/3mxdHGdjkTSY8DH2VjN7ph559vTBRo7TJWKD84cJffQdpzfwWWBKvYwDuIFwNkEkfZvQL/+70XXjOD0aSc8RGrM/VNd83cXkOI7jFNHQRmpJhykEB5tV1Edc0nhJd0p6TCEo15jMtnWxT/Kjkir1UXccx3EaRMNqELHf/dOEYeHzgSnAsbG3QCnNdYR+5b+U9G7gJDM7Pm5bbiHAWBIjR460CRMm1PMUHMdxej0PP/zwi2Y2qmhbI9sg9iUMc58DIOka4uCbTJpdCd1NIYxYvbmrB5swYQJTp07t6u6O4zibJJIqRm5opItpNB2Hkc+nPCzFdF4fCXoUMEzSVnF9kKSpcdh5YcOLpJNjmqmLFy+up+6O4zibPM0eKPclYKKkaYTIhQsIo1EhROVsIwwW+aGksiHmZnapmbWZWduoUYU1JMdxHKeLNNLFtICO8YXGkIuRYmYLiTUISZsRwtMui9sWxP85ku4m9FffmCBvjuM4Tg00sgYxBdhR0raSBgDHkIuYKWlkJhbQV4lD5GP8noGlNIQ5GbJtF47jOE6DaZiBiOFvTyXEGHqKEH52hqRzJR0Rkx0EzJT0NGHaz+9E+S7AVEnTCY3X52V7PzmO4ziNp9cMlGtrazPvxeQ4rcXCZau45J7ZTJ+3jD3GDmfSxO3ZZvjgZqvlZJD0cGzvLcNDbTg9Ci9weg4Ll63i0Av+yvLV7Rjw2PyXuemRBdx2xoF+z3oIbiCcliCl4F+4bBWHX3gvK1a3077emLHwFW55dCG3nn5Ah7T1NiLNym/a3KWcdcNjzF2yknEjhnDe0W9hr3FbNlS3enL+7TN5dfXrE60Z8Orqds6/fSbn//ueNefXyufaW3EXk9N08l+aAjYb2K/sS/PsW57g6gfn0r7+9We2fx9x7H7jOPfI3TfklTUi/fqIoQP7lRmRUtpajVJn+aUW6AdfcA8rVq/bIBs6sC93nDGxQ37T5i7lqJ+UT01y0+f235DnwmWreM8P7mbVmtfnKBo8oA93fvGgLp1rLelS2Pvbd/DSijVl8q2GDuDhbxxcU14Ll63i0B/+leWvZZ6RQf247QvltZFmGZKeasA6czE1exyE08tZuGwVZ9/yBEde9DfOvuUJFi5bVZam9KVZKvazX5pZps9b1sE4AKxdb0yft2zD+iX3zN5QmAO0rzdWrm7nkns69pAuFThX3f880+e/zFX3P8+hP/xrmX6X3DOb5a+t7ZDfitfWluVXKtBnLlrOqrXrmbloOUf95D6mze04Lfe3fz+jg3EAWLF6Hd/+/YwOstOvmVZ2nfLyr934eAfjALBqzXq+duPjXTrXkjG8+sG5TJ//Mlc/OJfDL7y38J6l3Nf2dcWT6+Xlyc/Ia7ln5LXyZ6SWc0hl2tylHHrBPezyjVs59IJ7yu5po47bCriBcBpG6ktz98ziUfB5+R5jh9OvT8cpkPv3EXuMfX0WzhQjAukFzpRnl7AuV8lutyDPctYNjxWeQ15+x4xFheny8rlLiguWrPyvzxRft7w89VxrMq4X5AzOBeUGJ2Xi9NRn5K5//Kswr7w89RxKx65mmFINfy3H7Um4gXAaRr1fmkkTt2fowH4bjET/PmLIwH5Mmvj6IPtKVfq8/M6nigucvHx9BRdsXv7cSysK0+Xl7RVKzbxcxck6yNdXyCsvTy1cazKuCTW+19Z2rCkVyYuekRUFz8jKNcV55eWp55BqmFINf+pxexpuIJyGkfrSHLRzcZiUvHyb4YO59fQD+Ph+49hjzBYcu9+4sraAmf98pTCvvHxVhcIrL+/Tp7iozssH9+9bmK6SvBpjRxQbukryzkgtXItqaP1yNTRIr/GlGNcpzy0pe0ba1xtTnutYQ0u9vju8oTgAdF6eapjmLllZmF9enlK77Ym4gdiESKlS15PUl/XMQ3Zm2KB+G76OBQwb1I8zD9m5bN9thg/m3CN355ZT38m5R+5e0ED5WuEx8/IhA4oLnLx8nwkjCgvNfSaM6CAbOWxQYX6jKsir8Y3371pVPrTCOeTlA/sWv+Z5+VF7jS4srI/aKx9jM4324iaIDvLVFQx1Xv6eXd5QmK6SvBqphmnciCGF++flKbXbnogbiBamngV6KzeibTN8MLd94UCOf/t49hizBce/fXxh75QUBvUvfqTz8ne/ubhgycuLXvyhBS/+HmO2KMzvLTl53thUkt8768UyN5OivMTwwf0L88rLR24+sDBdXv6rB4qjPufl+247ojBdXp7iJlu6Ym1hmrz8zEN2ZtjA3EfEwPKPiFn/Wl6YX16+uoL1ysvPO/otheny8pTabU/Ex0G0KKl9/lPprD2g1EW03qS+rPB6zWBj2WJIf5atai+UZznzkJ25/clFZd0m8wXONsMHc8VJ+5R1X83fg5Vryo9ZJD9416259Yl/lqU7eNetO6zfN/vFskZei/ISi18tri3l5QP7VahB5OSprqPU2tfYEYMLG9uzbrJUF942wwdz2xkHVu1GusfY4cxY+EpZV+i8q2fJ8vLut0XyrTcfxNCBfcu6JW+9eXnNsF7PcCvhNYgWpd4NvM1oRGuEX7Zaraq9QitwXp5aa1m4bBUn/mIKsxevYNXa9cxevIITfzGl7LgPPVve9bFI/o0P7MrQgR0L0qED+/KND3R0Kb346urC/LLyNRVcOHn5PhNG0DdXDvcVZW6yVJ58obidJy9PcZOltj9BdfcipLt6VKF6k5dfcs9sVq/teEHXrF3f43snpeIGokWpd4HejEa0evtlU9xkL7xS/FVdJE8pcOptqLcZPpg7zpjICdEwnfD28WWD5ABWrS0u/bPyCt60MvlRe40u66q7zihrW0gtrFesLq4t5eX3znqx3DD16egmO+5t4wvzqiSvRqqrZ9jA4lpQXt7qvZMa3a7oLqYWJbWqnMqkidtzy6MLNxR23dGIVnpZ6zW6NMVNltr1M5VaemLd8EiH6U42yPOkuCIG9hVF5fDATIk7oF9f1hb0UBrQr2Mh11nbQnak95mH7MytT/yzQ++mIQP6lrndKhVCefn0ecvKDdN6Oly7m6YtoK/okK6fgryrYUVSru+rrxU3jufl9X4P60m93dBFuIFoUepdoNe7sK7luCl+2ZSXP6WwHjKgDysLfC9DBnStsrzN8EFMn/9yoTzLcW8bX2gguvolrAo+kKw8tatuatvColdeK+v6unLNOha98lqHe5HSOwmKC9d8t9kiI9JuVBy3kFIYpjxLqW0fzfiwSqU72hXdxdSiNKJXRIpLpRmk9rBKcZOl9uxJZeai4ob2vDy1BxCkuQX6VeiaWkleD1IHhaWS0m021fVZyyjvlGeplrE3V5y0D9uPGsrg/n3YbtRQrjhpn5Z4d7rD/eUGooVp1QK93qS+/JMmbs/gAX07dHUcNKBvh6+5pSsrdJusIK9Gqjsl9Su9ngXYsIHFDoC8PLUwfL7CoLC8fLMKtbG8PMVoprZTpRaGqc9SatvHwmWrOOHyh3g6htp4etFyTrj8oZboHt4d7YpuIJymU9OXUFHfzwypA5tSGVShi2gleTVSC7AzD9m5sLdTtj3gvbmusZXkqQMRU8/1go/tVZguL08xmqk15dTCMPVZKrV9ZCm1fWRJjWPVDLpjcJ4bCKehpETCrCU8wso1HV/WVWs6Fq6pA5tSGblZhUFmOXnqV3o93QKpBX9ql97Ucz14tzdy2fF7M3xwOPbwwf247Pi9OXi3N3ZIlxrHqp7dV2sxJCltH6k1w2bQHYPzvJHaaRj5OQ1KkTCzcxrUQkpk1b3Gbcllx+/Nf10/nZdXtbPF4H587yN7FB4vpTFzj7HDmbW4PBBfvsBJbaTe4Q2bFTZ6543h+bfPLAwLnp1sp1Tw16tnz/47jGTO4hVk25r7Rnme3UZvwRF7jt5w3N1Gl48kHzaoX6Frb9igjsVOyn1I7WSR2qjcyr2TaqHRg/PcQDhdIuWl7qzR87YzJm5YTw6PUGF+gax84bJVfOn6x1ixeh0GLF+9ji9d/xi3jt6ibNa5enYRvGnaAkRHj1cfirtrppD65VrPXmKTJm7PTdMWdBhdPmRQeeGaeu1eLRjRnpfXch9SzrXehqSW7su9EXcxOTWT2tC6MZEwi6KILi2YnSwvT/Xxp04ElGq87ptVHhpjfZRnSR2FXE9qicO1fr11cOGtLxhAknqNU7qSNmIehRSXVap7JjUGVG/FDYRTM6mhklMbjFOjiPapMDYgK0/18adOBJTq035xeYXQGDl5apC4WkJQVCO1ED7/9pmsyI2DWLFmXZdm9ks9h2aOVE41JLedkWu/OaNrgSR7Im4gnJpJDZWc2mBc1KOkb0GPkpQCJz+ArZI8tQE1tXH0tQoFf16eGiSulhDo1UgthOs5s1/pHKr1xKqlq2Z3h6svsal0Ny+ioQZC0mGSZkqaJemsgu3jJd0p6TFJd0sak9n2CUnPxN8nGqmnUxtFbocieanBON/bJe+Tn/Jc+df8OqPM4KT0XU8d2FZLFNEUV8T4CrWlSvJq1DMEer37y9fSvbIP6mDk+uSCgKfm1crh6nszDTMQkvoCFwOHA7sCx0rKh3f8PnClmb0FOBf4btx3BPBNYD9gX+Cbkmpv6XMaQoqrB15vMF6eazDOv9SpBuemaQsK2yqyNY0XKhQYeXnqRECQ9gWZWltKDZUNIfTFg3Ne4ulFr/LgnJdYVCEQYTVSC+F6zuwHwbW1au26Dm0ar61d18G1VUtevXHO51ankb2Y9gVmmdkcAEnXAEcCT2bS7Ap8MS7/Bbg5Lh8K3GFmS+K+dwCHAb+pt5KpAcBame4+h322HcHTi14tC7C2T27CmNRYMakGp8hV0p5zlbxp+GDmFHRLfVMXe7GksvXmgxg6oG8HH/7QAeXzBqS2VaR2Ea5nF9EzD9mZP834Z9ncB53N7NcZqa6teubl1JdGuphGA/My6/OjLMt04MNx+ShgmKStEvdF0smSpkqaunhx7QNXekO1tRnnMGni9mw2qH/HWdYG9e9yeIR9ti2fr6DI4KS4SnbeunjQXV5e7xg7l9wzu6yheU17+bwBa/K+tArylLhItdz7VD96H+VcQpUmTkignq6t3jrnc6vT7EbqLwETJU0DJgILgOIQlQWY2aVm1mZmbaNGNa53RyvTjHOod3iEVIOTEospdU7q1ImASmmrNY6mGsOBeUtYQZ7SRbje9/6Se2azak3OJbRmXZfzq2coiN4653Or00gX0wJgbGZ9TJRtwMwWEmsQkjYDjjazZZIWAAfl9r273gr2hmprs84hxS2Q6sapKRR5lVhMqSNkU91fqQO5Uo/77l2Kpxx99y4d4yeNGzGksME920W43ve+3vnVM8R8s8LVb+o0sgYxBdhR0raSBgDHAJOzCSSNlFTS4avA5XH5NuAQSVvGxulDoqyu9IZqayufQyPcONUaPSdN3J6BuSnVBvTv0/DooKlfuN/4wK4MzkU9HTygT9mUoymN3qkxrFKpd36waXcR7Q00zECYWTtwKqFgfwq41sxmSDpX0hEx2UHATElPA1sD34n7LgG+TTAyU4BzSw3W9aQ3VFtb+RxS3TipvvSUQn3RK68VxjDK9wDaY+zwsoe/L+UxlmppaE01hnm/fpGff69xW3LT5/Zn5603Y3D/Puy89WZdjmEFzRtDUC96Q3thT6ShbRBm9kcz28nMtjezUuF/tplNjsvXm9mOMc1/mtnqzL6Xm9kO8feLRujXHdEQG00rn0MtYS9S0qXUlk759cOFuuTlB+wwkvzQtnVRXusxId0YdhaEL89e47bktjMm8tS3D+e2MyaWGYcnF1YI25GTpxauqfk1g97QXtgT2eSD9TU6GmJ30KrnkPr1nZquKJhcWSP1y8XdSPPy71eI5//922d2CFudckxIb9O46x//KjxuJXlnpI4GL8WdKnWUysadyuqWml8z6A3thT2RTd5AOI0jteE2NQQ2ULWROpXnXiofK1FRnnDM1AIsP+dzNXlnpI4GT407lZofdP/Ym1rCc/eGsU2tQrO7uTrdSD390Cl5TZq4PYP757ql9i//+k4lZcKgVAb2Kx7RnJenNIxDKMCKxnLkC7BBFeaUriTvjNTR4Kk1g9T86t0ekPospXQ+8LaK+uIGYhOhni9OTXnlPz4LPlJTQ2qnfAlvPWxAYV55eaXxX3l5as3gqL1GF+qWj0i7VQX9Ksk7o6iDwtCCDgqpNYPU/OrZHpD6LKV2PvC2ivriBmIToZ4vTi2NzykDr1IbglO+hN86vriXT17er0/xo5+Xp+qWEicKanPjVCO1g8Kub9q8cP+8PLUnVj3bA1KfpZSR5fXWzfE2iCR6g0+zni9OIxqfUwbUpRSuqSOpU2cKS9UtJU4UQN8KVZcieWqcpXp1UCj1xCqda6knVlcHBaaQ+ozUMvlUb5hKtFXwGkQVeotPs56DoFK/qlOPWc8v4VrmKkiZKazeYUVSaxD1fO5SXXj1HhSYQup1S518qpXHBfVE3EBUIXVqyk2JRryE9Rpxm6rbNsMHc+Wn9mWnOBBtp60348pP7VtxVrFquqUeN9XdU0+XYGohXMugwNSxN9UaoFOvW2o49VYeF9QTcRdTFVK7CLY6qV+RKZR81Wfd8Bhzl6xk3IghnHf0W8pewnoeMzW/1Jg9qe6UVFKvSSr1dAmmuslqcc+kuLZS4lil3q/SyPL89S0aWd6q44J6Im4gqtDKg4dqoZ6+2Wb4qmvJL6WASB3YlkrqNUk1mvW8dqmFcL3nyEi9xqkFemlkudN9uIupCvXsddJM6ukWaoavut751bu3Sz3DhUD9r12Km6ze7hnvUdTz8RpEFfaZMIJnFi3v8KBXmpqylalnuORafdX16gFWz/xqGr2dQL17bDUrvHU93TPeo6jn4waiCvWudjeTer389fZV10Kr+pdrcX+lFvyteq6p9KZ3Z1PFDUQVfKKSciZN3J6bHlnA8tWZAHYbEUKjGdS7Ab2WwrCnF/yp+LvT86lqICR9EPiDmeWjI28ybCovdE0khNBoZert/vDCsBh/d3o2KTWIjwE/lHQDcLmZ/aPBOjktziX3zGbl6lzQvI3oAdQMGuH+8MLQ6W1UNRBmdpykzYFjgSskGfAL4Ddm9mqjFXRaj94wNqQnfPH3hhAvTs8mqQ3CzF6RdD0wGPgCcBTwX5J+ZGY/bqSCTuvRW8aGtPIXf8ogM8dpNFXHQUg6QtJNwN1Af2BfMzsc2AM4s7HqOa1Ibxkb0sp42GqnFUipQRwNXGBmf80KzWylpE81Ri2nlektY0NaGR9k5r96v6sAABumSURBVLQCKSOpzwEeKq1IGixpAoCZ3dkQrZyWJnViGafrpI64dpxGkmIgrgOyXVzXRZmzieIRMxuPh612WoEUF1M/M1tTWjGzNZJqnx/R6VW0cgNvb6An9LJyej8pBmKxpCPMbDKApCOBF1Myl3QYcCHQF/i5mZ2X2z4O+CUwPKY5y8z+GF1YTwEzY9IHzGxSyjEdp7fgRthpNikGYhLwa0kXEcbLzgNOqLaTpL7AxcDBwHxgiqTJZvZkJtnXgWvN7KeSdgX+CEyI22ab2Z7JZ+I4juPUlZSBcrOBt0naLK6nBqvZF5hlZnMAJF0DHAlkDYQBpam0tgAWJubtOI7jNJikgXKS3g/sBgxSnFzdzM6tsttoQm2jxHxgv1yac4DbJX0eGAq8N7NtW0nTgFeAr5vZvSm6Oo7jOPUhZaDcJYR4TJ8nuJg+Coyv0/GPBa4wszHA+4CrJPUBXgDGmdlewBeBq2O4j7xuJ0uaKmnq4sWL66SS4ziOA2ndXPc3sxOApWb2LeDtwE4J+y0AxmbWx0RZlk8B1wKY2f3AIGCkma02s5ei/GFgdtExzexSM2szs7ZRo0YlqOQ4juOkkmIgXov/KyVtA6wF3pSw3xRgR0nbxm6xxwCTc2nmAu8BkLQLwUAsljQqNnIjaTtgR2BOwjEdx3GcOpHSBvE7ScOB7wGPEBqWL6u2k5m1SzoVuI3QhfVyM5sh6Vxgauw2eyZwmaQzYr4nmplJOhA4V9JawiC9SWbWc0KFOo7j9AJknUTgjO0BbzOz++L6QGCQmZVP5ttk2trabOrUqc1Woyl4WOhNF7/3zsYi6WEzayvc1pmBiDtPi43FLc2maiDyYaH7xbhIrRL6opULsFbWDarr18x7P23uUs664THmLlnJuBFDOO/ot7DXuC27lFer34fezsYaiO8D9wM3WrXETWRTNRBn3/IEVz84t2zqzGP3G9f0UbitbLxaWbdU/Wq59/UshKfNXcpRP7mvTH7T5/av2UjUch/ckDSGzgxESiP1ZwjB+VZLekXSq5JeqauGTpdp5bDQrTynQSvrBmn6pd77UiF89YNzmT7/Za5+cC6HX3gvC5et6pJuZ93wWE3yzki9D/U+ByeNqgbCzIaZWR8zG2Bmm8f1sjEJTnNo5bDQrWy8Wlk3SNMv9d7X2xjOXbKyJnlnpN6HVjfovZWUgXIHFv26QzmnOq0cFroRxmvhslWcfcsTHHnR3zj7lie6/AXZyoYV0vRLvff1NobjRgypSd4Zqfeh1Q16byXFxfRfmd83gN8RQmQ4LUArz81Qb+NVTzdDKxtWSNMv9d7X2xied/RbapJ3Rup9aHWD3lup2khdtoM0FvihmR3dGJW6xqbaSN3q1LNhsd4N8q3e6Fkv/fINwaVCeGM+JLq7F1MjzsEJbFQvpoLMBMwws13roVy9cAPR+znyor8xfX75EJw9xmzBLae+swka9Rxa3Rim0BvOoRXpzEBUHUkt6ceEUc4QXFJ7EkZUO063ssfY4Tyx4GXWZb5p+gl3MyTQGyYf6g3n0NNICbWR/SxvB35jZn9vkD6OU5Gj9hrNlfc/30HWbkHuOE79STEQ1wOvmdk6CDPFSRpiZrX3aXOcjeCmaQvo10cd2iD69RE3TVvQZf93K+MuFafZpBiIOwkT+ZRmkhsM3A7s3yilHKeIoq6O7S3U1bGeBXq+UXbGwle45dGF3ijrdCsp3VwHZacZjcu1d3h2nI2klbs61nukrw8Mc1qBFAOxQtJbSyuS9gZ8fLvT7bTy2IV6F+g+MKyceg2SdNJJcTF9AbhO0kLClKNvJExB6jjdSmlgWCv65etdoO/whs0Ku/Tu8IbNupRfT8ddbs2hqoEwsymS3gzsHEUzzWxtY9VynGJatavjHmOHM2PhK2WD+FrB/dUb6KyG1orPQ28hJRbTKcBQM3vCzJ4ANpP0ucar5jg9h3q7v558oThgciV5b8ddbs0hpQ3i02a24S6Y2VLg041TyXF6HvWOibV+fXGEg0ry3k4rd1DozaS0QfSVpNJkQZL6AgMaq5bj9Dzq6f7qI9Uk7+1Mmrg9tzy6sCwWUyt0UOjNpBiIPwG/lfSzuP6ZKHMcp0Hss+0Inl70allYkX22HdE8pZpIK3dQ6M2kGIivEIzCZ+P6HcDPG6aR4zj+xVxAq3ZQ6M3UHM21VfFork5vw0NtON3BxkZz3RH4LrArMKgkN7Pt6qah4zhl+Bez02xSejH9AvgpIZLru4ArgV81UinHcRyn+aQYiMFmdifBHfW8mZ0DvD8lc0mHSZopaZakswq2j5P0F0nTJD0m6X2ZbV+N+82UdGjqCTmO4zj1IaWRerWkPsAzkk4FFgBVx/vH7rAXAwcD84Epkiab2ZOZZF8HrjWzn0raFfgjMCEuHwPsBmwD/FnSTqWQ447jOE7jSalBnE6I3noasDdwHPCJhP32BWaZ2RwzWwNcAxyZS2PA5nF5C2BhXD4SuMbMVpvZs8CsmJ/jOI7TTSTFYoqLy4GTash7NDAvsz4f2C+X5hzgdkmfB4YS5p0o7ftAbt+yacMknQycDDBu3LgaVHMcx3GqkVKDaCTHAleY2RjgfcBV0Z2VhJldamZtZtY2atSohinpOI6zKZLSBtFVFgBjM+tjoizLp4DDAMzsfkmDgJGJ+zqO4zgNpJE1iCnAjpK2lTSA0Og8OZdmLvAeAEm7EMZZLI7pjpE0UNK2wI7AQw3U1XEcx8mRMlBuFCF664RsejP7ZGf7mVl77PV0G9AXuNzMZkg6F5hqZpOBM4HLJJ1BaLA+MQYFnCHpWuBJwviLU7wHk+M4TvdSNdSGpPuAe4GHgQ2FtJnd0FjVasNDbTiO49TORoXaAIaY2VfqrJPjOI7T4qS0Qfw+O8LZcRzH2TSoWIOQ9CqhXUDA1yStBtbGdTOzzSvt6ziO4/R8KhoIMxvWnYo4juM4rUVVF5OkO1NkjuM4Tu+iMxfTIEL4i5GStiS4liDETioLe+E4juP0LjrrxfQZ4AuEaKqPZOSvABc1UinHcRyn+XTWBnEhcKGkz5vZj7tRJ8dxHKcFSBkH8bKkE/JCM7uyAfo4juM4LUKKgdgnszyIEDvpEcLUo47jOE4vJWU+iM9n1yUNJ0z+4ziO4/RiuhLNdQWwbb0VcRzHcVqLlGiuvyOMqIZgUHYFrm2kUo7jOE7zSWmD+H5muR143szmN0gfx3Ecp0Xo1EBI6gucY2bv6iZ9HMdxnBah0zaIOEnPeklbdJM+juM4TouQ4mJaDjwu6Q5CAzUAZnZaw7RyHMdxmk6Kgbgx/rJ0Pg2d4ziO0+NJMRDDY9iNDUg6vUH6OI7jOC1CyjiITxTITqyzHo7jOE6L0Vm472OBjwPbSpqc2TQMWNJoxRzHcZzm0pmL6T7gBWAkcH5G/irwWCOVchzHcZpPZ+G+nweeB97efeo4juM4rUJXYjElI+kwSTMlzZJ0VsH2CyQ9Gn9PS1qW2bYus21yfl/HcRynsaT0YuoScRT2xcDBwHxgiqTJZvZkKY2ZnZFJ/3lgr0wWq8xsz0bp5ziO43ROI2sQ+wKzzGyOma0hhAg/spP0xwK/aaA+juM4Tg101ovpcToZEGdmb6mS92hgXmZ9PrBfhWONJ4QQvysjHiRpKiFA4HlmdnPBficDJwOMGzeuijqO4zhOLXTmYvpA/D8l/l8V//+jAXocA1wfYz+VGG9mCyRtB9wl6XEzm53dycwuBS4FaGtr89HdjuM4daRaLyYkHWxm2baBsyQ9ApQ1OudYAIzNrI+JsiKO4XVDVDr+gvg/R9LdhPaJ2eW7Oo7jOI0gpQ1Ckt6RWdk/cb8pwI6StpU0gGAEynojSXozsCVwf0a2paSBcXkk8A7gyfy+juM4TuNI6cX0KeDyGPJbwFLgk9V2MrN2SacCtwF9gcvNbIakc4GpZlYyFscA15hZ1kW0C/AzSesJxui8bO8nx3Ecp/GoY7ncScI4J4SZvdxQjbpIW1ubTZ06tdlqOI7j9CgkPWxmbUXbUuakHggcDUwA+kkCwMzOraOOjuM4TouR4mK6BXgZeBhY3Vh1HMdxnFYhxUCMMbPDGq6J4ziO01Kk9Ea6T9K/NVwTx3Ecp6VIqUG8EzhR0rMEF5MASxhJ7TiO4/RgUgzE4Q3XwnEcx2k5UgyEh7BwHMfZBEkxEH8gGAkBgwhB9WYCuzVQL8dxHKfJVDUQZtahgVrSW4HPNUwjx3EcpyWoeT4IM3uECmG7HcdxnN5DykjqL2ZW+wBvBRY2TCPHcRynJUhpgxiWWW4ntEnc0Bh1HMdxnFYhpQ3iWwCSNovryxutlOM4jtN8qrZBSNpd0jRgBjBD0sOSdm+8ao7jOE4zSWmkvhT4opmNN7PxwJlR5jiO4/RiUgzEUDP7S2nFzO4GhjZMI8dxHKclSGmkniPpG8BVcf04YE7jVHIcx3FagZQaxCeBUcCNhN5LI0mYctRxHMfp2XRag5DUF7jRzN7VTfo4juM4LUKnNQgzWwesL81H7TiO42w6pLRBLAcel3QHsKIkNLPTGqaV4ziO03RSDMSN8ec4juNsQqSMpP5ldyjiOI7jtBYV2yAkHSnplMz6g5LmxN9HUjKXdJikmZJmSTqrYPsFkh6Nv6clLcts+4SkZ+LvE7WemOM4jrNxdFaD+DJwTGZ9ILAPYZDcL4DrO8s49oC6GDgYmA9MkTTZzJ4spTGzMzLpPw/sFZdHAN8E2giTFT0c912afmqO4zjOxtBZL6YBZjYvs/43M3vJzOaSNpJ6X2CWmc0xszXANcCRnaQ/FvhNXD4UuMPMlkSjcAdwWMIxHcdxnDrRmYHYMrtiZqdmVkcl5D0ayBqY+VFWhqTxhKlM76plX0knS5oqaerixYsTVHIcx3FS6cxAPCjp03mhpM8AD9VZj2OA6+O4i2TM7FIzazOztlGjUmyW4ziOk0pnbRBnADdL+jjwSJTtTWiL+FBC3guAsZn1MVFWxDHAKZn1BcBBuX3vTjim4ziOUycqGggz+xewv6R3A7tF8R/M7K5K++SYAuwoaVtCgX8M8PF8IklvJriz7s+IbwP+n6SSm+sQ4KuJx3Ucx3HqQMo4iLt4vW0gGTNrl3QqobDvC1xuZjMknQtMNbPJMekxwDVmZpl9l0j6NsHIAJxrZktq1cFxHMfpOsqUyz2atrY2mzp1arPVcBzH6VFIetjM2oq2pYT7dhzHcTZB3EA4juM4hbiBcBzHcQpxA+E4juMU4gbCcRzHKcQNhOM4jlOIGwjHcRynEDcQjuM4TiFuIBzHcZxC3EA4juM4hbiBcBzHcQpxA+E4juMU4gbCcRzHKcQNhOM4jlOIGwjHcRynEDcQjuM4TiFuIBzHcZxC3EA4juM4hbiBcBzHcQpxA+E4juMU4gbCcRzHKcQNhOM4jlOIGwjHcRynkIYaCEmHSZopaZaksyqk+XdJT0qaIenqjHydpEfjb3Ij9XQcx3HK6deojCX1BS4GDgbmA1MkTTazJzNpdgS+CrzDzJZKekMmi1Vmtmej9HMcx3E6p5E1iH2BWWY2x8zWANcAR+bSfBq42MyWApjZvxqoj+M4jlMDjTQQo4F5mfX5UZZlJ2AnSX+X9ICkwzLbBkmaGuUfKjqApJNjmqmLFy+ur/aO4zibOA1zMdVw/B2Bg4AxwF8l/ZuZLQPGm9kCSdsBd0l63MxmZ3c2s0uBSwHa2tqse1V3HMfp3TSyBrEAGJtZHxNlWeYDk81srZk9CzxNMBiY2YL4Pwe4G9irgbo6juM4ORppIKYAO0raVtIA4Bgg3xvpZkLtAUkjCS6nOZK2lDQwI38H8CSO4zhOt9EwF5OZtUs6FbgN6AtcbmYzJJ0LTDWzyXHbIZKeBNYB/2VmL0naH/iZpPUEI3ZetveT4ziO03hk1jtc921tbTZ16tRmq+E4jtOjkPSwmbUVbfOR1I7jOE4hbiAcx3GcQtxAOI7jOIW4gXAcx3EKafZAOccBYOGyVVxyz2ymz1vGHmOHM2ni9mwzfHCz1XKcTRo3EE7TWbhsFYdfeC8rVrfTvt6YsfAVbnl0IbeefoAbCcdpIu5icprOJffM3mAcANrXGytXt3PJPbOr7Ok4TiNxA+E0nenzlm0wDiXWrjemz1vWJI0cxwE3EE4LsMfY4fTrow6y/n3EHmOHN0kjx3HADYTTAkyauD1DB/bbYCT69xFDBvZj0sTtm6yZ42zaeCO103S2GT6YW08/wHsxOU6L4QbCaQm2GT6Yc4/cvdlqOI6TwV1MjuM4TiFuIBzHcZxC3EA4juM4hbiBcBzHcQpxA+E4juMU0mtmlJO0GHi+2XpERgIvNluJTmhl/Vy3rtHKukFr67ep6zbezEYVbeg1BqKVkDS10hR+rUAr6+e6dY1W1g1aWz/XrTLuYnIcx3EKcQPhOI7jFOIGojFc2mwFqtDK+rluXaOVdYPW1s91q4C3QTiO4ziFeA3CcRzHKcQNhOM4jlOIG4gakXSYpJmSZkk6q2D7gZIekdQu6SO5beMk3S7pKUlPSprQQrr9n6QZUbcfSVJ+/wbr9sV4TR6TdKek8Zltn5D0TPx9op56bax+kvaUdH+8do9J+lir6JbZvrmk+ZIuaiXdGv0+1EG/Zr8TkyQ9LulRSX+TtGtm21fjfjMlHVpPvTpgZv5L/AF9gdnAdsAAYDqway7NBOAtwJXAR3Lb7gYOjsubAUNaQTdgf+DvMY++wP3AQd2s27tK1wP4LPDbuDwCmBP/t4zLWzbhvlbSbydgx7i8DfACMLwVdMtsvxC4GrioVa5bo9+HOtzXVngnNs8sHwH8KS7vGtMPBLaN+fSt57Ur/bwGURv7ArPMbI6ZrQGuAY7MJjCz58zsMWB9Vh6tfz8zuyOmW25mK1tBN8CAQYQHdSDQH1jUzbr9JXM9HgDGxOVDgTvMbImZLQXuAA6ro24bpZ+ZPW1mz8TlhcC/gMJRqd2tG4CkvYGtgdvrqNNG69YN78NG6UdrvBOvZFaHRp2I6a4xs9Vm9iwwK+ZXd9xA1MZoYF5mfX6UpbATsEzSjZKmSfqepL6toJuZ3Q/8hfD1+wJwm5k91UTdPgXc2sV9u8LG6LcBSfsSCpTZraCbpD7A+cCX6qhPXXSj8e/DRunXKu+EpFMkzQb+Dzitln3rgRuI7qMfcADhZd2HULU8sZkKlZC0A7AL4etpNPBuSQc0SZfjgDbge804fjUq6SfpTcBVwElmlq+hNUu3zwF/NLP5zdAnS4FuLfU+5PVrlXfCzC42s+2BrwBf7+7ju4GojQXA2Mz6mChLYT7waKxStgM3A29tEd2OAh6I1fzlhK+ot3e3bpLeC/w3cISZra5l3ybqh6TNgT8A/21mD7SQbm8HTpX0HPB94ARJ57WIbo1+HzZWv5Z4JzJcA3yoi/t2nUY0bPTWH+GrZw6hYajUsLRbhbRX0LEhuG9MPyqu/wI4pUV0+xjw55hHf+BO4IPdqRuwF8E1s2NOPgJ4ltBAvWVcHtHd97UT/QbE6/WFZj1zlXTLpTmR+jdSb8x1a+j7UAf9WuGd2DGz/EFgalzejY6N1HNoUCN13TPs7T/gfcDT8aH67yg7l/D1AaG6PB9YAbwEzMjsezDwGPA4oZAe0Aq6xZf1Z8BTwJPAD5pw3f5MaAR8NP4mZ/b9JKEhbhbBhdOM+1qoH3AcsDYjfxTYsxV0y+VxInU2EHW4rw19HzbyvrbCO3EhMCPq9RcyBoRQ45kNzAQOb8Q7YWYeasNxHMcpxtsgHMdxnELcQDiO4ziFuIFwHMdxCnED4TiO4xTiBsJxHMcpxA2E0y1IMknnZ9a/JOmcuHyOpJWS3pDZvjyzvLWkqyXNkfRwjJ56VNx2kKSXY8TLpyR9sw66TpD0RFxuk/SjCumekzSySl5fy63ft7H6OU534QbC6S5WAx/upEB9ETgzL4whlm8G/mpm25nZ3sAxZALSAfea2Z6EUAnHSarbiFwzm2pmp1VPWZEOBsLM9t9IlboVSf2arYPTPNxAON1FO2F+3TMqbL8c+JikETn5u4E1ZnZJSWBmz5vZj/MZmNkK4GFgh6xc0jWS3p9Zv0LSR2JN4V6FOTIekVRWeMcayu/j8lZx/oIZkn4OKJPu5li7mSHp5Cg7Dxgcaze/jrLl8V8xQN0TMeb/xzLHu1vS9ZL+IenXRfMQSPq0pCmSpku6QdKQKN9a0k1RPr10TpJOUJjzYLqkq7LXIZPn8owO90qaTBgkVnh+UX5YvHbTFeZT6KMwb8eouL2PwrwF9Yxw63QXjRqB5z//ZX/AcmBz4DlgC0KQtnPitnPi+tnAt0rp4/9pwAWd5HsQ8Pu4vFXMPx+y4Cjgl3F5ACES5mBgCDAoynfk9VAGE4AnCvL/EXB2XH4/IfzyyLg+Iv4PBp4AtsqeR/Y6xP+jCaHL+xLCcc8F3hSP9zKhhtSHMA/BOwvOe6vM8v8An4/LvyWG/Yh5b0EIzfB0ga5X0DHkyvLMOa8Ats1sKzs/QljzeaV0mTTfzOhwCHBDs58//3Xt5zUIp9uwEN/+Sl4PW5znR8AnJA2rlIeki+PX6pSM+ABJ0whzHpxnZjNyu90KvEvSQOBwgrtqFSHGzmWSHgeuI0zE0hkHAr+K5/IHYGlm22mSphPmFBhLMDid8U7gN2a2zswWAfcQQqEAPGRm8y1EhX2UYLDy7B6/8h8H/oNgBCDUuH4adVxnZi9H2XVm9mKUL6miW0mHZ6uc39sI1/LZXL6XAyfE5U8S4iw5PRD3LzrdzQ+BRygoNMxsmaSrgVMy4hmEr+1SmlNiO8bUTJp7zewDlQ5oZq9Jupsw+dDHCJExIbi7FgF7EL7WX+vKCUk6CHgv8HYzWxmPNagreUVWZ5bXUfyeXgF8yMymSzqR8NVfK+1EN7PC3BEDMttWlBZqPT8zmydpkaR3Eyay+Y8u6Oa0AF6DcLqV+JV5LWFyliJ+AHyG1wvFu4BBkj6bSTOkC4f+LXASYQ6CP0XZFsAL8Uv9eIJLpjP+CnwcQNLhhOiypXyWxsLzzYQv6xJrJfUvyOteQptL3+ifPxB4qIbzGQa8EPPOFsB3EqbOJOa9BeEaflTSVlFeaud5Dtg7Lh9BqFEVUen8HgAOlLRtLl+AnxNqW9eZ2boazstpIdxAOM3gfKCwN1N0g9xECGWMmRkhDv5ESc9Kegj4JWEClVq4HZgI/NnCFI8APyG4tKYDbybz1VyBbxEKxBnAhwntBhAMTj9JTwHnEQrOEpcCj5UaqTPcRIhkOp1QgH/ZzP5Zw/l8A3iQMG/yPzLy0wnutMcJDfa7Rpfbd4B74rn+IKa9jHBdpxPmOqh0/oXnZ2aLgZOBG2Mev83sM5kwz7S7l3owHs3VcZy6I6mN0LmgKTMTOvXB2yAcx6krks4iuLm87aGH4zUIx3EcpxBvg3Acx3EKcQPhOI7jFOIGwnEcxynEDYTjOI5TiBsIx3Ecp5D/H5Rb5tNSddNmAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "\u003cFigure size 432x288 with 1 Axes\u003e"
            ]
          },
          "metadata": {
            "needs_background": "light",
            "tags": []
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Compare ground-truth accuracy against NNGP \n",
        "plt.title(\"Ground truth accurcay vs. NNGP validation accuracy\")\n",
        "plt.scatter(nngp_val, ground_truth, s=30)\n",
        "plt.xlabel(\"NNGP validation accuracy\")\n",
        "plt.ylabel(\"Ground truth accuracy\")\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lffziq4HsKeY"
      },
      "source": [
        "### Metrics\n",
        "\n",
        "We compute metrics used in [**Towards NNGP-guided Neural Architecture Search**]() to evaluate how well NNGP accuracy predicts the ground truth performance of the 250-network subset selected.\n",
        "\n",
        "See paper for more information on the metrics."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "executionInfo": {
          "elapsed": 1363844,
          "status": "ok",
          "timestamp": 1602036547414,
          "user": {
            "displayName": "Jaehoon Lee",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhxHqEwdpweGC4ZrtdwaxYD0GM-tam3aLuSS-I4aQ=s64",
            "userId": "07296832508386080109"
          },
          "user_tz": 420
        },
        "id": "FlC0Tt1HsLEJ",
        "outputId": "567e31f2-f734-49ff-8370-82d1e3fe4b87"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Kendall's Tau: 0.1827665761509062\n",
            "Correlation Coefficient: 0.21632201735448991\n",
            "Discovered Performance: 0.9337606827418009\n"
          ]
        }
      ],
      "source": [
        "# Kendall's tau.\n",
        "tau, _ = stats.kendalltau(ground_truth, nngp_val)\n",
        "print(\"Kendall's Tau: {}\".format(tau))\n",
        "\n",
        "# Pearson Correlation\n",
        "corr, _ = stats.pearsonr(ground_truth, nngp_val)\n",
        "print(\"Correlation Coefficient: {}\".format(corr))\n",
        "\n",
        "# Discovered Performance\n",
        "# Sort networks according to NNGP performance\n",
        "sorted_networks = np.argsort(nngp_val) \n",
        "# Get best ground-truth performance of the top-10 networks according to NNGP performance.\n",
        "discovered_performance = np.max(np.array(ground_truth)[sorted_networks[-10:]]) \n",
        "print(\"Discovered Performance: {}\".format(discovered_performance))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "executionInfo": {
          "elapsed": 1363226,
          "status": "ok",
          "timestamp": 1602036547416,
          "user": {
            "displayName": "Jaehoon Lee",
            "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhxHqEwdpweGC4ZrtdwaxYD0GM-tam3aLuSS-I4aQ=s64",
            "userId": "07296832508386080109"
          },
          "user_tz": 420
        },
        "id": "l28eOBNmtAiT",
        "outputId": "12e2164a-686d-45c3-f93a-d79ad3574dbb"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hUZdrH8e+dhNBbIHSkBiF0CFiwF8SGiIhgo6xdXuyrrLr2spa1YmUpNkARFWzYXQERJpRAqCEgHULoJf1+/zgn7hgDYWBmziS5P9c1F2ee034DIfec9jyiqhhjjDFHKsrrAMYYY0oXKxzGGGMCYoXDGGNMQKxwGGOMCYgVDmOMMQGxwmGMMSYgVjiMMcYExAqHMaWEiKiItA7DftaKyDlHue4hM4rIUBGZeWzpTCSwwmEiivtL66CI7BORrSIyXkSq+c2/SETmish+EckUkfdEpLHf/KEiku+u7/9qVOR9gd9+9onIVSLysIjkuu93ichsETkpTJ/7H35Zsop8htRwZDDmSFnhMJHoYlWtBnQDkoAHAERkAPAB8CJQF2gP5AC/iEgtv/V/VdVqRV6b/N8D6wr3477ed9ed7M6PB2YCU0VEQv2BVfVJv2w3FfkM7QPdnojEBD+lMQ4rHCZiqepG4Cugg/vL+3ngcVX9QFUPquoW4DrgAHBbkPedC0wAGgB1/Oe5Ry8HRSTOr62riGwXkQoi0lpEfhaR3W7b5CBGO0dEVrlHRKMLi5p7pDVLRF4QkUzgYRGpKCLPicg69+jtDRGp7C5fV0Q+d7ezQ0R+ERH/3wddRCTF/QyTRaSS32e9XkTS3PWmiUij4oKKSB13/h4RmQu0CuLfg/GQFQ4TsUSkKXABsAA4HjgO+Mh/GVUtAD4Gegd53xWBocB6Vd1eZJ+bgF+By/yarwSmuAXnMeAboDbQBHgliNEuAnoAnYCBwHl+804A0oH6wBPA00AboAvQGmgM/NNd9i5gA86RVX3gH4B/x3UDgT5AC3dfQwFE5CzgKXd+Q+B3YNIhso4GstzlhrsvUwZY4TCR6FMR2YVzquhn4EmcU1MAm4tZfjPOL8BCJ7rfpAtfqwPY90B33+uB7sClh1juA2AwgPutf5DbBpALNAMaqWqWqgbzgvDTqrpLVdcBP+IUhUKbVPUVVc3D+YV9A3CHqu5Q1b04f4+D/DI2BJqpaq6q/qJ/7vH0Zff03g5gut9+rgLGqup8Vc0GRgEniUhz/5AiEo1TWP+pqvtVdQnOEZwpA6xwmEjUT1VrqWozVb1FVQ8Chd/6GxazfEO/+QBz3PULX4GcIvnQXaeeqp6lqsmHWO5jnF+YDYHTgALgF3fe3wEB5opIqogE85v2Fr/pA0A1v/fr/abjgSpAcmEBBb7mfwX2WSAN+EZE0kXkviPcTyOcowwAVHUfkIlzNOMvHogpkul3TJlghcOUFitwTq1c7t/onpe/DPgpnGFUdSfO6agrcE5TTSr8xq6qW1T1elVtBNwIvBaO22j586mm7cBBoL1fAa3pXnxHVfeq6l2q2hLoC9wpImcfwT424RxNASAiVXGuAW0sslwGkAc09Ws7LuBPZCKSFQ5TKri/lO8GHhCRK0Wkkog0AMbgnMYK5nWEI/UBcC1QeLcXACJyuYg0cd/uxPmFXhDOYO61n7eBF0SknpursYic505f5F7EF2A3kH+EGScCw0Ski3sd6EngN1VdW2T/+cBUnIv0VUQkERgSpI9nPGaFw5QaqjoZuAa4A9iBc20jCThdVf2vfZxUzHMcPUIQaRqQAGxR1UV+7T2A30Rkn7vMbaqaDuCeuroqBFmKcy/O6ag5IrIH+A7nJgPc3N8B+3Au9L+mqj+WtEFV/Q54EOdU3WacO6UGHWLxETinuLYA44FxR/tBTGQRGwHQlFYi0hvnm/45qrrQ6zzGlBdWOEypJiIXA41V9Q2vsxhTXljhMMYYExC7xmGMMSYg5aI/m7p162rz5s29jmGMMaVKcnLydlWNL9peLgpH8+bN8fl8XscwxphSRUSKfWjTTlUZY4wJiBUOY4wxAbHCYYwxJiBWOIwxxgTECocxxpiAlIu7qowxpVfS49+yfV/OX9rrVovF98C5HiQydsRhjIloxRWNw7Wb0LPCYYwxJiBWOIwxxgTECocxJmJZJ6yRyQqHMSYiZeXmc9dHi0pe0ISd3VVljIk4W3ZnceN7ySxav4sqsdEcyMn/yzIVY6JQVZzRb004hbRwiEgf4CUgGhijqk8Xs8xA4GGccZkXqeqVInIm8ILfYm2BQar6qYiMB07HGScZYKiN/mZM2TF/3U5uejeZ/dl5vHlNd85r3+Avy4z+MY1nZ6xg8rz1DOp5nAcpy7eQFQ4RiQZGA+cCG4B5IjJNVZf6LZMAjAJ6qepOEakH4I593MVdJg5n3ORv/DZ/j6pOCVV2Y4w3PvSt54FPltCgZiXe/dsJHN+gerHL3Xx6K35dncnD01Pp1qw2beoXv5wJjVBe4+gJpKlquqrmAJOAS4oscz0wWlV3AqjqtmK2MwD4SlUPhDCrMcZDefkFPDI9lb9PSaFnizimjeh1yKIBEBUl/PuKzlSrGMOt78/nYDGnskzohLJwNAbW+73f4Lb5awO0EZFZIjLHPbVV1CBgYpG2J0QkRUReEJGKxe1cRG4QEZ+I+DIyMo72MxhjQmzn/hyuHTuXcbPWMrxXC8YP60GtKrElrleveiVeuKILaRn7eGR6ahiSmkJe31UVAyQAZwCDgbdFpFbhTBFpCHQEZvitMwrnmkcPIA64t7gNq+pbqpqkqknx8X8ZwMoYEwGWb9lD39Ez8a3dybMDOvHPixOJiT7yX0unJsRz8+mtmDRvPZ8t3BjCpMZfKAvHRqCp3/smbpu/DcA0Vc1V1TXASpxCUmgg8Imq5hY2qOpmdWQD43BOiRljSpmvl2ym/2uzyc4tYPKNJ3J5UtOSVyrGnee2IalZbf4xdTFrt+8PckpTnFAWjnlAgoi0EJFYnFNO04os8ynO0QYiUhfn1FW63/zBFDlN5R6FIM49eP2AJaEIb4wJjYIC5YVvV3LTe/NpU7860//vFLoeV/uotxcTHcVLg7sSEx3FiInzyc6z6x2hFrLCoap5wAic00zLgA9VNVVEHhWRvu5iM4BMEVkK/Ihzt1QmgIg0xzli+bnIpt8XkcXAYqAu8HioPoMxJrj2Zedx03vJvPT9Ki7r1oRJN5xI/RqVjnm7jWtV5rnLO7Nk4x6e/mp5EJKaw5Hy8Eh/UlKS+nw+r2MYU679nrmf69/xsTpjP/df0I5hvZoH/eG9h6elMn72Wt6+NolzE+sHddvlkYgkq2pS0XavL44bY8qBmau20/fVWWzdk82EYT0ZfkqLkDzxPeqCtnRoXIN7pixi066DQd++cVjhMMaEjKryn5lruHbsb9SvUZFpI3pxSkLdkO2vYkw0rwzuRm5eASMnLiAvvyBk+yrPrHAYY0IiKzefuz9K4bHPl3JuYn2m3tKLZnWqhny/LepW5cn+HfH9vpMXv1sV8v2VR9bJoTEm6LbuyeKGd51OCm8/J4GRZyUQFRW+zggv6dKYWWnbGf1TGie2rBPSo5zyyI44jDFBNX/dTi5+ZSartu7ljau7c/s5bcJaNAo93Lc9reKrcfvkhWTszQ77/ssyKxzGmKD5yLeeQW/OoWKFKKbecjJ9Ovy1Z9twqRIbw+gru7E3K5c7P1xIQUHZv4M0XKxwGGOOWWEnhfdMSaFHi9pMu/UU2jao4XUsjm9QnYcubs8vq7bzxn9Xex2nzLBrHMaYY7Jzfw4jJs5nVlomw3u14B8XtA2ov6lQG9yzKbNWb+f5b1ZyQos4ujeL8zpSqRc5/7rGmFJnxZa9XDJ6FvPWHF0nheEgIjzVvyONa1Xm/z5YwK4DOV5HKvUi61/YGFNqfL1kC5e+Nous3HwmHUMnheFQo1IFXhnclYx92fx9SgrloceMULLCYYwJSEGB8uJ3K7npvWQS3E4Kux1DJ4Xh0rlpLe7t05Zvlm5lwuy1Xscp1ewahzHmiO3LzuOuDxcyI3Url3VrwhOXdqBShWivYx2xv53Sgl9XZ/Lkl8tJah5Hh8Y1vY5UKtkRhzHmiKzLPMBlr83m26VbefCiRJ67vFOpKhrgXO949vLOxFWNZcQH89mXned1pFLJCocxpkSz0rbTd/RMtuzJ4p3hJ/C3EHVSGA5xVWN5eXBX1u04wAOfLLbrHUfBCocx5pBUlbEz13Dt2LnUqx76TgrDpWeLOG4/pw2fLtzER8kbvI5T6tg1DmNMsbJy83ng0yVMSd5A78T6/PuKLlSrWHZ+Zdx6ZmvmpGfy0GepdDuuFq3rVfc6UqlhRxzGmL/YuieLQW/NYUryBm47O4E3ru5epooGQHSU8OIVXagSG82t7y8gK9eGnD1SVjiMMX+ywO2kcOXWvbxxdTfuONebTgrDoV6NSjw/sDMrtu7l0c+Xeh2n1LDCYYz5w5TkDVzxp04KG3odKeTOOL4eN57ekg9+W8cXKZu9jlMqhLRwiEgfEVkhImkict8hlhkoIktFJFVEPvBrzxeRhe5rml97CxH5zd3mZBGJDeVnMKY8yMsv4NHpS7n7o0UkNY+cTgrD5e7ex9P1uFrc93EK6zIPeB0n4oWscIhINDAaOB9IBAaLSGKRZRKAUUAvVW0P3O43+6CqdnFfff3a/wW8oKqtgZ3A30L1GYwpD3buz2HIuLmMnbWGYb2a887wntSuWr6+j1WIjuLlQV0Rgf+bOJ+cPBty9nBCecTRE0hT1XRVzQEmAZcUWeZ6YLSq7gRQ1W2H26A4N46fBUxxmyYA/YKa2phyxL+TwmcGdOKhi9tHXCeF4dI0rgrPDOjEog27eebr5V7HiWih/AlpDKz3e7/BbfPXBmgjIrNEZI6I9PGbV0lEfG57YXGoA+xS1cLHPYvbJgAicoO7vi8jI+PYP40xZUxhJ4UH3U4KB0ZwJ4Xh0qdDQ645sRljZq7hh+VbvY4Tsbz+ahEDJABnAIOBt0WkljuvmaomAVcCL4pIq0A2rKpvqWqSqibFx8cHM7MxpVpBgfLSd6v+10nhiNLRSWG43H9hO9o1rMFdHy5i8+6DXseJSKEsHBsB/68wTdw2fxuAaaqaq6prgJU4hQRV3ej+mQ78BHQFMoFaIhJzmG0aYw5hf3Yet7w/nxe+W0n/bo2ZfMOJNKhZyetYEaVShWhevbIr2XkF3DZpIXn5dr2jqFAWjnlAgnsXVCwwCJhWZJlPcY42EJG6OKeu0kWktohU9GvvBSxVp1OZH4EB7vpDgM9C+BmMKTPWZR6g/2uz+WbpFh68KJHnL+9c6jopDJdW8dV4vF8H5q7Zwcs/pHkdJ+KErHC41yFGADOAZcCHqpoqIo+KSOFdUjOATBFZilMQ7lHVTKAd4BORRW7706pa+HTOvcCdIpKGc83jP6H6DMaUFf6dFE4Y3rNUd1IYLv27NeGybk145YdVzF693es4EUXKQ8+QSUlJ6vP5vI5hTNipKuNnr+XxL5bRsm5V3r42ieZ1q3odq9TYn53Hxa/OZF9WHl/edip1q1X0OlJYiUiye635T7y+OG6MCZHsvHz+PiWFR6Yv5ay29fjk1l5WNAJUtWIMo6/sxq6Dudz14SIKCsr+F+0jYYXDmDJom9tJ4UfJGxh5dgJvlsFOCsOlXcMaPHhRIj+vzODtX9K9jhMR7CfJmDJm4fpd3Piuj71Zebx+VTfO71j2+5sKtatPOI7Zadt5dsYKerSIK/e3L9sRhzFlyMfJGxj45q9UiI7i45tPtqIRJCLC05d1okHNSoycuIDdB3O9juQpKxzGlAF5+QU89vlS7vpoEUnNajNtxCm0a1h+OikMh5qVK/DK4K5s2Z3FfR+nlOshZ61wGFPK7TqQw9Bx8/jPzP91UhhXzjopDJeux9XmnvOO56slW3hvzu9ex/GMXeMwphRbuXUv103wsWV3Fs8M6GT9TYXB9ae25Nf0TB77YhndmtWmfaOaXkcKOzviMKaUmpG6hUtHWyeF4RYVJTx/eWdqVa7A/32wgP3ZeSWvVMZY4TCmlCnspPDGd5NpbZ0UeqJOtYq8OKgLazL38+BnS7yOE3ZWOIwpRayTwshxcqu6/N9ZCUydv5GPkzd4HSes7BqHMaXEuswD3PCuj5Vb9/LgRYkM79Xc+pvy2MizWjMnPZMHP1tCl+Nq0Sq+mteRwsKOOIwpBWa7nRRu3m2dFEaSGHfI2YoxUdz6/nyycvO9jhQWVjiMiWCqyvhZa7hm7Fziq1Xks1t7cWqCDUwWSRrUrMTzAzuzfMtenvhimddxwsIKhzERKjsvn3s/TuFh66Qw4p3Vtj7Xn9qCd+f8zleLN3sdJ+TsGocxEWjbnixufC+ZBet2MfLsBG4/O4GoKDs1FcnuOa8tc9fu5O8fp9ChcU2axlXxOlLI2BGHMRFm4fpdXPzqTFZs2cvrV3XjznPbWNEoBWJjonhlUFdQGDlpAblleMhZKxzGRBDrpLB0O65OFZ6+rBML1u3iuW9WeB0nZOxUlTERIC+/gKe/Ws6YmWs4qWUdRl/VzfqbKqUu7NSQWauP482f0zmpZR3OOL6e15GCLqSFQ0T6AC8B0cAYVX26mGUGAg8DCixS1StFpAvwOlADyAeeUNXJ7vLjgdOB3e4mhqrqwlB+DmOCKenxb9m+L6fYeUNPbs79F7ajQrSdDCjN/nlRIvN/38mdHy7iq9tOpX6NsvWQZsh+OkUkGhgNnA8kAoNFJLHIMgnAKKCXqrYHbndnHQCuddv6AC+KSC2/Ve9R1S7uy4qGKVUOVTQAHu7b3opGGVCpQjSvXtmVgzn53D5pIfllbMjZUP6E9gTSVDVdVXOAScAlRZa5HhitqjsBVHWb++dKVV3lTm8CtgF287oxptRoXa86j1zSnl/TM3n1hzSv4wRVKAtHY2C93/sNbpu/NkAbEZklInPcU1t/IiI9gVhgtV/zEyKSIiIviEjF4nYuIjeIiE9EfBkZGcf2SYwx5ihc3r0J/bo04qXvVzInPdPrOEHj9TFxDJAAnAEMBt72PyUlIg2Bd4Fhqlp4b9sooC3QA4gD7i1uw6r6lqomqWpSfLwdrBhjwk9EePzSjjSrU5XbJi1gx/5Dn6YsTUJZODYC/gMENHHb/G0ApqlqrqquAVbiFBJEpAbwBXC/qs4pXEFVN6sjGxiHc0rMmFKhvI9VXR5VqxjDK4O7snN/Lnd/tKhMDDkbysIxD0gQkRYiEgsMAqYVWeZTnKMNRKQuzqmrdHf5T4B3VHWK/wruUQji9PDWDyh/neGbUiknr4Cb30s+5Py61ez227KqQ+Oa/OOCtvywfBv/mbnG6zjHLGS346pqnoiMAGbg3I47VlVTReRRwKeq09x5vUVkKc5tt/eoaqaIXA2cBtQRkaHuJgtvu31fROIBARYCN4XqMxgTLKrKfVNTmL06k+cv78xl3Zt4HcmE2ZCTmzN7dSb/+no5PZrH0blprZJXilBSFg6bSpKUlKQ+n8/rGKYce+Hblbz0/SruOKcNt52T4HUc45FdB3K48OWZREXBFyNPpUalCl5HOiwRSVbVpKLtXl8cN6bM+9C3npe+X8Xl3Zsw8uzWXscxHqpVJZaXB3dh064sRk1dXGqvd1jhMCaEZq7azj+mLubUhLo82b+jDb5k6N4sjrt6t+GLlM1MnLu+5BUikBUOY0Jk+ZY93PxeMq3rVeO1q7rZE+HmDzed1opTE+ryyPRUlm/Z43WcgNlPsjEhsGV3FsPGzaNKxWjGDetB9Qg/l23CKypK+PfALtSoXIERHyzgQE6e15ECYoXDmCDbm5XLsPHz2JuVx7ihPWlYs7LXkUwEiq9ekRev6MLqjH089Fmq13ECYoXDmCDKzS/g1g8WsHLrXkZf1Y3ERjW8jmQiWK/Wdbn1jNZ8lLyBTxcUfT46clnhMCZIVJUHP13Cf1dm8OSlHTi9jXV1Y0p2+zkJ9Ghem/s/Wcya7fu9jnNErHAYEySv/bSaSfPWM+LM1lzR4ziv45hSIiY6ipcGdaVCTBQjPphPdl6+15FKZIXDmCD4dMFGnp2xgku7Nuau3m28jmNKmUa1KvPsgM6kbtrDU18u9zpOiaxwGHOMfl2dyT1TFnFiyzj+dVkne1bDHJVzE+szrFdzxs9ey4zULV7HOSwrHMYcg1Vb93Ljuz6a1anKm1cnERtj/6XM0bvv/LZ0aFyDv09JYeOug17HOST7KTfmKG3bm8XQcfOIjYlm3NAe1Kxiz2qYY1MxJppXB3cjv0AZOXEBufkFJa/kASscxhyF/dl5/G28jx37cxg3tAdN46p4HcmUEc3rVuWJSzuQ/PtOXvh2pddximWFw5gA5eUXMHLiAlI37ebVK7vSsUlNryOZMuaSLo25Iqkpr/+8ml9WRd7Q11Y4jAmAqvLw9FS+X76NRy7pwNnt6nsdyZRRD/dtT+v4atwxeSHb9mZ5HedPrHAYE4C3/pvOe3PWcePpLbnmxGZexzFlWOXYaEZf1Y192XncMXkh+QWR0wW7FQ5jjtDnKZt46qvlXNSpIfee19brOKYcaFO/Og9f3J5ZaZm8/lOa13H+YIXDmCPgW7uDOz9cRI/mtXnu8s5ERdmzGiY8rujRlIs7N+Lf365k3todXscBSigcIlJJRG4XkVdF5EYRCdkY5cZEqvSMfVz3jo8mtSrz1jVJVKoQ7XUkU46ICE9e2oGmcVUYOXEBO/fneB2pxCOOCUASsBg4H3g+kI2LSB8RWSEiaSJy3yGWGSgiS0UkVUQ+8GsfIiKr3NcQv/buIrLY3ebLYo/pmhDavi+boePmES3CuGE9qF011utIphyqXqkCrwzuyvZ92dwzJcXzIWdLKhyJqnq1qr4JDABOPdINi0g0MBqn4CQCg0UkscgyCcAooJeqtgdud9vjgIeAE4CewEMiUttd7XXgeiDBffU50kzGBOJgTj7XTfCxbW8WY4Yk0axOVa8jmXKsU5Na3Hd+O75btpVxs9Z6mqWkwpFbOKGqgQ5R1RNIU9V0Vc0BJgGXFFnmemC0qu5097HNbT8P+FZVd7jzvgX6iEhDoIaqzlGn5L4D9AswlzElyi9Qbp+8gEUbdvHSoK50Pa52ySsZE2LDezXnnHb1eOqrZSzesNuzHCUVjs4issd97QU6FU6LSEkD5TYG/Edi3+C2+WsDtBGRWSIyR0T6lLBuY3f6cNsEQERuEBGfiPgyMiLvARoT2Z74YhkzUrfy4IWJnNe+gddxjAGc6x3PDuhM3WoVGTFxPnuzckteKQRKKhyVVLWG+6quqjF+08EY2iwG53TTGcBg4G0RqRWE7aKqb6lqkqomxcfbgDrmyI2duYaxs9YwvFcLhp/Swus4xvxJ7aqxvDSoK+t3HOD+T5Z4cr2jpMLx2zFseyPQ1O99E7fN3wZgmqrmquoaYCVOITnUuhvd6cNt05ij9vWSLTz2xVLOa1+f+y9s53UcY4rVs0Ucd5zThmmLNvGhb33JKwRZSYXjWO5YmgckiEgLEYkFBgHTiizzKc7RBiJSF+fUVTowA+gtIrXdi+K9gRmquhnYIyInundTXQt8dgwZjfnD/HU7uW3SAjo3qcWLV3Ql2p7VMBHsljNbc3KrOjw0LZWVW/eGdd8lPZcRLyJ3Hmqmqv77MPPyRGQEThGIBsaqaqqIPAr4VHUa/ysQS4F84B5VzQQQkcdwig/Ao6pa+OTLLcB4oDLwlfsy5pj8nrmf6yb4qF+jEmOGJFE51p7VMJEtOkp48YounP/SL4z4YD6f3XpK2H5u5XDnx0RkM87tr8V+9VLVR0KUK6iSkpLU5/N5HcNEqJ37c+j/+mx2Hshh6s0n0zK+mteRjDliP6/MYMjYuQzu2ZSn+ncK6rZFJFlVk4q2l3TEsVlVHw1qEmMiSFZuPte/42PjroN8cN0JVjRMqXN6m3huOr0Vb/y8mpNa1aVv50Yh32cor3EYE9EKCpS7PlqE7/edvDCwC0nN47yOZMxRuat3G7odV4t/TF3M2u37Q76/kgrHU4UTIvKn+xJFpH9IEhkTJv/6ejlfpGxm1PltubBTQ6/jGHPUKkRH8fLgrkQJ/N/EBWTn5Yd0fyUVDv/+pT4uMu+BIGcxJmzenfM7b/43nWtObMYNp7X0Oo4xx6xJ7So8e3lnFm/czb++WhHSfZV0jUMOMV3ce2NKhe+XbeWhz5Zwdtt6PHRxItZPpikrzmvfgEoVohg7y3mI1V/darH4Hjg3KPsp6YhDDzFd3HtjIl7Khl2M+GAB7RvV5JUruxITbUPSmLIlK7eg2Pbt+4LXHXtJRxwtRWQaztFF4TTue+uLwZQq63ccYPh4H3FVY/nP0CSqxNrwMsYcjZL+5/j3ZvtckXlF3xsTsXYfyGXY+Hnk5OUz8foTqFe9kteRjCm1Dls4VPVnABGpArR2m1eoanaogxkTLNl5+dz4no/fM/fz7t9OIKF+da8jGVOqlTR0bAUReRGnM8JxOF19pBeO5iciXUKe0JhjoKrcOyWFOek7eHZAZ05sWcfrSMaUeiVdGXweqAY0U9XuqtoNaIdzveN14JNQBzTmWDz/zUo+XbiJu3u3oV/XYoduMaZMqVut+OGND9V+NEq6xnEBkKB+HVqp6h4RuRnYjjMsrDERadLcdbz6YxqDejTl1jNbl7yCMWVAsG65PZySjjgKtJheEFU1H8hQ1TmhiWXMsfl5ZQb3f7qE09rE81i/DvashjFBVFLhWCoi1xZtFJGrgWWhiWTMsUndtJtb3kumTf3qvHZVNyrYsxrGBFVJp6puBaaKyHAg2W1LwhkL49JQBjPmaGzadZDh4+dRo3IFxg3tQbWK9qyGMcFW0u24G4ETROQsoL3b/KWqfh/yZMYEaE9WLsPHz+NAdj4f3XwSDWrasxrGhMJhC4eIVAJuwnmGYzHwH1XNC0cwYwKRm1/ALe/NJ23bPsYP60nbBjW8jmRMmVXSyd8JOKemFuPcQWVPi5uIo6qMmrqYmWnbeap/R05JqOt1JGPKtJIKR6KqXq2qbwIDgNMC2biI9BGRFSKSVvjQYJH5Q0UkQ0QWuq/r3PYz/doWikiWiPRz540XkTV+8+whxHLu5Q9B4/YAABZsSURBVO/TmJK8gdvOTuDypKZexzGmzCvpymFu4YSq5gVyS6OIRAOjgXNxnjyfJyLTVHVpkUUnq+oI/wZV/RHo4m4nDkgDvvFb5B5VnXLEYUyZ9XHyBl74biWXdWvC7eckeB3HmHKhpMLRWUT2uNMCVHbfC6CqergTyT2BNFVNBxCRSTidJhYtHCUZAHylqgcCXM+UcbPStnPvxymc3KoOT/XvaM9qGBMmhz1VparRqlrDfVVX1Ri/6ZKuPjYG1vu93+C2FXWZiKSIyBQRKe48wyBgYpG2J9x1XhCRisXtXERuEBGfiPgyMjJKiGpKmxVb9nLTu8m0jK/K61d3JzbGntUwJly8/t82HWiuqp2Ab3Euxv9BRBoCHYEZfs2jgLZADyAOuLe4DavqW6qapKpJ8fHxochuPLJ1TxbDxs2lcmw044b1pGblCl5HMqZcCWXh2Aj4H0E0cdv+oKqZfl20jwG6F9nGQOATVfW/1rJZHdk4Pfb2DHpyE7H2ZecxfPw8dh/MZezQHjSuVdnrSMaUO6EsHPOABBFpISKxOKecpvkv4B5RFOrLX7sxGUyR01SF64hzQrsfsCTIuU2Eyssv4Nb357N8y15evaobHRrX9DqSMeVSyPpjcO/CGoFzmikaGKuqqSLyKOBT1WnASBHpC+QBO4ChheuLSHOcI5afi2z6fRGJx7lAvxDnAUVTxqkqD36Wys8rM3jy0o6ceXw9ryMZU25JMZ3fljlJSUnq8/m8jmGOwWs/pfHM1yu45YxW/L1PW6/jGFMuiEiyqiYVbff64rgxJfps4Uae+XoFfTs34u7ex3sdx5hyzwqHiWi/pWdyz0cp9GwRx7OXdyIqyp7VMMZrVjhMxErbto8b3k2mSVxl3rqmOxVjor2OZIzBCoeJUBl7sxk6bi4VooUJw3pSq0rwxks2xhwbG+XGRJwDOXlcN2EemftymHTDiTSNq+J1JGOMHzviMBElv0AZOXEBizfu5uXBXenctJbXkYwxRVjhMBFDVXl0eirfLdvGQxe359zE+l5HMsYUwwqHiRj/mbmGCb/+zvWntmDIyc29jmOMOQQrHCYifLl4M098uYwLOjZg1PntvI5jjDkMKxzGc8m/7+D2yQvp2rQW/x7YxZ7VMCbCWeEwnlqzfT/XTfDRqGYlxgzpQaUK9qyGMZHOCofxTOa+bIaNm4uIMH5YT+Kq2rMaxpQGVjiMJ7Jy87nuHR+bd2fx9rVJNK9b1etIxpgjZA8AmrArKFDumLyQhet38dqV3ejerLbXkYwxAbAjDhN2T365jK+WbOH+C9pxfseGJa9gjIkoVjhMWI2ftYYxM9cw9OTm/O2UFl7HMcYcBSscJmy+Sd3CI58v5dzE+jx4USLO6L/GmNLGCocJi4XrdzFy0gI6Na7Jy4O6Em3PahhTalnhMCG3fscBrpswj/jqFRkzpAeVY+1ZDWNKs5AWDhHpIyIrRCRNRO4rZv5QEckQkYXu6zq/efl+7dP82luIyG/uNieLiN38H8F2HchhyLi55OYr44f1JL56Ra8jGWOOUcgKh4hEA6OB84FEYLCIJBaz6GRV7eK+xvi1H/Rr7+vX/i/gBVVtDewE/haqz2COTVZuPje8k8yGHQd5+9okWsVX8zqSMSYIQnnE0RNIU9V0Vc0BJgGXHMsGxbmaehYwxW2aAPQ7ppQmJAoKlHumpDB37Q6eG9iZni3ivI5kjAmSUBaOxsB6v/cb3LaiLhORFBGZIiJN/doriYhPROaISGFxqAPsUtW8EraJiNzgru/LyMg4xo9iAvXsNyuYvmgT9/ZpS9/OjbyOY4wJIq+fHJ8OTFTVbBG5EecI4ix3XjNV3SgiLYEfRGQxsPtIN6yqbwFvASQlJWmQcxs/SY9/y/Z9OX9pr1QhiptOb+lBImNMKIXyiGMj4H8E0cRt+4OqZqpqtvt2DNDdb95G98904CegK5AJ1BKRwoL3l22a8CuuaABk5RbYsxrGlEGhLBzzgAT3LqhYYBAwzX8BEfHvb6IvsMxtry0iFd3pukAvYKmqKvAjMMBdZwjwWQg/gzHGmCJCdqpKVfNEZAQwA4gGxqpqqog8CvhUdRowUkT6AnnADmCou3o74E0RKcApbk+r6lJ33r3AJBF5HFgA/CdUn8EYY8xfhfQah6p+CXxZpO2fftOjgFHFrDcb6HiIbabj3LFlIsDs1du9jmCMCTOvL46bUmr3gVye/HIZk33rS17YGFOmWJcjJiCqyhcpmzn73z8zZf4Gbjy9JXWrFf/w/qHajTGlmx1xmCO2efdBHvx0Cd8t20aHxjUYP6wHHRrXZNT57byOZowJIyscpkQFBcp7v/3OM1+vIK+ggPsvaMewXs2JibYDVmPKIysc5rBWbd3LfVMXk/z7Tk5NqMsT/TpyXJ0qXscyxnjICocpVnZePq/9uJrXfkqjasUYnr+8M/27NbYH+owxVjjMX/nW7uC+qYtJ27aPfl0a8eBFidSpZt2hG2McVjjMH/Zk5fLM18t5b846GteqzLhhPTjz+HpexzLGRBgrHAZwxgP/52epbN2bxfBeLbirdxuqVrQfD2PMX9lvhnJu254sHp6eypeLt9C2QXXeuKY7XZrW8jqWMSaCWeEop1SVyfPW88SXy8jOK+Ce847nhtNaUsFusTXGlMAKRzmUnrGPUVMX89uaHZzQIo6n+nekpQ3raow5QlY4ypHc/ALe+m86L32/iooxUTzdvyMDk5oSFWW32BpjjpwVjnJi4fpd3PdxCsu37OWCjg14+OL21KtRyetYxphSyApHGbc/O4/nvlnB+NlrqV+9Em9d053e7Rt4HcsYU4pZ4SjDflyxjQc+WcLGXQe55sRm/L3P8VSvVMHrWMaYUs4KRxmUuS+bRz9fymcLN9Eqviof3XQSPZrHeR3LGFNGWOEoQ1SVqfM38vgXS9mXncdtZydwy5mtqBgT7XU0Y0wZEtKb9kWkj4isEJE0EbmvmPlDRSRDRBa6r+vc9i4i8quIpIpIiohc4bfOeBFZ47dOl1B+htJiXeYBrh07l7s+WkSLulX5YuSp3HFuGysaxpigC9kRh4hEA6OBc4ENwDwRmaaqS4ssOllVRxRpOwBcq6qrRKQRkCwiM1R1lzv/HlWdEqrspUlefgFjZ63h39+uJCYqiscuac9VJzSzW2yNMSETylNVPYE0VU0HEJFJwCVA0cLxF6q60m96k4hsA+KBXYdeq/xZsnE3901NYcnGPZzTrh6P9etAw5qVvY5ljCnjQnmqqjGw3u/9BretqMvc01FTRKRp0Zki0hOIBVb7NT/hrvOCiJS7/r4P5uTz1FfLuGT0LLbszmb0ld14+9okKxrGmLDwumOi6UBzVe0EfAtM8J8pIg2Bd4FhqlrgNo8C2gI9gDjg3uI2LCI3iIhPRHwZGRmhyh92s9K2c96L/+XNn9MZ0K0J3995Ohd2amgDLBljwiaUhWMj4H8E0cRt+4OqZqpqtvt2DNC9cJ6I1AC+AO5X1Tl+62xWRzYwDueU2F+o6luqmqSqSfHx8UH5QF7auT+Huz9axFVjfiNK4IPrT+BfAzpRs4o9l2GMCa9QXuOYBySISAucgjEIuNJ/ARFpqKqb3bd9gWVueyzwCfBO0YvgheuI8xW7H7AkhJ/Bc6rK9JTNPDo9lZ0HcrnljFaMPDuBShXsbiljjDdCVjhUNU9ERgAzgGhgrKqmisijgE9VpwEjRaQvkAfsAIa6qw8ETgPqiEhh21BVXQi8LyLxgAALgZtC9Rm8tnHXQR78dAk/LN9GpyY1eWf4CSQ2quF1LGNMOSeq6nWGkEtKSlKfz+d1jCOWX6C8++tanp2xggKFu3q3YVivFkTbLbbGmDASkWRVTSrabk+OR5gVW/Zy78cpLFy/i9PaxPNEvw40javidSxjjPmDFY4IkZWbz+gf03j9p9XUqFyBF6/owiVdGtndUsaYiGOFIwL8lp7JqE8Wk56xn/5dG/PARYnEVY31OpYxxhTLCoeHdh/M5emvljNx7jqa1K7MO8N7clqb0n/rsDGmbLPC4ZGvl2zmn5+lsn1fNtef2oI7zm1DlVj75zDGRD77TRVmW/dk8c/PljAjdSvtGtZgzJAkOjWp5XUsY4w5YlY4wqSgQJk4bx1Pf7mcnPwC7u3TlutObUGFaK97fTHGmMBY4QiDtG37+MfUxcxdu4OTWtbhqf4daV63qtexjDHmqFjhCKGcvALe+Hk1r/6QRuXYaJ4Z0InLuzexW2yNMaWaFY4Qmb9uJ/d9nMLKrfu4qFNDHrq4PfHVy10P8MaYMsgKR5Dty87juRkrmPDrWhrUqMR/hiRxdrv6XscyxpigscIRRN8v28qDny5h854srj2xGff0aUu1ivZXbIwpW+y3WhBk7M3mkempfJ6ymYR61Zhy08l0b1bb61jGGBMSVjiOgaryUfIGnvhiGQdz8rnjnDbcfEYrYmPsFltjTNllheMo/Z65n1FTFzN7dSY9mtfmqf4daV2vutexjDEm5KxwBCgvv4AxM9fwwrcriY2O4vF+Hbiy53FE2VgZxphywgpHABZv2M29H6ewdPMeeifW59FLOtCgZiWvYxljTFhZ4ShG0uPfsn1fTrHz4qtX5I2ru9GnQ8MwpzLGmMhghaMYhyoaAN/deTo1K1cIYxpjjIksIb39R0T6iMgKEUkTkfuKmT9URDJEZKH7us5v3hARWeW+hvi1dxeRxe42X5Yw999hRcMYU96FrHCISDQwGjgfSAQGi0hiMYtOVtUu7muMu24c8BBwAtATeEhECh+MeB24HkhwX31C9RmMMcb8VSiPOHoCaaqarqo5wCTgkiNc9zzgW1Xdoao7gW+BPiLSEKihqnNUVYF3gH6hCG+MMaZ4oSwcjYH1fu83uG1FXSYiKSIyRUSalrBuY3e6pG0iIjeIiE9EfBkZGUf7GYwxxhTh9SPO04HmqtoJ56hiQrA2rKpvqWqSqibFxwc2jnfdarEBtRtjTHkSyruqNgJN/d43cdv+oKqZfm/HAM/4rXtGkXV/ctubHG6bweB74Nxgb9IYY8qMUB5xzAMSRKSFiMQCg4Bp/gu41ywK9QWWudMzgN4iUtu9KN4bmKGqm4E9InKiezfVtcBnIfwMxhhjigjZEYeq5onICJwiEA2MVdVUEXkU8KnqNGCkiPQF8oAdwFB33R0i8hhO8QF4VFV3uNO3AOOBysBX7ssYY0yYiHNzUtmWlJSkPp/P6xjGGFOqiEiyqiYVbff64rgxxphSxgqHMcaYgJSLU1UikgH8fpSr1wW2BzFOsFiuwFiuwFiuwJTVXM1U9S/PM5SLwnEsRMRX3Dk+r1muwFiuwFiuwJS3XHaqyhhjTECscBhjjAmIFY6SveV1gEOwXIGxXIGxXIEpV7nsGocxxpiA2BGHMcaYgFjhMMYYE5ByXTiOYGjb40TkRxFZ4I4ZcoHb3lxEDvoNeftGJORy53USkV9FJNUdYreS17lE5Cq/v6uFIlIgIl0iIFcFEZng/j0tE5FRwcp0jLliRWScm2uRiJwR5lzNROR7N9NPItLEb16xQzpHQK6vRWSXiHwezEzHkktEuvj9X0wRkSsiJFczEZnv/l9MFZGbAt65qpbLF07Hi6uBlkAssAhILLLMW8DN7nQisNadbg4sicBcMUAK0Nl9XweI9jpXkWU6Aqsj5O/rSmCSO10FWIszPozXuW4FxrnT9YBkICqMuT4ChrjTZwHvutNxQLr7Z213urbXudz3ZwMXA58H62crCH9fbYAEd7oRsBmoFQG5YoGK7nQ19+e+USD7L89HHEcytK0CNdzpmsCmCM/VG0hR1UXgjHeiqvkRkMvfYHfdYDmWXApUFZEYnN6Wc4A9EZArEfgBQFW3AbuAYD3EdSS5/tg/8KPf/GKHdI6AXKjq98DeIGUJSi5VXamqq9zpTcA2ILBR5UKTK0dVs932ihzFmafyXDiOZGjbh4GrRWQD8CXwf37zWrinGH4WkVMjJFcbQEVkhnso+vcIyeXvCmBihOSaAuzH+Sa4DnhO/9d9v5e5FgF9RSRGRFoA3fnzoGihzrUI6O9OXwpUF5E6R7iuF7lCKSi5RKQnzjf91ZGQS0SaikiKu41/uYXtiJXnwnEkBgPjVbUJcAHwrohE4fyiOU5VuwJ3Ah+ISI3DbCdcuWKAU4Cr3D8vFZGzIyAXACJyAnBAVZeEMdPhcvUE8nFOI7QA7hKRlhGQayzOLwIf8CIw280ZLncDp4vIAuB0nFE2w7n/QymVucQZsO5dYJiqFkRCLlVdr86Q3a2BISJSP5ANh3Lo2EhX4tC2wN9wD8VV9VdxLjTXdU8fZLvtySKyGufbfjAG/TjqXDi/bP6rqtsBRORLoBvwvce5trnzBxHco41jzXUl8LWq5gLbRGQWzimhdC9zuT9fdxQuJCKzgZVByHREudxvn/3dfVcDLlPVXSJyqCGdPc0VpP2HJJf7hfIL4H5VnRMpufyXEZElwKk4R+BHJpgXkkrTC6dopuN80yy8uNS+yDJfAUPd6XY456AF5zxltNve0v0Hi4uAXLWB+TgXemOA74ALvc7lvo9y/55aRtC/47387yJ0VWAp0CkCclUBqrrt5+J8GQjn31dd3IvxwBM4I3CCc1F8jftzVtudDufPfbG5/OafQfAvjh/L31cszpe224OZKQi5mgCV3enaOF9KOga0/2B/oNL0wjk9sBLnvOP9btujQF93OhGY5f6jLAR6u+2XAalu23zg4kjI5c672s22BHgmgnKdAcyJsH/Hajh3nqTiFI17IiRXc2AFsAyn+DcLc64BwCp3mTG4d+C484YDae5rWATl+gXIAA7iHHmf53Uu9/9irvtvW/jqEgG5zsW5+3KR++cNge7buhwxxhgTELs4bowxJiBWOIwxxgTECocxxpiAWOEwxhgTECscxhhjAmKFw5jDEJE68r9efbeIyEZ3epeILA3B/h4WkbsDXGffIdrHi8iA4CQz5n+scBhzGOp0FNlFVbsAbwAvuNNdgBK7j3A7UDSmTLHCYczRixaRt90xDb4RkcoA7tgHL4qID7hNRLq7nWEmux1QNnSXGykiS93xEvx7DE50t5EuIiMLG0XkThFZ4r5uLxpGHK+6YzR8h9MluzFBZ9+GjDl6CcBgVb1eRD7E6VHgPXderKomiUgF4GfgElXNcAfzeQLnCez7gBaqmi0itfy22xY4E6gOrBCR14FOwDDgBJxuSX4TkZ9VdYHfepcCx+M8kV4f52n4sSH55KZcs8JhzNFbo6oL3elknK5CCk12/zwe6AB8KyLgDMCz2Z2XArwvIp8Cn/qt+4U64yVki8g2nCJwCvCJqu4HEJGpOB3T+ReO04CJ6ozBsklEfsCYELDCYczRy/abzscZDKrQfvdPAVJV9aRi1r8Q55f9xcD9ItLxENu1/6cmotg1DmNCawUQLyInwR/jnLd3x91oqqo/4vTSWxOn08VD+QXoJyJVRKQqzmmpX4os81/gChGJdq+jnBnsD2MM2DcZY0JKVXPcW2JfFpGaOP/nXsTpsfQ9t02Al9UZ8+JQ25kvIuOBuW7TmCLXNwA+wRlbeinOiIa/BvvzGANY77jGGGMCY6eqjDHGBMQKhzHGmIBY4TDGGBMQKxzGGGMCYoXDGGNMQKxwGGOMCYgVDmOMMQH5f441Cq9E8DD2AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "\u003cFigure size 432x288 with 1 Axes\u003e"
            ]
          },
          "metadata": {
            "needs_background": "light",
            "tags": []
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Prediction Quality for Exceedance of Threshold Performance p_T (PQETP-p_T)\n",
        "def pqetp(ground_truth, performance_metric, threshold):\n",
        "  \"\"\"Returns PQETP_threshold with respect to the performance metric.\"\"\"\n",
        "  return sklearn.metrics.roc_auc_score(\n",
        "      (np.array(ground_truth) \u003e threshold), performance_metric)\n",
        "\n",
        "# PQETP plot.\n",
        "threshold_values = [0.85, 0.87, 0.89, 0.91, 0.93]\n",
        "pqetp_threshold = [\n",
        "                   pqetp(ground_truth, nngp_val, p_T) \n",
        "                   for p_T in threshold_values\n",
        "                   ]\n",
        "\n",
        "plt.title(\"PQETP vs. Threshold\")\n",
        "plt.plot(threshold_values, pqetp_threshold, marker='s')\n",
        "plt.xlabel(\"Threshold\")\n",
        "plt.ylabel(\"PQETP\")\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y_ttpKKtAogJ"
      },
      "source": [
        "The quality of prediction improves as one increases NUM_SPECS, TRAIN_SIZE, VAL_SIZE, and ENSEMBLE_NUMBER. Evaluating on the full (423k) NASBench-101 models as done in the paper would require distributed computation. "
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "last_runtime": {
        "build_target": "//learning/deepmind/dm_python:dm_notebook3",
        "kind": "private"
      },
      "name": "NNGP_on_NASBench101.ipynb",
      "provenance": [
        {
          "file_id": "1mk7bNsV5tbqKL9l7Mr8r19X0vOA_Hu3c",
          "timestamp": 1602097293687
        },
        {
          "file_id": "1OTCP0vAZ7x46Q9fPZxUOpgaZbDB1iZZe",
          "timestamp": 1602097090952
        },
        {
          "file_id": "1HGCyoCf7d8_473V6mQn0Ux34Lq_tkJ4B",
          "timestamp": 1594772313378
        }
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "6e6ccad3e71b4fc593be0143edf845c7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ec14e1007b04450eb6dc7147a52508d6",
              "IPY_MODEL_8b475a3beaca4ab39771ff5ef2afe5f4"
            ],
            "layout": "IPY_MODEL_b619a80c5d794002b34faba9e210b393"
          }
        },
        "8b475a3beaca4ab39771ff5ef2afe5f4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8cc0d1a4e21046f99d657b015a1315a4",
            "placeholder": "​",
            "style": "IPY_MODEL_c7430782e31946dd9b2b60510840d8b4",
            "value": " 250/250 [22:40\u0026lt;00:00,  5.44s/it]"
          }
        },
        "8cc0d1a4e21046f99d657b015a1315a4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b619a80c5d794002b34faba9e210b393": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c7430782e31946dd9b2b60510840d8b4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "cb151c02cf624186b3489757a8e1e0eb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": "initial"
          }
        },
        "e20930b90d27465fb7464a1f77b46620": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ec14e1007b04450eb6dc7147a52508d6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "100%",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e20930b90d27465fb7464a1f77b46620",
            "max": 250,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_cb151c02cf624186b3489757a8e1e0eb",
            "value": 250
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}