notebooks/2_yago_topk_prediction.ipynb

{
   "cells": [
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "# Knowledge Graph Completion on YAGO3-10\n",
            "\n",
            "<em>Copyright (c) 2023 Graphcore Ltd. All rights reserved.</em>\n",
            "\n",
            "BESS-KGE (`besskge`) is a PyTorch library for knowledge graph embedding (KGE) models on IPUs implementing the distribution framework [BESS](https://arxiv.org/abs/2211.12281), with embedding tables stored in the IPU SRAM.\n",
            "\n",
            "In this notebook we will show how to use the BESS-KGE package to perform knowledge graph completion on the YAGO3-10 dataset, a subset of [YAGO3](https://yago-knowledge.org/downloads/yago-3) (Yet Another Great Ontology 3) containing only entities with at least ten relations associated to them.\n",
            "\n",
            "In particular, we will learn how to use the `TopKQueryBessKGE` BESS distribution scheme to efficiently score queries against all entities in the knowledge graph and retrieve the most likely triple completions. \n",
            "\n",
            "If you have not already done so, we suggest running the [KGE Training and Inference on OGBL-BioKG](1_biokg_training_inference.ipynb) notebook to get familiar with the basic functionalities of the BESS-KGE library."
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Environment setup"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "To run the demo using IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to do this. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine."
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Dependencies"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "We recommend that you install `besskge` directly from the GitHub sources:"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 4,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Found existing installation: besskge 0.1\n",
                  "Uninstalling besskge-0.1:\n",
                  "  Successfully uninstalled besskge-0.1\n"
               ]
            }
         ],
         "source": [
            "import sys\n",
            "!{sys.executable} -m pip uninstall -y besskge\n",
            "!pip install -q git+https://github.com/graphcore-research/bess-kge.git\n",
            "\n",
            "!pip install -q matplotlib"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Next, import the necessary dependencies. "
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 1,
         "metadata": {},
         "outputs": [],
         "source": [
            "import os\n",
            "import pathlib\n",
            "import time\n",
            "\n",
            "import matplotlib.pyplot as plt\n",
            "import numpy as np\n",
            "import poptorch\n",
            "import torch\n",
            "\n",
            "from besskge.batch_sampler import RigidShardedBatchSampler\n",
            "from besskge.bess import EmbeddingMovingBessKGE, TopKQueryBessKGE\n",
            "from besskge.dataset import KGDataset\n",
            "from besskge.loss import LogSigmoidLoss\n",
            "from besskge.metric import Evaluation\n",
            "from besskge.negative_sampler import (\n",
            "    PlaceholderNegativeSampler,\n",
            "    RandomShardedNegativeSampler,\n",
            ")\n",
            "from besskge.scoring import ComplEx\n",
            "from besskge.sharding import PartitionedTripleSet, Sharding\n",
            "\n",
            "dataset_directory = os.getenv(\"DATASET_DIR\", \"../datasets/\") + \"/yago310/\""
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Entity sharding and triple partitioning\n",
            "\n",
            "Download and preprocess the dataset with the built-in method of `KGDataset`, `build_yago310`.\n",
            "\n",
            "You can use the `entity_dict` and `relation_dict` attributes of the `KGDataset` object to go from entity/relation IDs to labels."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 2,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Number of entities: 123,182\n",
                  "\n",
                  "Number of relation types: 37\n",
                  "\n",
                  "Number of triples: \n",
                  " training: 1,079,040 \n",
                  " validation/test: 5,000\n",
                  "\n",
                  "Example triple: ('Charles_VIII_of_France', 'isMarriedTo', 'Anne_of_Brittany')\n"
               ]
            }
         ],
         "source": [
            "yago = KGDataset.build_yago310(root=pathlib.Path(dataset_directory))\n",
            "\n",
            "print(f\"Number of entities: {yago.n_entity:,}\\n\")\n",
            "print(f\"Number of relation types: {yago.n_relation_type}\\n\")\n",
            "print(\n",
            "    f\"Number of triples: \\n training: {yago.triples['train'].shape[0]:,} \\n validation/test: {yago.triples['valid'].shape[0]:,}\\n\"\n",
            ")\n",
            "\n",
            "# Print example triple retrieving labels from yago.entity_dict and yago.relation_dict\n",
            "ex_triple_id = 2500\n",
            "ex_triple = yago.triples[\"train\"][ex_triple_id]\n",
            "print(\n",
            "    f\"Example triple: {yago.entity_dict[ex_triple[0]], yago.relation_dict[ex_triple[1]], yago.entity_dict[ex_triple[2]]}\"\n",
            ")"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "We want to train on 4 IPUs, so we construct a sharding of the entity table in 4 parts. The entity sharding induces a sharding of the triples into 4*4=16 shard-pairs, based on the shard of head and tail entities."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 3,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Global entity IDs on 4 shards:\n",
                  "[[     1      4      5 ... 123174 123176 123177]\n",
                  " [    12     22     25 ... 123179 123180 123182]\n",
                  " [     6      7     11 ... 123166 123168 123175]\n",
                  " [     0      2      3 ... 123178 123181 123183]]\n",
                  "\n",
                  "Reconstructed global entity IDs:\n",
                  "[     0      1      2 ... 123179 123180 123181]\n",
                  "\n",
                  "Number of triples per (h,t) shard-pair:\n",
                  "[[80014 68072 62429 58933]\n",
                  " [79964 68351 62451 59671]\n",
                  " [80045 69389 62231 58903]\n",
                  " [79919 67988 62046 58634]]\n"
               ]
            }
         ],
         "source": [
            "seed = 12345\n",
            "n_shard = 4\n",
            "\n",
            "sharding = Sharding.create(yago.n_entity, n_shard=n_shard, seed=seed)\n",
            "print(f\"Global entity IDs on {n_shard} shards:\")\n",
            "print(sharding.shard_and_idx_to_entity)\n",
            "\n",
            "# The global entity IDs can be recovered, as a function of the shard ID and the local ID on the shard, by\n",
            "print(\"\\nReconstructed global entity IDs:\")\n",
            "print(\n",
            "    sharding.shard_and_idx_to_entity[sharding.entity_to_shard, sharding.entity_to_idx]\n",
            ")\n",
            "\n",
            "train_triples = PartitionedTripleSet.create_from_dataset(yago, \"train\", sharding)\n",
            "\n",
            "print(\"\\nNumber of triples per (h,t) shard-pair:\")\n",
            "print(train_triples.triple_counts)"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "To iterate over the sharded set of triples we use a batch sampler.\n",
            "At each step, `RigidShardedBatchSampler` consumes the same number of triples from all 16 shard-pairs\n",
            "(resampling from the shorter ones, until the longest one is completed).\n",
            "\n",
            "To sample negatives during training we use a negative sampler. \n",
            "`RandomShardedNegativeSampler` constructs, for each triple, negative samples by sampling random corrupted entities.\n",
            "\n",
            "See the [KGE Training and Inference on OGBL-BioKG](1_biokg_training_inference.ipynb) notebook for more details on the options for these class."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 4,
         "metadata": {},
         "outputs": [],
         "source": [
            "device_iterations = 20\n",
            "accum_factor = 2\n",
            "shard_bs = 720\n",
            "neg_sampler = RandomShardedNegativeSampler(\n",
            "    n_negative=1,\n",
            "    sharding=sharding,\n",
            "    seed=seed,\n",
            "    corruption_scheme=\"ht\",\n",
            "    local_sampling=False,\n",
            "    flat_negative_format=False,\n",
            ")\n",
            "bs = RigidShardedBatchSampler(\n",
            "    partitioned_triple_set=train_triples,\n",
            "    negative_sampler=neg_sampler,\n",
            "    shard_bs=shard_bs,\n",
            "    batches_per_step=device_iterations * accum_factor,\n",
            "    seed=seed,\n",
            ")"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 5,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "head         torch.Size([40, 4, 4, 180])   \n",
                  "relation     torch.Size([40, 4, 4, 180])   \n",
                  "tail         torch.Size([40, 4, 4, 180])   \n",
                  "triple_mask  torch.Size([40, 4, 4, 180])   \n",
                  "negative     torch.Size([40, 4, 4, 720, 1])\n"
               ]
            }
         ],
         "source": [
            "options = poptorch.Options()\n",
            "options.replication_factor = sharding.n_shard\n",
            "options.deviceIterations(device_iterations)\n",
            "options.Training.gradientAccumulation(accum_factor)\n",
            "options._popart.setPatterns(dict(RemoveAllReducePattern=True))\n",
            "\n",
            "# Construct the dataloader with the dedicated utility function\n",
            "train_dl = bs.get_dataloader(\n",
            "    options=options, shuffle=True, num_workers=5, persistent_workers=True\n",
            ")\n",
            "\n",
            "# Example batch\n",
            "batch = next(iter(train_dl))\n",
            "for k, v in batch.items():\n",
            "    print(f\"{k:<12} {str(v.shape):<30}\")"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Training\n",
            "\n",
            "Let's train the **ComplEx** KGE model with complex embedding size of 128 and the **logsigmoid** loss function, using the `EmbeddingMovingBessKGE` distribution scheme."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 6,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "# model parameters: 31,544,576\n"
               ]
            }
         ],
         "source": [
            "loss_fn = LogSigmoidLoss(margin=12.0, negative_adversarial_sampling=True)\n",
            "complex_score_fn = ComplEx(\n",
            "    negative_sample_sharing=True,\n",
            "    sharding=sharding,\n",
            "    n_relation_type=yago.n_relation_type,\n",
            "    embedding_size=128,\n",
            ")\n",
            "model = EmbeddingMovingBessKGE(\n",
            "    negative_sampler=neg_sampler, score_fn=complex_score_fn, loss_fn=loss_fn\n",
            ")\n",
            "\n",
            "print(f\"# model parameters: {model.n_embedding_parameters:,}\")"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "To follow training evolution, let us compute at regular intervals the MRR for tail prediction on a small random subset of 500 validation queries. Since no candidate tails are provided in this dataset, we score each query against all entities in the knowledge graph and select the top-10 candidate tails with largest scores. We then rank the ground truth tail among the ordered predictions. By setting `worst_rank_infty=True` in the `Evaluation` module, we assign a reciprocal rank of 0 (or a rank of infinity) if the ground truth is not among the top-10 predictions made by the model.\n",
            "\n",
            "As the number of entities in the knowledge graph is not so large, we can afford to score the queries on the CPU. To see how to speed up this task on IPUs (which is necessary on graphs with more entities, or when scoring a large number of queries), see the [Top-k inference on IPU](#top-k-inference-on-ipu) section."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 7,
         "metadata": {},
         "outputs": [],
         "source": [
            "def evaluate_mrr_cpu(triples, evaluation):\n",
            "    # Unshard entity embedding table\n",
            "    ent_table = complex_score_fn.entity_embedding.detach()[\n",
            "        sharding.entity_to_shard, sharding.entity_to_idx\n",
            "    ]\n",
            "\n",
            "    # Score query (h,r,?) against all entities in the knowledge graph and select top-10 scores\n",
            "    scores = complex_score_fn.score_tails(\n",
            "        ent_table[triples[:, 0]],\n",
            "        torch.from_numpy(triples[:, 1]),\n",
            "        ent_table.unsqueeze(0),\n",
            "    )\n",
            "    top_k = torch.topk(scores, dim=-1, k=10)\n",
            "\n",
            "    # Use evaluation.ranks_from_indices to rank the ground truth, if present, among the predictions\n",
            "    ranks = evaluation.ranks_from_indices(\n",
            "        torch.from_numpy(triples[:, 2]), top_k.indices.squeeze()\n",
            "    )\n",
            "    return {\n",
            "        k: v / triples.shape[0]\n",
            "        for k, v in evaluation.dict_metrics_from_ranks(ranks).items()\n",
            "    }\n",
            "\n",
            "\n",
            "# Sample validation queries\n",
            "n_val_triples = 500\n",
            "val_triple_subset = yago.triples[\"valid\"][\n",
            "    np.random.default_rng(seed=1000).choice(\n",
            "        yago.triples[\"valid\"].shape[0], n_val_triples\n",
            "    )\n",
            "]\n",
            "evaluation = Evaluation([\"mrr\"], worst_rank_infty=True, reduction=\"sum\")"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 8,
         "metadata": {},
         "outputs": [
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "Graph compilation: 100%|██████████| 100/100 [00:47<00:00]\n"
               ]
            }
         ],
         "source": [
            "opt = poptorch.optim.AdamW(\n",
            "    model.parameters(),\n",
            "    lr=0.0016,\n",
            ")\n",
            "\n",
            "poptorch_model = poptorch.trainingModel(model, options=options, optimizer=opt)\n",
            "\n",
            "# The variable entity_embedding needs to hold different values on each replica,\n",
            "# corresponding to the shards of the entity embedding table\n",
            "poptorch_model.entity_embedding.replicaGrouping(\n",
            "    poptorch.CommGroupType.NoGrouping,\n",
            "    0,\n",
            "    poptorch.VariableRetrievalMode.OnePerGroup,\n",
            ")\n",
            "\n",
            "# Graph compilation\n",
            "_ = batch.pop(\"triple_mask\")\n",
            "res = poptorch_model(**{k: v.flatten(end_dim=1) for k, v in batch.items()})"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 9,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Epoch 1 loss: 4.582457 --- positive triples processed: 1.38e+06\n",
                  "Epoch duration (sec): 0.59385 (average step time: 0.03974)\n",
                  "Epoch 1 sample MRR: 0.0000\n",
                  "Epoch 2 loss: 0.980555 --- positive triples processed: 2.76e+06\n",
                  "Epoch duration (sec): 0.47770 (average step time: 0.03968)\n",
                  "Epoch 3 loss: 0.749413 --- positive triples processed: 4.15e+06\n",
                  "Epoch duration (sec): 0.47638 (average step time: 0.03960)\n",
                  "Epoch 4 loss: 0.707479 --- positive triples processed: 5.53e+06\n",
                  "Epoch duration (sec): 0.47575 (average step time: 0.03955)\n",
                  "Epoch 5 loss: 0.675585 --- positive triples processed: 6.91e+06\n",
                  "Epoch duration (sec): 0.47491 (average step time: 0.03948)\n",
                  "Epoch 6 loss: 0.673483 --- positive triples processed: 8.29e+06\n",
                  "Epoch duration (sec): 0.47486 (average step time: 0.03949)\n",
                  "Epoch 6 sample MRR: 0.0809\n",
                  "Epoch 7 loss: 0.660370 --- positive triples processed: 9.68e+06\n",
                  "Epoch duration (sec): 0.47620 (average step time: 0.03958)\n",
                  "Epoch 8 loss: 0.651325 --- positive triples processed: 1.11e+07\n",
                  "Epoch duration (sec): 0.47522 (average step time: 0.03949)\n",
                  "Epoch 9 loss: 0.650032 --- positive triples processed: 1.24e+07\n",
                  "Epoch duration (sec): 0.47702 (average step time: 0.03964)\n",
                  "Epoch 10 loss: 0.662921 --- positive triples processed: 1.38e+07\n",
                  "Epoch duration (sec): 0.48052 (average step time: 0.03987)\n",
                  "Epoch 11 loss: 0.669976 --- positive triples processed: 1.52e+07\n",
                  "Epoch duration (sec): 0.47592 (average step time: 0.03954)\n",
                  "Epoch 11 sample MRR: 0.1628\n",
                  "Epoch 12 loss: 0.675865 --- positive triples processed: 1.66e+07\n",
                  "Epoch duration (sec): 0.47705 (average step time: 0.03962)\n",
                  "Epoch 13 loss: 0.677261 --- positive triples processed: 1.80e+07\n",
                  "Epoch duration (sec): 0.47530 (average step time: 0.03952)\n",
                  "Epoch 14 loss: 0.681144 --- positive triples processed: 1.94e+07\n",
                  "Epoch duration (sec): 0.47456 (average step time: 0.03946)\n",
                  "Epoch 15 loss: 0.677323 --- positive triples processed: 2.07e+07\n",
                  "Epoch duration (sec): 0.47523 (average step time: 0.03950)\n",
                  "Epoch 16 loss: 0.687818 --- positive triples processed: 2.21e+07\n",
                  "Epoch duration (sec): 0.47560 (average step time: 0.03955)\n",
                  "Epoch 16 sample MRR: 0.2150\n",
                  "Epoch 17 loss: 0.684768 --- positive triples processed: 2.35e+07\n",
                  "Epoch duration (sec): 0.47711 (average step time: 0.03966)\n",
                  "Epoch 18 loss: 0.687305 --- positive triples processed: 2.49e+07\n",
                  "Epoch duration (sec): 0.47555 (average step time: 0.03951)\n",
                  "Epoch 19 loss: 0.695313 --- positive triples processed: 2.63e+07\n",
                  "Epoch duration (sec): 0.47482 (average step time: 0.03947)\n",
                  "Epoch 20 loss: 0.709156 --- positive triples processed: 2.76e+07\n",
                  "Epoch duration (sec): 0.47474 (average step time: 0.03945)\n",
                  "Epoch 21 loss: 0.709045 --- positive triples processed: 2.90e+07\n",
                  "Epoch duration (sec): 0.47610 (average step time: 0.03956)\n",
                  "Epoch 21 sample MRR: 0.2206\n",
                  "Epoch 22 loss: 0.702439 --- positive triples processed: 3.04e+07\n",
                  "Epoch duration (sec): 0.47680 (average step time: 0.03962)\n",
                  "Epoch 23 loss: 0.708583 --- positive triples processed: 3.18e+07\n",
                  "Epoch duration (sec): 0.47523 (average step time: 0.03950)\n",
                  "Epoch 24 loss: 0.708993 --- positive triples processed: 3.32e+07\n",
                  "Epoch duration (sec): 0.47482 (average step time: 0.03946)\n",
                  "Epoch 25 loss: 0.710253 --- positive triples processed: 3.46e+07\n",
                  "Epoch duration (sec): 0.47490 (average step time: 0.03945)\n",
                  "Final sample MRR: 0.2176\n"
               ]
            },
            {
               "data": {
                  "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmYAAAGwCAYAAAADo6klAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAB6d0lEQVR4nO3dd3RU1d7G8e+kkwohkNBDb9KkSRN8QYqggooIKEWEixdU4NrwIqCi2MWCgI2iIigqei0gIogCinSko3QSenqfOe8fhwxEEkgmk5yU57PWLKecOfOb4zDzZO999rYZhmEgIiIiIpbzsLoAERERETEpmImIiIgUEQpmIiIiIkWEgpmIiIhIEaFgJiIiIlJEKJiJiIiIFBEKZiIiIiJFhJfVBeRHRkYGW7ZsITw8HA8PZUwREZHiwOFwcPLkSVq0aIGXV7GOIm5XrI/Gli1baNOmjdVliIiIiAs2bNhA69atrS6jSCnWwSw8PBww/8dWqlTJ4mpEREQkN6KiomjTpo3zd1wuKtbBLLP7slKlSlStWtXiakRERCQvNAzpcjoiIiIiIkWEgpmIiIhIEaFgJiIiIlJEFOsxZrllt9tJT0+3uoxix8fHR/3/IiIihahEBzPDMIiOjiYmJsbqUoolDw8PatasiY+Pj9WliIiIlAolOphlhrKKFSvi7++PzWazuqRiw+FwcOLECaKioqhevbqOnYiISCEoscHMbrc7Q1n58uWtLqdYqlChAidOnCAjIwNvb2+ryxERESnxSuwAoswxZf7+/hZXUnxldmHa7XaLKxERESkdSmwwy6QuONfp2ImIiBSuEh/MRERERIoLy4PZ8ePHufvuuylfvjxlypShSZMmbNy40eqyRERERAqdpYP/z58/T4cOHbjhhhv4/vvvqVChAvv376dcuXJWliUiIiJiCUuD2QsvvEC1atWYO3eu876aNWtaWNFFcSnpBPl6WTLOatiwYcTExLB06dJCf20RkWIhLQ3OnAEfHwgKAl9fqysScQtLg9nXX39Njx496N+/Pz///DNVqlTh3//+NyNHjsx2+9TUVFJTU5234+PjC6SuuOR0Dp1NxN/Hi+qhZfDx8iyQ1xERkUsYBpw/D9HRFy9RUdlfP3s263O9vc2AltdLcPDl9wUGgqe+98Ualgazv//+m1mzZjFhwgSeeOIJ/vjjDx588EF8fHwYOnToZdtPnz6dp556yuXXMwyD5PSrT/2QmJZBeoaDc+mpxCanUbtCQL7CWRlvT7e1vP3888888sgjbNu2jdDQUIYOHcq0adPw8jL/Vy5ZsoSnnnqKAwcO4O/vT4sWLfjqq68ICAhg9erVPProo+zcuRNvb28aN27MwoULqVGjhltqExHJVmoqnDyZc8i69HpaWu736+kJmdP5pKfDuXPmxR38/fMf8DIvZcqAznKXXLI0mDkcDlq1asVzzz0HQIsWLfjzzz+ZPXt2tsFs4sSJTJgwwXn7+PHjNGrUKNevl5xup9Hk5fkvPI92Pd0Df5/8H+rjx49z0003MWzYMBYsWMCePXsYOXIkfn5+TJ06laioKAYOHMiLL75Iv379iI+P55dffsEwDDIyMujbty8jR47kk08+IS0tjQ0bNmhKDBFxjWGYIehKISvz+vnzedt3uXJQqRJERJiXnK6HhoLDAQkJEB+f/SUuLufHsrtkrquclGReTp7M/7Hy8HBfyAsKMlsHpcSyNJhVqlTpsmDVsGFDPv/882y39/X1xfeScQRxcXEFWl9R8/bbb1OtWjXeeustbDYbDRo04MSJEzz22GNMnjyZqKgoMjIyuO2225ytYE2aNAHg3LlzxMbG0qdPH2rXrg2Yx1pEJIuUlKu3akVFmYElM8Tkho/PlUNW5vXw8LyNF/P0hJAQ8+IOqamuh7rsLmCGx9hY8+IOvr7uCXhBQRAQYAZHKTIsDWYdOnRg7969We7bt29fgXWtlfH2ZNfTPXK9vWEYHDyTRFJaBmGBvkSE+Ln8uu6we/du2rVrl6WVq0OHDiQkJHDs2DGaNWtG165dadKkCT169KB79+7ccccdlCtXjtDQUIYNG0aPHj248cYb6datG3feeSeVKlVyS20iUoQ5HOaYrNwErryGh/LlLw9Z2YWusmWLR3eer695CQvL/74cDkhMdC3QZRcKM8dYp6aalzNn8l+jzWaGsyuFt7ZtYfjw/L+W5IqlwWz8+PG0b9+e5557jjvvvJMNGzbwzjvv8M477xTI69lstjx3KVYLLcPhs0mk2w23dEcWJE9PT1asWMG6dev44YcfePPNN/nvf//L77//Ts2aNZk7dy4PPvggy5YtY/HixUyaNIkVK1Zw3XXXWV26iLgiKSl3A+VPnoSMjNzv19c3d12J4eFmS5hk79IuTHdIT89fsPvnxeEwu6QTEsxLVFT2rxsXp2BWiCxNGq1bt+bLL79k4sSJPP3009SsWZMZM2YwePBgK8vKIuBCGEvNsJNhd+DlaV2Tb2Y3r2EYzlaztWvXEhQURNWqVQEzfHbo0IEOHTowefJkatSowZdffukcm9eiRQtatGjBxIkTadeuHQsXLlQwEylKHA6zJSQ3A+XzOpwjLCznkHXp7ZCQ4tG6Vdp4e5vj6kJD878vw4Dk5NwFu2uuyf/rSa5Z3gTUp08f+vTpY3UZOfLy9MDXy5PUDDvJ6XaCCimYxcbGsnXr1iz3jRo1ihkzZvDAAw8wduxY9u7dy5QpU5gwYQIeHh78/vvvrFy5ku7du1OxYkV+//13Tp8+TcOGDTl48CDvvPMOt9xyC5UrV2bv3r3s37+fIUOGFMr7ESn1EhNzN1D+1KmLZxrmhp/fxUB1pVauihU1aFwustnMM0/9/c2WTykyLA9mxYG/jxnMktLsBPkVzhfb6tWradGiRZb7RowYwXfffccjjzxCs2bNCA0NZcSIEUyaNAmA4OBg1qxZw4wZM4iLi6NGjRq88sor9OrVi5MnT7Jnzx7mz5/P2bNnqVSpEmPGjOFf//pXobwfkVJj1Sr48svLuxgTEnK/D5sNKlS4eldipUpmN5lat0RKDJthGIbVRbjq2LFjVKtWjaNHjzq78jKlpKRw8OBBatasiZ+fa4P2M51NSOV4TDJBft7UDAvI176KE3ceQ5FS4Y03YNw4s5soO/7+uetKrFBBrVtSol3p97u0U4tZLvj7mGdVJqVlZBnfJSICmOPCHn4YXnvNvH3nndChw+WBKzBQrVsickUKZrng5+2Jh82G3WGQmuHAz03TX4hICZCSAvfcA0uWmLenT4fHHlMAExGXKJjlgs1mw8/bk6S0DFIz7ApmImI6exZuvRXWrjW7HufNg0GDrK5KRIoxBbNc8vY0//pNyyi2Q/JExJ3+/ht69YJ9+8zpJZYuhS5drK5KRIo5BbNc8vEyp8lItzssrkRELLdhA/TpA6dPQ/Xq8N130Lix1VWJSAmgBbJyydtTwUxEgP/9z2wZO30amjeH9esVykTEbRTMcsnnQjBLy1AwEym1Zs2Cvn3NGdN79IA1a6ByZaurEpESRMEsly62mGmMmUip43CYZ1r++9/m9REjzJYzd62BKCJygcaY5ZKPlzn4P8PhwO4w8PTQqfAipUJqKgwbBosWmbeffhomTdJ0GCJSINRilkueHh54XvgiLuhxZsOGDcNmszF69OjLHhszZgw2m41hw4Zl2dZms+Ht7U3NmjV59NFHSUlJyfK8zG1sNhvBwcG0bt2ar776qkDfh0ixd/48dO9uhjIvL3M6jCefVCgTkQKjYJYH3oV4Zma1atVYtGgRycnJzvtSUlJYuHAh1atXz7Jtz549iYqK4u+//+a1115jzpw5TJky5bJ9zp07l6ioKDZu3EiHDh2444472LFjR4G/F5Fi6fBhc/b+NWvMLsvvv4ehQ62uSkRKOAWzPCjMEwCuvfZaqlWrxhdffOG874svvqB69eqXLW7u6+tLREQE1apVo2/fvnTr1o0VK1Zcts+yZcsSERFBvXr1eOaZZ8jIyGDVqlUF/l5Eip3Nm+G662D3bqhSBX79Fbp1s7oqESkFSlcwMwxITHT54pOSjC0pkYy4+Lw918V14u+9917mzp3rvP3BBx8wfPjwKz7nzz//ZN26dfj4+OS4TUZGBu+//z7AFbcTKZW+/x6uvx6io6FJE/jtN2ja1OqqRKSUKF2D/5OSzEWEXVT5wiXPEhIgICDPT7v77ruZOHEihw8fBmDt2rUsWrSI1atXZ9num2++ITAwkIyMDFJTU/Hw8OCtt966bH8DBw7E09OT5ORkHA4HkZGR3Hnnna68I5GS6d134f77wW6Hrl3h88/NWf1FRApJ6QpmxUyFChXo3bs38+bNwzAMevfuTVhY2GXb3XDDDcyaNYvExERee+01vLy8uP322y/b7rXXXqNbt278/fffjB8/njfeeIPQ0NDCeCsiRZthwOTJMG2aeXvIEDOkqUVZRApZ6Qpm/v5m65WLElIyOHg2AV8vT+qF52H+In9/l1/z3nvvZezYsQDMnDkz220CAgKoU6cOYHZ3NmvWjPfff58RI0Zk2S4iIoI6depQp04d5s6dy0033cSuXbuoWLGiy/WJFHtpaXDfffDhh+btyZNh6lSdeSkilihdwcxmc6lLMZOXjx0j2SDDI3/7yYuePXuSlpaGzWajR48eV93ew8ODJ554ggkTJjBo0CDKlCmT7XZt2rShZcuWPPvss7z++uvuLlukeIiNhdtug59+Ak9PmDPHnDxWRMQipWvwfz5lTiqb4TAwXBzQn+fX9PRk9+7d7Nq1C09Pz1w9p3///nh6eubYwpZp3LhxzJkzh+PHj7ujVJHi5ehR6NjRDGWBgfDNNwplImI5BbM88Lpktv8MR+EtzRQcHExwcHCut/fy8mLs2LG8+OKLJCYm5rhdz549qVmzJs8++6w7yhQpPrZtg3bt4M8/ISLCnKusZ0+rqxIRUTDLC5vN5mw1sxdgMJs3bx5Lly7N8fGlS5cyb968K277+OOPc+rUKQIudLkahkHfvn2zbGOz2di9ezdvv/22myoXKQZWrIBOneD4cWjY0JwO4x9zA4pI0TRz5kwiIyPx8/Ojbdu2bNiwIcdt3333XTp16kS5cuUoV64c3bp1u2x7wzCYPHkylSpVokyZMnTr1o39+/cX9Nu4IgWzPPLyMA9ZhhYzFyl+5s2Dm26C+Hjo3BnWroUaNayuSkRyYfHixUyYMIEpU6awefNmmjVrRo8ePTh16lS2269evZqBAweyatUq1q9fT7Vq1ejevXuW4Tsvvvgib7zxBrNnz+b3338nICCAHj16XLasYWFSMMsjL2eLWcHP/i8ibmIY5uLjw4dDRgYMHAjLl0O5clZXJlKqxcfHExcX57ykpqbmuO2rr77KyJEjGT58OI0aNWL27Nn4+/vzwQcfZLv9xx9/zL///W+aN29OgwYNeO+993A4HKxcuRIwW8tmzJjBpEmTuPXWW2natCkLFizgxIkTV+y1KmgKZnnk5XnxBAARKQbS083pMDLXj338cfjoI/D1tbYuEaFRo0aEhIQ4L9OnT892u7S0NDZt2kS3S5ZG8/DwoFu3bqxfvz5Xr5WUlER6erpz/s6DBw8SHR2dZZ8hISG0bds21/ssCKVrugw3uPTMTBEp4uLioH9/+OEH8PCAmTNh9GirqxKRC3bt2kWVKlWct31z+IPpzJkz2O12wsPDs9wfHh7Onj17cvVajz32GJUrV3YGsejoaOc+/rnPzMesUOKDmbuntXCOMSsFwaywpgQRKRAnTpjjybZtMyd5XrwY+vSxuioRuURQUFCeZh1w1fPPP+9c0tDPz6/AXy8/SmxXpre3N2A2XbqT86xMe8kfY5aWlgaQ6/nTRIqMnTvhuuvMUFaxIqxerVAmUoyFhYXh6enJyZMns9x/8uRJIiIirvjcl19+meeff54ffviBpk2bOu/PfJ4r+yxIJbbFzNPTk7JlyzrP1vD398fmhiVWHBlpGBlppKbaSUkpuYHF4XBw+vRp/P398fIqsR8TKYlWrYJ+/cxZ/evXh++/h5o1ra5KRPLBx8eHli1bsnLlSufUT5kD+TOXLczOiy++yLPPPsvy5ctp1apVlsdq1qxJREQEK1eupHnz5gDExcXx+++/c//99xfUW7mqEv2Lm5l4czqV1hUp6XbOJKTh7WkjI7ZoN4fml4eHB9WrV3dLoBUpFB9/bJ55mZ4OHTrAV19B+fJWVyUibjBhwgSGDh1Kq1ataNOmDTNmzCAxMZHhw4cDMGTIEKpUqeI8geCFF15g8uTJLFy4kMjISOe4scDAQAIDA7HZbIwbN45p06ZRt25datasyZNPPknlypUvm/ezMJXoYGaz2ahUqRIVK1YkPT3dLfvcdzKeqf/bRGiAD5+Nbu+WfRZVPj4+eHiU2N5uKUkMA55/Hp54wrzdvz8sWABFfCyJiOTegAEDOH36NJMnTyY6OprmzZuzbNky5+D9I0eOZPnNmjVrFmlpadxxxx1Z9jNlyhSmTp0KwKOPPkpiYiKjRo0iJiaGjh07smzZMkvHodmMYjzC+9ixY1SrVo2jR49StWrVQnnNqNhk2k3/CW9PG/um9VJrkojVMjJg7FhzAXKA//wHXnzRPAtTRIokK36/i4sS3WJWEMr5+wCQbjeIS8kgpIy3xRWJlGIJCTBgAHz3Hdhs8Prr8MADVlclIuIyBbM88vP2JMDHk8Q0O+cS0xTMRKwSHW2eablpk9lluXChOehfRKQYU1u/C0IDzVazc4k5Lx0hIgVozx5o184MZWFhF8/EFBEp5hTMXJDZnRmT5J4TCkQkD375Bdq3h0OHoE4dWL/enLNMRKQEUDBzQaCv2QOckJphcSUipczixdCtG5w/b4axdevMcCYiUkIomLlAwUykkBkGvPwy3HUXpKVB376wciVUqGB1ZSIibqVg5oJAvwvBLEXBTKTA2e3mmZaPPGLefuABWLLEXP9SRKSE0VmZLghSi5lI4UhKgkGDzBn8AV55BcaPN6fGEBEpgRTMXBBwIZjFq8VMpOCcOgU33wwbNoCvL3z4oTmjv4hICaZg5oLMrsxEtZiJFIx9+6BXL/j7bwgNNVvMOna0uioRkQKnYOYCdWWKFKB16+CWW+DsWahZE77/HurXt7oqEZFCocH/LnAO/lcwE3GvL76Arl3NUNaqlTlHmUKZiJQiCmYuCPQ1l2HSGDMRN3r9dbjjDkhJMZdaWr0awsOtrkpEpFApmLkgwNcT0BgzEbdwOMwzLceNM+crGz0avvwSAgKsrkxEpNBpjJkLgi60mKkrUySfkpPhnnvg88/N288/D48+qukwRKTUUjBzgSaYFXGDM2fg1lvNwf7e3jBvnjlnmYhIKaZg5gLnkkxpGRiGgU1/3YvkzV9/mdNh7N8PISGwdCl06WJ1VSIiltMYMxcEXWgxMwxISrNbXI1IMbNhA7RrZ4ay6tVh7VqFMhGRCxTMXODr5YGnh9lKpnFmInnw9ddmCDt9Glq0MKfDaNzY6qpERIoMS4PZ1KlTsdlsWS4NGjSwsqRcsdlszu5MTZkhkktvvw39+pkD/nv2hJ9/hsqVra5KRKRIsXyMWePGjfnxxx+dt728LC8pVwJ9vYhNTleLmcjVOBwwcSK8+KJ5e8QImDXLHPAvIiJZWJ6CvLy8iIiIsLqMPAvSmZkiV5eSAsOGweLF5u1nnoH//lfTYYiI5MDyYLZ//34qV66Mn58f7dq1Y/r06VSvXj3bbVNTU0lNTXXejo+PL6wyLxOo9TJFruzcObPrcs0a8PKC99+HIUOsrkpEpEizdIxZ27ZtmTdvHsuWLWPWrFkcPHiQTp065Ri4pk+fTkhIiPPSqFGjQq74ogAFM5GcHToEHTuaoSwoyFyIXKFMROSqLA1mvXr1on///jRt2pQePXrw3XffERMTw6effprt9hMnTiQ2NtZ52bVrVyFXfNHFSWbTLatBpEjatMmcDmP3bqhSBX79Fbp1s7oqEZFiwfKuzEuVLVuWevXqceDAgWwf9/X1xdfX13k7Li6usEq7TJBazEQu9913cOedkJgITZqYt6tWtboqEZFio0jNY5aQkMBff/1FpUqVrC7lqi6OMdMEsyIAvPsu3HKLGcq6doVfflEoExHJI0uD2cMPP8zPP//MoUOHWLduHf369cPT05OBAwdaWVauOLsyU9WVKaWcYcCkSTBqFNjt5liy774zl1oSEZE8sbQr89ixYwwcOJCzZ89SoUIFOnbsyG+//UaFChWsLCtXnC1mmi5DSrO0NHNeso8+Mm9PngxTp2o6DBERF1kazBYtWmTly+eLpsuQUi8mBm67DVatAk9PmDPHDGkiIuKyIjX4vzjJ7MrUkkxSKh09Cr16wc6dEBgIn31mLrMkIiL5omDmosx5zBLTFMyklNm2DW66CU6cgIgIczxZixZWVyUiUiIUqbMyixN/b08AktJ0VqaUIj/8AJ06maGsUSP47TeFMhERN1Iwc5G/j9lilqxgJqXFvHnQuzfEx0PnzubEsTVqWF2ViEiJomDmojI+5qFTi5mUeIYBTz0Fw4dDRgYMHAjLl0O5clZXJiJS4miMmYvKZLaYpSuYSQmWng7/+hfMnWvefvxxePZZ8NDfdCIiBUHBzEWZY8zSMhzYHQaeHpq3SUqYuDi44w5YscIMYjNnwujRVlclIlKiKZi5qIyPp/N6UloGQX7eFlYj4mbHj5vjybZtA39/WLwY+vSxuioRkRJP/REu8vXyILORTCcASIny55/Qrp0ZyipWhNWrFcpERAqJgpmLbDab88xMnQAgJcZPP0HHjuYEsvXrm9NhtG5tdVUiIqWGglk+ZHZnKphJifDRR+bs/bGxZjhbuxZq1rS6KhGRUkXBLB/8LwSz5HTN/i/FmGHAc8/BPfeYZ2H2728O+C9f3urKRERKHQWzfCij2f+luMvIMM+0/O9/zdv/+Q8sWgR+ftbWJSJSSumszHzwV1emFGcJCTBggLnWpc0Gr78ODzxgdVUiIqWaglk+aFkmKbaio83pMDZvNlvHFi6Efv2srkpEpNRTMMsHP3VlSnG0ezf06gWHD0NYGPzvf3DddVZXJSIiaIxZvlzsytTgfykm1qyBDh3MUFanDqxfr1AmIlKEKJjlg/OsTLWYSXGwaBHceCOcP2+GsXXrzHAmIiJFhoJZPjjnMdNC5lLUff45DBwIaWnQty+sXAkVKlhdlYiI/IOCWT6oxUyKhcOH4b77zOsjR8KSJeb6lyIiUuQomOXDxSWZNMZMiqiMDBg8GGJioE0bmDkTPD2trkpERHKgYJYPmRPMJqc7LK5EJAfTpplLKwUFmVNieHtbXZGIiFyBglk+XOzKVIuZFEFr1sAzz5jXZ8+G2rWtrUdERK5KwSwftIi5FFnnzpldmA4HDBkCgwZZXZGIiOSCglk+XBxjpmAmRYhhmIP8jx0zp8N46y2rKxIRkVxSMMsHnZUpRdI778AXX5jjyT75xBxfJiIixYKCWT5cnMdMY8ykiNi5E8aNM68/9xy0amVpOSIikjcKZvngPCtTLWZSFCQnm5PIpqRA9+4wYYLVFYmISB4pmOWDvwb/S1HyyCOwYwdUrAjz54OH/nmLiBQ3+ubOh8yuzOR0O4ZhWFyNlGpff21OHgswbx5ERFhajoiIuEbBLB8yz8o0DEjRJLNilePHYfhw8/r48dCrl7X1iIiIyxTM8iFzjBloWSaxiN0O99xjzlvWogVMn251RSIikg8KZvng6WHD18s8hBpnJpZ44QVYtQoCAmDRIvD1tboiERHJBwWzfPK/ZJyZSKFavx4mTzavv/km1KtnbT0iIpJvCmb5pNn/xRKxseYyS3Y73HUXDBtmdUUiIuIGCmb5dHG9TI0xk0JiGDB6NBw6BJGR5gLlNpvVVYmIFLiZM2cSGRmJn58fbdu2ZcOGDTluu3PnTm6//XYiIyOx2WzMmDHjsm2mTp2KzWbLcmnQoEEBvoOrUzDLJy3LJIVu/nxzPJmnJyxcCCEhVlckIlLgFi9ezIQJE5gyZQqbN2+mWbNm9OjRg1OnTmW7fVJSErVq1eL5558n4gpTCDVu3JioqCjn5ddffy2ot5ArCmb55OdlBjNNlyGFYu9eGDvWvP7009CunbX1iIjkQ3x8PHFxcc5Lampqjtu++uqrjBw5kuHDh9OoUSNmz56Nv78/H3zwQbbbt27dmpdeeom77roL3yucGOXl5UVERITzEhYWlu/3lR8KZvnk620ewhQN/peClppqLrmUmAhdusBjj1ldkYhIvjRq1IiQkBDnZXoOU/6kpaWxadMmunXr5rzPw8ODbt26sX79+nzVsH//fipXrkytWrUYPHgwR44cydf+8svL0lcvAfwuzGWWkqFgJgXsiSdgyxYIDYWPPjK7MkVEirFdu3ZRpUoV5+2cWrbOnDmD3W4nPDw8y/3h4eHs2bPH5ddv27Yt8+bNo379+kRFRfHUU0/RqVMn/vzzT4KCglzeb34omOVT5iSz6sqUArVsGbz6qnn9gw/gki8yEZHiKigoiODgYMtev9clK6U0bdqUtm3bUqNGDT799FNGjBhhSU3qyswnP3VlSkGLjoahQ83rY8bArbdaW4+ISCELCwvD09OTkydPZrn/5MmTVxzYn1dly5alXr16HDhwwG37zCsFs3zK7MpMVTCTguBwmKHs1Clo0gReesnqikRECp2Pjw8tW7Zk5cqVzvscDgcrV66knRtPgkpISOCvv/6iUqVKbttnXqkrM58yg5lm/pcC8eqr8MMP4OdnTpFRpozVFYmIWGLChAkMHTqUVq1a0aZNG2bMmEFiYiLDhw8HYMiQIVSpUsV5AkFaWhq7du1yXj9+/Dhbt24lMDCQOnXqAPDwww9z8803U6NGDU6cOMGUKVPw9PRk4MCB1rxJFMzyzc8rsytTY8zEzTZuNAf8A8yYAY0aWVqOiIiVBgwYwOnTp5k8eTLR0dE0b96cZcuWOU8IOHLkCB4eFzsCT5w4QYsWLZy3X375ZV5++WU6d+7M6tWrATh27BgDBw7k7NmzVKhQgY4dO/Lbb79RoUKFQn1vl1Iwyydf5+B/tZiJG8XHm1NjpKfDbbfBqFFWVyQiYrmxY8cyNnMux3/IDFuZIiMjMQzjivtbtGiRu0pzG40xy6eL02WoxUzc6IEH4MABqFoV3n1XSy6JiJQSCmb5VEYtZuJuCxeayy55eMDHH5vzlomISKmgYJZPmi5D3Orvv80FygEmTYLrr7e2HhERKVQKZvl0cboMdWVKPqWnm+PK4uOhQwd48kmrKxIRkUKmYJZPmS1mmi5D8m3yZNiwAUJCzC5ML52bIyJS2iiY5ZOfl8aYiRusXAkvvGBef/ddqFHD2npERMQSRSaYPf/889hsNsaNG2d1KXniq0XMJb9On4Z77gHDgJEjoX9/qysSERGLFIlg9scffzBnzhyaNm1qdSl5dnHwv8aYiQsMA+69F6KioEEDeO01qysSERELWR7MEhISGDx4MO+++y7lypW74rapqanExcU5L/Hx8YVUZc40XYbky1tvwTffgI+PueRSQIDVFYmIiIUsD2Zjxoyhd+/edOvW7arbTp8+nZCQEOelURFYokZnZYrLtm2Dhx82r7/0EjRrZm09IiJiOUuD2aJFi9i8ebNzwdGrmThxIrGxsc5L5uKkVsoMZml2B3bHlZd+EHFKTIS77oK0NOjd25zpX0RESj3Lzsc/evQoDz30ECtWrMDPzy9Xz/H19cXX19d5Oy4urqDKy7XMMWZgdmcG+GqKA8mF8eNhzx6oVAnmztWSSyIiJcjx48epUqWKS8+1rMVs06ZNnDp1imuvvRYvLy+8vLz4+eefeeONN/Dy8sJuLx5jtjKnywCNM5NcWrLk4vqXH34IFSpYXZGIiLhBdHQ0DzzwAHXr1nV5H5YFs65du7Jjxw62bt3qvLRq1YrBgwezdetWPD09r76TIsDDw4aP54UzM7WQuVzN4cPmlBgAjz0GXbtaW4+IiOTJ+fPnGThwIGFhYVSuXJk33ngDh8PB5MmTqVWrFn/88Qdz5851ef+W9bsFBQVxzTXXZLkvICCA8uXLX3Z/Uefr7UGa3aEWM7myjAwYPBhiYqBNG3j6aasrEhGRPHr88cdZt24dw4YNY/ny5YwfP55ly5bh4eHBTz/9xHXXXZev/Vt+VmZJoCkzJFemTYO1ayEoCBYuBG9vqysSEZE8+v7775k7dy4vv/wy//vf/zAMg+bNm/PNN9/kO5SBhS1m2Vm9erXVJbjEzxnM1JUpOfjlF3jmGfP67NlQu7a19YiIiEtOnDhBw4YNAYiMjMTPz4+7777bbftXi5kbXJz9Xy1mko1z58wuTIcDhgyBQYOsrkhERFxkGAZeXhfbtTw9PSlTpozb9l+kWsyKKz91ZUpOMte/PHoU6tQxZ/oXEZFiyzAMunbt6gxnycnJ3Hzzzfj4+GTZbvPmzS7tX8HMDTKnzFBXplzmnXfgiy/M8WSffGKOLxMRkWJrypQpWW7feuutbt2/gpkb+KorU7KzcyeMG2def+45aNXK0nJERCT//hnM3E1jzNzA2ZWZoWAmF6SkwMCB5n+7d4cJE6yuSERECkFKSgovv/yyy89XMHODMjorU/7pkUdgxw6oWBHmzwcP/VMTESkpTp8+zTfffMMPP/zgXKkoPT2d119/ncjISJ5//nmX962uTDfQWZmSxddfXxzkP28eRERYWo6IiLjPr7/+Sp8+fYiLi8Nms9GqVSvmzp1L37598fLyYurUqQwdOtTl/evPeDfQWZnidPw4DB9uXh8/Hnr1srYeERFxq0mTJnHTTTexfft2JkyYwB9//EG/fv147rnn2LVrF6NHj87X9BkKZm6gYCYA2O1wzz3mvGUtWsD06VZXJCIibrZjxw4mTZrENddcw9NPP43NZuPFF1/kjjvucMv+FczcwM8rsytTY8xKtRdegFWrICAAFi0CX1+rKxIRETc7f/48YWFhAJQpUwZ/f3+3rvGtMWZu4KsWM/ntN5g82bz+5ptQr5619YiISIHZtWsX0dHRgDnh7N69e0lMTMyyTdOmTV3at4KZGzjPysxQi1mpFBtrTo1ht8Ndd8GwYVZXJCIiBahr164YhuG83adPHwBsNhuGYWCz2Zxna+aVgpkbaIxZKWYYMHo0HDoEkZHmAuU2m9VViYhIATl48GCB7l/BzA00XUYpNn++OZ7M0xMWLoSQEKsrEhGRAlSjRo0C3b+CmRuoxayU2rcPxo41rz/9NLRrZ209IiJS4I4cOZKr7apXr+7S/hXM3OBii5nGmJUaqanmeLLEROjSBR57zOqKRESkENSsWdN5PXOcme2SISwaY1YE+HmpxazUeeIJ2LIFQkPho4/MrkwRESnxbDYbVatWZdiwYdx88814ebk3SimYuYGvFjEvXZYtg1dfNa9/8AFUqWJtPSIiUmiOHTvG/PnzmTt3LrNnz+buu+9mxIgRNGzY0C371wSzbqBFzEuR6GjIXANtzBi49VZr6xERkUIVERHBY489xp49e1iyZAnnz5+nbdu2XHfddbz77rs4HPnLAgpmbqCzMksJh8MMZadOQZMm8NJLVlckIiIW6tixI++//z779+/H39+f0aNHExMTk699Kpi5QeZZmalqMSvZXnsNfvgB/PzMKTLysUitiIgUf+vWreO+++6jXr16JCQkMHPmTMqWLZuvfWqMmRtkBrM0uwO7w8DTQxOMljibNsHEieb1GTOgUSNLyxEREWtERUWxYMEC5s6dy/nz5xk8eDBr165123qZCmZukNmVCWZ3ZoCvDmuJEh9vTo2Rng633QajRlldkYiIWKR69epUqVKFoUOHcsstt+Dt7Y3D4WD79u1ZttNamRbKnC4DFMxKpAcegAMHoGpVePddLbkkIlKK2e12jhw5wjPPPMO0adMAsqybCWgeM6t5eNjw8fQgze7QQuYlzcKF5rJLHh7w8cfmvGUiIlJqFcm1Mo8ePeqcYA1gw4YNLFy4kEaNGjGqlHbz+HlfCGY6M7Pk+Ptvc4FygEmT4Prrra1HREQsV9BrZbp0VuagQYNYtWoVANHR0dx4441s2LCB//73vzz99NNuLbC40HqZJUx6OgwcaI4v69ABnnzS6opERKQUcCmY/fnnn7Rp0waATz/9lGuuuYZ169bx8ccfM2/ePHfWV2z4aZLZkmXKFNiwAUJCzC5MNy+5ISIikh2Xgll6ejq+vr4A/Pjjj9xyyy0ANGjQgKioKPdVV4xoktkSZOVKeP558/q770IBN1uLiIhkcimYNW7cmNmzZ/PLL7+wYsUKevbsCcCJEycoX768WwssLtSVWUKcPg333AOGASNHQv/+VlckIiKliEvB7IUXXmDOnDl06dKFgQMH0qxZMwC+/vprZxdnaZM5ZYa6Mosxw4B774WoKGjQwJzpX0REJAcZGRn8+OOPzJkzh/j4eMBspEpISHB5ny4NnOnSpQtnzpwhLi6OcuXKOe8fNWoU/v7+LhdTnPmqK7P4e+st+OYb8PExl1wKCLC6IhERKaIOHz5Mz549OXLkCKmpqdx4440EBQXxwgsvkJqayuzZs13ar0stZsnJyaSmpjpD2eHDh5kxYwZ79+6lYsWKLhVS3JXJ7MrMUDArlrZtg0ceMa+/9BJcaAUWERHJzkMPPUSrVq04f/48ZS5ZO7lfv36sXLnS5f261GJ26623cttttzlXUW/bti3e3t6cOXOGV199lfvvv9/lgoornZVZjCUmmksupaZC797mTP8iIiJX8Msvv7Bu3Tp8fHyy3B8ZGcnx48dd3q9LLWabN2+mU6dOACxZsoTw8HAOHz7MggULeOONN1wupjjTWZnF2PjxsGcPVKoEc+dqySUREbkqh8OR7bJLx44dIygoyOX9uhTMkpKSnC/6ww8/cNttt+Hh4cF1113H4cOHXS6mONNZmcXUkiUX17/88EOoUMHqikREpBjo3r07M2bMcN622WwkJCQwZcoUbrrpJpf361Iwq1OnDkuXLuXo0aMsX76c7t27A3Dq1CmCg4NdLqY4UzArho4cMafEAHjsMeja1dp6RESk2HjllVdYu3YtjRo1IiUlhUGDBjm7MV944QWX9+vSGLPJkyczaNAgxo8fz//93//Rrl07wGw9a9GihcvFFGd+XpldmRpjVixkZMDgwRATA23aQCldSkxERFxTtWpVtm3bxqJFi9i+fTsJCQmMGDGCwYMHZzkZIK9cCmZ33HEHHTt2JCoqyjmHGUDXrl3p16+fy8UUZ75qMStepk2DX3+FoCBYuBC8va2uSEREihkvLy/uvvtu9+7T1SdGREQQERHBsWPHADM5ltbJZeHS6TLUYlbk/fILPPOMeX32bKhd29p6RESkWPj6669zvW3mcpV55VIwczgcTJs2jVdeecU5u21QUBD/+c9/+O9//4uHh0tD14o1jTErJs6dM7swHQ4YMgQGDbK6IhERKSb69u2bq+1sNlu2Z2zmhkvB7L///S/vv/8+zz//PB06dADg119/ZerUqaSkpPDss8+6VExxpukyioHM9S+PHoU6dcyZ/kVERHLJ4Sj4XjGXgtn8+fN57733sjTTNW3alCpVqvDvf/+7lAYztZgVee++C198YY4n++QTc3yZiIhIEeJSn+O5c+do0KDBZfc3aNCAc+fO5buo4uhii5nGmBVJO3fCQw+Z1597Dlq1srYeEREp9lauXEmfPn2oXbs2tWvXpk+fPvz444/52qdLwaxZs2a8lU030FtvvUXTpk3zVVBx5eelFrMiKyUFBg40/9u9O0yYYHVFIiJSzL399tv07NmToKAgHnroIR566CGCg4O56aabmDlzpsv7dakr88UXX6R37978+OOPzjnM1q9fz9GjR/nuu+9cLqY489Ui5kXXI4/Ajh1QsSLMnw+l8OQUERFxr+eee47XXnuNsWPHOu978MEH6dChA8899xxjxoxxab8u/UJ17tyZffv20a9fP2JiYoiJieG2225j586dfPjhhy4VUtyV0SLmRdPXX18c5D9vHkREWFqOiIiUDDExMfTs2fOy+7t3705sbKzL+3V5HrPKlStfNsh/27ZtvP/++7zzzjsuF1Rc6azMIuj4cbj3XvP6+PHQq5e19YiISIlxyy238OWXX/LII49kuf+rr76iT58+Lu9XfTpuorMyixi7He65B86ehRYtYPp0qysSEZF8mjlzJpGRkfj5+dG2bVs2bNiQ47Y7d+7k9ttvJzIyEpvNlmXBcVf3ealGjRrx7LPP0rt3b6ZNm8a0adPo06cPzz77LNdccw1vvPGG85IXLreYSVaZwSzdbmB3GHh62CyuqJR74QVYtQoCAmDRIvD1tboiERHJh8WLFzNhwgRmz55N27ZtmTFjBj169GDv3r1UrFjxsu2TkpKoVasW/fv3Z/z48W7Z56Xef/99ypUrx65du9i1a5fz/rJly/L+++87b9tsNh588MFcv08FMzfJ7MoEs9UswFeH1jK//QaTJ5vX33wT6tWzth4REcm3V199lZEjRzJ8+HAAZs+ezbfffssHH3zA448/ftn2rVu3pnXr1gDZPu7KPi918ODB/LydHOUpPdx2221XfDwmJiZPLz5r1ixmzZrFoUOHAGjcuDGTJ0+mVzEcC5Q5XQYomFkqNtacGsNuh7vugmHDrK5IRERyEB8fT1xcnPO2r68vvtn0cKSlpbFp0yYmTpzovM/Dw4Nu3bqxfv16l167IPbpDnlKDyEhIVd9fMiQIbneX9WqVXn++eepW7cuhmEwf/58br31VrZs2ULjxo3zUprlPDxs+Hh6kGZ3aCFzqxgGjB4Nhw5BZKS5QLlNXcoiIkVVo0aNstyeMmUKU6dOvWy7M2fOYLfbCQ8Pz3J/eHg4e/bscem187tPwzBYsmQJq1at4tSpU5ct1/TFF1+4VFeegtncuXNdepGc3HzzzVluP/vss8yaNYvffvut2AUzMLsz0+wOnQBglfnzzfFknp6wcCFc5Q8JERGx1q5du6hSpYrzdnatZUXVuHHjmDNnDjfccAPh4eHY3NQQUGT62+x2O5999hmJiYnOSWv/KTU1ldTUVOft+Pj4wiovV/y8PYlLyVAws8K+fZA5yd/TT0MOnyERESk6goKCCA4Ovup2YWFheHp6cvLkySz3nzx5kggX56fM7z4//PBDvvjiC2666SaXXj8nlk+XsWPHDgIDA/H19WX06NF8+eWXlzVtZpo+fTohISHOS07bWUVTZlgkNdUcT5aYCF26wGOPWV2RiIi4kY+PDy1btmTlypXO+xwOBytXrsyxMaeg9xkSEkKtWrVceu0rsTyY1a9fn61bt/L7779z//33M3To0CynnV5q4sSJxMbGOi85bWcVLWRukSeegC1bIDQUPvrI7MoUEZESZcKECbz77rvMnz+f3bt3c//995OYmOg8o3LIkCFZBvKnpaWxdetWtm7dSlpaGsePH2fr1q0cOHAg1/u8kqlTp/LUU0+RnJzs1vdpeVemj48PderUAaBly5b88ccfvP7668yZM+eybf95tsalZ3IUBWoxs8CyZfDqq+b1Dz6AS8YqiIhIyTFgwABOnz7N5MmTiY6Opnnz5ixbtsw5eP/IkSN4XLIW8okTJ2jRooXz9ssvv8zLL79M586dWb16da72eSV33nknn3zyCRUrViQyMhJvb+8sj2/evNml92l5MPsnh8ORZRxZcZK5XmZSmoJZoTh5EoYONa+PGQO33mptPSIiUqDGjh2bZdHwS2WGrUyRkZEYhpGvfV7J0KFD2bRpE3fffXfJGfw/ceJEevXqRfXq1YmPj2fhwoWsXr2a5cuXW1mWywIvzF2WmJphcSWlgMNhhrJTp6BJE3jpJasrEhGRUuTbb79l+fLldOzY0a37tTSYnTp1iiFDhhAVFUVISAhNmzZl+fLl3HjjjVaW5bLMSWUTFMwK3muvwfLl4OdnTpFRpozVFYmISClSrVq1XJ1RmleWBrNL15IqCQKcLWbqyixQmzZB5gDPGTOgiJ2dKyIiJd8rr7zCo48+yuzZs4mMjHTbfovcGLPiLNDXHGOWkJpucSUlWHy8OTVGejrcdhuMGmV1RSIiUgrdfffdJCUlUbt2bfz9/S8b/H/u3DmX9qtg5kYXuzLVYlZgHngADhyAqlXh3Xe15JKIiFhixowZBbJfBTM30uD/ArZwobnskocHfPyxOW+ZiIiIBYZmzgrgZgpmbqRgVoD+/ttcoBxg0iS4/npr6xEREbkgJSWFtLS0LPe5emKA5TP/lyQ6K7OApKfDwIHm+LIOHeDJJ62uSERESrnExETGjh1LxYoVCQgIoFy5clkurlIwcyNni1magplbTZkCGzZASIjZhemlhl4REbHWo48+yk8//cSsWbPw9fXlvffe46mnnqJy5cosWLDA5f3qF86NNF1GAfjpJ3j+efP6u+9CjRrW1iMiIgL873//Y8GCBXTp0oXhw4fTqVMn6tSpQ40aNfj4448ZPHiwS/tVi5kbBTiny1CLmVucOQN33w2GASNHQv/+VlckIiICmNNh1KpVCzDHk2VOj9GxY0fWrFnj8n4VzNxIg//dyDBg+HCIioIGDcyZ/kVERIqIWrVqcfDgQQAaNGjAp59+CpgtaWXLlnV5vwpmbpTZlZmUZsfhuPrCqXIFb70F33wDPj7mkksBAVZXJCIi4jR8+HC2bdsGwOOPP87MmTPx8/Nj/PjxPPLIIy7vV2PM3CizxQzMEwCC/LyvsLXkaNs2yPxQv/QSNGtmbT0iIiL/MH78eOf1bt26sXv3bjZv3kydOnVo2rSpy/tVMHMjXy8PPD1s2B0Gial2BTNXJCWZSy6lpkLv3uZM/yIiIkVcZGSkW9bMVFemG9lsNgJ8dAJAvowfD3v2QKVKMHeullwSEZEiZf369XzzzTdZ7luwYAE1a9akYsWKjBo1itTUVJf3r2DmZoGaZNZ1S5bAO++YYezDD6FCBasrEhERyeLpp59m586dzts7duxgxIgRdOvWjccff5z//e9/TJ8+3eX9K5i5WYDOzHTNkSPmlBgAjz0GXbtaW4+IiEg2tm7dStdLfqMWLVpE27Zteffdd5kwYQJvvPGG8wxNVyiYuZmWZXJBRgYMHgwxMdCmDTz9tNUViYiIZOv8+fOEh4c7b//888/06tXLebt169YcPXrU5f0rmLmZ5jJzwbRp8OuvEBQECxeCt06aEBGRoik8PNw5f1laWhqbN2/muuuucz4eHx+Pdz5+xxTM3Cxz9n8Fs1z65Rd45hnz+uzZULu2tfWIiIhcwU033cTjjz/OL7/8wsSJE/H396dTp07Ox7dv307tfPyWaboMNwv0NVNygtbLvLpz58wuTIcDhgyBQYOsrkhEROSKnnnmGW677TY6d+5MYGAg8+fPx8fHx/n4Bx98QPfu3V3ev4KZmwWqxSx3DAPuvReOHoU6dcyZ/kVERIq4sLAw1qxZQ2xsLIGBgXh6emZ5/LPPPiMwMNDl/SuYuZkG/+fSzJnw1VfmeLLFi83xZSIiIsVESEhItveHhobma78aY+Zmmi4jF7Zuhf/8x7z+0ktw7bWWliMiIlJUKJi5mfOszDQFs2wlJMCAAZCWBn36wIMPWl2RiIhIkaFg5mYXuzI1+D9bY8fCvn1QpYqWXBIREfkHBTM30+D/K/jwQ5g/Hzw84OOPISzM6opERESKFAUzN3O2mKUomGWxbx/cf795ffJk6NzZ2npERESKIAUzNwv2M+cxi09Jt7iSIiQ1Fe66CxITzUA2aZLVFYmIiBRJCmZuFlzGDGZxajG76LHHYMsWKF8ePvoI/jHni4iIiJgUzNwsyO/iPGZ2h2FxNUXA11/D66+b1+fNg6pVLS1HRESkKFMwc7PMYAYaZ8axYzB8uHl93DhzegwRERHJkYKZm/l6eeLnbR7WuNI8ziwjw1z78tw5aNkSnn/e6opERESKPAWzApB5AkBscikOZtOmwS+/QGAgLFoEvr5WVyQiIlLkKZgVgMzuzPjS2pW5ejU884x5fc4cc5FyERERuSoFswJw8czMUthiduYMDB4MDoc5vmzQIKsrEhERKTYUzApAZldmXGnryjQMGDYMTpyA+vXhzTetrkhERKRYUTArAKV2LrPXX4dvvzXHky1eDAEBVlckIiJSrCiYFYCLY8xKUYvZpk3w6KPm9VdfhWbNrK1HRESkGFIwKwAXuzJLSYtZXBwMGADp6dCv38U1MUVERCRPFMwKQHAZs8WsVAz+NwwziP31F1SvDu+/Dzab1VWJiIgUSwpmBaBULWQ+fz4sXGiuf7lwIZQrZ3VFIiIixZaCWQFwDv4v6V2Ze/bAmDHm9aefhg4drK1HRESkmFMwKwCZg/9LdFdmSoo5riwpCbp2hcces7oiERGRYk/BrAA4B/+X5GD28MOwfTtUqAAffmh2ZYqIiEi+KJgVgJAyJXxJpi+/hJkzzesLFkClStbWIyIiUkIomBWAS2f+NwzD4mrc7PBhuPde8/ojj0DPntbWIyIiUoIomBWAoAvBzGFAYprd4mrcKCPDXPsyJgbatIFp06yuSEREpERRMCsAft4eeHuac3mVqPUyp06FdesgOBg++QR8fKyuSEREpERRMCsANpvtkrnMSsg4s5Ur4bnnzOvvvAO1allbj4iISAmkYFZALi5kXgJazE6dgrvvNmf5HznSnCZDRERE3E7BrIA45zIr7l2ZDgcMHQrR0dCoEcyYYXVFIiIiJZaCWQEpMV2Zr74Ky5aBnx8sXgz+/lZXJCIiUmIpmBWQErGQ+YYNMHGief311+Gaa6ytR0REpISzNJhNnz6d1q1bExQURMWKFenbty979+61siS3uXQus2IpNhbuusucIqN/f3NsmYiIiBQoS4PZzz//zJgxY/jtt99YsWIF6enpdO/encTERCvLcouL62UWw65Mw4BRo+DgQYiMNM/CtNmsrkpERKTE87LyxZctW5bl9rx586hYsSKbNm3i+uuvv2z71NRUUlNTnbfj4+MLvEZXXRxjVgxbzN5/Hz79FLy8zPnKypa1uiIREZFSoUiNMYuNjQUgNDQ028enT59OSEiI89KoUaPCLC9PnNNlJBezFrOdO+HBB83rzz4L111nbT0iIiKlSJEJZg6Hg3HjxtGhQweuyWGQ+cSJE4mNjXVedu3aVchV5l6xHPyfnGzOUZacDN27w8MPW12RiIhIqWJpV+alxowZw59//smvv/6a4za+vr74+vo6b8fFxRVGaS4J8i2Gg//HjzdbzMLDYcEC8CgyuV1ERKRUKBLBbOzYsXzzzTesWbOGqlWrWl2OW2R2ZRabecw++wzmzDEH+X/0kRnOREREpFBZGswMw+CBBx7gyy+/ZPXq1dSsWdPKctyqWHVlHjx4cTqMxx+Hbt2srUdERKSUsjSYjRkzhoULF/LVV18RFBREdHQ0ACEhIZQpU8bK0vLt4jxmRbzFLD0dBg405y1r1w6eesrqikREREotSwcRzZo1i9jYWLp06UKlSpWcl8WLF1tZlltkzmOWZneQkm63uJorePJJ+P13c0qMhQvB29vqikREREotS4OZYRjZXoYNG2ZlWW4R4OOFx4U5WYtsd+YPP8ALL5jX33vPnExWRESkiJo5cyaRkZH4+fnRtm1bNmzYcMXtP/vsMxo0aICfnx9NmjThu+++y/L4sGHDsNlsWS49e/YsyLdwVTrtroB4eNgIKsrdmdHRcM895vX774fbb7e2HhERkStYvHgxEyZMYMqUKWzevJlmzZrRo0cPTp06le3269atY+DAgYwYMYItW7bQt29f+vbty59//pllu549exIVFeW8fPLJJ4XxdnKkYFaAiuwJAA6HGcpOnYImTeCVV6yuSERE5IpeffVVRo4cyfDhw2nUqBGzZ8/G39+fDz74INvtX3/9dXr27MkjjzxCw4YNeeaZZ7j22mt56623smzn6+tLRESE81KuXLnCeDs5UjArQBeXZSpiLWYvvgg//gj+/rB4MRTzEy1ERKR4io+PJy4uznm5dNnFS6WlpbFp0ya6XTJrgIeHB926dWP9+vXZPmf9+vVZtgfo0aPHZduvXr2aihUrUr9+fe6//37Onj2bz3eVPwpmBci5kHlRmmR2/XqYNMm8/uab0LChtfWIiEip1ahRoyxLLU6fPj3b7c6cOYPdbif8H3NshoeHO2d0+Kfo6Oirbt+zZ08WLFjAypUreeGFF/j555/p1asXdrt1J+0ViQlmSyrnlBlFpSvz/Hlzagy73fzv8OFWVyQiIqXYrl27qFKlivP2pav7FIa77rrLeb1JkyY0bdqU2rVrs3r1arp27VqotWRSi1kBKlILmRuGOYns4cNQqxbMnm3O8i8iImKRoKAggoODnZecgllYWBienp6cPHkyy/0nT54kIiIi2+dERETkaXuAWrVqERYWxoEDB/L4TtxHwawAXRxjVgRazObMgc8/N+cpW7wYgoOtrkhERCRXfHx8aNmyJStXrnTe53A4WLlyJe3atcv2Oe3atcuyPcCKFSty3B7g2LFjnD17lkqVKrmncBcomBUg5xgzq4PZ9u0wbpx5/fnnoVUrS8sRERHJqwkTJvDuu+8yf/58du/ezf33309iYiLDLwzLGTJkCBMnTnRu/9BDD7Fs2TJeeeUV9uzZw9SpU9m4cSNjx44FICEhgUceeYTffvuNQ4cOsXLlSm699Vbq1KlDjx49LHmPoDFmBapIdGUmJsJdd0FqKtx008WAJiIiUowMGDCA06dPM3nyZKKjo2nevDnLli1zDvA/cuQIHh4X25vat2/PwoULmTRpEk888QR169Zl6dKlXHPNNQB4enqyfft25s+fT0xMDJUrV6Z79+4888wzhT7W7VI2wzAMy149n44dO0a1atU4evQoVatWtbqcy3y28SiPLNnODfUrMHd4G2uKuO8+eP99qFQJtm2DChWsqUNEROSCov77bSV1ZRYgZ4uZVfOYffKJGcpsNvj4Y4UyERGRIk7BrABlDv6PtWIes7/+gn/9y7w+aRLccEPh1yAiIiJ5omBWgEIDfAA4l5hWuC+clmaOK4uPh44dYfLkwn19ERERcYmCWQHKDGbnk9KwOwpxKN8TT8DGjVCuHCxcCF46x0NERKQ4UDArQOX8vbHZzLldY5IKqdXsu+8uLko+dy5Uq1Y4rysiIiL5pmBWgLw8PSh74QSAs4XRnXniBAwdal5/4AG49daCf00RERFxGwWzApbZnXk2oYCDmd0Od98NZ85A8+bw4osF+3oiIiLidgpmBax8oDlJ3dnE1IJ9oeeeg1WrICAAFi0CP7+CfT0RERFxOwWzAla+MM7M/OUXmDrVvP7221C/fsG9loiIiBQYBbMCVuBdmWfPwqBB4HDAPffAkCEF8zoiIiJS4BTMCliBdmUaBtx7Lxw7BnXrwsyZ7n8NERERKTQKZgWsQLsy33oLvv4afHzMcWVBQe5/DRERESk0CmYFrHygGczOuLsrc8sWePhh8/pLL8G117p3/yIiIlLoFMwKWIEsyxQfDwMGmEsv3XyzOWeZiIiIFHsKZgWsfIA5xsytwWzsWNi/H6pUMWf3t9nct28RERGxjIJZAcvsynTbepkLFpgXDw9zHczy5fO/TxERESkSFMwKWDl/H+d6mefzu17mvn3w73+b16dMgeuvz3+BIiIiUmQomBUwTw/bxfUy83MCQGqqOa4sMRE6d4b//tdNFYqIiEhRoWBWCNwyl9mjj8LWrWbX5ccfg6ene4oTERGRIkPBrBDk+8zMr7+GN94wr8+fbw76FxERkRJHwawQlM/PskxHj8Lw4eb18eOhd283ViYiIiJFiYJZIcg8M/NsXlvMMjJg8GA4dw5atoTnny+A6kRERKSoUDArBKEX5jI7m5DHMWbPPAO//GIutbRokbn0koiIiJRYCmaFICzQhTFmq1aZwQxgzhyoU6cAKhMREZGiRMGsEGQO/s91V+bp03D33ebkZ/feCwMHFmB1IiIiUlQomBUCZzDLTVemYcCwYXDiBDRocPFsTBERESnxFMwKQVhgHtbLnDEDvvsOfH1h8WIICCjY4kRERKTIUDArBJktZueT0smwO3LecONGeOwx8/prr0HTpoVQnYiIiBQVCmaFIHO9TDDDWbbi4uCuuyA9HW67DUaPLrwCRUREpEhQMCsEnh42yvlf4cxMwzCD2F9/QfXq8N57OJOciIiIlBoKZoXkiicAzJsHn3xirn/5ySdQrlzhFiciIiJFgoJZISmf05QZu3fD2LHm9WeegfbtC7kyERERKSoUzApJ+ewmmU1ONseVJSVBt24XB/6LiIhIqaRgVkiy7cp8+GHYvh0qVoQPPwQP/e8QEREpzZQECkn5zPUyM1vMvvgC3n7bvL5gAUREWFSZiIiIFBUKZoUksyvzbEIaHD4MI0aYDzz6KPToYWFlIiIiUlQomBWSzK7M2Lgkc+3LmBho2xamTbO2MBERESkyFMwKSWZX5k1fzIH16yE42Jwaw9vb4spERESkqFAwKyR1wwPpcGgrg39aaN7x3ntQs6a1RYmIiEiRomBWSMISY3j9u1fxwODkXUOgf3+rSxIREZEiRsGsMDgcMHQoYfHn2BtWnW+GPWJ1RSIiIlIEWRrM1qxZw80330zlypWx2WwsXbrUynIKziuvwPLlpPv6MfaWx9h8Npv1MkVERKTUszSYJSYm0qxZM2bOnGllGQXr99/hiScAODTpOfZXqMH2YzHW1iQiIiJFkpeVL96rVy969eplZQkFKybGXHIpIwPuvJOKE8bA0ys4ei6Z84lplLswhYaIiIgIFLMxZqmpqcTFxTkv8fHxVpeUM8OAUaPg0CGIjIR33iHE34eaYQEAbFOrmYiIiPxDsQpm06dPJyQkxHlp1KiR1SXl7L334LPPwMsLFi2CkBAAmlcrC8Dmw+ctLE5ERESKomIVzCZOnEhsbKzzsmvXLqtLyt7OnfDgg+b1554zZ/i/oHVkKAAbDp2zojIREREpwiwdY5ZXvr6++Pr6Om/HxcVZWE0OkpJgwABISTHXwPzPf7I83KamGcy2HIkhNcOOr5enFVWKiIhIEVSsWsyKhfHjzRaziAhYsAA8sh7i2hUCKB/gQ2qGgx3HYi0qUkRERIoiS4NZQkICW7duZevWrQAcPHiQrVu3cuTIESvLct2nn8I774DNBh9+CBUrXraJzWZTd6aIiIhky9JgtnHjRlq0aEGLFi0AmDBhAi1atGDy5MlWluWagwdh5Ejz+sSJ0K1bjpu2vtCdueGggpmIiIhcZOkYsy5dumAYhpUluEd6ujlfWVwctG8PU6decfP2tcsDsO7AWU7FpVAx2K8QihQREZGiTmPM3GHSJNiwAcqWhYULwdv7ips3rBRMqxrlSLM7mL/+UKGUKCIiIkWfgll+LV8OL75oXn//fahRI1dPu69TLQAWrDtM7zd+4en/7SoZrYciIiLiMgWz/IiKgnvuMa//+99w2225fuqNjcKJLO9PfGoGO0/E8cHag3y19YTz8bQMh7urFRERkSJOwcxVDocZyk6fhqZN4ZVX8vR0Tw8brw1ozj3X1eD2a6sC8ORXf3LkbBJTvvqT5k//wLI/owuichERESmiitUEs0XKCy/AypXg728uueSX9wH8LaqXo0X1cmTYHfx9JoEtR2Lo/cYvxKdmAPDY59tpWjWEsEBffLyUoUVEREo6/dq7Yt06ePJJ8/pbb0HDhvnanZenB7PvbkmtsABnKAsN8CE2OZ32z/9E4ynLWLHrZH6rFhERkSJOwSyvzp+HgQPBbodBg2DYMLfsNjzYj0WjrqNbw3DG3FCbz0a3I9jPbNBMtxs8smQbUbHJANgdBhsOnuOj3w6TcCHInUtM48FPtvDYku2cjk91S00iIkXNybgUvtp6nFV7TllditOx80ks/uMIDsfVT+A6ei6Jg2cS3X6y19ajMdw3/w92nSiCSxVKnqgrMy8MA+67D44cgdq1YdYsc5Z/N6kY7Md7Q1s5b6+f2JXE1AzuW7CR7cdi6T97Pc2rlWXdX2c5l5gGwMLfj3Bfp5q89uM+jp4zg9tX247j5eFBmt1BOX9vJvVuRIc6Yazac4r/a1CRcgE+bqtZRORKVu09RVqGgx6NI/L83F0n4khKy6DVhdVSlu+MZuzCzaTbzVDz44TrqVMxKNf7i01O571f/ubgmURubBROj8YR+Hnnb71iwzAYuWATu6Pi8Pb04LYLY4azc/RcEt1e/ZnUDAfVQ/3p16IKvZpEULtCIN6eHqRm2NlyJIY2kaF4eOT+t+VUfAojF2zkdHwqqRkOPhzRlnOJaYTm47v+vV/+5vDZJB7r1YBAX0WFwmQzivEcDceOHaNatWocPXqUqlVz/sfgNrNmmWdfenub3ZmtWl39OW5w8Ewit729lvNJ6c77gv288PCwEXPJfdVD/Qny82LnP/5i8vSwEVLGm3OJaYQF+vBsvyYufUkWRYZh8PW2EzSqFEzd8Nx/QUveZNgd/Pb3OeJT0ul5TQQ2N/5BYgXDMFiz/wxHziVROyyA9nXCrC7J7Y6cTWLUhxu5rlZ5pt7SONfP+3nfaRasO8SkPo2oGRaQrxqOnkui80urcBjwZJ9G3NmqKmW8PfHyvHJnTXRsCi8u28MXW44DMKl3Q/q3rEbXV1dzJiENb08b6XaDqTc3YliHmrmq5ZvtJ5i09M8s35nhwb6M61aPgW2qu/wef9pzknvnbQSgX4sqvDageY7bzlx1gJeW773s/iBfL6b1u4ZPNhzht7/P8Vy/Jgxqe/Wa7A6D/207wbu//J3le79/y6p8tukY/72pISOvr3XV/cQkpfHtjihubBhOxWA/1uw7zZAPNgDQrFpZFgxvQ4j/lefnzKtC//0uRhTMcmv7dmjTBlJT4dVXzcXKC1Fscjpr9p3mr9MJtIkMpXXNUE7EJDNm4WbSMwy6NqzIyE61CC7jzb6T8fh6eeDr7cmMFfv4bNMxAHw8zVY0mw0m92lEcrqdAB8vhrSrgc1mIzXDztFzyUSW9+dUfCrv/vI3f51OpHyADyM71aJR5WAMw+BsYhrlA3yKxI/z/7ad4IFPttCwUjDfP9TJpX0kp9nZcuQ8bWqGXvUHozjacSyWI+eS6N20kkvP3xsdz/C5GzgRmwLAvOGt6VL/8nVgc3LwTCJ+3h5UCinj0utn53R8Ki8v38uvB87wxsDmtKwRetXnRMUmM2/tIfq3qsr8dYf58LfDzseGtKvBpN6N8nSSTUxSGgmpGVQt54/dYZCQmkGwn9cV/10kpGYwb+1B2tYq71wzF8ygmPm8v04n8NZPB+h5TYTLf0DFJqVz26y1/HU6EQ8b/PHfbpQP9L3q83adiOP2WetITrfTvFpZPr+/PZ55aLn5p+e+2807a/7Ocl9keX+Wjbs+25aqDLuDd385yJs/7ScpzZ7lsbBAX84kpFK7QgB9mlbm9ZX76dk4gtn3tLxsP5nHc8PBc0xauoPzSenOIR51Kwbyfw0q8vW2E0Rd+Ex/MvI62tYMJTEtgyC/nAPIqr2nWLzhKL7eHiSm2rE7HBw5l8RfpxMBiAj2Y/3E/8vxM9Dr9V/YHRXHlJsbERrgw5JNx9hyJMY5JCXT9fUqsODeNpe9py82HycpLYOe11SiQpAvE7/YzicbjgIQ4ONJ3fAgth6NcT6njLcnPz3c+bJ/e5uPnGfFrpOkpjsID/blw98Oc+x8MjXK+/P+0FbcN38jh84mObdvVq0sX97fPk+teFejYJYzBbPcSEw0W8f27IHeveF//3NrF2ZBsjsM3vrpAJ4eMLR9JM9/v4ePf8+6SHzvJpXw9rSxcvcp4lMzaBARxJmENM4kZB2rdl2tUFLSHWw9GsODXesy4cZ6eapl/8l4pn+/h9aRofxfg4qs2nuKGxuFU7tCYI7PiUlKI8jPO8uPw/6T8Ty0aCvdGlbk94Pn+P3CmqObn7zRpab7e+f9wU97TtGlfgXeGnStpc32x2OSSctw5KulIt3u4M/jsXh7elDGx5Ob3/yVpDQ7S8d0oHm1srnaR3Kanf9tP4GnzcarK/ZxPCbZ+diIjjV5sk+jHJ+772Q8b686QJOqZalWrgyjP9qEw4CmVUPo0TiCsv7eOAy4q3U1vHMZhO0Ogw9+PUia3UGwnxcvLt9LfIr5Y3ZLs8q8MbDFFZ9vGAZ3vfMbvx88R5Cvl/Mkm451wvj1wBkAHvi/Ovyne/3LnvfIku1Ex6bwzpCW+PuYn43E1Ax6zFjDsfPJ3Nq8MhsOniMqNoVgPy8m9WnEna2qXVaDw2Ew6sON/Lj7FB42eKxnA/q2qMJjn29n0+Hz3Nq8MgG+Xny0/jCJaXbK+nvz28Su+Hl7cuhMItuOxdCnaWUS0zLYcSyWsEBf6lYMvOzH8sCpeMYu3MKe6HjnfdNva5KlVSgl3Y6Xh835h8iKXSd5eflejpxLIjn9YiC6rUUVyvr7EBrgTce6FbL9/KzYdZJZqw/g7elB82plublZZeavOwTAD7tOEpuczvX1KvDr/tNkDsF6f2grujYMz3Kc1/91lueX7WH7sVgArq1elik3N+bXA2d4+Ye9GIb5tbt4VDs8PWzcPmsd5fy92TTpRtLsDj7beJTaFQLZEx3Pyz/spW7FQPadTHC+H5sNxnSpw0Pd6jq7DR/8ZAvLd55kZKeanIhNYcXOkzzb7xpuv7YqMcnpnIxLYeaqA+w6EcfNzSozc9UBMrIZR+Zz4Tim2R2sergLNcMC+Pt0AmFBvgRfCHp/nU6g6ys/4+VhY+OkbpT1N7+r7A6DJ77YweKNR5378/XyYNuU7nh62Ji/7hBeHjbiUzJ4ZcU+wOwFualJJf637QQ2G/y7S20GtKrO0fNJDH7vd8AMZcnpdq6pEkytsEA616tA76aVOHAqgVtnrsV+lfFwFYN8mXX3tYxduIWnbmlMdzf3siiY5UzBLDdGjIAPPoDKlWHrVqhQoeBeq4AZhsEz3+zmg7UHaVgpmH0n47P8A7XZzKF0YC4dNaRdDdYeOMO3O6K49JPi5+3BmkdvoGLQ5dOEJKRmsGD9IU7GpnD4XBK7o+JoUa0cGw+f40xCWpZta1UI4Idx12fbUrVy90lGf7SJaqH+TO/XhLa1yhObnE7fmWs5eCbxsu3n3NMy1y0MCakZ7I2Ow9PDg74z1zrvb1erPJ+Mui5X+zgVl0JqhoOq5cpgs9mISUpj6ZbjxKVk0LJGOTrksnvseEwyGXYHial2bp+1jjS7g5mDrqVJ1RC8PWy5Xkt1b3Q8C9YfYumW4yReaG24NISM61aXcd1yDtNpGQ4+3XiUw2cT+W5HdJYwVjMsgOEdIpn81U6uqRLMNw9c3jr5w85ovtxynB92nXR+pjxskNP3/1O3NGZo+8jL7t985DyfbTzGnug4nripIddWL8ejS7bz+eZjWbarFlqGo+eSCQ3wYeN/uzkDSmqGnZ0n4mhetazzvuU7o/nXh5uyPP/eDjWZfHMjPtlwhIlf7KBSiB9rH/u/LEHn0i6dCTfWo3HlYE7EpnD0XNJlLUGZfDw9+ObBjtS70LV++GwiE7/YwdmENPaejM/yb+xqXr2zGTc3q8yNr/7MobNJDO8QyboDZ9l70gxdneqGMbFXQ0Yu2Eh4sC8ta5RjwfrDpGY4CAv0oVPdCny55Tid6obx4Yi2HDyTyKsr9rF8ZzS1wgKYN7wNQX5edH5ptfMPsfrhQdzSvPJlXW5lvD1Z8+gNVAjydR7nr7aeYOIXO674I1891J9VD3chISWD55ft4ZMNR7izVVVevKMZYLZkPvTJVjYcMv/ACvbzYvLNjbn92irOlqeo2GR2R8VR1t+Ha6uXI93uoOnUH0hOt7N83PXMX3+Ihf/4gzNTp7phPNqjAeUDfahcNmvLUWaLe+UQP6LjUpyfVX8fz8ta7DJ1axhO25qhlPHxJDXDwYaDZ7mhfkW+2HKcDQfP8Wy/a6hctgwj5v1BWKAv7wxpxcvL97I7Ko6ziWl0qV+BecOztoY5HAafbjxKeIgfT3yxg6jYFF68oymfbTzKH4fOZ9m2TsVADpxKcN4e1j7S2VVtGAaTlv5JWoaDwdfVoN/ba7N81ioE+VLG25Mj55JoWaMcrWqU4/DZJCJC/LijZVXunfcHp+JTqVsxkGf7NaFNzVBS0u35HoeXHQWznCmYXc3ChTB4sJlYfvoJunQpmNcpZLFJ6QSX8eLnfad5e9VfNKsWQs9rKlGjvD9v/XQAu8Pg8V4NCLjQenQ8Jplvt5/AYcD3O6LYdiyW+zrWZNKFlpNNh8/x2cZjdG0Yzoe/HWbNvtPZvm7dioHEJKdzJiEVH08PUjMczr/m0zLM+dy8PGwkpzm454Pfs4wHua1FFXZHx7M7Kg4vD9tlf7leWk92TsQk89fpBFLSHUz9eifHY5KdX8Dta5dn4+HzpGU4+HpsB5pWLXvF4/fHoXMMeX8Dyel2IoL9+PcNtfnot8PsO2l+Yfp4evD7E12veKJFSrqd11bs4/1fD2I3DIJ8vYi70AqU+eMd6OvFsnGdqFrOP8tzHQ6D3w+eI6SMN4fPJjJ//SF++/uc8/Gy/t4kpGRkOUYta5Tj8/vbZ1uLYRj857NtfLH5uPO+yiF+VAz2I93u4K1B1+Lv40nb51biYYOtU7pTxtvT2eL16R9HefTz7c7ntq0Zyh+HzuEwzLD72oDmrNp7ip/2nOJETDI7T8TRqkY5ltzf3nl22pr9Z5j50wHnDzSYYaBOxUB2HI/F08NGm8hQTsQmc1fr6gzvEEnLZ1aQmGbnmwc6ck2VEE7GpTBqwUa2HYvljpZV+XeX2sxc9Rc/7jZbboa1jyQqNhm7A94a1AI/b09S0u20nvYj8akZfPqvdrSpGYphGKTbDQa++xubDps/jJnjmi419oY67IqKo12t8tzZqhrjP93KT3tO0axqCIv/1Q7DgH5vr83SevXagGYkptqZtfovjsckUy20DBNurMdvf53Dx8uDJlVCOBGbzIwf99OyRjnubFWVxz7fkeV1/X08ybAbpNkd+HqZ/44u1aluGK/c2YyElAz+75Wf8fSw8cX97Rm5YCOnLjlru0rZMjSvXpZvt0dRo7w/s+9uabbC2Wy89uM+omJTKB/gw4pdJ/n7TCK3Nq/M4bNJ7ImOw+EwW4gAbm1emevrVmDeukPsOB5Lm8hQAv28WL33FK/c2Yx+Lczv53UHzjDovd8JDfBh5qBr+WFXNF9vPcHZxDT8vD3o37IaY26oQ0TI1f8Yuef93/ll/xm61K/A6r3m902QrxdpdgeP9WyAwzCIS8ng311q5xgsziWm0XLaCmd4Cfa7+G8QzH/HneqGUS8iiPd/OUiryHJ8MKx1tvt7bcU+Xl+5n9aR5Th4JvGyP0IzvTWoBX2aVs7xfV3aPZn5nkIDfTh8NolR19fiiZsa8uOuk0z+6k9CA3349F/tnC25/7Ri10l2nYjD7nCwZNMx53CE0AAffhh/PWH/6N6OTU4nITWDKmXdN+wgJwpmOVMwu5IDB+DaayE+HiZPhqeecv9rFEOr955i2Nw/AKgXHojDIMtfcGC2qI3oWPNCd0sQ3+44QVxyBtP6XkMZH0/iUtL5ZlsUT3+zi4pBvkzq08jZlXKpa6oE07Rq2Sx/DQf5ejHv3tZM/monO0/EOQe6Nqsa4gwemw6f5+El26hdIZB+Laowa/VfWX4cL2WzwYrxnXnrp/0s3XqCu1pX4/nbmwJZx/44HAZLtx5nb3Q8CzcccXalXapikC+eHjaiYlMu6z5KSM0gPcPhDGsPLdqSZRkugBrl/bmmcgjf7ohy3ndr88q8flfWrrpnvtnF+78ezHKfp4eN7o3CuaddDa6rWZ6/zyTw8e9HaB0Zyr8/3oynh40tk28k2M8bh8Pg5R/28sehc7SKDOXIuSS+3R6Fp4eNe66rQY3y/gxoXe2yL/wuL63i0NkkGkQEER2XwntDWlE+0Jfeb/xCUpqd/i2rMrR9JNdUCWHtgTP8sv8MozvXcnbbgDmwu93zKzEMGNy2Op9uPIq/jxexyWYI9/a0cXPTypyKT3V2M/p5e/BK/+aXjZO7b/4f/Lj7FBNurEdIGW/eWLmfs4kXfxAvDS21KwTw1diO2XZVP/zZNpZsOsadrapyXa3yWc5y9vXyoGZYgPPzU9bfm5ikdDrWCePDEW2yjCeKjk3hxtd+Jj4lg9aR5fDy8GD932cJC/ThqVuuoXbFABpEBANmF9aO47HUqRh4WU2n4lJo//xPZDgMAn29SEjNoFZYAH9fGK/36b/acfx8Mvd/vBkwA1b3xuFsv/AH06UnaPScsSbLZ79eeCATb2rIM//bxd+XtDzPHHRtjuMQf91/hrvf//2y+ysE+dK/ZVX+070+nh42HA6D4zHJzlZkh8PI0gKZYXfQ6tkfs/zBBWbr/Jy7W1K9vP8/XyJH/xxIP6JjTR7tWZ/UDIez+zA3bnnrV2f36TN9r6FTnTAyHA6qhwZkGXOYnGbH18sjx3FWGw+d447Z652361QM5GRcCvEpGUSW9+fZfk0I9PWiadWQK45DXPZnFKM/Mv+/Vg/1Z/69bahWrgxRsSlUC714fAzDwDDI9bivtAwHC38/zDfbo3ioW1061bW258fV3++ZM2fy0ksvER0dTbNmzXjzzTdp06ZNjtt/9tlnPPnkkxw6dIi6devywgsvcNNNNzkfNwyDKVOm8O677xITE0OHDh2YNWsWdevWzdf7yw8Fs5ykpUH79rBpE3TqZLaWeemUYTA/yA9/lrVrycMGHeqEsfbAGRzG1f8qBLMrpPtrazh8ySBTfx9PfLw8SMtwUD7Qhw/vbUtkWAB/HDrHGyv30yAiiNGda1M+0JeE1AyOnE0ixN+bDs//5NxHkK8XKRn2y1o3PGxml1xKuoN2tctz93U1eGfNXzSrWpZ/da7NhoPnuHPOevx9PFk/sStvrtzPJxuOcEfLqrSpWZ4Pf8vaKtU6shzvDmnFJxuO8uqKvQT4erF4VDt+2nOKF5btoX3t8iwcaXaLHjgVz13v/MaZhLQsrRSeHjZmDrqWKmXL8P2fUdzVujpVy5Vhy9HzpKY7GPz+7xgGvNy/GTc1icDfx4vPNh7lkSVm61Swnxe+3p4MaFWNQW2rX9ZVk+n/Xl7N32cSeXvwtdSqEMBbPx3gm+1Rl22XU/dipseWbM8yFqZK2TL4eHlw8Ewi19UK5eP7rsvVYPE7Z6/P0jIGZuvYoLbVua9TTSqFlCEl3c6rK/bh6+XB0PaRl/11DzBv7UGm/m9XlvvqhQfSqW4FZ3BtHVmOh7rWo1VkuRxbTn7ed5qhF7os/2l059r0a1GFp/63k1uaVeaW5pVZs+8MHeuGZRvy1v91llELNjq7kL08bMy/t02uu7YzTf9uN3MudJdWDPLlx/905svNx2lerSzNLoz1mrnK/P/40h1NuaZKSLb72XT4HA9/tp2DZxKJCPbji3+3p3LZMsQmpfPB2oMs/uMoTaqG8M49LXMMDIZhMOCd39hw8ByR5f2ZOfhagv28nQEsLzJDMJit4F0aVKR7o/A8d5edT0xj+ve7iY5LJTzIl2f6XuNSl9tLy/cwc9VfeHnY+OO/3VyeTsgwDBZuOMKqPac4m5jGS3c0IyE1g+92RHFfx5q5HpKQkJpBj9fWEHjhD1B3njRTlLjy+7148WKGDBnC7Nmzadu2LTNmzOCzzz5j7969VKx4+QlJ69at4/rrr2f69On06dOHhQsX8sILL7B582auueYaAF544QWmT5/O/PnzqVmzJk8++SQ7duxg165d+Lmwoo87KJjl5D//Mc++DA01x5VVu3wwb2l3Oj6V7cdi8PP2pGZYAJXLluHAqQTiUtK5tnq5XO3jeEwyr/+4jy82H6dzvQq8cmezLK0rudV++kpnM32mHo3DOZ+UzubD5xnWPpKx/1fnivs2DIMbX1vDgVMJBPl5ZdsiVsbbkztbVaV6+QDubFXVeQbXybgUfDw9KBfgw9FzSXR6cRU2G3z2r3aciE3huW93Ex2Xctn+rjbu69IfMR8vD2pd0nIz5obaPNy9fq5+GCd/9ScL1h/Ocp+Xh41/d6lNdFwKoQG+dKhT/qp/RX+x+RgTPt2GzQah/j7O1qkqZcvw2eh2OQbDf5q/7hBTvt4JwD3X1eCuNtWoWtY/z6fkZw6oBvOsvYe61eWu1tXw8rDx9uq/cDgM/tW59lXPtky3O+j+2hoOnkmkStkyDGpbnTtaViUxNYPI8gF5Phtt54lYnv9+D3UqBjKoTXWXp3LZHRXHqr2n6FSnAk2qZh+8csMwDPaejCc8yM/l4HEqPoWlW47Tr0VV5zgzV/x9OoFXV+zj9pZVuSEPZ/cWlL3R8dz81q/c1qKKs6Xcahl2BzabLV9nxBZ1rvx+t23bltatW/PWW28B4HA4qFatGg888ACPP/74ZdsPGDCAxMREvvnmG+d91113Hc2bN2f27NkYhkHlypX5z3/+w8MPPwxAbGws4eHhzJs3j7vuussN7zTvFMyy8+230KePef2rr+CWW9y3b8lWWoYjX+uBfrj+EJ9tOsbozrWpVs6fM4mpdK5bAZsNUjMcuf5Lev1fZ3ngky2cSUjFwwYP/F9dNh0+T3xqBo0qBfGv62sTmYszJvu9vZYtR2Ky3FcvPJD3h7ZmT3Q8n248Stky3ky/rckVp+hITM3g9ZX7+f7PKGfXmrenjaHtIpl4U8Ncf3FvOnyO/rPX4zDMcNm6ZiijO9eife28teKkpNt5cumfdKwbRuWyZRj6wQZqhgXwwbDWhOeyRQDgbEIqN73xC7XCApl/b5t8/b//dONRElIyuKvN5V2veZGUlkFCSkauWzak5EhJt+Pt6VGig1BRk/n7vWvXLqpUqeK839fXF1/fy4N/Wloa/v7+LFmyhL59+zrvHzp0KDExMXz11VeXPad69epMmDCBcePGOe+bMmUKS5cuZdu2bfz999/Url2bLVu20Lx5c+c2nTt3pnnz5rz++utuea95pb657KSnQ3CwudySQlmhyO8i7fe0i+SedpHZPpaX7o12tcvzy6M38PW241QPDaBd7fIu1TO6c20evzAYvnLZMrSvXZ5/da5NWKAv1UL9ubFR+FX2YArw9eKJmxoysVcD9p1MYPuxGNrWLJ+nsTgALWuEsnVKd+x2g+Ay3i7/APl5e/JS/2bO2xv+240AH888d2eVD/Rl/eNdgdyPkclJdlNTuMLfxytfwU6Kr4I461Byp1GjrCdsTZkyhalTp1623ZkzZ7Db7YSHZ/3uDA8PZ8+ePdnuOzo6Otvto6OjnY9n3pfTNlbQt1B2+vaFbdugkmsTckrxVsbHkwGtXZ8JHKBHY9cnB82OzWajfkQQ9SNcX90gLwOicys/c765c7JKESmesmsxK+1K3jTn7hIZCfqAiIiIFJigoCCCg4Odl5yCWVhYGJ6enpw8eTLL/SdPniQiIvs/giMiIq64feZ/87LPwqBgJiIiIkWaj48PLVu2ZOXKlc77HA4HK1eupF27dtk+p127dlm2B1ixYoVz+5o1axIREZFlm7i4OH7//fcc91kY1JUpIiIiRd6ECRMYOnQorVq1ok2bNsyYMYPExESGDx8OwJAhQ6hSpQrTp08H4KGHHqJz58688sor9O7dm0WLFrFx40beeecdwBwiMm7cOKZNm0bdunWd02VUrlw5ywkGhU3BTERERIq8AQMGcPr0aSZPnkx0dDTNmzdn2bJlzsH7R44cwcPjYkdg+/btWbhwIZMmTeKJJ56gbt26LF261DmHGcCjjz5KYmIio0aNIiYmho4dO7Js2TLL5jADTZchIiIihUy/3znTGDMRERGRIkLBTERERKSIUDATERERKSIUzERERESKCAUzERERkSJCwUxERESkiFAwExERESkiFMxEREREiggFMxEREZEiolgvyeRwOACIioqyuBIRERHJrczf7czfcbmoWAezkydPAtCmTRuLKxEREZG8OnnyJNWrV7e6jCKlWK+VmZGRwZYtWwgPD8+ycGl+xMfH06hRI3bt2kVQUJBb9lkc6TjoGGTScdAxAB2DTDoO7jkGDoeDkydP0qJFC7y8inUbkdsV62BWEOLi4ggJCSE2Npbg4GCry7GMjoOOQSYdBx0D0DHIpOOgY1DQNPhfREREpIhQMBMREREpIhTM/sHX15cpU6bg6+trdSmW0nHQMcik46BjADoGmXQcdAwKmsaYiYiIiBQRajETERERKSIUzERERESKCAUzERERkSJCwUxERESkiCiVwWzmzJlERkbi5+dH27Zt2bBhwxW3/+yzz2jQoAF+fn40adKE7777rpAqLVh5OQ7z5s3DZrNlufj5+RVite63Zs0abr75ZipXrozNZmPp0qVXfc7q1au59tpr8fX1pU6dOsybN6/A6yxIeT0Gq1evvuxzYLPZiI6OLpyCC8D06dNp3bo1QUFBVKxYkb59+7J3796rPq+kfS+4chxK2vfCrFmzaNq0KcHBwQQHB9OuXTu+//77Kz6npH0O8noMStpnoCgodcFs8eLFTJgwgSlTprB582aaNWtGjx49OHXqVLbbr1u3joEDBzJixAi2bNlC37596du3L3/++WchV+5eeT0OAMHBwURFRTkvhw8fLsSK3S8xMZFmzZoxc+bMXG1/8OBBevfuzQ033MDWrVsZN24c9913H8uXLy/gSgtOXo9Bpr1792b5LFSsWLGAKix4P//8M2PGjOG3335jxYoVpKen0717dxITE3N8Tkn8XnDlOEDJ+l6oWrUqzz//PJs2bWLjxo383//9H7feeis7d+7MdvuS+DnI6zGAkvUZKBKMUqZNmzbGmDFjnLftdrtRuXJlY/r06dluf+eddxq9e/fOcl/btm2Nf/3rXwVaZ0HL63GYO3euERISUkjVFT7A+PLLL6+4zaOPPmo0btw4y30DBgwwevToUYCVFZ7cHINVq1YZgHH+/PlCqckKp06dMgDj559/znGbkvq9cKncHIeS/r1gGIZRrlw547333sv2sdLwOTCMKx+D0vAZKGylqsUsLS2NTZs20a1bN+d9Hh4edOvWjfXr12f7nPXr12fZHqBHjx45bl8cuHIcABISEqhRowbVqlW76l9QJVFJ/Cy4qnnz5lSqVIkbb7yRtWvXWl2OW8XGxgIQGhqa4zal4bOQm+MAJfd7wW63s2jRIhITE2nXrl2225T0z0FujgGU3M+AVUpVMDtz5gx2u53w8PAs94eHh+c4RiY6OjpP2xcHrhyH+vXr88EHH/DVV1/x0Ucf4XA4aN++PceOHSuMkouEnD4LcXFxJCcnW1RV4apUqRKzZ8/m888/5/PPP6datWp06dKFzZs3W12aWzgcDsaNG0eHDh245pprctyuJH4vXCq3x6Ekfi/s2LGDwMBAfH19GT16NF9++SWNGjXKdtuS+jnIyzEoiZ8Bq3lZXYAUD+3atcvyF1P79u1p2LAhc+bM4ZlnnrGwMilM9evXp379+s7b7du356+//uK1117jww8/tLAy9xgzZgx//vknv/76q9WlWCq3x6Ekfi/Ur1+frVu3Ehsby5IlSxg6dCg///xzjsGkJMrLMSiJnwGrlapgFhYWhqenJydPnsxy/8mTJ4mIiMj2OREREXnavjhw5Tj8k7e3Ny1atODAgQMFUWKRlNNnITg4mDJlylhUlfXatGlTIoLM2LFj+eabb1izZg1Vq1a94rYl8XshU16Owz+VhO8FHx8f6tSpA0DLli35448/eP3115kzZ85l25bUz0FejsE/lYTPgNVKVVemj48PLVu2ZOXKlc77HA4HK1euzLH/vF27dlm2B1ixYsUV+9uLOleOwz/Z7XZ27NhBpUqVCqrMIqckfhbcYevWrcX6c2AYBmPHjuXLL7/kp59+ombNmld9Tkn8LLhyHP6pJH4vOBwOUlNTs32sJH4OsnOlY/BPJfEzUOisPvugsC1atMjw9fU15s2bZ+zatcsYNWqUUbZsWSM6OtowDMO45557jMcff9y5/dq1aw0vLy/j5ZdfNnbv3m1MmTLF8Pb2Nnbs2GHVW3CLvB6Hp556yli+fLnx119/GZs2bTLuuusuw8/Pz9i5c6dVbyHf4uPjjS1bthhbtmwxAOPVV181tmzZYhw+fNgwDMN4/PHHjXvuuce5/d9//234+/sbjzzyiLF7925j5syZhqenp7Fs2TKr3kK+5fUYvPbaa8bSpUuN/fv3Gzt27DAeeughw8PDw/jxxx+tegv5dv/99xshISHG6tWrjaioKOclKSnJuU1p+F5w5TiUtO+Fxx9/3Pj555+NgwcPGtu3bzcef/xxw2azGT/88INhGKXjc5DXY1DSPgNFQakLZoZhGG+++aZRvXp1w8fHx2jTpo3x22+/OR/r3LmzMXTo0Czbf/rpp0a9evUMHx8fo3Hjxsa3335byBUXjLwch3Hjxjm3DQ8PN2666SZj8+bNFlTtPplTP/zzkvm+hw4danTu3Pmy5zRv3tzw8fExatWqZcydO7fQ63anvB6DF154wahdu7bh5+dnhIaGGl26dDF++ukna4p3k+zeP5Dl/21p+F5w5TiUtO+Fe++916hRo4bh4+NjVKhQwejataszkBhG6fgc5PUYlLTPQFFgMwzDKLz2ORERERHJSakaYyYiIiJSlCmYiYiIiBQRCmYiIiIiRYSCmYiIiEgRoWAmIiIiUkQomImIiIgUEQpmIiIiIkWEgpmIiIhcZs2aNdx8881UrlwZm83G0qVL8/T8qVOnYrPZLrsEBAQUTMElhIKZiOTJ6tWrsdlsxMTEXHG7yMhIZsyYUSg15cawYcPo27dvnp5T1N6DSGFKTEykWbNmzJw506XnP/zww0RFRWW5NGrUiP79+7u50pJFwUykBBo2bJjzr1MfHx/q1KnD008/TUZGRr733b59e6KioggJCQFg3rx5lC1b9rLt/vjjD0aNGpXv17uSvASn119/nXnz5hVoPSIlSa9evZg2bRr9+vXL9vHU1FQefvhhqlSpQkBAAG3btmX16tXOxwMDA4mIiHBeTp48ya5duxgxYkQhvYPiycvqAkSkYPTs2ZO5c+eSmprKd999x5gxY/D29mbixIn52q+Pjw8RERFX3a5ChQr5eh13sdvt2Gw2Z5AUEfcYO3Ysu3btYtGiRVSuXJkvv/ySnj17smPHDurWrXvZ9u+99x716tWjU6dOFlRbfKjFTKSE8vX1JSIigho1anD//ffTrVs3vv76awDOnz/PkCFDKFeuHP7+/vTq1Yv9+/c7n3v48GFuvvlmypUrR0BAAI0bN+a7774DsnZlrl69muHDhxMbG+tsoZs6dSqQtTVr0KBBDBgwIEt96enphIWFsWDBAgAcDgfTp0+nZs2alClThmbNmrFkyZIc31+XLl04fPgw48ePd742XGzB+/rrr2nUqBG+vr4cOXLksq7MLl26MHbsWMaOHUtISAhhYWE8+eSTXGn54JiYGO677z4qVKhAcHAw//d//8e2bducj2/bto0bbriBoKAggoODadmyJRs3brzK/ymR4ufIkSPMnTuXzz77jE6dOlG7dm0efvhhOnbsyNy5cy/bPiUlhY8//litZbmgFjORUqJMmTKcPXsWMLs69+/fz9dff01wcDCPPfYYN910E7t27cLb25sxY8aQlpbGmjVrCAgIYNeuXQQGBl62z/bt2zNjxgwmT57M3r17AbLdbvDgwfTv35+EhATn48uXLycpKcnZTTJ9+nQ++ugjZs+eTd26dVmzZg133303FSpUoHPnzpft84svvqBZs2aMGjWKkSNHZnksKSmJF154gffee4/y5ctTsWLFbI/J/PnzGTFiBBs2bGDjxo2MGjWK6tWrX7a/TP3796dMmTJ8//33hISEMGfOHLp27cq+ffsIDQ1l8ODBtGjRglmzZuHp6cnWrVvx9vbO6X+JSLG1Y8cO7HY79erVy3J/amoq5cuXv2z7L7/8kvj4eIYOHVpYJRZbCmYiJZxhGKxcuZLly5fzwAMPOAPZ2rVrad++PQAff/wx1apVY+nSpfTv358jR45w++2306RJEwBq1aqV7b59fHwICQnBZrNdsXuzR48eBAQE8OWXX3LPPfcAsHDhQm655RaCgoJITU3lueee48cff6Rdu3bO1/z111+ZM2dOtsEsNDQUT09PgoKCLnvt9PR03n77bZo1a3bFY1OtWjVee+01bDYb9evXZ8eOHbz22mvZBrNff/2VDRs2cOrUKXx9fQF4+eWXWbp0KUuWLGHUqFEcOXKERx55hAYNGgBk250jUhIkJCTg6enJpk2b8PT0zPJYdn+cvffee/Tp04fw8PDCKrHYUjATKaG++eYbAgMDSU9Px+FwMGjQIKZOncrKlSvx8vKibdu2zm3Lly9P/fr12b17NwAPPvgg999/Pz/88APdunXj9ttvp2nTpi7X4uXlxZ133snHH3/MPffcQ2JiIl999RWLFi0C4MCBAyQlJXHjjTdmeV5aWhotWrTI8+v5+Pjkqt7rrrvO2QUK0K5dO1555RXsdvtlPzbbtm0jISHhstaA5ORk/vrrLwAmTJjAfffdx4cffki3bt3o378/tWvXznP9IkVdixYtsNvtnDp16qpjxg4ePMiqVaucQynkyhTMREqoG264gVmzZuHj40PlypXx8sr9P/f77ruPHj168O233/LDDz8wffp0XnnlFR544AGX6xk8eDCdO3fm1KlTrFixgjJlytCzZ0/A/Osb4Ntvv6VKlSpZnpfZOpUXZcqUyRK43CEhIYFKlSplOessU+ZZqVOnTmXQoEF8++23fP/990yZMoVFixbleFabSFGWkJDAgQMHnLcPHjzI1q1bCQ0NpV69egwePJghQ4bwyiuv0KJFC06fPs3KlStp2rQpvXv3dj7vgw8+oFKlSvTq1cuKt1HsKJiJlFABAQHUqVPnsvsbNmxIRkYGv//+u7Mr8+zZs+zdu5dGjRo5t6tWrRqjR49m9OjRTJw4kXfffTfbYObj44Pdbr9qPe3bt6datWosXryY77//nv79+zvHX106SD+7bsuc5Pa1c/L7779nuf3bb79Rt27dy1rLAK699lqio6Px8vIiMjIyx33Wq1ePevXqMX78eAYOHMjcuXMVzKRY2rhxIzfccIPz9oQJEwAYOnQo8+bNY+7cuUybNo3//Oc/HD9+nLCwMK677jr69OnjfI7D4WDevHkMGzYs239XcjkFM5FSpm7dutx6662MHDmSOXPmEBQUxOOPP06VKlW49dZbARg3bhy9evWiXr16nD9/nlWrVtGwYcNs9xcZGUlCQgIrV66kWbNm+Pv74+/vn+22gwYNYvbs2ezbt49Vq1Y57w8KCuLhhx9m/PjxOBwOOnbsSGxsLGvXriU4ODjHAcORkZGsWbOGu+66C19fX8LCwvJ0LI4cOcKECRP417/+xebNm3nzzTd55ZVXst22W7dutGvXjr59+/Liiy9Sr149Tpw4wbfffku/fv1o3LgxjzzyCHfccQc1a9bk2LFj/PHHH9x+++15qkmkqOjSpcsVz1L29vbmqaee4qmnnspxGw8PD44ePVoQ5ZVYmi5DpBSaO3cuLVu2pE+fPrRr1w7DMPjuu++cLVh2u50xY8bQsGFDevbsSb169Xj77bez3Vf79u0ZPXo0AwYMoEKFCrz44os5vu7gwYPZtWsXVapUoUOHDlkee+aZZ3jyySeZPn2683W//fZbatasmeP+nn76aQ4dOkTt2rVdmjdtyJAhJCcn06ZNG8aMGcNDDz2U46S4NpuN7777juuvv57hw4dTr1497rrrLg4fPkx4eDienp6cPXuWIUOGUK9ePe6880569ep1xR8tEZF/shlXisMiIiVUly5daN68uZZcEpEiRS1mIiIiIkWEgpmIiIhIEaGuTBEREZEiQi1mIiIiIkWEgpmIiIhIEaFgJiIiIlJEKJiJiIiIFBEKZiIiIiJFhIKZiIiISBGhYCYiIiJSRCiYiYiIiBQR/w87NSpZKeJsFwAAAABJRU5ErkJggg==",
                  "text/plain": [
                     "<Figure size 640x480 with 2 Axes>"
                  ]
               },
               "metadata": {},
               "output_type": "display_data"
            }
         ],
         "source": [
            "# Train on IPU with interleaved validation on CPU\n",
            "\n",
            "n_epochs = 25\n",
            "val_ep_interval = 5\n",
            "\n",
            "cumulative_triples = 0\n",
            "training_loss = []\n",
            "val_mrr = []\n",
            "for ep in range(n_epochs):\n",
            "    ep_start_time = time.time()\n",
            "    ep_log = []\n",
            "    for batch in train_dl:\n",
            "        step_start_time = time.time()\n",
            "        triple_mask = batch.pop(\"triple_mask\")\n",
            "        cumulative_triples += triple_mask.numel()\n",
            "        res = poptorch_model(**{k: v.flatten(end_dim=1) for k, v in batch.items()})\n",
            "        ep_log.append(\n",
            "            dict(\n",
            "                loss=float(torch.sum(res[\"loss\"])) / triple_mask[-1].numel(),\n",
            "                step_time=(time.time() - step_start_time),\n",
            "            )\n",
            "        )\n",
            "    ep_loss = [v[\"loss\"] for v in ep_log]\n",
            "    training_loss.extend([v[\"loss\"] for v in ep_log])\n",
            "    print(\n",
            "        f\"Epoch {ep+1} loss: {np.mean(ep_loss):.6f} --- positive triples processed: {cumulative_triples:.2e}\"\n",
            "    )\n",
            "    print(\n",
            "        f\"Epoch duration (sec): {(time.time() - ep_start_time):.5f} (average step time: {np.mean([v['step_time'] for v in ep_log]):.5f})\"\n",
            "    )\n",
            "    if ep % val_ep_interval == 0:\n",
            "        ep_mrr = evaluate_mrr_cpu(val_triple_subset, evaluation)[\"mrr\"]\n",
            "        val_mrr.append(ep_mrr)\n",
            "        print(f\"Epoch {ep+1} sample MRR: {ep_mrr:.4f}\")\n",
            "ep_mrr = evaluate_mrr_cpu(val_triple_subset, evaluation)[\"mrr\"]\n",
            "val_mrr.append(ep_mrr)\n",
            "print(f\"Final sample MRR: {ep_mrr:.4f}\")\n",
            "\n",
            "# Plot loss and sample MRR as a function of the number of positive triples processed\n",
            "total_triples = np.cumsum(n_epochs * len(train_dl) * [triple_mask.numel()])\n",
            "ax0, ax1 = plt.gca(), plt.twinx()\n",
            "(line0,) = ax0.plot(total_triples, training_loss)\n",
            "(line1,) = ax1.plot(\n",
            "    np.concatenate(\n",
            "        [total_triples[:: val_ep_interval * len(train_dl)], total_triples[-1:]]\n",
            "    ),\n",
            "    val_mrr,\n",
            "    color=\"r\",\n",
            ")\n",
            "ax0.set_xlabel(\"Positive triples\")\n",
            "ax0.set_ylabel(\"Loss\")\n",
            "ax1.set_ylabel(\"Sample MRR\")\n",
            "plt.legend([line0, line1], [\"Loss\", \"MRR\"])\n",
            "\n",
            "poptorch_model.detachFromDevice()\n",
            "del train_dl"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "The fact that we are seeing lower metric values compared to the [KGE Training and Inference on OGBL-BioKG](1_biokg_training_inference.ipynb) case study is not surprising, as we are now making predictions choosing from all 123k+ entities in the knowledge graph, while for OGB-BioKG we only had to rank the ground truth against a small set of 500 candidate heads/tails."
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Top-k inference on IPU\n",
            "\n",
            "On the IPU we use the BESS `TopKQueryBessKGE` distribution scheme to predict, for each validation query, the top-k most likely tails among all the entities in the knowledge graph. Since there are no specific tail candidates to sample, our dataloader does not need to pass negative indices to the IPUs. We therefore use the `PlaceholderNegativeSampler` class.\n",
            "\n",
            "When using `TopKQueryBessKGE` we partition triples based just on the shard of the head entity (or the tail entity, if we wanted to predict heads), specifying `partition_mode='h_shard'` when constructing the `PartitionedTripleSet`.\n",
            "\n",
            "Since we want to inspect the predictions made by the model, we ask the dataloader to return the indices (with respect to the ordered list `validation_triples.triples`) of the triples in the batches, by setting the flag `return_triple_idx=True` when creating the `RigidShardedBatchSampler` batch sampler."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 16,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Number of triples per h_shard:\n",
                  "[1196 1273 1286 1245]\n"
               ]
            }
         ],
         "source": [
            "device_iterations = 1\n",
            "shard_bs = 1440\n",
            "\n",
            "validation_triples = PartitionedTripleSet.create_from_dataset(\n",
            "    yago, \"valid\", sharding, partition_mode=\"h_shard\"\n",
            ")\n",
            "candidate_sampler = PlaceholderNegativeSampler(corruption_scheme=\"t\", seed=seed)\n",
            "bs_valid = RigidShardedBatchSampler(\n",
            "    partitioned_triple_set=validation_triples,\n",
            "    negative_sampler=candidate_sampler,\n",
            "    shard_bs=shard_bs,\n",
            "    batches_per_step=device_iterations,\n",
            "    seed=seed,\n",
            "    duplicate_batch=False,\n",
            "    return_triple_idx=True,\n",
            ")\n",
            "\n",
            "print(\"Number of triples per h_shard:\")\n",
            "print(validation_triples.triple_counts)"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 17,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "head         torch.Size([1, 4, 1440])      \n",
                  "relation     torch.Size([1, 4, 1440])      \n",
                  "tail         torch.Size([1, 4, 1440])      \n",
                  "triple_mask  torch.Size([1, 4, 1440])      \n",
                  "triple_idx   torch.Size([1, 4, 1440])      \n"
               ]
            }
         ],
         "source": [
            "val_options = poptorch.Options()\n",
            "val_options.replication_factor = sharding.n_shard\n",
            "val_options.deviceIterations(bs_valid.batches_per_step)\n",
            "val_options.outputMode(poptorch.OutputMode.All)\n",
            "\n",
            "valid_dl = bs_valid.get_dataloader(\n",
            "    options=val_options, shuffle=False, num_workers=5, persistent_workers=True\n",
            ")\n",
            "\n",
            "# Example batch\n",
            "batch = next(iter(valid_dl))\n",
            "for k, v in batch.items():\n",
            "    print(f\"{k:<12} {str(v.shape):<30}\")"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "As explained above, `triple_idx` contains the indices of the triple in the batch.  \n",
            "\n",
            "Notice that, when constructing the partitioned triple set with `partition_mode='h_shard'`, the tensor `head` will contain the **local** entity IDs in the corresponding shard, while `tail` contains the **global** IDs of the ground truth tails:"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 18,
         "metadata": {},
         "outputs": [
            {
               "data": {
                  "text/plain": [
                     "True"
                  ]
               },
               "execution_count": 18,
               "metadata": {},
               "output_type": "execute_result"
            }
         ],
         "source": [
            "# Put original validation triple in the same order as validation_triples.triples\n",
            "triple_sorted = yago.triples[\"valid\"][validation_triples.triple_sort_idx]\n",
            "# Pass from global IDs to local IDs just for the heads\n",
            "triple_sorted[:, 0] = sharding.entity_to_idx[triple_sorted[:, 0]]\n",
            "# Compare with validation_triples.triples\n",
            "np.all(triple_sorted == validation_triples.triples)"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Let us now compile the `TopKQueryBessKGE` inference model. The parameter `window_size` represents the size of the **sliding window** used to score the queries in the batch against all candidate tails stored on the processing IPU. An **on-device loop** scores the queries in the batch against `window_size` entities in the knowledge graph at each iteration, and the results are used to update the list of the $k$ highest-scoring tails seen until that moment. As the `window_size` parameter determines the number of loop iterations, it should be tweaked together with the micro-batch size to optimize memory utilization."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 19,
         "metadata": {},
         "outputs": [
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "[10:12:08.722] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 477\n",
                  "[10:12:08.740] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 534\n",
                  "[10:12:08.743] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 557\n",
                  "Graph compilation: 100%|██████████| 100/100 [00:51<00:00]\n"
               ]
            }
         ],
         "source": [
            "evaluation = Evaluation(\n",
            "    [\"mrr\", \"hits@3\", \"hits@10\"], worst_rank_infty=True, reduction=\"sum\"\n",
            ")\n",
            "inf_model = TopKQueryBessKGE(\n",
            "    k=10,\n",
            "    candidate_sampler=candidate_sampler,\n",
            "    score_fn=complex_score_fn,\n",
            "    evaluation=evaluation,\n",
            "    window_size=500,\n",
            ")\n",
            "\n",
            "poptorch_inf_model = poptorch.inferenceModel(inf_model, options=val_options)\n",
            "\n",
            "poptorch_inf_model.entity_embedding.replicaGrouping(\n",
            "    poptorch.CommGroupType.NoGrouping,\n",
            "    0,\n",
            "    poptorch.VariableRetrievalMode.OnePerGroup,\n",
            ")\n",
            "\n",
            "_ = batch.pop(\"triple_idx\")\n",
            "res = poptorch_inf_model(**{k: v.flatten(end_dim=1) for k, v in batch.items()})"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Iterate over the validation set to compute the predictions and the corresponding metrics."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 20,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Validation time (sec): 0.02274\n",
                  "\n",
                  "hits@3 : 0.207600\n",
                  "hits@10 : 0.564800\n",
                  "mrr : 0.205530\n"
               ]
            }
         ],
         "source": [
            "val_log = []\n",
            "start_time = time.time()\n",
            "n_val_queries = 0\n",
            "for batch_val in valid_dl:\n",
            "    triple_idx = batch_val.pop(\"triple_idx\")\n",
            "    step_start_time = time.time()\n",
            "    # triple_mask is now passed to the model to filter out the metrics of padding triples\n",
            "    res = poptorch_inf_model(**{k: v.flatten(end_dim=1) for k, v in batch_val.items()})\n",
            "    n_val_queries += batch_val[\"triple_mask\"].sum()\n",
            "    val_log.append(\n",
            "        {\n",
            "            k: v.sum()\n",
            "            for k, v in zip(\n",
            "                evaluation.metrics.keys(),\n",
            "                res[\"metrics\"].T,\n",
            "            )\n",
            "        }\n",
            "    )\n",
            "\n",
            "print(f\"Validation time (sec): {(time.time() - start_time):.5f}\\n\")\n",
            "\n",
            "for metric in val_log[0].keys():\n",
            "    reduced_metric = sum([l[metric] for l in val_log]) / n_val_queries\n",
            "    print(\"%s : %f\" % (metric, reduced_metric))\n",
            "\n",
            "poptorch_inf_model.detachFromDevice()\n",
            "del valid_dl"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Let us check the results (and compare the validation time) against the CPU function defined earlier:"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 21,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "CPU Validation time (sec): 0.65354\n",
                  "\n",
                  "CPU validation MRR: 0.205530\n"
               ]
            }
         ],
         "source": [
            "start_time = time.time()\n",
            "cpu_res = evaluate_mrr_cpu(yago.triples[\"valid\"], evaluation)\n",
            "\n",
            "print(f\"CPU Validation time (sec): {(time.time() - start_time):.5f}\\n\")\n",
            "print(f\"CPU validation MRR: {cpu_res['mrr']:.6f}\")"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Now, we can have a look at some of the predictions made by the model:"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 36,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Example query: (Juiz_de_Fora, isLocatedIn, ?)\n",
                  "\n",
                  "Correct tail: Southeast_(Brazil)\n",
                  "\n",
                  "10 most likely predicted tails:\n",
                  "1) Southeast_Region,_Brazil\n",
                  "2) Minas_Gerais\n",
                  "3) Southeast_(Brazil)   <-----\n",
                  "4) Brazil\n",
                  "5) Northeast_(Brazil)\n",
                  "6) Italy\n",
                  "7) Poland\n",
                  "8) France\n",
                  "9) Province_of_Ravenna\n",
                  "10) Argentina\n",
                  "\n",
                  "\n",
                  "Example query: (Billy_Stark, playsFor, ?)\n",
                  "\n",
                  "Correct tail: St._Mirren_F.C.\n",
                  "\n",
                  "10 most likely predicted tails:\n",
                  "1) Kilmarnock_F.C.\n",
                  "2) Celtic_F.C.\n",
                  "3) Scotland_national_under-21_football_team\n",
                  "4) St._Mirren_F.C.   <-----\n",
                  "5) Scotland_national_football_team\n",
                  "6) Aberdeen_F.C.\n",
                  "7) Hamilton_Academical_F.C.\n",
                  "8) Stockport_County_F.C.\n",
                  "9) Preston_North_End_F.C.\n",
                  "10) Dundee_F.C.\n",
                  "\n",
                  "\n"
               ]
            }
         ],
         "source": [
            "def check_prediction(val_triple_id):\n",
            "    # Recover the non-padding triples seen in the last batch using triple_idx and triple_mask\n",
            "    triples = yago.triples[\"valid\"][validation_triples.triple_sort_idx][\n",
            "        triple_idx[batch_val[\"triple_mask\"]]\n",
            "    ]\n",
            "    h, r, t = triples[val_triple_id]\n",
            "    # res[\"topk_global_id\"] contains the top-10 tails predicted by the KGE model\n",
            "    top10_t = res[\"topk_global_id\"][batch_val[\"triple_mask\"].flatten()][val_triple_id]\n",
            "\n",
            "    print(f\"Example query: ({yago.entity_dict[h]}, {yago.relation_dict[r]}, ?)\\n\")\n",
            "    print(f\"Correct tail: {yago.entity_dict[t]}\\n\")\n",
            "    print(f\"10 most likely predicted tails:\")\n",
            "    for i, pt in enumerate(top10_t):\n",
            "        print(f\"{i+1}) {yago.entity_dict[pt]}\" + (\"   <-----\" if pt == t else \"\"))\n",
            "    print(\"\\n\")\n",
            "\n",
            "\n",
            "check_prediction(10)\n",
            "check_prediction(1000)"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Scaling analysis\n",
            "\n",
            "What speed-up did we achieve for top-k inference by using 4 IPUs, compared to running on a single chip? To measure that, we first change the number of shards of the model we trained as follows:"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 37,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Current embedding table (4 shards): torch.Size([4, 30796, 256])\n",
                  "Refactored embedding table (1 shard): torch.Size([1, 123182, 256])\n"
               ]
            }
         ],
         "source": [
            "# complex_score_fn.entity_embedding has shape [n_shard, max_entity_per_shard, embedding_size]\n",
            "\n",
            "print(\n",
            "    f\"Current embedding table ({sharding.n_shard} shards): {complex_score_fn.entity_embedding.shape}\"\n",
            ")\n",
            "\n",
            "# New entity sharding with a single shard - to use on 1 IPU\n",
            "new_val_sharding = Sharding.create(yago.n_entity, n_shard=1, seed=seed)\n",
            "\n",
            "# Update sharding of embedding tables, stored in the scoring function\n",
            "complex_score_fn.update_sharding(new_sharding=new_val_sharding)\n",
            "\n",
            "print(\n",
            "    f\"Refactored embedding table (1 shard): {complex_score_fn.entity_embedding.shape}\"\n",
            ")"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Then we re-instantiate the dataloader and PopTorch model exactly as before:"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 38,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Number of triples per h_shard:\n",
                  "[5000]\n",
                  "Example batch:\n",
                  "head         torch.Size([4, 1, 1440])      \n",
                  "relation     torch.Size([4, 1, 1440])      \n",
                  "tail         torch.Size([4, 1, 1440])      \n",
                  "triple_mask  torch.Size([4, 1, 1440])      \n",
                  "triple_idx   torch.Size([4, 1, 1440])      \n"
               ]
            }
         ],
         "source": [
            "# Multiply device_iterations by 4 and keep same shard batch size as before\n",
            "device_iterations = 4\n",
            "shard_bs = 1440\n",
            "\n",
            "validation_triples = PartitionedTripleSet.create_from_dataset(\n",
            "    yago, \"valid\", new_val_sharding, partition_mode=\"h_shard\"\n",
            ")\n",
            "bs_valid = RigidShardedBatchSampler(\n",
            "    partitioned_triple_set=validation_triples,\n",
            "    negative_sampler=candidate_sampler,\n",
            "    shard_bs=shard_bs,\n",
            "    batches_per_step=device_iterations,\n",
            "    seed=seed,\n",
            "    duplicate_batch=False,\n",
            "    return_triple_idx=True,\n",
            ")\n",
            "\n",
            "print(\"Number of triples per h_shard:\")\n",
            "print(validation_triples.triple_counts)\n",
            "\n",
            "val_options = poptorch.Options()\n",
            "val_options.replication_factor = new_val_sharding.n_shard\n",
            "val_options.deviceIterations(bs_valid.batches_per_step)\n",
            "val_options.outputMode(poptorch.OutputMode.All)\n",
            "\n",
            "valid_dl = bs_valid.get_dataloader(\n",
            "    options=val_options, shuffle=False, num_workers=5, persistent_workers=True\n",
            ")\n",
            "\n",
            "print(\"Example batch:\")\n",
            "batch = next(iter(valid_dl))\n",
            "for k, v in batch.items():\n",
            "    print(f\"{k:<12} {str(v.shape):<30}\")"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 39,
         "metadata": {},
         "outputs": [
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "[10:15:51.754] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 688\n",
                  "[10:15:51.772] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 745\n",
                  "[10:15:51.784] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 768\n",
                  "Graph compilation: 100%|██████████| 100/100 [01:31<00:00]\n"
               ]
            },
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Validation time (sec): 0.12073\n",
                  "\n",
                  "hits@3 : 0.207600\n",
                  "hits@10 : 0.564800\n",
                  "mrr : 0.205530\n"
               ]
            }
         ],
         "source": [
            "inf_model = TopKQueryBessKGE(\n",
            "    k=10,\n",
            "    candidate_sampler=candidate_sampler,\n",
            "    score_fn=complex_score_fn,\n",
            "    evaluation=evaluation,\n",
            "    window_size=500,\n",
            ")\n",
            "\n",
            "poptorch_inf_model = poptorch.inferenceModel(inf_model, options=val_options)\n",
            "\n",
            "poptorch_inf_model.entity_embedding.replicaGrouping(\n",
            "    poptorch.CommGroupType.NoGrouping,\n",
            "    0,\n",
            "    poptorch.VariableRetrievalMode.OnePerGroup,\n",
            ")\n",
            "\n",
            "# Compile model\n",
            "_ = batch.pop(\"triple_idx\")\n",
            "res = poptorch_inf_model(**{k: v.flatten(end_dim=1) for k, v in batch.items()})\n",
            "\n",
            "# Run inference\n",
            "val_log = []\n",
            "start_time = time.time()\n",
            "n_val_queries = 0\n",
            "for batch_val in valid_dl:\n",
            "    triple_idx = batch_val.pop(\"triple_idx\")\n",
            "    step_start_time = time.time()\n",
            "    res = poptorch_inf_model(**{k: v.flatten(end_dim=1) for k, v in batch_val.items()})\n",
            "\n",
            "    n_val_queries += batch_val[\"triple_mask\"].sum()\n",
            "    val_log.append(\n",
            "        {\n",
            "            k: v.sum()\n",
            "            for k, v in zip(\n",
            "                evaluation.metrics.keys(),\n",
            "                res[\"metrics\"].T,\n",
            "            )\n",
            "        }\n",
            "    )\n",
            "\n",
            "print(f\"Validation time (sec): {(time.time() - start_time):.5f}\\n\")\n",
            "\n",
            "for metric in val_log[0].keys():\n",
            "    reduced_metric = sum([l[metric] for l in val_log]) / n_val_queries\n",
            "    print(\"%s : %f\" % (metric, reduced_metric))\n",
            "\n",
            "poptorch_inf_model.detachFromDevice()\n",
            "del valid_dl"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "By comparing the validation time, we see that **by moving from 1 IPU to 4 IPUs we speed up inference by a factor of >5X**. This more than linear speed-up is explained by the fact that, on top of the usual advantages offered by data parallelism, when using multiple IPUs each of them needs to score queries only against the set of entities stored in its memory, which is a smaller and smaller fraction of the total number of entities the more IPUs are used."
         ]
      }
   ],
   "metadata": {
      "kernelspec": {
         "display_name": ".venv",
         "language": "python",
         "name": "python3"
      },
      "language_info": {
         "codemirror_mode": {
            "name": "ipython",
            "version": 3
         },
         "file_extension": ".py",
         "mimetype": "text/x-python",
         "name": "python",
         "nbconvert_exporter": "python",
         "pygments_lexer": "ipython3",
         "version": "3.8.10"
      },
      "orig_nbformat": 4
   },
   "nbformat": 4,
   "nbformat_minor": 2
}