notebooks/3_wikikg2_fp16.ipynb

{
   "cells": [
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "# FP16 embeddings on OGBL-WikiKG2\n",
            "\n",
            "<em>Copyright (c) 2023 Graphcore Ltd. All rights reserved.</em>\n",
            "\n",
            "BESS-KGE (`besskge`) is a PyTorch library for knowledge graph embedding (KGE) models on IPUs implementing the distribution framework [BESS](https://arxiv.org/abs/2211.12281), with embedding tables stored in the IPU SRAM.\n",
            "\n",
            "The aim of this notebook is to show how to use FP16 weights to reduce memory requirements for KGE models and speed up computations. This is especially important when dealing with knowledge graphs with a large number of entities, whose embeddings - if stored in FP32 - would require too many IPUs.\n",
            "\n",
            "As a study case we look at the [ogbl-wikikg2](https://ogb.stanford.edu/docs/linkprop/#ogbl-wikikg2) dataset, a knowledge graph containing 2.5M entities.\n",
            "\n",
            "If you have not already done so, we suggest running the [KGE Training and Inference on OGBL-BioKG](1_biokg_training_inference.ipynb) notebook to get familiar with the basic functionalities of the BESS-KGE library."
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Environment setup\n",
            "\n",
            "To run the demo using IPU hardware, you need to have the Poplar SDK enabled and a PopTorch wheel installed. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to do this. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine."
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Dependencies"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "We recommend that you install `besskge` directly from the GitHub sources:"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 1,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Found existing installation: besskge 0.1\n",
                  "Uninstalling besskge-0.1:\n",
                  "  Successfully uninstalled besskge-0.1\n"
               ]
            }
         ],
         "source": [
            "import sys\n",
            "!{sys.executable} -m pip uninstall -y besskge\n",
            "!pip install -q git+https://github.com/graphcore-research/bess-kge.git\n",
            "\n",
            "!pip install -q matplotlib"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Next, import the necessary dependencies. "
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 1,
         "metadata": {},
         "outputs": [],
         "source": [
            "import os\n",
            "import pathlib\n",
            "import time\n",
            "\n",
            "import matplotlib.pyplot as plt\n",
            "import numpy as np\n",
            "import poptorch\n",
            "import torch\n",
            "\n",
            "from besskge.batch_sampler import RandomShardedBatchSampler, RigidShardedBatchSampler\n",
            "from besskge.bess import EmbeddingMovingBessKGE, ScoreMovingBessKGE, TopKQueryBessKGE\n",
            "from besskge.dataset import KGDataset\n",
            "from besskge.embedding import init_KGE_normal\n",
            "from besskge.loss import SampledSoftmaxCrossEntropyLoss\n",
            "from besskge.metric import Evaluation\n",
            "from besskge.negative_sampler import (\n",
            "    PlaceholderNegativeSampler,\n",
            "    RandomShardedNegativeSampler,\n",
            "    TripleBasedShardedNegativeSampler,\n",
            ")\n",
            "from besskge.scoring import TransE\n",
            "from besskge.sharding import PartitionedTripleSet, Sharding\n",
            "\n",
            "dataset_directory = os.getenv(\"DATASET_DIR\", \"../datasets/\") + \"/wikikg2/\""
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Sharding entities and triples\n",
            "\n",
            "The OGBL-WikiKG2 dataset can be downloaded and preprocessed with the built-in method of `KGDataset`, `build_ogbl_wikikg2`. Sharding of entities and triples is performed as shown in the [KGE Training and Inference on OGBL-BioKG](1_biokg_training_inference.ipynb) notebook."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 2,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Number of entities: 2,500,604\n",
                  "\n",
                  "Number of relation types: 535\n",
                  "\n",
                  "Number of triples: \n",
                  " training: 16,109,182 \n",
                  " validation/test: 429,456\n",
                  "\n",
                  "Number of negative heads/tails for validation/test triples: 500\n"
               ]
            }
         ],
         "source": [
            "wikikg = KGDataset.build_ogbl_wikikg2(root=pathlib.Path(dataset_directory))\n",
            "\n",
            "print(f\"Number of entities: {wikikg.n_entity:,}\\n\")\n",
            "print(f\"Number of relation types: {wikikg.n_relation_type}\\n\")\n",
            "print(\n",
            "    f\"Number of triples: \\n training: {wikikg.triples['train'].shape[0]:,} \\n validation/test: {wikikg.triples['valid'].shape[0]:,}\\n\"\n",
            ")\n",
            "print(\n",
            "    f\"Number of negative heads/tails for validation/test triples: {wikikg.neg_heads['valid'].shape[-1]}\"\n",
            ")"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 28,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Number of total entities: 2,500,604\n",
                  "\n",
                  "Number of shards: 4\n",
                  "\n",
                  "Number of entities in each shard: 625,151\n",
                  "\n",
                  "Global entity IDs on 4 shards:\n",
                  " [[      0       5      10 ... 2500598 2500599 2500603]\n",
                  " [      2       9      11 ... 2500593 2500600 2500601]\n",
                  " [      6       7       8 ... 2500594 2500596 2500602]\n",
                  " [      1       3       4 ... 2500563 2500570 2500587]]\n",
                  "\n"
               ]
            }
         ],
         "source": [
            "# Train on 4 IPUs\n",
            "n_shard = 4\n",
            "\n",
            "seed = 432\n",
            "\n",
            "sharding = Sharding.create(n_entity=wikikg.n_entity, n_shard=n_shard, seed=seed)\n",
            "\n",
            "print(f\"Number of total entities: {sharding.n_entity:,}\\n\")\n",
            "\n",
            "print(f\"Number of shards: {sharding.n_shard}\\n\")\n",
            "\n",
            "print(f\"Number of entities in each shard: {sharding.max_entity_per_shard:,}\\n\")\n",
            "\n",
            "print(f\"Global entity IDs on {n_shard} shards:\\n {sharding.shard_and_idx_to_entity}\\n\")\n"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 29,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Number of triples per (h,t) shard-pair:\n",
                  " [[1113036  854970  876020 1182313]\n",
                  " [1111203  854342  876685 1182587]\n",
                  " [1113802  856550  877509 1185590]\n",
                  " [1111410  855467  875729 1181969]]\n"
               ]
            }
         ],
         "source": [
            "train_triples = PartitionedTripleSet.create_from_dataset(\n",
            "    dataset=wikikg, part=\"train\", sharding=sharding, partition_mode=\"ht_shardpair\"\n",
            ")\n",
            "\n",
            "print(f\"Number of triples per (h,t) shard-pair:\\n {train_triples.triple_counts}\")"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Interleaved training and validation (vs all)\n",
            "\n",
            "We use the `RandomShardedBatchSampler` batch sampler class, which creates a batch by randomly sampling (with replacement) a fixed number (=`batch_sampler.positive_per_partition`) of triples from each of the 16 shard-pairs.\n",
            "\n",
            "Negative entities, used to construct negative samples, are also sampled randomly using the `RandomShardedNegativeSampler` class. Here we construct negative samples by always corrupting the tail of positive triples, as specified by the choice `corruption_scheme=\"t\"`."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 30,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "# triples per shard-pair per step: 128 \n",
                  "\n",
                  "head         torch.Size([100, 4, 4, 128])   torch.int32;\n",
                  "relation     torch.Size([100, 4, 4, 128])   torch.int32;\n",
                  "tail         torch.Size([100, 4, 4, 128])   torch.int32;\n",
                  "negative     torch.Size([100, 4, 4, 1, 32]) torch.int32;\n"
               ]
            }
         ],
         "source": [
            "device_iterations = 100\n",
            "accum_factor = 1\n",
            "shard_bs = 512\n",
            "\n",
            "neg_sampler = RandomShardedNegativeSampler(\n",
            "    n_negative=32,\n",
            "    sharding=sharding,\n",
            "    seed=seed,\n",
            "    corruption_scheme=\"t\",\n",
            "    local_sampling=False,\n",
            "    flat_negative_format=True,\n",
            ")\n",
            "\n",
            "batch_sampler = RandomShardedBatchSampler(\n",
            "    partitioned_triple_set=train_triples,\n",
            "    negative_sampler=neg_sampler,\n",
            "    shard_bs=shard_bs,\n",
            "    batches_per_step=device_iterations * accum_factor,\n",
            "    seed=seed,\n",
            ")\n",
            "\n",
            "\n",
            "print(f\"# triples per shard-pair per step: {batch_sampler.positive_per_partition} \\n\")\n",
            "\n",
            "# Example batch\n",
            "idx_sampler = iter(batch_sampler.get_dataloader_sampler(shuffle=True))\n",
            "for k, v in batch_sampler[next(idx_sampler)].items():\n",
            "    print(f\"{k:<12} {str(v.shape):<30} {v.dtype};\")"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Notice from the `negative` shape that we are **decoupling** the number of negative samples from the batch size, by using the negative sampler option `flat_negative_format=True`. Instead of sampling negative entities on a triple basis, we sample them on a shard-pair basis. In this case, each pair of devices will exchange 32 negative entities in both directions (while, if using `flat_negative_format=False`, this number would need to be a multiple of the shard batch size - see for instance the shape of `negative` in the [KGE Training and Inference on OGBL-BioKG](1_biokg_training_inference.ipynb) notebook). This of course requires the use of negative sample sharing.\n",
            "\n",
            "Thanks to this decoupling we can increase the shard batch size without increasing the number of negative samples to score in each batch. This is important as we don't intend to use gradient accumulation (as specified by `accum_factor = 1`), thus saving on memory which would have been used by the weight accumulation tensors."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 31,
         "metadata": {},
         "outputs": [],
         "source": [
            "options = poptorch.Options()\n",
            "options.replication_factor = sharding.n_shard\n",
            "options.deviceIterations(device_iterations)\n",
            "options._popart.setPatterns(dict(RemoveAllReducePattern=True))\n",
            "\n",
            "# Enable stochastic rounding on IPU for more stable half-precision training\n",
            "options.Precision.enableStochasticRounding(True)\n",
            "\n",
            "train_dl = batch_sampler.get_dataloader(\n",
            "    options=options, shuffle=True, num_workers=3, persistent_workers=True\n",
            ")"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "We will train a simple **TransE** KGE model with an embedding size of 100, in order to fit the embedding tables in the SRAM of 4 IPUs, and use **sampled softmax cross entropy loss**.\n",
            "\n",
            "We specify a custom initialization scheme for entity and relation embedding tables using the `entity_initializer` and `relation_initializer` arguments of the scoring function. We can either pass directly the tensors of parameters (if we want to load a checkipoint) or a list of intialization functions (either from `torch.nn.init` or `besskge.embedding`).\n",
            "\n",
            "For this dataset we find it to be beneficial to construct negative samples by corrupting the tail of a positive triple using the randomly sampled negative entities in `negative` and also the **tails of the other positive triples** in the same micro-batch. This can be done simply by setting the flag `augment_negative=True` when instantiating the `EmbeddingMovingBessKGE` distribution scheme. This strategy also has the advantage of increasing the number of negative samples used in the contrastive loss without increasing the costs of gathering and communicating negative entities across IPUs."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 32,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "# model parameters: 250,113,900\n"
               ]
            }
         ],
         "source": [
            "loss_fn = SampledSoftmaxCrossEntropyLoss(n_entity=wikikg.n_entity)\n",
            "# Initializer for entity and relation embeddings\n",
            "emb_initializer = [init_KGE_normal]\n",
            "transe_score_fn = TransE(\n",
            "    negative_sample_sharing=True,\n",
            "    scoring_norm=1,\n",
            "    sharding=sharding,\n",
            "    n_relation_type=wikikg.n_relation_type,\n",
            "    embedding_size=100,\n",
            "    entity_initializer=emb_initializer,\n",
            "    relation_initializer=emb_initializer,\n",
            ")\n",
            "\n",
            "model = EmbeddingMovingBessKGE(\n",
            "    negative_sampler=neg_sampler,\n",
            "    score_fn=transe_score_fn,\n",
            "    loss_fn=loss_fn,\n",
            "    augment_negative=True,\n",
            ")\n",
            "\n",
            "print(f\"# model parameters: {model.n_embedding_parameters:,}\")"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "As in standard PyTorch, we can use `model.half()` to **cast the embedding tables to FP16**, before wrapping the model in `poptorch.trainingModel`.\n",
            "\n",
            "We train with SGD momentum instead of Adam, for reduced optimizer state memory usage."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 33,
         "metadata": {},
         "outputs": [
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "[10:58:06.333] [poptorch:cpp] [warning] [DISPATCHER] Tensor (ptr 0xa0c3650) type coerced from Long to Int\n",
                  "Graph compilation: 100%|██████████| 100/100 [02:19<00:00]\n"
               ]
            }
         ],
         "source": [
            "# FP16 weights\n",
            "model.half()\n",
            "\n",
            "opt = poptorch.optim.SGD(\n",
            "    model.parameters(),\n",
            "    lr=0.001,\n",
            "    momentum=0.95,\n",
            "    velocity_accum_type=torch.float16,\n",
            ")\n",
            "\n",
            "poptorch_model = poptorch.trainingModel(model, options=options, optimizer=opt)\n",
            "\n",
            "# The variable entity_embedding needs to hold different values on each replica,\n",
            "# corresponding to the shards of the entity embedding table\n",
            "poptorch_model.entity_embedding.replicaGrouping(\n",
            "    poptorch.CommGroupType.NoGrouping,\n",
            "    0,\n",
            "    poptorch.VariableRetrievalMode.OnePerGroup,\n",
            ")\n",
            "\n",
            "# Compile model\n",
            "batch = next(iter(train_dl))\n",
            "res = poptorch_model(**{k: v.flatten(end_dim=1) for k, v in batch.items()})\n",
            "\n",
            "poptorch_model.detachFromDevice()"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "That's all we need to do to use FP16 weights! The triple scoring in TransE via L1 distance is also going to be performed in FP16, while - for stability reasons - the loss is always computed in FP32.\n",
            "\n",
            "To track the performance evolution during training, we will validate at a regular cadence on a set of 4000 triples sampled randomly from the validation set. We provide **no tail candidates** (hence, scoring each (h,r,?) query against all 2.5 million entities in the knowledge graph). As this would be too slow to do on a CPU, we will perform the task on the IPU using the BESS `TopKQueryBessKGE` (see the [Knowledge Graph Completion on YAGO3-10](2_yago_topk_prediction.ipynb) notebook for more details on this class and how to use it).\n",
            "\n",
            "To partition a custom set of triples (that is, not one of the \"official\" triple parts specified when creating `KGDataset`) on the fly, use the `PartitionedTripleSet.create_from_queries` function as follows."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 34,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Number of triples per h_shard:\n",
                  "[ 987 1035  990  988]\n"
               ]
            }
         ],
         "source": [
            "n_sample_queries = 4000\n",
            "\n",
            "val_device_iterations = 2\n",
            "val_shard_bs = 512\n",
            "\n",
            "# Partition a random subset of n_sample_queries triples taken from wikikg.triples[\"valid\"]\n",
            "subset_val_triples = wikikg.triples[\"valid\"][\n",
            "    np.random.default_rng(seed=seed).choice(\n",
            "        wikikg.triples[\"valid\"].shape[0], n_sample_queries\n",
            "    )\n",
            "]\n",
            "sample_val_triples = PartitionedTripleSet.create_from_queries(\n",
            "    wikikg,\n",
            "    sharding,\n",
            "    queries=subset_val_triples[:, :2],\n",
            "    query_mode=\"hr\",\n",
            "    ground_truth=subset_val_triples[:, 2],\n",
            ")\n",
            "\n",
            "candidate_sampler = PlaceholderNegativeSampler(corruption_scheme=\"t\", seed=seed)\n",
            "bs_sample = RigidShardedBatchSampler(\n",
            "    partitioned_triple_set=sample_val_triples,\n",
            "    negative_sampler=candidate_sampler,\n",
            "    shard_bs=val_shard_bs,\n",
            "    batches_per_step=val_device_iterations,\n",
            "    seed=seed,\n",
            "    duplicate_batch=False,\n",
            "    return_triple_idx=False,\n",
            ")\n",
            "\n",
            "print(\"Number of triples per h_shard:\")\n",
            "print(sample_val_triples.triple_counts)"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 35,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "head         torch.Size([2, 4, 512])       \n",
                  "relation     torch.Size([2, 4, 512])       \n",
                  "tail         torch.Size([2, 4, 512])       \n",
                  "triple_mask  torch.Size([2, 4, 512])       \n"
               ]
            }
         ],
         "source": [
            "val_options = poptorch.Options()\n",
            "val_options.replication_factor = sharding.n_shard\n",
            "val_options.deviceIterations(bs_sample.batches_per_step)\n",
            "val_options.outputMode(poptorch.OutputMode.All)\n",
            "\n",
            "sample_valid_dl = bs_sample.get_dataloader(\n",
            "    options=val_options, shuffle=False, num_workers=2, persistent_workers=True\n",
            ")\n",
            "\n",
            "# Example batch\n",
            "val_batch = next(iter(sample_valid_dl))\n",
            "for k, v in val_batch.items():\n",
            "    print(f\"{k:<12} {str(v.shape):<30}\")"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 36,
         "metadata": {},
         "outputs": [
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "[11:01:13.333] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 185\n",
                  "[11:01:13.351] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 242\n",
                  "[11:01:13.354] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 265\n",
                  "Graph compilation: 100%|██████████| 100/100 [02:10<00:00]\n"
               ]
            }
         ],
         "source": [
            "# Inference model\n",
            "\n",
            "evaluation = Evaluation([\"mrr\"], worst_rank_infty=True, reduction=\"sum\")\n",
            "\n",
            "inf_model = TopKQueryBessKGE(\n",
            "    k=10,\n",
            "    candidate_sampler=candidate_sampler,\n",
            "    score_fn=transe_score_fn,\n",
            "    evaluation=evaluation,\n",
            "    window_size=500,\n",
            ")\n",
            "\n",
            "poptorch_inf_model = poptorch.inferenceModel(inf_model, options=val_options)\n",
            "\n",
            "poptorch_inf_model.entity_embedding.replicaGrouping(\n",
            "    poptorch.CommGroupType.NoGrouping,\n",
            "    0,\n",
            "    poptorch.VariableRetrievalMode.OnePerGroup,\n",
            ")\n",
            "\n",
            "# Compile inference model\n",
            "val_res = poptorch_inf_model(**{k: v.flatten(end_dim=1) for k, v in val_batch.items()})\n",
            "\n",
            "poptorch_inf_model.detachFromDevice()"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Let us train the model for 70 epochs, performing validation on the selected random set of triples every 10 epochs. This should take less than 10 minutes.\n",
            "\n",
            "Be aware that the interleaved execution scheme introduces some overhead for detaching and attaching the executables of the two different models to the available IPUs. This overhead is **not** included in the time measurements that we print out, as it can be significantly cut down by using two separate IPU-POD4 systems, one for training and one for validation. If you are running a session with at least 8 IPUs:\n",
            "\n",
            "* remove the `model.detachFromDevice()`, `model.attachToDevice()` parts of the code in the cells above\n",
            "* remove the `model.detachFromDevice()`, `model.attachToDevice()` parts of the code in the next cell\n",
            "* uncomment the 2 lines of code (`poptorch_model.copyWeightsToHost()` and `poptorch_inf_model.copyWeightsToDevice()`) in the next cell"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 37,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Epoch 1 loss: 11.539560 --- positive triples processed: 1.90e+07\n",
                  "Epoch duration (sec): 4.60964 (average step time: 0.04855)\n",
                  "Epoch 1 sample MRR: 0.0526 (validation time: 0.89469)\n",
                  "Epoch 2 loss: 10.939403 --- positive triples processed: 3.81e+07\n",
                  "Epoch duration (sec): 4.46646 (average step time: 0.04796)\n",
                  "Epoch 3 loss: 10.406051 --- positive triples processed: 5.71e+07\n",
                  "Epoch duration (sec): 4.46946 (average step time: 0.04799)\n",
                  "Epoch 4 loss: 9.870989 --- positive triples processed: 7.62e+07\n",
                  "Epoch duration (sec): 4.46372 (average step time: 0.04792)\n",
                  "Epoch 5 loss: 9.366117 --- positive triples processed: 9.52e+07\n",
                  "Epoch duration (sec): 4.46765 (average step time: 0.04797)\n",
                  "Epoch 6 loss: 8.915543 --- positive triples processed: 1.14e+08\n",
                  "Epoch duration (sec): 4.46673 (average step time: 0.04793)\n",
                  "Epoch 7 loss: 8.497296 --- positive triples processed: 1.33e+08\n",
                  "Epoch duration (sec): 4.46252 (average step time: 0.04791)\n",
                  "Epoch 8 loss: 8.151726 --- positive triples processed: 1.52e+08\n",
                  "Epoch duration (sec): 4.46434 (average step time: 0.04794)\n",
                  "Epoch 9 loss: 7.814506 --- positive triples processed: 1.71e+08\n",
                  "Epoch duration (sec): 4.47936 (average step time: 0.04806)\n",
                  "Epoch 10 loss: 7.515382 --- positive triples processed: 1.90e+08\n",
                  "Epoch duration (sec): 4.46475 (average step time: 0.04792)\n",
                  "Epoch 11 loss: 7.257089 --- positive triples processed: 2.10e+08\n",
                  "Epoch duration (sec): 4.48401 (average step time: 0.04813)\n",
                  "Epoch 11 sample MRR: 0.1915 (validation time: 0.88983)\n",
                  "Epoch 12 loss: 7.029889 --- positive triples processed: 2.29e+08\n",
                  "Epoch duration (sec): 4.46270 (average step time: 0.04793)\n",
                  "Epoch 13 loss: 6.819004 --- positive triples processed: 2.48e+08\n",
                  "Epoch duration (sec): 4.46902 (average step time: 0.04793)\n",
                  "Epoch 14 loss: 6.635813 --- positive triples processed: 2.67e+08\n",
                  "Epoch duration (sec): 4.46475 (average step time: 0.04789)\n",
                  "Epoch 15 loss: 6.465908 --- positive triples processed: 2.86e+08\n",
                  "Epoch duration (sec): 4.48516 (average step time: 0.04816)\n",
                  "Epoch 16 loss: 6.318337 --- positive triples processed: 3.05e+08\n",
                  "Epoch duration (sec): 4.46776 (average step time: 0.04796)\n",
                  "Epoch 17 loss: 6.191060 --- positive triples processed: 3.24e+08\n",
                  "Epoch duration (sec): 4.47195 (average step time: 0.04800)\n",
                  "Epoch 18 loss: 6.085122 --- positive triples processed: 3.43e+08\n",
                  "Epoch duration (sec): 4.53764 (average step time: 0.04864)\n",
                  "Epoch 19 loss: 5.934304 --- positive triples processed: 3.62e+08\n",
                  "Epoch duration (sec): 4.57562 (average step time: 0.04905)\n",
                  "Epoch 20 loss: 5.842280 --- positive triples processed: 3.81e+08\n",
                  "Epoch duration (sec): 4.45862 (average step time: 0.04787)\n",
                  "Epoch 21 loss: 5.762109 --- positive triples processed: 4.00e+08\n",
                  "Epoch duration (sec): 4.47655 (average step time: 0.04803)\n",
                  "Epoch 21 sample MRR: 0.2267 (validation time: 0.88997)\n",
                  "Epoch 22 loss: 5.660895 --- positive triples processed: 4.19e+08\n",
                  "Epoch duration (sec): 4.46483 (average step time: 0.04795)\n",
                  "Epoch 23 loss: 5.580583 --- positive triples processed: 4.38e+08\n",
                  "Epoch duration (sec): 4.45884 (average step time: 0.04788)\n",
                  "Epoch 24 loss: 5.500723 --- positive triples processed: 4.57e+08\n",
                  "Epoch duration (sec): 4.46727 (average step time: 0.04793)\n",
                  "Epoch 25 loss: 5.435099 --- positive triples processed: 4.76e+08\n",
                  "Epoch duration (sec): 4.46389 (average step time: 0.04793)\n",
                  "Epoch 26 loss: 5.349809 --- positive triples processed: 4.95e+08\n",
                  "Epoch duration (sec): 4.46493 (average step time: 0.04794)\n",
                  "Epoch 27 loss: 5.294559 --- positive triples processed: 5.14e+08\n",
                  "Epoch duration (sec): 4.46032 (average step time: 0.04790)\n",
                  "Epoch 28 loss: 5.230184 --- positive triples processed: 5.33e+08\n",
                  "Epoch duration (sec): 4.45850 (average step time: 0.04788)\n",
                  "Epoch 29 loss: 5.184251 --- positive triples processed: 5.52e+08\n",
                  "Epoch duration (sec): 4.45941 (average step time: 0.04789)\n",
                  "Epoch 30 loss: 5.149366 --- positive triples processed: 5.71e+08\n",
                  "Epoch duration (sec): 4.46138 (average step time: 0.04791)\n",
                  "Epoch 31 loss: 5.091528 --- positive triples processed: 5.90e+08\n",
                  "Epoch duration (sec): 4.46497 (average step time: 0.04794)\n",
                  "Epoch 31 sample MRR: 0.2525 (validation time: 0.88975)\n",
                  "Epoch 32 loss: 5.036900 --- positive triples processed: 6.09e+08\n",
                  "Epoch duration (sec): 4.52223 (average step time: 0.04852)\n",
                  "Epoch 33 loss: 4.999296 --- positive triples processed: 6.29e+08\n",
                  "Epoch duration (sec): 4.58633 (average step time: 0.04916)\n",
                  "Epoch 34 loss: 4.959585 --- positive triples processed: 6.48e+08\n",
                  "Epoch duration (sec): 4.46112 (average step time: 0.04790)\n",
                  "Epoch 35 loss: 4.929678 --- positive triples processed: 6.67e+08\n",
                  "Epoch duration (sec): 4.45927 (average step time: 0.04789)\n",
                  "Epoch 36 loss: 4.881487 --- positive triples processed: 6.86e+08\n",
                  "Epoch duration (sec): 4.46107 (average step time: 0.04790)\n",
                  "Epoch 37 loss: 4.871422 --- positive triples processed: 7.05e+08\n",
                  "Epoch duration (sec): 4.46823 (average step time: 0.04796)\n",
                  "Epoch 38 loss: 4.798804 --- positive triples processed: 7.24e+08\n",
                  "Epoch duration (sec): 4.46421 (average step time: 0.04793)\n",
                  "Epoch 39 loss: 4.770441 --- positive triples processed: 7.43e+08\n",
                  "Epoch duration (sec): 4.45899 (average step time: 0.04788)\n",
                  "Epoch 40 loss: 4.784256 --- positive triples processed: 7.62e+08\n",
                  "Epoch duration (sec): 4.46355 (average step time: 0.04793)\n",
                  "Epoch 41 loss: 4.722328 --- positive triples processed: 7.81e+08\n",
                  "Epoch duration (sec): 4.46442 (average step time: 0.04794)\n",
                  "Epoch 41 sample MRR: 0.2612 (validation time: 0.88990)\n",
                  "Epoch 42 loss: 4.717527 --- positive triples processed: 8.00e+08\n",
                  "Epoch duration (sec): 4.46442 (average step time: 0.04794)\n",
                  "Epoch 43 loss: 4.710725 --- positive triples processed: 8.19e+08\n",
                  "Epoch duration (sec): 4.46190 (average step time: 0.04792)\n",
                  "Epoch 44 loss: 4.656087 --- positive triples processed: 8.38e+08\n",
                  "Epoch duration (sec): 4.46244 (average step time: 0.04792)\n",
                  "Epoch 45 loss: 4.656974 --- positive triples processed: 8.57e+08\n",
                  "Epoch duration (sec): 4.46762 (average step time: 0.04798)\n",
                  "Epoch 46 loss: 4.636364 --- positive triples processed: 8.76e+08\n",
                  "Epoch duration (sec): 4.47196 (average step time: 0.04802)\n",
                  "Epoch 47 loss: 4.627054 --- positive triples processed: 8.95e+08\n",
                  "Epoch duration (sec): 4.46888 (average step time: 0.04799)\n",
                  "Epoch 48 loss: 4.614621 --- positive triples processed: 9.14e+08\n",
                  "Epoch duration (sec): 4.46070 (average step time: 0.04790)\n",
                  "Epoch 49 loss: 4.598107 --- positive triples processed: 9.33e+08\n",
                  "Epoch duration (sec): 4.46274 (average step time: 0.04791)\n",
                  "Epoch 50 loss: 4.578415 --- positive triples processed: 9.52e+08\n",
                  "Epoch duration (sec): 4.46991 (average step time: 0.04799)\n",
                  "Epoch 51 loss: 4.555379 --- positive triples processed: 9.71e+08\n",
                  "Epoch duration (sec): 4.47375 (average step time: 0.04804)\n",
                  "Epoch 51 sample MRR: 0.2627 (validation time: 0.89033)\n",
                  "Epoch 52 loss: 4.540395 --- positive triples processed: 9.90e+08\n",
                  "Epoch duration (sec): 4.46183 (average step time: 0.04791)\n",
                  "Epoch 53 loss: 4.519772 --- positive triples processed: 1.01e+09\n",
                  "Epoch duration (sec): 4.46083 (average step time: 0.04790)\n",
                  "Epoch 54 loss: 4.536430 --- positive triples processed: 1.03e+09\n",
                  "Epoch duration (sec): 4.51554 (average step time: 0.04841)\n",
                  "Epoch 55 loss: 4.481946 --- positive triples processed: 1.05e+09\n",
                  "Epoch duration (sec): 4.52571 (average step time: 0.04854)\n",
                  "Epoch 56 loss: 4.512139 --- positive triples processed: 1.07e+09\n",
                  "Epoch duration (sec): 4.46114 (average step time: 0.04789)\n",
                  "Epoch 57 loss: 4.490931 --- positive triples processed: 1.09e+09\n",
                  "Epoch duration (sec): 4.46138 (average step time: 0.04791)\n",
                  "Epoch 58 loss: 4.508470 --- positive triples processed: 1.10e+09\n",
                  "Epoch duration (sec): 4.46369 (average step time: 0.04790)\n",
                  "Epoch 59 loss: 4.457185 --- positive triples processed: 1.12e+09\n",
                  "Epoch duration (sec): 4.46205 (average step time: 0.04788)\n",
                  "Epoch 60 loss: 4.467912 --- positive triples processed: 1.14e+09\n",
                  "Epoch duration (sec): 4.48002 (average step time: 0.04809)\n",
                  "Epoch 61 loss: 4.459184 --- positive triples processed: 1.16e+09\n",
                  "Epoch duration (sec): 4.48192 (average step time: 0.04811)\n",
                  "Epoch 61 sample MRR: 0.2626 (validation time: 0.89042)\n",
                  "Epoch 62 loss: 4.466573 --- positive triples processed: 1.18e+09\n",
                  "Epoch duration (sec): 4.47011 (average step time: 0.04799)\n",
                  "Epoch 63 loss: 4.442837 --- positive triples processed: 1.20e+09\n",
                  "Epoch duration (sec): 4.46524 (average step time: 0.04795)\n",
                  "Epoch 64 loss: 4.448936 --- positive triples processed: 1.22e+09\n",
                  "Epoch duration (sec): 4.47425 (average step time: 0.04805)\n",
                  "Epoch 65 loss: 4.419045 --- positive triples processed: 1.24e+09\n",
                  "Epoch duration (sec): 4.47169 (average step time: 0.04802)\n",
                  "Epoch 66 loss: 4.409024 --- positive triples processed: 1.26e+09\n",
                  "Epoch duration (sec): 4.48787 (average step time: 0.04820)\n",
                  "Epoch 67 loss: 4.434240 --- positive triples processed: 1.28e+09\n",
                  "Epoch duration (sec): 4.48682 (average step time: 0.04819)\n",
                  "Epoch 68 loss: 4.422602 --- positive triples processed: 1.30e+09\n",
                  "Epoch duration (sec): 4.46942 (average step time: 0.04800)\n",
                  "Epoch 69 loss: 4.415661 --- positive triples processed: 1.31e+09\n",
                  "Epoch duration (sec): 4.46357 (average step time: 0.04793)\n",
                  "Epoch 70 loss: 4.391059 --- positive triples processed: 1.33e+09\n",
                  "Epoch duration (sec): 4.47337 (average step time: 0.04804)\n"
               ]
            },
            {
               "data": {
                  "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAGwCAYAAAAQdOnRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAB8sElEQVR4nO3dd3xN9/8H8NfNzd6ITCFix4hYsWdIzNKiRa0qpbRKq6U2rVXKr6Uotfo1qqqt1hZiE8SOvSJkWNky7/n9cZub3NybSG7uvefem9fz8bgP95x7xjunmvv2Ge+PRBAEAURERERk8MzEDoCIiIiIioeJGxEREZGRYOJGREREZCSYuBEREREZCSZuREREREaCiRsRERGRkWDiRkRERGQkzMUOQNeys7Nx8eJFuLm5wcyMeSoREZExkMlkiIuLQ0BAAMzNTT5dKTaTfxIXL15Es2bNxA6DiIiINBAeHo6mTZuKHYbBMPnEzc3NDYD8P7yHh4fI0RAREVFxxMTEoFmzZorvcZIz+cQtt3vUw8MDlSpVEjkaIiIiKgkOc1LGp0FERERkJJi4ERERERkJJm5ERERERsLkx7gVV05ODrKyssQOw6hYWFhAKpWKHQYREVGZUeYTN0EQEBsbi4SEBLFDMUrOzs5wd3eHRCIROxQiIiKTV+YTt9ykzdXVFba2tkxAikkQBKSlpSE+Ph4AWGqFiIhID8p04paTk6NI2ipUqCB2OEbHxsYGABAfHw9XV1d2mxIREelYmZ6ckDumzdbWVuRIjFfus+P4QCIiIt0r04lbLnaPao7PjoiISH+YuBEREREZCSZuREREREaCiRsRERGRkSjTs0pLIztHBpkgwMxMAnMRFsAdNmwYEhIS8Ndff+n93kRUxslkwOvXQFqa/E9BACQS+cvMLO99/pe29nNcLZVxTNw0FJuUjpepmXB3tIaro7XY4RARyWVnyxOq3FdqqvJ2Ya+SHJeeLu7PWFhCp4+ksbDPzp0DWKGA9ICJWz6CIOB1Vk6xjk3PzEF6Vg7SMnOQlpld6nvbWEi1NkPz6NGjmDRpEi5fvozy5ctj6NCh+Oabb2BuLv/PvWPHDsyePRt3796Fra0tAgIC8Pfff8POzg5hYWH48ssvcf36dVhYWKBu3brYsmULqlSpopXYiMosQQAyMrSbQKnbr+/SPJaW8sRFEPJeMlnee13Q5bU1ZWjxkMli4pbP66wc+M3YL8q9I+cEw9ay9P85njx5gm7dumHYsGHYtGkTbt68iZEjR8La2hqzZs1CTEwMBgwYgEWLFqFPnz5ITk7G8ePHIQgCsrOz0bt3b4wcORJbt25FZmYmwsPDWfKDqKDkZODCBeDsWeDBg+InWzKZ/mKUSOQtQLkvOzvl7aJexT3WxgYoTuHtwpK6gi9tf6bPe1mz54X0g4mbifnpp5/g7e2N5cuXQyKRoHbt2nj69Cm++uorzJgxAzExMcjOzsbbb7+taEWrX78+AODly5dITExEjx49UK1aNQBAnTp1RPtZiAxCTg5w/bo8Sct9RUaWLgmzsNBO4lTUcVZWhjMeLP/YNK6wQlQqTNzysbGQInJOcLGOjU1Mx/OUDFSws4KHc+n/pWVjoZ1fZjdu3ECLFi2UWslatWqFlJQUREdHw9/fH506dUL9+vURHByMLl26oG/fvihXrhzKly+PYcOGITg4GJ07d0ZQUBD69+/PdUipbHnyRDlJO39e3qJWkLc3EBgI1K0LODgUP8mysZEnbkREGmDilo9EIil2d6WtpTmsLbJhbSnVShenvkilUhw8eBCnTp3CgQMH8OOPP2Lq1Kk4e/YsqlativXr1+PTTz/Fvn378Ntvv2HatGk4ePAgmjdvLnboRNqXkiJPzM6eBcLD5X8+eaJ6nIMD0LQp0KyZPFkLDAT4DxoiEoHxZBwGJrdBSzCwAal16tTBH3/8AUEQFK1uJ0+ehIODAypVqgRAnqC2atUKrVq1wowZM1ClShX8+eefmDhxIgAgICAAAQEBmDJlClq0aIEtW7YwcSPjl5Mj7+LM35p2/bpql6eZGVC/fl6CFhgI1K7NLj4iMghM3DSk6IgUMW9LTEzEpUuXlPaNGjUKy5YtwyeffIJx48bh1q1bmDlzJiZOnAgzMzOcPXsWoaGh6NKlC1xdXXH27Fk8e/YMderUwYMHD/Dzzz+jV69e8PT0xK1bt3Dnzh0MGTJEnB+QqDSePMlrRcvt8kxJUT2uUiXlJK1xY3m3JhGRAWLipqncFjcRQwgLC0NAQIDSvhEjRmDPnj2YNGkS/P39Ub58eYwYMQLTpk0DADg6OuLYsWNYtmwZkpKSUKVKFSxZsgRdu3ZFXFwcbt68iY0bN+LFixfw8PDA2LFj8dFHH4nx4xEVX2pqXpdn7ktdl6e9PdCkiXKi5ump/3iJiDQkEQytr0/LoqOj4e3tjcePHyu6CnOlp6fjwYMHqFq1KqxLOJX7WXIGYhJfw9nGEpUrlN2ii6V5hkQayckBbtxQTtKuXVPf5VmvnnKSVqcOuzyJjERR399lGVvcNKQY4yZqmxtRGfD0aV6CFh4ur1CvrsvTy0u1y9PeXv/xEhHpEBM3DeWOcTPt9koiPUtNzStsm/uKjlY9zs5OtcvTy0v/8RIR6ZmoiduxY8fw3Xff4cKFC4iJicGff/6J3r17AwCysrIwbdo07NmzB/fv34eTkxOCgoKwYMECeBrAmJTcGZvM24g0lJMD3Lyp2uWZU2DZOTMzea20/Emanx+7PImoTBI1cUtNTYW/vz8++OADvP3220qfpaWlISIiAtOnT4e/vz9evXqF8ePHo1evXjh//rxIEecx+6/JTcYmN6LiiYlRLWybnKx6nKenapeng4P+4yUiMkCiJm5du3ZF165d1X7m5OSEgwcPKu1bvnw5mjVrhqioKFSuXFkfIRYqt8UtPbN4i9ITlSlpaapdno8fqx5na6va5clByEREhTKqMW6JiYmQSCRwdnYu9JiMjAxkZGQotpPV/YteCzKy5AlbDlvcqKyTyeSzPPPXTLt6VbXLUyJR3+VpblS/hoiIRGU0vzHT09Px1VdfYcCAAXB0dCz0uPnz52P27Nk6jyc1X0tbVo4MFlIznd+TyKBcugRMnQocP66+y9PDQzlJa9KEXZ5ERKVkFIlbVlYW+vfvD0EQsHLlyiKPnTJlimLpJgB48uQJ/Pz8tB6TJN97NrpRmfL6NTBnDvDdd3mtara28rFoBbs8JZKir0VERCVi8M1EuUnbo0ePcPDgwSJb2wDAysoKjo6OipeDjv6FLzXL+0Iy8RrGRHmOHAEaNAAWLJAnbX37AhERQGIicOyYPJnr2xfw9mbSRkSiWLFiBXx8fGBtbY3AwECEh4cXeuyaNWvQpk0blCtXDuXKlUNQUJDK8cOGDYNEIlF6hYSE6PrHKJRBJ265SdudO3dw6NAhVKhQQeyQFNwc81YJkImQt+X+RRo9erTKZ2PHjoVEIsGwYcOUjpVIJLCwsEDVqlXx5ZdfIj09Xem8/H8pHR0d0bRpU/z999/6+HHI0L16BYwcCXTsCNy9K5/5+ddfwO+/AwEBHKdGRAbht99+w8SJEzFz5kxERETA398fwcHBiI+PV3t8WFgYBgwYgCNHjuD06dPw9vZGly5d8KTAknkhISGIiYlRvLZu3aqPH0ctURO3lJQUXLp0SbFQ+oMHD3Dp0iVERUUhKysLffv2xfnz57F582bk5OQgNjYWsbGxyMzMFDNsAICled6jE2uCgre3N7Zt24bXr18r9qWnp2PLli0qs25z/9Ldv38fS5cuxerVqzFz5kyVa65fvx4xMTE4f/48WrVqhb59++Lq1as6/1nIQAkC8Mcf8kkEa9fK940ZA0RGAm+9JW5sRFQmJCcnIykpSfHKPwGxoO+//x4jR47E8OHD4efnh1WrVsHW1hbr1q1Te/zmzZvx8ccfo2HDhqhduzbWrl0LmUyG0NBQpeOsrKzg7u6ueJUrV06rP2NJiJq4nT9/HgEBAYqF0idOnIiAgADMmDEDT548wa5duxAdHY2GDRvCw8ND8Tp16pSYYSvYWMgLgGbnyN5wpG40atQI3t7e2Llzp2Lfzp07UblyZZXF53P/0nl7e6N3794ICgpSKbcCAM7OznB3d0fNmjUxd+5cZGdn48iRIzr/WcgAPXkCvP22vOszNhaoVUveHfrTT4CTk9jREVEZ4efnBycnJ8Vr/vz5ao/LzMzEhQsXEBQUpNhnZmaGoKAgnD59ulj3SktLQ1ZWFsqXL6+0PywsDK6urqhVqxbGjBmDFy9eaP4DlZKo/Rvt27cvcnyY3seOCYK8/lQxpScmQQIgSZYJ5wp2pbu3ra1GY4I++OADrF+/HoMGDQIArFu3DsOHD0dYWFih51y7dg2nTp1ClSpVCj0mOzsbv/zyCwDA0tKyxHGREZPJgDVrgC+/BJKS5N2gU6YAX38NWFu/+XwiIi2KjIyEV74l7aysrNQe9/z5c+Tk5MDNzU1pv5ubG27evFmse3311Vfw9PRUSv5CQkLw9ttvo2rVqrh37x6+/vprdO3aFadPn4ZUhBVcODAlv7S0Ei1KXV+b905Jka+/WELvv/8+pkyZgkePHgEATp48iW3btqkkbv/++y/s7e2RnZ2NjIwMmJmZYfny5SrXGzBgAKRSKV6/fg2ZTAYfHx/0799fox+JjNCtW/KxbMePy7cDA+VJXH2t/m0nIio2BweHN05M1IYFCxYovj+t8/0j9b333lO8r1+/Pho0aIBq1aohLCwMnTp10nlcBTFxM3IVK1ZE9+7dsWHDBgiCgO7du8PFxUXluA4dOmDlypVITU3F0qVLYW5ujnfeeUfluKVLlyIoKAj379/HhAkT8MMPP6g0GZMJysyUzwidM0f+3s4OmDcPGDuWa4ISkVFwcXGBVCpFXFyc0v64uDi4u7sXee7ixYuxYMECHDp0CA0aNCjyWF9fX7i4uODu3btM3ERnaytv+Sqmq08SFO/rezmX/t4a+uCDDzBu3DgA8mnQ6tjZ2aF69eoA5N2p/v7++OWXXzBixAil49zd3VG9enVUr14d69evR7du3RAZGQlXV1eN4yMDFx4OfPihfLUDAOjaFVi5EiiiK52IyNBYWlqicePGCA0NRe/evQFAMdEg9ztSnUWLFuHbb7/F/v370aRJkzfeJzo6Gi9evICHh4e2Qi8Rgy4HoncSibyloZgvwTbvVZLz1L5KUfMqJCQEmZmZyMrKQnBw8BuPNzMzw9dff41p06YpzUgtqFmzZmjcuDG+/fZbjWMjA5aSAkyYADRvLk/aXFyAzZuB3buZtBGRUZo4cSLWrFmDjRs34saNGxgzZgxSU1MxfPhwAMCQIUMwZcoUxfELFy7E9OnTsW7dOvj4+CiqV6T814iTkpKCSZMm4cyZM3j48CFCQ0Px1ltvoXr16sX6vtUFJm4mQCqV4saNG4iMjCz2QMl+/fpBKpUW2kKX67PPPsPq1atVatqQkdu3D6hXD1i2TD4pZ/Bg+XqjAweycC4RGa13330XixcvxowZM9CwYUNcunQJ+/btU0xYiIqKQkxMjOL4lStXIjMzE3379lWqXrF48WIA8u/XK1euoFevXqhZsyZGjBiBxo0b4/jx44VOktA1dpWWgplEAtl/M18FQYBExC+8kg7cNDc3x7hx47Bo0SKMGTMGdoVMjAgJCUHVqlXx7bff4qefftJGqCSm58/lrWz/+598u0oVYPVqQKR/ORIRadu4ceMK7RotOHHv4cOHRV7LxsYG+/fv11Jk2iERTHy9pujoaHh7e+Px48eoVKmS0mfp6el48OABqlatqjSDpLgys2W4GZuk2K7v5SRq8iaG0j5D0hNBALZsAT77TJ68mZkB48fLJyOUYCY1EZG+FPX9XZaxxa0UzM2Uk7SsHAGW5mUrcSMj8OgRMHq0vHsUkJf2WLsWaNZM3LiIiKjEOMatFAo2rgkw6cZLMjY5OcD//R9Qt648abOyAr79FrhwgUkbEZGRYotbKRTsFn2ZkgkPZxuRoiHK5+pVeYmP8HD5dtu2wM8/y5etIiIio8UWNy1KycgWOwQq69LTgenTgUaN5Embo6N88sGRI0zaiIhMAFvcULo1Ub3K2eDJK3kttNdZOdoKyWiY+NwW43L8uHy5qlu35Nt9+gDLlwOenuLGRUREWlOmW9wsLCwAAGklWFi+IBsL5bppsjKWyOQ+u9xnSSJITATGjJF3h966Bbi7A3/8AezcyaSNiMjElOkWN6lUCmdnZ8THxwMAbG1tS1zOIzs7B0J2pmI7+lkiXB1NvyyGIAhIS0tDfHw8nJ2di134l7Ts77+Bjz8Gnj6Vb48cCSxaBDg7ixoWERHpRplO3AAoFp7NTd40Ef8qb9moRHMzpDqIU01ZDM7Ozm9cvJd0ICYG+PRTYMcO+XaNGvLJB+3bixoWERHpVplP3CQSCTw8PODq6oqsrCyNrvHhzjCl7f+NCCwTs0stLCzY0qZvggCsWwd88QWQkABIpcCXX8onJNiY/t85IqKyrswnbrmkUqnGSci7zavh+4O3FdunHyVjoHs5bYVGJHfnDvDRR/IZogDQpIm8kK6/v7hxERGR3pTpyQna0sRHOUnbey2mkCOJNJCVBSxYADRoIE/abG2BJUuA06eZtBERlTFscdOCCnbKY9qO33kuUiRkci5ckBfSvXRJvt25s7wuW9WqooZFRETiYIubFtRyd4CdJcd6kRalpQGTJsmXprp0CShfHti4Edi/n0kbEVEZxsRNS/ZPaKu0nV4Gi/GSlhw8CNSrByxeDMhkwIABwI0bwJAhqgvkEhFRmcLETUsKFuLdGh4lUiRktF68AIYNA7p0AR48ALy9gX//BbZsAVxdxY6OiIgMABM3LbEwV36Us/+JFCkSMjqCAGzbBvj5ybtDJRLgk0+A69eB7t3Fjo6IiAwIJydoiZ2l6qNMz8qBtQXHvlERHj+Wr3zw77/ybT8/eYmPFi3EjYuIiAwSW9y0RGomQcT0zkr7vtnNVjcqhEwGrFghT9T+/RewtARmzwYuXmTSRkREhWLipkXl7SyVtv93huPcSI3r14HWrYFx44CUFKBVK/nM0Rkz5AkcERFRIZi46Vj0qzSxQyBDkZEBzJoFBATIi+c6OAA//QQcOwbUqSN2dEREZASYuOnYr2ceiR0CGYJTp4BGjeTdoVlZQM+e8pa3MWMAM/5vSERExcNvDC07/Hk7pe2fj90XKRIyCElJ8i7R1q2ByEh5WY/t24G//5aX+yAiIioBzirVMt+K9krbggAIggAJC6eWPf/+K29Ri46Wb3/wAfDdd/JVEIiIiDTAFjcd8HSyVtrecSFapEhIFHFxwHvvybtDo6MBX1/g0CHgl1+YtBERUakwcdOB95pVVtqetOMK4pPTRYqG9EYQgA0b5BMNfvsNkEqBL78Erl4FOnUSOzoiIjIBTNx0wKNAixsArD7KsW4m7f59oHNnYPhw4NUr+czR8HBg4ULA1lbs6IiIyEQwcdOBPgFeKvsevWBZEJOUnS1fDL5ePSA0FLC2BhYtkidtjRqJHR0REZkYJm46YC41w7CWPkr73J2sxAmGdOfaNSAwEJg0CXj9GujYUd4tOmkSYM55P0REpH1M3HRkVFtfpe2nCRzjZlL27ZMvTRURATg7A+vWyScgVK8udmRERGTCmLjpiKezDZxtLRTbh2/GixgNadUvvwA9esiXq+rQAbhxQz62jSVfiIhIx5i46ZCfh6PStiAIIkVCWiEIwPTpwIcfAjk5wODB8pY3d3exIyMiojKCiZsOuTooj2s7fue5SJFQqWVmAkOHAt98I9+ePh3YuJGLwhMRkV4xcdOhqd39UMM1byWFIevCRYyGNJaQAISEAL/+Kq/NtnYtMGcOu0aJiEjvmLjpUEUHK/w5tpXSvhVH7uLkXba8GY2oKPk6o0eOAPb2wO7dwIgRYkdFRERlFGsW6JiFVLlV5rv9twAADxd0FyMcKomLF4Hu3YGYGMDDA9izB2jYUOyoiIioDGOLm45ZSvmIjdK+fUDbtvKkrW5d4MwZJm1ERCQ6ZhU6JuE4KOOzdm1euY+OHYETJ4DKld98HhERkY4xcdODgqsokIESBGDaNGDkSHm5jyFDgL175QV2iYiIDAATNz2YFFxL7BDoTTIz5Ynat9/Kt6dPBzZsYLkPIiIyKEzc9MDOyhy/jWqutO/ak0SRoiEVueU+/vc/ebmPX35huQ8iIjJITNz0xMJc+VH3+PGESJGQEnXlPj74QOyoiIiI1GI5ED0J8HYWOwQqKH+5D09PedLGmaNERGTA2OKmJxKJBFUq2IodBuXauzev3Ee9eiz3QURERoGJmx6ter+x0vbzlAyRIinj1qwBevaUl/vo1Ele7sPbW+yoiIiI3oiJmx7ZWyn3TA9bz7VL9UoQgKlTgVGj8sp97NkDODmJHRkREVGxMHHTI+/yyl2l154kYd+1WKRlZosUURmSmQkMHgzMmyffnjGD5T6IiMjoMHHTs/XDmiptj/7fBUz+46pI0ZQRueU+Nm8GzM3l5T5mz2a5DyIiMjpM3PSsiU85lX27Lj8VIZIy4tEjoFUrlvsgIiKTwHIgeuZgbSF2CGVHRIS83EdsLMt9EBGRSWCLG5mm3HIfsbFA/fos90FERCaBiZsI2tasKHYIpu3nn+XlPlJT5eU+jh9nuQ8iIjIJTNxEMKqNr8q+HJkgQiQmJrfcx0cfyct9DB3Kch9ERGRSmLiJoHUNF5V9f196IkIkJiQjQ7ncx8yZwPr1LPdBREQmhZMTDMS9Zylih2C8Xr0C3n4bCAuTl/v4+Wdg+HCxoyIiItI6trgZiBVH7okdgnF69Aho3VqetDk4yGeOMmkjIiITxcRNJDs/bil2CMYvIgJo3hyIjAS8vOSTELp0ETsqIiIinWHiJhI/D0eVfZygUAJ79qiW+/D3FzsqIiIinWLiJhJrCymszJUf/7S/uPRVsfz8M9Crl7zcR1CQvKWtUiWxoyIiItI5Jm4i2v1pa6XtreGPRYrESKgr97F7N8t9EBFRmcFZpSKq7uogdgjGIyNDvsboli3y7Zkz5S8uFE9ERGUIW9xE1qp6BaXtqBdpIkViwF69AoKD5UmbuTmwbh0waxaTNiIiKnOYuIlscT/lAfXbzkWJFImBevQIaNUKOHpUXu5jzx6W+yAiojKLiZvIPJxslLZ/CmM9N4ULF+TlPm7ckJf7OHEC6NxZ7KiIiIhEI2riduzYMfTs2ROenp6QSCT466+/lD4XBAEzZsyAh4cHbGxsEBQUhDt37ogTLOnXnj1Au3bK5T4aNBA7KiIiIlGJmrilpqbC398fK1asUPv5okWL8MMPP2DVqlU4e/Ys7OzsEBwcjPT0dD1Hql+CUMbrubHcBxERkVqiJm5du3bFN998gz59+qh8JggCli1bhmnTpuGtt95CgwYNsGnTJjx9+lSlZc7YDW1RRWn7i9+viBSJyGQy4Ouv88p9DBsmb3ljuQ8iIiIABjzG7cGDB4iNjUVQUJBin5OTEwIDA3H69OlCz8vIyEBSUpLilZycrI9wS2VWr7poVrW8YvuPiGgkpmWJGJEIMjKAwYOB+fPl27NmyWePWliIGhYREZEhMdjELTY2FgDg5uamtN/NzU3xmTrz58+Hk5OT4uXn56fTOLVBIpHg0441lPb5zzlQdpK3guU+1q9njTYiIiI1DDZx09SUKVOQmJioeEVGRoodUrEEVHZW2XcpOkHvcejdw4fK5T727pV3kRIREWlgxYoV8PHxgbW1NQIDAxEeHl7osWvWrEGbNm1Qrlw5lCtXDkFBQSrHG9pESYNN3Nzd3QEAcXFxSvvj4uIUn6ljZWUFR0dHxcvBwThWJ7CzKoOLWKgr95Gva5yIiKgkfvvtN0ycOBEzZ85EREQE/P39ERwcjPj4eLXHh4WFYcCAAThy5AhOnz4Nb29vdOnSBU+ePFEcY2gTJQ02catatSrc3d0RGhqq2JeUlISzZ8+iRYsWIkamP0PXFf6vBKO3ezfQti0QFycv88FyH0REpEZycrLS2PWMjIxCj/3+++8xcuRIDB8+HH5+fli1ahVsbW2xbt06tcdv3rwZH3/8MRo2bIjatWtj7dq1kMlkitzDECdKipq4paSk4NKlS7h06RIA+YSES5cuISoqChKJBJ999hm++eYb7Nq1C1evXsWQIUPg6emJ3r17ixm2zrSp4aKyL/JpkgiR6Njq1fJyH2lp8oK6LPdBRESF8PPzUxq7Pj93ElsBmZmZuHDhgtKkRjMzMwQFBRU5qTG/tLQ0ZGVloXx5+YRBTSdK6pKo/XPnz59Hhw4dFNsTJ04EAAwdOhQbNmzAl19+idTUVIwaNQoJCQlo3bo19u3bB2tra7FC1qlNHzRD1Sl7lPZ9+cdl/PtJG5Ei0jKZDJg6FViwQL49fLg8iePMUSIiKkRkZCS8vLwU21ZWVmqPe/78OXJyctROarx582ax7vXVV1/B09NTkahpOlFSl0RN3Nq3b19ksVmJRII5c+Zgzpw5eoxKPBI1syhfpGSKEIkOZGTIE7WtW+Xbs2cD06dz5igRERXJwcEBjo6OOr/PggULsG3bNoSFhRl0A5HBjnEjuZjEdFx7kih2GKWTW+5j61Z5uY8NG4AZM5i0ERGR1ri4uEAqlZZ4UiMALF68GAsWLMCBAwfQIN94a00nSuoSEzcD878RgSr7evx4QoRItERduY+hQ8WOioiITIylpSUaN26sNKkxd6JBUZMaFy1ahLlz52Lfvn1o0qSJ0meGOFGyDNagMGyt1UxQMFoXLgDdu8tnjnp5yZev4sxRIiLSkYkTJ2Lo0KFo0qQJmjVrhmXLliE1NRXDhw8HAAwZMgReXl6KCQ4LFy7EjBkzsGXLFvj4+CjGrdnb28Pe3l5pomSNGjVQtWpVTJ8+XdSJkkzcDFD7WhURduuZ0r7E11lwsjGiQfy7dwP9+8tnjjZoIN/mzFEiItKhd999F8+ePcOMGTMQGxuLhg0bYt++fYrJBVFRUTAzy+tsXLlyJTIzM9G3b1+l68ycOROzZs0CAIObKCkRipodYAKio6Ph7e2Nx48fo5KRJA4L993EyrB7SvtsLKS4MTdEpIhKaNUqYOxY+SzSzp2BHTsAPQwsJSIi02GM39/6wDFuBkiqZtD+66wcESIpIZkMmDwZGDNG/n74cHlLG5M2IiIirWBXqQFyc1Rfo8agZWTI1xjdtk2+PWcOMG0aZ44SERFpEVvcDNC7TSvjnUZG1Cz88iXQpYs8acst98EabURERFrHxM0AWZqbYUl/f5X9BjkcMbfcx7Fj8i7RfftY7oOIiEhHmLgZkdn/RIodgrLz54HmzYGbN+UzRk+cADp1EjsqIiIik8XEzYDdn9dNaXvDqYc4e/+FSNEU8OyZPEmLiwP8/YEzZ4D69cWOioiIyKQxcTNgZmaqY8R+OfFAhEjU+OcfICkJqFNH3k2abwFgIiIi0g0mbkbmQGTcmw/Sh9275X++9x7LfRAREekJEzcD9/Pgxir7El9niRBJPpmZwMGD8vfduhV9LBEREWkNEzcD16Wuu8o+/9kHcCcuWYRo/nPyJJCcDLi6Ao0aiRcHERFRGcPEzUhtO/dYvJvv2SP/s2tXwIx/hYiIiPSF37pGoFpFO5V9WTkyESL5T+74tu7dxYuBiIioDGLiZgTWDm2qsm/3lRgRIgHw4AFw4wYglcoXkCciIiK9YeJmBKq6qLa4vUjNFCESAHv3yv9s1QpwdhYnBiIiojKKiZsRe5GSof+b5naTcjYpERGR3jFxMxJTu9VR2bfnWqx+g3j9Gjh8WP6e49uIiIj0jombkfigdVWVfcdvP9NvEGFhQHo64O0N1K2r33sTEREREzdjIVWz/JXeV1HILQPSrRsgUY2HiIiIdIuJmxHZO76Nyj5BEPRzc0FgGRAiIiItefLkiUbnMXEzInU8VNcErTplD07de677m9+6JS8FYmkJdOyo+/sRERGZoNjYWHzyySeoUaOGRuczcTMBA9ec1f1NcrtJ27cH7FTLkxAREZHcq1evMGDAALi4uMDT0xM//PADZDIZZsyYAV9fX5w7dw7r16/X6NrmWo6VdMzR2hxJ6dn6vzHLgBARERXL5MmTcerUKQwbNgz79+/HhAkTsG/fPpiZmeHw4cNo3ry5xtdmi5uROTixndr9154k6u6mSUnA8ePy9xzfRkREVKS9e/di/fr1WLx4Mf755x8IgoCGDRvi33//LVXSBjBxMzpujtZq9/f48QSydbV+aWgokJUF1KgBVK+um3sQERGZiKdPn6JOHXn9VR8fH1hbW+P999/XyrWZuJmQ7eejdXPh/GVAiIiIqEiCIMDcPG80mlQqhY2NjVauzTFuJiTqZZr2LyoIeYkbu0mJiIjeSBAEdOrUSZG8vX79Gj179oSlpaXScRERESW+NhM3E7Lr0hNM7lpbuxe9fBl4+hSwtQXattXutYmIiEzQzJkzlbbfeustrV2biZsRcrKxQOLrLJX9TxPTtX+z3Na2oCDAykr71yciIjIxBRM3beIYNyO0/7O2+L6/v35uxjIgREREWpWeno7FixdrdC4TNyPk7mSNtxtVUvuZTKbFJbBevADOnJG/Z+JGRERUbM+ePcO///6LAwcOICcnBwCQlZWF//u//4OPjw8WLFig0XXZVWpiLkS9QlOf8tq52IEDgEwG1K8PeHtr55pEREQm7sSJE+jRoweSkpIgkUjQpEkTrF+/Hr1794a5uTlmzZqFoUOHanRttriZmH6rTmvvYiwDQkREVGLTpk1Dt27dcOXKFUycOBHnzp1Dnz59MG/ePERGRmL06NEalwdh4mbEbswJQfcGHir7fSbvRlpmKZfFyskB9u6Vv2cZECIiomK7evUqpk2bhnr16mHOnDmQSCRYtGgR+vbtW+prM3EzYjaWUqwY2EjtZ1vORpXu4ufOyce4OTkBLVqU7lpERERlyKtXr+Di4gIAsLGxga2tLerVq6eVa3OMmwno5e+JXZefKu3bdPoRPmzjq/lFc7tJg4MBc/41ISIiKonIyEjExsYCkBfkvXXrFlJTU5WOadCgQYmvy29kEzD3rXoqiVupV1FgGRAiIiKNderUCYKQV+mhR48eAACJRAJBECCRSBSzTUuCiZsJcLK1ULt/46mHGNrSp+QXjIkBcpfh6NpV88CIiIjKoAcPHujs2kzcTMRHbX2x+th9pX0zd13XLHHbt0/+Z9OmgKtr6YMjIiIqQ6pUqaKzazNxMxFTutVRSdw0xm5SIiIijUVFFW+CYOXKlUt8bSZuJi4rRwYLaQkmD2dlyQvvAiwDQkREpIGqVasq3ueOc5NIJEr7OMaN8MOAAHy69aLSvhpT92LPp23g5+lYvIucPAkkJwMVKwKNG+sgSiIiItMmkUhQqVIlDBs2DD179oS5FqszsI6bCenl76l2f/cfjxf/IrllQLp2Bcz414OIiKikoqOjMWbMGGzbtg3du3fHr7/+CktLS/j7+yu9NMFv5jJAKMm68xzfRkREVCru7u746quvcPPmTezYsQOvXr1CYGAgmjdvjjVr1kAmk2l8bSZulOfhQyAyEpBKgS5dxI6GiIjI6LVu3Rq//PIL7ty5A1tbW4wePRoJCQkaX4+JG+XJXZu0ZUugXDlxYyEiIjIBp06dwocffoiaNWsiJSUFK1asgLOzs8bX4+QEEyM1kyBHpto3Gv7gJaRmQHVXBzjZqC/Yy25SIiKi0ouJicGmTZuwfv16vHr1CoMGDcLJkye1sl4pEzcT8+fHLfHx5ghEv3qttL//6tMAAE8na5ya0kn1xNevgcOH5e9ZBoSIiEhjlStXhpeXF4YOHYpevXrBwsICMpkMV65cUTpOk7VKJYJQoqHrRic6Ohre3t54/PgxKlWqJHY4euMzeXehnz1coCYx27dPPpO0UiUgKgrIV2+GiIhI34z5+9ssX1WG3PptBdMt1nGj0sktA9KtG5M2IiKiUjC4tUofP36sKC4HAOHh4diyZQv8/PwwatQorQZIeiAIHN9GRESkJbpcq1SjWaUDBw7EkSNHAACxsbHo3LkzwsPDMXXqVMyZM0erAZJmPu1YvfgH374N3L8PWFoCndSMfyMiIiKDoFHidu3aNTRr1gwAsH37dtSrVw+nTp3C5s2bsWHDBm3GRxqa0LlmoZ/N+SdSeUduN2m7doC9vQ6jIiIiotLQKHHLysqClZUVAODQoUPo1asXAKB27dqIiYnRXnSkMYlEglXvN1L72bqTDxD5NClvB7tJiYiIjIJGiVvdunWxatUqHD9+HAcPHkRISAgA4OnTp6hQoYJWAyTNhdTzKPSzbj8cx934FPmC8seOyXeyDAgREZFB0yhxW7hwIVavXo327dtjwIABioVSd+3apehCJcM35n8X8PyvPUBWFlC9OlCjhtghERERmZTs7GwcOnQIq1evRnJyMgB5Q1dKSopG19NoVmn79u3x/PlzJCUloVy+pZFGjRoFW1tbjQIh3Qiq44ZDN+LUfnYnPgUHNq3HQIDdpERERFr26NEjhISEICoqChkZGejcuTMcHBywcOFCZGRkYNWqVSW+pkYtbq9fv0ZGRoYiaXv06BGWLVuGW7duwdXVVZNLko6sGdIYk4Jrqf9QENDh3nn5eyZuREREWjV+/Hg0adIEr169go2NjWJ/nz59EBoaqtE1NUrc3nrrLWzatAkAkJCQgMDAQCxZsgS9e/fGypUrNQqEdEMikWBIC/X1ZOo8ewCPlBeAra18RikRERFpzfHjxzFt2jRYWloq7ffx8cGTJ080uqZGiVtERATatGkDANixYwfc3Nzw6NEjbNq0CT/88INGgZDuOFirX1Q+t7UtpXU7CP/NEiYiIiLtkMlkape1io6OhoODg0bX1ChxS0tLU9zwwIEDePvtt2FmZobmzZvj0aNHGgVC+pebuM1HVaw4clfkaIiIiExLly5dsGzZMsW2RCJBSkoKZs6ciW4aDlHSKHGrXr06/vrrLzx+/Bj79+9Hly5dAADx8fFwdHTUKBDSrYkFCvI6vU5Go6c3AQBHqjXBkoO3xQiLiIjIZC1ZsgQnT56En58f0tPTMXDgQEU36cKFCzW6pkazSmfMmIGBAwdiwoQJ6NixI1q0aAFA3voWEBCgUSCkW592qoHa7g4Y9esFAEDbBxGQCjLcdKmCp46ugCBygERERCamUqVKuHz5MrZt24YrV64gJSUFI0aMwKBBg5QmK5SERolb37590bp1a8TExChquAFAp06d0KdPH40CId3rUtdd8b7DfXk3aVi1JmKFQ0REZPLMzc3x/vvva+96mp7o7u4Od3d3REdHA5BnlSy+a/jWDWuCD9edRbv78pa3I755idvjl2nwLs86fERERJratWtXsY/NXTK0JDRK3GQyGb755hssWbJEUfnXwcEBn3/+OaZOnQozM42GzpEedKzthiVVs1DhdRKSrOxwwauO4rM2i47g8swucLJRPwuViIiIita7d+9iHSeRSNTOOH0TjTKsqVOnYvny5ViwYAEuXryIixcvYt68efjxxx8xffp0TS6pVk5ODqZPn46qVavCxsYG1apVw9y5cyEIHJBVGn1irwAAjvkEIFuqnLtHv0oTIyQiIiKTIJPJivXSJGkDNGxx27hxI9auXavUxNegQQN4eXnh448/xrfffqtRMAUtXLgQK1euxMaNG1G3bl2cP38ew4cPh5OTEz799FOt3KNM2r0bgPrxbVIzib6jISIiomLSqMXt5cuXqF27tsr+2rVr4+XLl6UOKtepU6fw1ltvoXv37vDx8UHfvn3RpUsXhIeHa+0eZU5sLHBBPr4tzLexysdn7r3A2M0RuBKdoOfAiIiITE9oaCh69OiBatWqoVq1aujRowcOHTqk8fU0Stz8/f2xfPlylf3Lly9HgwYNNA6moJYtWyI0NBS3b8trjF2+fBknTpxA165dCz0nIyMDSUlJildycrLW4jEJ+/bJ/2zSBM/tyql8POufSOy+GoNey0/qOTAiIiLT8tNPPyEkJAQODg4YP348xo8fD0dHR3Tr1g0rVqzQ6JoaJW6LFi3CunXr4OfnhxEjRmDEiBHw8/PDhg0bsHjxYo0CUWfy5Ml47733ULt2bVhYWCAgIACfffYZBg0aVOg58+fPh5OTk+Ll5+entXhMwn/dpFxUnoiITNGKFSvg4+MDa2trBAYGFtlLd/36dbzzzjvw8fGBRCJRWuUg16xZsyCRSJRe6nod1Zk3bx6WLl2KrVu34tNPP8Wnn36KLVu2YOnSpZg3b55GP59GiVu7du1w+/Zt9OnTBwkJCUhISMDbb7+N69ev49dff9UoEHW2b9+OzZs3Y8uWLYiIiMDGjRuxePFibNy4sdBzpkyZgsTERMUrMjJSa/EYvaws4MAB+ftu3bC4n3/RxxMRERmR3377DRMnTsTMmTMREREBf39/BAcHIz4+Xu3xaWlp8PX1xYIFC+Du7q72GACoW7cuYmJiFK8TJ04UK56EhASEhISo7O/SpQsSExOL90MVoHEdN09PT5VJCJcvX8Yvv/yCn3/+WdPLKpk0aZKi1Q0A6tevj0ePHmH+/PkYOnSo2nOsrKxglW/B9KSkJK3EYhJOnQKSkoCKFYGmTeF694XYERERERUpOTlZ6bu84Pd8ft9//z1GjhyJ4cOHAwBWrVqF3bt3Y926dZg8ebLK8U2bNkXTpk0BQO3nuczNzYtM7ArTq1cv/Pnnn5g0aZLS/r///hs9evQo8fWAUiRu+pCWlqZSE04qlUImk4kUkZHbs0f+Z0gIUIxae1k5MlhIWZOPiIjEU3DI08yZMzFr1iyV4zIzM3HhwgVMmTJFsc/MzAxBQUE4ffp0qWK4c+cOPD09YW1tjRYtWmD+/PmoXLlysWL/9ttvERYWplge9MyZMzh58iQ+//xz/PDDD4pji1stw6ATt549e+Lbb79F5cqVUbduXVy8eBHff/89PvjgA7FDM04lHN9Wc9pefBlcG2PaV9NhUERERIWLjIyEl5eXYruw1rbnz58jJycHbm5uSvvd3Nxw8+ZNje8fGBiIDRs2oFatWoiJicHs2bPRpk0bXLt2DQ4ODkWe+8svv6BcuXKIjIxUGrrl7OyMX375RbEtkUhMI3HLLej78ccfIz4+Hp6envjoo48wY8YMsUMzPo8eAdevy1vagoMBAG6O1kWeIgjAwn03mbgREZFoHBwc4OjoKNr981eyaNCgAQIDA1GlShVs374dI0aMKPLcBw8eaD2eEiVub7/9dpGfJyQklCYWFQ4ODli2bJnaWR5UQnv3yv9s2RIoJy8DUsu96H8p5MrMlsHSnF2mRERkuFxcXCCVShEXF6e0Py4uTqPxaYVxdnZGzZo1cffuXa1dsyRKlLg5OTm98fMhQ4aUKiDSkUK6Sf/9pDV6/HgCrg5WiE/OUHvq+G0XsfJ91WK9REREhsLS0hKNGzdGaGioYr1QmUyG0NBQjBs3Tmv3SUlJwb179zB48OA3HisIAnbs2IEjR44gPj5eZYz+zp07S3z/EiVu69evL/ENyACkpwOhofL3BRK3el5OuDi9M56lZKDL0mNqT997LVbXERIREZXaxIkTMXToUDRp0gTNmjXDsmXLkJqaqphlOmTIEHh5eWH+/PkA5BMacseeZWZm4smTJ7h06RLs7e1RvXp1AMAXX3yBnj17okqVKnj69ClmzpwJqVSKAQMGvDGezz77DKtXr0aHDh3g5uYGiaT0y0oa9Bg30pKjR4HXrwEvL0DNyhbl7CzhZGNR5CUm/nYJI9v6oo6HeOMMiIiIivLuu+/i2bNnmDFjBmJjY9GwYUPs27dPMWEhKipKqVrF06dPERAQoNhevHgxFi9ejHbt2iEsLAwAEB0djQEDBuDFixeoWLEiWrdujTNnzqBixYpvjOfXX3/Fzp070U2LRe+ZuJUFuWVAunUDCsn2zcwk6NHAA/9eiVH7+c6LT7Dz4hM8XNBdV1ESERGV2rhx4wrtGs1NxnL5+PhAEIQir7dt2zaNY3FycoKvr6/G56vDEeemThCKXQZkZs+6qP2GCQsrjtzFhpPanyVDRERkambNmoXZs2fj9evXWrsmW9xM3Z07wL17gIUFEBRU5KEVHayw77O28Jm8u9Bjvtt/CwAwtKWPVvrqiYiITFX//v2xdetWuLq6wsfHBxYWysOSIiIiSnxNJm6mLrebtF07wN6+WKfc/bYrev90EteeFL5cmCAU2utKREREAIYOHYoLFy7g/fff5+QEKqYSrpYAAOZSM/z7SZsiW95kggAzMHMjIiIqzO7du7F//360bt1aa9fkGDdTlpIin1EKlChxK46bsclavR4REZGp8fb21vqqD0zcTFloKJCVBVSrBtSsqdVL9/jxBLacjYLP5N345/JTrV6biIjIFCxZsgRffvklHj58qLVrsqvUlBWjDEhpfP3nVQDAJ1svoqe/p9avT0REZMzef/99pKWloVq1arC1tVWZnPDy5csSX5OJm6kSBOXEjYiIiPRKF2utM3EzVVevAtHRgI0N0L69RpcIquOGQzfi3nwggOwcGcyl7HknIiLKNXToUK1fk9+0piq3ta1TJ8DaWqNLLB8YgHebeBfr2Ixs2ZsPIiIiKqPS09ORlJSk9NIEEzdTpUEZkIKsLaRY2LdBsZa5Ov/oFZ4kaK8yNBERkbFLTU3FuHHj4OrqCjs7O5QrV07ppQkmbqbo1Svg1Cn5ez2Nbxu6LhytFhzWy72IiIiMwZdffonDhw9j5cqVsLKywtq1azF79mx4enpi06ZNGl2TY9xM0YEDgEwG1K0LVKkidjRERERl0j///INNmzahffv2GD58ONq0aYPq1aujSpUq2Lx5MwYNGlTia7LFzRRpoZu0oI/bV9PatYiIiMqCly9fwtfXFwDg6OioKP/RunVrHDt2TKNrMnEzNTIZsHev/L0WE7cvQ2rj+uxgVHWxK/K4i1GvIJMJWrsvERGRsfL19cWDBw8AALVr18b27dsByFvinJ2dNbomEzdTc/488Pw54OgItGql1UvbWZnj8Oftijymz0+n4Pv1Hny3/6ZW701ERGRshg8fjsuXLwMAJk+ejBUrVsDa2hoTJkzApEmTNLomx7iZmtwyIF26AAUqNGuDpJgrMKw4cg9d/Nzh7+2s2JedI0P0q9fweUOrHRERkSmYMGGC4n1QUBBu3LiBiIgIVK9eHQ0aNNDomkzcTI0OxrcVZG4mQXYxukPDH7xUStw+3HQeYbeeYcXARujewENn8RERERkiHx8f+Pj4lOoa7Co1JXFx8q5SAOjaVWe3qe5qX6zjcgR5cvcsOQN34pIRdusZAGDdyQc6i42IiEhsp0+fxr///qu0b9OmTahatSpcXV0xatQoZGRkaHRtJm6mZN8++Z+NGwPu7jq7zerBjdG9vge+61t0M+/1p0nIzJah6beH0Hlp3uwZ7S93T0REZDjmzJmD69evK7avXr2KESNGICgoCJMnT8Y///yD+fPna3RtJm6mRA/dpABQpYIdVgxqhH5vWA7rn8tPUXPaXp3GQkREZGguXbqETp06Kba3bduGwMBArFmzBhMnTsQPP/ygmGFaUkzcTEVWlrzwLqC31RI0Vcz5DUREREbp1atXcHNzU2wfPXoUXfMNYWratCkeP36s0bWZuJmK06eBxETAxQVo2lRvt7WUlvyvkISdpUREZMLc3NwU9dsyMzMRERGB5s2bKz5PTk6GhYaVH5i4mYrcMiAhIYBUqrfbXprZWW/3IiIiMgbdunXD5MmTcfz4cUyZMgW2trZo06aN4vMrV66gWjXNViRi4mYq9DS+rSBbS3Pc/qaEM1jZ4EZERCZs7ty5MDc3R7t27bBmzRqsWbMGlpaWis/XrVuHLl26aHRt1nEzBVFRwLVrgJkZEBys99tbmpsVu7Zbftk5Mphr0NVKRERkyFxcXHDs2DEkJibC3t4e0gI9Yb///jvs7YtXWqsgfmuagty1SVu0AMqXFyWEXeNaF/tYCYB+q06h+tS9iHyapLugiIiIROTk5KSStAFA+fLllVrgSoKJmykQqZs0Pz9PR/w8uHGxjj338CXOPXwFAJi/94YuwyIiIjIpTNyMXXo6EBoqfy9yGZAudd2LtapC/h7VpNdZkJWwi5WIiKisYuJm7I4dA9LSAE9PwN9f7Gjw99hWJTr+cnQihqwL11E0REREpoWJm7HLLQPSrZtBVLa10GCywYm7z3UQCRERkelh4mbsDGB8W36a5o4vUjRbbJeIiKgsYeJmzO7cAe7eBSwsgKAgsaMBoFyi7b2mRa9lml/jbw4hO0em/YCIiIhMCBM3Y5bbTdq2LeDgIG4s/5Hka3Ib0boqjn/ZodjnVp+6F2mZ2boIi4iIyCQwcTNmBtZNCgBmEqChtzOqutihqosdvMvbluj89Scf6iYwIiIiE8CVE4xVSgpw9Kj8vQElbhKJBH9+3BIyAZCalXzA23f7b+HUvedo6O2MzzvXgpkG1yAiIjJVTNyM1eHDQGYmULUqUKuW2NEokUgkkJYi3zp59wVO3n2BO3EpaFPDBYNb+GgtNiIiImPGrlJjldtN2r27QZQB0YUDkXGY/vd1RL9KEzsUIiIig8DEzRgJgnL9NhOXnM4JC0RERAATN+N07RoQHQ3Y2ADt24sdTbG1qeGi0Xkm2qBIRERUYkzcjFFua1vHjvLkzcB907sevJxtMLtXXY3O/2rHFSSnZ2k5KiIiIuPDxM0YGWAZkKK837wKTk7uCN+K9viub4MSn385OhH9Vp1GqwWH8eWOy0jPylH6PCtHhozsnELOJiIiMh1M3IzNq1fAqVPy9127ihuLBvo1Kf5qCvndjE3Gk4TX2H4+Gk2/PaTYLwgC2iw8gsB5ocjiygtERGTimLgZm4MHgZwcoE4deSmQMij/ZIXXWTmITUpHQloWYhPTRYyKiIhI95i4GZv8ZUCM1N7xbQAA5loorpsjExTvNSn4S0REZEyYuBkTmQzYu1f+3kjGt6lTx8MRDxd0x915mv8MuWPamLgREVFZwsTNmFy4ADx7Jl9QvlUrsaMRVditZ/CZvBtbwqMU+5i2ERGRqWPiZkxyy4B06QJYWoobi8g++vUCAGDRvluKfTJBPlmBiIjIVHGtUmNiZGVAiuPAhLaIT8pA6xouaDz3IF6kZmp8rebzQwEAN+eGwNpCqq0QiYiIDAZb3IxFXBxw7pz8fUiIuLFoUU03B7T+b0WF41910Mo1t59/rJXrEBERGRombsZi/375nwEBgKenuLHoiK2lOcZ2qAYA8HCy1vg6rzNZjJeIiEwTEzdjYQJlQIrjk441sLifP/4eq/nki8iYJJy8+1yLURERERkGjnEzBtnZeS1uJjS+TR1rCyn6Nq5Uqmv8fekp/r70FADwaaca+KxTDZixVAgREZkAtrgZg9OngcREoEIFoFkzsaMxKj+E3sHuqzFih0FERKQVTNyMQW4ZkJAQQMrZkiX14Hmq2CEQERFpBRM3Y2CCZUBKqoufm8bnfn/wNmISX2sxGiIiInEwcTN0jx8DV68CEgkQHCx2NHo1KbgWAKC5b3n8PKRJqa7VYv5h/N+hO0r7BEHAiTvP8bIUteOIiIj0iZMTDF3u2qTNm8vHuJUho9tVQ+Mq5dCgkpNWrrf00G0sPXQbAHBwQlu8/8tZxCVlwMXeCuenBWnlHkRERLrEFjdDV0bKgKgjNZOguW8F2FrK/32x6J0GWrt256XHEJeUAQB4npKh8nni6yxEPk3S2v2IiIi0gYmbIcvIAA4dkr8vw+PbcvVv6o3Vgxvr5V5tFh5Gtx+O4/zDl4p9v52LwvLDd4o4i4iISLeYuBmyY8eAtDTAwwNo2FDsaAxCcF13XJ8djPa1Kmr1un1XnoLP5N3YGh4FAEhKzwYAhN6MVxzz1R9XsfjAbdyJS9bqvYmIiIqLiZshyy0D0q2bfHICAQDsrMyxYbh269mdf/QKADBl51WVz47cjMfn2y8rtpPSs7R6byIiouLi5ARDxjIgBmH4hnNK24IgUiBERFTmscXNUN25I3+ZmwNBnPGoTzJZ0ZkZ8zYiIhILEzdDlVsGpE0bwNFR3FjKGN+v9xT5OVvciIhILEzcDFUZLgNiSFaG3VPZJzBzIyIikRh84vbkyRO8//77qFChAmxsbFC/fn2cP39e7LB0KzUVCAuTv+f4NiIiIvqPQSdur169QqtWrWBhYYG9e/ciMjISS5YsQbly5cQOTbcOHwYyMwEfH6B2bbGjMVi7P22ttO3pZK2X+7K9jYiIxGLQidvChQvh7e2N9evXo1mzZqhatSq6dOmCatWqiR2abuXvJmUZkELV9XRCc9/yiu1TUzrh3rxuqO3uoHRc13ruWr1vQloWzt5/oegyvROXjD4/ncTR28+0eh8iIqKCDDpx27VrF5o0aYJ+/frB1dUVAQEBWLNmTZHnZGRkICkpSfFKTjayYqmCoFy/jYq0pH9DBNVxw7ZRzQHIl8naNKIZpvfwUxxjb2WOhwu0N1Zw9P8u4N2fz+DfKzFYe/w+Oi89hotRCRi6Llxr9yAiIs2sWLECPj4+sLa2RmBgIMLDC//dfP36dbzzzjvw8fGBRCLBsmXLSn1NXTPoxO3+/ftYuXIlatSogf3792PMmDH49NNPsXHjxkLPmT9/PpycnBQvPz+/Qo81SNevA48fA9bWQPv2Ykdj8LycbbB2aBM0962g2OfqYI0Rrasqtj101IW65vh9fLP7hk6uTUREJffbb79h4sSJmDlzJiIiIuDv74/g4GDEx8erPT4tLQ2+vr5YsGAB3N3V986U9Jq6JhEMeIqcpaUlmjRpglOnTin2ffrppzh37hxOnz6t9pyMjAxkZOQtGv7kyRP4+fnh8ePHqFSpks5jLrVFi4CvvgK6ds1reSONHLgei73XYvFtn3qwtTSHz+TdOr+nmQSQCYB/JSd8/25DVKtor/N7EhGZoujoaHh7eyMyMhJeXl6K/VZWVrCyslJ7TmBgIJo2bYrly5cDAGQyGby9vfHJJ59g8uTJRd7Px8cHn332GT777DOtXVMXDLrFzcPDQ6XFrE6dOoiKiir0HCsrKzg6OipeDg4OhR5rkFgGRGu61HXH0ncbwtZSdYGQ/40I1Mk9c2v3Xo5OxPhtFyEIAu7Gp7yxqC8REann5+en1JM2f/58tcdlZmbiwoULCMpXtN7MzAxBQUGFNva8iS6uWVoGnbi1atUKt27dUtp3+/ZtVKlSRaSIdCwhATh5Uv6+a1dRQzF1rWu4YFr3Ojq9x7UnSVhy4DaCvj+Khftu6vReRESmKjIyEomJiYrXlClT1B73/Plz5OTkwM3NTWm/m5sbYmNjNbq3Lq5ZWgaduE2YMAFnzpzBvHnzcPfuXWzZsgU///wzxo4dK3ZounHwIJCTIy8B4usrdjQm78M2un/Gy4/cBQCsPnafhXuJiDTg4OCg1JNWWDdpWWHQiVvTpk3x559/YuvWrahXrx7mzp2LZcuWYdCgQWKHphvsJjVpP6lZhYGIiLTDxcUFUqkUcXFxSvvj4uIKnXggxjVLy6ATNwDo0aMHrl69ivT0dNy4cQMjR44UOyTdkMny1idlGRCd6FjbFQDQvb6HYp+jter4N135bv+tQlvdBEHA45dpeouFiMjUWFpaonHjxggNDVXsk8lkCA0NRYsWLQzmmqVl8IlbmRERAcTHA/b2QOvWbz6eSuz/3muIHwYEYFHfBop9v49uieGtfHBmSie9xHD8znMAwJn7LzBx+yXEJqYDAKb9dQ1tFh3B5rOP9BIHEZEpmjhxItasWYONGzfixo0bGDNmDFJTUzF8+HAAwJAhQ5TGyGVmZuLSpUu4dOkSMjMz8eTJE1y6dAl3794t9jX1TX/NDVS03NIfnTsDlpbixmKiHKwt0MvfU2lfLXcHzOxZV28xDFkXjhqu9rgTnwIA2BnxBOemBmHzWflM6al/XsMvxx/g4w7V0bexEZSvISIyIO+++y6ePXuGGTNmIDY2Fg0bNsS+ffsUkwuioqJgZpbXZvX06VMEBAQothcvXozFixejXbt2CPtvzfA3XVPfDLqOmzbk1oEx+DpugYFAeDiwdi0wYoTY0ZRJf118gs9/v4wcAyndoc3VHoiIjI3RfH/rGVvcDEF8PHDunPw9y4CIpneAF3oHeCE+KR09l59AXFLGm08iIiLSI45xMwT798vXKG3YEPD0fOPhpFuujtY4/mVHXJsdLHYoRERESpi4GQKWATE4luZmsLcyR213+cobk4JriRrPsdvPMGvXdWRk5xR5XHJ6Fg5Gxr3xOCIiMk7sKhVbdra8xQ1gGRAD9NuoFoiIeoU2NVzw3f5bbz5Bix69SEWVCnaQyQQMWRcOANhw6iG6+LmhcnlbfN6lFgQIOPfwFZ4lZ6Bv40r4cON5nH3wEh+0qooZPf3ecAciIjI2TNzEduaMfKmr8uXlExTIoDjZWqDDf/Xf9K3dd2Fq9x+IlBeClAnAupMPFPvrejri7IOXAIDfLzxm4kZEZILYVSq23DIgwcGAVCpuLFQqwXX1OzX8nytPlbafJrxWvDeTSPQaCxER6QcTN7FxfJvRsDTP+9/F3CwvMarpZo9POlbHioGN9BpPwUI++beZtxERmSZ2lYopOhq4ckX+LRvMGYwGL19idHNuCMylyv/u0XdJxOcpyuVKcvLdPyEtS+mzs/df4HVWDtrXEqfbl4iItIMtbmLKXZs0MBBwcRE3FnqjsR2qAwD6BHipJG0AIBG5mevcf+PbciW+zsK/V55iz9UYvPvzGQxbf04l2cvPxGtxExGZBLa4iYndpEblk47V0amOK2r9VyJEnQp2lniRmqnHqPKsPfFAadt/9gGVY16mZsLF3kpl/7UniXj/l7P4vEstDG5eRWcxEhFR6XDJK7FkZAAVKgCpqcCFC0Aj/Y6PIt3IyM5BVo6ArGwZol6mYc3x+zh2+xmS0rPFDg0AUNXFDnvHt4G1hfJEmC5Lj+J2nHz9VC61RUSGwGC/v0XGFjexHD8uT9rc3eUrJpBJsDKXwsocgBVQzs4SPw4IQLZMwLmHLzFwzVmxw8OD56moPX0fXOytEFDZGT8OCIC1hRT5l2cVBAGLD9yCr4s93iliofv0rBw8fpmGGm6Ft0ASEZF2cYybWHLLgHTtCpjxP4OpkkgksJCaoa6Hk9ihKHmekoGDkXHYfv4xAPmKC7nOPXyFFUfu4fPfLxd5jXd/PoPOS49h37VYncZKRER5mDGIhePbyhQBhjki4bdz8sQtLilv0sKLfBMYPtl6UbF81h8XotH+uyO4G58MALj8OAEAFMkfERHpHhM3Mdy9C9y+DZibA0FBYkdDelBwJGntIiY46NP1p0kq+/J3m/5z+akiufv898t4+CINX/1xtcDxhpmUEhGZIiZuYsgtA9K6NeBkWF1opBtm+Qr2/jSoEfZ91hZvB3iJGFGetEzliRNjt0Qobc/4+zr+vvREsV1wAXtBkI+LG7jmDD7YcI5lRYiIdIiTE8SQ203KReXLDCcbC7zfvDJyZEC3+h4AgDm96yGgSjlM/+uaqLENWvvmSRPjt11SvL/2JAmJr/PGxMkEAauO3sepey8AyLtf32tWWetxEhERy4HoX2qqvAxIRgZw/Trgx4XAy7pdl5/i060XMbyVD9affCh2OCVWztYCrwqs1PCmkiKxien47dxjDAysjIoOqnXliIgM7vvbQLDFTd+OHJEnbVWqAHXqiB0NGYBe/p5oXd0F5WwtkJaRg9/OP0afAC/8efHJm082AAWTtuIY/MtZ3IlPwfE7z7BjTEsdREVEZJo4xk3f8neTciVw+k95O0tIJBLMf7s+Dn/eDt/394etpXKRXH9vZ3GC08D5h8rLb+2MiMaGk3krO9yJlxf7Pf/olWJfRnYOcmQm3QFARFRqbHHTJ0HIq9/GMiCkhpmZBL4V7QEARyd1wJFb8fhyxxUAgIWZ8ST6wzecQ8T0zkh6nYWeP57A08R0AEDnuu44dfe5yvFpmdloNPcgqrrYY+/4NvoOl4jIaLDFTZ8iI4GoKMDKCujQQexoyMBVdLBC/ybeiu3GPuVEjKZkktOz0XLBYTT+5pAiaQOAO3HJmPRfIprr3rMURDxKQHqWDDdiVMuTqPPnxWgcvf1MqzETERkDJm76lNva1qEDYGsrbixkdN5rWhkrBhrPmrbPkjNU9g1bf05l38TfLuHa00TF9pOE1xix4RxO3ZO3zB2KjMOsXdeRlSMDADx6kYoJv13G0HXhimLAxWXic7GIqAxgV6k+sQwIaWD5wAC8SMlEVRc7VHWxw9gt8v39GlfC7xeixQ1OCy5HJ+JydF7i9vXOqzh6+xlCb8bDwdocyenyOnPVXO0xuHkVPE/JVBwb9P0x/DaqOQJ9K6hcN0cmQJqve/mfy08xa9d1LO7vjw61XHX4ExER6Q5b3PQlMRE4cUL+nuPbqAR6NPDE0JY+KvvfauiFi9M7F3peT39PzH2rLi7P6KLD6LQvPl9LXW7SBgBPE14jPSsHcUnpSsfvvhoDmUxQWm816kUa/GcfwMJ9NxX7Ptl6ES9SMzF8/TlsP8dluojIOLHFTV8OHgRycoBatQBfX7GjIRNgJgHK2Vmq/Wzv+Dao5eagtGKDsShsnNvKsHtYGXZPZX+2TMDAtWdw5v5LTOteBx+28cXSQ7eRkpGNlWH38FVIbZVzvvzjCvo39VbaF5+UDolEorauXEziazjZWMDWkr8yiUhcbHHTF3aTkpY09y0PVwcrNKqiOlnhwrQgnJsahDoejkpJW4daFRXvLaWm9b+9TCbgzH15+ZFvdt8AACSkZRZ1CgDg6O1neJkqPy49KwfN5oWi6beHkP3fWLpcj1+mocX8w2i54LCWIyciKjn+81EfZLK89UnZTUqltHVkc+TIBJirScAq2KtfhSB/K9Ktb0LQYNYBJGdkqz3W2DxJeK20PWLDORy59eYZp0PXhaOigxXOTQ3CLyfyasylZeXAUWqGPVdjIAhQLO+VUMxCw3FJ6XiekoG6nlyHmIi0z7T+6W2oLl4E4uIAe3v5wvJEpSCRSNQmbSU5f+9nbfB1t9q4Msu4xr+pc/yOcl240JvxSts+k3cjtZAkNXfm66EbcYp9giCvK/fx5giM3RKBpHxj5zKyc5CcnoWPN1/Avmsxaq8ZOC8U3X84gfvPUtR+/uhFKrosPYo/TGBiCRHpHxM3fcgtAxIUJK/hRqRFf49tBX9vZ/w+ukWhxxSsglGpnC1Gta0GR2sL/PtJawRWLY92NSticPMqOo5WHAPXnCn+wQLwOjNHsZmSb4LEvfhU/Hj4LvZcjcXo/0UUeZkr+WbK5jftr2u4HZeCz3+/XPyYiIj+w65SfeD4NtIhf29n/D22VZHHFFW9rJ6XE377KC/pm9u7HlYdvYcFe28WcZZxuVxIEgUAJ+48x8WoBMV2Rk4OQpYdV2xvOxeleP/oRSpi8xUU/jH0Dvo18Ya7k7XKdQtb0S5/UkhEVFJscdO1Z8+A8HD5eyZuJJLK5UtW8Hl0u2p4ML9s/H19/5ezStutFx5RTFoAoFQ3bszmCOy6/FSxveTgbQxZdxavUjMxZedVRES9UrrW3qsxKuu2aoOsFGu6ZmbLcP9ZCracjVKZiEFEho8tbrq2f7+8n8rfH/DyEjsaKqNGtfXFy9RMBNVxK/Y5EokEbzfyws6IJzqMzPBkZpcsmbkdl4KAuQcBAFvD81rnVh29ryht8knH6hjeqipsLaWFtsTlt/3cY/xz5Sl+GtQIDtYWAOQzXz///TLOPXgJmSBgw/BmsDI3Qw03B5XzLz9OwA+hdzClW21Ud837/GBkHEZuOq/YzsqRqa0RSESGiy1uusZuUjIA1hZSzOpVF61ruGjlegGVnbVyHVOWvx7dj4fvotHcgwiYc1BlvKE6X/5xBcfvPMfqo/cRn5yO6Fdp6LvqFHZfiUF8cgaep2Six48n0HnpMTx6kYr0rBxcfpyAjOwcTPr9Mt5acRKhN+PxwYbzStf9ePMFpe0Lj/JaCGMSX2PUpvM4fe9FkbGlZWbji98vIzTfhI6irAy7h6HrwkucEBORemxx06XsbHmLG8AyIGSU6rg7AlBtcdv+UQt8vfOqSSy5pU+vs3JwPl+y9O+Vp3iWnIG/Lj6BlbkUzatVQPt8NfeWH7mL5UfuFnnNdt+FKd7XdnfAzdi89VujXqYVeW7+1r9Jv1/BibvPcSAyDjfmhCA1MxsV7CwhKdBEuCrsHnZciMaOC9F4uODNv9dyV6/Ydfkp+jau9MbjC7oTl4wv/7iCz4Jqol3Nim8+gcjEMXHTpbNngVevgHLlgMBAsaMhKrGhLX2QkZ2DpPRs/HzsPgD55AULqRm+6+ePC49e4f7zVADA8FY++N+ZR8jK4ULuxTVuy0Wl7fCHL/FD6B2Nr5c/aQNUJ0gU/G8jgbzu3KEbcbgdl3dunRn7AADvNKqEJf39kfg6C2fuv0D7WhURk6i85FhxpWdpNinj480RuBOfgqHrwouVKBqTFykZmP1PJN5r5o2W1bTTGk6mj12lupRbBiQ4GDBnjkzGx9LcDOM61sDX3eoo9lnkX0Yr39uZPevi9jdd8XYjL4zvVKPI67o7qs7CJO3Ln7fdiUtW/Vwiwds/ncLUP68prRGb648IeYvqez+fwUe/XkCtafuKnKFcFLPiDO5T41UxVsEozJGb8Wi14DBO3Xv+5oNFMOffSOy6/BQD15x988FE/2Hipksc30Ym5O1GXvBwskYPf89Cj5FIJPi+f0NM6FyzyGuNavvm9XonvuEa9GYyAWg09yCWH76DL3ZcUfn8z4tPVFaeKCjxdZbSeD1ZcQbpqZEjkyFHzWxYQRAQ/uAlnqdkQBAEpXj+uBCtNKu3JJ4lZ2D4hnN4kvDaYBOjx2/oyiZSh4mbrjx5Aly+LO+rCAkROxqiUvu+f0Oc/Koj7K2K13r8ScfqcLa1QHBd1ZmsPf9L/toWMWbJycZCs0BJycvUTCw+cBuXHydodL7/7ANK2/lnGWdkF7/7c/rf11Ht6z0QCiR+Ybeeof/q02i98DDm772JVgsOY+1xebd8aYoUd156VONzi+NufApO3TXMljwybUzcdCV3bdJmzYCKHFBLpiH/wvUAUNNVtRRFrs+71ELEtM7wLpdXQ65RZWdM614HFR2scHNuCDYOb6p0Tvf6Hor3Bb/gyfB8t+8Whq8Px5az8jIogiDgRUoGwm7F4+PNF5Tq4eXK3+qWkJaJaX9dAwCkZ8kU4yi/2X2jyPsKgoC4JOWxdhnZOYhJzGutK7i27Jv+Pq0+eg8+k3fjXiFLlRUU9P1RDFx7FrfjkrHvWixuxap2Rb+JKfwNFwSBRaX1jAOvdIXdpFQGfNunHsrZWeK9pt5qPy+Y6O38OG+FB2sLKQB5y9yPh+/i2uxgXI1OxO6r8jVApWaajYki/Vl74gEA4MitZ8jIzsGD56nYdPqR4vNTakqLdFxyFP2bVMLAwCpo9F/9O3W+P3hbZd/zlAxYmpvhh0N3sPbEA8zrUx/vNfWGmZkEXf/vOO4/S8W+z9qgtrujyrkBcw9ickhttKruAu//ClK/TM3E6F8voG+TSpj/30ohnZYcxaTgWvj9/GPMf7sBWlSrUOQz2HjqITb/l7i2ql4BQ1v4IKByObyz8hT6Na6ET/KN9zx6+xkm/X4Zi/o2QPtarmpLwzxLzoCLfd5s3tSMbNj918p97uFLnL3/AmPaV9fa/x+CICBbJsBCw/WPv/rjCrafj8buT1ujrqeTVmKiorHFTRcyMoBDh+TvWQaETFgFeyvMf7s+/L2dCz1GKi36C+bzLrXwcEF32FuZw8E679+SfRpVQpUKtnivqTfebuQFBytz/DGmpdK5/pX4RWEoZv8TqZS0AaqtXoC8RMniA7cR/qDoFSXUza5t8s0hNJh1QJEwfv3nVTT+5iAev0zD/Wfy2c17rsTgyK14lXMT0rIweedVtFl0RLHv+4O3EP7wJb4sMP7vu/238PBFGgaoWeM2NjFdqfXu2tO88X8n777AqF8v4Kewu4h6mYYl/yWfC/fdRL9VpzB0XTjikzMwbP05AKotbr+eeYSm3x7C0kPyn73Hj8dRd+Z+xfJz/VadxuIDt7HjwmO1z2xnRDQ+334ZWSVYEWPkpvNoNOcgktJV/1sduB6LM/fV1/WTyQQcvhmH7eflE1hWHb1f7HtS6bDFTRdOnABSUgA3NyAgQOxoiET1Udtq2Hs1Fr0D3rxySF1PRwxv5QNPJxvYW5kj7Iv2ipaH7BwZzAu0Cvw9rjX6/HRSaa3RXP/3XkOM33ZJGz8C6cDn2y9p5Tqv0rKUkrEfDhdd9w4Aol6k4ds9kdh/vXhFhHP936E7WHroNizy/WNE3djB7HxlVyb8dgl/Xize6iPT/+s2/iH0Dj5o5YNrT+RJ4aqj9zC5a23Fcff+S1IB4EnCa/zfodv4oHVVTNwuHxPYrGo5vNu0cqH3yc6R4ds9N9CqmgsO3ZAnuQeuxynV2Xua8BqjfpUXbC5YhiU7R4YPN51H2K1nin1sINcfJm66kNtN2rUrYMZGTSrbyttZ4tiXHYp1rEQiwcyedZW2cxVM2nI5FzKJoaqLHXo39MTR28+QkS3DxM418U6jSorlqUhcqSKOi2r73ZE3H/Sfxy/TsPb4fbSv7Yqlh+QtaG+qVfjrmbyWx8KStqcFZvO2WXRYabvg88nfypf//cebI3D5cQL+upi3hq66ls78fr8QjfUnH2L9yYeFHvNMTXmYXL+ceKCUtAHKpWdIt5i46UJu/TaObyPSua+61saRAl8igHyJ4GXvyVu8c2SCYkyQpbkZl1+iYhu6Lhz3n6diY4Fu4NJqvfAw6nvldfU/fqmcyLVaoJzI5R/zl78n9PqTRABAZr6dbxr/tvHUQ5V9X/x+GZ5O1mhZXV4I+KewvJZLQRCU/hG173qsyvkFV9gg3WFzkLbduwfcugVIpUCXLmJHQ2RyOtZ2BQC0/G/QeG13R6wvMDu1oPxfZA7FLGdCBECxMoi2yQT5q7h+zNcFvP96LLafl49zy1ZzkYuPEyAIAk7ceY7nKfKWs36rTqHO9H0If/BSZYWNXAPX5tW7y9+NnDsT+KNfz2PwL2cR9UK1/hzTNv3hbzBtyy0D0ro14MSB00TatvTdhth7NQYh9dwV+zrUckXo5+0glUjQfnFYkec39HZG6E3Vwetvsv2jFqjr6Yi6M/eX+Fwida7+11pWUk8SXuPLHVcUrW0F7b4SA2cbC8Vs10MT2+LcQ/kauf1Xn37j9ZMLTFTIEQRkZeYUPSaQmZvesMVN21gGhEinnGws8F6zynC2tVTaX62iPXxc7BTbro5Was9f1LeB0vaQFlXUHudb0U5pu5a7A+yszHF0UnsNoibSvqK6b3OTNkB9aZXCrDhyF/VnKRddzijG0AIJMze9YYubNqWlAUf+G/TKMiBEotgxugWS07Ph4WSj9vMK9lZ4v3ll/O9MFOytzDGrZ13UcndAdo6AF6mZ+CH0DsZ3qoG78SmKEhNA3oLtVSrY4fiXHZRmMhIZsj1XVcekFea7/bdU9jWYdQCfvmH9YdIfJm7adOSIvIZb5cqAn5/Y0RCVSU18yr/xmCld68Cngh2C67rDzEyCQYF5rW4ftqkKR2sLvEjJUBQDBgBpvsHX3uVtceubENSatk+7wRMZKHV19fLj3AT9YVepNuXvJuXfYiKDZWdljg/b+Coq6OfnaC0vL1LB3gpze9dT7LcyV/51aWUuLfT61hZ5x37Rpabi/aYPmilt51r1fiP8PLhx8X8AIgOTnsVlr/SFLW7aIggsA0JkYno18FQURS2sjlxBH7XzRf8m3gh/8BIHI+MworUvQuq543ZcCtrWrIi2NStiZFtfRWtdXU9HhNSTr9G677M2OHv/JWbuuq6bH4hIR67nW0GCdIuJm7bcuAE8egRYWQEdO4odDRFpgZOtBQ5/3g42loW3ruVnbWGGKV3rAJBPlhjQTF69vrqrA6q7OiiOszKX4vKMLvjzYjR6+nsq9td2d0RcUuGFT4kMFfuY9IddpdqS29rWvj1gZ1fkoURkPHwr2hc60SH/MkQl5WRrgWGtqqKCvfLs17Y1XDCxs2p3amEKrt9aXL+Naq7ReURqMXPTGyZu2sIyIERlzuh21XBuapBi20wLY1slEgk+7VQD4VM74cPWVRH6ebsij6/pZq+yz6mQZcByBVR2RqBvhVLFSUTiYOKmDYmJ8oXlASZuRGVMRQcr9GggH6P2UdtqWruuq4M1pvXwQ7WKqolZ/lp06pY3ujAtCGemdIKfh6Paa8/rU19rceb6KkTz1kcyfmxw0x8mbtpw6BCQnQ3UrAlUry52NESkZ9/3b4i/x7bCJx118///zJ5+6Oznptiu65mXkEkgwaDAyvAub4OZPf3w99hWMJeawd3JGrs/ba1ScBgA6vyX0P0zrjUaVFJd4cXeyhwnvuqAU5PzxuvO7V0PHWpVVDl2ROuqeDC/G8a0r4aF72g/ISTjwLVK9YeTE7SB3aREZZqluRn8vZ11dv3hrapieKuqWHZIXgHfwUq5K/TbPvVVFgIH5F+mRXXf1q/khF3jWsNnsvx3WDOf8mhZvQJ6N/RCpXLKpVKqV7RH74aeOHwzHjsuROP4necAgOk98mpWlrdTv1oFAJyfFoQm3xwqxk9beg7W5pjzVl38GHpXZ2uNkjJBKMHCq1QqTNxKSybLW5+UiRsR6dBnQfJJC4mv89aSNJfKE7PCWjy61/fAyrC7aO5bAUF13FDdVbXrdUiLKth0+hFm9PRDPS/lFri/xrbCnbhktKgmHxP3VkMvmJuZ4fid5/CpoJzcdaztipC67mjg7QQnGwtM/fOa4jMXeyv8PbYVJu+8ihsxSUrX33zmEdrVqohxWy4q9j+Y3w2RMUn435koSCTAlnxLOAHA/0YEwsPZGp2WHFX5eS7P6AIzMwn8Kzmjo5rPSftc7AtP2km7mLiV1qVLQGysfCZp27ZiR0NEZYCTjQV+HdEMFlIzWLyhvpyNpRSHJrYrsitrzlv18HW3OrC2UC170tDbGQ0LtCZ2q++OPz9uqZIESs0kWPVfIeGz918o9v86ohkAwN/bGXvHt8GhyDhM2H4JS/s3VLr+/WepinU1JRIJ6no6Yf7b8u7Xgolb6xouStvBdd2w/3oc6ng4wuy/cX++asYHlsaZKZ3QfH6oVq9pKlpWc3nzQaQVTNxKK7cMSFCQvIYbEZEetKmhOt6sMMUZf6QuaSvqegGVyxV5TKMq5VDb3QG+Fe1UYg3yc1O0iuU3qq0vXqZmKo3nU2fXuFaK9042Fkh8nYVPOtbArF51UaGI7tr82tasiIXv1EeL+YeLdfwvQ5vA3cm6WMeqY2cpRWqmeKsLdG/ggd1XYt58oIaqVFBdhYR0g5MTSovj24iIVFhIzbB3fBv8NEj9Ul4FkzZAnjzO6lUXraqrtt5M6y4vbPxRO180qOSs2H/iqw44NLEd6nk5wcPJBpYFliZb9X4jtfcXBKHQ+nwF/TGmJTrVKTqZLMyq9xuhtrsDtoxsDn81E0H04cH8blgxsBEGBlYu8ri733bF+82LPqYwvQO8NDqPSo4tbqXx/Dlw9qz8PRM3IiIl2pxp+GEbX/Rr4q1So87B2gIO1oXXrctdTqykHK3NMf/tBmhb00Xt9dvUcEGr6i4wkwDz9txU+mxa9zrIyJbho7a+MJeaKWLo3sADl6MTAQDz366PNjVcMOPv6zh8Mx5ezjZ4kvC6yJhyWxcL+qidL1YfvV/oebn/Hea+VQ+tqrnA0twMIzedVzrG3dEa5lIzfNO7PlpXd8F3+2/h3jP5xI7Qz9upHUtI4mDiVhr798vXKG3QAKhUSexoiIhM2psKC5fWkn7+6FzXDRZmZrC2MFObeL7dyAs7I55gZs+6qO5qjxyZgGyZgPvPUrHjQjQAeZKpTvcGnookr6e/J+ytzLGknz+2hEehd4AXWi1Q7bb193ZG0ussPHieis5+bop75Jrdqy7srfK+yqtUsMWjF2lq7y81k6B7Aw9ciU5Q+ezIF+0V70PqeaB9LVf0/PEEmviUR7WK9hjaogo2nn6k9rqkX0zcSoPdpERERim3esWSfv44evsZvuvXAFbmbx7nt6SfP+b1qa8YEyg1k+Dj9tWR+DoLoTfi1Hbz5lLX/ljOzhJjOxRe/2/ROw1QztYCByLj0DvACyfuPEdsUrri83pejrj0OFGx3aOBB1YcuQdAPqO4Y21XlWtmy/JKd3wZUgvVK9qrrMdrbSHFgQltFcnrJ51q4NS9F3Cxt8LpfBNPAGB4K59C4yftY+KmqZwcYN8++XsmbkREBsungi0evkhDHQ9HRSmS3LIn7zSuhHcaF7/HRCKRqJ3I4WRjgfPTOqtdySKXvXXeV66FVPW4+W/Xx5SdV7FiYCM421ogJjEdtdwdAADvN68CQN5tWXfm/vwR4a2Gnlh68Dba1HBBL38vReI2/536cFTTzVvJOW9s38ftC08a87c4uthb4eBE+fJruXX/cnkWc6wgaQcTN02dPQu8egU4OwMtWogdDRERFeLXEYH49cwjDG/lg8TXWThwPQ4jC+nOLI2ikjYAcLS2wLphTSA1M1PbujegWWW83ciryJY/dfdwsbfC+WlBsDKXd+8emtgOdlZStUkbALg6WmPnxy3hYKWdFICLJuiXUc0qXbBgASQSCT777DOxQ8krAxIcDJgz/yUiMlTe5W3xdbc68HCyQW13R3zaqYZK16C+dKzthnY1Cy/l8qbu2oJJUm4iZ20hVbSQVXe1f+OM2UaVy6GGm0MxIlb1++gW+CyoBrrX94C9lTnebsQx3vpkNInbuXPnsHr1ajRooLrunigSEgALC3aTEhGR3liZSxVr4jb3LY8GXvovMdLUpzw+C6qJ5QMDcGlGZ5S3s9R7DEVZsWIFfHx8YG1tjcDAQISHhxd5/O+//47atWvD2toa9evXx57chpn/DBs2DBKJROkVEhKiyx+hSEaRuKWkpGDQoEFYs2YNypUruuij3ixfDrx4AfTtK3YkRERUhnzepRYeLuiObaNaqK2Hpy8SiQTmb1i5Q99+++03TJw4ETNnzkRERAT8/f0RHByM+Ph4tcefOnUKAwYMwIgRI3Dx4kX07t0bvXv3xrVr15SOCwkJQUxMjOK1detWffw4ahnWEy/E2LFj0b17dwQFBb3x2IyMDCQlJSleycnJugvMwQGwZbVoIiIiQ/D9999j5MiRGD58OPz8/LBq1SrY2tpi3bp1ao//v//7P4SEhGDSpEmoU6cO5s6di0aNGmH58uVKx1lZWcHd3V3xErMRyeATt23btiEiIgLz588v1vHz58+Hk5OT4uXn56fjCImIiEhXkpOTlRpkMjIy1B6XmZmJCxcuKDXymJmZISgoCKdPn1Z7zunTp1UahYKDg1WODwsLg6urK2rVqoUxY8bgxQvlkij6ZNCJ2+PHjzF+/Hhs3rwZ1tbFWyNuypQpSExMVLwiIyN1HCURERHpip+fn1KDTGENOc+fP0dOTg7c3JSXJ3Nzc0NsbKzac2JjY994fEhICDZt2oTQ0FAsXLgQR48eRdeuXZGTI87aswY9HfLChQuIj49Ho0Z5a83l5OTg2LFjWL58OTIyMiCVKs/AsbKyglW+xd6TkpL0Fi8RERFpV2RkJLy88tZCzf8drw/vvfee4n39+vXRoEEDVKtWDWFhYejUqZNeYwEMPHHr1KkTrl69qrRv+PDhqF27Nr766iuVpI2IiIhMi4ODAxwdHd94nIuLC6RSKeLi4pT2x8XFwd3dXe057u7uJToeAHx9feHi4oK7d++KkrgZdFepg4MD6tWrp/Sys7NDhQoVUK9ePbHDIyIiIgNhaWmJxo0bIzQ0VLFPJpMhNDQULQoplN+iRQul4wHg4MGDhR4PANHR0Xjx4gU8PDy0E3gJGXTiRkRERFRcEydOxJo1a7Bx40bcuHEDY8aMQWpqKoYPHw4AGDJkCKZMmaI4fvz48di3bx+WLFmCmzdvYtasWTh//jzGjRsHQF6ObNKkSThz5gwePnyI0NBQvPXWW6hevTqCg4NF+RkNuqtUnbCwMLFDICIiIgP07rvv4tmzZ5gxYwZiY2PRsGFD7Nu3TzEBISoqCmZmeW1WLVu2xJYtWzBt2jR8/fXXqFGjBv766y9Fr55UKsWVK1ewceNGJCQkwNPTE126dMHcuXP1PtYul0QQBEGUO+tJdHQ0vL298fjxY1SqxGU5iIiIjAG/v9VjVykRERGRkWDiRkRERGQkmLgRERERGQkmbkRERERGgokbERERkZEwunIgJSWTyQAAMTExIkdCRERExZX7vZ37PU5yJp+45S5l0axZM5EjISIiopKKi4tD5cqVxQ7DYJh8Hbfs7GxcvHgRbm5uSkX3Sis5ORl+fn6IjIyEg4OD1q5rrPg8lPF55OGzUMbnoYzPIw+fhTKZTIa4uDgEBATA3Nzk25mKzeQTN11JSkqCk5MTEhMTi7X4ranj81DG55GHz0IZn4cyPo88fBZUHJycQERERGQkmLgRERERGQkmbhqysrLCzJkzRVtk1tDweSjj88jDZ6GMz0MZn0cePgsqDo5xIyIiIjISbHEjIiIiMhJM3IiIiIiMBBM3IiIiIiPBxI2IiIjISDBxK8KKFSvg4+MDa2trBAYGIjw8vMjjf//9d9SuXRvW1taoX78+9uzZo6dI9aMkz2PNmjVo06YNypUrh3LlyiEoKOiNz8/YlPTvR65t27ZBIpGgd+/eug1Qj0r6LBISEjB27Fh4eHjAysoKNWvWNKn/X0r6PJYtW4ZatWrBxsYG3t7emDBhAtLT0/UUre4cO3YMPXv2hKenJyQSCf766683nhMWFoZGjRrBysoK1atXx4YNG3Qep76U9Hns3LkTnTt3RsWKFeHo6IgWLVpg//79+gmWDJdAam3btk2wtLQU1q1bJ1y/fl0YOXKk4OzsLMTFxak9/uTJk4JUKhUWLVokREZGCtOmTRMsLCyEq1ev6jly3Sjp8xg4cKCwYsUK4eLFi8KNGzeEYcOGCU5OTkJ0dLSeI9eNkj6PXA8ePBC8vLyENm3aCG+99ZZ+gtWxkj6LjIwMoUmTJkK3bt2EEydOCA8ePBDCwsKES5cu6Tly3Sjp89i8ebNgZWUlbN68WXjw4IGwf/9+wcPDQ5gwYYKeI9e+PXv2CFOnThV27twpABD+/PPPIo+/f/++YGtrK0ycOFGIjIwUfvzxR0EqlQr79u3TT8A6VtLnMX78eGHhwoVCeHi4cPv2bWHKlCmChYWFEBERoZ+AySAxcStEs2bNhLFjxyq2c3JyBE9PT2H+/Plqj+/fv7/QvXt3pX2BgYHCRx99pNM49aWkz6Og7OxswcHBQdi4caOuQtQrTZ5Hdna20LJlS2Ht2rXC0KFDTSZxK+mzWLlypeDr6ytkZmbqK0S9KunzGDt2rNCxY0elfRMnThRatWql0zj1rTiJypdffinUrVtXad+7774rBAcH6zAycRTneajj5+cnzJ49W/sBkdFgV6kamZmZuHDhAoKCghT7zMzMEBQUhNOnT6s95/Tp00rHA0BwcHChxxsTTZ5HQWlpacjKykL58uV1FabeaPo85syZA1dXV4wYMUIfYeqFJs9i165daNGiBcaOHQs3NzfUq1cP8+bNQ05Ojr7C1hlNnkfLli1x4cIFRXfq/fv3sWfPHnTr1k0vMRsSU/49qg0ymQzJyckm8XuUNGcudgCG6Pnz58jJyYGbm5vSfjc3N9y8eVPtObGxsWqPj42N1Vmc+qLJ8yjoq6++gqenp8ovZWOkyfM4ceIEfvnlF1y6dEkPEeqPJs/i/v37OHz4MAYNGoQ9e/bg7t27+Pjjj5GVlYWZM2fqI2yd0eR5DBw4EM+fP0fr1q0hCAKys7MxevRofP311/oI2aAU9ns0KSkJr1+/ho2NjUiRGYbFixcjJSUF/fv3FzsUEhFb3EjnFixYgG3btuHPP/+EtbW12OHoXXJyMgYPHow1a9bAxcVF7HBEJ5PJ4Orqip9//hmNGzfGu+++i6lTp2LVqlVihyaKsLAwzJs3Dz/99BMiIiKwc+dO7N69G3PnzhU7NDIgW7ZswezZs7F9+3a4urqKHQ6JiC1uari4uEAqlSIuLk5pf1xcHNzd3dWe4+7uXqLjjYkmzyPX4sWLsWDBAhw6dAgNGjTQZZh6U9Lnce/ePTx8+BA9e/ZU7JPJZAAAc3Nz3Lp1C9WqVdNt0Dqiyd8NDw8PWFhYQCqVKvbVqVMHsbGxyMzMhKWlpU5j1iVNnsf06dMxePBgfPjhhwCA+vXrIzU1FaNGjcLUqVNhZlZ2/n1d2O9RR0fHMt3atm3bNnz44Yf4/fffTaLXgkqn7PxGKAFLS0s0btwYoaGhin0ymQyhoaFo0aKF2nNatGihdDwAHDx4sNDjjYkmzwMAFi1ahLlz52Lfvn1o0qSJPkLVi5I+j9q1a+Pq1au4dOmS4tWrVy906NABly5dgre3tz7D1ypN/m60atUKd+/eVSSvAHD79m14eHgYddIGaPY80tLSVJKz3KRWKGNLSZvy71FNbd26FcOHD8fWrVvRvXt3scMhQyD27AhDtW3bNsHKykrYsGGDEBkZKYwaNUpwdnYWYmNjBUEQhMGDBwuTJ09WHH/y5EnB3NxcWLx4sXDjxg1h5syZJlcOpCTPY8GCBYKlpaWwY8cOISYmRvFKTk4W60fQqpI+j4JMaVZpSZ9FVFSU4ODgIIwbN064deuW8O+//wqurq7CN998I9aPoFUlfR4zZ84UHBwchK1btwr3798XDhw4IFSrVk3o37+/WD+C1iQnJwsXL14ULl68KAAQvv/+e+HixYvCo0ePBEEQhMmTJwuDBw9WHJ9bDmTSpEnCjRs3hBUrVphUOZCSPo/NmzcL5ubmwooVK5R+jyYkJIj1I5ABYOJWhB9//FGoXLmyYGlpKTRr1kw4c+aM4rN27doJQ4cOVTp++/btQs2aNQVLS0uhbt26wu7du/UcsW6V5HlUqVJFAKDymjlzpv4D15GS/v3Iz5QSN0Eo+bM4deqUEBgYKFhZWQm+vr7Ct99+K2RnZ+s5at0pyfPIysoSZs2aJVSrVk2wtrYWvL29hY8//lh49eqV/gPXsiNHjqj9PZD78w8dOlRo166dyjkNGzYULC0tBV9fX2H9+vV6j1tXSvo82rVrV+TxVDZJBKGMtcUTERERGSmOcSMiIiIyEkzciIiIiIwEEzciIiIiI8HEjYiIiMhIMHEjIiIiMhJM3IiIiIiMBBM3IiIiIiPBxI2IiIhK7NixY+jZsyc8PT0hkUjw119/lfga27dvR8OGDWFra4sqVargu+++036gJoaJGxFpXVhYGCQSCRISEoo8zsfHB8uWLdNLTMUxbNgw9O7du0TnGNrPQKQvqamp8Pf3x4oVKzQ6f+/evRg0aBBGjx6Na9eu4aeffsLSpUuxfPlyLUdqWpi4EZVRw4YNg0QigUQigaWlJapXr445c+YgOzu71Ndu2bIlYmJi4OTkBADYsGEDnJ2dVY47d+4cRo0aVer7FaUkidX//d//YcOGDTqNh8hUdO3aFd988w369Omj9vOMjAx88cUX8PLygp2dHQIDAxEWFqb4/Ndff0Xv3r0xevRo+Pr6onv37pgyZQoWLlwILupUOCZuRGVYSEgIYmJicOfOHXz++eeYNWuWVroqLC0t4e7uDolEUuRxFStWhK2tbanvV1o5OTmQyWRwcnJSm2ASUcmNGzcOp0+fxrZt23DlyhX069cPISEhuHPnDgB5Ymdtba10jo2NDaKjo/Ho0SMxQjYKTNyIyjArKyu4u7ujSpUqGDNmDIKCgrBr1y4AwKtXrzBkyBCUK1cOtra26Nq1q+IXLgA8evQIPXv2RLly5WBnZ4e6detiz549AJS7SsPCwjB8+HAkJiYqWvhmzZoFQLk1bODAgXj33XeV4svKyoKLiws2bdoEAJDJZJg/fz6qVq0KGxsb+Pv7Y8eOHYX+fO3bt8ejR48wYcIExb2BvBbAXbt2wc/PD1ZWVoiKilLpKm3fvj3GjRuHcePGwcnJCS4uLpg+fXqRrQEJCQn48MMPUbFiRTg6OqJjx464fPmy4vPLly+jQ4cOcHBwgKOjIxo3bozz58+/4b8UkXGJiorC+vXr8fvvv6NNmzaoVq0avvjiC7Ru3Rrr168HAAQHB2Pnzp0IDQ2FTCbD7du3sWTJEgBATEyMmOEbNHOxAyAiw2FjY4MXL14AkHel3rlzB7t27YKjoyO++uordOvWDZGRkbCwsMDYsWORmZmJY8eOwc7ODpGRkbC3t1e5ZsuWLbFs2TLMmDEDt27dAgC1xw0aNAj9+vVDSkqK4vP9+/cjLS1N0RUzf/58/O9//8OqVatQo0YNHDt2DO+//z4qVqyIdu3aqVxz586d8Pf3x6hRozBy5Eilz9LS0rBw4UKsXbsWFSpUgKurq9pnsnHjRowYMQLh4eE4f/48Ro0ahcqVK6tcL1e/fv1gY2ODvXv3wsnJCatXr0anTp1w+/ZtlC9fHoMGDUJAQABWrlwJqVSKS5cuwcLCorD/JERG6erVq8jJyUHNmjWV9mdkZKBChQoAgJEjR+LevXvo0aMHsrKy4OjoiPHjx2PWrFkwM2O7UmGYuBERBEFAaGgo9u/fj08++USRsJ08eRItW7YEAGzevBne3t7466+/0K9fP0RFReGdd95B/fr1AQC+vr5qr21paQknJydIJBK4u7sXGkNwcDDs7Ozw559/YvDgwQCALVu2oFevXnBwcEBGRgbmzZuHQ4cOoUWLFop7njhxAqtXr1abuJUvXx5SqRQODg4q987KysJPP/0Ef3//Ip+Nt7c3li5dColEglq1auHq1atYunSp2sTtxIkTCA8PR3x8PKysrAAAixcvxl9//YUdO3Zg1KhRiIqKwqRJk1C7dm0AQI0aNYq8P5ExSklJgVQqxYULFyCVSpU+y/2HmUQiwcKFCzFv3jzExsaiYsWKCA0NBVD47xNi4kZUpv3777+wt7dHVlYWZDIZBg4ciFmzZiE0NBTm5uYIDAxUHFuhQgXUqlULN27cAAB8+umnGDNmDA4cOICgoCC88847aNCggcaxmJubo3///ti8eTMGDx6M1NRU/P3339i2bRsA4O7du0hLS0Pnzp2VzsvMzERAQECJ72dpaVmseJs3b640Vq9FixZYsmQJcnJyVL6QLl++jJSUFEWLQq7Xr1/j3r17AICJEyfiww8/xK+//oqgoCD069cP1apVK3H8RIYsICAAOTk5iI+PR5s2bYo8ViqVwsvLCwCwdetWtGjRAhUrVtRHmEaJiRtRGdahQwesXLkSlpaW8PT0hLl58X8lfPjhhwgODsbu3btx4MABzJ8/H0uWLMEnn3yicTyDBg1Cu3btEB8fj4MHD8LGxgYhISEA5P+CB4Ddu3crfsnnym3dKgkbG5s3Tp4oqZSUFHh4eCjNnMuVO+lh1qxZGDhwIHbv3o29e/di5syZ2LZtW6Ez84gMVUpKCu7evavYfvDgAS5duoTy5cujZs2aGDRoEIYMGYIlS5YgICAAz549Q2hoKBo0aIDu3bvj+fPn2LFjB9q3b4/09HTFmLijR4+K+FMZPiZuRGWYnZ0dqlevrrK/Tp06yM7OxtmzZxVdpS9evMCtW7fg5+enOM7b2xujR4/G6NGjMWXKFKxZs0Zt4mZpaYmcnJw3xtOyZUt4e3vjt99+w969e9GvXz/F+K/8kwjUdYsWprj3LszZs2eVts+cOYMaNWqotLYBQKNGjRAbGwtzc3P4+PgUes2aNWuiZs2amDBhAgYMGID169czcSOjc/78eXTo0EGxPXHiRADA0KFDsWHDBqxfvx7ffPMNPv/8czx58gQuLi5o3rw5evTooThn48aN+OKLLyAIAlq0aIGwsDA0a9ZM7z+LMWHiRkQqatSogbfeegsjR47E6tWr4eDggMmTJ8PLywtvvfUWAOCzzz5D165dUbNmTbx69QpHjhxBnTp11F7Px8cHKSkpCA0Nhb+/P2xtbQstAzJw4ECsWrUKt2/fxpEjRxT7HRwc8MUXX2DChAmQyWRo3bo1EhMTcfLkSTg6OmLo0KGF3vvYsWN47733YGVlBRcXlxI9i6ioKEycOBEfffQRIiIi8OOPPypmvhUUFBSEFi1aoHfv3li0aBFq1qyJp0+fYvfu3ejTpw/q1q2LSZMmoW/fvqhatSqio6Nx7tw5vPPOOyWKicgQtG/fvsgZ1hYWFpg9ezZmz56t9nMXFxecPn1aV+GZLE7bICK11q9fj8aNG6NHjx5o0aIFBEHAnj17FC1gOTk5GDt2LOrUqYOQkBDUrFkTP/30k9prtWzZEqNHj8a7776LihUrYtGiRYXed9CgQYiMjISXlxdatWql9NncuXMxffp0zJ8/X3Hf3bt3o2rVqoVeb86cOXj48CGqVaum0biZIUOG4PXr12jWrBnGjh2L8ePHF1o0WCKRYM+ePWjbti2GDx+OmjVr4r333sOjR4/g5uYGqVSKFy9eYMiQIahZsyb69++Prl27FvrFRkRUkERgeWIiIrXat2+Phg0bckkrIjIYbHEjIiIiMhJM3IiIiIiMBLtKiYiIiIwEW9yIiIiIjAQTNyIiIiIjwcSNiIiIyEgwcSMiIiIyEkzciIiIiIwEEzciIiIiI8HEjYiIiMhIMHEjIiIiMhL/D10ee3QXL7f/AAAAAElFTkSuQmCC",
                  "text/plain": [
                     "<Figure size 640x480 with 2 Axes>"
                  ]
               },
               "metadata": {},
               "output_type": "display_data"
            }
         ],
         "source": [
            "n_epochs = 70\n",
            "val_ep_interval = 10\n",
            "\n",
            "cumulative_triples = 0\n",
            "training_loss = []\n",
            "val_mrr = []\n",
            "poptorch_model.attachToDevice()\n",
            "for ep in range(n_epochs):\n",
            "    ep_start_time = time.time()\n",
            "    ep_log = []\n",
            "    for batch in train_dl:\n",
            "        step_start_time = time.time()\n",
            "        cumulative_triples += batch[\"head\"].numel()\n",
            "        res = poptorch_model(**{k: v.flatten(end_dim=1) for k, v in batch.items()})\n",
            "        ep_log.append(\n",
            "            dict(\n",
            "                loss=float(torch.sum(res[\"loss\"])) / batch[\"head\"][0].numel(),\n",
            "                step_time=(time.time() - step_start_time),\n",
            "            )\n",
            "        )\n",
            "    ep_loss = [v[\"loss\"] for v in ep_log]\n",
            "    training_loss.extend([v[\"loss\"] for v in ep_log])\n",
            "    print(\n",
            "        f\"Epoch {ep+1} loss: {np.mean(ep_loss):.6f} --- positive triples processed: {cumulative_triples:.2e}\"\n",
            "    )\n",
            "    print(\n",
            "        f\"Epoch duration (sec): {(time.time() - ep_start_time):.5f} (average step time: {np.mean([v['step_time'] for v in ep_log]):.5f})\"\n",
            "    )\n",
            "    if ep % val_ep_interval == 0:\n",
            "        poptorch_model.detachFromDevice()\n",
            "        poptorch_inf_model.attachToDevice()\n",
            "        # If running training and inference on separate devices, uncomment the following 2 lines:\n",
            "        # poptorch_model.copyWeightsToHost()\n",
            "        # poptorch_inf_model.copyWeightsToDevice()\n",
            "        val_start_time = time.time()\n",
            "        ep_mrr = 0.0\n",
            "        for batch_val in sample_valid_dl:\n",
            "            ep_mrr += poptorch_inf_model(\n",
            "                **{k: v.flatten(end_dim=1) for k, v in batch_val.items()}\n",
            "            )[\"metrics\"].sum()\n",
            "        ep_mrr /= n_sample_queries\n",
            "        val_mrr.append(ep_mrr)\n",
            "        print(\n",
            "            f\"Epoch {ep+1} sample MRR: {ep_mrr:.4f} (validation time: {(time.time() - val_start_time):.5f})\"\n",
            "        )\n",
            "        poptorch_inf_model.detachFromDevice()\n",
            "        poptorch_model.attachToDevice()\n",
            "\n",
            "# Plot loss and sample MRR as a function of the number of positive triples processed\n",
            "total_triples = np.cumsum(n_epochs * len(train_dl) * [batch[\"head\"].numel()])\n",
            "ax0, ax1 = plt.gca(), plt.twinx()\n",
            "(line0,) = ax0.plot(total_triples, training_loss)\n",
            "(line1,) = ax1.plot(\n",
            "    total_triples[:: val_ep_interval * len(train_dl)], val_mrr, color=\"r\"\n",
            ")\n",
            "ax0.set_xlabel(\"Positive triples\")\n",
            "ax0.set_ylabel(\"Loss\")\n",
            "ax1.set_ylabel(\"Sample MRR\")\n",
            "plt.legend([line0, line1], [\"Loss\", \"MRR\"], loc=\"upper left\")\n",
            "\n",
            "poptorch_model.detachFromDevice()\n",
            "del train_dl\n",
            "del sample_valid_dl"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Let us compute the final MRR on the whole validation set, containing 429k+ triples. Once again, even though the dataset provides 500 candidate tails for each triple, here we are actually scoring queries against **all entities** in the dataset."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 38,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Number of triples per h_shard:\n",
                  "[106601 107205 108197 107453]\n"
               ]
            }
         ],
         "source": [
            "validation_triples = PartitionedTripleSet.create_from_dataset(\n",
            "    wikikg, \"valid\", sharding, partition_mode=\"h_shard\"\n",
            ")\n",
            "bs_valid = RigidShardedBatchSampler(\n",
            "    partitioned_triple_set=validation_triples,\n",
            "    negative_sampler=candidate_sampler,\n",
            "    shard_bs=val_shard_bs,\n",
            "    batches_per_step=val_device_iterations,\n",
            "    seed=seed,\n",
            ")\n",
            "\n",
            "print(\"Number of triples per h_shard:\")\n",
            "print(validation_triples.triple_counts)\n",
            "\n",
            "valid_dl = bs_valid.get_dataloader(options=val_options, shuffle=False, num_workers=3)"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 39,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "Validation MRR: 0.2668512463569641\n",
                  "Validation time (sec): 47.12475\n"
               ]
            }
         ],
         "source": [
            "poptorch_inf_model.attachToDevice()\n",
            "\n",
            "val_mrr = 0.0\n",
            "start_time = time.time()\n",
            "n_val_queries = 0\n",
            "for batch_val in valid_dl:\n",
            "    n_val_queries += batch_val[\"triple_mask\"].sum()\n",
            "    val_mrr += poptorch_inf_model(\n",
            "        **{k: v.flatten(end_dim=1) for k, v in batch_val.items()}\n",
            "    )[\"metrics\"].sum()\n",
            "\n",
            "print(f\"Validation MRR: {val_mrr / n_val_queries}\")\n",
            "print(f\"Validation time (sec): {(time.time() - start_time):.5f}\")\n",
            "\n",
            "poptorch_inf_model.detachFromDevice()\n",
            "del valid_dl"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "## Validation (vs candidate tails)\n",
            "\n",
            "Now we can check the metrics when scoring validation queries only against the provided candidates.\n",
            "\n",
            "Similarly to what we did in the [KGE Training and Inference on OGBL-BioKG](1_biokg_training_inference.ipynb) notebook, we use `TripleBasedShardedNegativeSampler` to provide the model with the triple-specific candidate tails."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 40,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "head            torch.Size([10, 4, 4, 64])          torch.int32;\n",
                  "relation        torch.Size([10, 4, 4, 64])          torch.int32;\n",
                  "tail            torch.Size([10, 4, 4, 64])          torch.int32;\n",
                  "triple_mask     torch.Size([10, 4, 4, 64])          torch.bool;\n",
                  "negative        torch.Size([10, 4, 4, 256, 173])    torch.int32;\n",
                  "negative_mask   torch.Size([10, 4, 256, 4, 173])    torch.bool;\n"
               ]
            }
         ],
         "source": [
            "validation_triples = PartitionedTripleSet.create_from_dataset(\n",
            "    dataset=wikikg, part=\"valid\", sharding=sharding, partition_mode=\"ht_shardpair\"\n",
            ")\n",
            "ns_valid = TripleBasedShardedNegativeSampler(\n",
            "    negative_heads=validation_triples.neg_heads,\n",
            "    negative_tails=validation_triples.neg_tails,\n",
            "    sharding=sharding,\n",
            "    corruption_scheme=\"t\",\n",
            "    seed=seed,\n",
            ")\n",
            "# We do not need to duplicate_batch as we only want to score negative tails\n",
            "bs_valid = RigidShardedBatchSampler(\n",
            "    partitioned_triple_set=validation_triples,\n",
            "    negative_sampler=ns_valid,\n",
            "    shard_bs=256,\n",
            "    batches_per_step=10,\n",
            "    seed=seed,\n",
            "    duplicate_batch=False,\n",
            ")\n",
            "\n",
            "# Example batch\n",
            "idx_sampler = iter(bs_valid.get_dataloader_sampler(shuffle=False))\n",
            "for k, v in bs_valid[next(idx_sampler)].items():\n",
            "    print(f\"{k:<15} {str(v.shape):<35} {v.dtype};\")"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "As we see from the shape of `negative` and `negative_mask`, the 500 negative tails for each validation triple are retrieved in blocks of 173 entities from each of the 4 IPUs (with padding added where needed).\n",
            "\n",
            "We use the `ScoreMovingBessKGE` version of BESS, as recommended for performing inference against triple-specific sets of candidates."
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 41,
         "metadata": {},
         "outputs": [
            {
               "name": "stderr",
               "output_type": "stream",
               "text": [
                  "[11:12:23.084] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 369\n",
                  "[11:12:23.086] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 375\n",
                  "Graph compilation: 100%|██████████| 100/100 [01:11<00:00]\n"
               ]
            }
         ],
         "source": [
            "val_options = poptorch.Options()\n",
            "val_options.replication_factor = sharding.n_shard\n",
            "val_options.deviceIterations(bs_valid.batches_per_step)\n",
            "val_options.outputMode(poptorch.OutputMode.All)\n",
            "\n",
            "valid_dl = bs_valid.get_dataloader(\n",
            "    options=val_options, shuffle=False, num_workers=3, persistent_workers=True\n",
            ")\n",
            "\n",
            "# Each triple is now to be scored against a specific set of negatives, so we turn off negative sample sharing\n",
            "transe_score_fn.negative_sample_sharing = False\n",
            "\n",
            "evaluation = Evaluation([\"mrr\", \"hits@1\", \"hits@5\", \"hits@10\"], reduction=\"sum\")\n",
            "model_inf = ScoreMovingBessKGE(\n",
            "    negative_sampler=ns_valid, score_fn=transe_score_fn, evaluation=evaluation\n",
            ")\n",
            "\n",
            "poptorch_model_inf = poptorch.inferenceModel(model_inf, options=val_options)\n",
            "\n",
            "poptorch_model_inf.entity_embedding.replicaGrouping(\n",
            "    poptorch.CommGroupType.NoGrouping,\n",
            "    0,\n",
            "    poptorch.VariableRetrievalMode.OnePerGroup,\n",
            ")\n",
            "\n",
            "# Compile model\n",
            "batch = next(iter(valid_dl))\n",
            "res = poptorch_model_inf(**{k: v.flatten(end_dim=1) for k, v in batch.items()})"
         ]
      },
      {
         "cell_type": "code",
         "execution_count": 42,
         "metadata": {},
         "outputs": [
            {
               "name": "stdout",
               "output_type": "stream",
               "text": [
                  "hits@1 : 0.857026\n",
                  "hits@5 : 0.934648\n",
                  "hits@10 : 0.949573\n",
                  "mrr : 0.892919\n",
                  "Validation time (sec): 1.06543\n"
               ]
            }
         ],
         "source": [
            "val_log = []\n",
            "start_time = time.time()\n",
            "n_val_queries = 0\n",
            "for batch_val in valid_dl:\n",
            "    res = poptorch_model_inf(**{k: v.flatten(end_dim=1) for k, v in batch_val.items()})\n",
            "    n_val_queries += batch_val[\"triple_mask\"].sum()\n",
            "    # By transposing res[\"metrics\"] we separate the outputs for the different metrics\n",
            "    val_log.append(\n",
            "        {\n",
            "            k: v.sum()\n",
            "            for k, v in zip(\n",
            "                evaluation.metrics.keys(),\n",
            "                res[\"metrics\"].T,\n",
            "            )\n",
            "        }\n",
            "    )\n",
            "\n",
            "for metric in val_log[0].keys():\n",
            "    reduced_metric = sum([l[metric] for l in val_log]) / n_val_queries\n",
            "    print(\"%s : %f\" % (metric, reduced_metric))\n",
            "print(f\"Validation time (sec): {(time.time() - start_time):.5f}\")\n",
            "\n",
            "poptorch_model_inf.detachFromDevice()\n",
            "del valid_dl"
         ]
      },
      {
         "attachments": {},
         "cell_type": "markdown",
         "metadata": {},
         "source": [
            "Notice that, in this dataset, predicting tails is easier than predicting heads. \n",
            "\n",
            "Want to see how the model we trained performs on the complete task (scoring queries against candidate heads + candidate tails), to compare with the TransE (100dim) result on the [OGB leaderboard](https://ogb.stanford.edu/docs/leader_linkprop/#ogbl-wikikg2)? Just set `corruption_scheme=\"ht\"` when creating the `TripleBasedShardedNegativeSampler` object and `duplicate_batch=True` in the `RigidShardedBatchSampler` and re-run the last three code cells. If this is the aim, however, you might want to corrupt both heads and tails already during training. You can do this by changing the corruption scheme of `RandomShardedNegativeSampler` accordingly before [starting training](#Interleaved-training-and-validation-(vs-all))."
         ]
      }
   ],
   "metadata": {
      "kernelspec": {
         "display_name": ".venv3.2",
         "language": "python",
         "name": "python3"
      },
      "language_info": {
         "codemirror_mode": {
            "name": "ipython",
            "version": 3
         },
         "file_extension": ".py",
         "mimetype": "text/x-python",
         "name": "python",
         "nbconvert_exporter": "python",
         "pygments_lexer": "ipython3",
         "version": "3.8.10"
      },
      "orig_nbformat": 4
   },
   "nbformat": 4,
   "nbformat_minor": 2
}