From 1e0c5c042f4f18e901dba4c3c71df5650b78e79b Mon Sep 17 00:00:00 2001 From: "L. M. Riza Rizky" <42672299+zaRizk7@users.noreply.github.com> Date: Mon, 14 Jul 2025 19:13:34 +0100 Subject: [PATCH 1/3] add visualisation outputs to drug tutorial --- .../tutorial-drug.ipynb | 5130 +++++++++++++---- 1 file changed, 3934 insertions(+), 1196 deletions(-) diff --git a/tutorials/drug-target-interaction/tutorial-drug.ipynb b/tutorials/drug-target-interaction/tutorial-drug.ipynb index 1cb3501..9f22516 100644 --- a/tutorials/drug-target-interaction/tutorial-drug.ipynb +++ b/tutorials/drug-target-interaction/tutorial-drug.ipynb @@ -1,1198 +1,3936 @@ { - "nbformat": 4, - "nbformat_minor": 5, - "metadata": { - "kernelspec": { - "display_name": "mmai-drug-tutorial", - "language": "python", - "name": "python3" - } - }, - "cells": [ - { - "metadata": {}, - "source": [ - "# Drug–Target Interaction Prediction\n", - "\n", - "![](images/drugban-pyakle-api.png)\n", - "\n", - "\n", - "In this tutorial, we will train models to predict the interaction between **two data modalities**: **molecules (drug)** and **proteins (target)** using `PyKale`. Drug-target interaction (DTI) plays a key role in drug discovery and identifying potential therapeutic targets. This example is based on the **DrugBAN** framework by [**Bai et al. (_Nature Machine Intelligence_, 2023)**](https://www.nature.com/articles/s42256-022-00605-1).\n", - "\n", - "The DTI prediction problem is formulated as a **binary classification task**, where the goal is to predict whether a given **drug–protein pair interacts or not**. The DrugBAN framework tackles this problem using two key ideas:\n", - "\n", - "- **Bilinear Attention Network (BAN)**, which learns detailed feature representations for both drugs and proteins and captures local interaction patterns between them.\n", - "\n", - "- **Adversarial Domain Adaptation**, which helps the model generalise to out-of-distribution datasets, i.e., in clustering-based cross-validation instead of random splits, improving its ability to predict interactions on unseen drug–target pairs.\n", - "\n", - "With `PyKale`, implementing such a multimodal DTI prediction pipeline is straightforward. The library provides ready-to-use modules and configuration support, making it easy to apply advanced techniques with minimal custom coding." - ], - "cell_type": "markdown", - "id": "8c1bf9c7" - }, - { - "metadata": {}, - "source": [ - "## Step 0: Environment Preparation\n", - "\n", - "As a starting point, we will install the required packages and load a set of helper functions to assist throughout this tutorial.\n", - "\n", - "To prepare the helper functions and necessary materials, we download them from the GitHub repository.\n", - "\n", - "Moreover, we provide helper functions that can be inspected directly in the `.py` files located in the notebook's current directory. The additional helper script is:\n", - "- [`config.py`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/drug-target-interaction/configs.py): Defines the base configuration settings, which can be overridden using a custom `.yaml` file." - ], - "cell_type": "markdown", - "id": "745ccdcf" - }, - { - "metadata": {}, - "source": [ - "import os\n", - "\n", - "!rm -rf /content/mmai-tutorials\n", - "!git clone --single-branch -b main https://github.com/pykale/mmai-tutorials.git\n", - "%mv /content/mmai-tutorials/tutorials/drug-target-interaction /content/\n", - "%cd /content/drug-target-interaction\n", - "\n", - "print(\"Changed working directory to:\", os.getcwd())" - ], - "cell_type": "code", - "outputs": [], - "id": "a6028209", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "### Package Installation\n", - "\n", - "The main package required for this tutorial is `PyKale`.\n", - "\n", - "`PyKale` is an open-source interdisciplinary machine learning library developed at the University of Sheffield, with a focus on applications in biomedical and scientific domains.\n", - "\n", - "Then, we install `PyG` (PyTorch Geometric) and related packages.\n", - "\n", - "Please **do not** re-run this session after installation completed. Runing this installation multiple times will trigger issues related to `PyG`. If you want to re-run this installation, please click the `Runtime` on the top menu and choose `Disconnect and delete runtime` before installing." - ], - "cell_type": "markdown", - "id": "c52c6334" - }, - { - "metadata": {}, - "source": [ - "!pip install --quiet \\\n", - " git+https://github.com/pykale/pykale@main \\\n", - " yacs==0.1.8 \\\n", - " rdkit \\\n", - " torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric \\\n", - " -f https://data.pyg.org/whl/torch-2.6.0+cu124.html \\\n", - " && echo \"pykale,yacs and wfdb installed successfully ✅\" \\\n", - " || echo \"Failed to install pykale,yacs ❌\"" - ], - "cell_type": "code", - "outputs": [], - "id": "53e3b14e", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "We then hide the warnings messages to get a clear output." - ], - "cell_type": "markdown", - "id": "69f50b6a" - }, - { - "metadata": {}, - "source": [ - "import os\n", - "import warnings\n", - "\n", - "warnings.filterwarnings(\"ignore\")\n", - "os.environ[\"PYTHONWARNINGS\"] = \"ignore\"" - ], - "cell_type": "code", - "outputs": [], - "id": "6e871c63", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "Exercise: Check NumPy Version" - ], - "cell_type": "markdown", - "id": "6606e3fb" - }, - { - "metadata": {}, - "source": [ - "import numpy as np\n", - "\n", - "print(\"NumPy version:\", np.__version__) # numpy should be 2.0.0 or higher" - ], - "cell_type": "code", - "outputs": [], - "id": "0d384020", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "### Configuration\n", - "\n", - "To minimize the footprint of the notebook when specifying configurations, we provide a [`config.py`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/drug-target-interaction/configs.py) file that defines default parameters. These can be customized by supplying a `.yaml` configuration file, such as [`configs/DA_cross_domain.yaml`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/drug-target-interaction/configs/DA_cross_domain.yaml) as an example." - ], - "cell_type": "markdown", - "id": "cabd3406" - }, - { - "metadata": {}, - "source": [ - "from configs import get_cfg_defaults\n", - "\n", - "%cd /content/drug-target-interaction\n", - "\n", - "cfg = get_cfg_defaults() # Load the default settings from config.py\n", - "cfg.merge_from_file(\n", - " \"configs/DA_cross_domain.yaml\"\n", - ") # Update (or override) some of those settings using a custom YAML file" - ], - "cell_type": "code", - "outputs": [], - "id": "55c13b48", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "In this tutorial, we list the hyperparameters we would like users to play with outside the `.yaml` file:\n", - "- `cfg.SOLVER.MAX_EPOCH`: Number of epochs in training stage. You can reduce the number of training epochs to shorten runtime.\n", - "- `cfg.DATA.DATASET`: The dataset used in the study. This can be `bindingdb` or `biosnap`.\n", - "\n", - "As a quick exercise, please take a moment to review and understand the parameters in [`config.py`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/drug-target-interaction/configs.py)." - ], - "cell_type": "markdown", - "id": "74ffdbc2" - }, - { - "metadata": {}, - "source": [ - "cfg.SOLVER.MAX_EPOCH = 2" - ], - "cell_type": "code", - "outputs": [], - "id": "424c7286", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "You can also switch to a different dataset." - ], - "cell_type": "markdown", - "id": "97c088fd" - }, - { - "metadata": {}, - "source": [ - "cfg.DATA.DATASET = \"biosnap\"" - ], - "cell_type": "code", - "outputs": [], - "id": "c69376fa", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "Exercise: Now print the full configuration to check all current hyperparameter and dataset settings." - ], - "cell_type": "markdown", - "id": "d3d41633" - }, - { - "metadata": {}, - "source": [ - "print(cfg)" - ], - "cell_type": "code", - "outputs": [], - "id": "45874296", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "## Step 1: Data Loading and Preparation\n", - "\n", - "In this tutorial, we use the **Biosnap** dataset for the main demonstration and the **BindingDB** dataset for the exercise at the end." - ], - "cell_type": "markdown", - "id": "17558d0c" - }, - { - "metadata": {}, - "source": [ - "### Data Downloading\n", - "\n", - "Please run the following cell to download necessary datasets." - ], - "cell_type": "markdown", - "id": "6c6071b9" - }, - { - "metadata": {}, - "source": [ - "!rm -rf data\n", - "!mkdir data\n", - "!cd data\n", - "\n", - "!pip install -q gdown\n", - "!gdown --id 1ogOcxZn-1q418LOT-gQ94aHQV0Y1sOmk --output data/drug-target-interaction.zip\n", - "!unzip data/drug-target-interaction.zip -d data/\n", - "!mv data/drug-target-interaction/checkpoint ./" - ], - "cell_type": "code", - "outputs": [], - "id": "56f9f58e", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "Exercise: Check the data is ready" - ], - "cell_type": "markdown", - "id": "c39b3e39" - }, - { - "metadata": {}, - "source": [ - "import os\n", - "import shutil\n", - "\n", - "print(\"Contents of the data folder:\")\n", - "for item in os.listdir(\"data/drug-target-interaction\"):\n", - " print(item)" - ], - "cell_type": "code", - "outputs": [], - "id": "a6258d1f", - "execution_count": null - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": [ - "The data content is structured as follows:\n", - "```sh\n", - " ├───data\n", - " │ ├───checkpoint\n", - " │ ├───bindingdb\n", - " │ ├───biosnap" - ], - "id": "9ab0b5f833dc40f8" - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "The `data` folder contains two datasets: `bindingdb` and `biosnap`. Each dataset folder contains the following files. The `checkpoint` folder contains the saved model checkpoint, which are used later in the interpretation section.", - "id": "5be1dcc62b7d5649" - }, - { - "metadata": {}, - "cell_type": "code", - "outputs": [], - "execution_count": null, - "source": [ - "print(\"Contents of bindingdb folder:\")\n", - "for item in os.listdir(\"data/drug-target-interaction/bindingdb\"):\n", - " print(item)" - ], - "id": "a93303c51c8b974e" - }, - { - "metadata": {}, - "source": [ - "Each dataset folder follows the structure:\n", - "\n", - "```sh\n", - " ├───dataset_name\n", - " │ ├───cluster\n", - " │ │ ├───source_train.csv\n", - " │ │ ├───target_train.csv\n", - " │ │ ├───target_test.csv\n", - " │ ├───random\n", - " │ │ ├───test.csv\n", - " │ │ ├───train.csv\n", - " │ │ ├───val.csv\n", - " │ ├───full.csv\n", - "```" - ], - "cell_type": "markdown", - "id": "79cbc1c1" - }, - { - "metadata": {}, - "source": [ - "We use the cluster dataset folder for cross-domain prediction, containing three parts:\n", - "\n", - "- Train samples from the source domain: Drug–protein pairs the model learns from.\n", - "\n", - "- Train samples from the target domain: Additional training data from a different distribution to improve generalisation.\n", - "\n", - "- Test samples from the target domain: Unseen drug–protein pairs used to evaluate model performance on new data.\n", - "\n", - "The source and target sets are defined based on the clustering results." - ], - "cell_type": "markdown", - "id": "d35e04f9" - }, - { - "metadata": {}, - "source": "### Data Loading", - "cell_type": "markdown", - "id": "98acf744" - }, - { - "metadata": {}, - "source": [ - "Here’s what each csv file looks like in a table format:\n", - "\n", - "| SMILES | Protein Sequence | Y |\n", - "|--------------------|--------------------------|---|\n", - "| Fc1ccc(C2(COC…) | MDNVLPVDSDLS… | 1 |\n", - "| O=c1oc2c(O)c(…) | MMYSKLLTLTTL… | 0 |\n", - "| CC(C)Oc1cc(N…) | MGMACLTMTEME… | 1 |\n", - "\n", - "Each row of the dataset contains three key pieces of information:\n", - "\n", - "**Drugs**: \n", - "Drugs are often written as SMILES strings, which are like chemical formulas in text format (for example, `\"CC(=O)OC1=CC=CC=C1C(=O)O\"` is aspirin). \n", - "\n", - "\n", - "**Protein Sequence** \n", - "This is a string of letters where each letter stands for an amino acid, the building blocks of proteins. For example, `MGYTSLLT...` is a short protein sequence.\n", - "\n", - "\n", - "**Y (Labels)**: \n", - "Each drug–protein pair is given a label:\n", - "- `1` if they interact\n", - "- `0` if they do not\n", - "\n", - "\n", - "Each row shows one drug–protein pair. The goal of our machine learning model is to predict the last column (**Y**) — whether or not the drug and protein interact." - ], - "cell_type": "markdown", - "id": "1e5f4f44" - }, - { - "metadata": {}, - "source": "You can load CSV files into Python using tools like `pandas`. The output shows a sample of the data, including the SMILES string for the drug, the protein sequence, the interaction label (Y) and the cluster ID.", - "cell_type": "markdown", - "id": "b7590daf" - }, - { - "metadata": {}, - "source": [ - "import pandas as pd\n", - "\n", - "dataFolder = os.path.join(\n", - " f\"data/drug-target-interaction/{cfg.DATA.DATASET}\", str(cfg.DATA.SPLIT)\n", - ")\n", - "\n", - "df_train_source = pd.read_csv(os.path.join(dataFolder, \"source_train.csv\"))\n", - "df_train_target = pd.read_csv(os.path.join(dataFolder, \"target_train.csv\"))\n", - "df_test_target = pd.read_csv(os.path.join(dataFolder, \"target_test.csv\"))\n", - "\n", - "print(\"Sample example:\", df_train_source.iloc[0]))" - ], - "cell_type": "code", - "outputs": [], - "id": "0c709e31", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "### Data Preprocessing\n", - "\n", - "We convert drug SMILES strings into molecular graphs using `kale.loaddata.molecular_datasets.smiles_to_graph`, encoding atom-level features as node attributes and bond types as edges.\n", - "\n", - "\n", - "Protein sequences are transformed into fixed-length integer arrays using `kale.prepdata.chem_transform.integer_label_protein`, with each amino acid mapped to an integer and sequences padded or truncated to a uniform length.\n", - "\n", - "Finally, the `kale.loaddata.molecular_datasets.DTIDataset` class packages drugs, proteins, and labels into a PyTorch-ready dataset." - ], - "cell_type": "markdown", - "id": "542d4e69" - }, - { - "metadata": {}, - "source": [ - "**Note:** If you encounter an error related to requiring numpy `<2.0`, simply ignore it and re-run this block until it completes successfully." - ], - "cell_type": "markdown", - "id": "981d5520" - }, - { - "metadata": {}, - "source": [ - "from kale.loaddata.molecular_datasets import DTIDataset\n", - "\n", - "# Create preprocessed datasets\n", - "train_dataset = DTIDataset(df_train_source.index.values, df_train_source)\n", - "train_target_dataset = DTIDataset(df_train_target.index.values, df_train_target)\n", - "test_target_dataset = DTIDataset(df_test_target.index.values, df_test_target)" - ], - "cell_type": "code", - "outputs": [], - "id": "ae5af8eb", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "We load data in small, manageable pieces called batches to save memory and speed up training. We use `kale.loaddata.sampler.MultiDataLoader` from PyKale to load one batch from the source domain and one from the target domain at each training step." - ], - "cell_type": "markdown", - "id": "a0a510ce" - }, - { - "metadata": {}, - "source": [ - "First, we specify a few DataLoader parameters:\n", - "- Batch size: Number of samples per batch\n", - "- Shuffle: Randomly shuffle data\n", - "- Number of workers: Parallel data loading\n", - "- Drop last: Discard the last incomplete batch for consistent batch sizes\n", - "- Collate function: Use graph_collate_func to batch variable-sized molecular graphs" - ], - "cell_type": "markdown", - "id": "c09084c0" - }, - { - "metadata": {}, - "source": [ - "from torch.utils.data import DataLoader\n", - "from kale.loaddata.molecular_datasets import graph_collate_func\n", - "from kale.loaddata.sampler import MultiDataLoader\n", - "\n", - "params = {\n", - " \"batch_size\": cfg.SOLVER.BATCH_SIZE,\n", - " \"shuffle\": True,\n", - " \"num_workers\": cfg.SOLVER.NUM_WORKERS,\n", - " \"drop_last\": True,\n", - " \"collate_fn\": graph_collate_func,\n", - "}\n", - "\n", - "params" - ], - "cell_type": "code", - "outputs": [], - "id": "94a15868", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "Then, we create a DataLoader from both the source and target datasets for training." - ], - "cell_type": "markdown", - "id": "e884ed07" - }, - { - "metadata": {}, - "source": [ - "print(\"Using domain adaptation:\", cfg.DA.USE)\n", - "\n", - "if not cfg.DA.USE:\n", - " training_generator = DataLoader(train_dataset, **params)\n", - "else:\n", - " source_generator = DataLoader(train_dataset, **params)\n", - " target_generator = DataLoader(train_target_dataset, **params)\n", - "\n", - " # Get the number of batches in the longer dataset to align both\n", - " n_batches = max(len(source_generator), len(target_generator))\n", - "\n", - " # Combine the source and target data loaders using MultiDataLoader\n", - " training_generator = MultiDataLoader(\n", - " dataloaders=[source_generator, target_generator], n_batches=n_batches\n", - " )" - ], - "cell_type": "code", - "outputs": [], - "id": "24ba12b5", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "Lastly, we set up DataLoaders for validation and testing. Since we don’t want to shuffle or drop any samples, we adjust the parameters accordingly." - ], - "cell_type": "markdown", - "id": "649301de" - }, - { - "metadata": {}, - "source": [ - "# Update parameters for validation/testing (no shuffling, keep all data)\n", - "params.update({\"shuffle\": False, \"drop_last\": False})\n", - "\n", - "# Create validation and test data loaders\n", - "valid_generator = DataLoader(test_target_dataset, **params)\n", - "test_generator = DataLoader(test_target_dataset, **params)" - ], - "cell_type": "code", - "outputs": [], - "id": "b4cf543a", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "### Exercise: Dataset Inspection\n", - "\n", - "Once the dataset is ready, let’s inspect one sample from the training data to check the input graph, protein sequence, and label format." - ], - "cell_type": "markdown", - "id": "e474eea2" - }, - { - "metadata": {}, - "source": [ - "# Get the first batch (contains one batch from source and one from target)\n", - "first_batch = next(iter(training_generator))\n", - "\n", - "# Unpack source and target batches\n", - "source_batch, target_batch = first_batch\n", - "\n", - "# Inspect the first sample from the source batch\n", - "print(\"First sample from source batch:\")\n", - "print(\"Drug graph:\", source_batch[0][0])\n", - "print(\"Protein sequence:\", source_batch[1][0])\n", - "print(\"Label:\", source_batch[2][0])" - ], - "cell_type": "code", - "outputs": [], - "id": "31b8a93f", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "This sample is a tuple with three parts:\n", - "\n", - "1. **Drug Graph**\n", - "- `x=[290, 7]`: Feature matrix with 290 atoms (nodes) and 7 features per atom.\n", - "- `edge_index=[2, 58]`: Shows 146 edges, with source and target node indices.\n", - "- `edge_attr=[58, 1]`: Each edge has 1 bond feature, such as bond type.\n", - "- `num_nodes=290`: Confirms the graph has 290 nodes.\n", - "\n", - "2. **Protein Features (array)**\n", - "- Example values: `[11., 1., 18., ..., 0., 0., 0.]`: A fixed-length numeric array representing the protein sequence. Each position holds an integer-encoded amino acid, with zeros for padding.\n", - "\n", - "3. **Label (float)**\n", - "- `0.0`; The ground-truth interaction label indicating no interaction." - ], - "cell_type": "markdown", - "id": "cb0b269b" - }, - { - "metadata": {}, - "source": [ - "## Step 2: Model Definition" - ], - "cell_type": "markdown", - "id": "8eaf5c8f" - }, - { - "metadata": {}, - "source": [ - "### Embed\n", - "\n", - "DrugBAN consists of three main components: a Graph Convolutional Network (GCN) for extracting structural features from drug molecular graphs, a Convolutional Neural Network (CNN) for encoding protein sequences, and a Bilinear Attention Network (BAN) for fusing drug and protein features. The fused representation is then passed through a Multi-Layer Perceptron (MLP) classifier to predict interaction scores.\n", - "\n", - "We define the DrugBAN class in `kale.embed.ban`." - ], - "cell_type": "markdown", - "id": "b2819549" - }, - { - "metadata": {}, - "source": [ - "from kale.embed.ban import DrugBAN\n", - "\n", - "model = DrugBAN(**cfg)\n", - "print(model)" - ], - "cell_type": "code", - "outputs": [], - "id": "1c8f3acc", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "### Predict\n", - "We use the PyKale pipeline API `kale.pipeline.drugban_trainer` to connect dataloaders, encoders and outcoders for model training and evaluation." - ], - "cell_type": "markdown", - "id": "32084f24" - }, - { - "metadata": {}, - "source": [ - "from kale.pipeline.drugban_trainer import DrugbanTrainer\n", - "\n", - "drugban_trainer = DrugbanTrainer(\n", - " model=DrugBAN(**cfg),\n", - " solver_lr=cfg.SOLVER.LEARNING_RATE,\n", - " num_classes=cfg.DECODER.BINARY,\n", - " batch_size=cfg.SOLVER.BATCH_SIZE,\n", - " is_da=cfg.DA.USE,\n", - " solver_da_lr=cfg.SOLVER.DA_LEARNING_RATE,\n", - " da_init_epoch=cfg.DA.INIT_EPOCH,\n", - " da_method=cfg.DA.METHOD,\n", - " original_random=cfg.DA.ORIGINAL_RANDOM,\n", - " use_da_entropy=cfg.DA.USE_ENTROPY,\n", - " da_random_layer=cfg.DA.RANDOM_LAYER,\n", - " da_random_dim=cfg.DA.RANDOM_DIM,\n", - " decoder_in_dim=cfg.DECODER.IN_DIM,\n", - ")" - ], - "cell_type": "code", - "outputs": [], - "id": "46e2b9b4", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "We want to save the best model during training so we can reuse it later without needing to retrain. PyTorch Lightning’s `ModelCheckpoint` does this by automatically saving the model whenever it achieves a new best validation AUROC score." - ], - "cell_type": "markdown", - "id": "a48c86b9" - }, - { - "metadata": {}, - "source": [ - "import pytorch_lightning as pl\n", - "from pytorch_lightning.callbacks import ModelCheckpoint\n", - "\n", - "checkpoint_callback = ModelCheckpoint(\n", - " filename=\"{epoch}-{step}-{val_BinaryAUROC:.4f}\",\n", - " monitor=\"val_BinaryAUROC\",\n", - " mode=\"max\",\n", - ")" - ], - "cell_type": "code", - "outputs": [], - "id": "7754bd38", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "We now create the `Trainer`." - ], - "cell_type": "markdown", - "id": "969beac0" - }, - { - "metadata": {}, - "source": [ - "import torch\n", - "\n", - "trainer = pl.Trainer(\n", - " callbacks=[checkpoint_callback],\n", - " devices=\"auto\",\n", - " accelerator=\"auto\",\n", - " max_epochs=cfg.SOLVER.MAX_EPOCH,\n", - " deterministic=True,\n", - ")" - ], - "cell_type": "code", - "outputs": [], - "id": "e68e07bc", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "## Step 3: Model Training" - ], - "cell_type": "markdown", - "id": "1f9a4714" - }, - { - "metadata": {}, - "source": [ - "### Train\n", - "\n", - "After setting up the model and data loaders, we now start training the full DrugBAN model using the PyTorch Lightning Trainer via calling `trainer.fit()`.\n", - "\n", - "#### What Happens Here?\n", - "- The model receives batches of drug-protein pairs from the training data loader.\n", - "\n", - "- During each step, the GCN, CNN, BAN layer, and MLP classifier are updated to improve interaction prediction.\n", - "\n", - "- Validation is automatically run at the end of each epoch to track performance and save the best model based on AUROC.\n", - "\n", - "\n", - "This code block takes approximately 5 minutes to complete." - ], - "cell_type": "markdown", - "id": "b72634ee" - }, - { - "metadata": {}, - "source": [ - "trainer.fit(\n", - " drugban_trainer,\n", - " train_dataloaders=training_generator,\n", - " val_dataloaders=valid_generator,\n", - ")" - ], - "cell_type": "code", - "outputs": [], - "id": "0624b0c6", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "## Step 4: Evaluation\n", - "\n", - "Once training is complete, we evaluate the model on the test set using `trainer.test()`.\n", - "\n", - "### What is included in this step?\n", - "- The best model checkpoint (based on validation AUROC) is automatically loaded.\n", - "\n", - "- The model runs on the test data to generate predictions.\n", - "\n", - "- Final classification metrics, including AUROC, F1 score, accuracy, sensitivity, and specificity, are calculated and logged." - ], - "cell_type": "markdown", - "id": "23b3975c" - }, - { - "metadata": {}, - "source": [ - "trainer.test(drugban_trainer, dataloaders=test_generator, ckpt_path=\"best\")" - ], - "cell_type": "code", - "outputs": [], - "id": "c1415c02", - "execution_count": null - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": [ - "### Performance Comparison\n", - "\n", - "The earlier example was a simple demonstration. To properly evaluate DrugBAN against baseline models, we train it for 100 epochs across multiple random seeds.\n", - "\n", - "We provide a checkpoint trained for 100 epochs in the `checkpoint` for your test after the tutorial. We will also use the provided checkpoint for the interpretation section for a better visualization.\n" - ], - "id": "bb0a08bec91d2bd9" - }, - { - "metadata": {}, - "source": [ - "The figure below shows the performance of different models on the BioSNAP and BindingDB datasets:\n", - "- Left plot: AUROC (Area Under the ROC Curve)\n", - "- Right plot: AUPRC (Area Under the Precision–Recall Curve)\n", - "\n", - "![](https://media.springernature.com/full/springer-static/image/art%3A10.1038%2Fs42256-022-00605-1/MediaObjects/42256_2022_605_Fig3_HTML.png?as=webp)\n", - "\n", - "The box plots show the median as the centre lines and the mean as green triangles. The minima and lower percentile represent the worst and second-worst scores. The maxima and upper percentile indicate the best and second-best scores. Supplementary Table 2 provides the data statistics of the BindingDB and BioSNAP datasets." - ], - "cell_type": "markdown", - "id": "37dbe9f3" - }, - { - "metadata": {}, - "source": [ - "## Step 5: Interpretation\n", - "\n", - "We interpret the trained models by analyzing the learned attention weights. In this step, we will use PyKale's API to\n", - "1) draw the attention maps of the Bilinear Attention Network (BAN) layer, and\n", - "2) generate molecule images with attention highlights.\n", - "\n", - "This helps us understand which parts of the drug contribute to the interaction with the target protein." - ], - "cell_type": "markdown", - "id": "02e3c73e" - }, - { - "metadata": {}, - "source": [ - "### Extracting Attention Weights\n", - "First, we need to load the test dataset and create a DataLoader for it. This will allow us to process the test samples in batches. We define functions to create the test dataset and DataLoader." - ], - "cell_type": "markdown", - "id": "4a56f260141b7368" - }, - { - "metadata": {}, - "source": [ - "def get_test_dataset(dataFolder):\n", - " df_test_target = pd.read_csv(dataFolder)\n", - " test_target_dataset = DTIDataset(df_test_target.index.values, df_test_target)\n", - " return test_target_dataset\n", - "\n", - "\n", - "def get_test_dataloader(dataset, batchsize, num_workers, collate_fn):\n", - " test_dataloader = DataLoader(\n", - " dataset,\n", - " batch_size=batchsize,\n", - " num_workers=num_workers,\n", - " collate_fn=collate_fn,\n", - " shuffle=False,\n", - " drop_last=True,\n", - " )\n", - " return test_dataloader" - ], - "cell_type": "code", - "id": "2c67553408592b2", - "outputs": [], - "execution_count": null - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "We load a small subset of samples for testing from the provided `.csv` file. You can create your own `.csv` file with the same format to test your drug–protein pairs.", - "id": "ecdab66ee05da10c" - }, - { - "metadata": {}, - "source": [ - "test_dataFolder = \"/content/drug-target-interaction/data/drug-target-interaction/bindingdb/interpretation_samples.csv\"" - ], - "cell_type": "code", - "outputs": [], - "id": "7ef1867541d2577a", - "execution_count": null - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "We then build the test dataset and DataLoader using the functions defined above. The `batchsize` is set to 1 to ensure we process one sample at a time for attention visualization later.", - "id": "7fec5dc00a7b4aa4" - }, - { - "metadata": {}, - "cell_type": "code", - "outputs": [], - "execution_count": null, - "source": [ - "test_dataset = get_test_dataset(test_dataFolder)\n", - "test_dataloader = get_test_dataloader(\n", - " test_dataset,\n", - " batchsize=1,\n", - " num_workers=cfg.SOLVER.NUM_WORKERS,\n", - " collate_fn=graph_collate_func,\n", - ")" - ], - "id": "c99a558c96a1ffd" - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "Then, we use the following function to load the trained model with the PyKale API.", - "id": "e1ff543d132abc42" - }, - { - "metadata": {}, - "source": [ - "def get_model_from_ckpt(ckpt_path, config):\n", - " return DrugbanTrainer.load_from_checkpoint(\n", - " checkpoint_path=ckpt_path,\n", - " model=DrugBAN(**config),\n", - " solver_lr=config.SOLVER.LEARNING_RATE,\n", - " num_classes=config.DECODER.BINARY,\n", - " batch_size=config.SOLVER.BATCH_SIZE,\n", - " # --- domain adaptation parameters ---\n", - " is_da=config.DA.USE,\n", - " solver_da_lr=config.SOLVER.DA_LEARNING_RATE,\n", - " da_init_epoch=config.DA.INIT_EPOCH,\n", - " da_method=config.DA.METHOD,\n", - " original_random=config.DA.ORIGINAL_RANDOM,\n", - " use_da_entropy=config.DA.USE_ENTROPY,\n", - " da_random_layer=config.DA.RANDOM_LAYER,\n", - " # --- discriminator parameters ---\n", - " da_random_dim=config.DA.RANDOM_DIM,\n", - " decoder_in_dim=config.DECODER.IN_DIM,\n", - " )" - ], - "cell_type": "code", - "outputs": [], - "id": "3b7f12b12b139799", - "execution_count": null - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "Once the model and test data are prepared, we extract attention maps from the trained model. We set the directory to the provided checkpoint file, load the trained model, and set it to evaluation mode.", - "id": "c0678dddcdf076fc" - }, - { - "metadata": {}, - "source": [ - "checkpoint_path = \"/content/drug-target-interaction/checkpoint/best.ckpt\"\n", - "model = get_model_from_ckpt(checkpoint_path, cfg)\n", - "model.model.eval()" - ], - "cell_type": "code", - "outputs": [], - "id": "d2a8931099b73c01", - "execution_count": null - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "We then iterate through the test DataLoader, passing each batch of drug and protein pairs to the model. The model's forward method returns the attention weights. After processing all batches, we concatenate the attention tensors into a single tensor.", - "id": "159d3fa67b29c9e9" - }, - { - "metadata": {}, - "source": [ - "from tqdm import tqdm\n", - "\n", - "all_attentions = []\n", - "for batch in tqdm(test_dataloader):\n", - " drug, protein, _ = batch\n", - " drug, protein = drug.to(model.device), protein.to(model.device)\n", - "\n", - " _, _, _, _, attention = model.model.forward(\n", - " drug, protein, mode=\"eval\"\n", - " ) # [B, H, V, Q]\n", - "\n", - " attention = attention.detach().cpu()\n", - " all_attentions.append(attention)\n", - "\n", - "# Concatenate into one tensor: [N, H, V, Q]\n", - "all_attentions = torch.cat(all_attentions, dim=0)\n", - "torch.save(all_attentions, \"attention_maps.pt\")\n", - "\n", - "all_attentions.shape" - ], - "cell_type": "code", - "outputs": [], - "id": "781a7762c36c72be", - "execution_count": null - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "The attention has shape [B, H, V, Q] (Number of drug-target pairs, Heads of attentions, Drug tokens, Protein tokens).", - "id": "78dc763b6c0eef0" - }, - { - "metadata": {}, - "source": "### Visualize Attention Maps and Molecule Images", - "cell_type": "markdown", - "id": "8f72ea4d93f640cb" - }, - { - "metadata": {}, - "source": "Once attention maps are saved, run the visualization script:", - "cell_type": "markdown", - "id": "383c342a7c31d7ae" - }, - { - "metadata": {}, - "source": [ - "This script will:\n", - "\n", - "1) Load the attention weights and the corresponding SMILES + protein data.\n", - "\n", - "2) Plot:\n", - "\n", - " a) A heatmap of attention over drug–protein tokens.\n", - "\n", - " b) Molecular structures with atoms highlighted by attention values.\n", - "\n", - "The output images are saved in the `visualization` directory. You can also modify the `data_file` to use your own input in the same format as `target_test.csv`.\n", - "\n" - ], - "cell_type": "markdown", - "id": "d8a746169def8da5" - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "We first import the necessary PyKale APIs and set the output directory.", - "id": "aac54bfc67ce32eb" - }, - { - "metadata": {}, - "source": [ - "%pip install nilearn\n", - "from kale.interpret.visualize import draw_attention_map, draw_mol_with_attention\n", - "from kale.prepdata.tensor_reshape import normalize_tensor\n", - "\n", - "out_dir = \"./visualization\"\n", - "os.makedirs(out_dir, exist_ok=True)" - ], - "cell_type": "code", - "outputs": [], - "id": "d3c1d2e4cab69107", - "execution_count": null - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "We then load the attention maps, data, and SMILES strings from the test dataset.", - "id": "126b62034111d92a" - }, - { - "metadata": {}, - "cell_type": "code", - "outputs": [], - "execution_count": null, - "source": [ - "attention = torch.load(\"attention_maps.pt\", map_location=\"cpu\")\n", - "data_df = pd.read_csv(test_dataFolder)\n", - "smiles = data_df[\"SMILES\"]\n", - "proteins = data_df[\"Protein\"]" - ], - "id": "7f70a6810c1c5e60" - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "We select the first sample from the attention maps and corresponding SMILES and protein sequence for visualization.", - "id": "d1a009bbb9f4a0f9" - }, - { - "metadata": {}, - "cell_type": "code", - "outputs": [], - "execution_count": null, - "source": [ - "index = 0\n", - "att_path = os.path.join(out_dir, f\"att_map_{index}.png\")\n", - "mol_path = os.path.join(out_dir, f\"mol_{index}.svg\")" - ], - "id": "e808c255fe862925" - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "We crop the attention map to the actual lengths of the drug and protein sequences. This is important because the attention map may include padding tokens.", - "id": "438e6aa218e6b51d" - }, - { - "metadata": {}, - "cell_type": "code", - "outputs": [], - "execution_count": null, - "source": [ - "from rdkit import Chem\n", - "\n", - "\n", - "def get_real_length(smile, protein_sequence):\n", - " \"\"\"Get the real length of the drug and protein sequences.\"\"\"\n", - " mol = Chem.MolFromSmiles(smile)\n", - " return mol.GetNumAtoms(), len(protein_sequence)\n", - "\n", - "\n", - "att = attention[index] # [H, V, Q]\n", - "smile = smiles[index]\n", - "protein = proteins[index]\n", - "real_drug_len, real_prot_len = get_real_length(smile, protein)\n", - "att = att[:, :real_drug_len, :real_prot_len].mean(0) # [V, Q]\n", - "\n", - "# Normalize\n", - "att = normalize_tensor(att)" - ], - "id": "af15baa1c8caabc0" - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "Finally, we save the attention map and the molecule image with attention highlights.", - "id": "60a4ce71146a721e" - }, - { - "metadata": {}, - "source": [ - "draw_attention_map(\n", - " att,\n", - " att_path,\n", - " title=f\"Drug {index} Attention\",\n", - " xlabel=\"Drug Tokens\",\n", - " ylabel=\"Protein Tokens\",\n", - ")" - ], - "cell_type": "code", - "outputs": [], - "id": "403f77ada0ecc446", - "execution_count": null - }, - { - "metadata": {}, - "source": [ - "draw_mol_with_attention(att.mean(dim=1), smile, mol_path)" - ], - "cell_type": "code", - "outputs": [], - "id": "b1003372361a66d6", - "execution_count": null - }, - { - "metadata": {}, - "cell_type": "markdown", - "source": "The output images are saved in the `visualization` directory. The attention map shows how much each drug token attends to each protein token, while the molecule image highlights the atoms based on their attention values.", - "id": "1999d67d6d14b263" - }, - { - "metadata": {}, - "source": [ - "## Extension Tasks" - ], - "cell_type": "markdown", - "id": "eeb308c3" - }, - { - "metadata": {}, - "source": [ - "### Task 1\n", - "\n", - "To use the BindingDB dataset, modify the relevant line in the Configuration section of Step 0 as shown below.\n", - "\n", - "```python\n", - "cfg.DATA.DATASET = \"bindingdb\"\n", - "```\n", - "\n", - "Reload the dataset and re-run training and testing.\n", - "\n", - "> Tip: See if the model struggles more or less with the new dataset. It can reveal how generalisable DrugBAN is.\n" - ], - "cell_type": "markdown", - "id": "aa2a83d8" - }, - { - "metadata": {}, - "source": [ - "### Task 2\n", - "\n", - "Turn off domain adaptation by updating the config file and re-running training and testing.\n", - "\n", - "Replace `configs/DA_cross_domain.yaml` with `configs/non_DA_cross_domain.yaml` in the Configuration section of Step 0 as shown below.\n", - "\n", - "```python\n", - "cfg.merge_from_file(\"configs/non_DA_cross_domain.yaml\")\n", - "```\n", - ">Tip: Compare the results with and without domain adaptation to see how it affects model performance." - ], - "cell_type": "markdown", - "id": "c94f174c" - } - ] + "cells": [ + { + "cell_type": "markdown", + "id": "8c1bf9c7", + "metadata": { + "id": "8c1bf9c7" + }, + "source": [ + "# Drug–Target Interaction Prediction\n", + "\n", + "![](https://github.com/pykale/mmai-tutorials/blob/main/tutorials/drug-target-interaction/images/drugban-pyakle-api.png?raw=1)\n", + "\n", + "\n", + "In this tutorial, we will train models to predict the interaction between **two data modalities**: **molecules (drug)** and **proteins (target)** using `PyKale`. Drug-target interaction (DTI) plays a key role in drug discovery and identifying potential therapeutic targets. This example is based on the **DrugBAN** framework by [**Bai et al. (_Nature Machine Intelligence_, 2023)**](https://www.nature.com/articles/s42256-022-00605-1).\n", + "\n", + "The DTI prediction problem is formulated as a **binary classification task**, where the goal is to predict whether a given **drug–protein pair interacts or not**. The DrugBAN framework tackles this problem using two key ideas:\n", + "\n", + "- **Bilinear Attention Network (BAN)**, which learns detailed feature representations for both drugs and proteins and captures local interaction patterns between them.\n", + "\n", + "- **Adversarial Domain Adaptation**, which helps the model generalise to out-of-distribution datasets, i.e., in clustering-based cross-validation instead of random splits, improving its ability to predict interactions on unseen drug–target pairs.\n", + "\n", + "With `PyKale`, implementing such a multimodal DTI prediction pipeline is straightforward. The library provides ready-to-use modules and configuration support, making it easy to apply advanced techniques with minimal custom coding." + ] + }, + { + "cell_type": "markdown", + "id": "745ccdcf", + "metadata": { + "id": "745ccdcf" + }, + "source": [ + "## Step 0: Environment Preparation\n", + "\n", + "As a starting point, we will install the required packages and load a set of helper functions to assist throughout this tutorial.\n", + "\n", + "To prepare the helper functions and necessary materials, we download them from the GitHub repository.\n", + "\n", + "Moreover, we provide helper functions that can be inspected directly in the `.py` files located in the notebook's current directory. The additional helper script is:\n", + "- [`config.py`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/drug-target-interaction/configs.py): Defines the base configuration settings, which can be overridden using a custom `.yaml` file." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "a6028209", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "a6028209", + "outputId": "bdba9c0d-e5a6-4dba-981d-b12e21d2c463" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Cloning into 'mmai-tutorials'...\n", + "remote: Enumerating objects: 610, done.\u001b[K\n", + "remote: Counting objects: 100% (242/242), done.\u001b[K\n", + "remote: Compressing objects: 100% (146/146), done.\u001b[K\n", + "remote: Total 610 (delta 156), reused 121 (delta 96), pack-reused 368 (from 2)\u001b[K\n", + "Receiving objects: 100% (610/610), 23.16 MiB | 16.03 MiB/s, done.\n", + "Resolving deltas: 100% (309/309), done.\n", + "mv: cannot move '/content/mmai-tutorials/tutorials/drug-target-interaction' to '/content/drug-target-interaction': Directory not empty\n", + "/content/drug-target-interaction\n", + "Changed working directory to: /content/drug-target-interaction\n" + ] + } + ], + "source": [ + "import os\n", + "\n", + "!rm -rf /content/mmai-tutorials\n", + "!git clone --single-branch -b main https://github.com/pykale/mmai-tutorials.git\n", + "%mv /content/mmai-tutorials/tutorials/drug-target-interaction /content/\n", + "%cd /content/drug-target-interaction\n", + "\n", + "print(\"Changed working directory to:\", os.getcwd())" + ] + }, + { + "cell_type": "markdown", + "id": "c52c6334", + "metadata": { + "id": "c52c6334" + }, + "source": [ + "### Package Installation\n", + "\n", + "The main package required for this tutorial is `PyKale`.\n", + "\n", + "`PyKale` is an open-source interdisciplinary machine learning library developed at the University of Sheffield, with a focus on applications in biomedical and scientific domains.\n", + "\n", + "Then, we install `PyG` (PyTorch Geometric) and related packages.\n", + "\n", + "Please **do not** re-run this session after installation completed. Runing this installation multiple times will trigger issues related to `PyG`. If you want to re-run this installation, please click the `Runtime` on the top menu and choose `Disconnect and delete runtime` before installing." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "53e3b14e", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "53e3b14e", + "outputId": "7890c8c4-6c6c-4156-bdf4-55ffbf9ee2b5" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n", + " Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n", + " Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n", + "pykale, gdown, pyg and yacs installed successfully ✅\n" + ] + } + ], + "source": [ + "%pip install --quiet \\\n", + " \"pykale[example]@git+https://github.com/pykale/pykale@main\" \\\n", + " gdown==5.2.0 torch-geometric==2.6.0 torch_sparse torch_scatter \\\n", + " -f https://data.pyg.org/whl/torch-2.6.0+cu124.html \\\n", + " && echo \"pykale, gdown, pyg and yacs installed successfully ✅\" \\\n", + " || echo \"Failed to install pykale, gdown, pyg and yacs ❌\"" + ] + }, + { + "cell_type": "markdown", + "id": "69f50b6a", + "metadata": { + "id": "69f50b6a" + }, + "source": [ + "We then hide the warnings messages to get a clear output." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "6e871c63", + "metadata": { + "id": "6e871c63" + }, + "outputs": [], + "source": [ + "import os\n", + "import warnings\n", + "\n", + "warnings.filterwarnings(\"ignore\")\n", + "os.environ[\"PYTHONWARNINGS\"] = \"ignore\"" + ] + }, + { + "cell_type": "markdown", + "id": "6606e3fb", + "metadata": { + "id": "6606e3fb" + }, + "source": [ + "Exercise: Check NumPy Version" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "0d384020", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "0d384020", + "outputId": "d1ecfffa-1567-4b1c-dc8e-cbbeda3728c0" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "NumPy version: 2.0.2\n" + ] + } + ], + "source": [ + "import numpy as np\n", + "\n", + "print(\"NumPy version:\", np.__version__) # numpy should be 2.0.0 or higher" + ] + }, + { + "cell_type": "markdown", + "id": "cabd3406", + "metadata": { + "id": "cabd3406" + }, + "source": [ + "### Configuration\n", + "\n", + "To minimize the footprint of the notebook when specifying configurations, we provide a [`config.py`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/drug-target-interaction/configs.py) file that defines default parameters. These can be customized by supplying a `.yaml` configuration file, such as [`configs/DA_cross_domain.yaml`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/drug-target-interaction/configs/DA_cross_domain.yaml) as an example." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "55c13b48", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "55c13b48", + "outputId": "dd3df032-9263-4bcf-d47c-3059d1f66830" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "/content/drug-target-interaction\n" + ] + } + ], + "source": [ + "from configs import get_cfg_defaults\n", + "\n", + "%cd /content/drug-target-interaction\n", + "\n", + "cfg = get_cfg_defaults() # Load the default settings from config.py\n", + "cfg.merge_from_file(\n", + " \"configs/DA_cross_domain.yaml\"\n", + ") # Update (or override) some of those settings using a custom YAML file" + ] + }, + { + "cell_type": "markdown", + "id": "74ffdbc2", + "metadata": { + "id": "74ffdbc2" + }, + "source": [ + "In this tutorial, we list the hyperparameters we would like users to play with outside the `.yaml` file:\n", + "- `cfg.SOLVER.MAX_EPOCH`: Number of epochs in training stage. You can reduce the number of training epochs to shorten runtime.\n", + "- `cfg.DATA.DATASET`: The dataset used in the study. This can be `bindingdb` or `biosnap`.\n", + "\n", + "As a quick exercise, please take a moment to review and understand the parameters in [`config.py`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/drug-target-interaction/configs.py)." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "424c7286", + "metadata": { + "id": "424c7286" + }, + "outputs": [], + "source": [ + "cfg.SOLVER.MAX_EPOCH = 2" + ] + }, + { + "cell_type": "markdown", + "id": "97c088fd", + "metadata": { + "id": "97c088fd" + }, + "source": [ + "You can also switch to a different dataset." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "c69376fa", + "metadata": { + "id": "c69376fa" + }, + "outputs": [], + "source": [ + "cfg.DATA.DATASET = \"biosnap\"" + ] + }, + { + "cell_type": "markdown", + "id": "d3d41633", + "metadata": { + "id": "d3d41633" + }, + "source": [ + "Exercise: Now print the full configuration to check all current hyperparameter and dataset settings." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "45874296", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "45874296", + "outputId": "5c4738ff-85e8-463f-dac7-6e183643a223" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "BCN:\n", + " HEADS: 2\n", + "COMET:\n", + " API_KEY: \n", + " EXPERIMENT_NAME: DA_cross_domain\n", + " PROJECT_NAME: drugban-23-May\n", + " TAG: DrugBAN_CDAN\n", + " USE: False\n", + "DA:\n", + " INIT_EPOCH: 10\n", + " LAMB_DA: 1\n", + " METHOD: CDAN\n", + " ORIGINAL_RANDOM: True\n", + " RANDOM_DIM: 256\n", + " RANDOM_LAYER: True\n", + " TASK: True\n", + " USE: True\n", + " USE_ENTROPY: False\n", + "DATA:\n", + " DATASET: biosnap\n", + " SPLIT: cluster\n", + "DECODER:\n", + " BINARY: 2\n", + " HIDDEN_DIM: 512\n", + " IN_DIM: 256\n", + " NAME: MLP\n", + " OUT_DIM: 128\n", + "DRUG:\n", + " HIDDEN_LAYERS: [128, 128, 128]\n", + " MAX_NODES: 290\n", + " NODE_IN_EMBEDDING: 128\n", + " NODE_IN_FEATS: 7\n", + " PADDING: True\n", + "PROTEIN:\n", + " EMBEDDING_DIM: 128\n", + " KERNEL_SIZE: [3, 6, 9]\n", + " NUM_FILTERS: [128, 128, 128]\n", + " PADDING: True\n", + "RESULT:\n", + " SAVE_MODEL: True\n", + "SOLVER:\n", + " BATCH_SIZE: 32\n", + " DA_LEARNING_RATE: 5e-05\n", + " LEARNING_RATE: 0.0001\n", + " MAX_EPOCH: 2\n", + " NUM_WORKERS: 0\n", + " SEED: 20\n" + ] + } + ], + "source": [ + "print(cfg)" + ] + }, + { + "cell_type": "markdown", + "id": "17558d0c", + "metadata": { + "id": "17558d0c" + }, + "source": [ + "## Step 1: Data Loading and Preparation\n", + "\n", + "In this tutorial, we use the **Biosnap** dataset for the main demonstration and the **BindingDB** dataset for the exercise at the end." + ] + }, + { + "cell_type": "markdown", + "id": "6c6071b9", + "metadata": { + "id": "6c6071b9" + }, + "source": [ + "### Data Downloading\n", + "\n", + "Please run the following cell to download necessary datasets." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "56f9f58e", + "metadata": { + "id": "56f9f58e" + }, + "outputs": [], + "source": [ + "!rm -rf data\n", + "!mkdir data\n", + "!cd data\n", + "\n", + "!pip install -q gdown\n", + "!gdown --id 1ogOcxZn-1q418LOT-gQ94aHQV0Y1sOmk --output data/drug-target-interaction.zip\n", + "!unzip data/drug-target-interaction.zip -d data/\n", + "!mv data/drug-target-interaction/checkpoint ./" + ] + }, + { + "cell_type": "markdown", + "id": "c39b3e39", + "metadata": { + "id": "c39b3e39" + }, + "source": [ + "Exercise: Check the data is ready" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "a6258d1f", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "a6258d1f", + "outputId": "2813ad1f-5971-44ca-f9b6-c90a496fe4b6" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Contents of the data folder:\n", + "biosnap\n", + "bindingdb\n", + "checkpoint\n" + ] + } + ], + "source": [ + "import os\n", + "import shutil\n", + "\n", + "print(\"Contents of the data folder:\")\n", + "for item in os.listdir(\"data/drug-target-interaction\"):\n", + " print(item)" + ] + }, + { + "cell_type": "markdown", + "id": "9ab0b5f833dc40f8", + "metadata": { + "id": "9ab0b5f833dc40f8" + }, + "source": [ + "The data content is structured as follows:\n", + "```sh\n", + " ├───data\n", + " │ ├───checkpoint\n", + " │ ├───bindingdb\n", + " │ ├───biosnap" + ] + }, + { + "cell_type": "markdown", + "id": "5be1dcc62b7d5649", + "metadata": { + "id": "5be1dcc62b7d5649" + }, + "source": [ + "The `data` folder contains two datasets: `bindingdb` and `biosnap`. Each dataset folder contains the following files. The `checkpoint` folder contains the saved model checkpoint, which are used later in the interpretation section." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "a93303c51c8b974e", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "a93303c51c8b974e", + "outputId": "93e0c029-b489-4f4d-8889-8b99f868039c" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Contents of bindingdb folder:\n", + "random\n", + "full.csv\n", + "interpretation_samples.csv\n", + "cluster\n" + ] + } + ], + "source": [ + "print(\"Contents of bindingdb folder:\")\n", + "for item in os.listdir(\"data/drug-target-interaction/bindingdb\"):\n", + " print(item)" + ] + }, + { + "cell_type": "markdown", + "id": "79cbc1c1", + "metadata": { + "id": "79cbc1c1" + }, + "source": [ + "Each dataset folder follows the structure:\n", + "\n", + "```sh\n", + " ├───dataset_name\n", + " │ ├───cluster\n", + " │ │ ├───source_train.csv\n", + " │ │ ├───target_train.csv\n", + " │ │ ├───target_test.csv\n", + " │ ├───random\n", + " │ │ ├───test.csv\n", + " │ │ ├───train.csv\n", + " │ │ ├───val.csv\n", + " │ ├───full.csv\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "d35e04f9", + "metadata": { + "id": "d35e04f9" + }, + "source": [ + "We use the cluster dataset folder for cross-domain prediction, containing three parts:\n", + "\n", + "- Train samples from the source domain: Drug–protein pairs the model learns from.\n", + "\n", + "- Train samples from the target domain: Additional training data from a different distribution to improve generalisation.\n", + "\n", + "- Test samples from the target domain: Unseen drug–protein pairs used to evaluate model performance on new data.\n", + "\n", + "The source and target sets are defined based on the clustering results." + ] + }, + { + "cell_type": "markdown", + "id": "98acf744", + "metadata": { + "id": "98acf744" + }, + "source": [ + "### Data Loading" + ] + }, + { + "cell_type": "markdown", + "id": "1e5f4f44", + "metadata": { + "id": "1e5f4f44" + }, + "source": [ + "Here’s what each csv file looks like in a table format:\n", + "\n", + "| SMILES | Protein Sequence | Y |\n", + "|--------------------|--------------------------|---|\n", + "| Fc1ccc(C2(COC…) | MDNVLPVDSDLS… | 1 |\n", + "| O=c1oc2c(O)c(…) | MMYSKLLTLTTL… | 0 |\n", + "| CC(C)Oc1cc(N…) | MGMACLTMTEME… | 1 |\n", + "\n", + "Each row of the dataset contains three key pieces of information:\n", + "\n", + "**Drugs**: \n", + "Drugs are often written as SMILES strings, which are like chemical formulas in text format (for example, `\"CC(=O)OC1=CC=CC=C1C(=O)O\"` is aspirin). \n", + "\n", + "\n", + "**Protein Sequence** \n", + "This is a string of letters where each letter stands for an amino acid, the building blocks of proteins. For example, `MGYTSLLT...` is a short protein sequence.\n", + "\n", + "\n", + "**Y (Labels)**: \n", + "Each drug–protein pair is given a label:\n", + "- `1` if they interact\n", + "- `0` if they do not\n", + "\n", + "\n", + "Each row shows one drug–protein pair. The goal of our machine learning model is to predict the last column (**Y**) — whether or not the drug and protein interact." + ] + }, + { + "cell_type": "markdown", + "id": "b7590daf", + "metadata": { + "id": "b7590daf" + }, + "source": [ + "You can load CSV files into Python using tools like `pandas`. The output shows a sample of the data, including the SMILES string for the drug, the protein sequence, the interaction label (Y) and the cluster ID." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "0c709e31", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "0c709e31", + "outputId": "5e663c15-f59f-4a0f-acc3-b545b6392a1e" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Sample example: SMILES CC1=CN=C2N1C=CN=C2NCC1=CC=NC=C1\n", + "Protein MARSLLLPLQILLLSLALETAGEEAQGDKIIDGAPCARGSHPWQVA...\n", + "Y 0.0\n", + "drug_cluster 1904\n", + "target_cluster 1528\n", + "Name: 0, dtype: object\n" + ] + } + ], + "source": [ + "import pandas as pd\n", + "\n", + "dataFolder = os.path.join(\n", + " f\"data/drug-target-interaction/{cfg.DATA.DATASET}\", str(cfg.DATA.SPLIT)\n", + ")\n", + "\n", + "df_train_source = pd.read_csv(os.path.join(dataFolder, \"source_train.csv\"))\n", + "df_train_target = pd.read_csv(os.path.join(dataFolder, \"target_train.csv\"))\n", + "df_test_target = pd.read_csv(os.path.join(dataFolder, \"target_test.csv\"))\n", + "\n", + "print(\"Sample example:\", df_train_source.iloc[0])" + ] + }, + { + "cell_type": "markdown", + "id": "542d4e69", + "metadata": { + "id": "542d4e69" + }, + "source": [ + "### Data Preprocessing\n", + "\n", + "We convert drug SMILES strings into molecular graphs using `kale.loaddata.molecular_datasets.smiles_to_graph`, encoding atom-level features as node attributes and bond types as edges.\n", + "\n", + "\n", + "Protein sequences are transformed into fixed-length integer arrays using `kale.prepdata.chem_transform.integer_label_protein`, with each amino acid mapped to an integer and sequences padded or truncated to a uniform length.\n", + "\n", + "Finally, the `kale.loaddata.molecular_datasets.DTIDataset` class packages drugs, proteins, and labels into a PyTorch-ready dataset." + ] + }, + { + "cell_type": "markdown", + "id": "981d5520", + "metadata": { + "id": "981d5520" + }, + "source": [ + "**Note:** If you encounter an error related to requiring numpy `<2.0`, simply ignore it and re-run this block until it completes successfully." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "ae5af8eb", + "metadata": { + "id": "ae5af8eb" + }, + "outputs": [], + "source": [ + "from kale.loaddata.molecular_datasets import DTIDataset\n", + "\n", + "# Create preprocessed datasets\n", + "train_dataset = DTIDataset(df_train_source.index.values, df_train_source)\n", + "train_target_dataset = DTIDataset(df_train_target.index.values, df_train_target)\n", + "test_target_dataset = DTIDataset(df_test_target.index.values, df_test_target)" + ] + }, + { + "cell_type": "markdown", + "id": "a0a510ce", + "metadata": { + "id": "a0a510ce" + }, + "source": [ + "We load data in small, manageable pieces called batches to save memory and speed up training. We use `kale.loaddata.sampler.MultiDataLoader` from PyKale to load one batch from the source domain and one from the target domain at each training step." + ] + }, + { + "cell_type": "markdown", + "id": "c09084c0", + "metadata": { + "id": "c09084c0" + }, + "source": [ + "First, we specify a few DataLoader parameters:\n", + "- Batch size: Number of samples per batch\n", + "- Shuffle: Randomly shuffle data\n", + "- Number of workers: Parallel data loading\n", + "- Drop last: Discard the last incomplete batch for consistent batch sizes\n", + "- Collate function: Use graph_collate_func to batch variable-sized molecular graphs" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "94a15868", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "94a15868", + "outputId": "a4c14890-db12-45b7-bcbe-1a94b4f846b2" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'batch_size': 32,\n", + " 'shuffle': True,\n", + " 'num_workers': 0,\n", + " 'drop_last': True,\n", + " 'collate_fn': }" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from torch.utils.data import DataLoader\n", + "from kale.loaddata.molecular_datasets import graph_collate_func\n", + "from kale.loaddata.sampler import MultiDataLoader\n", + "\n", + "params = {\n", + " \"batch_size\": cfg.SOLVER.BATCH_SIZE,\n", + " \"shuffle\": True,\n", + " \"num_workers\": cfg.SOLVER.NUM_WORKERS,\n", + " \"drop_last\": True,\n", + " \"collate_fn\": graph_collate_func,\n", + "}\n", + "\n", + "params" + ] + }, + { + "cell_type": "markdown", + "id": "e884ed07", + "metadata": { + "id": "e884ed07" + }, + "source": [ + "Then, we create a DataLoader from both the source and target datasets for training." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "24ba12b5", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "24ba12b5", + "outputId": "47a5c085-d2d0-48f8-cad2-4483e3fe2efa" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Using domain adaptation: True\n" + ] + } + ], + "source": [ + "print(\"Using domain adaptation:\", cfg.DA.USE)\n", + "\n", + "if not cfg.DA.USE:\n", + " training_generator = DataLoader(train_dataset, **params)\n", + "else:\n", + " source_generator = DataLoader(train_dataset, **params)\n", + " target_generator = DataLoader(train_target_dataset, **params)\n", + "\n", + " # Get the number of batches in the longer dataset to align both\n", + " n_batches = max(len(source_generator), len(target_generator))\n", + "\n", + " # Combine the source and target data loaders using MultiDataLoader\n", + " training_generator = MultiDataLoader(\n", + " dataloaders=[source_generator, target_generator], n_batches=n_batches\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "649301de", + "metadata": { + "id": "649301de" + }, + "source": [ + "Lastly, we set up DataLoaders for validation and testing. Since we don’t want to shuffle or drop any samples, we adjust the parameters accordingly." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "b4cf543a", + "metadata": { + "id": "b4cf543a" + }, + "outputs": [], + "source": [ + "# Update parameters for validation/testing (no shuffling, keep all data)\n", + "params.update({\"shuffle\": False, \"drop_last\": False})\n", + "\n", + "# Create validation and test data loaders\n", + "valid_generator = DataLoader(test_target_dataset, **params)\n", + "test_generator = DataLoader(test_target_dataset, **params)" + ] + }, + { + "cell_type": "markdown", + "id": "e474eea2", + "metadata": { + "id": "e474eea2" + }, + "source": [ + "### Exercise: Dataset Inspection\n", + "\n", + "Once the dataset is ready, let’s inspect one sample from the training data to check the input graph, protein sequence, and label format." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "31b8a93f", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "31b8a93f", + "outputId": "74ae5660-3d74-4c5e-f5ed-065c1f099516" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "First sample from source batch:\n", + "Drug graph: Data(x=[290, 7], edge_index=[2, 106], edge_attr=[106, 1], num_nodes=290)\n", + "Protein sequence: tensor([11., 7., 18., ..., 0., 0., 0.], dtype=torch.float64)\n", + "Label: tensor(0., dtype=torch.float64)\n" + ] + } + ], + "source": [ + "# Get the first batch (contains one batch from source and one from target)\n", + "first_batch = next(iter(training_generator))\n", + "\n", + "# Unpack source and target batches\n", + "source_batch, target_batch = first_batch\n", + "\n", + "# Inspect the first sample from the source batch\n", + "print(\"First sample from source batch:\")\n", + "print(\"Drug graph:\", source_batch[0][0])\n", + "print(\"Protein sequence:\", source_batch[1][0])\n", + "print(\"Label:\", source_batch[2][0])" + ] + }, + { + "cell_type": "markdown", + "id": "cb0b269b", + "metadata": { + "id": "cb0b269b" + }, + "source": [ + "This sample is a tuple with three parts:\n", + "\n", + "1. **Drug Graph**\n", + "- `x=[290, 7]`: Feature matrix with 290 atoms (nodes) and 7 features per atom.\n", + "- `edge_index=[2, 58]`: Shows 146 edges, with source and target node indices.\n", + "- `edge_attr=[58, 1]`: Each edge has 1 bond feature, such as bond type.\n", + "- `num_nodes=290`: Confirms the graph has 290 nodes.\n", + "\n", + "2. **Protein Features (array)**\n", + "- Example values: `[11., 1., 18., ..., 0., 0., 0.]`: A fixed-length numeric array representing the protein sequence. Each position holds an integer-encoded amino acid, with zeros for padding.\n", + "\n", + "3. **Label (float)**\n", + "- `0.0`; The ground-truth interaction label indicating no interaction." + ] + }, + { + "cell_type": "markdown", + "id": "8eaf5c8f", + "metadata": { + "id": "8eaf5c8f" + }, + "source": [ + "## Step 2: Model Definition" + ] + }, + { + "cell_type": "markdown", + "id": "b2819549", + "metadata": { + "id": "b2819549" + }, + "source": [ + "### Embed\n", + "\n", + "DrugBAN consists of three main components: a Graph Convolutional Network (GCN) for extracting structural features from drug molecular graphs, a Convolutional Neural Network (CNN) for encoding protein sequences, and a Bilinear Attention Network (BAN) for fusing drug and protein features. The fused representation is then passed through a Multi-Layer Perceptron (MLP) classifier to predict interaction scores.\n", + "\n", + "We define the DrugBAN class in `kale.embed.ban`." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "1c8f3acc", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "1c8f3acc", + "outputId": "65f82225-7219-4391-abf2-2194a00c3af0" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "DrugBAN(\n", + " (drug_extractor): MolecularGCN(\n", + " (init_transform): Linear(in_features=7, out_features=128, bias=False)\n", + " (gcn_layers): ModuleList(\n", + " (0-2): 3 x GCNConv(128, 128)\n", + " )\n", + " )\n", + " (protein_extractor): ProteinCNN(\n", + " (embedding): Embedding(26, 128, padding_idx=0)\n", + " (conv1): Conv1d(128, 128, kernel_size=(3,), stride=(1,))\n", + " (bn1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " (conv2): Conv1d(128, 128, kernel_size=(6,), stride=(1,))\n", + " (bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " (conv3): Conv1d(128, 128, kernel_size=(9,), stride=(1,))\n", + " (bn3): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " )\n", + " (bcn): BANLayer(\n", + " (v_net): FCNet(\n", + " (main): Sequential(\n", + " (0): Dropout(p=0.2, inplace=False)\n", + " (1): Linear(in_features=128, out_features=768, bias=True)\n", + " (2): ReLU()\n", + " )\n", + " )\n", + " (q_net): FCNet(\n", + " (main): Sequential(\n", + " (0): Dropout(p=0.2, inplace=False)\n", + " (1): Linear(in_features=128, out_features=768, bias=True)\n", + " (2): ReLU()\n", + " )\n", + " )\n", + " (p_net): AvgPool1d(kernel_size=(3,), stride=(3,), padding=(0,))\n", + " (bn): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " )\n", + " (mlp_classifier): MLPDecoder(\n", + " (fc1): Linear(in_features=256, out_features=512, bias=True)\n", + " (bn1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " (fc2): Linear(in_features=512, out_features=512, bias=True)\n", + " (bn2): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " (fc3): Linear(in_features=512, out_features=128, bias=True)\n", + " (bn3): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " (fc4): Linear(in_features=128, out_features=2, bias=True)\n", + " )\n", + ")\n" + ] + } + ], + "source": [ + "from kale.embed.ban import DrugBAN\n", + "\n", + "model = DrugBAN(**cfg)\n", + "print(model)" + ] + }, + { + "cell_type": "markdown", + "id": "32084f24", + "metadata": { + "id": "32084f24" + }, + "source": [ + "### Predict\n", + "We use the PyKale pipeline API `kale.pipeline.drugban_trainer` to connect dataloaders, encoders and outcoders for model training and evaluation." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "46e2b9b4", + "metadata": { + "id": "46e2b9b4" + }, + "outputs": [], + "source": [ + "from kale.pipeline.drugban_trainer import DrugbanTrainer\n", + "\n", + "drugban_trainer = DrugbanTrainer(\n", + " model=DrugBAN(**cfg),\n", + " solver_lr=cfg.SOLVER.LEARNING_RATE,\n", + " num_classes=cfg.DECODER.BINARY,\n", + " batch_size=cfg.SOLVER.BATCH_SIZE,\n", + " is_da=cfg.DA.USE,\n", + " solver_da_lr=cfg.SOLVER.DA_LEARNING_RATE,\n", + " da_init_epoch=cfg.DA.INIT_EPOCH,\n", + " da_method=cfg.DA.METHOD,\n", + " original_random=cfg.DA.ORIGINAL_RANDOM,\n", + " use_da_entropy=cfg.DA.USE_ENTROPY,\n", + " da_random_layer=cfg.DA.RANDOM_LAYER,\n", + " da_random_dim=cfg.DA.RANDOM_DIM,\n", + " decoder_in_dim=cfg.DECODER.IN_DIM,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "a48c86b9", + "metadata": { + "id": "a48c86b9" + }, + "source": [ + "We want to save the best model during training so we can reuse it later without needing to retrain. PyTorch Lightning’s `ModelCheckpoint` does this by automatically saving the model whenever it achieves a new best validation AUROC score." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "7754bd38", + "metadata": { + "id": "7754bd38" + }, + "outputs": [], + "source": [ + "import pytorch_lightning as pl\n", + "from pytorch_lightning.callbacks import ModelCheckpoint\n", + "\n", + "checkpoint_callback = ModelCheckpoint(\n", + " filename=\"{epoch}-{step}-{val_BinaryAUROC:.4f}\",\n", + " monitor=\"val_BinaryAUROC\",\n", + " mode=\"max\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "969beac0", + "metadata": { + "id": "969beac0" + }, + "source": [ + "We now create the `Trainer`." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "e68e07bc", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "e68e07bc", + "outputId": "08a9b744-3fb5-48c7-863f-dd64afc4dd80" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:pytorch_lightning.utilities.rank_zero:GPU available: True (cuda), used: True\n", + "INFO:pytorch_lightning.utilities.rank_zero:TPU available: False, using: 0 TPU cores\n", + "INFO:pytorch_lightning.utilities.rank_zero:HPU available: False, using: 0 HPUs\n" + ] + } + ], + "source": [ + "import torch\n", + "\n", + "trainer = pl.Trainer(\n", + " callbacks=[checkpoint_callback],\n", + " devices=\"auto\",\n", + " accelerator=\"auto\",\n", + " max_epochs=cfg.SOLVER.MAX_EPOCH,\n", + " deterministic=True,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "1f9a4714", + "metadata": { + "id": "1f9a4714" + }, + "source": [ + "## Step 3: Model Training" + ] + }, + { + "cell_type": "markdown", + "id": "b72634ee", + "metadata": { + "id": "b72634ee" + }, + "source": [ + "### Train\n", + "\n", + "After setting up the model and data loaders, we now start training the full DrugBAN model using the PyTorch Lightning Trainer via calling `trainer.fit()`.\n", + "\n", + "#### What Happens Here?\n", + "- The model receives batches of drug-protein pairs from the training data loader.\n", + "\n", + "- During each step, the GCN, CNN, BAN layer, and MLP classifier are updated to improve interaction prediction.\n", + "\n", + "- Validation is automatically run at the end of each epoch to track performance and save the best model based on AUROC.\n", + "\n", + "\n", + "This code block takes approximately 5 minutes to complete." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "0624b0c6", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 424, + "referenced_widgets": [ + "bc415b5a5635482eb20a65601866febf", + "98992d70c9d74d3fa794edf0dc9333b1", + "32de4bd89e034c35adc198247950d4bf", + "12c8f4410cc74d4f822c779c724bce94", + "fe6bdb21df9c4415b9899aaa96969502", + "65bb75f11b1f4064b715a166bed1e215", + "a893676474004fb0a24023403f5acf46", + "941d8002127f499c9868abffea2a2429", + "2fc478f35c3e48f2b0c13bd9a73f3dc6", + "79368bfcf5cc4067a59a46c242a77e2d", + "8632ffdd1b5844f183cc28e824cd117e", + "9a8deecaaef543fbb18d0f50b83b5abc", + "b113d7ba3f50417fb93de1118b4e4dec", + "4b0bcd88167b469da36e8433a4d47377", + "1ed13da64943461ab42ab18495a6246b", + "7d4fd3d9c5ed4cc6af20be7384a758ac", + "635b3ed7aa264fa6907187152359a63f", + "9d5ebfa060ac49d6bd7a8c4837b4fc29", + "a80aad59381c42edaa2adb89d781ddd6", + "b71c219972474c2c89ed04fa48ec637d", + "8f16c1ee593d406c88c9bfcf32fdd3df", + "16cdd1f5e14e405490575177c52f4408", + "07d966a71f604f63b00707e6d3a0bfe6", + "37f06572693f4972b468c3997fd0687c", + "a0cec4295099427bb8e97229bd49d620", + "931700f44cb0491ca187cbc58dc67476", + "dd7ad2a05e22470ab00e2118ff994147", + "1aa1b891fe6b447586d3e87ee62768a2", + "8618e343c4f1435ebf099bd70605606b", + "33a0e7cd7f7d49ffa57e962ca2bf0b66", + "f223529cf08b4235b8e74ad1049c5a60", + "304a1cadd7c048a2a34568301fb1c4dd", + "e809f1b951f740ddabc3a3e56d4dd903", + "f60cfca35cab44e5801bbb2b04f9cc6f", + "2f468dcdec6d4c6db229fcab061fd0d8", + "530bec1df42241a9be6a34c561be4a36", + "700bde0d67f64f9d8680c483d232fa5b", + "abf0b363155c4b1fb8de818ae41606eb", + "301141b0856941cba6a0a1f496267c4e", + "b4f64adc9af34262bb35ea42f0f58899", + "3c34cab162424e51b4dcf0008c39c7a0", + "b2e51980648f49f2963ba66745b82b57", + "b9b44b6940884c359ebb60834280b401", + "807b512710c848eabb6816cf8e9ecc75" + ] + }, + "id": "0624b0c6", + "outputId": "8a7fb16d-707f-4a4e-edd8-dec451f5fa5f" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:pytorch_lightning.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", + "INFO:pytorch_lightning.callbacks.model_summary:\n", + " | Name | Type | Params | Mode \n", + "---------------------------------------------------------------------\n", + "0 | model | DrugBAN | 1.0 M | train\n", + "1 | domain_discriminator | DomainNetSmallImage | 133 K | train\n", + "2 | random_layer | RandomLayer | 66.0 K | train\n", + "3 | valid_metrics | MetricCollection | 0 | train\n", + "4 | test_metrics | MetricCollection | 0 | train\n", + "---------------------------------------------------------------------\n", + "1.2 M Trainable params\n", + "0 Non-trainable params\n", + "1.2 M Total params\n", + "4.847 Total estimated model params size (MB)\n", + "64 Modules in train mode\n", + "0 Modules in eval mode\n" + ] + }, + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "bc415b5a5635482eb20a65601866febf", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Sanity Checking: | | 0/? [00:00┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n", + "┃ Test metric ┃ DataLoader 0 ┃\n", + "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n", + "│ test_BinaryAUROC │ 0.48449382185935974 │\n", + "│ test_BinaryAccuracy │ 0.5071665048599243 │\n", + "│ test_BinaryF1Score │ 0.0933062881231308 │\n", + "│ test_BinaryRecall │ 0.050549451261758804 │\n", + "│ test_BinarySpecificity │ 0.9668141603469849 │\n", + "│ test_accuracy_sklearn │ 0.5038588643074036 │\n", + "│ test_auroc_sklearn │ 0.48449382185935974 │\n", + "│ test_f1_sklearn │ 0.6671618223190308 │\n", + "│ test_loss │ 0.8901852369308472 │\n", + "│ test_optim_threshold │ 0.07649494707584381 │\n", + "│ test_sensitivity │ 0.006637168116867542 │\n", + "│ test_specificity │ 0.997802197933197 │\n", + "└───────────────────────────┴───────────────────────────┘\n", + "\n" + ], + "text/plain": [ + "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n", + "┃\u001b[1m \u001b[0m\u001b[1m Test metric \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m DataLoader 0 \u001b[0m\u001b[1m \u001b[0m┃\n", + "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n", + "│\u001b[36m \u001b[0m\u001b[36m test_BinaryAUROC \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.48449382185935974 \u001b[0m\u001b[35m \u001b[0m│\n", + "│\u001b[36m \u001b[0m\u001b[36m test_BinaryAccuracy \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.5071665048599243 \u001b[0m\u001b[35m \u001b[0m│\n", + "│\u001b[36m \u001b[0m\u001b[36m test_BinaryF1Score \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.0933062881231308 \u001b[0m\u001b[35m \u001b[0m│\n", + "│\u001b[36m \u001b[0m\u001b[36m test_BinaryRecall \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.050549451261758804 \u001b[0m\u001b[35m \u001b[0m│\n", + "│\u001b[36m \u001b[0m\u001b[36m test_BinarySpecificity \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.9668141603469849 \u001b[0m\u001b[35m \u001b[0m│\n", + "│\u001b[36m \u001b[0m\u001b[36m test_accuracy_sklearn \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.5038588643074036 \u001b[0m\u001b[35m \u001b[0m│\n", + "│\u001b[36m \u001b[0m\u001b[36m test_auroc_sklearn \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.48449382185935974 \u001b[0m\u001b[35m \u001b[0m│\n", + "│\u001b[36m \u001b[0m\u001b[36m test_f1_sklearn \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.6671618223190308 \u001b[0m\u001b[35m \u001b[0m│\n", + "│\u001b[36m \u001b[0m\u001b[36m test_loss \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.8901852369308472 \u001b[0m\u001b[35m \u001b[0m│\n", + "│\u001b[36m \u001b[0m\u001b[36m test_optim_threshold \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.07649494707584381 \u001b[0m\u001b[35m \u001b[0m│\n", + "│\u001b[36m \u001b[0m\u001b[36m test_sensitivity \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.006637168116867542 \u001b[0m\u001b[35m \u001b[0m│\n", + "│\u001b[36m \u001b[0m\u001b[36m test_specificity \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.997802197933197 \u001b[0m\u001b[35m \u001b[0m│\n", + "└───────────────────────────┴───────────────────────────┘\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "[{'test_loss': 0.8901852369308472,\n", + " 'test_auroc_sklearn': 0.48449382185935974,\n", + " 'test_accuracy_sklearn': 0.5038588643074036,\n", + " 'test_f1_sklearn': 0.6671618223190308,\n", + " 'test_optim_threshold': 0.07649494707584381,\n", + " 'test_sensitivity': 0.006637168116867542,\n", + " 'test_specificity': 0.997802197933197,\n", + " 'test_BinaryAUROC': 0.48449382185935974,\n", + " 'test_BinaryF1Score': 0.0933062881231308,\n", + " 'test_BinaryRecall': 0.050549451261758804,\n", + " 'test_BinarySpecificity': 0.9668141603469849,\n", + " 'test_BinaryAccuracy': 0.5071665048599243}]" + ] + }, + "execution_count": 23, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "trainer.test(drugban_trainer, dataloaders=test_generator, ckpt_path=\"best\")" + ] + }, + { + "cell_type": "markdown", + "id": "bb0a08bec91d2bd9", + "metadata": { + "id": "bb0a08bec91d2bd9" + }, + "source": [ + "### Performance Comparison\n", + "\n", + "The earlier example was a simple demonstration. To properly evaluate DrugBAN against baseline models, we train it for 100 epochs across multiple random seeds.\n", + "\n", + "We provide a checkpoint trained for 100 epochs in the `checkpoint` for your test after the tutorial. We will also use the provided checkpoint for the interpretation section for a better visualization.\n" + ] + }, + { + "cell_type": "markdown", + "id": "37dbe9f3", + "metadata": { + "id": "37dbe9f3" + }, + "source": [ + "The figure below shows the performance of different models on the BioSNAP and BindingDB datasets:\n", + "- Left plot: AUROC (Area Under the ROC Curve)\n", + "- Right plot: AUPRC (Area Under the Precision–Recall Curve)\n", + "\n", + "![](https://media.springernature.com/full/springer-static/image/art%3A10.1038%2Fs42256-022-00605-1/MediaObjects/42256_2022_605_Fig3_HTML.png?as=webp)\n", + "\n", + "The box plots show the median as the centre lines and the mean as green triangles. The minima and lower percentile represent the worst and second-worst scores. The maxima and upper percentile indicate the best and second-best scores. Supplementary Table 2 provides the data statistics of the BindingDB and BioSNAP datasets." + ] + }, + { + "cell_type": "markdown", + "id": "02e3c73e", + "metadata": { + "id": "02e3c73e" + }, + "source": [ + "## Step 5: Interpretation\n", + "\n", + "We interpret the trained models by analyzing the learned attention weights. In this step, we will use PyKale's API to\n", + "1) draw the attention maps of the Bilinear Attention Network (BAN) layer, and\n", + "2) generate molecule images with attention highlights.\n", + "\n", + "This helps us understand which parts of the drug contribute to the interaction with the target protein." + ] + }, + { + "cell_type": "markdown", + "id": "4a56f260141b7368", + "metadata": { + "id": "4a56f260141b7368" + }, + "source": [ + "### Extracting Attention Weights\n", + "First, we need to load the test dataset and create a DataLoader for it. This will allow us to process the test samples in batches. We define functions to create the test dataset and DataLoader." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "2c67553408592b2", + "metadata": { + "id": "2c67553408592b2" + }, + "outputs": [], + "source": [ + "def get_test_dataset(dataFolder):\n", + " df_test_target = pd.read_csv(dataFolder)\n", + " test_target_dataset = DTIDataset(df_test_target.index.values, df_test_target)\n", + " return test_target_dataset\n", + "\n", + "\n", + "def get_test_dataloader(dataset, batchsize, num_workers, collate_fn):\n", + " test_dataloader = DataLoader(\n", + " dataset,\n", + " batch_size=batchsize,\n", + " num_workers=num_workers,\n", + " collate_fn=collate_fn,\n", + " shuffle=False,\n", + " drop_last=True,\n", + " )\n", + " return test_dataloader" + ] + }, + { + "cell_type": "markdown", + "id": "ecdab66ee05da10c", + "metadata": { + "id": "ecdab66ee05da10c" + }, + "source": [ + "We load a small subset of samples for testing from the provided `.csv` file. You can create your own `.csv` file with the same format to test your drug–protein pairs." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "7ef1867541d2577a", + "metadata": { + "id": "7ef1867541d2577a" + }, + "outputs": [], + "source": [ + "test_dataFolder = \"/content/drug-target-interaction/data/drug-target-interaction/bindingdb/interpretation_samples.csv\"" + ] + }, + { + "cell_type": "markdown", + "id": "7fec5dc00a7b4aa4", + "metadata": { + "id": "7fec5dc00a7b4aa4" + }, + "source": [ + "We then build the test dataset and DataLoader using the functions defined above. The `batchsize` is set to 1 to ensure we process one sample at a time for attention visualization later." + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "c99a558c96a1ffd", + "metadata": { + "id": "c99a558c96a1ffd" + }, + "outputs": [], + "source": [ + "test_dataset = get_test_dataset(test_dataFolder)\n", + "test_dataloader = get_test_dataloader(\n", + " test_dataset,\n", + " batchsize=1,\n", + " num_workers=cfg.SOLVER.NUM_WORKERS,\n", + " collate_fn=graph_collate_func,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "e1ff543d132abc42", + "metadata": { + "id": "e1ff543d132abc42" + }, + "source": [ + "Then, we use the following function to load the trained model with the PyKale API." + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "3b7f12b12b139799", + "metadata": { + "id": "3b7f12b12b139799" + }, + "outputs": [], + "source": [ + "def get_model_from_ckpt(ckpt_path, config):\n", + " return DrugbanTrainer.load_from_checkpoint(\n", + " checkpoint_path=ckpt_path,\n", + " model=DrugBAN(**config),\n", + " solver_lr=config.SOLVER.LEARNING_RATE,\n", + " num_classes=config.DECODER.BINARY,\n", + " batch_size=config.SOLVER.BATCH_SIZE,\n", + " # --- domain adaptation parameters ---\n", + " is_da=config.DA.USE,\n", + " solver_da_lr=config.SOLVER.DA_LEARNING_RATE,\n", + " da_init_epoch=config.DA.INIT_EPOCH,\n", + " da_method=config.DA.METHOD,\n", + " original_random=config.DA.ORIGINAL_RANDOM,\n", + " use_da_entropy=config.DA.USE_ENTROPY,\n", + " da_random_layer=config.DA.RANDOM_LAYER,\n", + " # --- discriminator parameters ---\n", + " da_random_dim=config.DA.RANDOM_DIM,\n", + " decoder_in_dim=config.DECODER.IN_DIM,\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "c0678dddcdf076fc", + "metadata": { + "id": "c0678dddcdf076fc" + }, + "source": [ + "Once the model and test data are prepared, we extract attention maps from the trained model. We set the directory to the provided checkpoint file, load the trained model, and set it to evaluation mode." + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "d2a8931099b73c01", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "d2a8931099b73c01", + "outputId": "1dbad556-912d-44f3-88ad-059cf7876a36" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "DrugBAN(\n", + " (drug_extractor): MolecularGCN(\n", + " (init_transform): Linear(in_features=7, out_features=128, bias=False)\n", + " (gcn_layers): ModuleList(\n", + " (0-2): 3 x GCNConv(128, 128)\n", + " )\n", + " )\n", + " (protein_extractor): ProteinCNN(\n", + " (embedding): Embedding(26, 128, padding_idx=0)\n", + " (conv1): Conv1d(128, 128, kernel_size=(3,), stride=(1,))\n", + " (bn1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " (conv2): Conv1d(128, 128, kernel_size=(6,), stride=(1,))\n", + " (bn2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " (conv3): Conv1d(128, 128, kernel_size=(9,), stride=(1,))\n", + " (bn3): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " )\n", + " (bcn): BANLayer(\n", + " (v_net): FCNet(\n", + " (main): Sequential(\n", + " (0): Dropout(p=0.2, inplace=False)\n", + " (1): Linear(in_features=128, out_features=768, bias=True)\n", + " (2): ReLU()\n", + " )\n", + " )\n", + " (q_net): FCNet(\n", + " (main): Sequential(\n", + " (0): Dropout(p=0.2, inplace=False)\n", + " (1): Linear(in_features=128, out_features=768, bias=True)\n", + " (2): ReLU()\n", + " )\n", + " )\n", + " (p_net): AvgPool1d(kernel_size=(3,), stride=(3,), padding=(0,))\n", + " (bn): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " )\n", + " (mlp_classifier): MLPDecoder(\n", + " (fc1): Linear(in_features=256, out_features=512, bias=True)\n", + " (bn1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " (fc2): Linear(in_features=512, out_features=512, bias=True)\n", + " (bn2): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " (fc3): Linear(in_features=512, out_features=128, bias=True)\n", + " (bn3): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " (fc4): Linear(in_features=128, out_features=2, bias=True)\n", + " )\n", + ")" + ] + }, + "execution_count": 28, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "checkpoint_path = \"/content/drug-target-interaction/checkpoint/best.ckpt\"\n", + "model = get_model_from_ckpt(checkpoint_path, cfg)\n", + "model.model.eval()" + ] + }, + { + "cell_type": "markdown", + "id": "159d3fa67b29c9e9", + "metadata": { + "id": "159d3fa67b29c9e9" + }, + "source": [ + "We then iterate through the test DataLoader, passing each batch of drug and protein pairs to the model. The model's forward method returns the attention weights. After processing all batches, we concatenate the attention tensors into a single tensor." + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "781a7762c36c72be", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "781a7762c36c72be", + "outputId": "fade7fb6-bea8-438a-9688-8cffd8dc9c47" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 6/6 [00:00<00:00, 65.52it/s]\n" + ] + }, + { + "data": { + "text/plain": [ + "torch.Size([6, 2, 290, 1185])" + ] + }, + "execution_count": 29, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from tqdm import tqdm\n", + "\n", + "all_attentions = []\n", + "for batch in tqdm(test_dataloader):\n", + " drug, protein, _ = batch\n", + " drug, protein = drug.to(model.device), protein.to(model.device)\n", + "\n", + " _, _, _, _, attention = model.model.forward(\n", + " drug, protein, mode=\"eval\"\n", + " ) # [B, H, V, Q]\n", + "\n", + " attention = attention.detach().cpu()\n", + " all_attentions.append(attention)\n", + "\n", + "# Concatenate into one tensor: [N, H, V, Q]\n", + "all_attentions = torch.cat(all_attentions, dim=0)\n", + "torch.save(all_attentions, \"attention_maps.pt\")\n", + "\n", + "all_attentions.shape" + ] + }, + { + "cell_type": "markdown", + "id": "78dc763b6c0eef0", + "metadata": { + "id": "78dc763b6c0eef0" + }, + "source": [ + "The attention has shape [B, H, V, Q] (Number of drug-target pairs, Heads of attentions, Drug tokens, Protein tokens)." + ] + }, + { + "cell_type": "markdown", + "id": "8f72ea4d93f640cb", + "metadata": { + "id": "8f72ea4d93f640cb" + }, + "source": [ + "### Visualize Attention Maps and Molecule Images" + ] + }, + { + "cell_type": "markdown", + "id": "383c342a7c31d7ae", + "metadata": { + "id": "383c342a7c31d7ae" + }, + "source": [ + "Once attention maps are saved, run the visualization script:" + ] + }, + { + "cell_type": "markdown", + "id": "d8a746169def8da5", + "metadata": { + "id": "d8a746169def8da5" + }, + "source": [ + "This script will:\n", + "\n", + "1) Load the attention weights and the corresponding SMILES + protein data.\n", + "\n", + "2) Plot:\n", + "\n", + " a) A heatmap of attention over drug–protein tokens.\n", + "\n", + " b) Molecular structures with atoms highlighted by attention values.\n", + "\n", + "The output images are saved in the `visualization` directory. You can also modify the `data_file` to use your own input in the same format as `target_test.csv`.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "aac54bfc67ce32eb", + "metadata": { + "id": "aac54bfc67ce32eb" + }, + "source": [ + "We first import the necessary PyKale APIs and set the output directory." + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "d3c1d2e4cab69107", + "metadata": { + "id": "d3c1d2e4cab69107" + }, + "outputs": [], + "source": [ + "from kale.interpret.visualize import draw_attention_map, draw_mol_with_attention\n", + "from kale.prepdata.tensor_reshape import normalize_tensor\n", + "\n", + "out_dir = \"./visualization\"\n", + "os.makedirs(out_dir, exist_ok=True)" + ] + }, + { + "cell_type": "markdown", + "id": "126b62034111d92a", + "metadata": { + "id": "126b62034111d92a" + }, + "source": [ + "We then load the attention maps, data, and SMILES strings from the test dataset." + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "7f70a6810c1c5e60", + "metadata": { + "id": "7f70a6810c1c5e60" + }, + "outputs": [], + "source": [ + "attention = torch.load(\"attention_maps.pt\", map_location=\"cpu\")\n", + "data_df = pd.read_csv(test_dataFolder)\n", + "smiles = data_df[\"SMILES\"]\n", + "proteins = data_df[\"Protein\"]" + ] + }, + { + "cell_type": "markdown", + "id": "d1a009bbb9f4a0f9", + "metadata": { + "id": "d1a009bbb9f4a0f9" + }, + "source": [ + "We select the first sample from the attention maps and corresponding SMILES and protein sequence for visualization." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "e808c255fe862925", + "metadata": { + "id": "e808c255fe862925" + }, + "outputs": [], + "source": [ + "index = 0\n", + "att_path = os.path.join(out_dir, f\"att_map_{index}.png\")\n", + "mol_path = os.path.join(out_dir, f\"mol_{index}.svg\")" + ] + }, + { + "cell_type": "markdown", + "id": "438e6aa218e6b51d", + "metadata": { + "id": "438e6aa218e6b51d" + }, + "source": [ + "We crop the attention map to the actual lengths of the drug and protein sequences. This is important because the attention map may include padding tokens." + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "af15baa1c8caabc0", + "metadata": { + "id": "af15baa1c8caabc0" + }, + "outputs": [], + "source": [ + "from rdkit import Chem\n", + "\n", + "\n", + "def get_real_length(smile, protein_sequence):\n", + " \"\"\"Get the real length of the drug and protein sequences.\"\"\"\n", + " mol = Chem.MolFromSmiles(smile)\n", + " return mol.GetNumAtoms(), len(protein_sequence)\n", + "\n", + "\n", + "att = attention[index] # [H, V, Q]\n", + "smile = smiles[index]\n", + "protein = proteins[index]\n", + "real_drug_len, real_prot_len = get_real_length(smile, protein)\n", + "att = att[:, :real_drug_len, :real_prot_len].mean(0) # [V, Q]\n", + "\n", + "# Normalize\n", + "att = normalize_tensor(att)" + ] + }, + { + "cell_type": "markdown", + "id": "60a4ce71146a721e", + "metadata": { + "id": "60a4ce71146a721e" + }, + "source": [ + "Finally, we save the attention map and the molecule image with attention highlights." + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "403f77ada0ecc446", + "metadata": { + "id": "403f77ada0ecc446" + }, + "outputs": [], + "source": [ + "draw_attention_map(\n", + " att,\n", + " att_path,\n", + " title=f\"Drug {index} Attention\",\n", + " xlabel=\"Drug Tokens\",\n", + " ylabel=\"Protein Tokens\",\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "b1003372361a66d6", + "metadata": { + "id": "b1003372361a66d6" + }, + "outputs": [], + "source": [ + "draw_mol_with_attention(att.mean(dim=1), smile, mol_path)" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "4mHWCbJmGMgG", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 921 + }, + "id": "4mHWCbJmGMgG", + "outputId": "5b3f1960-85a5-4ca3-bbbb-f48f6bc67401", + "tags": [ + "hide-input" + ] + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA+gAAAJYCAYAAADxHswlAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAA0fhJREFUeJzs3Xt8FNX9//H37G6yCRCiAcJVBKVeUaR4KWIxKAXRolRbr1XEW7VJLVCtpq2itTbaVottKfy0CrZKpVVBqxaLXOUrqKBRaRVFUbwQEJVAAtnsZX5/zMyS2SSws9klm+T15LGPZWdnPvM5Z86c2ZOZnTVM0zQFAAAAAABala+1EwAAAAAAAAzQAQAAAADICgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQDYD5YtWybDMLRs2bLWTgUAAGQpBugA0AbMmTNHhmHEH3l5eerTp4/Gjh2rP/zhD9q5c2drp9isUCikm266SX369FF+fr5OOukkLVq0yHOc888/X4Zh6Kabbmry/eeee0633XZbo+m7du3Sbbfdtt8Gxn/+8581Z86c/bIuAADQvhimaZqtnQQAYO/mzJmjSZMm6Ze//KUGDhyocDisqqoqLVu2TIsWLVL//v319NNP69hjj23tVBu56KKL9Pjjj2vy5Mn62te+pjlz5ujVV1/V0qVLdcoppyQVY8eOHerZs6d69eqlaDSqjz76SIZhuOYpKyvTjBkzlHhY27Ztm3r06KFp06Y1OYBPt8GDB6t79+6N/iAQi8VUX1+v3Nxc+Xz8fRwAADQWaO0EAADJGzdunI4//vj46/Lyci1ZskTf/va3dfbZZ+vtt99Wfn5+s8vX1taqc+fO+yNVSdIrr7yixx57TL/97W91ww03SJIuu+wyDR48WD/96U/10ksvJRXniSeeUDQa1UMPPaTTTjtNK1as0KmnnprJ1NPO5/MpLy+vtdMAAABZjD/hA0Abd9ppp+mWW27RRx99pEceeSQ+/fLLL1eXLl30/vvv68wzz1RBQYEuueQSSdKAAQN0+eWXN4pVUlKikpIS17SPPvpIZ599tjp37qzi4mJNmTJFzz//fFLfp3788cfl9/t1zTXXxKfl5eXpyiuv1KpVq/Txxx8nVcZHH31U3/rWtzRq1CgdeeSRevTRR13vX3755ZoxY4Ykub4K8OGHH6pHjx6SpNtvvz0+veGZ9HfeeUff/e53VVRUpLy8PB1//PF6+umnXfGdrxj83//9n6ZOnaoePXqoc+fO+s53vqPPP/88Pt+AAQP03//+V8uXL4+vy6nP5r6D/s9//lPDhg1Tfn6+unfvru9///v69NNPG5WvS5cu+vTTTzVhwgR16dJFPXr00A033KBoNJpUHQIAgOzHAB0A2oFLL71UkvSf//zHNT0SiWjs2LEqLi7W7373O5133nme4tbW1uq0007TCy+8oOuvv14///nP9dJLLzX7PfBEr7/+ug477DB17drVNf3EE0+UJFVWVu4zxmeffaalS5fqoosukrTnkvn6+vr4PD/4wQ/0rW99S5L0t7/9Lf7o0aOHZs6cKUn6zne+E59+7rnnSpL++9//6hvf+Ibefvtt3XzzzbrnnnvUuXNnTZgwQfPnz2+Uy49+9CO98cYbmjZtmq677jr961//UllZWfz96dOnq1+/fjriiCPi6/r5z3/ebNnmzJmj888/X36/XxUVFbr66qv15JNP6pRTTtH27dtd80ajUY0dO1bdunXT7373O5166qm65557dP/99++zDgEAQNvAJe4A0A7069dPhYWFev/9913TQ6GQvve976mioiKluP/v//0/ffDBB1qwYIHOOeccSdZgeOjQoUktv3nzZvXu3bvRdGfaZ599ts8Yf//73xUMBuPrv/DCC3Xrrbfqueee04QJEyRJw4cP12GHHaZFixbp+9//vmv57373u7ruuut07LHHNnrvxz/+sfr3769XX31VwWBQkvTDH/5Qp5xyim666SZ95zvfcc3frVs3/ec//4l//z0Wi+kPf/iDqqurVVhYqAkTJugXv/hF/Ez43oTDYd10000aPHiwVqxYEb/8/ZRTTtG3v/1t/f73v9ftt98en7+urk4XXHCBbrnlFknStddeq69//et68MEHdd111+2zHgEAQPbjDDoAtBNdunRp8m7uLRm8LVy4UH379tXZZ58dn5aXl6err746qeV3794dH/g25AxGd+/evc8Yjz76qM466ywVFBRIkr72ta9p2LBhjS5z9+rLL7/UkiVLdP7552vnzp3atm2btm3bpi+++EJjx47Ve++91+hS82uuucZ1c7pvfvOb8ZvWebVmzRpt3bpVP/zhD13fTT/rrLN0xBFH6Nlnn220zLXXXut6/c1vflMffPCB53UDAIDsxBl0AGgnampqVFxc7JoWCATUr1+/lGN+9NFHOvTQQxvdMX3QoEFJLZ+fn69QKNRoel1dXfz9vXn77bf1+uuv67LLLtOGDRvi00tKSjRjxgzt2LGj0eXzydqwYYNM09Qtt9wSPyudaOvWrerbt2/8df/+/V3vH3jggZKkr776yvP6nUH94Ycf3ui9I444QitXrnRNy8vLi3+fvuH6U1k3AADITgzQAaAd+OSTT1RdXd1o4BwMBpv8Sa/EAbcjGo3K7/enLa/evXs3OgstWZe+S1KfPn32urxz07spU6ZoypQpjd5/4oknNGnSpJRyi8VikqQbbrhBY8eObXKexPpsrm72xy+WpnO7AACA7MQAHQDagb/97W+S1OxAM9GBBx7Y6CZkknVW95BDDom/Pvjgg/W///1Ppmm6BvUNz2bvzXHHHaelS5c2OtP98ssvx99vjmmamjt3rkaNGqUf/vCHjd6/44479Oijj8YH6M390aG56U45c3JyNHr06KTKk4zm1pfo4IMPliStX79ep512muu99evXx98HAAAdB99BB4A2bsmSJbrjjjs0cODA+M+o7cuhhx6q1atXu+6E/swzzzT62bOxY8fq008/df3sWF1dnR544IGk1vPd735X0WjUdafxUCik2bNn66STTtJBBx3U7LL/93//pw8//FCTJk3Sd7/73UaPCy64QEuXLo3faM75fffEPzx06tSpyenFxcUqKSnR//t//y9+Rr+hhj+f5kXnzp2b/ONHouOPP17FxcWaNWuW62sA//73v/X222/rrLPOSmn9AACg7eIMOgC0If/+97/1zjvvKBKJaMuWLVqyZIkWLVqkgw8+WE8//bTrZmN7c9VVV+nxxx/XGWecofPPP1/vv/++HnnkER166KGu+X7wgx/oT3/6ky666CL9+Mc/Vu/evfXoo4/G17Ovs8UnnXSSvve976m8vFxbt27VoEGD9PDDD+vDDz/Ugw8+uNdlH330Ufn9/mYHqmeffbZ+/vOf67HHHtPUqVM1bNgwSdL111+vsWPHyu/368ILL1R+fr6OOuoozZs3T4cddpiKioo0ePBgDR48WDNmzNApp5yiY445RldffbUOOeQQbdmyRatWrdInn3yiN954I6n6bGjYsGGaOXOmfvWrX2nQoEEqLi5udIZcss7c33333Zo0aZJOPfVUXXTRRdqyZYvuu+8+DRgwoMlL+gEAQDtnAgCy3uzZs01J8Udubq7Zq1cv81vf+pZ53333mTt27Gi0zMSJE83OnTs3G/Oee+4x+/btawaDQXPEiBHmmjVrzFNPPdU89dRTXfN98MEH5llnnWXm5+ebPXr0MH/yk5+YTzzxhCnJXL169T5z3717t3nDDTeYvXr1MoPBoHnCCSeYCxcu3Osy9fX1Zrdu3cxvfvObe51v4MCB5tChQ03TNM1IJGL+6Ec/Mnv06GEahmE2PMS99NJL5rBhw8zc3FxTkjlt2rT4e++//7552WWXmb169TJzcnLMvn37mt/+9rfNxx9/PD6PU/+vvvqqa/1Lly41JZlLly6NT6uqqjLPOusss6CgwJQUr8+m5jVN05w3b545dOhQMxgMmkVFReYll1xifvLJJ655mtuW06ZNMzmUAwDQfhimuR/ubAMAaFemT5+uKVOm6JNPPnHd5RwAAACpY4AOANir3bt3u34Ora6uTkOHDlU0GtW7777bipkBAAC0L3wHHQCwV+eee6769++v4447TtXV1XrkkUf0zjvv6NFHH23t1AAAANoVBugAgL0aO3as/vKXv+jRRx9VNBrVUUcdpccee0wXXHBBa6cGAADQrvAzawCAvZo8ebLWrVunmpoa7d69W2vXrmVwDgBAK1uxYoXGjx+vPn36yDAMLViwYJ/LLFu2TF//+tcVDAY1aNAgzZkzJ+N5whsG6AAAAADQxtTW1mrIkCGaMWNGUvNv3LhRZ511lkaNGqXKykpNnjxZV111lZ5//vkMZwovuEkcAAAAALRhhmFo/vz5mjBhQrPz3HTTTXr22We1bt26+LQLL7xQ27dv18KFC/dDlkgGZ9ABAAAAIAuEQiHt2LHD9QiFQmmJvWrVKo0ePdo1bezYsVq1alVa4iM9OuxN4o654feSJOPUryRJAw/8UpL031WHeI7V43XrIoS6A6y/d1Qfbr321XuLE+kcs/5jWE9d3/VLkmpO2G3F+zQv6VjhoogkKeeLHElSLGjldMDbVvBwJ2+51Rxi5XbxyP+TJP3zX9+04uZ6vwAj92s7JEmB5YVWDCtF1R9gPUeDycXs+r5Vlq9OtCo6//1cSVLk2FpJkvlh8oV0cuj0mRXTP9JqDzu2WzFyNyZf95LU5fhtkqQvNh1oLd/d2oaB17t4iiNJuw62tqWC1jboss4qZ7jAmhzNS66+Ip2s+Q55MixJ+vhbQSvOAVErt53J/73OqS+jV50kyfeh9RNc4W5WroHt/qRjSVKXTVa9j7ryZUnSs8+dZK/He/uK5lvLdD/0C+u5s9UeNvzfAE8xDXu2aB/roGhG9tSPr9rqOn27k6szJ9bJp1l/sV7+ztckSf5ca5saH3trX/46q776La2LT6v6sZVn6N2unmJF7O1v5NvtzO5/zGprIwdqkyujedCeXHp1q5YkffpZkSSpW7G1z1ev65ZULCNqJdH3hE8lSR9+2l2SdMCrVput6e+9XQz6m7VPr7/a2idVZPUbvqqgpzhOvxMY/7kkqfMfDrByPNfuO3Yk3/ajXaztX3ywldv2tT0kSfXd7X1yR3J17wtb667vYW3DW099SpK0K2b1FX94cnzSOTk6f2I9R+ymWTz+Y0nSR2v6eYrT9Wi7bO9ZbaHzx1auO46wj1FfJV9fwW3WsrUHW/VzwP/sY+RIax/XR8n1+abdxjsdZrVTn8/aDqF6q81H3vfeT3d7y3rOqbFi/fUP90iSxj7yU2ud/uTabK/V1vK1Pa2yVX/TOnZ0P7BGkvTlW92TzinSw+rr87pafUP4A6tcTn9bc3CSfaH90aTHkK2SpC1fWH1MrN7K0b8tJ+mcHEX/tZ6rD7FyGXr6eknSy6/ZfaPdx+2Lae8ih8222tlnp+/pY2r7eusnovbxK6fKKs9BL1j1tuunVjvZ9mYPT/GcMnxj7J4zla982l+SFN5Q4ClW36VWbptPtnJ74PszJUnX/O06SXuO7/uSe8hOSVL9xj3rz99s5dnlM2tD7+yfXL8TtneT+mIrtz+e/jdJ0pTHJ1lvGHvPacONU5NaTzaKVR2W0fgVsy7W7bff7po2bdo03XbbbS2OXVVVpZ49e7qm9ezZUzt27Gj0k6poPR12gA4AAAAA2aS8vFxTp7r/gBEMevtjMtq2NnuJ+4wZMzRgwADl5eXppJNO0iuvvNLaKQEAAABox2IZ/hcMBtW1a1fXI10D9F69emnLli2uaVu2bFHXrl05e55F2uQAfd68eZo6daqmTZum1157TUOGDNHYsWO1devW1k4NAAAAALLO8OHDtXjxYte0RYsWafjw4a2UEZrSJgfo9957r66++mpNmjRJRx11lGbNmqVOnTrpoYceau3UAAAAALRTUTOW0YcXNTU1qqysVGVlpSTrZ9QqKyu1adMmSdbl8pdddll8/muvvVYffPCBfvrTn+qdd97Rn//8Z/3jH//QlClT0lY/aLk2N0Cvr6/X2rVrXXcg9Pl8Gj16NHcgBAAAANAhrFmzRkOHDtXQoUMlSVOnTtXQoUN16623SpI2b94cH6xL0sCBA/Xss89q0aJFGjJkiO655x795S9/0dixY1slfzStzd0kbtu2bYpGo03egfCdd95ppawAAAAAtHcxef8lkUwpKSmRaTafz5w5c5pc5vXXX89gVmipNjdAT0UoFGr0+4GxSES+QIcoPgAAAACgDWhzl7h3795dfr+/yTsQ9urVq8llKioqVFhY6Hp8/soL+yNdAAAAAO1Epu/iDrS5AXpubq6GDRvmugNhLBbT4sWLm70DYXl5uaqrq12PHieObnJeAAAAAABaQ5u8xnvq1KmaOHGijj/+eJ144omaPn26amtrNWnSpCbnDwaDjX4/kMvbAQAAAHgR3ct3voF0aJOj1AsuuECff/65br31VlVVVem4447TwoULG904DgAAAACAtqJNDtAlqaysTGVlZa2dBgAAAIAOIpvu4o72qc19Bx0AAAAAgPaozZ5BBwAAAID9KcoZdGQYZ9ABAAAAAMgCHfYMuq/eevb7rd8b7JpTZ00wDc+xcndGJUn1na2/dxgRpRbLmd/+w5w/ZOca8P6biEbin16cmPXWf6K5XnOznopzd1jxnZRSqC+/zwoW2GW9Dnd2ryPpOCF7gbDPfm299OVaGyDkIZZTHl/EnaMvYLreT1Z+Ttj6j1PvPitAvG14kVgvZuJzstvAWiBnh1Mz9i8b+E2PcSTZdzD12fuPsy3ri1P7q3JOrbVc3+BXkiQjar8R8N6+lHB31U4Ba1s429BIMkUjaq07kGslE4402Kns95KOZa+7R7DGem23L3/Aih3zuB857Siwsz4+LS/XKmcohX3SFdvOzWsYpyyS1CXXbmN2jAL7dXWyQRP2GyeOv85533sZjS+qredokSQpFkutngK7reTy7fr21cfslPzeg0WsHHID9gZ1UvLavzq7sL0vD8j9XJK0M5bvjudBvH817Bz9UW85Jabo9Kn19vLJ7jwNOP2zYW87X9SVoswkczPsCjPsHJz+Psduw6l00067yKmxYgwIFNhJ2TMkmVug1lo+UGd/nrBz65Rj7etfeqh/0+6znM85ETsXX/wzStKhrOXs+orZx9x4e02h2w/UOcdWK0iB8xnM5wRLcls6bcDZv2Pd9uQbtmKYgWQ7atMV07/bqqjOdv+1Lbkoe8LZ9dw5sOfTiNPmvO5Hudut7e+L5EiSevhr7TjecnLaeH2D5ZzjU26N8zkluXN38eO0Haubv8YVz/S17FiUzfgOOjKNM+gAAAAAAGSBNjlAX7FihcaPH68+ffrIMAwtWLCgtVMCAAAA0M5FTTOjD6BNDtBra2s1ZMgQzZgxo7VTAQAAAAAgLdrkd9DHjRuncePGtXYaAAAAADqQFG7rAXjSJgfoAAAAALC/8TNryLQ2eYk7AAAAAADtTYc4gx4KhRQKuX90KxaNyOfvEMUHAAAAkAZRTqAjwzrEGfSKigoVFha6HlvXvNDaaQEAAAAAENchBujl5eWqrq52PYqPH93aaQEAAABoQ2IZfgBt8hrvmpoabdiwIf5648aNqqysVFFRkfr3799o/mAwqGAw6JrG5e0AAAAAgGzSJkepa9as0ahRo+Kvp06dKkmaOHGi5syZ00pZAQAAAGjPojJaOwW0c21ygF5SUiLT5A4NAAAAAID2o00O0AEAAABgf4txjhAZ1iFuEgcAAAAAQLbjDDoAAAAAJIHvoCPTOuwA3R+2nmvrciVJn9QWtjhmLMd6DuyydtxocC8zN5XTbms5I+re8WMR60IHT5c7RNwxjKj1bDpBPF474a+z4r1V08/bgk2orbEqplu9dY1QuIudq8f+zvRbC+R8YTVjI2KHMbxfe2TY9eUsuX17J+u1aU/3mNuueqtd+eqsig7VWY0jL4VrVpwYMTsJp52Zfo9x7DJG86368lqmhpz2FN5lJdPlC+v1roNTi5dTa9X8hl3FkiRfvTXd6z4k7Wmr274ssHJ1yumxWfjsPsJ02lPdngp39vGkt4Ed4uNdB0qSYiFrwWiO9YMqXjeF366fcNc9FRSORD1GcRa0yxK1tqWZb8VJ7If2JVK/pzK+2m3vP3bsUNTbocap8k+/OsD6j1P3LflM1LWLJClQawWpz8lJKYzP7meqd+dJkjoVW3GMsPd+x2lHn++wcovmWjH8u7x1FGbC7K/tHiBJ2uXsQCnUm+mzj0d2eTd9daAdy1s5v/qysytHn91MjYj3ztCJ4RwrneI5/VDSrczu/HbY/XxuvrWzRyNWO0ulmUWDdv8ctJJctNvOxgmWZL1F7YOE0z9H7f1qRyjPc07OsTHU1V0zqfSrkrT5c+tzkv9z6/gW7epsTO+xojnO5x3r9Ybq7tZ/PO5GAbstOPt3NLfBe7us53DXJIOFrbo37N+5qj/QClYfzm1uib3y11nP//uqV3zarh3WdvT64dsMWOV0jo1/336i9R+PdV8Xcj5A7JnmfKZwpiX2J81x+kLnM8qT20+wUkr8vAnAsw47QAcAAAAALziDjkxrk3/fqqio0AknnKCCggIVFxdrwoQJWr9+fWunBQAAAABAytrkAH358uUqLS3V6tWrtWjRIoXDYY0ZM0a1tbWtnRoAAACAdipmGhl9AG3yEveFCxe6Xs+ZM0fFxcVau3atRo4c2UpZAQAAAACQujY5QE9UXV0tSSoqKmrlTAAAAAC0V3wHHZnWJi9xbygWi2ny5MkaMWKEBg8e3NrpAAAAAACQkjZ/Br20tFTr1q3TypUrm50nFAopFAq5psWiEfn8bb74AAAAAPaTaNs/v4ks16ZbWFlZmZ555hktXbpU/fo1//vcFRUVKiwsdD22vPbCfswUAAAAAIC9a5MDdNM0VVZWpvnz52vJkiUaOHDgXucvLy9XdXW169Hz66P3U7YAAAAA2gPu4o5Ma5PXeJeWlmru3Ll66qmnVFBQoKqqKklSYWGh8vPzG80fDAYVDAZd07i8HQAAAACQTdrkKHXmzJmSpJKSEtf02bNn6/LLL9//CQEAAABo97iLOzKtTQ7QTdNs7RQAAAAAAEirNjlABwAAAID9LWq2yVt4oQ1hgA4AAAAASYi1zXtsow2hhQEAAAAAkAU67Bn0wG7re+xmws8ZmD7v32/374pay9p3hs/ZYU2P9PQWK7DbyiVQa8ett5b358QkSbEU7klh+u1y+p11WK8jed6C+eqt+aN2ffnqrenRPO/1FavNkSTl1FrL1h1oxfT6yxJOWfwhO6eINb2w8y5J0i6jMOlYfrs8Tj3FItZ/AnlhSZIR85Zbrt9qE4Fddr3Z7cqIeosjST67fLG8mDtHZ+81ktwGdhn8tVZhjVgnO9mYtziSDLsxmlHrudMWq2Bf2W3V68bs/LG1zUJ2oZz25SUnR7ytfpVrpVLk5CxPufmsTa/cPCuZ+uieX4gI7Laew52Ty8lZd8Bn1ZNh11swaK2k3mM5/XV2GZr4E6vnPsyuD8Mur7rEXHFMI7n6MvyN12tErAS7BuskSZ8nXU5rnaHdVl+heiuOETUbvu3NjhpJUs7OYklSNJjaTX6C1dY23BG22qq/0M4t4rGTkOQL232XXS9Ru+6DW/12jkkGcppDrpXbAX5rf+rkc/Z1z6nJF7FyCoSs5+q6HNe6kmXutJYzg1YSznEu/r6HeE7/6ezjMbsvNCN2kCTbV3ydu6xtGLX7rfjtbVJoGj77eK2Y9fxp2Op4nCthky2ns61ya9xlMZyyeegrcqvt/ShmJeGz6yl+dW6SOTlrNL+wGmTel9aCtckfYhvxRd3PEScpj6eNAjX2f2qtTjmat+e93J3Wc7jA2wZ1tkE0z0om6Lc+XHj+jBKynr/a3eDXhUJ2o/XY5+dssQqTu8M6br9Z3dc9g8du37V6+//BL63+whdNcmhgH6f9tVY9fbrbbhBOPaVw/G4ruEkcMo0z6AAAAAAAZIE2OUCfOXOmjj32WHXt2lVdu3bV8OHD9e9//7u10wIAAADQjkVNX0YfQJtsBf369dNdd92ltWvXas2aNTrttNN0zjnn6L///W9rpwYAAAAAQEra5HfQx48f73p95513aubMmVq9erWOPvroVsoKAAAAQHsW4zvoyLA2OUBvKBqN6p///Kdqa2s1fPjw1k4HAAAAAICUtNkB+ltvvaXhw4errq5OXbp00fz583XUUUe1dloAAAAA2qlo2/yGMNqQNjtAP/zww1VZWanq6mo9/vjjmjhxopYvX97kID0UCikUCrmmxaIR+fxttvgAAAAAgHamzf4JKDc3V4MGDdKwYcNUUVGhIUOG6L777mty3oqKChUWFroem99avJ8zBgAAANCWcRd3ZFq7aQWxWKzRWXJHeXm5qqurXY/ex5y+nzMEAAAAAKB5bfIa7/Lyco0bN079+/fXzp07NXfuXC1btkzPP/98k/MHg0EFg0HXNC5vBwAAAOBFrP2c30SWapOj1K1bt+qyyy7T5s2bVVhYqGOPPVbPP/+8vvWtb7V2agAAAAAApKRNDtAffPDB1k4BAAAAQAcTNfkddGQW12gAAAAAAJAF2uQZdAAAAADY3/gddGQaLQwAAAAAgCzQYc+gR3Ot749EQn5JUk0ouLfZ9x6rkxUjaodI+ScMTevJiNpxg1aOsVgK33VpZhnTb8fM8RjPDrd1d4H10s4xpa/h1FsVFO5i2DnZsTzWWyzgXs55HY76U0jKjpFrxww79eRLKbdd9VYFO/XkMFuyx9l17eRi+k1vi9uzRwqCrtdOu/OyLY2EVcfs/am5drcvZsBazmlfzp8OU2lfRsx+jlgL14Xtxu7ESky+uZx8CStvKhmP+e2oz7P+E7bblR3Tcznt+eu77mlQdSGvO3VCyEhCf+MxKTO6Z/66sJ2X3f5r6oOeQsbbeMT6j8/OLdLJqS9vbV+S1LWLpD37pGGX0/R5ixUNWjmF6qz6zunkjueFU85w2O6z7H3aF7HXlZdsIPspZgXcWNfDimPXUyrHJOf44+xPMbvNeg3ltCsFrFxiAaev8J5TvP80nBztl1GP+5Ezn11vMXt5I5V2JScX53hmbct3dve21+UtZiTPZz+7+9Td9v7tabd0QtjbwDlkeD7+23whu57i+1BqcSQpkvB5Kf4ZzOsmcLqrgs7Wc4NjbLzdJ7sNnD7B3h3ru1gB6sO5HpOyw9m51O1u8PkyklqfH+vkNHbraXNN15TiROy+xrUfO/tTvvVeLMnPKU49Oe1hW11n1/Lt+WvaMX6rHBlGCwMAAAAAIAu0+QH6XXfdJcMwNHny5NZOBQAAAEA7FpUvow+gTbeCV199Vf/v//0/HXvssa2dCgAAAAAALdJmB+g1NTW65JJL9MADD+jAAw9s7XQAAAAAtHNR08joA2izA/TS0lKdddZZGj16dGunAgAAAKADiMmX0QfQJu/i/thjj+m1117Tq6++2tqpAAAAAACQFm1ugP7xxx/rxz/+sRYtWqS8vOR+gyYUCikUCrmmxaIR+fxtrvgAAAAAWkmUn1lDhrW5FrZ27Vpt3bpVX//61xUIBBQIBLR8+XL94Q9/UCAQUDQabbRMRUWFCgsLXY+qyhdaIXsAAAAAAJrW5k4hn3766Xrrrbdc0yZNmqQjjjhCN910k/x+f6NlysvLNXXqVNe0U6f8v4zmCQAAAKB9iYkbuSGz2twAvaCgQIMHD3ZN69y5s7p169ZouiMYDCoYDLqmcXk7AAAAACCbMEoFAAAAgCTwHXRkWrsYoC9btqy1UwAAAAAAoEXaxQAdAAAAADIt2vbusY02hhYGAAAAAEAW4Aw6AAAAACQhZnIXd2RWhx2gB+pMSZIvEJMkdc6tlyR9mUIs/y7rt9eNmFWdPtP9vmEmLtE0I2Y/2z/l7qu349s5Nv6F971ITMIW2G2XO+yxc7FzOyC428rRDp9s2VzshQJ19suo4YplJhnT2Yayf+7CZ1dQl9yQJGmrl5Scuo+4Qsrnj7neT1ZejhUoZL92tmE8fgpMe5s65fRa987sgZ1WVkY035rgSyGes/3tevI726KZdrcvgR1WTkXBXZKkj1IsY8PcnFxy7W0RdrZhkgdWZ5t3yg1LkqobJmMmPCeZ0wG51v4jvzUhJ2AVNOyxnE47Cuza0zADdixP/YQrqP1k52Ya7v1rn4v79xTCKZfTtpx9MuntGV+1ewFfuMnJydn2lbVsrNh+tlfl8TqygN3fB3Lsfj/eZ3irL2vl1lOOHcvpluP9jdd24bMW7J27XZLks3Py2n9Jks9ulP56O0aK+7bD2e2cuKlcv+dz6toJYbcH02MfFp/N7y6b00+nUF3x45G/3lr60Dz7COQUPMnkAnXW8mbALlS8H7PaSL2HzRD/LBGwF0r4jOGZUxR7+ZbcJyvgHBzt1PICEdc6kk7JOVZs32FP6d7ovWT7/MTjl7NNg3b/9YW31OLtNSdnz4E/nMrxVpJvp3Vs9IUPkCQVBK0PUNUe4zhtvOFiTv8QqLF2KF8kqGTEEj4HHpBrf6hryedDAJI68AAdAAAAALzgO+jItDbZwm677TYZhuF6HHHEEa2dFgAAAAAAKWuzZ9CPPvpovfDCC/HXgUCbLQoAAACANiDG76Ajw9rsqDYQCKhXr16tnQYAAAAAAGnRZv8E9N5776lPnz465JBDdMkll2jTpk2tnRIAAACAdiwqI6MPoE0O0E866STNmTNHCxcu1MyZM7Vx40Z985vf1M6dO1s7NQAAAAAAUtImL3EfN25c/P/HHnusTjrpJB188MH6xz/+oSuvvLLR/KFQSKFQyDUtFo3I52+TxQcAAADQCvgOOjKtXbSwAw44QIcddpg2bNjQ5PsVFRUqLCx0PT5bt3g/ZwkAAAAAQPPaxQC9pqZG77//vnr37t3k++Xl5aqurnY9+gw+fT9nCQAAAKAt4zvoyLQ2eY33DTfcoPHjx+vggw/WZ599pmnTpsnv9+uiiy5qcv5gMKhgMOiaxuXtAAAAAIBs0iZHqZ988okuuugiffHFF+rRo4dOOeUUrV69Wj169Gjt1AAAAAC0U3wHHZnWJgfojz32WGunAAAAAABAWrXJAToAAAAA7G9RzqAjw2hhAAAAAJCEmIyMPlIxY8YMDRgwQHl5eTrppJP0yiuv7HX+6dOn6/DDD1d+fr4OOuggTZkyRXV1dSmtG+nHAB0AAAAA2qB58+Zp6tSpmjZtml577TUNGTJEY8eO1datW5ucf+7cubr55ps1bdo0vf3223rwwQc1b948/exnP9vPmaM5HfcSd9N68gdikiTDsCYYMe9/uYrlGPay1utovlKKFbO3RqSz9ZxbY0+POvGTj2caTg6JudrTTU+pxf+UE7ADRvKdgN7ry8yzkzL9Vi6JMyQb0ilj1A5n57gzlGdNTyE3X9iJaS0bi6b2NyyfXcF2ERWNWHFyUojltAsnJ6e8vrCdoz+5OGauvdENu16cNhB1XidfX/H2Y/8nku8sm9pffsMHWNusKLfWSsUpUwrb0Ekh2slqZ6ZdLg/Fc+UQjdltoEFTiOZ5T8sVO2DVWyhsb1yP5TTtxfyhaHxabo71/3qPsYyYlUssx+4D49O99TsNuxSnzsxcu79w6jDJWE778uXE7NzcfUZK7cJnLRO1f9AjFvDaCdrL2f19p/x6K2y9Pd1OzR9OKawVa7fPlaNXpl0vYbuB7Iyk3lCdY0ckz4rps4+Vntuq3QaMsFP/7uW99NNGQjMI7Lan+01PuTldoPKsfcbnd/oK+/0UmkY013rO2WnF+lqwypWz177RqXfns4lzTPHSkcVndT7fOOVzuo0ky+ksFz4wYuW2I8c1Pdk4TeXmHLcDfudA7i1mzNlX8u223uAzj7NNvObkrNsXcde9588Udpyu+aH4pF2+zgkrSzJUJ6ugzme4fp2rJUmf6CBvOe1FNM8+sCS7Pe0iOJ9ROgXsPtHelKmeCW4Lsu0S93vvvVdXX321Jk2aJEmaNWuWnn32WT300EO6+eabG83/0ksvacSIEbr44oslSQMGDNBFF12kl19+eb/mjeZlVwtL0qeffqrvf//76tatm/Lz83XMMcdozZo1rZ0WAAAAAOwX9fX1Wrt2rUaPHh2f5vP5NHr0aK1atarJZU4++WStXbs2fhn8Bx98oOeee05nnnnmfskZ+9bmzqB/9dVXGjFihEaNGqV///vf6tGjh9577z0deOCBrZ0aAAAAgHYs5vWSPI9CoZBCoZBrWjAYVDDY+PKqbdu2KRqNqmfPnq7pPXv21DvvvNNk/Isvvljbtm3TKaecItM0FYlEdO2113KJexZpc2fQ7777bh100EGaPXu2TjzxRA0cOFBjxozRoYce2tqpAQAAAEDKKioqVFhY6HpUVFSkLf6yZcv061//Wn/+85/12muv6cknn9Szzz6rO+64I23rQMu0uTPoTz/9tMaOHavvfe97Wr58ufr27asf/vCHuvrqq1s7NQAAAADtWDTD5zdvLS/X1KlTXdOaOnsuSd27d5ff79eWLVtc07ds2aJevXo1ucwtt9yiSy+9VFdddZUk6ZhjjlFtba2uueYa/fznP5fP1+bO37Y7bW4LfPDBB5o5c6a+9rWv6fnnn9d1112n66+/Xg8//HBrpwYAAAAAKQsGg+ratavr0dwAPTc3V8OGDdPixYvj02KxmBYvXqzhw4c3ucyuXbsaDcL9fuvum6aZ2g1UkV5t7gx6LBbT8ccfr1//+teSpKFDh2rdunWaNWuWJk6c2OQyTX2XIxaNyOdvc8UHAAAA0Eoy/R10r6ZOnaqJEyfq+OOP14knnqjp06ertrY2flf3yy67TH379o1fJj9+/Hjde++9Gjp0qE466SRt2LBBt9xyi8aPHx8fqKN1tbkRau/evXXUUUe5ph155JF64oknml2moqJCt99+u2tan2PGqN+QsRnJEQAAAAAy7YILLtDnn3+uW2+9VVVVVTruuOO0cOHC+I3jNm3a5Dpj/otf/EKGYegXv/iFPv30U/Xo0UPjx4/XnXfe2VpFQII2N0AfMWKE1q9f75r27rvv6uCDD252mfImvstx+rWzMpIfAAAAgPYploXfEC4rK1NZWVmT7y1btsz1OhAIaNq0aZo2bdp+yAypaHMD9ClTpujkk0/Wr3/9a51//vl65ZVXdP/99+v+++9vdpmmfpqAy9sBAAAAANmkzY1STzjhBM2fP1/l5eX65S9/qYEDB2r69Om65JJLWjs1AAAAAO1YNMu+g472p80N0CXp29/+tr797W+3dhoAAAAAAKRNmxygAwAAAMD+lm13cUf7k313OQAAAAAAoAPiDDoAAAAAJCFmcn4TmdVhB+h1B1qXp4TrrCr4ancn6w3D9Bxrd3GOJCli3yg+YofyGivm3GjeZ+VW38WOV2fFD3iIZ0Savvym7kCrUwl3avLt5nOzW8pHOw+0Xuc4K/IWx5VLkZVLJM967fWKIacsUbveonacL3d635axHMMVIz693m89e9xTvqrNt+LlWjlEQ3acYLOLNM9vlyNq2LlaL+P15UuynHZ97OrX2VreKVO9z/V+Mky//WzntKvYjhHyHkuSdg6wKv6dHdZvdsa3Q7JlayCaa7iWra3LtXK1y2smGTMWsOJsr8m3F2ywDmc7Jpmes+6NO4sk7ameUK2VW8BjOSOdrNzquuXGp9XuisoV3CNnMdNu8/EwScZz2rgk7fJbeRl2+9i6s8B6I8lymnbjNmPuTiHSWZ7iNBQ+4iArht09mPZ+ZES8xdlVbG3MXXa76uTESaHaYznWQiG7j/eFnX7IW7B4fdnbbs0O62dH66JWXK/9lyTVd7Vj2rt0LGL9x+exoKbdf/nC1vL1dlNQzH72EC+SZ/eBAbtfzXPaiT2D13Zh11s07HRodpgUjmv1XZ2+z6rzlTWHWyGdz/FJlrOuyMolVGiXLWoFqNkd9BRH2tNPxOy6d9pB/DiUZDnjXYFdlvh+6OyfKdWXXT47p+pdVj/r9BnJxnQ+P9Qd0t2K52/wnv0ZKtk+3/nc5MQI2du02um/PLb9cGe7/kO5jd/0GKv2kEJJe/afL0NWR7anfSUXJ2L3EQ3beMxOr7av1TASPwc1x/ksErM/52yt6+LOKYV+GoClww7QAQAAAMCLaEvOTgFJaJPXaAwYMECGYTR6lJaWtnZqAAAAAACkpE2eQX/11VcVjUbjr9etW6dvfetb+t73vteKWQEAAABoz7iLOzKtTQ7Qe/To4Xp911136dBDD9Wpp57aShkBAAAAANAybXKA3lB9fb0eeeQRTZ06VYbBX7QAAAAAZAZ3cUemtfkWtmDBAm3fvl2XX355a6cCAAAAAEDK2vwZ9AcffFDjxo1Tnz59mp0nFAopFAq5psWiEfn8bb74AAAAAPaTGHdxR4a16TPoH330kV544QVdddVVe52voqJChYWFrsfWNS/spywBAAAAtAdR08joA2jTA/TZs2eruLhYZ5111l7nKy8vV3V1tetRfPzo/ZQlAAAAAAD71mav8Y7FYpo9e7YmTpyoQGDvxQgGgwoGg65pXN4OAAAAwAtuEodMa7Mt7IUXXtCmTZt0xRVXtHYqAAAAAAC0WJs9jTxmzBiZptnaaQAAAADoIGJ8TxwZ1mbPoAMAAAAA0J602TPoAAAAALA/8TNryDTOoAMAAAAAkAU67Bn03J3W99d9gZj12h9NOVZOrRXDMP2SJH+dNT0a9PYXNl+99exPePblOLnleAjW9Pfzc2qt6bEcb7kZdgqdc6ykfBHrddTvKYwVK2z9XSiwy8ol3Cm1v0Q6ZfFFrOUNO6dO+VaONeqcdCxf2H62Yyhq52TXoxHzllterhVwd70Vx8yJuXL0xMklmJCLU23JfhfKbhK5OyJ2Ltbub/pNb3EaxHJyCFbbEwIpxJLUaYtVX93zaiVJH9ttX/ne24az30TsdhHwWxUWsduwkeStK5x67twpJEn6Uvnx95z2YibZ/p39pzDX6hw+sacHgnaDMIONF9oLp4/xRfYUJhCwVhL2FKkJdp8ow9vO7fSlkuSz9xunGXQOWhtlt8d2Ydpt34g/O294bxc5W3dKkgK7re0Yy7X3Tb+3e5nk7rTK6fdZzzEP3XIip+9y6jySY+WSW2v1kVFvzUKG3c8UB62y1kasAF77L0nyh9yvzRS/c2k4faDdJuJxUwjn7HeN2oMj2RwTOgHDee1r8u2k+EN2+eqsyu6ZU23n5KzEY5yQ4VouYH9GqfewHZy6DjuHM+f447SHZPtCZ756+9htddOKHOD090mnFOd83nH47X7a9HmrMKe/z/1yt71Y7p73dtl5dko2KXd5fM7nHqf/SjJMfP2hJibG7HJ53J/ytlk55OyyjtsR5y7iHuveaOqzod0egl/Znw36JtepOWf4nLbqU0J7aMff0+Y76Mg0zqADAAAAAJAF2twAPRqN6pZbbtHAgQOVn5+vQw89VHfccQd3dAcAAACQUTHTl9EH0OYucb/77rs1c+ZMPfzwwzr66KO1Zs0aTZo0SYWFhbr++utbOz0AAAAAAFLS5gboL730ks455xydddZZkqQBAwbo73//u1555ZVWzgwAAABAe8Z30JFpbe46ipNPPlmLFy/Wu+++K0l64403tHLlSo0bN66VMwMAAAAAIHVt7gz6zTffrB07duiII46Q3+9XNBrVnXfeqUsuuaS1UwMAAADQjvE76Mi0NjdA/8c//qFHH31Uc+fO1dFHH63KykpNnjxZffr00cSJE5tcJhQKKRRy/95FLBqRz9/mig8AAAAAaKfa3CXuN954o26++WZdeOGFOuaYY3TppZdqypQpqqioaHaZiooKFRYWuh6b31y8H7MGAAAA0NbFTCOjD6DNDdB37doln8+dtt/vVywWa3aZ8vJyVVdXux69jz0906kCAAAAAJC0NneN9/jx43XnnXeqf//+Ovroo/X666/r3nvv1RVXXNHsMsFgUMFg0DWNy9sBAAAAeMFZbmRamxul/vGPf9Qtt9yiH/7wh9q6dav69OmjH/zgB7r11ltbOzUAAAAAAFLW5gboBQUFmj59uqZPn97aqQAAAADoQDiDjkxrc99BBwAAAACgPWpzZ9ABAAAAoDVwBh2ZxgAdAAAAAJIQEwN0ZFaHHaAb9q+ymfZfwUwz/bFleAtq2LkYEffiRgv6gfgf+YyE3FKUlk4pmvA6IcdkxbehUza7vnJ8iSvY/+LbzM7R8KWhgSXE2LNtvcU27eTii8Ubmoc4zsqdRVvYrnz1VoBIzO+K67Vs9kLWU8x69qUUY4+Ar3Hh4m3Pn2JQOyWf3/pPzGuOTr/V4EtKZkv/oh/vb1KsryZWbyTGTDq2074S22pCYC9i1jKG3T2kug2d5aMxq/IDzjZoQTPbUz/udXgP5H4ZaElfmJCLGUutfRlRu60GrDK2qHtOqONGTT7JdmEm9M/OvuPzx5qO6yE302ctHHYaVoq7ZbwoiUXy0PaNxIIk7kdJih9jw3Y9RRJX5C1eU8uk+hnM2Y+Neiuphn1iyvuRk5MdK+VjiL1YJNogqRQ/HzoV5JSpLtLCj+9G4/97Po47RbCXizgV1pJ+GoCkDjxABwAAAAAvuMQdmdYmbxK3c+dOTZ48WQcffLDy8/N18skn69VXX23ttAAAAAAASFmbPIN+1VVXad26dfrb3/6mPn366JFHHtHo0aP1v//9T3379m3t9AAAAAC0Q5xBR6a1uTPou3fv1hNPPKHf/OY3GjlypAYNGqTbbrtNgwYN0syZM1s7PQAAAAAAUtLmzqBHIhFFo1Hl5eW5pufn52vlypWtlBUAAACA9o4z6Mi0NncGvaCgQMOHD9cdd9yhzz77TNFoVI888ohWrVqlzZs3t3Z6AAAAAACkpM0N0CXpb3/7m0zTVN++fRUMBvWHP/xBF110kXy+posTCoW0Y8cO1yMWTfydEAAAAABoXsw0MvoA2uQA/dBDD9Xy5ctVU1Ojjz/+WK+88orC4bAOOeSQJuevqKhQYWGh67H5rcX7OWsAAAAAAJrXJgfojs6dO6t379766quv9Pzzz+ucc85pcr7y8nJVV1e7Hr2POX0/ZwsAAACgLTNNI6MPoM3dJE6Snn/+eZmmqcMPP1wbNmzQjTfeqCOOOEKTJk1qcv5gMKhgMOia5vO3yaIDAAAAANqpNjlKra6uVnl5uT755BMVFRXpvPPO05133qmcnJzWTg0AAABAOxUTZ7mRWW1ygH7++efr/PPPb+00AAAAAABImzY5QAcAAACA/Y07rSPT2vRN4gAAAAAAaC84gw4AAAAASeBO68i0jjtAt/ctM2r9x7nhQyr7nBE1E17LFcswlRQjZj37IolvmCnn5ojnEnO/TpZTBueyHl+99Tqal2ThGsaKZqhjs1MJ+K1CplJfpnNNiVNfPu/lkyTTXsypNyPZRtCEZreZ1/LFnJzs9mSX1UjDdTSN2qzn5a3k6qJWl+SUOSVOnYdb2M7sOM62VINN6Ozj8nhfSp/TDux9wHnttbhOGNO3p4zOvplqP+Hsl/HiGu7nfWpQPz6fXaKEcnoWM9zPqe9GUszKydl28TbmMWZ8E0asHcdrP+8Sb2N23eeYrhy9xnH47WSCLd0xJfnCTpL2k9djh73tTLsvNZyUUomXUNdmwL0Oz8J25xdI8cDYMDWnPdkhqqOd7Nfejt9OHCNh06U0IEhsk0Yzz0nG8dUbTeaWkoR+NRZzDkjewsT350jjXrRFxxE16ANb2LdGoy0/yBphu/+yy1QXSfGmyE31U84xM+qtE4t/vrGfIzF3nwggdR13gA4AAAAAHvAddGRa1n0HfcWKFRo/frz69OkjwzC0YMEC1/umaerWW29V7969lZ+fr9GjR+u9995rnWQBAAAAAEiTrBug19bWasiQIZoxY0aT7//mN7/RH/7wB82aNUsvv/yyOnfurLFjx6qurm4/ZwoAAACgIzFNI6MPIOsucR83bpzGjRvX5HumaWr69On6xS9+oXPOOUeS9Ne//lU9e/bUggULdOGFF+7PVAEAAAAASJusO4O+Nxs3blRVVZVGjx4dn1ZYWKiTTjpJq1atasXMAAAAALR3MdPI6ANoUwP0qqoqSVLPnj1d03v27Bl/DwAAAACAtijrLnHPhFAopFAo5JoWi0bk83eI4gMAAABIA7MlP/kJJKFNnUHv1auXJGnLli2u6Vu2bIm/15SKigoVFha6HpvfWpzRXAEAAAC0LzEZGX0AbWqAPnDgQPXq1UuLF+8ZXO/YsUMvv/yyhg8f3uxy5eXlqq6udj16H3P6/kgZAAAAAICkZN013jU1NdqwYUP89caNG1VZWamioiL1799fkydP1q9+9St97Wtf08CBA3XLLbeoT58+mjBhQrMxg8GggsGgaxqXtwMAAADwgp9CQ6Zl3Sh1zZo1GjVqVPz11KlTJUkTJ07UnDlz9NOf/lS1tbW65pprtH37dp1yyilauHCh8vLyWitlAAAAAABaLOsG6CUlJTL3cvcFwzD0y1/+Ur/85S/3Y1YAAAAAOjp+Cg2Z1qa+gw4AAAAAQHuVdWfQAQAAACAb8TNryDTOoAMAAAAAkAU67Bl0I2o/+60/gxXkhiRJ21OIlbsjIknyRazqTPmrKTHryRe2n+utZ3/AeiPqJVZiDvZrf13MXoc/pdy659VKkj6p97Z4U7kEQlbd+yLWBMNeh5nkn4389vKGXeE+azOoa7BOkrS1BSmaOVYygZyoK7dkBQPWciF7OWcbGp42otzr9tl/snX+cmt4+xOuM3ugus6eYt9Y0e+xcA1j2vtPTq1T0NRiBb7aJUnqmmPllko9xXOyy2nErHaRm2M1jLCTWpI7qNOecgONk/Fb3YWiwUZvNc3O6YDc3a7XeUFrZ69JMkw8N7uPCNTuyc3vS6Gf0J79zYnptDMz3r6Sqy+nLUhSvl2uXfakLnb/uiXppOznhPbkb0G/Y361XZLki/S2XqfYTwd2WTXss+vJH97b3Mnx2+Wsj/cTdv+c5C4eb/N2Gzgkz937ee2/JMlXb+/b9kb0BVLbt02nPdntI1DnJOs9VrycCY3c87ZMmN8XsMtob4dUShqvJ7uj6R7Y6U4uyf46sNta3nQO0Xb9BXPtfsxDTs529/nd6zYiHoI0XM6ud6dvTPZY3RSnHfjCVv10Dlo7906P2zLeb33xlZ1bj8bvJdtAEjaVs02DOal1PD6n/n0N6j/Ffsf/+XYrVrjAyslvbYR4s0qyr/Dbn23MBo3cqaec7VY/7YvmJhUrHsJed1HQOr59lFwqbRp3cUemcQYdAAAAAIAskHUD9BUrVmj8+PHq06ePDMPQggULXO8/+eSTGjNmjLp16ybDMFRZWdkqeQIAAADoWEzTyOgDyLoBem1trYYMGaIZM2Y0+/4pp5yiu+++ez9nBgAAAABA5mTdd9DHjRuncePGNfv+pZdeKkn68MMP91NGAAAAAMDvoCPzsu4MOgAAAAAAHVHWnUEHAAAAgGzE76Aj0zrEAD0UCikUCrmmxaIR+fwdovgAAAAAgDagQ1ziXlFRocLCQtfjs3WLWzstAAAAAG0Id3FHpnWIAXp5ebmqq6tdjz6DT2/ttAAAAAAAiMu6a7xramq0YcOG+OuNGzeqsrJSRUVF6t+/v7788ktt2rRJn332mSRp/fr1kqRevXqpV69eTcYMBoMKBoOuaVzeDgAAAMALznIj07LuDPqaNWs0dOhQDR06VJI0depUDR06VLfeeqsk6emnn9bQoUN11llnSZIuvPBCDR06VLNmzWq1nAEAAAAAaKmsO41cUlIicy+3R7z88st1+eWX77+EAAAAAEASN3FHpmXdGXQAAAAAADqirDuDDgAAAADZiO+gI9M4gw4AAAAAQBbosGfQY3bJfb70fZPEiFnPZkKtJv2HNvvPJaY/YXoqKTrLGO7nWI7hnu6Rzwns5JpCHDPHnVy8eIk574OzDRPrpz6aWIFJ5OSs045ltLBdOLdRiNmpmNEW/C3MSS5m15e/6beTDRN/HY9jeIojSUa8vuxlW/inPjPX2pgxueOl1L7S9WdHexvWRRp3k57zcjah8x87x6jdLrzGc7adL7qnnRpGim3WXs7023Ufa/mZgXg/kdjmkgxtJPzH9Ds7lPe2Gg+Vk2PHkuvZK6eenPo2IqnFaYrhlK+FR+aYXUE5RjT1XOwqj+9PKTaL+LZzXqehK2zumJh0u3CWd+rbaV4t6adtTr3VmTnuCV7jNNOuvLR9Z16nfE6bT7WrcJaP5aS2fENODs7nJr8vllpOTn3Yv9TjavJe26zpfo5/pkvxbKnT1n0Ny+ZsE68h/VblO597cvyp79vNJWAGPLZ/pywJixnu7rp94kvoyLAOO0AHAAAAAC+4xB2ZlnWXuK9YsULjx49Xnz59ZBiGFixYEH8vHA7rpptu0jHHHKPOnTurT58+uuyyy+K/iQ4AAAAAHcmMGTM0YMAA5eXl6aSTTtIrr7yy1/m3b9+u0tJS9e7dW8FgUIcddpiee+65/ZQt9iXrBui1tbUaMmSIZsyY0ei9Xbt26bXXXtMtt9yi1157TU8++aTWr1+vs88+uxUyBQAAANCRmGZmH17NmzdPU6dO1bRp0/Taa69pyJAhGjt2rLZu3drk/PX19frWt76lDz/8UI8//rjWr1+vBx54QH379m1hzSBdsu4S93HjxmncuHFNvldYWKhFixa5pv3pT3/SiSeeqE2bNql///77I0UAAAAAaHX33nuvrr76ak2aNEmSNGvWLD377LN66KGHdPPNNzea/6GHHtKXX36pl156STn2/VkGDBiwP1PGPmTdGXSvqqurZRiGDjjggNZOBQAAAEA7ZppGRh+hUEg7duxwPUKhUJO51NfXa+3atRo9enR8ms/n0+jRo7Vq1aoml3n66ac1fPhwlZaWqmfPnho8eLB+/etfKxpt4c0HkTYZG6Bv3749U6Hj6urqdNNNN+miiy5S165dM74+AAAAAMiUiooKFRYWuh4VFRVNzrtt2zZFo1H17NnTNb1nz56qqqpqcpkPPvhAjz/+uKLRqJ577jndcsstuueee/SrX/0q7WVBatIyQL/77rs1b968+Ovzzz9f3bp1U9++ffXGG2+kYxWNhMNhnX/++TJNUzNnztzrvE39JSoWTeNv4wAAAABo/0wjo4/y8nJVV1e7HuXl5WlLPxaLqbi4WPfff7+GDRumCy64QD//+c81a9astK0DLZOWAfqsWbN00EEHSZIWLVqkRYsW6d///rfGjRunG2+8MR2rcHEG5x999JEWLVq0z7PnTf0lavObi9OeFwAAAACkKhgMqmvXrq5HMBhsct7u3bvL7/dry5YtrulbtmxRr169mlymd+/eOuyww+T3++PTjjzySFVVVam+vj59BUHK0jJAr6qqig/Qn3nmGZ1//vkaM2aMfvrTn+rVV19NxyrinMH5e++9pxdeeEHdunXb5zJN/SWq97GnpzUvAAAAAO1bNt3FPTc3V8OGDdPixXtOPMZiMS1evFjDhw9vcpkRI0Zow4YNisVi8Wnvvvuuevfurdzc3JTqBOmVlgH6gQceqI8//liStHDhwviNCkzT9HzDgZqaGlVWVqqyslKStHHjRlVWVmrTpk0Kh8P67ne/qzVr1ujRRx9VNBpVVVXVPv/i09Rfonz+rLuBPQAAAAAkberUqXrggQf08MMP6+2339Z1112n2tra+F3dL7vsMtcl8tddd52+/PJL/fjHP9a7776rZ599Vr/+9a9VWlraWkVAgrSMUs8991xdfPHF+trXvqYvvvgi/jNpr7/+ugYNGuQp1po1azRq1Kj466lTp0qSJk6cqNtuu01PP/20JOm4445zLbd06VKVlJSkXggAAAAA2JsUfqs8ky644AJ9/vnnuvXWW1VVVaXjjjtOCxcujN84btOmTfL59pyTPeigg/T8889rypQpOvbYY9W3b1/9+Mc/1k033dRaRUCCtAzQf//732vAgAH6+OOP9Zvf/EZdunSRJG3evFk//OEPPcUqKSmRuZfrO/b2HgAAAAB0JGVlZSorK2vyvWXLljWaNnz4cK1evTrDWSFVaRmg5+Tk6IYbbmg0fcqUKekIDwAAAACtzjSN1k4B7Vzavoj93nvvaenSpdq6davrpgOSdOutt6ZrNQAAAAAAtEtpGaA/8MADuu6669S9e3f16tVLhrHnL0uGYTBABwAAAND28W1bZFhaBui/+tWvdOedd7apmwtEc60/IkQj1k0TautT/1mB+sIcSVLM/jnBWE5qcUyfk5sdx36OhK3Ani6oSbz8xu5MInnW9JjXLW/nti3U2ZVjS0SCVi6mk4vHK4acsjjLOWXaEcrznowdw7S3oRm124dd9w1+KjIpdeEcV7xo2KpAM5U9zrA2nmHn5LST+DY2kjxS2LNHCt31Y0ZSv1TLqaf6LnZSMY852eqL8iVJO+qt3FKqJycnn/Ns5bA7lOOanmxuZsBwLd9wn3L2Ta9tdltdZzuW9RSqT62gTluPdNrTMCNRj43UZsScdmUlZUadCvRWuFhkzw1oakPuDsLzPumsOiGHWAv6HbO4SFKDvstpBx63YbizvS+b7v7LdHdHyXH2Safu7H0xfgxJMpjTop1t92Goh/v9FH6vJer0z07faO/rnnsLO4Bh/6BLJN8J6D0npz9VQhP12N004rRdww6Uys/bhDs79WUtvbW+a0q5OO0rXk/2/plKX5F4PHPqK5ZaVyEzYNVPLNfZpqkfOyJ2l+C09fhnsFjT8zfHWT7W80ArxwZla7Sv70vCTuxsg/oUPx869RyNNWhRKbbVaA+rPTl94O6IfVxzNkGSm8L5zGv49iQSy7HLeYAVPNnPh/H2ZT9vr0/hsxeAJqXlZ9a++uorfe9730tHKK1YsULjx49Xnz59ZBiGFixY4Hr/tttu0xFHHKHOnTvrwAMP1OjRo/Xyyy+nZd0AAAAA0BzTNDL6ANIyQP/e976n//znP+kIpdraWg0ZMkQzZsxo8v3DDjtMf/rTn/TWW29p5cqVGjBggMaMGaPPP/88LesHAAAAAKA1pOUS90GDBumWW27R6tWrdcwxxygnx32N9/XXX590rHHjxsV/R70pF198sev1vffeqwcffFBvvvmmTj/9dG+JAwAAAECy+A46MiwtA/T7779fXbp00fLly7V8+XLXe4ZheBqge1FfX6/7779fhYWFGjJkSEbWAQAAAADA/pCWAfrGjRvTESZpzzzzjC688ELt2rVLvXv31qJFi9S9e/f9mgMAAACAjobviSOz0vIddEd9fb3Wr1+vSCSSzrCNjBo1SpWVlXrppZd0xhln6Pzzz9fWrVubnT8UCmnHjh2uRyya2RwBAAAAAPAiLQP0Xbt26corr1SnTp109NFHa9OmTZKkH/3oR7rrrrvSsQqXzp07a9CgQfrGN76hBx98UIFAQA8++GCz81dUVKiwsND12PL6C2nPCwAAAEA7Zmb4gQ4vLQP08vJyvfHGG1q2bJny8vb8DuLo0aM1b968dKxir2KxmEKh0F7zq66udj16Dh2d8bwAAAAAtCMM0JFhafkO+oIFCzRv3jx94xvfkGHs+V7G0Ucfrffff99TrJqaGm3YsCH+euPGjaqsrFRRUZG6deumO++8U2effbZ69+6tbdu2acaMGfr000/3+jvswWBQwWDQNc3nT0vRAQAAAABIi7SMUj///HMVFxc3ml5bW+sasCdjzZo1GjVqVPz11KlTJUkTJ07UrFmz9M477+jhhx/Wtm3b1K1bN51wwgl68cUXdfTRR7esEAAAAACwNyY3iUNmpWWAfvzxx+vZZ5/Vj370I0mKD8r/8pe/aPjw4Z5ilZSUyDSbv77jySefTD1RAAAAAACyVFoG6L/+9a81btw4/e9//1MkEtF9992n//3vf3rppZca/S46AAAAALRFezmPCKRFWm4Sd8opp6iyslKRSETHHHOM/vOf/6i4uFirVq3SsGHD0rEKAAAAAADatbScQV+3bp0GDx6sBx54oNF7CxYs0IQJE9KxGgAAAABoPZxBR4alZYA+duxYrVy5UgMHDnRNf+KJJ3TZZZeptrY2HatJq0CdtXcFcqKSpM659ZKkr1LY6fx1MUmSYfqtCXYMI9aym0gYEevZ54+54ibFZybkYj0HQtaEcNhjbnacA3J3u3NMoYym3wrmr7eefZHU6snZhk4Ohp1jji/mep0Mp358Yee1HdOuRyPqLbe8HCtQyKl/u8zONvXCdK5zsQu0J0dnjuTqz7SX99dYAYyo9ZOIzn0cPW1Lp1wBKwlfxOdOxeMNVPx1VsX0zt8hSfogmkJOcpaxU8ixksy19/HdKR5Q84NWfe3y7wkQ3ybJxrTnKwrusv7jtAefuw0ny2mPgdo9DTOYY9Wh53I68zvtwO5v4pOTjOcLxBuk/Ha5zBxrWl7AbnPJljO+38Rcufns/SeldvFFtb1sUZPrSpbT7+TZ7cJX76zAc0rxdQfsuos45TLd7yfNrvfiHGs/qovlWKnFml2i+VBO/2w3MV/Abqtec/K7F4jn4nQZHuL5EvrPgP3rqqbX/chZaZ5VOKftxvvCFPoKX9haKMfeJ4/t9LEkaV7CsWRfArvttpBnfZ5w9oFce/8Oe2j78eNajtNPy/WcbPty6iOa4/5ckfg5wwvnOObIt4+Z21P82OSrqbNz6Rqf5nk7Jmwjv72v56f4+dCp58559fFpIV++lZvHPsxXazV2I9JFktSzU40k6XOPfYXfbusNL9GOH0922/tDJCepWM6u7NRz9zzrs/5HTrfdws/AQEeWlkvcr7rqKo0ePVpVVVXxafPmzdNll12mOXPmpGMVAAAAANC6TCOzD3R4aRmg33777TrzzDM1evRoffnll5o7d64mTZqkv/71r3v9ffKmrFixQuPHj1efPn1kGIYWLFjQ7LzXXnutDMPQ9OnTW1YAAAAAAABaWVoG6JL0xz/+UUOGDNE3vvENXX311fr73/+u8847z3Oc2tpaDRkyRDNmzNjrfPPnz9fq1avVp0+fVFMGAAAAgKQZZmYfQMrfQX/66acbTTv33HP14osv6qKLLpJhGPF5zj777KTjjhs3TuPGjdvrPJ9++ql+9KMf6fnnn9dZZ53lLXEAAAAAALJQygP0vd2Z/aGHHtJDDz0kSTIMQ9Goxzts7UUsFtOll16qG2+8UUcffXTa4gIAAADAXnGWGxmW8gA9FkvhtrBpcPfddysQCOj6669vlfUDAAAAAJAJafmZtf1l7dq1uu+++/Taa6/JMJK/y2EoFFIoFHJNi0Uj8vnbVPEBAAAAtCbutI4MS9tN4pYvX67x48dr0KBBGjRokM4++2y9+OKL6QovSXrxxRe1detW9e/fX4FAQIFAQB999JF+8pOfaMCAAc0uV1FRocLCQtdj81uL05obAAAAAAAtkZYB+iOPPKLRo0erU6dOuv7663X99dcrPz9fp59+uubOnZuOVUiSLr30Ur355puqrKyMP/r06aMbb7xRzz//fLPLlZeXq7q62vXofczpacsLAAAAQAdgZviBDi8t13jfeeed+s1vfqMpU6bEp11//fW69957dccdd+jiiy9OOlZNTY02bNgQf71x40ZVVlaqqKhI/fv3V7du3Vzz5+TkqFevXjr88MObjRkMBhUMBl3TuLwdAAAAAJBN0nIG/YMPPtD48eMbTT/77LO1ceNGT7HWrFmjoUOHaujQoZKkqVOnaujQobr11lvTkSoAAAAApIYz6MiwtJxGPuigg7R48WINGjTINf2FF17QQQcd5ClWSUmJTDP51vnhhx96ig8AAAAAQDZq0QD9iiuu0H333aef/OQnuv7661VZWamTTz5ZkvR///d/mjNnju677760JAoAAAAArYqz3MiwFg3QH374Yd1111267rrr1KtXL91zzz36xz/+IUk68sgjNW/ePJ1zzjlpSRQAAAAAgPasRQP0hpeif+c739F3vvOdFicEAAAAAFmJ30FHhrX4O+g7d+5UXl7eXufp2rVrS1eTdrEc6zkS9kuSautzU44V7mzday9m16bp3HrP8HYNjGnYO7zPnWMs4ms4OclgTU+OBA1X7KTD2Sv/MtTJeu233/BYRkkyolYOkTw7F797HcmK5hqu5Zw4u8Pem7XT15rOonaOTt17za0u7K5gM2bn6m9q7iSZ7vLGjw8et0Gk0N3WzVgKcZxcYu5tadr15jmnTlbFf1Hf2YrTgvYV349s9fY+Lo/HU6eewxGngTZ400h43hd7Pmf/ceovarcvz32F31o+3GVPg6qPtKRxKV4+Z5t6rfr4tpcUsvdBw45VW2//ikayQQ2nPflci0WcQ00q7aLIOg7F92VnB/J5ixXuZC3n1Lfh5JTKJY92CtGor8npXtus7Pr+uK5IkhQ2U+u/JClm968xu3+IRazXnltZvH+xnqJO95NCfcX7voQ+P74tk2wX8b7Tnj1+jPXHXLl64fSB/k5WUh/Vd08pVqSTlUvU6VNj1utQvX1M8dD2TZ/7OOZwPqskm1t8jc5xzOd+3ZL6MuPH7ZyElSWZm7189ICEzyZqUM5kOeWzyxOx9/VIONc1Pelw9vrr6hskkuox8oB8Kwc71Jd1+e6ckswtGrb71AbzO3UW6RJw5b0viZ9n4p8PU/wMDGCPFg/QDzvssGbfM01ThmEoGo22dDUAAAAA0Kr42wMyrcUD9Mcff1xFRUXpyEWStGLFCv32t7/V2rVrtXnzZs2fP18TJkyIv3/55Zfr4Ycfdi0zduxYLVy4MG05AAAAAEAjDNCRYS0eoI8YMULFxcXpyEWSVFtbqyFDhuiKK67Queee2+Q8Z5xxhmbPnh1/HQwG07Z+AAAAAABaQ1p+Bz2dxo0bp3Hjxu11nmAwqF69eu2njAAAAAAAyLwUbh2zx8EHHyy/v4U3JkrBsmXLVFxcrMMPP1zXXXedvvjii/2eAwAAAAAA6dSiM+gbN25MVx5JO+OMM3Tuuedq4MCBev/99/Wzn/1M48aN06pVq1rljwUAAAAAOgZuEodMy7pL3PflwgsvjP//mGOO0bHHHqtDDz1Uy5Yt0+mnn97kMqFQSKFQyDUtFo3I529zxQcAAAAAtFMtusQ9GxxyyCHq3r27NmzY0Ow8FRUVKiwsdD2q3li8H7MEAAAA0OaZRmYf6PDa/AD9k08+0RdffKHevXs3O095ebmqq6tdj15Dmj7bDgAAAABAa8i6a7xrampcZ8M3btyoyspKFRUVqaioSLfffrvOO+889erVS++//75++tOfatCgQRo7dmyzMYPBYKOfYuPydgAAAACe8B10ZFjaRqmLFy/W4sWLtXXrVsViMdd7Dz30UNJx1qxZo1GjRsVfT506VZI0ceJEzZw5U2+++aYefvhhbd++XX369NGYMWN0xx138FvoAAAAAIA2LS0D9Ntvv12//OUvdfzxx6t3794yjNS/P1FSUiLTbP5PU88//3zKsQEAAAAgZZxBR4alZYA+a9YszZkzR5deemk6wgEAAAAA0OGkZYBeX1+vk08+OR2hAAAAACAr8TvoyLS03MX9qquu0ty5c9MRCgAAAACADiktZ9Dr6up0//3364UXXtCxxx6rnJwc1/v33ntvOlaTXhn465fz04Wp/mUtvlxsr7N5SybhZUv/6hezA7UoTjrKt5ccnFsgePkpyaz+a6iTWzYmaafU0tQMu03E0vD7n4n7kdnCmEYm6t2pN8P10vPyalC0lpaz2VWlEraFVRZfZ2L7SkMRW9xW7eXNmJVMWluHHdxoYR8ZTUNFOdvA55Q31faVGCgdEtpDyts0saG1QLxd2CFT3QYZ2Y0Ti5fiOpx2mZEusaWfm5oMmlpMp35Mn+tlymKxlp8Li/cJ9nPUzMCvJMecRpzq4h3o97uz8OMY2pe0DNDffPNNHXfccZKkdevWud5ryQ3jAAAAAADoKNIyQF+6dGk6wkiSVqxYod/+9rdau3atNm/erPnz52vChAmued5++23ddNNNWr58uSKRiI466ig98cQT6t+/f9ryAAAAAAAXzqAjwzJwjUzL1NbWasiQIZoxY0aT77///vs65ZRTdMQRR2jZsmV68803dcsttygvL28/ZwoAAAAAQPqkfAb93HPP1Zw5c9S1a1ede+65e533ySefTDruuHHjNG7cuGbf//nPf64zzzxTv/nNb+LTDj300KTjAwAAAEAqsvGWQGhfUj6DXlhYGP9+eWFh4V4f6RKLxfTss8/qsMMO09ixY1VcXKyTTjpJCxYsSNs6AAAAAABoDSmfQZ89e3aT/8+krVu3qqamRnfddZd+9atf6e6779bChQt17rnnaunSpTr11FP3Sx4AAAAAOqCOdMd6tIq03CROkiKRiJYtW6b3339fF198sQoKCvTZZ5+pa9eu6tKlS1rWEYtZvy9xzjnnaMqUKZKk4447Ti+99JJmzZrV7AA9FAopFAq5Y0Uj8vnTVnwAAAAAAFokLTeJ++ijj3TMMcfonHPOUWlpqT7//HNJ0t13360bbrghHauQJHXv3l2BQEBHHXWUa/qRRx6pTZs2NbtcRUVFo8vuq95YnLa8AAAAAHQAZoYf6PDSMkD/8Y9/rOOPP15fffWV8vPz49O/853vaPHi9A2Ec3NzdcIJJ2j9+vWu6e+++64OPvjgZpcrLy9XdXW169FryOlpywsAAAAAgJZKyzXeL774ol566SXl5ua6pg8YMECffvqpp1g1NTXasGFD/PXGjRtVWVmpoqIi9e/fXzfeeKMuuOACjRw5UqNGjdLChQv1r3/9S8uWLWs2ZjAYVDAYdE3j8nYAAAAAXnAXd2RaWkapsVhM0Wi00fRPPvlEBQUFnmKtWbNGo0aNir+eOnWqJGnixImaM2eOvvOd72jWrFmqqKjQ9ddfr8MPP1xPPPGETjnllJYVAgAAAAD2hgE6MiwtA/QxY8Zo+vTpuv/++yVJhmGopqZG06ZN05lnnukpVklJiUxz7y3/iiuu0BVXXJFyvgAAAAAAZJu0DNDvuecejR07VkcddZTq6up08cUX67333lP37t3197//PR2rAAAAAIBWxSXuyLS0DND79eunN954Q/PmzdMbb7yhmpoaXXnllbrkkktcN40DAAAAAABNS8sAfcWKFTr55JN1ySWX6JJLLolPj0QiWrFihUaOHJmO1QAAAABA6+EMOjIsLQP0UaNGafPmzSouLnZNr66u1qhRo5q8gVxrC4SsZ8Nn7WX5uWFJ0lcpxMqpjUmSfBG/HTS1nAwrjPz19rOdoz/HesNTf+Breu7ALmu6r4u3JJ3cuuZaSfnC9htmCoW1U3O2gS/qnp6swG7TlYMvYr3sErQC13iJ58xrl9PZhr6ANcEpf7Jyc6xk6u2yBXKs/xgp7ApGzErGtH8UMZUYVgA7l+p6O671ywaGP/UjjWHn5GwLZ3/y2i5yv6qTJHUJJLSvVDjFsXNwtkVdwvR9ceo5316+usFiTn6xZHtQe90H5Na5Juc47STJMPH12209sGtPw/T7rP97bKpxTjuT3R5Me1saydZXoEEufrvPshftbPcbXyabi5nwH2cfb0G7ML6otmLUd7Ne29vX9Phjo06d++z68bXk8GYXz6mvsHMIibjfT5qdU9/gdtdkr/2XJPnrrVjOccjnT61lJWxCBZxdIIVDh7PNnPI4z2Yzx7tkk3P6Lae/T+XDd8DuYAJ1VoyegR0ppZRjH6OjuaYrR6cfizS5VNOcfsIXcDaCe3qy5XS2oRFJ2A8Nd1wvnPoyolbMYMDZuN7iGHYu/i9r7Vy6xN/ze/2c4vR5zmexkPU6L8fqob/0WE6nb/CnuO80FLDL5w91kiQF/dZGjPeVSebmtPGGszvtIWenVWG+aE5SseIfl+xyOp8PAbRcWgbopmnKMBp3gF988YU6d+6cjlUAAAAAQOviDDoyrEUD9HPPPVeSddf2yy+/3PVb49FoVG+++aZOPvlkTzFXrFih3/72t1q7dq02b96s+fPna8KECfH3m/pDgCT95je/0Y033ui9EAAAAAAAZIEWDdALCwslWWfQCwoKXDeEy83N1Te+8Q1dffXVnmLW1tZqyJAhuuKKK+J/AGho8+bNrtf//ve/deWVV+q8885LoQQAAAAAkBzu4o5Ma9EAffbs2ZKkAQMG6IYbbkjL5ezjxo3TuHHjmn2/V69ertdPPfWURo0apUMOOaTF6wYAAAAAoLWk5Tvo06ZNkyR9/vnnWr9+vSTp8MMPV48ePdIRvllbtmzRs88+q4cffjij6wEAAAAAINM83r+2abt27dIVV1yh3r17a+TIkRo5cqT69OmjK6+8Urt27UrHKpr08MMPq6CgoMlL4QEAAAAAaEvSMkCfMmWKli9frn/961/avn27tm/frqeeekrLly/XT37yk3SsokkPPfSQLrnkEuXl5e11vlAopB07drgesaiXHysBAAAA0OGZGX6gw0vLAP2JJ57Qgw8+qHHjxqlr167q2rWrzjzzTD3wwAN6/PHH07GKRl588UWtX79eV1111T7nraioUGFhoevx2X8XZyQvAAAAAABSkbZL3Hv27NloenFxccYucX/wwQc1bNgwDRkyZJ/zlpeXq7q62vXoc/TpGckLAAAAQPtkmJl9AGkZoA8fPlzTpk1TXV1dfNru3bt1++23a/jw4Z5i1dTUqLKyUpWVlZKkjRs3qrKyUps2bYrPs2PHDv3zn/9M6uy5JAWDwfiZfefh86fl/ngAAAAAAKRFWkap06dP1xlnnKF+/frFz2i/8cYbysvL0/PPP+8p1po1azRq1Kj466lTp0qSJk6cqDlz5kiSHnvsMZmmqYsuuigd6QMAAADAvnGWGxmWlgH6Mccco/fee0+PPvqo3nnnHUnSRRddpEsuuUT5+fmeYpWUlMg0997yr7nmGl1zzTUp5wsAAAAAQLZp8QA9HA7riCOO0DPPPKOrr746HTkBAAAAQPbhDDoyrMXfQc/JyXF99xwAAAAAAHiXlpvElZaW6u6771Ykwm+LAwAAAGifuIs7Mi0t30F/9dVXtXjxYv3nP//RMccco86dO7vef/LJJ9OxmswwDUmSrwXXqxixhGVTDRVz4iXEt/fWdOyzjXJNVsJiRtT5j/d4RsywYzrLGntdV7I5OXJ80abf8MIul2HsY75mxNtTM0VMSWJde41pL27ErAZmOsunITdfuIWtM2LlFHOSaUE4I15O+9lwT086Tnz5hG2pxvtosjn5nAXt135fagVtav2m2cIN6bFMe+Ozy+Xk6fNa+fHZ7TLFEqenwP4Dcrx9pBjLKVPM6cdSbF9NcurNYxeWuO4crwH2Fts+ZqTcvuJ9oFO2xvtTquIppZibaW/DeFfoNLcUwhkJ5yfCpj+lnJz6TsdhLH4YSvyIkuKmdJpVOppXvM0mtgOPucX7wvr65t9LUbyvcD6DpXjMjaXSoBKF3f1XJNay82tN9VdGpGU7ZaylxyAAcWkZoB9wwAE677zz0hEKAAAAALITZ7mRYWkZoM+ePTsdYSRJK1as0G9/+1utXbtWmzdv1vz58zVhwoT4+zU1Nbr55pu1YMECffHFFxo4cKCuv/56XXvttWnLAQAAAAAScRk6Mq1F18jEYjHdfffdGjFihE444QTdfPPN2r17d4sSqq2t1ZAhQzRjxowm3586daoWLlyoRx55RG+//bYmT56ssrIyPf300y1aLwAAAAAAralFA/Q777xTP/vZz9SlSxf17dtX9913n0pLS1uU0Lhx4/SrX/1K3/nOd5p8/6WXXtLEiRNVUlKiAQMG6JprrtGQIUP0yiuvtGi9AAAAALBXZoYf6PBaNED/61//qj//+c96/vnntWDBAv3rX//So48+qlgsjXcbSnDyySfr6aef1qeffirTNLV06VK9++67GjNmTMbWCQAAAADZaMaMGRowYIDy8vJ00kknJX3i8rHHHpNhGK6vE6P1tWiAvmnTJp155pnx16NHj5ZhGPrss89anFhz/vjHP+qoo45Sv379lJubqzPOOEMzZszQyJEjM7ZOAAAAAMi2M+jz5s3T1KlTNW3aNL322msaMmSIxo4dq61bt+51uQ8//FA33HCDvvnNb3pfKTKqRQP0SCSivLw817ScnByFw+EWJbU3f/zjH7V69Wo9/fTTWrt2re655x6VlpbqhRdeaHaZUCikHTt2uB6xKL/ZDgAAAKDtuvfee3X11Vdr0qRJOuqoozRr1ix16tRJDz30ULPLRKNRXXLJJbr99tt1yCGH7MdskYwW3cXdNE1dfvnlCgaD8Wl1dXW69tprXb+Fnq7fQd+9e7d+9rOfaf78+TrrrLMkSccee6wqKyv1u9/9TqNHj25yuYqKCt1+++2uaX2PGaN+x45NS14AAAAA2r9M38U9FAopFAq5pgWDQdd4y1FfX6+1a9eqvLw8Ps3n82n06NFatWpVs+v45S9/qeLiYl155ZV68cUX05c80qJFZ9AnTpyo4uJiFRYWxh/f//731adPH9e0dAmHwwqHw/L53Gn7/f69fu+9vLxc1dXVrkefo09PW14AAAAA0FIVFRWucVRhYaEqKiqanHfbtm2KRqPq2bOna3rPnj1VVVXV5DIrV67Ugw8+qAceeCDtuSM9WnQGPZ2/f+6oqanRhg0b4q83btyoyspKFRUVqX///jr11FN14403Kj8/XwcffLCWL1+uv/71r7r33nubjdnUX518/rT8BDwAAACAjiLDZ9DLy8s1depU17Smzp6nYufOnbr00kv1wAMPqHv37mmJifTLulHqmjVrNGrUqPhrp4FOnDhRc+bM0WOPPaby8nJdcskl+vLLL3XwwQfrzjvv1LXXXttaKQMAAABAizV3OXtTunfvLr/fry1btrimb9myRb169Wo0//vvv68PP/xQ48ePj09zrkIOBAJav369Dj300BZkj3TIugF6SUmJTLP5P0316tUrI2fuAQAAAGCvsui3ynNzczVs2DAtXrw4/lNpsVhMixcvVllZWaP5jzjiCL311luuab/4xS+0c+dO3XfffTrooIP2R9rYh6wboAMAAAAA9m3q1KmaOHGijj/+eJ144omaPn26amtrNWnSJEnSZZddpr59+6qiokJ5eXkaPHiwa/kDDjhAkhpNR+thgA4AAAAAScj0Xdy9uuCCC/T555/r1ltvVVVVlY477jgtXLgwfuO4TZs2NbrBNrIbA3QAAAAAaKPKysqavKRdkpYtW7bXZefMmZP+hNAiHXaAHs2xnp2vu9fW56YcK9zFL0mKObVppBbHtJeP2veF8Ns/gRgJW/E9/e2rmb/uRTr5XOtKOpyVgnbUB105psK06ycatP4Ts2PH6y3J+ovkWzOadsU49b8jlOdaT1IS1x2z/hMNW8EDHv/wGIoEXLlFI3Ycf3NLNM/0WxvTsHNK+ccR7cXDhXnu+NEUG6wk0/51w3AXu13t5ecO9yZclC9J2hWx9kOv7dOVk10/Tr2FI37X9GT/9B3LserF2ZYNxdt/klXnrHt7fb5ruVA4tYJG7e4q3HlPY4h5avBNsOvLdNqZHS/ZsGZkTy5hv93Q7WVr6z12GPH90H529vEcb2Fcuh0oqUHfH28P3sJE8t07oNPvOPXkKZzTF0btmGG7T8xxv78vZkK/9WnoACucmdD2PXDaf9iw20Gshe3L6VPzWtDfxPdte0ILU4rHdXKz23AqYcOdnaWsGNXRTinl4uzT4U52PDs3px/zwqmneB9vuKcn3b6c51zrf07bSK3RW8LO8dvef+qcvtDjISRm94WxogIrXoNqisb3oyRPd9qzOe0sYrfV3eHUPh/G67+lfbOkSI+ukvYce8L2B6d46CRXEXPaeIM+welvwoVWOWNJNrV4+ex116RYT21Slp1BR/vD9Q4AAAAAAGSBrBugr1ixQuPHj1efPn1kGIYWLFjgen/Lli26/PLL1adPH3Xq1ElnnHGG3nvvvdZJFgAAAECHYZiZfQBZN0Cvra3VkCFDNGPGjEbvmaapCRMm6IMPPtBTTz2l119/XQcffLBGjx6t2traVsgWAAAAAID0yLrvoI8bN07jxo1r8r333ntPq1ev1rp163T00UdLkmbOnKlevXrp73//u6666qr9mSoAAACAjoSz3MiwrDuDvjehkHXXtLy8PTe58vl8CgaDWrlyZWulBQAAAABAi7WpAfoRRxyh/v37q7y8XF999ZXq6+t1991365NPPtHmzZtbOz0AAAAA7ZmZ4Qc6vDY1QM/JydGTTz6pd999V0VFRerUqZOWLl2qcePGyedrviihUEg7duxwPWLRyH7MHAAAAACAvWtTA3RJGjZsmCorK7V9+3Zt3rxZCxcu1BdffKFDDjmk2WUqKipUWFjoemx+c/F+zBoAAABAW2dk+AG0uQG6o7CwUD169NB7772nNWvW6Jxzzml23vLyclVXV7sevY89fT9mCwAAAADA3mXdXdxramq0YcOG+OuNGzeqsrJSRUVF6t+/v/75z3+qR48e6t+/v9566y39+Mc/1oQJEzRmzJhmYwaDQQWDQdc0nz/rig4AAAAgm/E9cWRY1o1S16xZo1GjRsVfT506VZI0ceJEzZkzR5s3b9bUqVO1ZcsW9e7dW5dddpluueWW1koXAAAAQAdhMEBHhmXdAL2kpESm2XzLv/7663X99dfvx4wAAAAAAMi8rBugAwAAAEBW4gw6MqzN3iQOAAAAAID2hDPoAAAAAJAMzqAjwzrsAN0XsZ/9sRbHMhN/tLCFP2JoJlzXYKQSL2GZ+A0tWtipBHxWfZl+J14KyRkx97Ip1peRsOmcMvrsQqZyE494zBY2C58vfb13vK7jExKek62/TLQJO2Z8U6ba9u02n+cPu+K1iF1vvhbezSXXH7X+0yCnVPMLOA3Mn5720bBoTjlTbbrxbei0XbPxOpJZXpIMe6E97SLF8joxDa+NvQmRxttRkvf279SLz13GlqTm1I8Ra1mf6CwXbwuxlu9Iicej1APZTy2pLyeGnVO8v06xvzXs/dDZlvHPA2nYPbv461JbMKGMjpT2ocS6TtyNkgzprDqaeCwy3H1FS6TaR8TbQriJ/TvV5p9wbIzvn15TtJuTz9egV27hvu3I8UVTDNSExH0zSYn10dJjLYA9OuwAHQAAAAC84G8RyLSs+w56RUWFTjjhBBUUFKi4uFgTJkzQ+vXrXfPU1dWptLRU3bp1U5cuXXTeeedpy5YtrZQxAAAAAAAtl3UD9OXLl6u0tFSrV6/WokWLFA6HNWbMGNXW1sbnmTJliv71r3/pn//8p5YvX67PPvtM5557bitmDQAAAKDdMzP8QIeXdZe4L1y40PV6zpw5Ki4u1tq1azVy5EhVV1frwQcf1Ny5c3XaaadJkmbPnq0jjzxSq1ev1je+8Y3WSBsAAAAAgBbJujPoiaqrqyVJRUVFkqS1a9cqHA5r9OjR8XmOOOII9e/fX6tWrWqVHAEAAAC0f4aZ2QeQ1QP0WCymyZMna8SIERo8eLAkqaqqSrm5uTrggANc8/bs2VNVVVWtkCUAAAAAAC2XdZe4N1RaWqp169Zp5cqVLYoTCoUUCoVc02LRiHz+rC4+AAAAgGzCWW5kWNaeQS8rK9MzzzyjpUuXql+/fvHpvXr1Un19vbZv3+6af8uWLerVq1eTsSoqKlRYWOh6fLZucSbTBwAAAADAk6wboJumqbKyMs2fP19LlizRwIEDXe8PGzZMOTk5Wrx4zwB7/fr12rRpk4YPH95kzPLyclVXV7sefQafntFyAAAAAGhf+A46Mi3rrvEuLS3V3Llz9dRTT6mgoCD+vfLCwkLl5+ersLBQV155paZOnaqioiJ17dpVP/rRjzR8+PBm7+AeDAYVDAZd07i8HQAAAACQTbJulDpz5kxJUklJiWv67Nmzdfnll0uSfv/738vn8+m8885TKBTS2LFj9ec//3k/ZwoAAACgQ+EsNzIs6wboprnvVp+Xl6cZM2ZoxowZ+yEjAAAAAAAyL+sG6AAAAACQlTiDjgzLupvEAQAAAADQEXEGPQ3SdsdFM+E57StoOV86/mxoGi2PkWlpTtFIwzZMV7UZSXyNxHvMFgaIpTFec/tRitKx7RppaUh7+WzdldKell3QlrWLhIVTjNUoB6ew6WwmaYrlS0PbTXfzz6LDWfuXqW2XhrjpagfxONEmAnpdR7qP+04a6eio01X3TaViT0t1m3TEfbojlhn7F2fQAQAAAADIAlk3QK+oqNAJJ5yggoICFRcXa8KECVq/fr1rnvvvv18lJSXq2rWrDMPQ9u3bWydZAAAAAB2HmeEHOrysG6AvX75cpaWlWr16tRYtWqRwOKwxY8aotrY2Ps+uXbt0xhln6Gc/+1krZgoAAAAAQPpk3XfQFy5c6Ho9Z84cFRcXa+3atRo5cqQkafLkyZKkZcuW7efsAAAAAHRUmbiXD9BQ1g3QE1VXV0uSioqKWjkTAAAAAB0a43NkWNZd4t5QLBbT5MmTNWLECA0ePLi10wEAAAAAIGOy+gx6aWmp1q1bp5UrV7YoTigUUigUck2LRSPy+bO6+AAAAACyCD+zhkzL2jPoZWVleuaZZ7R06VL169evRbEqKipUWFjoeny2bnGaMgUAAAAAoOWyboBumqbKyso0f/58LVmyRAMHDmxxzPLyclVXV7sefQafnoZsAQAAAHQY/MwaMizrrvEuLS3V3Llz9dRTT6mgoEBVVVWSpMLCQuXn50uSqqqqVFVVpQ0bNkiS3nrrLRUUFKh///5N3kwuGAwqGAy6pnF5OwAAAAAgm2TdGfSZM2equrpaJSUl6t27d/wxb968+DyzZs3S0KFDdfXVV0uSRo4cqaFDh+rpp59urbQBAAAAtHOGmdkHkHWnkc0kflvwtttu02233Zb5ZAAAAAAA2E+yboAOAAAAAFmJs9zIsKy7xB0AAAAAgI6IM+gAAAAAkAS+J45M67ADdNO+diAWbflFBEbMfrZ3WDOWYhx7eV/EnpDQAZiGh2DNLJtqp+IsF7Erzhe2XkfzvAc0EusnXR2dHcfvs/7jqb6cEM4yTn2lWGHOrRTS0Ykb0cQJCc/JSqj3xLKmxCmnnWMSt5Boki9iJVcXzXHFbQkj0pKC7ckhGvO5XkuSzy5vNNUeNGbl5rMbiNcuI14yY08ZY6k0+IYxo9byzjZ0wiUdtkH9OPuNEzNlMSeJloVpyGmrLd03oxGrXeS0ZF936tquZDNg15vXBpGwbr+dTE6jzsM7X8RJ0n7yuEkNexuaPqdN2G/EUojn9Mv2sjG//TrFdmaGrW1o+K2AZvxA6T1WfJvZy9ZE81LKyYkT/xxgM1M6oCUG38frfYQxwtYCzR6TvDDdz6nW/Z792d1OpZbvR43e9pibEy5+DEliHc0x7GOk08/Up3rwaWr98eN4yzrFiF3OFh6KAKgDD9ABAAAAwBPOoCPDsu476BUVFTrhhBNUUFCg4uJiTZgwQevXr4+//+WXX+pHP/qRDj/8cOXn56t///66/vrrVV1d3YpZAwAAAADQMlk3QF++fLlKS0u1evVqLVq0SOFwWGPGjFFtba0k6bPPPtNnn32m3/3ud1q3bp3mzJmjhQsX6sorr2zlzAEAAAC0Z/wOOjIt6y5xX7hwoev1nDlzVFxcrLVr12rkyJEaPHiwnnjiifj7hx56qO688059//vfVyQSUSCQdUUCAAAAAGCfsn4061y6XlRUtNd5unbtyuAcAAAAQOakekdcIElZd4l7Q7FYTJMnT9aIESM0ePDgJufZtm2b7rjjDl1zzTX7OTsAAAAAANInq085l5aWat26dVq5cmWT7+/YsUNnnXWWjjrqKN12223NxgmFQgqFQq5psWhEPn9WFx8AAABAFuF74si0rD2DXlZWpmeeeUZLly5Vv379Gr2/c+dOnXHGGSooKND8+fOVk5PTbKyKigoVFha6HpvfWpzJ9AEAAAAA8CTrBuimaaqsrEzz58/XkiVLNHDgwEbz7NixQ2PGjFFubq6efvpp5eXl7TVmeXm5qqurXY/ex5yeqSIAAAAAaI/MDD/Q4WXdNd6lpaWaO3eunnrqKRUUFKiqqkqSVFhYqPz8/PjgfNeuXXrkkUe0Y8cO7dixQ5LUo0cP+f3+RjGDwaCCwaBrGpe3AwAAAACySdaNUmfOnClJKikpcU2fPXu2Lr/8cr322mt6+eWXJUmDBg1yzbNx40YNGDBgf6QJAAAAoIMxYq2dAdq7rBugm/v46YKSkpJ9zgMAAAAAaccwBBmWdd9BBwAAAACgI8q6M+gAAAAAkI34mTVkGmfQAQAAAADIAh32DLovYv35y/BZz51z6yVJX6UQK6cmYsWKWneQT/Uva0bUevbX2zmG7dcB624UMS9xjaYnB3ZZsYxI47vd75V9Q4wDcuskSZ+E7OkF3sK4cglZBfJF7WSd8iVZzkCdvQ1Na3mftRlUmLdbkrTNS30lzuu3JvhzrY3i9YYgwYC1XMjepgF7Gzrb2Iv4uu226mwLrw3NmT1QXWdPsX6e0Ah4b7BOLGfZ3Bq7fP7UGn/gy1pJUpeA1bCcbZkKp76c59wcK1jYqTezmZ0jgZNDXk648Xt2+4/mesupKLjL+o/9Oi9oxa5JLsye9dt9RGDXngbl98Uahk6e4c7RaWem4d6/9hkmsGfNwVyr8nbbzaFr0Gpznyebk9OM7Pbk5OZrvCmSZm770gpZ39u9Do+cOjfsavE5myDJemrI2Y98fquA9fb+FC9nkjnG90e7DRwU/ML9fgo3NPLVW0Fzau1t4GwLj/UW79bt5QNOo3DanYd4ift2fB3eq961cqdsfns7pNBNK2eXHaPeilEU8LpXWwK77RxynQZm97FOP+YhVny/SeiXDad/9dq+wvax1k7CdE7xpLAvOcdvX8SKmWeXb4fHben0hfrC+vRmxHrseS++HyUZNKFNOp9ROuXUN7PAPnJz6r9hI0+xrfq3Wb9W5A91kSTl2B1PPHSS28Bn99MNb+Xk1FPOdvv4m+SBzYnhtLMD7H7+o+RSadu4FxYyjDPoAAAAAABkgawboFdUVOiEE05QQUGBiouLNWHCBK1fv941zw9+8AMdeuihys/PV48ePXTOOefonXfeaaWMAQAAAHQEhpnZB5B1A/Tly5ertLRUq1ev1qJFixQOhzVmzBjV1tbG5xk2bJhmz56tt99+W88//7xM09SYMWMUjaZyYRoAAAAAAK0v676DvnDhQtfrOXPmqLi4WGvXrtXIkSMlSddcc038/QEDBuhXv/qVhgwZog8//FCHHnrofs0XAAAAQAfBWW5kWNadQU9UXV0tSSoqKmry/draWs2ePVsDBw7UQQcdtD9TAwAAAAAgbbJ6gB6LxTR58mSNGDFCgwcPdr335z//WV26dFGXLl3073//W4sWLVJubpK3VAYAAAAAj/gOOjItqwfopaWlWrdunR577LFG711yySV6/fXXtXz5ch122GE6//zzVVdX10QUKRQKaceOHa5HLNqC33ECAAAAACDNsnaAXlZWpmeeeUZLly5Vv379Gr1fWFior33taxo5cqQef/xxvfPOO5o/f36TsSoqKlRYWOh6fPbfxZkuAgAAAID2xDQz+0CHl3UDdNM0VVZWpvnz52vJkiUaOHBgUsuYpqlQKNTk++Xl5aqurnY9+hx9erpTBwAAAAAgZVl3F/fS0lLNnTtXTz31lAoKClRVVSXJOmOen5+vDz74QPPmzdOYMWPUo0cPffLJJ7rrrruUn5+vM888s8mYwWBQwWDQNc3nz7qiAwAAAMhifE8cmZZ1Z9Bnzpyp6upqlZSUqHfv3vHHvHnzJEl5eXl68cUXdeaZZ2rQoEG64IILVFBQoJdeeknFxcWtnD0AAAAAAKnJutPI5j6+e9GnTx8999xz+ykbAAAAALBxBh0ZlnVn0AEAAAAA6Iiy7gw6AAAAAGQjvoOOTOMMOgAAAAAAWaDDnkH3h6w/f/l81rPRgj+HGRE7Rgv/ombE7P/Yz4b9ffyAPypJqvcQyzSanu6Lppik/aecTgH7p+yaiZ8Mp558YXtCiikZdlmcshpWNalLTr1renLB3LnFJ6dYztxARNKeMgZyrNfRFGLF7L3U8LWwndnLGYltoCUN164fX3280brWlXSYiLV8Z6d9xfYy875i2cs62z8/x9oItSkWs1veLknSpgZtwSmm16rL99t7sR0rz24nNR5zilezf09STl/mVXw/ccrkN5t+fx98DebrErS243Y7VJ4/4imnxHpN3MdTEauxatlnp2Kk2Mbi/Y6dY8xvPZvx/TP5TiNeLrvATt2nXE47Xq9AtSSpNhbcy8z7CBVLeE6xn4iXxa4ff73peu0pltPuW3D8sQLJFcgfsArp91vPqVS/EbPbhb0jFPp3pZZaxF0vTr0H/Ck0WCdUwrYzUzw142xLZx9qybHDyaHRtkzoj5IV+XybHfew+DSft26ncS52DgFfap2F0dT6U227MavynTLl+FPrJJzjRLRhHk4/FLZjJln38b7BXizgTEjxWNSmxDpAGdGqOIMOAAAAAEAWyLoBekVFhU444QQVFBSouLhYEyZM0Pr165uc1zRNjRs3ToZhaMGCBfs3UQAAAAAdi5nhBzq8rBugL1++XKWlpVq9erUWLVqkcDisMWPGqLa2ttG806dPl5HqNcgAAAAAAGSRrPsO+sKFC12v58yZo+LiYq1du1YjR46MT6+srNQ999yjNWvWqHfv3vs7TQAAAAAdDHdxR6Zl3QA9UXW1dbOboqKi+LRdu3bp4osv1owZM9SrV6/WSg0AAABAR2IyQkdmZd0l7g3FYjFNnjxZI0aM0ODBg+PTp0yZopNPPlnnnHNOK2YHAAAAAED6ZPUZ9NLSUq1bt04rV66MT3v66ae1ZMkSvf7660nHCYVCCoVCrmmxaEQ+f1YXHwAAAEAW4RJ3ZFrWnkEvKyvTM888o6VLl6pfv37x6UuWLNH777+vAw44QIFAQIGANcg+77zzVFJS0mSsiooKFRYWuh6frF+yP4oBAAAAABkzY8YMDRgwQHl5eTrppJP0yiuvNDvvAw88oG9+85s68MADdeCBB2r06NF7nR/7X9YN0E3TVFlZmebPn68lS5Zo4MCBrvdvvvlmvfnmm6qsrIw/JOn3v/+9Zs+e3WTM8vJyVVdXux79Dj8t00UBAAAA0J5k2c+szZs3T1OnTtW0adP02muvaciQIRo7dqy2bt3a5PzLli3TRRddpKVLl2rVqlU66KCDNGbMGH366afeV46MyLprvEtLSzV37lw99dRTKigoUFVVlSSpsLBQ+fn56tWrV5M3huvfv3+jwbwjGAwqGAy6pnF5OwAAAIC27N5779XVV1+tSZMmSZJmzZqlZ599Vg899JBuvvnmRvM/+uijrtd/+ctf9MQTT2jx4sW67LLL9kvO2LusO4M+c+ZMVVdXq6SkRL17944/5s2b19qpAQAAAOjADNPM6MOL+vp6rV27VqNHj45P8/l8Gj16tFatWpVUjF27dikcDrt+MQutK+tOI5sp/HRBKssAAAAAQDZp6ubWTV0NLEnbtm1TNBpVz549XdN79uypd955J6n13XTTTerTp49rkI/WlXVn0AEAAAAgK8Uy+2jq5tYVFRUZKcpdd92lxx57TPPnz1deXl5G1gHvsu4MOgAAAAB0ROXl5Zo6daprWlNnzyWpe/fu8vv92rJli2v6li1bmrxnV0O/+93vdNddd+mFF17Qscce27KkkVacQQcAAACAJGT6O+jBYFBdu3Z1PZoboOfm5mrYsGFavHhxfFosFtPixYs1fPjwZsvwm9/8RnfccYcWLlyo448/Pu11hJbpsGfQYzmG9RyznmvrrYZvGt5jhQusajTtP3fEnz3GcpaL5VjP0VwrQCTq9xzPcL6Wn7BMJOhzrcNrbjXhPNfyqdSXs1DU7mtiTiv0GCsatBdIqLeacG4KSblSk+x2EY1YwX0e/5QVCrvbRCRq/8efQlJ+05WcmZCL120QKbQvYUooq6c48WXtmPlOozfd7ycpWpgvSdoZsXIz7XpKpX3F68detj4SSCknp13WRRt3k857idui2ZzsdX9V39mVS52dm9dyOm093HlPArFUtqOa6CucAB4DxaJ7cnHav/NzMTs89q9mQvuKryOQ8L4H/qIDrRgt6bskRfOtxhmLRq3nxO4mhbhRu+5Mu7ymx34ivsmi1n+qIoWSpK8inV3ve8rJOf7Y3YWzfQ2vsZxtaLcF59gb7888xIslHmPtZ8PdRSbNsHNzyhbzx/Yy994l9oFbwoWu5JLNLZpnxYnk2fVjFzoc8f45IN6PRt11HW9fye6Pzn+cIjpdohMwlfZlH7+dXOoS+oxkYzp9gr+w0BVP8v45J96OfO4ca1P8TOHkEm3QNyrFthrr1tWKZe+XuyMeC2eL2u2oYf3GP3N2tfrpmMeRgVOW7fV5TU5H5k2dOlUTJ07U8ccfrxNPPFHTp09XbW1t/K7ul112mfr27Ru/TP7uu+/Wrbfeqrlz52rAgAHxX8zq0qWLunTp0mrlwB4ddoAOAAAAAJ5k2b2pL7jgAn3++ee69dZbVVVVpeOOO04LFy6M3zhu06ZN8jU40zRz5kzV19fru9/9rivOtGnTdNttt+3P1NGMrLvEvaKiQieccIIKCgpUXFysCRMmaP369a55SkpKZBiG63Httde2UsYAAAAA0DrKysr00UcfKRQK6eWXX9ZJJ50Uf2/ZsmWaM2dO/PWHH34o0zQbPRicZ4+sG6AvX75cpaWlWr16tRYtWqRwOKwxY8aotrbWNd/VV1+tzZs3xx+/+c1vWiljAAAAAB2CaWb2gQ4v6y5xX7hwoev1nDlzVFxcrLVr12rkyJHx6Z06ddrn3QkBAAAAAGgrsu4MeqLq6mpJUlFRkWv6o48+qu7du2vw4MEqLy/Xrl27WiM9AAAAAB2EYWb2AWTdGfSGYrGYJk+erBEjRmjw4MHx6RdffLEOPvhg9enTR2+++aZuuukmrV+/Xk8++WQrZgsAAAAAQOqyeoBeWlqqdevWaeXKla7p11xzTfz/xxxzjHr37q3TTz9d77//vg499NBGcUKhkEKhkGtaLBqRz5/VxQcAAACQTfieODIsay9xLysr0zPPPKOlS5eqX79+e53XuVPhhg0bmny/oqJChYWFrsen/1uc9pwBAAAAAEhV1g3QTdNUWVmZ5s+fryVLlmjgwIH7XKayslKS1Lt37ybfLy8vV3V1tevR96jT05k2AAAAgHbOiGX2AWTdNd6lpaWaO3eunnrqKRUUFKiqqkqSVFhYqPz8fL3//vuaO3euzjzzTHXr1k1vvvmmpkyZopEjR+rYY49tMmYwGFQwGHRN4/J2AAAAAEA2ybpR6syZMyVJJSUlrumzZ8/W5ZdfrtzcXL3wwguaPn26amtrddBBB+m8887TL37xi1bIFgAAAECHwXfQkWFZN0A399HoDzroIC1fvnw/ZQMAAAAANsbnyLCs+w46AAAAAAAdUdadQQcAAACAbGRwiTsyjDPoAAAAAABkgQ57Bt0Xtv765fNl0V/BnFQM+8l+bRjeczQNdyxHPGaKxQ74ok3GTYURixcwteXtVMyExX2pFk4N6qWFzcKf0K6cIqYS1kz8M1qquaVhm2UkVgMB+/dFWrAJ4/XjtIscfzS1OPbynQJh12vX/z3mGd9/7PaRcltN6CuklvdlTjvz+Vr+Gy/x9m/nl3I5m+m/UmGGI+mJZZ85MZrpY1NhJG47r7kl5JBjpNjmG4ZM06HRiDkHNHtCvO16X4GR8B8nhJlq23fi2G3ebz9HmpvfgwJ/XWopNVOUVD4H7Fk4IXaqbdaX0Lm2QGKIFpVPklkXamKixxjN9OuBFH/3qsndMNWqS/iclGpOTiMwmzyepbYN0tVXtCmcQUeGcQYdAAAAAIAskHUD9IqKCp1wwgkqKChQcXGxJkyYoPXr1zeab9WqVTrttNPUuXNnde3aVSNHjtTu3btbIWMAAAAAHUIsww90eFk3QF++fLlKS0u1evVqLVq0SOFwWGPGjFFtbW18nlWrVumMM87QmDFj9Morr+jVV19VWVmZfL6sKw4AAAAAAEnJuu+gL1y40PV6zpw5Ki4u1tq1azVy5EhJ0pQpU3T99dfr5ptvjs93+OGH79c8AQAAAHQs3MUdmZb1p5yrq6slSUVFRZKkrVu36uWXX1ZxcbFOPvlk9ezZU6eeeqpWrlzZmmkCAAAAANAiWT1Aj8Vimjx5skaMGKHBgwdLkj744ANJ0m233aarr75aCxcu1Ne//nWdfvrpeu+991ozXQAAAADtmWlm9oEOL+sucW+otLRU69atc50dj8Wsuyf84Ac/0KRJkyRJQ4cO1eLFi/XQQw+poqKiUZxQKKRQyP0THLFoRD5/VhcfAAAAANCBZO0Z9LKyMj3zzDNaunSp+vXrF5/eu3dvSdJRRx3lmv/II4/Upk2bmoxVUVGhwsJC1+OT9UsylzwAAACA9ocz6MiwrBugm6apsrIyzZ8/X0uWLNHAgQNd7w8YMEB9+vRp9NNr7777rg4++OAmY5aXl6u6utr16Hf4aRkrAwAAAAAAXmXdNd6lpaWaO3eunnrqKRUUFKiqqkqSVFhYqPz8fBmGoRtvvFHTpk3TkCFDdNxxx+nhhx/WO++8o8cff7zJmMFgUMFg0DWNy9sBAAAAeMJvlSPDsm6UOnPmTElSSUmJa/rs2bN1+eWXS5ImT56suro6TZkyRV9++aWGDBmiRYsW6dBDD93P2QIAAAAAkB5ZN0A3k/zuxc033+z6HXQAAAAAyCR+Bx2ZlnXfQQcAAAAAoCPKujPoAAAAAJCVOIOODOMMOgAAAAAAWYAz6AAAAACQDM6gI8M4gw4AAAAAQBbIugF6RUWFTjjhBBUUFKi4uFgTJkzQ+vXr4+9/+OGHMgyjycc///nPVswcAAAAQLtmmpl9oMPLugH68uXLVVpaqtWrV2vRokUKh8MaM2aMamtrJUkHHXSQNm/e7Hrcfvvt6tKli8aNG9fK2QMAAAAAkJqs+w76woULXa/nzJmj4uJirV27ViNHjpTf71evXr1c88yfP1/nn3++unTpsj9TBQAAANCRxFo7AbR3WTdAT1RdXS1JKioqavL9tWvXqrKyUjNmzNifaQEAAADoYAwuQ0eGZd0l7g3FYjFNnjxZI0aM0ODBg5uc58EHH9SRRx6pk08+eT9nBwAAAABA+mT1GfTS0lKtW7dOK1eubPL93bt3a+7cubrlllv2GicUCikUCrmmxaIR+fxZXXwAAAAA2YQz6MiwrD2DXlZWpmeeeUZLly5Vv379mpzn8ccf165du3TZZZftNVZFRYUKCwtdj0/WL8lE2gAAAAAApCTrBuimaaqsrEzz58/XkiVLNHDgwGbnffDBB3X22WerR48ee41ZXl6u6upq16Pf4aelO3UAAAAA7VnMzOwDHV7WXeNdWlqquXPn6qmnnlJBQYGqqqokSYWFhcrPz4/Pt2HDBq1YsULPPffcPmMGg0EFg0HXNC5vBwAAAABkk6w7gz5z5kxVV1erpKREvXv3jj/mzZvnmu+hhx5Sv379NGbMmFbKFAAAAECHYpqZfaDDy7rTyGaSDfPXv/61fv3rX2c4GwAAAAAA9o+sG6ADAAAAQFbiLDcyLOsucQcAAAAAoCPiDDoAAAAAJIMz6MiwDjtAj+UY1nPMaOVMGnBSsfd7035tmt5zNJy+w3l2YiU8exWJ+d1xW8D0OcmkuLydipGwfCzVwqlBvbSwWUQT2lVL+nIjZseIT0gxUDqPJxk6NkVM66KeFmzCeP047SIc9acWx15+VyTH9dr1f495xvcfu32k3FYT+gqp5X1ZvJ3FWn5hVbz92/mlXM6EdtaSdmHkuA93KccyrAXNxD62BczEbec1t4QcwmaKbb5hyDQdGk2fc0CzJ8TbrvcVmAn/cUIYqbZ9J47d5qNpaPuOndG8lJZrrlpS+RywZ+GE2Km2Waee09A2Eo/bLSqfJCMv2MREjzGa6ded45JXTe6GqdZ9woeIVHNyGoHR5PEstW2Qrr4CwB5Zd4l7RUWFTjjhBBUUFKi4uFgTJkzQ+vXrXfNUVVXp0ksvVa9evdS5c2d9/etf1xNPPNFKGQMAAADoEPgddGRY1g3Qly9frtLSUq1evVqLFi1SOBzWmDFjVFtbG5/nsssu0/r16/X000/rrbfe0rnnnqvzzz9fr7/+eitmDgAAAABA6rLuEveFCxe6Xs+ZM0fFxcVau3atRo4cKUl66aWXNHPmTJ144omSpF/84hf6/e9/r7Vr12ro0KH7PWcAAAAAHYAZa+0M0M5l3Rn0RNXV1ZKkoqKi+LSTTz5Z8+bN05dffqlYLKbHHntMdXV1KikpaaUsAQAAAABomaw7g95QLBbT5MmTNWLECA0ePDg+/R//+IcuuOACdevWTYFAQJ06ddL8+fM1aNCgVswWAAAAQLvGXdyRYVk9QC8tLdW6deu0cuVK1/RbbrlF27dv1wsvvKDu3btrwYIFOv/88/Xiiy/qmGOOaRQnFAopFAq5psWiEfn8WV18AAAAAEAHkrUj1LKyMj3zzDNasWKF+vXrF5/+/vvv609/+pPWrVuno48+WpI0ZMgQvfjii5oxY4ZmzZrVKFZFRYVuv/1217S+R39LBx0zNrOFAAAAANB+cKd1ZFjWfQfdNE2VlZVp/vz5WrJkiQYOHOh6f9euXZIkn8+dut/vVyzW9E0bysvLVV1d7Xr0Per0zBQAAAAAAIAUZN0Z9NLSUs2dO1dPPfWUCgoKVFVVJUkqLCxUfn6+jjjiCA0aNEg/+MEP9Lvf/U7dunXTggULtGjRIj3zzDNNxgwGgwoGg65pXN4OAAAAwBO+g44My7oz6DNnzlR1dbVKSkrUu3fv+GPevHmSpJycHD333HPq0aOHxo8fr2OPPVZ//etf9fDDD+vMM89s5ewBAAAAAEhN1p1GNpP4q9TXvvY1PfHEE/shGwAAAACwcQYdGZZ1Z9ABAAAAAOiIsu4MOgAAAABkJc6gI8MYoAMAAABAMpr51SggXTrsAD3cyZAkmTH72f5jmJHCH8WiuVaMmN+e4EstVizHjmffcN4fsp4jISuwz0s8u1xKWCZil9szO872+jzrpd1yUqsvq2OLBJv5hkWSKUaD1oxG1Hods3PaUlPgPSmHk5JdrlgktW+B1IWtjRnNt15HdudKkvJSiGXU2xUSttuqk5K9jY0kG4bTvmK57jKZETtOCtvSjFrL1nf12a/tg5bHYOGuVqPfFcm1c0wpjJWDvR+adr3s3B20X8tbUMMq22c7utoB97zl5Jc0exM6+4+jdlfQU0rx9dtl9NfvWTAa86UUK97Wc6z/mPWpxTEbfF6prrUbvh2ztt5bOfdsq//f3n2HRXFufwA/u0tZem9KswOxE0HQCBIUlCgYry1RrNEYsSUae+9Xb2LXG6MYsXeDCVbU/OyKiiXYKGoULKCIIEX3+/sDdy4rZWdhFRLP53nm0d139+yZZXZmzjsz77z5V1b4RoVeBZZVOysiInr1JjXhb6jhKvGVQeEbdHQLVzyyl5rnIlCupvFmvvIKZ1y53tA0N3oT50aOPRERKd481jgO/e93pJxfJU2/+1eGhW+QFBTOW6656rZJo3jKbeubZU36WrVZ9PKl/M+b5fP1m9zwZiNSnq3k6zfLk+6Lwn+zFG9+68q/7dsb41Iov2+8tXeWX/AmNw2+L2FdqFD93UjKWVu8Nix8Y4Hxm3XE63LuT5DwtQhftkSZnIa5Qbldc3MhIiJpwf/aXilXt2K/NIXq/CjXQ/mvyrerrFwmZLL/zVRBOfc1FQaFwZTzJH0T4O3vUR3lelplGXiTS4GxZvOpXEYVb/ZZlesbYbmQ8VFmxsrrgy3QGWOMMcYYY0wjfIo7e8eq3CBxc+bMoWbNmpGJiQnZ2tpSWFgY3bhxQ+U1iYmJ1KlTJ7KxsSFTU1Pq2rUrPXz4sJIyZowxxhhjjDHGKq7KFejHjh2jIUOG0OnTp+ngwYNUUFBAbdu2pezsbCIiys7OprZt25JEIqHY2Fg6ceIE5efnU4cOHUjB14QwxhhjjDHG3hXg3U7sg1flTnHft2+fyuO1a9eSra0txcXFUatWrejEiROUkpJCFy9eJFPTwmtDf/nlF7KwsKDY2FgKDAysjLQZY4wxxhhjjLEKqXJH0N+WmZlJRESWlpZERJSXl0cSiYT09fWF18jlcpJKpXT8+PFKyZExxhhjjDH2AVDg3U7sg1elC3SFQkEjRoygFi1aUP369YmIqHnz5mRkZERjxoyhnJwcys7OplGjRtHr168pNTW1kjNmjDHGGGOMMcbKp0oX6EOGDKGrV6/S5s2bhedsbGxo27ZtFB0dTcbGxmRmZkbPnj2jpk2bklRa8uzk5eXR8+fPVSbF61fvazYYY4wxxhhj/wCA4p1OjFXZAj0iIoL27t1LR44cIUdHR5W2tm3bUmJiIj169IiePHlCUVFRdP/+fapZs2aJsebMmUNmZmYqU1r84fcxG4wxxhhjjDHGmChVrkAHQBEREbRr1y6KjY2lGjVqlPpaa2trMjc3p9jYWHr06BF17NixxNeNGzeOMjMzVSb7Rp++q1lgjDHGGGOM/RPxNejsHatyo7gPGTKENm7cSHv27CETExNKS0sjIiIzMzMyMDAgIqLIyEhyd3cnGxsbOnXqFA0fPpxGjhxJ9erVKzGmvr6+yqByRERSWZWbdcYYY4wxxhhjH7AqV6WuWLGCiIj8/f1Vno+MjKQ+ffoQEdGNGzdo3LhxlJGRQa6urjRhwgQaOXLke86UMcYYY4wx9kHhe5Wzd6zKFegQsdDPnTuX5s6d+x6yYYwxxhhjjDHG3o8qV6AzxhhjjDHGWJWk4JHW2btV5QaJY4wxxhhjjDHGPkR8BJ0xxhhjjDHGxOBr0Nk79uEW6JJ3EPLN71VRzthQvk8b5zVIUOJDVHC+pW8CVSiOls7bKC0H5XpTosH6s6LfyzulzK0qJvkmpYqmhjfLhFSTP1ppsd76HUkqGBPv4ntXfm/lTU1YJoo8pYXvrsSPEhlW5WUV/MqEz3x7+dLCLFZ4WX3zfom0MBmtLh1vgqOC60jZmy9KUYHs3t5mlHv5EgKVO5USYpJKzHL/TbX4mxG+pzchZeWc4XfyM377+ynnZyiXy3eySqzoflOJQcsXU/n9SLR0FrNUWvFAwjrhzb8ybSVXlFS5EJf37Vy0MqYtH26BzhhjjDHGGGMaAF+Dzt6xKncN+ooVK6hhw4ZkampKpqam5OPjQzExMUJ7bm4uDRkyhKysrMjY2Jg6d+5MDx8+rMSMGWOMMcYYY4yxiqtyBbqjoyPNnTuX4uLi6Pz58xQQEEChoaF07do1IiIaOXIkRUdH07Zt2+jYsWP04MED+vzzzys5a8YYY4wxxtg/HvBuJ/bBq3KnuHfo0EHl8axZs2jFihV0+vRpcnR0pNWrV9PGjRspICCAiIgiIyPJ3d2dTp8+Tc2bN6+MlBljjDHGGGOMsQqrckfQi3r9+jVt3ryZsrOzycfHh+Li4qigoIACAwOF17i5uZGzszOdOnWqEjNljDHGGGOM/eMp8G4n9sGrckfQiYiuXLlCPj4+lJubS8bGxrRr1y7y8PCgS5cukZ6eHpmbm6u83s7OjtLS0ionWcYYY4wxxtiHATxIHHu3qmSBXq9ePbp06RJlZmbS9u3bqXfv3nTs2LFyx8vLy6O8vDyV5xSvX5FUViVnnzHGGGOMMcbYB6hKnuKup6dHtWvXJk9PT5ozZw41atSIFi1aRPb29pSfn0/Pnj1Tef3Dhw/J3t6+1Hhz5swhMzMzlSkt/vA7ngvGGGOMMcbYPwkUeKcTY1WyQH+bQqGgvLw88vT0JF1dXTp8+H/F9Y0bN+ju3bvk4+NT6vvHjRtHmZmZKpN9o0/fR+qMMcYYY4wxxpgoVe4c73HjxlG7du3I2dmZsrKyaOPGjXT06FHav38/mZmZUf/+/enbb78lS0tLMjU1paFDh5KPj0+ZI7jr6+uTvr6+ynN8ejtjjDHGGGNMI3wNOnvHqlyV+ujRIwoPD6fU1FQyMzOjhg0b0v79+6lNmzZERPTjjz+SVCqlzp07U15eHgUFBdHy5csrOWvGGGOMMcYYY6xiqtwp7qtXr6aUlBTKy8ujR48e0aFDh4TinIhILpfTsmXLKCMjg7Kzs2nnzp1lXn/OGGOMMcYYY9pQFa9BX7ZsGbm6upJcLidvb286e/Zsma/ftm0bubm5kVwupwYNGtDvv/9ers9l70aVK9AZY4wxxhhjjKm3ZcsW+vbbb2nKlCl04cIFatSoEQUFBdGjR49KfP3JkyepR48e1L9/f7p48SKFhYVRWFgYXb169T1nzkrDBTpjjDHGGGOMiQHFu5009MMPP9BXX31Fffv2JQ8PD1q5ciUZGhrSmjVrSnz9okWLKDg4mEaPHk3u7u40Y8YMatq0KS1durSi3wzTEi7QGWOMMcYYY6wKyMvLo+fPn6tMeXl5Jb42Pz+f4uLiKDAwUHhOKpVSYGAgnTp1qsT3nDp1SuX1RERBQUGlvp5VAnygcnNzMWXKFOTm5laZWJwT58Q5cU6cE+fEOXFOnBPn9KHkxIqbMmUKiEhlmjJlSomvvX//PogIJ0+eVHl+9OjR8PLyKvE9urq62Lhxo8pzy5Ytg62trVbyZxX3wRbomZmZICJkZmZWmVicE+fEOXFOnBPnxDlxTpwT5/Sh5MSKy83NRWZmpspUWkcIF+j/TFXuNmuMMcYYY4wx9iHS19cnfX19Ua+1trYmmUxGDx8+VHn+4cOHpd7lyt7eXqPXs/ePr0FnjDHGGGOMsb8ZPT098vT0pMOHDwvPKRQKOnz4MPn4+JT4Hh8fH5XXExEdPHiw1Nez94+PoDPGGGOMMcbY39C3335LvXv3po8//pi8vLxo4cKFlJ2dTX379iUiovDwcKpevTrNmTOHiIiGDx9Ofn5+9J///IdCQkJo8+bNdP78efrpp58qczZYER9sga6vr09TpkwRfQrJ+4jFOXFOnBPnxDlxTpwT58Q5cU4fSk6s4rp160aPHz+myZMnU1paGjVu3Jj27dtHdnZ2RER09+5dkkr/d9K0r68vbdy4kSZOnEjjx4+nOnXq0O7du6l+/fqVNQvsLRIAqOwkGGOMMcYYY4yxDx1fg84YY4wxxhhjjFUBXKAzxhhjjDHGGGNVABfojDHGGGOMMcZYFcAFOmOMMcYYY4wxVgVwgc4YY4wxxhhjjFUBH8xt1p48eUJr1qyhU6dOUVpaGhER2dvbk6+vL/Xp04dsbGxExUlNTaUVK1bQ8ePHKTU1laRSKdWsWZPCwsKoT58+JJPJ3uVslGjHjh3Url07MjQ0fO+fLVZycjLdvn2bHBwcKvU2Dunp6XT58mVq1KgRWVpa0pMnT2j16tWUl5dHXbp0IXd390rLraLy8/Np9+7dJS7joaGhpKenV8kZMsZK8+TJE7K2ttZ63GfPntG2bdvo7t275OLiQl26dCEzM7MKxSwoKCBdXd0K5zZt2jQaMmSIVub74cOHlJeXR87OzhWOpW3Z2dkUFxdHrVq1EvX6V69e0bVr11TW4x4eHuX6zpWfXXR/pWnTpiSRSETHePToEV29epU8PT3JzMyMHj58SL/88gspFAoKCQmhBg0aiIoTFxdHnp6eGs8DY4x9cPABOHv2LCwsLFC9enX07t0b33//Pb7//nv07t0bjo6OsLS0xLlz59TGOXfuHMzMzODp6YmWLVtCJpOhV69e6NatG8zNzeHr64vnz5+/hzlSJZFIYGpqiq+++gqnT58ud5zc3Fzk5+cLj2/fvo3x48ejZ8+emDBhApKSkkTFGTx4MLKysgAAOTk56Ny5M6RSKSQSCaRSKVq3bi20l1fr1q2RkpKi0XvOnDkDMzMzSCQSWFhY4Pz586hRowbq1KmDWrVqwcDAAHFxcaLjvX79utTn79y5o1FupUlLS8O0adPUvu7WrVuoWbMm5HI5/Pz80LVrV3Tt2hV+fn6Qy+WoXbs2bt26Jfpzc3JysHr1avTt2xfBwcFo3749IiIicOjQIY3n4eHDhzh8+DCePXsmzNO8efMwZ84cXL58WVSM7du3Izs7W+PPFuPp06f46aefMHHiRKxatUrIszzy8/Nx8+bNCsUoqk+fPrh//77G73v16pXK49OnT+PYsWMqv2+x3vVy/uLFCxw7dkyj95w5cwYLFy7E2LFjMXbsWCxcuBBnzpzR+LMPHz6MadOm4euvv8Y333yDBQsW4ObNmxrHuXTpElavXo3ExEQAwNWrVzF48GAMGjQI+/btExVDKpUiICAAGzZsQG5ursY5KHXq1Anbtm0T8rC2toaNjQ28vb1hZ2cHe3t7/Pnnn6JibdmyBXl5ecLjJUuWwNnZGVKpFFZWVqLWTQCQmZlZbHr27Bl0dXVx5swZ4Tkxnj9/ji+//BLOzs4IDw9HXl4evvnmG2H70qpVK9Gxli1bhk8//RRdunQptm57/PgxatSoISqOOpcuXYJUKlX7utevX2PChAkwNzeHRCJRmczNzTFx4sRSf48lxRo9ejQMDQ0hlUqFbbBEIoGLiwt+/fVXUXGOHDkCIyMjSCQS2Nvb49KlS3B0dESdOnVQr1496OvrY//+/aJiSSQS1KpVC7NmzSrXeq00BQUFOHDgAH7++WccPHiw2PpPU9pej0+dOhWPHz+ucJy0tDSt7VtoW0FBQZXNjbG/ow+iQPf29sbAgQOhUCiKtSkUCgwcOBDNmzdXG6dFixaYOnWq8DgqKgre3t4AgIyMDDRu3BjDhg3TKLfo6GhMmjQJx48fB1C4w9iuXTsEBQXhv//9r6gYEokE06dPR5MmTSCRSPDRRx/hxx9/xJMnTzTKxc/PT9ixO378OPT19dGwYUN069YNTZo0gaGhIU6ePKk2jlQqxcOHDwEA48aNg6OjI2JjY5GdnY3jx4+jVq1aGDt2rKic9uzZU+Ikk8mwdOlS4bEYgYGBGDBgAJ4/f4758+fD0dERAwYMENr79u2LsLAwtXEyMzPRpUsXyOVy2NraYtKkSSo7BGlpaaJ2xsQQu2MXGBiI0NDQEndMMzMzERoairZt24r6zFu3bsHFxQW2trZwcnKCRCJBSEgIvL29IZPJ0KVLFxQUFIiKpa2dO211QgHaK2DmzZuHnJwcAIUF8XfffQc9PT1IpVLo6Oigb9++ogvi+Pj4EiddXV3s2rVLeKzOgwcP0KJFC8hkMrRq1QoZGRkICQkRdszr1q2LBw8eiMrpfS3nYpdxoLCzp2XLlkKR4eXlBS8vL7i4uEAikaBly5bCukddHC8vL+FvJZVK4enpCXt7e8hkMowePVp0/jt27IBMJoOVlRWMjY1x8OBBmJubIzAwEEFBQZDJZNiwYYPaOBKJBMHBwdDT04OFhQUiIiJw8eJF0XkoWVhYICEhAQDQrl07fPHFF0KRnZ+fj/79+4teFxRdl69ZswZyuRyTJ0/Gb7/9hpkzZ8LIyAirVq0SFaekSVlUK/8VIyIiAm5ubli8eDH8/f0RGhqK+vXr4/jx4zh27Bg8PDwwfvx4tXEWLVoEQ0NDDBkyBD179oSenh5mz54ttFfGenz06NGwsbHBypUrkZycjJycHOTk5CA5ORn//e9/YWtri++//17UZ44ZMwbu7u6Ijo7GwYMH0apVK8ybNw8JCQmYNGmS6HVvy5YtMWTIEGRlZWH+/PmoXr06hgwZIrSPGjUKvr6+onKSSCT46quvYGtrCx0dHYSEhGDXrl0aF9QRERGIjo4GANy7dw9ubm6QyWSws7ODTCZDgwYN8Ndff4mKpc31uLY6orTZCQW8n44oTdbjjDH1PogCXS6XCzssJUlISIBcLlcbx8DAQDhCAhT2UOvq6iItLQ0AcODAAVSrVk10XitXroSOjg48PT1hamqKqKgomJiYYMCAARg0aBAMDAywcOFCtXEkEomwE3X+/HkMHjwY5ubm0NfXR5cuXXDgwAFR+ZiamgpHj/z8/DBy5EiV9okTJ6JFixYa5VO/fn1s3LhRpX3Pnj2oW7euqJyK7sCVNondKFhYWAiFV35+PqRSqcpRt7i4OFSvXl1tnGHDhqFu3brYtm0bVq1aBRcXF4SEhAg7wWlpaZBIJKJyKq0wU05btmwRNX8GBga4cuVKqe2XL1+GgYGBqJzatWuHQYMGCR1ac+fORbt27QAAN2/ehKurK6ZMmSIqlrZ27rTVCQVor4ApWrzMnz8fFhYWWLNmDa5du4b169fD1tYW8+bNE5VTWcu5JgVMr1694Ovri19//RXdunWDr68vPvnkE/z111+4c+cOWrRoofL9l0Wby3lZNNmx69y5M3x8fHD9+vVibdevX4evry/+9a9/qY3TrVs3hIWFITMzE7m5uYiIiEB4eDiAwk5SKysrUeteAGjatClmzpwJANi0aRPMzc0xffp0oX3BggVo3Lix2jjK9ebjx4+xYMECeHh4QCqVomnTpli+fLnoHXIDAwPcvn0bAODg4IALFy6otN+4cQNmZmaiYhVdl3t5eeHf//63Svvy5cvRpEkTtXGqV6+OkJAQxMbG4ujRozh69CiOHDkCmUyGyMhI4TkxnJycEBsbCwC4f/8+JBKJUKwBwN69e1GvXj21cTw8PFQ6Tk6cOAEbGxtMmjQJgGYFuoWFRZmTqampqFh2dnZlnnGxb98+2NraisrJwcEBf/zxh/D4r7/+grGxsXB2xvTp0+Hj46M2jqmpqbA8FRQUQEdHR6Xj6ObNmxovTwUFBdi+fTvat28vFNbff/89bty4ISqOnZ2dsL3r2rUrAgMDhSPU6enp+Oyzz0StBwDtrse11RGlrU4o4P11RHGBzph2fRAFuqurK3755ZdS23/55Re4uLiojePi4iIc6QYKj1ZJJBKh9zU5OVlUoa/k4eGBn376CQAQGxsLuVyOZcuWCe2RkZFwd3dXG6foTpTSy5cvsW7dOvj7+0MqlcLV1VVtHCMjI6FwsbOzw6VLl1Tab9++DWNjY1H5PHr0CABgbW2Nq1evqrSnpKSILhaDg4MREhJSbP50dHRw7do1UTGUjIyMkJycLDw2NjZW6XC5c+eOqL+fs7Mzjhw5Ijx+/PgxvLy80LZtW+Tm5mq0wdNWYebg4KCyk/q2X3/9FQ4ODqJyMjQ0VDnNNy8vD7q6ukIxvHv3blHLE6C9nTttdUIB2itgiubUpEmTYme8rF+/Hh999JGonBo1aoSQkBAkJCQgJSUFKSkpSE5Oho6ODg4ePCg8p46DgwNOnToFoHBHVSKRqBwxOXz4MGrWrCkqJ20t59oqXoDC3+zbf6+izp8/L2odZWpqqrJeevHiBXR1dYUiOCoqSlSRB6iuVxQKBXR1dVUu3UhMTBS93nx7PXfy5En069cPJiYmMDQ0RK9evdTG8fb2FrYrTZo0wa5du1TaDxw4AHt7e7VxlDkVXZeXtE0wMTFRGyc9PR1hYWFo3bq1ypHN8qzH9fX1cffuXeGxoaGhSmGXkpICQ0NDtXEMDAxUtgcAcOXKFdjZ2WHs2LEarccNDQ3x3XffYe3atSVO06ZNExXL0NCwzMt+4uPjYWRkJConExOTYgcUdHR0kJqaCgC4du2aqO+p6DY8OzsbUqlUWMcoc7K2thaVU0nL+F9//YXp06ejZs2akEql+OSTT9TGkcvlwiV3jo6OxS5vuXLlSrlyquh6XFsdUdrqhAK01xHVpEmTMic3Nzcu0BnTog9ikLhRo0bRwIEDKS4ujj799FOys7MjosJBZQ4fPkyrVq2iBQsWqI0TFhZGX3/9Nc2fP5/09fVpxowZ5OfnRwYGBkREdOPGDapevbrovJKTkykoKIiIiFq3bk2vX79WGUTG39+fhgwZojZOSYO9yOVy6tWrF/Xq1Ytu375NkZGRauN4e3tTdHQ0ubm5Ua1atSg+Pp4aNWoktF+6dIksLS3FzBpNmjSJDA0NSSqV0oMHD+ijjz4S2tLT08nIyEhUnJiYGPrxxx/p448/puXLl9Nnn30m6n0lcXJyoqSkJHJ1dSUios2bN5ODg4PQnpqaKmqwosePH5OLi4vw2Nramg4dOkRBQUHUvn17+vnnn0XnZGlpSf/+97/p008/LbH92rVr1KFDB7VxBgwYQOHh4TRp0qQSl/GZM2fS0KFDReVkbm5OWVlZwuOcnBx69eqVMMhcw4YNKTU1VVQsPT09ys3NJaLCQewUCoXwmIjo5cuXGg985OnpSZ6envTDDz/Qtm3baM2aNRQcHEzOzs6UnJys9v0NGzak2NhYqlWrFtnb29OdO3eoSZMmQvudO3eE37Q6yt/e3bt3ydfXV6XN19dXVD5ERGfPnqXvv/+eOnfuTOvXr1fJp1q1airLW1mePn0qrIMsLS3J0NBQ5b21a9cW/bfT1nKel5dHgwcPLnUgqTt37tC0adNExdLX16fnz5+X2p6VlUX6+vqi4hRdb0qlUnr9+jW9evWKiAr/dikpKaJyMjExofT0dHJ1daVnz57Rq1evKD09XWhPT08nY2NjtXFKWo/7+PiQj48PLV68mDZv3kxr1qxRG2fSpEkUHh5Ourq6NGzYMBo5ciSlp6eTu7s73bhxg6ZMmUK9evUSNW9ERPv27SMzMzOSy+WUk5Oj0pabmytqsDFLS0vatWsXrVixgry8vGjBggXUo0cP0TkUZWVlRY8fPyYnJyciIgoNDSVzc3Oh/cWLF6KWAWtra7p3756wPSAiql+/PsXGxlJAQAA9ePBAdE6NGzcmJycn6t27d4nt8fHxopZxf39/GjVqFG3YsKHYtujJkyc0ZswY8vf3F5VTgwYNaNOmTTRhwgQiItq6dSsZGxuTvb09EREpFApR31OLFi1o7NixNHbsWFq3bh01bdqUZs6cSVu2bCGJREIzZsygjz/+WFROJS0r1atXp0mTJtGkSZPo8OHDopbxunXr0tmzZ6lGjRpkYmJSbJ2QlZVFCoVCVE5F86roevzy5cvUv39/mjFjBkVFRQnrYolEQl5eXuTh4SEqzqNHj6h27dpEVLj+NzAwoLp16wrt9evXp3v37omKlZycrDJPvr6+FBsbS4GBgVRQUEAjRowQFefPP/+k7t27U40aNUpsT01NpZs3b4qKxRgTobJ7CN6XzZs3w9vbGzo6OsIRSh0dHXh7e2PLli2iYmRlZaFr165CDF9fX5WB0/bv34+tW7eKzsnR0VE4BU3ZS/rbb78J7UePHoWjo6PaOCX1SpfHyZMnYWZmhilTpmDJkiWwtrbGxIkTsWHDBkyePBnm5uaiTvXy8/ODv7+/ML19jeKMGTPg5+enUW4XL16Eh4cHBg4ciOzs7HIdeZk6dSo2bdpUavv48ePx+eefq41Tr149lb+TUlZWFnx8fNCoUSPRPclt27bFjBkzSm2/dOmS6NOI586dCwcHB+Gou/IIvIODg+hT9ACgd+/e8PPzQ0JCApKSkoQxCJSOHj0KJycnUbFCQ0Px2Wef4fjx4xg4cCA+/vhjhISE4MWLF8jOzsa//vUvBAcHq41T9DTEkty6dUv0KX979+6FpaUlIiMjERkZCVdXV/z88884ceIE1qxZAycnJ1HXIEskEsyaNQuLFi2Cg4NDsYHO4uPjYWFhISonpd9//x2Ojo6YPXu2cMRLk+Xc2dlZ5WjSmDFjkJ6eLjy+dOmS6CNL2lrOfX19yzxdXJNTI7/55hu4uLhg586dKqd8Z2ZmYufOnXB1dUVERITaOJ06dULnzp3x4sUL5OfnY8SIEahdu7bQfvr0adFHmXv27Alvb2+sX78eHTp0QFBQEJo3b46EhARcv34dfn5+ok631dZ6HCgcVNHR0bHY2TlyuRwjRowQfc3v22f1KE/lV/r5559FneJe1LVr19CoUSP06NGjXOvx4OBgrFy5stT2yMhIUZfN9OjRAyNGjCix7erVq7CxsRG9XM6aNUtlfJq33b17F3369FEb5+7du6hfvz50dHTQpEkTBAcHIzg4GE2aNIGOjg4aNmyocvZAWQ4dOgR9fX14eXmhVatW0NHRwY8//ii0z58/HwEBAWrj3Lx5E3Xq1IFEIoG7uzv++usvdOzYETo6OtDR0YGNjY3owVW1tYxHRkbC0dERR44cwbp16+Du7o5Dhw7h/v37iI2NRYMGDVTGl1GXk7bX48uXL0e1atWEy/s0Xc6rVaum8p326NFD5Xu7evWq6JycnJxULnVQunbtGuzs7BAeHi5qOff09MTy5ctLbb948SIfQWdMiz6YAl0pPz8fDx48wIMHD8o1ojFQePp4RUchB4AhQ4agTp06mDlzJry8vNC7d2+4ubkhJiYG+/btQ4MGDdCvXz+1cVJSUkocAK88Tp48iebNmxfbMatevbroazJLo8wxMTER9+7d0/j9OTk5GDRoEOrUqQOZTKbxjp062dnZokZPjoiIKHWH+/nz5/D29ha9odq5cyeioqJKbc/IyMDatWtFxVJKSkrCyZMncfLkSdEj7xf18OFDlWXAxcVFZWdh27ZtWLx4sahY2tq502bxAmingHFxcYGrq6swFd35BYCFCxeKGnzybWlpaWjXrh0++eQTjXfsOnbsWObvdOnSpaJ2ygFg6NChWlnOtVW8AIV3mvj666+FQZzkcjnkcjmkUin09PQwePBgUb/hxMRE1KpVCzo6OtDV1YW5uTkOHjwotEdGRooeyDItLQ1t2rSBsbExgoKC8OzZM0RERAgdZXXq1BEuqSjL2rVrKzR6+9tevXqFs2fPYvPmzdi4cSOOHDmi9buMREdHix6lvqi8vDyMHDkSjRs31ngdlZ6ejqdPn5ba/vvvv6tcmlGa+Ph4rFmzptT2K1eulLncviuvX7/G77//jsmTJ2PgwIEYOHAgJk+ejJiYGNEjuCtdunQJ48ePx3fffafRZUAleXu8j0OHDiE6OlqjcUCOHj0qeoBRdf7zn//A0NAQBgYGwvpAOYWFhYneR3tX6/GKdERpqxMK0F5H1LBhwzB8+PBS22/fvg1/f39ROTHG1JMAQGUfxf9QZWdn08iRI+nUqVPk6+tLS5YsocWLF9OECROooKCA/Pz8aMuWLWRra/vec3v8+DElJSWRQqEgBwcHldMAy0tPT4/i4+MrfK/x6Ohoio2NpXHjxlXKd/P06dNip+0XlZWVRRcuXCA/P7/3mldqaiqtWLGCjh8/rnLP27CwMOrTpw/JZDKN4t26dYvy8vLI3d1d4/e+LT09naysrITHhw8fppcvX5KPj4/K86W5c+cOOTs7a3TvXnVev35NFy5cUFnOPT09ycTERCvxT58+Tfr6+iqnq2ti8eLFdOTIEVqyZAk5OjpqJaezZ8+SoaEh1a9fX+1rq+pyTkT0/PlziouLU7lPtKenJ5mamoqOkZOTQ8ePH6f8/Hxq3ry51u9BnpSURDk5OeTm5kY6Oh/E1WSMvVfPnj2jAwcOUHJysrAOb9GiBdWpU0drn1GR9Xh+fj6NHTuWjhw5Qjt37iz19PC3ZWRkkFQqVbl0o6iYmBgyMDAQdbnD5cuXKS4ujvr27Vti+9WrV2nHjh00ZcoUUbkxxt4PLtCroNzcXCooKNCoUHj58iXFxcWRpaVlseuccnNzaevWrRQeHq42TkJCAp0+fZp8fX2pXr16dP36dVq0aBHl5eVRz549KSAgQG2Mb7/9tsTnFy1aRD179hQKsh9++EHEnKnKzs6mrVu30u3bt8nBwYF69OghqsAjIrpw4QJZWFgIG8moqChauXIl3b17l1xcXCgiIoK6d++uNs7QoUOpa9eu9Mknn2ic/7ty/vx5CgwMpNq1a5OBgQGdOnWKvvjiC8rPz6f9+/eTh4cH7du3T/Qype1inzEmztmzZ+nUqVMqnQ8+Pj7k5eWllfhPnz6l6OhoUdsDJYVCQVKptNjzAOjevXvk7OysNgYASklJIScnJ9LR0aH8/HzatWsX5eXlUfv27SvcQRIQEECRkZGix2soSXJysrBtEdOJpZSXl0dSqVQYTyMxMZHWrFkjbFv69+8vqjjbsWMHtWvXjgwNDcs9D+8iVnx8PMXFxZG/vz/VrFmTrl27RsuWLSOFQkGdOnUSxtIRKzY2tti2pWPHjlotrBlj7G+tMg/fs7LdvXsXffv2Vfu6GzduCPcBVt4js+i9jsWO0hkTEwM9PT1YWlpCLpcjJiYGNjY2CAwMREBAAGQyGQ4fPqw2jkQiQePGjVWuQ/f394dEIkGzZs3g7++P1q1bq40DAO7u7sI1tHfv3oWLiwvMzMzQrFkzWFpawtbWVvQpkg0bNhROY121ahUMDAwwbNgwrFixAiNGjICxsTFWr14tav6Up67OnTtXGBW3vJYsWYJevXoJ18crr6mrV68exo0bJ+qUwBYtWqicjhkVFQVvb28AhafJN27cGMOGDROVz7lz52BmZgZPT0+0bNkSMpkMvXr1Qrdu3WBubg5fX1+NTpXNy8vDli1bMGLECHTv3h3du3fHiBEjsHXrVuGWXRWVlpaGadOmafSee/fulXgaZH5+frHrEEvz5MkTxMbGCsvo48ePMXfuXEybNk3UvdTVqVGjhsqI+ppSKBSIjY3FTz/9hOjoaI0u67l3755w6yIA+OOPP/DFF1+gZcuW+PLLL3Hy5ElRcRYsWCBqBHqxoqOjMWnSJOGOGocPH0a7du0QFBRUbATm9xEHKLz8ZvXq1ejbty+Cg4PRvn17REREFLvncFm0dY93dTS55j8zMxNdunSBXC6Hra0tJk2apHL5h9hty/Xr1+Hi4gKpVIratWsjKSkJnp6eMDIygqGhIaytrUUv53v27ClxkslkWLp0qfBYncGDBwu//5ycHHTu3FnlzhmtW7cWfZq0n58ftm3bBgA4fvw49PX10bBhQ2H8DkNDQ1G/F4lEAlNTU3z11Vc4ffq0qM9+17F27NgBmUwGKysrGBsb4+DBgzA3N0dgYCCCgoIgk8lURgkvy8OHD+Hl5SXcY1wqlcLT0xP29vaQyWSixv4o6syZM1i4cCHGjh2LsWPHYuHChcVGdK+ojIyMMu8CVJLSLkV4/fo17ty5IyqGQqFAUlKSsP3Py8vD5s2b8csvv6isl8urdevWFV4vJyUl4cCBA2Xe4pUxVj5coFdhYnekwsLCEBISgsePH+PWrVsICQlBjRo1hA2B2J0oHx8fTJgwAUDh/XwtLCxUBt4aO3Ys2rRpozbOnDlzUKNGjWLFfHkGBCp67fGXX34JX19fPHv2DEDhYFWBgYHo0aOHqFgGBgbCBqlJkybCrYiUNmzYAA8PD1E5HTp0CMOHD4e1tTV0dXXRsWNHREdHa3yN4IwZM2BiYoLOnTvD3t4ec+fOhZWVFWbOnInZs2fDxsYGkydPFjVvb99SR1dXF2lpaQAKb61UrVo1UTlps9i/desWatasCblcDj8/P3Tt2hVdu3aFn58f5HI5ateujVu3bomKVRZNio4HDx6gWbNmkEqlQudD0R1xsb+XM2fOwMzMDBKJBBYWFjh//jxq1KiBOnXqoFatWjAwMBA9eNKiRYtKnGQyGcaNGyc8Vqddu3bC7yM9PR3e3t6QSCTCdYZubm7CbbPU8fLyEm7ts3v3bkilUnTs2BFjxoxBp06doKurW+at/ZQkEglkMhkCAwOxefPmCnXKrFy5Ejo6OvD09ISpqSmioqJgYmKCAQMGYNCgQTAwMBA1Voa24gCFy7iLiwtsbW3h5OQEiUSCkJAQeHt7QyaToUuXLqI62bR1j/fMzMwyp//7v/8T/VsZNmwY6tati23btmHVqlVwcXFBSEiI8DdMS0sTNYhlaGgoOnbsiMuXL2PEiBFwd3dHaGgo8vPzkZubiw4dOqBnz56icirr1pRFb1GpTtGBJ8eNGwdHR0fExsYiOzsbx48fR61atUSPQ2Bqaip0MPj5+WHkyJEq7RMnTkSLFi1Ezdv06dPRpEkTSCQSfPTRR/jxxx81us5b27GaNm0qDA64adMmmJubY/r06UL7ggUL0LhxY1GxunXrhrCwMGRmZiI3NxcREREIDw8HUNhBZmVlJep39746swDu0FLSZocWY0w9LtArUWkrTuX0448/ilqZ29raqtw7VaFQ4Ouvv4azszMSExNFbxRMTU2FYkk5gnTRew4r7xErxtmzZ1G3bl189913wlG7ihboNWvWLDbQzYkTJ0SPKG5lZYXz588DKPzOSrqnr5j7sxfNKT8/H1u2bBGOJFSrVg3jx48XXXTWqlULO3bsAFC4IyCTybB+/XqhfefOnSqjS5fGxcVFOBIIFBahEokEOTk5AIDk5GRR93gHtFvsBwYGIjQ0VGXEbaXMzEyEhoaibdu2auPEx8eXOW3ZskX0TlR4eDi8vb1x7tw5HDx4EJ6envj444+RkZEBQHzRERgYiAEDBuD58+eYP38+HB0dVUYO7tu3L8LCwkTlJJFI4OjoqDJYkaurqzBAo6urK2rUqCEqjnLZHDx4MDw8PIQzTO7duwdPT098/fXXonIyMjIS3uvt7Y25c+eqtC9ZskTUCN4SiQSRkZEIDQ2Frq4urKysMHz48HIddfHw8BA61mJjYyGXy7Fs2TKhPTIyEu7u7u8tDlDYKTJo0CBhEMy5c+eiXbt2AAoHSXR1dcWUKVPUxtHWPd6L3sWhpElsAQsU3hWg6IBrjx8/hpeXF9q2bYvc3FzR2xYbGxtcvHgRQOE95yUSCf7v//5PaD9x4gScnZ1F5RQcHIyQkJBiBZim25eiv5X69esLI24r7dmzB3Xr1hUVy8jICAkJCQAAOzu7ErctYv92ypzOnz+PwYMHw9zcHPr6+ujSpYtGA71pK5aRkZFwv3iFQgFdXV2V/Y3ExERR8wYU7mMo76kOFC4Lurq6wvYhKipK1P29tdWZBXCHVmV0aDHG1OMCvRJpa8VpYmJS4um0Q4YMEW7lJrZALzrasLGxsUqhlpKSIrrIAwqPcIeHh6Nhw4a4cuUKdHV1y1WgK4/4VatWrdhOvSY59ezZE/379wcAdOnSBRMnTlRpnz17Nho0aCAqp5J65+/cuYMpU6YIPd9iGBgYqJzypqurq7IDk5KSAkNDQ7Vxhg8fjvr16yMmJgaxsbFo3bq1yoiq+/btQ61atUTlpO1iv6xC7PLly6I7RUr7rWhadFSrVk3lNEjlDk/jxo2Rnp4uuuiwsLAQfnf5+fmQSqUqcePi4lC9enVROQ0aNAiNGzcu9juuSNFRr169YkdGDh06JKrQBwAzMzPEx8cDKOzQUv5f6fbt26KWzaI5PXz4EPPmzYObmxukUimaNWuGn376SfQlEyX9XoouX8nJyaJy0lYcADA0NFQ5mpWXlwddXV3hSOXu3bvh6uqqNo6VlRWOHj1aavuRI0dgZWWlNo6pqSnmzZuHo0ePljitWrVKo/XT25cQPX/+HD4+PggICEBSUpKoWG9/38bGxirbmrt370JfX19UTgDwww8/wMnJSeUMjvL8VpTbFmtra5X1LlC47hWzbgKAgIAA/Pvf/wZQeFvBt0+J3r59u6gOiJK2LS9fvsS6devg7+8PqVQqalnSZix7e3uhYzsjIwMSiUSl0+bs2bOib0loY2Oj8jfKycmBVCoVLhFKTEwUtRxoqzML4A4tsbTZocUYU48L9EpUrVo17N69u9R2sfeVbNasGdatW1di25AhQ2Bubi4qTsOGDRETEyM8vnLlisqpmX/88YfonfuiNm3aBDs7O0il0nIV6A0aNECTJk1gbGyM7du3q7QfO3ZMdBF0//59uLq6olWrVvj2229hYGCAli1b4quvvkKrVq2gp6dX4n2fS8qprNPnFAqF6KMTNWrUEL7zmzdvQiqVYuvWrUL7b7/9JmonKisrC127doWOjg4kEgl8fX1Vdqz379+vErcs2iz2HRwcyjwN+tdff4WDg4PaOFZWVli9ejVSUlJKnH777TfRO1FGRkbFTg8sKChAWFgYGjZsiMuXL4uKVfTIElC8Q+vOnTsadWjt3LkTTk5OWLJkifBcRYoOW1vbEosOsYVQx44dhSMiQUFBxU6xX7VqFerUqSMqp5J+L3/88Qd69+4NIyMjGBkZicpJ2eEIFP6eJRKJym/26NGjcHR0fG9xgOL3LH769CkkEonQ6ZCUlCTqO9fWPd79/f0xb968UtsvXbok6igeUNjJU9I6MSsrCz4+PmjUqJGo30qtWrVUCozly5erdMrExcWJLvKULl68CA8PDwwcOBDZ2dnl+q0MGjQII0eOhK2tbbF1dlxcHKytrUXFOnnyJMzMzDBlyhQsWbIE1tbWmDhxIjZs2IDJkyfD3Ny8zL+JUtGjlCW5deuWymVn7yNWz5494e3tjfXr16NDhw4ICgpC8+bNkZCQgOvXr8PPz0/00epOnTqhc+fOePHiBfLz8zFixAiVM8ROnz4tajnQVmcWwB1aYmmzQ4sxph4X6JWoQ4cOmDRpUqntYnekZs+eLZxSWZLBgweLirNixQrs3bu31PZx48YJR6A1de/ePezevRsvXrzQ6H1Tp05Vmd6+5+6oUaPQvXt30fGePn2KMWPGwMPDA3K5HHp6enBxccEXX3yBc+fOiYrh6uparmsCSzJx4kTY2NhgwIABqFGjBsaOHQtnZ2esWLECK1euhJOTU7HrGcvy8uXLCl8Hps1if9KkSbCwsMAPP/yA+Ph4pKWlIS0tDfHx8fjhhx9gaWkp6vTftm3bYsaMGaW2a1J0NGjQoFhHD/C/It3Z2VnUTpSbm5vKOAt79+4VzjIACnc2xRZ5Sn/99RcCAgIQHByM1NTUcu1EtW/fHp06dYKFhUWxzpHTp0+Lvkzlzz//hJWVFcLDwzFjxgwYGxujZ8+emDVrFsLDw6Gvr4/IyEi1cdQVCpmZmcXGgyjNkCFDUKdOHcycORNeXl7o3bs33NzcEBMTg3379qFBgwbo16/fe4sDAL1794afnx8SEhKQlJQkDAymdPToUVGX4WjrHu8//fRTmeMVpKWlib6/99ChQ0stvp4/fw5vb29Rv5VBgwZh1apVpbbPmTMH7du3F5VTUTk5ORg0aBDq1KkDmUym0W/Fz89PZRDTt/ObMWMG/Pz8RMc7efIkmjdvXuwMn+rVq4sez0Bd568mtBUrLS0Nbdq0gbGxMYKCgvDs2TNERESoDJZatHgsS2JiImrVqgUdHR3o6urC3NxcGLgVKLy0RMxp0trqzAK4Q0ssbXZoMcbU4wK9Ev3xxx8qR6zf9uLFizJ7idnf3+vXrzFr1ix89tlnmD17NhQKBTZt2gQnJydYWVmhT58+GndqaIs2in2g8JpcBwcHlVMJJRIJHBwcRB1VAgqPLkdFRZXanpGRgbVr14qK9f3335d63XtBQQE6duwoaods6tSpwsj7JRk/fjw+//xzUTkVpVAoMHv2bGFkY012ovr06aMybdmyRaV99OjRCAoKEh3v9u3b6N69O0xMTISCQ1dXF76+vti1a5eoGNosOl68eIGvvvoK9evXx8CBA5GXl4f58+dDT08PEokE/v7+oj5LW3GAwtP2lYWZVCqFi4uLyum327Ztw+LFi0XPY2ZmJmJjY7Fx40Zs3LgRsbGxJY7h8D5kZGQUO1JW1PPnz7WyjUpKSlK584im9uzZgxEjRmhtOQMKi8l79+5p/L5Hjx7h9OnTOHnypMoZNmKkpKQIYxlUlDZjlSQxMbHYWXZiZGdnY//+/YiOji73aOTa6swCCju0yupA4Q6tQtru0GKMlY3vg84Yey+Sk5NV7u0s5r7A78KrV68oJyeHTE1NS22/f/9+he6nTESUk5NDMpmM9PX1y/X+uLg4On78OIWHh5OFhUWFclHKzs4mmUxGcrlco/cBoEePHpFCoSBra2vhfs9VRW5uLhUUFJCJiUmlxbl16xbl5eWRm5sb6ejoVCgPxph6z58/p7i4OJXtiqenZ6nr9nft6dOn9ODBA/roo49KbM/KyqILFy6Qn59fhT4nOTmZ5HI5OTg4lOv9v/76Kx05coTGjRtHtra2FcpFKSkpifT09MjR0VEr8Rj70EkrOwHG2IehRo0a5OPjQz4+PkJxfu/ePerXr1+FY2sSR0dHp8wduNTUVJo2bVqFc0pPT6fBgweX+/2enp40fPhwsrCw0Nr3lJGRQd98843G75NIJGRnZ0cODg5CcV4Zf7vSyOVyMjExqXCsisSpU6cO1a9fv1hxrkmsly9f0vHjx+nPP/8s1pabm0vr1q17r3E4J86pquaUkJBAO3bsIAcHB+rRowc1adKEtm7dSiNGjKDY2FjR+ShjRUZG0vXr14mI6Pr16zR48GDq16+fRrEsLCxIKpWWGuvcuXOii/OyckpOTtaoOH87Vt26denly5c0duxYjeZPGefGjRvFckpJSeHinDFtquQj+IyxD5gm95h9H3G0GYtzer9xtBmrMnK6ceOGcB9nqVSKVq1a4f79+0K72BGgS4pT9PRxsXG0GYtz4py0GSsmJgZ6enqwtLSEXC5HTEwMbGxsEBgYiICAAMhkMpXxQf5usf7pOTHG1OPz8Bhj78yvv/5aZntSUtJ7jcM5cU5VNacxY8ZQ/fr16fz58/Ts2TMaMWIEtWzZko4ePUrOzs6i8ykpTosWLTSOo81YnBPnpM1Y06dPp9GjR9PMmTNp8+bN9MUXX9DgwYNp1qxZREQ0btw4mjt3LgUEBPwtY/3Tc2KMiVDZPQSMsX+usu5fXvQ+5u8rDufEOVXVnGxtbXH58mXhsUKhwNdffw1nZ2ckJiaKPrqorTicE+dUVXMyNTXFrVu3ABQOtKqjo6MyMOOVK1dE362iKsb6p+fEGFOPr0FnjL0zDg4OtHPnTlIoFCVOFy5ceK9xOCfOqarm9PLlS5Xr1yUSCa1YsYI6dOhAfn5+dPPmzfcah3PinKpqTsr3EhFJpVKSy+VkZmYmtJmYmFBmZubfOtY/PSfGWNm4QGeMvTOenp4UFxdXartEIiGIuJGEtuJwTpxTVc3Jzc2Nzp8/X+z5pUuXUmhoKHXs2FFUPtqKwzlxTlU1J1dXV7p165bw+NSpUyqnyN+9e1f0IGpVMdY/PSfGmHpcoDPG3pnRo0eTr69vqe21a9emI0eOvLc4nBPnVFVz6tSpE23atKnEtqVLl1KPHj1EFfraisM5cU5VNafBgwfT69evhcdv3z0hJiZG9LXQVTHWPz0nxph6fB90xhhjjDHGGGOsCuAj6IwxxhhjjDHGWBXABTpjjDHGGGOMMVYFcIHOGGOMMcYYY4xVAVygM8YYY4wxxhhjVQAX6Iwxxv7x+vTpQ2FhYZWdBmOMMcZYmbhAZ4wxpqJPnz4kkUhIIpGQrq4u2dnZUZs2bWjNmjWkUCgqJaepU6cKOZU2McYYY4z93XGBzhhjrJjg4GBKTU2llJQUiomJodatW9Pw4cPps88+o1evXpX6voKCgneSz6hRoyg1NVWYHB0dafr06SrPMcYYY4z93XGBzhhjrBh9fX2yt7en6tWrU9OmTWn8+PG0Z88eiomJobVr1wqvk0gktGLFCurYsSMZGRnRrFmzaO3atWRubq4Sb/fu3cWOcs+cOZNsbW3JxMSEBgwYQGPHjqXGjRuXmI+xsTHZ29sLk0wmIxMTE+Hx48ePKSAggAwMDMjKyooGDhxIL168KHX+zp07RzY2NjRv3jwiInr27BkNGDCAbGxsyNTUlAICAig+Pl54/dSpU6lx48YUFRVFrq6uZGZmRt27d6esrCzhNdu3b6cGDRoIOQQGBlJ2drbIb5wxxhhjjAt0xhhjIgUEBFCjRo1o586dKs9PnTqVOnXqRFeuXKF+/fqJirVhwwaaNWsWzZs3j+Li4sjZ2ZlWrFhRrryys7MpKCiILCws6Ny5c7Rt2zY6dOgQRURElPj62NhYatOmDc2aNYvGjBlDRERdunShR48eUUxMDMXFxVHTpk3p008/pYyMDOF9iYmJtHv3btq7dy/t3buXjh07RnPnziUiotTUVOrRowf169ePEhIS6OjRo/T5558TgHLNE2OMMcY+TDqVnQBjjLG/Dzc3N7p8+bLKc1988QX17dtXozhLliyh/v37C++bPHkyHThwoMyj3qXZuHEj5ebm0rp168jIyIiIiJYuXUodOnSgefPmkZ2dnfDaXbt2UXh4OP3888/UrVs3IiI6fvw4nT17lh49ekT6+vpERLRgwQLavXs3bd++nQYOHEhERAqFgtauXUsmJiZERNSrVy86fPgwzZo1i1JTU+nVq1f0+eefk4uLCxERNWjQQON5YYwxxtiHjY+gM8YYEw1AsVPVP/74Y43j3Lhxg7y8vFSee/uxWAkJCdSoUSOhOCciatGiBSkUCrpx44bw3JkzZ6hLly4UFRUlFOdERPHx8fTixQuysrIiY2NjYUpOTqbExEThda6urkJxTkTk4OBAjx49IiKiRo0a0aeffkoNGjSgLl260KpVq+jp06flmh/GGGOMfbj4CDpjjDHREhISqEaNGirPFS2MiYikUmmxU7vf1eBxmqhVqxZZWVnRmjVrKCQkhHR1dYmI6MWLF+Tg4EBHjx4t9p6i19IrX68kkUiEUe1lMhkdPHiQTp48SQcOHKAlS5bQhAkT6MyZM8W+L8YYY4yx0vARdMYYY6LExsbSlStXqHPnzmW+zsbGhrKyslQGSLt06ZLKa+rVq0fnzp1Tee7tx2K5u7tTfHy8yuedOHGCpFIp1atXT3jO2tqaYmNj6fbt29S1a1eh06Bp06aUlpZGOjo6VLt2bZXJ2tpadB4SiYRatGhB06ZNo4sXL5Kenh7t2rWrXPPEGGOMsQ8TF+iMMcaKycvLo7S0NLp//z5duHCBZs+eTaGhofTZZ59ReHh4me/19vYmQ0NDGj9+PCUmJtLGjRtVRn4nIho6dCitXr2afvnlF7p16xbNnDmTLl++XK77mX/55Zckl8upd+/edPXqVTpy5AgNHTqUevXqpXL9ORGRra0txcbG0vXr16lHjx706tUrCgwMJB8fHwoLC6MDBw5QSkoKnTx5kiZMmEDnz58XlcOZM2do9uzZdP78ebp79y7t3LmTHj9+TO7u7hrPD2OMMcY+XFygM8YYK2bfvn3k4OBArq6uFBwcTEeOHKHFixfTnj17SCaTlfleS0tLWr9+Pf3+++/UoEED2rRpE02dOlXlNV9++SWNGzeORo0aRU2bNqXk5GTq06cPyeVyjXM1NDSk/fv3U0ZGBjVr1oz+9a9/0aeffkpLly4t8fX29vbC2QBffvklKRQK+v3336lVq1bUt29fqlu3LnXv3p3u3LlTrMAvjampKf3xxx/Uvn17qlu3Lk2cOJH+85//ULt27TSeH8YYY4x9uCTge8AwxhirAtq0aUP29vYUFRVV2akwxhhjjFUKHiSOMcbYe5eTk0MrV66koKAgkslktGnTJjp06BAdPHiwslNjjDHGGKs0fASdMcbYe/fy5Uvq0KEDXbx4kXJzc6levXo0ceJE+vzzzys7NcYYY4yxSsMFOmOMMcYYY4wxVgXwIHGMMcYYY4wxxlgVwAU6Y4wxxhhjjDFWBXCBzhhjjDHGGGOMVQFcoDPGGGOMMcYYY1UAF+iMMcYYY4wxxlgVwAU6Y4wxxhhjjDFWBXCBzhhjjDHGGGOMVQFcoDPGGGOMMcYYY1UAF+iMMcYYY4wxxlgVwAU6Y4wxxhhjjDFWBXCBzhhjjDHGGGOMVQFcoDPGGGOMMcYYY1UAF+iMMcYYY4wxxlgVwAU6Y4wxxhhjjDFWBfw/kqka5/QAPUMAAAAASUVORK5CYII=", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + " \n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "from IPython.display import Image, SVG\n", + "\n", + "attention_plot = Image(att_path)\n", + "molecular_img = SVG(mol_path)\n", + "display(attention_plot)\n", + "display(molecular_img)" + ] + }, + { + "cell_type": "markdown", + "id": "1999d67d6d14b263", + "metadata": { + "id": "1999d67d6d14b263" + }, + "source": [ + "The output images are saved in the `visualization` directory. The attention map shows how much each drug token attends to each protein token, while the molecule image highlights the atoms based on their attention values." + ] + }, + { + "cell_type": "markdown", + "id": "eeb308c3", + "metadata": { + "id": "eeb308c3" + }, + "source": [ + "## Extension Tasks" + ] + }, + { + "cell_type": "markdown", + "id": "aa2a83d8", + "metadata": { + "id": "aa2a83d8" + }, + "source": [ + "### Task 1\n", + "\n", + "To use the BindingDB dataset, modify the relevant line in the Configuration section of Step 0 as shown below.\n", + "\n", + "```python\n", + "cfg.DATA.DATASET = \"bindingdb\"\n", + "```\n", + "\n", + "Reload the dataset and re-run training and testing.\n", + "\n", + "> Tip: See if the model struggles more or less with the new dataset. It can reveal how generalisable DrugBAN is.\n" + ] + }, + { + "cell_type": "markdown", + "id": "c94f174c", + "metadata": { + "id": "c94f174c" + }, + "source": [ + "### Task 2\n", + "\n", + "Turn off domain adaptation by updating the config file and re-running training and testing.\n", + "\n", + "Replace `configs/DA_cross_domain.yaml` with `configs/non_DA_cross_domain.yaml` in the Configuration section of Step 0 as shown below.\n", + "\n", + "```python\n", + "cfg.merge_from_file(\"configs/non_DA_cross_domain.yaml\")\n", + "```\n", + ">Tip: Compare the results with and without domain adaptation to see how it affects model performance." + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "gpuType": "T4", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "widgets": { + "application/vnd.jupyter.widget-state+json": { + "07d966a71f604f63b00707e6d3a0bfe6": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_37f06572693f4972b468c3997fd0687c", + "IPY_MODEL_a0cec4295099427bb8e97229bd49d620", + "IPY_MODEL_931700f44cb0491ca187cbc58dc67476" + ], + "layout": "IPY_MODEL_dd7ad2a05e22470ab00e2118ff994147" + } + }, + "12c8f4410cc74d4f822c779c724bce94": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_79368bfcf5cc4067a59a46c242a77e2d", + "placeholder": "​", + "style": "IPY_MODEL_8632ffdd1b5844f183cc28e824cd117e", + "value": " 2/2 [00:01<00:00,  1.07it/s]" + } + }, + "16cdd1f5e14e405490575177c52f4408": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "1aa1b891fe6b447586d3e87ee62768a2": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "1ed13da64943461ab42ab18495a6246b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_8f16c1ee593d406c88c9bfcf32fdd3df", + "placeholder": "​", + "style": "IPY_MODEL_16cdd1f5e14e405490575177c52f4408", + "value": " 305/305 [01:07<00:00,  4.52it/s, v_num=2]" + } + }, + "2439bd152f3f40a190e80cb5156a8be7": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "2f468dcdec6d4c6db229fcab061fd0d8": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_301141b0856941cba6a0a1f496267c4e", + "placeholder": "​", + "style": "IPY_MODEL_b4f64adc9af34262bb35ea42f0f58899", + "value": "Validation DataLoader 0: 100%" + } + }, + "2fc478f35c3e48f2b0c13bd9a73f3dc6": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "301141b0856941cba6a0a1f496267c4e": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "304a1cadd7c048a2a34568301fb1c4dd": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "32de4bd89e034c35adc198247950d4bf": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_941d8002127f499c9868abffea2a2429", + "max": 2, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_2fc478f35c3e48f2b0c13bd9a73f3dc6", + "value": 2 + } + }, + "33a0e7cd7f7d49ffa57e962ca2bf0b66": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": "2", + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "37f06572693f4972b468c3997fd0687c": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_1aa1b891fe6b447586d3e87ee62768a2", + "placeholder": "​", + "style": "IPY_MODEL_8618e343c4f1435ebf099bd70605606b", + "value": "Validation DataLoader 0: 100%" + } + }, + "3c34cab162424e51b4dcf0008c39c7a0": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": "2", + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "4b0bcd88167b469da36e8433a4d47377": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_a80aad59381c42edaa2adb89d781ddd6", + "max": 305, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_b71c219972474c2c89ed04fa48ec637d", + "value": 305 + } + }, + "530bec1df42241a9be6a34c561be4a36": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_3c34cab162424e51b4dcf0008c39c7a0", + "max": 29, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_b2e51980648f49f2963ba66745b82b57", + "value": 29 + } + }, + "5436ad6633f7491898361cb877e5c96a": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "635b3ed7aa264fa6907187152359a63f": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "64151477b0f4444b852363f72540296e": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "65bb75f11b1f4064b715a166bed1e215": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "6692877437d14f56bdf0bc4e8793bb4b": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": "inline-flex", + "flex": null, + "flex_flow": "row wrap", + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": "100%" + } + }, + "700bde0d67f64f9d8680c483d232fa5b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_b9b44b6940884c359ebb60834280b401", + "placeholder": "​", + "style": "IPY_MODEL_807b512710c848eabb6816cf8e9ecc75", + "value": " 29/29 [00:02<00:00, 12.12it/s]" + } + }, + "79368bfcf5cc4067a59a46c242a77e2d": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "7d4fd3d9c5ed4cc6af20be7384a758ac": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": "inline-flex", + "flex": null, + "flex_flow": "row wrap", + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": "100%" + } + }, + "807b512710c848eabb6816cf8e9ecc75": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "8618e343c4f1435ebf099bd70605606b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "8632ffdd1b5844f183cc28e824cd117e": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "8f16c1ee593d406c88c9bfcf32fdd3df": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "931700f44cb0491ca187cbc58dc67476": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_304a1cadd7c048a2a34568301fb1c4dd", + "placeholder": "​", + "style": "IPY_MODEL_e809f1b951f740ddabc3a3e56d4dd903", + "value": " 29/29 [00:02<00:00, 12.61it/s]" + } + }, + "941d8002127f499c9868abffea2a2429": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": "2", + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "98992d70c9d74d3fa794edf0dc9333b1": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_65bb75f11b1f4064b715a166bed1e215", + "placeholder": "​", + "style": "IPY_MODEL_a893676474004fb0a24023403f5acf46", + "value": "Sanity Checking DataLoader 0: 100%" + } + }, + "9a8deecaaef543fbb18d0f50b83b5abc": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_b113d7ba3f50417fb93de1118b4e4dec", + "IPY_MODEL_4b0bcd88167b469da36e8433a4d47377", + "IPY_MODEL_1ed13da64943461ab42ab18495a6246b" + ], + "layout": "IPY_MODEL_7d4fd3d9c5ed4cc6af20be7384a758ac" + } + }, + "9d5ebfa060ac49d6bd7a8c4837b4fc29": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "a0cec4295099427bb8e97229bd49d620": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_33a0e7cd7f7d49ffa57e962ca2bf0b66", + "max": 29, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_f223529cf08b4235b8e74ad1049c5a60", + "value": 29 + } + }, + "a3aa520f98b042f29f1fca44f24b61e0": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": "2", + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "a80aad59381c42edaa2adb89d781ddd6": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": "2", + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "a893676474004fb0a24023403f5acf46": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "abf0b363155c4b1fb8de818ae41606eb": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": "inline-flex", + "flex": null, + "flex_flow": "row wrap", + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": "hidden", + "width": "100%" + } + }, + "af4b74bd9f13458c8dd9b74ad2004783": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "b113d7ba3f50417fb93de1118b4e4dec": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_635b3ed7aa264fa6907187152359a63f", + "placeholder": "​", + "style": "IPY_MODEL_9d5ebfa060ac49d6bd7a8c4837b4fc29", + "value": "Epoch 1: 100%" + } + }, + "b2e51980648f49f2963ba66745b82b57": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "b4f64adc9af34262bb35ea42f0f58899": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "b71c219972474c2c89ed04fa48ec637d": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "b9b44b6940884c359ebb60834280b401": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "bc415b5a5635482eb20a65601866febf": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_98992d70c9d74d3fa794edf0dc9333b1", + "IPY_MODEL_32de4bd89e034c35adc198247950d4bf", + "IPY_MODEL_12c8f4410cc74d4f822c779c724bce94" + ], + "layout": "IPY_MODEL_fe6bdb21df9c4415b9899aaa96969502" + } + }, + "be7949a38cfe4129abd0583b3b9c080e": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_e9d78f7f6ffd4d22b7f85f65da6e54c3", + "placeholder": "​", + "style": "IPY_MODEL_5436ad6633f7491898361cb877e5c96a", + "value": " 29/29 [00:02<00:00, 10.86it/s]" + } + }, + "d10203b9631842348ce5ee323f56d8c3": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_a3aa520f98b042f29f1fca44f24b61e0", + "max": 29, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_64151477b0f4444b852363f72540296e", + "value": 29 + } + }, + "d782f86576c9463eb171f866f007b9b3": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_f7b99f048d0b430b913f03382dcf6cae", + "IPY_MODEL_d10203b9631842348ce5ee323f56d8c3", + "IPY_MODEL_be7949a38cfe4129abd0583b3b9c080e" + ], + "layout": "IPY_MODEL_6692877437d14f56bdf0bc4e8793bb4b" + } + }, + "dd7ad2a05e22470ab00e2118ff994147": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": "inline-flex", + "flex": null, + "flex_flow": "row wrap", + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": "hidden", + "width": "100%" + } + }, + "e809f1b951f740ddabc3a3e56d4dd903": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "e9d78f7f6ffd4d22b7f85f65da6e54c3": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "f223529cf08b4235b8e74ad1049c5a60": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "f60cfca35cab44e5801bbb2b04f9cc6f": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_2f468dcdec6d4c6db229fcab061fd0d8", + "IPY_MODEL_530bec1df42241a9be6a34c561be4a36", + "IPY_MODEL_700bde0d67f64f9d8680c483d232fa5b" + ], + "layout": "IPY_MODEL_abf0b363155c4b1fb8de818ae41606eb" + } + }, + "f7b99f048d0b430b913f03382dcf6cae": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_2439bd152f3f40a190e80cb5156a8be7", + "placeholder": "​", + "style": "IPY_MODEL_af4b74bd9f13458c8dd9b74ad2004783", + "value": "Testing DataLoader 0: 100%" + } + }, + "fe6bdb21df9c4415b9899aaa96969502": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": "inline-flex", + "flex": null, + "flex_flow": "row wrap", + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": "hidden", + "width": "100%" + } + } + } + } + }, + "nbformat": 4, + "nbformat_minor": 5 } From cc3ecbf5ffdbd08010ff456721a9e90c79d2ad7a Mon Sep 17 00:00:00 2001 From: "L. M. Riza Rizky" <42672299+zaRizk7@users.noreply.github.com> Date: Mon, 14 Jul 2025 19:13:50 +0100 Subject: [PATCH 2/3] pre-commit --- .../tutorial-drug.ipynb | 3364 ++++------------- 1 file changed, 663 insertions(+), 2701 deletions(-) diff --git a/tutorials/drug-target-interaction/tutorial-drug.ipynb b/tutorials/drug-target-interaction/tutorial-drug.ipynb index 9f22516..bce1839 100644 --- a/tutorials/drug-target-interaction/tutorial-drug.ipynb +++ b/tutorials/drug-target-interaction/tutorial-drug.ipynb @@ -1,34 +1,36 @@ { + "nbformat": 4, + "nbformat_minor": 5, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, "cells": [ { - "cell_type": "markdown", - "id": "8c1bf9c7", - "metadata": { - "id": "8c1bf9c7" - }, + "metadata": {}, "source": [ - "# Drug–Target Interaction Prediction\n", + "# Drug\u2013Target Interaction Prediction\n", "\n", "![](https://github.com/pykale/mmai-tutorials/blob/main/tutorials/drug-target-interaction/images/drugban-pyakle-api.png?raw=1)\n", "\n", "\n", "In this tutorial, we will train models to predict the interaction between **two data modalities**: **molecules (drug)** and **proteins (target)** using `PyKale`. Drug-target interaction (DTI) plays a key role in drug discovery and identifying potential therapeutic targets. This example is based on the **DrugBAN** framework by [**Bai et al. (_Nature Machine Intelligence_, 2023)**](https://www.nature.com/articles/s42256-022-00605-1).\n", "\n", - "The DTI prediction problem is formulated as a **binary classification task**, where the goal is to predict whether a given **drug–protein pair interacts or not**. The DrugBAN framework tackles this problem using two key ideas:\n", + "The DTI prediction problem is formulated as a **binary classification task**, where the goal is to predict whether a given **drug\u2013protein pair interacts or not**. The DrugBAN framework tackles this problem using two key ideas:\n", "\n", "- **Bilinear Attention Network (BAN)**, which learns detailed feature representations for both drugs and proteins and captures local interaction patterns between them.\n", "\n", - "- **Adversarial Domain Adaptation**, which helps the model generalise to out-of-distribution datasets, i.e., in clustering-based cross-validation instead of random splits, improving its ability to predict interactions on unseen drug–target pairs.\n", + "- **Adversarial Domain Adaptation**, which helps the model generalise to out-of-distribution datasets, i.e., in clustering-based cross-validation instead of random splits, improving its ability to predict interactions on unseen drug\u2013target pairs.\n", "\n", "With `PyKale`, implementing such a multimodal DTI prediction pipeline is straightforward. The library provides ready-to-use modules and configuration support, making it easy to apply advanced techniques with minimal custom coding." - ] + ], + "cell_type": "markdown", + "id": "8c1bf9c7" }, { - "cell_type": "markdown", - "id": "745ccdcf", - "metadata": { - "id": "745ccdcf" - }, + "metadata": {}, "source": [ "## Step 0: Environment Preparation\n", "\n", @@ -38,23 +40,27 @@ "\n", "Moreover, we provide helper functions that can be inspected directly in the `.py` files located in the notebook's current directory. The additional helper script is:\n", "- [`config.py`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/drug-target-interaction/configs.py): Defines the base configuration settings, which can be overridden using a custom `.yaml` file." - ] + ], + "cell_type": "markdown", + "id": "745ccdcf" }, { + "metadata": {}, + "source": [ + "import os\n", + "\n", + "!rm -rf /content/mmai-tutorials\n", + "!git clone --single-branch -b main https://github.com/pykale/mmai-tutorials.git\n", + "%mv /content/mmai-tutorials/tutorials/drug-target-interaction /content/\n", + "%cd /content/drug-target-interaction\n", + "\n", + "print(\"Changed working directory to:\", os.getcwd())" + ], "cell_type": "code", - "execution_count": 1, - "id": "a6028209", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "a6028209", - "outputId": "bdba9c0d-e5a6-4dba-981d-b12e21d2c463" - }, "outputs": [ { - "name": "stdout", "output_type": "stream", + "name": "stdout", "text": [ "Cloning into 'mmai-tutorials'...\n", "remote: Enumerating objects: 610, done.\u001b[K\n", @@ -69,23 +75,11 @@ ] } ], - "source": [ - "import os\n", - "\n", - "!rm -rf /content/mmai-tutorials\n", - "!git clone --single-branch -b main https://github.com/pykale/mmai-tutorials.git\n", - "%mv /content/mmai-tutorials/tutorials/drug-target-interaction /content/\n", - "%cd /content/drug-target-interaction\n", - "\n", - "print(\"Changed working directory to:\", os.getcwd())" - ] + "id": "a6028209", + "execution_count": null }, { - "cell_type": "markdown", - "id": "c52c6334", - "metadata": { - "id": "c52c6334" - }, + "metadata": {}, "source": [ "### Package Installation\n", "\n", @@ -96,134 +90,98 @@ "Then, we install `PyG` (PyTorch Geometric) and related packages.\n", "\n", "Please **do not** re-run this session after installation completed. Runing this installation multiple times will trigger issues related to `PyG`. If you want to re-run this installation, please click the `Runtime` on the top menu and choose `Disconnect and delete runtime` before installing." - ] + ], + "cell_type": "markdown", + "id": "c52c6334" }, { + "metadata": {}, + "source": [ + "%pip install --quiet \\\n", + " \"pykale[example]@git+https://github.com/pykale/pykale@main\" \\\n", + " gdown==5.2.0 torch-geometric==2.6.0 torch_sparse torch_scatter \\\n", + " -f https://data.pyg.org/whl/torch-2.6.0+cu124.html \\\n", + " && echo \"pykale, gdown, pyg and yacs installed successfully \u2705\" \\\n", + " || echo \"Failed to install pykale, gdown, pyg and yacs \u274c\"" + ], "cell_type": "code", - "execution_count": 2, - "id": "53e3b14e", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "53e3b14e", - "outputId": "7890c8c4-6c6c-4156-bdf4-55ffbf9ee2b5" - }, "outputs": [ { - "name": "stdout", "output_type": "stream", + "name": "stdout", "text": [ " Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n", " Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n", " Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n", - "pykale, gdown, pyg and yacs installed successfully ✅\n" + "pykale, gdown, pyg and yacs installed successfully \u2705\n" ] } ], - "source": [ - "%pip install --quiet \\\n", - " \"pykale[example]@git+https://github.com/pykale/pykale@main\" \\\n", - " gdown==5.2.0 torch-geometric==2.6.0 torch_sparse torch_scatter \\\n", - " -f https://data.pyg.org/whl/torch-2.6.0+cu124.html \\\n", - " && echo \"pykale, gdown, pyg and yacs installed successfully ✅\" \\\n", - " || echo \"Failed to install pykale, gdown, pyg and yacs ❌\"" - ] + "id": "53e3b14e", + "execution_count": null }, { - "cell_type": "markdown", - "id": "69f50b6a", - "metadata": { - "id": "69f50b6a" - }, + "metadata": {}, "source": [ "We then hide the warnings messages to get a clear output." - ] + ], + "cell_type": "markdown", + "id": "69f50b6a" }, { - "cell_type": "code", - "execution_count": 3, - "id": "6e871c63", - "metadata": { - "id": "6e871c63" - }, - "outputs": [], + "metadata": {}, "source": [ "import os\n", "import warnings\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "os.environ[\"PYTHONWARNINGS\"] = \"ignore\"" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "6e871c63", + "execution_count": null }, { - "cell_type": "markdown", - "id": "6606e3fb", - "metadata": { - "id": "6606e3fb" - }, + "metadata": {}, "source": [ "Exercise: Check NumPy Version" - ] + ], + "cell_type": "markdown", + "id": "6606e3fb" }, { + "metadata": {}, + "source": [ + "import numpy as np\n", + "\n", + "print(\"NumPy version:\", np.__version__) # numpy should be 2.0.0 or higher" + ], "cell_type": "code", - "execution_count": 4, - "id": "0d384020", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "0d384020", - "outputId": "d1ecfffa-1567-4b1c-dc8e-cbbeda3728c0" - }, "outputs": [ { - "name": "stdout", "output_type": "stream", + "name": "stdout", "text": [ "NumPy version: 2.0.2\n" ] } ], - "source": [ - "import numpy as np\n", - "\n", - "print(\"NumPy version:\", np.__version__) # numpy should be 2.0.0 or higher" - ] + "id": "0d384020", + "execution_count": null }, { - "cell_type": "markdown", - "id": "cabd3406", - "metadata": { - "id": "cabd3406" - }, + "metadata": {}, "source": [ "### Configuration\n", "\n", "To minimize the footprint of the notebook when specifying configurations, we provide a [`config.py`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/drug-target-interaction/configs.py) file that defines default parameters. These can be customized by supplying a `.yaml` configuration file, such as [`configs/DA_cross_domain.yaml`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/drug-target-interaction/configs/DA_cross_domain.yaml) as an example." - ] + ], + "cell_type": "markdown", + "id": "cabd3406" }, { - "cell_type": "code", - "execution_count": 5, - "id": "55c13b48", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "55c13b48", - "outputId": "dd3df032-9263-4bcf-d47c-3059d1f66830" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "/content/drug-target-interaction\n" - ] - } - ], + "metadata": {}, "source": [ "from configs import get_cfg_defaults\n", "\n", @@ -233,81 +191,78 @@ "cfg.merge_from_file(\n", " \"configs/DA_cross_domain.yaml\"\n", ") # Update (or override) some of those settings using a custom YAML file" - ] + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "/content/drug-target-interaction\n" + ] + } + ], + "id": "55c13b48", + "execution_count": null }, { - "cell_type": "markdown", - "id": "74ffdbc2", - "metadata": { - "id": "74ffdbc2" - }, + "metadata": {}, "source": [ "In this tutorial, we list the hyperparameters we would like users to play with outside the `.yaml` file:\n", "- `cfg.SOLVER.MAX_EPOCH`: Number of epochs in training stage. You can reduce the number of training epochs to shorten runtime.\n", "- `cfg.DATA.DATASET`: The dataset used in the study. This can be `bindingdb` or `biosnap`.\n", "\n", "As a quick exercise, please take a moment to review and understand the parameters in [`config.py`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/drug-target-interaction/configs.py)." - ] + ], + "cell_type": "markdown", + "id": "74ffdbc2" }, { - "cell_type": "code", - "execution_count": 6, - "id": "424c7286", - "metadata": { - "id": "424c7286" - }, - "outputs": [], + "metadata": {}, "source": [ "cfg.SOLVER.MAX_EPOCH = 2" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "424c7286", + "execution_count": null }, { - "cell_type": "markdown", - "id": "97c088fd", - "metadata": { - "id": "97c088fd" - }, + "metadata": {}, "source": [ "You can also switch to a different dataset." - ] + ], + "cell_type": "markdown", + "id": "97c088fd" }, { - "cell_type": "code", - "execution_count": 7, - "id": "c69376fa", - "metadata": { - "id": "c69376fa" - }, - "outputs": [], + "metadata": {}, "source": [ "cfg.DATA.DATASET = \"biosnap\"" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "c69376fa", + "execution_count": null }, { - "cell_type": "markdown", - "id": "d3d41633", - "metadata": { - "id": "d3d41633" - }, + "metadata": {}, "source": [ "Exercise: Now print the full configuration to check all current hyperparameter and dataset settings." - ] + ], + "cell_type": "markdown", + "id": "d3d41633" }, { + "metadata": {}, + "source": [ + "print(cfg)" + ], "cell_type": "code", - "execution_count": 8, - "id": "45874296", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "45874296", - "outputId": "5c4738ff-85e8-463f-dac7-6e183643a223" - }, "outputs": [ { - "name": "stdout", "output_type": "stream", + "name": "stdout", "text": [ "BCN:\n", " HEADS: 2\n", @@ -359,42 +314,31 @@ ] } ], - "source": [ - "print(cfg)" - ] + "id": "45874296", + "execution_count": null }, { - "cell_type": "markdown", - "id": "17558d0c", - "metadata": { - "id": "17558d0c" - }, + "metadata": {}, "source": [ "## Step 1: Data Loading and Preparation\n", "\n", "In this tutorial, we use the **Biosnap** dataset for the main demonstration and the **BindingDB** dataset for the exercise at the end." - ] + ], + "cell_type": "markdown", + "id": "17558d0c" }, { - "cell_type": "markdown", - "id": "6c6071b9", - "metadata": { - "id": "6c6071b9" - }, + "metadata": {}, "source": [ "### Data Downloading\n", "\n", "Please run the following cell to download necessary datasets." - ] + ], + "cell_type": "markdown", + "id": "6c6071b9" }, { - "cell_type": "code", - "execution_count": 9, - "id": "56f9f58e", - "metadata": { - "id": "56f9f58e" - }, - "outputs": [], + "metadata": {}, "source": [ "!rm -rf data\n", "!mkdir data\n", @@ -404,33 +348,35 @@ "!gdown --id 1ogOcxZn-1q418LOT-gQ94aHQV0Y1sOmk --output data/drug-target-interaction.zip\n", "!unzip data/drug-target-interaction.zip -d data/\n", "!mv data/drug-target-interaction/checkpoint ./" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "56f9f58e", + "execution_count": null }, { - "cell_type": "markdown", - "id": "c39b3e39", - "metadata": { - "id": "c39b3e39" - }, + "metadata": {}, "source": [ "Exercise: Check the data is ready" - ] + ], + "cell_type": "markdown", + "id": "c39b3e39" }, { + "metadata": {}, + "source": [ + "import os\n", + "import shutil\n", + "\n", + "print(\"Contents of the data folder:\")\n", + "for item in os.listdir(\"data/drug-target-interaction\"):\n", + " print(item)" + ], "cell_type": "code", - "execution_count": 10, - "id": "a6258d1f", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "a6258d1f", - "outputId": "2813ad1f-5971-44ca-f9b6-c90a496fe4b6" - }, "outputs": [ { - "name": "stdout", "output_type": "stream", + "name": "stdout", "text": [ "Contents of the data folder:\n", "biosnap\n", @@ -439,55 +385,42 @@ ] } ], - "source": [ - "import os\n", - "import shutil\n", - "\n", - "print(\"Contents of the data folder:\")\n", - "for item in os.listdir(\"data/drug-target-interaction\"):\n", - " print(item)" - ] + "id": "a6258d1f", + "execution_count": null }, { - "cell_type": "markdown", - "id": "9ab0b5f833dc40f8", - "metadata": { - "id": "9ab0b5f833dc40f8" - }, + "metadata": {}, "source": [ "The data content is structured as follows:\n", "```sh\n", - " ├───data\n", - " │ ├───checkpoint\n", - " │ ├───bindingdb\n", - " │ ├───biosnap" - ] + " \u251c\u2500\u2500\u2500data\n", + " \u2502 \u251c\u2500\u2500\u2500checkpoint\n", + " \u2502 \u251c\u2500\u2500\u2500bindingdb\n", + " \u2502 \u251c\u2500\u2500\u2500biosnap" + ], + "cell_type": "markdown", + "id": "9ab0b5f833dc40f8" }, { - "cell_type": "markdown", - "id": "5be1dcc62b7d5649", - "metadata": { - "id": "5be1dcc62b7d5649" - }, + "metadata": {}, "source": [ "The `data` folder contains two datasets: `bindingdb` and `biosnap`. Each dataset folder contains the following files. The `checkpoint` folder contains the saved model checkpoint, which are used later in the interpretation section." - ] + ], + "cell_type": "markdown", + "id": "5be1dcc62b7d5649" }, { + "metadata": {}, + "source": [ + "print(\"Contents of bindingdb folder:\")\n", + "for item in os.listdir(\"data/drug-target-interaction/bindingdb\"):\n", + " print(item)" + ], "cell_type": "code", - "execution_count": 11, - "id": "a93303c51c8b974e", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "a93303c51c8b974e", - "outputId": "93e0c029-b489-4f4d-8889-8b99f868039c" - }, "outputs": [ { - "name": "stdout", "output_type": "stream", + "name": "stdout", "text": [ "Contents of bindingdb folder:\n", "random\n", @@ -497,77 +430,64 @@ ] } ], - "source": [ - "print(\"Contents of bindingdb folder:\")\n", - "for item in os.listdir(\"data/drug-target-interaction/bindingdb\"):\n", - " print(item)" - ] + "id": "a93303c51c8b974e", + "execution_count": null }, { - "cell_type": "markdown", - "id": "79cbc1c1", - "metadata": { - "id": "79cbc1c1" - }, + "metadata": {}, "source": [ "Each dataset folder follows the structure:\n", "\n", "```sh\n", - " ├───dataset_name\n", - " │ ├───cluster\n", - " │ │ ├───source_train.csv\n", - " │ │ ├───target_train.csv\n", - " │ │ ├───target_test.csv\n", - " │ ├───random\n", - " │ │ ├───test.csv\n", - " │ │ ├───train.csv\n", - " │ │ ├───val.csv\n", - " │ ├───full.csv\n", + " \u251c\u2500\u2500\u2500dataset_name\n", + " \u2502 \u251c\u2500\u2500\u2500cluster\n", + " \u2502 \u2502 \u251c\u2500\u2500\u2500source_train.csv\n", + " \u2502 \u2502 \u251c\u2500\u2500\u2500target_train.csv\n", + " \u2502 \u2502 \u251c\u2500\u2500\u2500target_test.csv\n", + " \u2502 \u251c\u2500\u2500\u2500random\n", + " \u2502 \u2502 \u251c\u2500\u2500\u2500test.csv\n", + " \u2502 \u2502 \u251c\u2500\u2500\u2500train.csv\n", + " \u2502 \u2502 \u251c\u2500\u2500\u2500val.csv\n", + " \u2502 \u251c\u2500\u2500\u2500full.csv\n", "```" - ] + ], + "cell_type": "markdown", + "id": "79cbc1c1" }, { - "cell_type": "markdown", - "id": "d35e04f9", - "metadata": { - "id": "d35e04f9" - }, + "metadata": {}, "source": [ "We use the cluster dataset folder for cross-domain prediction, containing three parts:\n", "\n", - "- Train samples from the source domain: Drug–protein pairs the model learns from.\n", + "- Train samples from the source domain: Drug\u2013protein pairs the model learns from.\n", "\n", "- Train samples from the target domain: Additional training data from a different distribution to improve generalisation.\n", "\n", - "- Test samples from the target domain: Unseen drug–protein pairs used to evaluate model performance on new data.\n", + "- Test samples from the target domain: Unseen drug\u2013protein pairs used to evaluate model performance on new data.\n", "\n", "The source and target sets are defined based on the clustering results." - ] + ], + "cell_type": "markdown", + "id": "d35e04f9" }, { - "cell_type": "markdown", - "id": "98acf744", - "metadata": { - "id": "98acf744" - }, + "metadata": {}, "source": [ "### Data Loading" - ] + ], + "cell_type": "markdown", + "id": "98acf744" }, { - "cell_type": "markdown", - "id": "1e5f4f44", - "metadata": { - "id": "1e5f4f44" - }, + "metadata": {}, "source": [ - "Here’s what each csv file looks like in a table format:\n", + "Here\u2019s what each csv file looks like in a table format:\n", "\n", "| SMILES | Protein Sequence | Y |\n", "|--------------------|--------------------------|---|\n", - "| Fc1ccc(C2(COC…) | MDNVLPVDSDLS… | 1 |\n", - "| O=c1oc2c(O)c(…) | MMYSKLLTLTTL… | 0 |\n", - "| CC(C)Oc1cc(N…) | MGMACLTMTEME… | 1 |\n", + "| Fc1ccc(C2(COC\u2026) | MDNVLPVDSDLS\u2026 | 1 |\n", + "| O=c1oc2c(O)c(\u2026) | MMYSKLLTLTTL\u2026 | 0 |\n", + "| CC(C)Oc1cc(N\u2026) | MGMACLTMTEME\u2026 | 1 |\n", "\n", "Each row of the dataset contains three key pieces of information:\n", "\n", @@ -580,39 +500,44 @@ "\n", "\n", "**Y (Labels)**: \n", - "Each drug–protein pair is given a label:\n", + "Each drug\u2013protein pair is given a label:\n", "- `1` if they interact\n", "- `0` if they do not\n", "\n", "\n", - "Each row shows one drug–protein pair. The goal of our machine learning model is to predict the last column (**Y**) — whether or not the drug and protein interact." - ] + "Each row shows one drug\u2013protein pair. The goal of our machine learning model is to predict the last column (**Y**) \u2014 whether or not the drug and protein interact." + ], + "cell_type": "markdown", + "id": "1e5f4f44" }, { - "cell_type": "markdown", - "id": "b7590daf", - "metadata": { - "id": "b7590daf" - }, + "metadata": {}, "source": [ "You can load CSV files into Python using tools like `pandas`. The output shows a sample of the data, including the SMILES string for the drug, the protein sequence, the interaction label (Y) and the cluster ID." - ] + ], + "cell_type": "markdown", + "id": "b7590daf" }, { + "metadata": {}, + "source": [ + "import pandas as pd\n", + "\n", + "dataFolder = os.path.join(\n", + " f\"data/drug-target-interaction/{cfg.DATA.DATASET}\", str(cfg.DATA.SPLIT)\n", + ")\n", + "\n", + "df_train_source = pd.read_csv(os.path.join(dataFolder, \"source_train.csv\"))\n", + "df_train_target = pd.read_csv(os.path.join(dataFolder, \"target_train.csv\"))\n", + "df_test_target = pd.read_csv(os.path.join(dataFolder, \"target_test.csv\"))\n", + "\n", + "print(\"Sample example:\", df_train_source.iloc[0])" + ], "cell_type": "code", - "execution_count": 12, - "id": "0c709e31", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "0c709e31", - "outputId": "5e663c15-f59f-4a0f-acc3-b545b6392a1e" - }, "outputs": [ { - "name": "stdout", "output_type": "stream", + "name": "stdout", "text": [ "Sample example: SMILES CC1=CN=C2N1C=CN=C2NCC1=CC=NC=C1\n", "Protein MARSLLLPLQILLLSLALETAGEEAQGDKIIDGAPCARGSHPWQVA...\n", @@ -623,26 +548,11 @@ ] } ], - "source": [ - "import pandas as pd\n", - "\n", - "dataFolder = os.path.join(\n", - " f\"data/drug-target-interaction/{cfg.DATA.DATASET}\", str(cfg.DATA.SPLIT)\n", - ")\n", - "\n", - "df_train_source = pd.read_csv(os.path.join(dataFolder, \"source_train.csv\"))\n", - "df_train_target = pd.read_csv(os.path.join(dataFolder, \"target_train.csv\"))\n", - "df_test_target = pd.read_csv(os.path.join(dataFolder, \"target_test.csv\"))\n", - "\n", - "print(\"Sample example:\", df_train_source.iloc[0])" - ] + "id": "0c709e31", + "execution_count": null }, { - "cell_type": "markdown", - "id": "542d4e69", - "metadata": { - "id": "542d4e69" - }, + "metadata": {}, "source": [ "### Data Preprocessing\n", "\n", @@ -652,26 +562,20 @@ "Protein sequences are transformed into fixed-length integer arrays using `kale.prepdata.chem_transform.integer_label_protein`, with each amino acid mapped to an integer and sequences padded or truncated to a uniform length.\n", "\n", "Finally, the `kale.loaddata.molecular_datasets.DTIDataset` class packages drugs, proteins, and labels into a PyTorch-ready dataset." - ] + ], + "cell_type": "markdown", + "id": "542d4e69" }, { - "cell_type": "markdown", - "id": "981d5520", - "metadata": { - "id": "981d5520" - }, + "metadata": {}, "source": [ "**Note:** If you encounter an error related to requiring numpy `<2.0`, simply ignore it and re-run this block until it completes successfully." - ] + ], + "cell_type": "markdown", + "id": "981d5520" }, { - "cell_type": "code", - "execution_count": 13, - "id": "ae5af8eb", - "metadata": { - "id": "ae5af8eb" - }, - "outputs": [], + "metadata": {}, "source": [ "from kale.loaddata.molecular_datasets import DTIDataset\n", "\n", @@ -679,24 +583,22 @@ "train_dataset = DTIDataset(df_train_source.index.values, df_train_source)\n", "train_target_dataset = DTIDataset(df_train_target.index.values, df_train_target)\n", "test_target_dataset = DTIDataset(df_test_target.index.values, df_test_target)" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "ae5af8eb", + "execution_count": null }, { - "cell_type": "markdown", - "id": "a0a510ce", - "metadata": { - "id": "a0a510ce" - }, + "metadata": {}, "source": [ "We load data in small, manageable pieces called batches to save memory and speed up training. We use `kale.loaddata.sampler.MultiDataLoader` from PyKale to load one batch from the source domain and one from the target domain at each training step." - ] + ], + "cell_type": "markdown", + "id": "a0a510ce" }, { - "cell_type": "markdown", - "id": "c09084c0", - "metadata": { - "id": "c09084c0" - }, + "metadata": {}, "source": [ "First, we specify a few DataLoader parameters:\n", "- Batch size: Number of samples per batch\n", @@ -704,21 +606,31 @@ "- Number of workers: Parallel data loading\n", "- Drop last: Discard the last incomplete batch for consistent batch sizes\n", "- Collate function: Use graph_collate_func to batch variable-sized molecular graphs" - ] + ], + "cell_type": "markdown", + "id": "c09084c0" }, { + "metadata": {}, + "source": [ + "from torch.utils.data import DataLoader\n", + "from kale.loaddata.molecular_datasets import graph_collate_func\n", + "from kale.loaddata.sampler import MultiDataLoader\n", + "\n", + "params = {\n", + " \"batch_size\": cfg.SOLVER.BATCH_SIZE,\n", + " \"shuffle\": True,\n", + " \"num_workers\": cfg.SOLVER.NUM_WORKERS,\n", + " \"drop_last\": True,\n", + " \"collate_fn\": graph_collate_func,\n", + "}\n", + "\n", + "params" + ], "cell_type": "code", - "execution_count": 14, - "id": "94a15868", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "94a15868", - "outputId": "a4c14890-db12-45b7-bcbe-1a94b4f846b2" - }, "outputs": [ { + "output_type": "execute_result", "data": { "text/plain": [ "{'batch_size': 32,\n", @@ -728,57 +640,23 @@ " 'collate_fn': }" ] }, - "execution_count": 14, "metadata": {}, - "output_type": "execute_result" + "execution_count": 14 } ], - "source": [ - "from torch.utils.data import DataLoader\n", - "from kale.loaddata.molecular_datasets import graph_collate_func\n", - "from kale.loaddata.sampler import MultiDataLoader\n", - "\n", - "params = {\n", - " \"batch_size\": cfg.SOLVER.BATCH_SIZE,\n", - " \"shuffle\": True,\n", - " \"num_workers\": cfg.SOLVER.NUM_WORKERS,\n", - " \"drop_last\": True,\n", - " \"collate_fn\": graph_collate_func,\n", - "}\n", - "\n", - "params" - ] + "id": "94a15868", + "execution_count": null }, { - "cell_type": "markdown", - "id": "e884ed07", - "metadata": { - "id": "e884ed07" - }, + "metadata": {}, "source": [ "Then, we create a DataLoader from both the source and target datasets for training." - ] + ], + "cell_type": "markdown", + "id": "e884ed07" }, { - "cell_type": "code", - "execution_count": 15, - "id": "24ba12b5", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "24ba12b5", - "outputId": "47a5c085-d2d0-48f8-cad2-4483e3fe2efa" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Using domain adaptation: True\n" - ] - } - ], + "metadata": {}, "source": [ "print(\"Using domain adaptation:\", cfg.DA.USE)\n", "\n", @@ -795,26 +673,30 @@ " training_generator = MultiDataLoader(\n", " dataloaders=[source_generator, target_generator], n_batches=n_batches\n", " )" - ] + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Using domain adaptation: True\n" + ] + } + ], + "id": "24ba12b5", + "execution_count": null }, { - "cell_type": "markdown", - "id": "649301de", - "metadata": { - "id": "649301de" - }, + "metadata": {}, "source": [ - "Lastly, we set up DataLoaders for validation and testing. Since we don’t want to shuffle or drop any samples, we adjust the parameters accordingly." - ] + "Lastly, we set up DataLoaders for validation and testing. Since we don\u2019t want to shuffle or drop any samples, we adjust the parameters accordingly." + ], + "cell_type": "markdown", + "id": "649301de" }, { - "cell_type": "code", - "execution_count": 16, - "id": "b4cf543a", - "metadata": { - "id": "b4cf543a" - }, - "outputs": [], + "metadata": {}, "source": [ "# Update parameters for validation/testing (no shuffling, keep all data)\n", "params.update({\"shuffle\": False, \"drop_last\": False})\n", @@ -822,43 +704,24 @@ "# Create validation and test data loaders\n", "valid_generator = DataLoader(test_target_dataset, **params)\n", "test_generator = DataLoader(test_target_dataset, **params)" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "b4cf543a", + "execution_count": null }, { - "cell_type": "markdown", - "id": "e474eea2", - "metadata": { - "id": "e474eea2" - }, + "metadata": {}, "source": [ "### Exercise: Dataset Inspection\n", "\n", - "Once the dataset is ready, let’s inspect one sample from the training data to check the input graph, protein sequence, and label format." - ] + "Once the dataset is ready, let\u2019s inspect one sample from the training data to check the input graph, protein sequence, and label format." + ], + "cell_type": "markdown", + "id": "e474eea2" }, { - "cell_type": "code", - "execution_count": 17, - "id": "31b8a93f", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "31b8a93f", - "outputId": "74ae5660-3d74-4c5e-f5ed-065c1f099516" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "First sample from source batch:\n", - "Drug graph: Data(x=[290, 7], edge_index=[2, 106], edge_attr=[106, 1], num_nodes=290)\n", - "Protein sequence: tensor([11., 7., 18., ..., 0., 0., 0.], dtype=torch.float64)\n", - "Label: tensor(0., dtype=torch.float64)\n" - ] - } - ], + "metadata": {}, "source": [ "# Get the first batch (contains one batch from source and one from target)\n", "first_batch = next(iter(training_generator))\n", @@ -871,14 +734,25 @@ "print(\"Drug graph:\", source_batch[0][0])\n", "print(\"Protein sequence:\", source_batch[1][0])\n", "print(\"Label:\", source_batch[2][0])" - ] + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "First sample from source batch:\n", + "Drug graph: Data(x=[290, 7], edge_index=[2, 106], edge_attr=[106, 1], num_nodes=290)\n", + "Protein sequence: tensor([11., 7., 18., ..., 0., 0., 0.], dtype=torch.float64)\n", + "Label: tensor(0., dtype=torch.float64)\n" + ] + } + ], + "id": "31b8a93f", + "execution_count": null }, { - "cell_type": "markdown", - "id": "cb0b269b", - "metadata": { - "id": "cb0b269b" - }, + "metadata": {}, "source": [ "This sample is a tuple with three parts:\n", "\n", @@ -893,47 +767,43 @@ "\n", "3. **Label (float)**\n", "- `0.0`; The ground-truth interaction label indicating no interaction." - ] + ], + "cell_type": "markdown", + "id": "cb0b269b" }, { - "cell_type": "markdown", - "id": "8eaf5c8f", - "metadata": { - "id": "8eaf5c8f" - }, + "metadata": {}, "source": [ "## Step 2: Model Definition" - ] + ], + "cell_type": "markdown", + "id": "8eaf5c8f" }, { - "cell_type": "markdown", - "id": "b2819549", - "metadata": { - "id": "b2819549" - }, + "metadata": {}, "source": [ "### Embed\n", "\n", "DrugBAN consists of three main components: a Graph Convolutional Network (GCN) for extracting structural features from drug molecular graphs, a Convolutional Neural Network (CNN) for encoding protein sequences, and a Bilinear Attention Network (BAN) for fusing drug and protein features. The fused representation is then passed through a Multi-Layer Perceptron (MLP) classifier to predict interaction scores.\n", "\n", "We define the DrugBAN class in `kale.embed.ban`." - ] + ], + "cell_type": "markdown", + "id": "b2819549" }, { + "metadata": {}, + "source": [ + "from kale.embed.ban import DrugBAN\n", + "\n", + "model = DrugBAN(**cfg)\n", + "print(model)" + ], "cell_type": "code", - "execution_count": 18, - "id": "1c8f3acc", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "1c8f3acc", - "outputId": "65f82225-7219-4391-abf2-2194a00c3af0" - }, "outputs": [ { - "name": "stdout", "output_type": "stream", + "name": "stdout", "text": [ "DrugBAN(\n", " (drug_extractor): MolecularGCN(\n", @@ -982,32 +852,20 @@ ] } ], - "source": [ - "from kale.embed.ban import DrugBAN\n", - "\n", - "model = DrugBAN(**cfg)\n", - "print(model)" - ] + "id": "1c8f3acc", + "execution_count": null }, { - "cell_type": "markdown", - "id": "32084f24", - "metadata": { - "id": "32084f24" - }, + "metadata": {}, "source": [ "### Predict\n", "We use the PyKale pipeline API `kale.pipeline.drugban_trainer` to connect dataloaders, encoders and outcoders for model training and evaluation." - ] + ], + "cell_type": "markdown", + "id": "32084f24" }, { - "cell_type": "code", - "execution_count": 19, - "id": "46e2b9b4", - "metadata": { - "id": "46e2b9b4" - }, - "outputs": [], + "metadata": {}, "source": [ "from kale.pipeline.drugban_trainer import DrugbanTrainer\n", "\n", @@ -1026,26 +884,22 @@ " da_random_dim=cfg.DA.RANDOM_DIM,\n", " decoder_in_dim=cfg.DECODER.IN_DIM,\n", ")" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "46e2b9b4", + "execution_count": null }, { - "cell_type": "markdown", - "id": "a48c86b9", - "metadata": { - "id": "a48c86b9" - }, + "metadata": {}, "source": [ - "We want to save the best model during training so we can reuse it later without needing to retrain. PyTorch Lightning’s `ModelCheckpoint` does this by automatically saving the model whenever it achieves a new best validation AUROC score." - ] + "We want to save the best model during training so we can reuse it later without needing to retrain. PyTorch Lightning\u2019s `ModelCheckpoint` does this by automatically saving the model whenever it achieves a new best validation AUROC score." + ], + "cell_type": "markdown", + "id": "a48c86b9" }, { - "cell_type": "code", - "execution_count": 20, - "id": "7754bd38", - "metadata": { - "id": "7754bd38" - }, - "outputs": [], + "metadata": {}, "source": [ "import pytorch_lightning as pl\n", "from pytorch_lightning.callbacks import ModelCheckpoint\n", @@ -1055,33 +909,38 @@ " monitor=\"val_BinaryAUROC\",\n", " mode=\"max\",\n", ")" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "7754bd38", + "execution_count": null }, { - "cell_type": "markdown", - "id": "969beac0", - "metadata": { - "id": "969beac0" - }, + "metadata": {}, "source": [ "We now create the `Trainer`." - ] + ], + "cell_type": "markdown", + "id": "969beac0" }, { + "metadata": {}, + "source": [ + "import torch\n", + "\n", + "trainer = pl.Trainer(\n", + " callbacks=[checkpoint_callback],\n", + " devices=\"auto\",\n", + " accelerator=\"auto\",\n", + " max_epochs=cfg.SOLVER.MAX_EPOCH,\n", + " deterministic=True,\n", + ")" + ], "cell_type": "code", - "execution_count": 21, - "id": "e68e07bc", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "e68e07bc", - "outputId": "08a9b744-3fb5-48c7-863f-dd64afc4dd80" - }, "outputs": [ { - "name": "stderr", "output_type": "stream", + "name": "stderr", "text": [ "INFO:pytorch_lightning.utilities.rank_zero:GPU available: True (cuda), used: True\n", "INFO:pytorch_lightning.utilities.rank_zero:TPU available: False, using: 0 TPU cores\n", @@ -1089,34 +948,19 @@ ] } ], - "source": [ - "import torch\n", - "\n", - "trainer = pl.Trainer(\n", - " callbacks=[checkpoint_callback],\n", - " devices=\"auto\",\n", - " accelerator=\"auto\",\n", - " max_epochs=cfg.SOLVER.MAX_EPOCH,\n", - " deterministic=True,\n", - ")" - ] + "id": "e68e07bc", + "execution_count": null }, { - "cell_type": "markdown", - "id": "1f9a4714", - "metadata": { - "id": "1f9a4714" - }, + "metadata": {}, "source": [ "## Step 3: Model Training" - ] + ], + "cell_type": "markdown", + "id": "1f9a4714" }, { - "cell_type": "markdown", - "id": "b72634ee", - "metadata": { - "id": "b72634ee" - }, + "metadata": {}, "source": [ "### Train\n", "\n", @@ -1131,70 +975,24 @@ "\n", "\n", "This code block takes approximately 5 minutes to complete." - ] + ], + "cell_type": "markdown", + "id": "b72634ee" }, { + "metadata": {}, + "source": [ + "trainer.fit(\n", + " drugban_trainer,\n", + " train_dataloaders=training_generator,\n", + " val_dataloaders=valid_generator,\n", + ")" + ], "cell_type": "code", - "execution_count": 22, - "id": "0624b0c6", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 424, - "referenced_widgets": [ - "bc415b5a5635482eb20a65601866febf", - "98992d70c9d74d3fa794edf0dc9333b1", - "32de4bd89e034c35adc198247950d4bf", - "12c8f4410cc74d4f822c779c724bce94", - "fe6bdb21df9c4415b9899aaa96969502", - "65bb75f11b1f4064b715a166bed1e215", - "a893676474004fb0a24023403f5acf46", - "941d8002127f499c9868abffea2a2429", - "2fc478f35c3e48f2b0c13bd9a73f3dc6", - "79368bfcf5cc4067a59a46c242a77e2d", - "8632ffdd1b5844f183cc28e824cd117e", - "9a8deecaaef543fbb18d0f50b83b5abc", - "b113d7ba3f50417fb93de1118b4e4dec", - "4b0bcd88167b469da36e8433a4d47377", - "1ed13da64943461ab42ab18495a6246b", - "7d4fd3d9c5ed4cc6af20be7384a758ac", - "635b3ed7aa264fa6907187152359a63f", - "9d5ebfa060ac49d6bd7a8c4837b4fc29", - "a80aad59381c42edaa2adb89d781ddd6", - "b71c219972474c2c89ed04fa48ec637d", - "8f16c1ee593d406c88c9bfcf32fdd3df", - "16cdd1f5e14e405490575177c52f4408", - "07d966a71f604f63b00707e6d3a0bfe6", - "37f06572693f4972b468c3997fd0687c", - "a0cec4295099427bb8e97229bd49d620", - "931700f44cb0491ca187cbc58dc67476", - "dd7ad2a05e22470ab00e2118ff994147", - "1aa1b891fe6b447586d3e87ee62768a2", - "8618e343c4f1435ebf099bd70605606b", - "33a0e7cd7f7d49ffa57e962ca2bf0b66", - "f223529cf08b4235b8e74ad1049c5a60", - "304a1cadd7c048a2a34568301fb1c4dd", - "e809f1b951f740ddabc3a3e56d4dd903", - "f60cfca35cab44e5801bbb2b04f9cc6f", - "2f468dcdec6d4c6db229fcab061fd0d8", - "530bec1df42241a9be6a34c561be4a36", - "700bde0d67f64f9d8680c483d232fa5b", - "abf0b363155c4b1fb8de818ae41606eb", - "301141b0856941cba6a0a1f496267c4e", - "b4f64adc9af34262bb35ea42f0f58899", - "3c34cab162424e51b4dcf0008c39c7a0", - "b2e51980648f49f2963ba66745b82b57", - "b9b44b6940884c359ebb60834280b401", - "807b512710c848eabb6816cf8e9ecc75" - ] - }, - "id": "0624b0c6", - "outputId": "8a7fb16d-707f-4a4e-edd8-dec451f5fa5f" - }, "outputs": [ { - "name": "stderr", "output_type": "stream", + "name": "stderr", "text": [ "INFO:pytorch_lightning.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n", "INFO:pytorch_lightning.callbacks.model_summary:\n", @@ -1215,6 +1013,7 @@ ] }, { + "output_type": "display_data", "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "bc415b5a5635482eb20a65601866febf", @@ -1225,10 +1024,10 @@ "Sanity Checking: | | 0/? [00:00┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n", - "┃ Test metric ┃ DataLoader 0 ┃\n", - "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n", - "│ test_BinaryAUROC │ 0.48449382185935974 │\n", - "│ test_BinaryAccuracy │ 0.5071665048599243 │\n", - "│ test_BinaryF1Score │ 0.0933062881231308 │\n", - "│ test_BinaryRecall │ 0.050549451261758804 │\n", - "│ test_BinarySpecificity │ 0.9668141603469849 │\n", - "│ test_accuracy_sklearn │ 0.5038588643074036 │\n", - "│ test_auroc_sklearn │ 0.48449382185935974 │\n", - "│ test_f1_sklearn │ 0.6671618223190308 │\n", - "│ test_loss │ 0.8901852369308472 │\n", - "│ test_optim_threshold │ 0.07649494707584381 │\n", - "│ test_sensitivity │ 0.006637168116867542 │\n", - "│ test_specificity │ 0.997802197933197 │\n", - "└───────────────────────────┴───────────────────────────┘\n", + "
\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n",
+              "\u2503        Test metric        \u2503       DataLoader 0        \u2503\n",
+              "\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n",
+              "\u2502     test_BinaryAUROC      \u2502    0.48449382185935974    \u2502\n",
+              "\u2502    test_BinaryAccuracy    \u2502    0.5071665048599243     \u2502\n",
+              "\u2502    test_BinaryF1Score     \u2502    0.0933062881231308     \u2502\n",
+              "\u2502     test_BinaryRecall     \u2502   0.050549451261758804    \u2502\n",
+              "\u2502  test_BinarySpecificity   \u2502    0.9668141603469849     \u2502\n",
+              "\u2502   test_accuracy_sklearn   \u2502    0.5038588643074036     \u2502\n",
+              "\u2502    test_auroc_sklearn     \u2502    0.48449382185935974    \u2502\n",
+              "\u2502      test_f1_sklearn      \u2502    0.6671618223190308     \u2502\n",
+              "\u2502         test_loss         \u2502    0.8901852369308472     \u2502\n",
+              "\u2502   test_optim_threshold    \u2502    0.07649494707584381    \u2502\n",
+              "\u2502     test_sensitivity      \u2502   0.006637168116867542    \u2502\n",
+              "\u2502     test_specificity      \u2502     0.997802197933197     \u2502\n",
+              "\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n",
               "
\n" ], "text/plain": [ - "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n", - "┃\u001b[1m \u001b[0m\u001b[1m Test metric \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m DataLoader 0 \u001b[0m\u001b[1m \u001b[0m┃\n", - "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n", - "│\u001b[36m \u001b[0m\u001b[36m test_BinaryAUROC \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.48449382185935974 \u001b[0m\u001b[35m \u001b[0m│\n", - "│\u001b[36m \u001b[0m\u001b[36m test_BinaryAccuracy \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.5071665048599243 \u001b[0m\u001b[35m \u001b[0m│\n", - "│\u001b[36m \u001b[0m\u001b[36m test_BinaryF1Score \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.0933062881231308 \u001b[0m\u001b[35m \u001b[0m│\n", - "│\u001b[36m \u001b[0m\u001b[36m test_BinaryRecall \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.050549451261758804 \u001b[0m\u001b[35m \u001b[0m│\n", - "│\u001b[36m \u001b[0m\u001b[36m test_BinarySpecificity \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.9668141603469849 \u001b[0m\u001b[35m \u001b[0m│\n", - "│\u001b[36m \u001b[0m\u001b[36m test_accuracy_sklearn \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.5038588643074036 \u001b[0m\u001b[35m \u001b[0m│\n", - "│\u001b[36m \u001b[0m\u001b[36m test_auroc_sklearn \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.48449382185935974 \u001b[0m\u001b[35m \u001b[0m│\n", - "│\u001b[36m \u001b[0m\u001b[36m test_f1_sklearn \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.6671618223190308 \u001b[0m\u001b[35m \u001b[0m│\n", - "│\u001b[36m \u001b[0m\u001b[36m test_loss \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.8901852369308472 \u001b[0m\u001b[35m \u001b[0m│\n", - "│\u001b[36m \u001b[0m\u001b[36m test_optim_threshold \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.07649494707584381 \u001b[0m\u001b[35m \u001b[0m│\n", - "│\u001b[36m \u001b[0m\u001b[36m test_sensitivity \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.006637168116867542 \u001b[0m\u001b[35m \u001b[0m│\n", - "│\u001b[36m \u001b[0m\u001b[36m test_specificity \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.997802197933197 \u001b[0m\u001b[35m \u001b[0m│\n", - "└───────────────────────────┴───────────────────────────┘\n" + "\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n", + "\u2503\u001b[1m \u001b[0m\u001b[1m Test metric \u001b[0m\u001b[1m \u001b[0m\u2503\u001b[1m \u001b[0m\u001b[1m DataLoader 0 \u001b[0m\u001b[1m \u001b[0m\u2503\n", + "\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n", + "\u2502\u001b[36m \u001b[0m\u001b[36m test_BinaryAUROC \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[35m \u001b[0m\u001b[35m 0.48449382185935974 \u001b[0m\u001b[35m \u001b[0m\u2502\n", + "\u2502\u001b[36m \u001b[0m\u001b[36m test_BinaryAccuracy \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[35m \u001b[0m\u001b[35m 0.5071665048599243 \u001b[0m\u001b[35m \u001b[0m\u2502\n", + "\u2502\u001b[36m \u001b[0m\u001b[36m test_BinaryF1Score \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[35m \u001b[0m\u001b[35m 0.0933062881231308 \u001b[0m\u001b[35m \u001b[0m\u2502\n", + "\u2502\u001b[36m \u001b[0m\u001b[36m test_BinaryRecall \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[35m \u001b[0m\u001b[35m 0.050549451261758804 \u001b[0m\u001b[35m \u001b[0m\u2502\n", + "\u2502\u001b[36m \u001b[0m\u001b[36m test_BinarySpecificity \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[35m \u001b[0m\u001b[35m 0.9668141603469849 \u001b[0m\u001b[35m \u001b[0m\u2502\n", + "\u2502\u001b[36m \u001b[0m\u001b[36m test_accuracy_sklearn \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[35m \u001b[0m\u001b[35m 0.5038588643074036 \u001b[0m\u001b[35m \u001b[0m\u2502\n", + "\u2502\u001b[36m \u001b[0m\u001b[36m test_auroc_sklearn \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[35m \u001b[0m\u001b[35m 0.48449382185935974 \u001b[0m\u001b[35m \u001b[0m\u2502\n", + "\u2502\u001b[36m \u001b[0m\u001b[36m test_f1_sklearn \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[35m \u001b[0m\u001b[35m 0.6671618223190308 \u001b[0m\u001b[35m \u001b[0m\u2502\n", + "\u2502\u001b[36m \u001b[0m\u001b[36m test_loss \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[35m \u001b[0m\u001b[35m 0.8901852369308472 \u001b[0m\u001b[35m \u001b[0m\u2502\n", + "\u2502\u001b[36m \u001b[0m\u001b[36m test_optim_threshold \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[35m \u001b[0m\u001b[35m 0.07649494707584381 \u001b[0m\u001b[35m \u001b[0m\u2502\n", + "\u2502\u001b[36m \u001b[0m\u001b[36m test_sensitivity \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[35m \u001b[0m\u001b[35m 0.006637168116867542 \u001b[0m\u001b[35m \u001b[0m\u2502\n", + "\u2502\u001b[36m \u001b[0m\u001b[36m test_specificity \u001b[0m\u001b[36m \u001b[0m\u2502\u001b[35m \u001b[0m\u001b[35m 0.997802197933197 \u001b[0m\u001b[35m \u001b[0m\u2502\n", + "\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n" ] }, - "metadata": {}, - "output_type": "display_data" + "metadata": {} }, { + "output_type": "execute_result", "data": { "text/plain": [ "[{'test_loss': 0.8901852369308472,\n", @@ -1430,51 +1203,41 @@ " 'test_BinaryAccuracy': 0.5071665048599243}]" ] }, - "execution_count": 23, "metadata": {}, - "output_type": "execute_result" + "execution_count": 23 } ], - "source": [ - "trainer.test(drugban_trainer, dataloaders=test_generator, ckpt_path=\"best\")" - ] + "id": "c1415c02", + "execution_count": null }, { - "cell_type": "markdown", - "id": "bb0a08bec91d2bd9", - "metadata": { - "id": "bb0a08bec91d2bd9" - }, + "metadata": {}, "source": [ "### Performance Comparison\n", "\n", "The earlier example was a simple demonstration. To properly evaluate DrugBAN against baseline models, we train it for 100 epochs across multiple random seeds.\n", "\n", "We provide a checkpoint trained for 100 epochs in the `checkpoint` for your test after the tutorial. We will also use the provided checkpoint for the interpretation section for a better visualization.\n" - ] + ], + "cell_type": "markdown", + "id": "bb0a08bec91d2bd9" }, { - "cell_type": "markdown", - "id": "37dbe9f3", - "metadata": { - "id": "37dbe9f3" - }, + "metadata": {}, "source": [ "The figure below shows the performance of different models on the BioSNAP and BindingDB datasets:\n", "- Left plot: AUROC (Area Under the ROC Curve)\n", - "- Right plot: AUPRC (Area Under the Precision–Recall Curve)\n", + "- Right plot: AUPRC (Area Under the Precision\u2013Recall Curve)\n", "\n", "![](https://media.springernature.com/full/springer-static/image/art%3A10.1038%2Fs42256-022-00605-1/MediaObjects/42256_2022_605_Fig3_HTML.png?as=webp)\n", "\n", "The box plots show the median as the centre lines and the mean as green triangles. The minima and lower percentile represent the worst and second-worst scores. The maxima and upper percentile indicate the best and second-best scores. Supplementary Table 2 provides the data statistics of the BindingDB and BioSNAP datasets." - ] + ], + "cell_type": "markdown", + "id": "37dbe9f3" }, { - "cell_type": "markdown", - "id": "02e3c73e", - "metadata": { - "id": "02e3c73e" - }, + "metadata": {}, "source": [ "## Step 5: Interpretation\n", "\n", @@ -1483,27 +1246,21 @@ "2) generate molecule images with attention highlights.\n", "\n", "This helps us understand which parts of the drug contribute to the interaction with the target protein." - ] + ], + "cell_type": "markdown", + "id": "02e3c73e" }, { - "cell_type": "markdown", - "id": "4a56f260141b7368", - "metadata": { - "id": "4a56f260141b7368" - }, + "metadata": {}, "source": [ "### Extracting Attention Weights\n", "First, we need to load the test dataset and create a DataLoader for it. This will allow us to process the test samples in batches. We define functions to create the test dataset and DataLoader." - ] + ], + "cell_type": "markdown", + "id": "4a56f260141b7368" }, { - "cell_type": "code", - "execution_count": 24, - "id": "2c67553408592b2", - "metadata": { - "id": "2c67553408592b2" - }, - "outputs": [], + "metadata": {}, "source": [ "def get_test_dataset(dataFolder):\n", " df_test_target = pd.read_csv(dataFolder)\n", @@ -1521,48 +1278,40 @@ " drop_last=True,\n", " )\n", " return test_dataloader" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "2c67553408592b2", + "execution_count": null }, { - "cell_type": "markdown", - "id": "ecdab66ee05da10c", - "metadata": { - "id": "ecdab66ee05da10c" - }, + "metadata": {}, "source": [ - "We load a small subset of samples for testing from the provided `.csv` file. You can create your own `.csv` file with the same format to test your drug–protein pairs." - ] + "We load a small subset of samples for testing from the provided `.csv` file. You can create your own `.csv` file with the same format to test your drug\u2013protein pairs." + ], + "cell_type": "markdown", + "id": "ecdab66ee05da10c" }, { - "cell_type": "code", - "execution_count": 25, - "id": "7ef1867541d2577a", - "metadata": { - "id": "7ef1867541d2577a" - }, - "outputs": [], + "metadata": {}, "source": [ "test_dataFolder = \"/content/drug-target-interaction/data/drug-target-interaction/bindingdb/interpretation_samples.csv\"" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "7ef1867541d2577a", + "execution_count": null }, { - "cell_type": "markdown", - "id": "7fec5dc00a7b4aa4", - "metadata": { - "id": "7fec5dc00a7b4aa4" - }, + "metadata": {}, "source": [ "We then build the test dataset and DataLoader using the functions defined above. The `batchsize` is set to 1 to ensure we process one sample at a time for attention visualization later." - ] + ], + "cell_type": "markdown", + "id": "7fec5dc00a7b4aa4" }, { - "cell_type": "code", - "execution_count": 26, - "id": "c99a558c96a1ffd", - "metadata": { - "id": "c99a558c96a1ffd" - }, - "outputs": [], + "metadata": {}, "source": [ "test_dataset = get_test_dataset(test_dataFolder)\n", "test_dataloader = get_test_dataloader(\n", @@ -1571,26 +1320,22 @@ " num_workers=cfg.SOLVER.NUM_WORKERS,\n", " collate_fn=graph_collate_func,\n", ")" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "c99a558c96a1ffd", + "execution_count": null }, { - "cell_type": "markdown", - "id": "e1ff543d132abc42", - "metadata": { - "id": "e1ff543d132abc42" - }, + "metadata": {}, "source": [ "Then, we use the following function to load the trained model with the PyKale API." - ] + ], + "cell_type": "markdown", + "id": "e1ff543d132abc42" }, { - "cell_type": "code", - "execution_count": 27, - "id": "3b7f12b12b139799", - "metadata": { - "id": "3b7f12b12b139799" - }, - "outputs": [], + "metadata": {}, "source": [ "def get_model_from_ckpt(ckpt_path, config):\n", " return DrugbanTrainer.load_from_checkpoint(\n", @@ -1611,31 +1356,31 @@ " da_random_dim=config.DA.RANDOM_DIM,\n", " decoder_in_dim=config.DECODER.IN_DIM,\n", " )" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "3b7f12b12b139799", + "execution_count": null }, { - "cell_type": "markdown", - "id": "c0678dddcdf076fc", - "metadata": { - "id": "c0678dddcdf076fc" - }, + "metadata": {}, "source": [ "Once the model and test data are prepared, we extract attention maps from the trained model. We set the directory to the provided checkpoint file, load the trained model, and set it to evaluation mode." - ] + ], + "cell_type": "markdown", + "id": "c0678dddcdf076fc" }, { + "metadata": {}, + "source": [ + "checkpoint_path = \"/content/drug-target-interaction/checkpoint/best.ckpt\"\n", + "model = get_model_from_ckpt(checkpoint_path, cfg)\n", + "model.model.eval()" + ], "cell_type": "code", - "execution_count": 28, - "id": "d2a8931099b73c01", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "d2a8931099b73c01", - "outputId": "1dbad556-912d-44f3-88ad-059cf7876a36" - }, "outputs": [ { + "output_type": "execute_result", "data": { "text/plain": [ "DrugBAN(\n", @@ -1684,57 +1429,23 @@ ")" ] }, - "execution_count": 28, "metadata": {}, - "output_type": "execute_result" + "execution_count": 28 } ], - "source": [ - "checkpoint_path = \"/content/drug-target-interaction/checkpoint/best.ckpt\"\n", - "model = get_model_from_ckpt(checkpoint_path, cfg)\n", - "model.model.eval()" - ] + "id": "d2a8931099b73c01", + "execution_count": null }, { - "cell_type": "markdown", - "id": "159d3fa67b29c9e9", - "metadata": { - "id": "159d3fa67b29c9e9" - }, + "metadata": {}, "source": [ "We then iterate through the test DataLoader, passing each batch of drug and protein pairs to the model. The model's forward method returns the attention weights. After processing all batches, we concatenate the attention tensors into a single tensor." - ] + ], + "cell_type": "markdown", + "id": "159d3fa67b29c9e9" }, { - "cell_type": "code", - "execution_count": 29, - "id": "781a7762c36c72be", - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "781a7762c36c72be", - "outputId": "fade7fb6-bea8-438a-9688-8cffd8dc9c47" - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "100%|██████████| 6/6 [00:00<00:00, 65.52it/s]\n" - ] - }, - { - "data": { - "text/plain": [ - "torch.Size([6, 2, 290, 1185])" - ] - }, - "execution_count": 29, - "metadata": {}, - "output_type": "execute_result" - } - ], + "metadata": {}, "source": [ "from tqdm import tqdm\n", "\n", @@ -1755,44 +1466,56 @@ "torch.save(all_attentions, \"attention_maps.pt\")\n", "\n", "all_attentions.shape" - ] + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 6/6 [00:00<00:00, 65.52it/s]\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "torch.Size([6, 2, 290, 1185])" + ] + }, + "metadata": {}, + "execution_count": 29 + } + ], + "id": "781a7762c36c72be", + "execution_count": null }, { - "cell_type": "markdown", - "id": "78dc763b6c0eef0", - "metadata": { - "id": "78dc763b6c0eef0" - }, + "metadata": {}, "source": [ "The attention has shape [B, H, V, Q] (Number of drug-target pairs, Heads of attentions, Drug tokens, Protein tokens)." - ] + ], + "cell_type": "markdown", + "id": "78dc763b6c0eef0" }, { - "cell_type": "markdown", - "id": "8f72ea4d93f640cb", - "metadata": { - "id": "8f72ea4d93f640cb" - }, + "metadata": {}, "source": [ "### Visualize Attention Maps and Molecule Images" - ] + ], + "cell_type": "markdown", + "id": "8f72ea4d93f640cb" }, { - "cell_type": "markdown", - "id": "383c342a7c31d7ae", - "metadata": { - "id": "383c342a7c31d7ae" - }, + "metadata": {}, "source": [ "Once attention maps are saved, run the visualization script:" - ] + ], + "cell_type": "markdown", + "id": "383c342a7c31d7ae" }, { - "cell_type": "markdown", - "id": "d8a746169def8da5", - "metadata": { - "id": "d8a746169def8da5" - }, + "metadata": {}, "source": [ "This script will:\n", "\n", @@ -1800,107 +1523,89 @@ "\n", "2) Plot:\n", "\n", - " a) A heatmap of attention over drug–protein tokens.\n", + " a) A heatmap of attention over drug\u2013protein tokens.\n", "\n", " b) Molecular structures with atoms highlighted by attention values.\n", "\n", "The output images are saved in the `visualization` directory. You can also modify the `data_file` to use your own input in the same format as `target_test.csv`.\n", "\n" - ] + ], + "cell_type": "markdown", + "id": "d8a746169def8da5" }, { - "cell_type": "markdown", - "id": "aac54bfc67ce32eb", - "metadata": { - "id": "aac54bfc67ce32eb" - }, + "metadata": {}, "source": [ "We first import the necessary PyKale APIs and set the output directory." - ] + ], + "cell_type": "markdown", + "id": "aac54bfc67ce32eb" }, { - "cell_type": "code", - "execution_count": 30, - "id": "d3c1d2e4cab69107", - "metadata": { - "id": "d3c1d2e4cab69107" - }, - "outputs": [], + "metadata": {}, "source": [ "from kale.interpret.visualize import draw_attention_map, draw_mol_with_attention\n", "from kale.prepdata.tensor_reshape import normalize_tensor\n", "\n", "out_dir = \"./visualization\"\n", "os.makedirs(out_dir, exist_ok=True)" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "d3c1d2e4cab69107", + "execution_count": null }, { - "cell_type": "markdown", - "id": "126b62034111d92a", - "metadata": { - "id": "126b62034111d92a" - }, + "metadata": {}, "source": [ "We then load the attention maps, data, and SMILES strings from the test dataset." - ] + ], + "cell_type": "markdown", + "id": "126b62034111d92a" }, { - "cell_type": "code", - "execution_count": 31, - "id": "7f70a6810c1c5e60", - "metadata": { - "id": "7f70a6810c1c5e60" - }, - "outputs": [], + "metadata": {}, "source": [ "attention = torch.load(\"attention_maps.pt\", map_location=\"cpu\")\n", "data_df = pd.read_csv(test_dataFolder)\n", "smiles = data_df[\"SMILES\"]\n", "proteins = data_df[\"Protein\"]" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "7f70a6810c1c5e60", + "execution_count": null }, { - "cell_type": "markdown", - "id": "d1a009bbb9f4a0f9", - "metadata": { - "id": "d1a009bbb9f4a0f9" - }, + "metadata": {}, "source": [ "We select the first sample from the attention maps and corresponding SMILES and protein sequence for visualization." - ] + ], + "cell_type": "markdown", + "id": "d1a009bbb9f4a0f9" }, { - "cell_type": "code", - "execution_count": 32, - "id": "e808c255fe862925", - "metadata": { - "id": "e808c255fe862925" - }, - "outputs": [], + "metadata": {}, "source": [ "index = 0\n", "att_path = os.path.join(out_dir, f\"att_map_{index}.png\")\n", "mol_path = os.path.join(out_dir, f\"mol_{index}.svg\")" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "e808c255fe862925", + "execution_count": null }, { - "cell_type": "markdown", - "id": "438e6aa218e6b51d", - "metadata": { - "id": "438e6aa218e6b51d" - }, + "metadata": {}, "source": [ "We crop the attention map to the actual lengths of the drug and protein sequences. This is important because the attention map may include padding tokens." - ] + ], + "cell_type": "markdown", + "id": "438e6aa218e6b51d" }, { - "cell_type": "code", - "execution_count": 33, - "id": "af15baa1c8caabc0", - "metadata": { - "id": "af15baa1c8caabc0" - }, - "outputs": [], + "metadata": {}, "source": [ "from rdkit import Chem\n", "\n", @@ -1919,26 +1624,22 @@ "\n", "# Normalize\n", "att = normalize_tensor(att)" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "af15baa1c8caabc0", + "execution_count": null }, { - "cell_type": "markdown", - "id": "60a4ce71146a721e", - "metadata": { - "id": "60a4ce71146a721e" - }, + "metadata": {}, "source": [ "Finally, we save the attention map and the molecule image with attention highlights." - ] + ], + "cell_type": "markdown", + "id": "60a4ce71146a721e" }, { - "cell_type": "code", - "execution_count": 34, - "id": "403f77ada0ecc446", - "metadata": { - "id": "403f77ada0ecc446" - }, - "outputs": [], + "metadata": {}, "source": [ "draw_attention_map(\n", " att,\n", @@ -1947,47 +1648,50 @@ " xlabel=\"Drug Tokens\",\n", " ylabel=\"Protein Tokens\",\n", ")" - ] - }, - { + ], "cell_type": "code", - "execution_count": 35, - "id": "b1003372361a66d6", - "metadata": { - "id": "b1003372361a66d6" - }, "outputs": [], + "id": "403f77ada0ecc446", + "execution_count": null + }, + { + "metadata": {}, "source": [ "draw_mol_with_attention(att.mean(dim=1), smile, mol_path)" - ] + ], + "cell_type": "code", + "outputs": [], + "id": "b1003372361a66d6", + "execution_count": null }, { - "cell_type": "code", - "execution_count": 36, - "id": "4mHWCbJmGMgG", "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 921 - }, - "id": "4mHWCbJmGMgG", - "outputId": "5b3f1960-85a5-4ca3-bbbb-f48f6bc67401", "tags": [ "hide-input" ] }, + "source": [ + "from IPython.display import Image, SVG\n", + "\n", + "attention_plot = Image(att_path)\n", + "molecular_img = SVG(mol_path)\n", + "display(attention_plot)\n", + "display(molecular_img)" + ], + "cell_type": "code", "outputs": [ { + "output_type": "display_data", "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA+gAAAJYCAYAAADxHswlAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAA0fhJREFUeJzs3Xt8FNX9//H37G6yCRCiAcJVBKVeUaR4KWIxKAXRolRbr1XEW7VJLVCtpq2itTbaVottKfy0CrZKpVVBqxaLXOUrqKBRaRVFUbwQEJVAAtnsZX5/zMyS2SSws9klm+T15LGPZWdnPvM5Z86c2ZOZnTVM0zQFAAAAAABala+1EwAAAAAAAAzQAQAAAADICgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQAAAADIAgzQAQDYD5YtWybDMLRs2bLWTgUAAGQpBugA0AbMmTNHhmHEH3l5eerTp4/Gjh2rP/zhD9q5c2drp9isUCikm266SX369FF+fr5OOukkLVq0yHOc888/X4Zh6Kabbmry/eeee0633XZbo+m7du3Sbbfdtt8Gxn/+8581Z86c/bIuAADQvhimaZqtnQQAYO/mzJmjSZMm6Ze//KUGDhyocDisqqoqLVu2TIsWLVL//v319NNP69hjj23tVBu56KKL9Pjjj2vy5Mn62te+pjlz5ujVV1/V0qVLdcoppyQVY8eOHerZs6d69eqlaDSqjz76SIZhuOYpKyvTjBkzlHhY27Ztm3r06KFp06Y1OYBPt8GDB6t79+6N/iAQi8VUX1+v3Nxc+Xz8fRwAADQWaO0EAADJGzdunI4//vj46/Lyci1ZskTf/va3dfbZZ+vtt99Wfn5+s8vX1taqc+fO+yNVSdIrr7yixx57TL/97W91ww03SJIuu+wyDR48WD/96U/10ksvJRXniSeeUDQa1UMPPaTTTjtNK1as0KmnnprJ1NPO5/MpLy+vtdMAAABZjD/hA0Abd9ppp+mWW27RRx99pEceeSQ+/fLLL1eXLl30/vvv68wzz1RBQYEuueQSSdKAAQN0+eWXN4pVUlKikpIS17SPPvpIZ599tjp37qzi4mJNmTJFzz//fFLfp3788cfl9/t1zTXXxKfl5eXpyiuv1KpVq/Txxx8nVcZHH31U3/rWtzRq1CgdeeSRevTRR13vX3755ZoxY4Ykub4K8OGHH6pHjx6SpNtvvz0+veGZ9HfeeUff/e53VVRUpLy8PB1//PF6+umnXfGdrxj83//9n6ZOnaoePXqoc+fO+s53vqPPP/88Pt+AAQP03//+V8uXL4+vy6nP5r6D/s9//lPDhg1Tfn6+unfvru9///v69NNPG5WvS5cu+vTTTzVhwgR16dJFPXr00A033KBoNJpUHQIAgOzHAB0A2oFLL71UkvSf//zHNT0SiWjs2LEqLi7W7373O5133nme4tbW1uq0007TCy+8oOuvv14///nP9dJLLzX7PfBEr7/+ug477DB17drVNf3EE0+UJFVWVu4zxmeffaalS5fqoosukrTnkvn6+vr4PD/4wQ/0rW99S5L0t7/9Lf7o0aOHZs6cKUn6zne+E59+7rnnSpL++9//6hvf+Ibefvtt3XzzzbrnnnvUuXNnTZgwQfPnz2+Uy49+9CO98cYbmjZtmq677jr961//UllZWfz96dOnq1+/fjriiCPi6/r5z3/ebNnmzJmj888/X36/XxUVFbr66qv15JNP6pRTTtH27dtd80ajUY0dO1bdunXT7373O5166qm65557dP/99++zDgEAQNvAJe4A0A7069dPhYWFev/9913TQ6GQvve976mioiKluP/v//0/ffDBB1qwYIHOOeccSdZgeOjQoUktv3nzZvXu3bvRdGfaZ599ts8Yf//73xUMBuPrv/DCC3Xrrbfqueee04QJEyRJw4cP12GHHaZFixbp+9//vmv57373u7ruuut07LHHNnrvxz/+sfr3769XX31VwWBQkvTDH/5Qp5xyim666SZ95zvfcc3frVs3/ec//4l//z0Wi+kPf/iDqqurVVhYqAkTJugXv/hF/Ez43oTDYd10000aPHiwVqxYEb/8/ZRTTtG3v/1t/f73v9ftt98en7+urk4XXHCBbrnlFknStddeq69//et68MEHdd111+2zHgEAQPbjDDoAtBNdunRp8m7uLRm8LVy4UH379tXZZ58dn5aXl6err746qeV3794dH/g25AxGd+/evc8Yjz76qM466ywVFBRIkr72ta9p2LBhjS5z9+rLL7/UkiVLdP7552vnzp3atm2btm3bpi+++EJjx47Ve++91+hS82uuucZ1c7pvfvOb8ZvWebVmzRpt3bpVP/zhD13fTT/rrLN0xBFH6Nlnn220zLXXXut6/c1vflMffPCB53UDAIDsxBl0AGgnampqVFxc7JoWCATUr1+/lGN+9NFHOvTQQxvdMX3QoEFJLZ+fn69QKNRoel1dXfz9vXn77bf1+uuv67LLLtOGDRvi00tKSjRjxgzt2LGj0eXzydqwYYNM09Qtt9wSPyudaOvWrerbt2/8df/+/V3vH3jggZKkr776yvP6nUH94Ycf3ui9I444QitXrnRNy8vLi3+fvuH6U1k3AADITgzQAaAd+OSTT1RdXd1o4BwMBpv8Sa/EAbcjGo3K7/enLa/evXs3OgstWZe+S1KfPn32urxz07spU6ZoypQpjd5/4oknNGnSpJRyi8VikqQbbrhBY8eObXKexPpsrm72xy+WpnO7AACA7MQAHQDagb/97W+S1OxAM9GBBx7Y6CZkknVW95BDDom/Pvjgg/W///1Ppmm6BvUNz2bvzXHHHaelS5c2OtP98ssvx99vjmmamjt3rkaNGqUf/vCHjd6/44479Oijj8YH6M390aG56U45c3JyNHr06KTKk4zm1pfo4IMPliStX79ep512muu99evXx98HAAAdB99BB4A2bsmSJbrjjjs0cODA+M+o7cuhhx6q1atXu+6E/swzzzT62bOxY8fq008/df3sWF1dnR544IGk1vPd735X0WjUdafxUCik2bNn66STTtJBBx3U7LL/93//pw8//FCTJk3Sd7/73UaPCy64QEuXLo3faM75fffEPzx06tSpyenFxcUqKSnR//t//y9+Rr+hhj+f5kXnzp2b/ONHouOPP17FxcWaNWuW62sA//73v/X222/rrLPOSmn9AACg7eIMOgC0If/+97/1zjvvKBKJaMuWLVqyZIkWLVqkgw8+WE8//bTrZmN7c9VVV+nxxx/XGWecofPPP1/vv/++HnnkER166KGu+X7wgx/oT3/6ky666CL9+Mc/Vu/evfXoo4/G17Ovs8UnnXSSvve976m8vFxbt27VoEGD9PDDD+vDDz/Ugw8+uNdlH330Ufn9/mYHqmeffbZ+/vOf67HHHtPUqVM1bNgwSdL111+vsWPHyu/368ILL1R+fr6OOuoozZs3T4cddpiKioo0ePBgDR48WDNmzNApp5yiY445RldffbUOOeQQbdmyRatWrdInn3yiN954I6n6bGjYsGGaOXOmfvWrX2nQoEEqLi5udIZcss7c33333Zo0aZJOPfVUXXTRRdqyZYvuu+8+DRgwoMlL+gEAQDtnAgCy3uzZs01J8Udubq7Zq1cv81vf+pZ53333mTt27Gi0zMSJE83OnTs3G/Oee+4x+/btawaDQXPEiBHmmjVrzFNPPdU89dRTXfN98MEH5llnnWXm5+ebPXr0MH/yk5+YTzzxhCnJXL169T5z3717t3nDDTeYvXr1MoPBoHnCCSeYCxcu3Osy9fX1Zrdu3cxvfvObe51v4MCB5tChQ03TNM1IJGL+6Ec/Mnv06GEahmE2PMS99NJL5rBhw8zc3FxTkjlt2rT4e++//7552WWXmb169TJzcnLMvn37mt/+9rfNxx9/PD6PU/+vvvqqa/1Lly41JZlLly6NT6uqqjLPOusss6CgwJQUr8+m5jVN05w3b545dOhQMxgMmkVFReYll1xifvLJJ655mtuW06ZNMzmUAwDQfhimuR/ubAMAaFemT5+uKVOm6JNPPnHd5RwAAACpY4AOANir3bt3u34Ora6uTkOHDlU0GtW7777bipkBAAC0L3wHHQCwV+eee6769++v4447TtXV1XrkkUf0zjvv6NFHH23t1AAAANoVBugAgL0aO3as/vKXv+jRRx9VNBrVUUcdpccee0wXXHBBa6cGAADQrvAzawCAvZo8ebLWrVunmpoa7d69W2vXrmVwDgBAK1uxYoXGjx+vPn36yDAMLViwYJ/LLFu2TF//+tcVDAY1aNAgzZkzJ+N5whsG6AAAAADQxtTW1mrIkCGaMWNGUvNv3LhRZ511lkaNGqXKykpNnjxZV111lZ5//vkMZwovuEkcAAAAALRhhmFo/vz5mjBhQrPz3HTTTXr22We1bt26+LQLL7xQ27dv18KFC/dDlkgGZ9ABAAAAIAuEQiHt2LHD9QiFQmmJvWrVKo0ePdo1bezYsVq1alVa4iM9OuxN4o654feSJOPUryRJAw/8UpL031WHeI7V43XrIoS6A6y/d1Qfbr321XuLE+kcs/5jWE9d3/VLkmpO2G3F+zQv6VjhoogkKeeLHElSLGjldMDbVvBwJ2+51Rxi5XbxyP+TJP3zX9+04uZ6vwAj92s7JEmB5YVWDCtF1R9gPUeDycXs+r5Vlq9OtCo6//1cSVLk2FpJkvlh8oV0cuj0mRXTP9JqDzu2WzFyNyZf95LU5fhtkqQvNh1oLd/d2oaB17t4iiNJuw62tqWC1jboss4qZ7jAmhzNS66+Ip2s+Q55MixJ+vhbQSvOAVErt53J/73OqS+jV50kyfeh9RNc4W5WroHt/qRjSVKXTVa9j7ryZUnSs8+dZK/He/uK5lvLdD/0C+u5s9UeNvzfAE8xDXu2aB/roGhG9tSPr9rqOn27k6szJ9bJp1l/sV7+ztckSf5ca5saH3trX/46q776La2LT6v6sZVn6N2unmJF7O1v5NvtzO5/zGprIwdqkyujedCeXHp1q5YkffpZkSSpW7G1z1ev65ZULCNqJdH3hE8lSR9+2l2SdMCrVput6e+9XQz6m7VPr7/a2idVZPUbvqqgpzhOvxMY/7kkqfMfDrByPNfuO3Yk3/ajXaztX3ywldv2tT0kSfXd7X1yR3J17wtb667vYW3DW099SpK0K2b1FX94cnzSOTk6f2I9R+ymWTz+Y0nSR2v6eYrT9Wi7bO9ZbaHzx1auO46wj1FfJV9fwW3WsrUHW/VzwP/sY+RIax/XR8n1+abdxjsdZrVTn8/aDqF6q81H3vfeT3d7y3rOqbFi/fUP90iSxj7yU2ud/uTabK/V1vK1Pa2yVX/TOnZ0P7BGkvTlW92TzinSw+rr87pafUP4A6tcTn9bc3CSfaH90aTHkK2SpC1fWH1MrN7K0b8tJ+mcHEX/tZ6rD7FyGXr6eknSy6/ZfaPdx+2Lae8ih8222tlnp+/pY2r7eusnovbxK6fKKs9BL1j1tuunVjvZ9mYPT/GcMnxj7J4zla982l+SFN5Q4ClW36VWbptPtnJ74PszJUnX/O06SXuO7/uSe8hOSVL9xj3rz99s5dnlM2tD7+yfXL8TtneT+mIrtz+e/jdJ0pTHJ1lvGHvPacONU5NaTzaKVR2W0fgVsy7W7bff7po2bdo03XbbbS2OXVVVpZ49e7qm9ezZUzt27Gj0k6poPR12gA4AAAAA2aS8vFxTp7r/gBEMevtjMtq2NnuJ+4wZMzRgwADl5eXppJNO0iuvvNLaKQEAAABox2IZ/hcMBtW1a1fXI10D9F69emnLli2uaVu2bFHXrl05e55F2uQAfd68eZo6daqmTZum1157TUOGDNHYsWO1devW1k4NAAAAALLO8OHDtXjxYte0RYsWafjw4a2UEZrSJgfo9957r66++mpNmjRJRx11lGbNmqVOnTrpoYceau3UAAAAALRTUTOW0YcXNTU1qqysVGVlpSTrZ9QqKyu1adMmSdbl8pdddll8/muvvVYffPCBfvrTn+qdd97Rn//8Z/3jH//QlClT0lY/aLk2N0Cvr6/X2rVrXXcg9Pl8Gj16NHcgBAAAANAhrFmzRkOHDtXQoUMlSVOnTtXQoUN16623SpI2b94cH6xL0sCBA/Xss89q0aJFGjJkiO655x795S9/0dixY1slfzStzd0kbtu2bYpGo03egfCdd95ppawAAAAAtHcxef8lkUwpKSmRaTafz5w5c5pc5vXXX89gVmipNjdAT0UoFGr0+4GxSES+QIcoPgAAAACgDWhzl7h3795dfr+/yTsQ9urVq8llKioqVFhY6Hp8/soL+yNdAAAAAO1Epu/iDrS5AXpubq6GDRvmugNhLBbT4sWLm70DYXl5uaqrq12PHieObnJeAAAAAABaQ5u8xnvq1KmaOHGijj/+eJ144omaPn26amtrNWnSpCbnDwaDjX4/kMvbAQAAAHgR3ct3voF0aJOj1AsuuECff/65br31VlVVVem4447TwoULG904DgAAAACAtqJNDtAlqaysTGVlZa2dBgAAAIAOIpvu4o72qc19Bx0AAAAAgPaozZ5BBwAAAID9KcoZdGQYZ9ABAAAAAMgCHfYMuq/eevb7rd8b7JpTZ00wDc+xcndGJUn1na2/dxgRpRbLmd/+w5w/ZOca8P6biEbin16cmPXWf6K5XnOznopzd1jxnZRSqC+/zwoW2GW9Dnd2ryPpOCF7gbDPfm299OVaGyDkIZZTHl/EnaMvYLreT1Z+Ttj6j1PvPitAvG14kVgvZuJzstvAWiBnh1Mz9i8b+E2PcSTZdzD12fuPsy3ri1P7q3JOrbVc3+BXkiQjar8R8N6+lHB31U4Ba1s429BIMkUjaq07kGslE4402Kns95KOZa+7R7DGem23L3/Aih3zuB857Siwsz4+LS/XKmcohX3SFdvOzWsYpyyS1CXXbmN2jAL7dXWyQRP2GyeOv85533sZjS+qredokSQpFkutngK7reTy7fr21cfslPzeg0WsHHID9gZ1UvLavzq7sL0vD8j9XJK0M5bvjudBvH817Bz9UW85Jabo9Kn19vLJ7jwNOP2zYW87X9SVoswkczPsCjPsHJz+Psduw6l00067yKmxYgwIFNhJ2TMkmVug1lo+UGd/nrBz65Rj7etfeqh/0+6znM85ETsXX/wzStKhrOXs+orZx9x4e02h2w/UOcdWK0iB8xnM5wRLcls6bcDZv2Pd9uQbtmKYgWQ7atMV07/bqqjOdv+1Lbkoe8LZ9dw5sOfTiNPmvO5Hudut7e+L5EiSevhr7TjecnLaeH2D5ZzjU26N8zkluXN38eO0Haubv8YVz/S17FiUzfgOOjKNM+gAAAAAAGSBNjlAX7FihcaPH68+ffrIMAwtWLCgtVMCAAAA0M5FTTOjD6BNDtBra2s1ZMgQzZgxo7VTAQAAAAAgLdrkd9DHjRuncePGtXYaAAAAADqQFG7rAXjSJgfoAAAAALC/8TNryLQ2eYk7AAAAAADtTYc4gx4KhRQKuX90KxaNyOfvEMUHAAAAkAZRTqAjwzrEGfSKigoVFha6HlvXvNDaaQEAAAAAENchBujl5eWqrq52PYqPH93aaQEAAABoQ2IZfgBt8hrvmpoabdiwIf5648aNqqysVFFRkfr3799o/mAwqGAw6JrG5e0AAAAAgGzSJkepa9as0ahRo+Kvp06dKkmaOHGi5syZ00pZAQAAAGjPojJaOwW0c21ygF5SUiLT5A4NAAAAAID2o00O0AEAAABgf4txjhAZ1iFuEgcAAAAAQLbjDDoAAAAAJIHvoCPTOuwA3R+2nmvrciVJn9QWtjhmLMd6DuyydtxocC8zN5XTbms5I+re8WMR60IHT5c7RNwxjKj1bDpBPF474a+z4r1V08/bgk2orbEqplu9dY1QuIudq8f+zvRbC+R8YTVjI2KHMbxfe2TY9eUsuX17J+u1aU/3mNuueqtd+eqsig7VWY0jL4VrVpwYMTsJp52Zfo9x7DJG86368lqmhpz2FN5lJdPlC+v1roNTi5dTa9X8hl3FkiRfvTXd6z4k7Wmr274ssHJ1yumxWfjsPsJ02lPdngp39vGkt4Ed4uNdB0qSYiFrwWiO9YMqXjeF366fcNc9FRSORD1GcRa0yxK1tqWZb8VJ7If2JVK/pzK+2m3vP3bsUNTbocap8k+/OsD6j1P3LflM1LWLJClQawWpz8lJKYzP7meqd+dJkjoVW3GMsPd+x2lHn++wcovmWjH8u7x1FGbC7K/tHiBJ2uXsQCnUm+mzj0d2eTd9daAdy1s5v/qysytHn91MjYj3ztCJ4RwrneI5/VDSrczu/HbY/XxuvrWzRyNWO0ulmUWDdv8ctJJctNvOxgmWZL1F7YOE0z9H7f1qRyjPc07OsTHU1V0zqfSrkrT5c+tzkv9z6/gW7epsTO+xojnO5x3r9Ybq7tZ/PO5GAbstOPt3NLfBe7us53DXJIOFrbo37N+5qj/QClYfzm1uib3y11nP//uqV3zarh3WdvT64dsMWOV0jo1/336i9R+PdV8Xcj5A7JnmfKZwpiX2J81x+kLnM8qT20+wUkr8vAnAsw47QAcAAAAALziDjkxrk3/fqqio0AknnKCCggIVFxdrwoQJWr9+fWunBQAAAABAytrkAH358uUqLS3V6tWrtWjRIoXDYY0ZM0a1tbWtnRoAAACAdipmGhl9AG3yEveFCxe6Xs+ZM0fFxcVau3atRo4c2UpZAQAAAACQujY5QE9UXV0tSSoqKmrlTAAAAAC0V3wHHZnWJi9xbygWi2ny5MkaMWKEBg8e3NrpAAAAAACQkjZ/Br20tFTr1q3TypUrm50nFAopFAq5psWiEfn8bb74AAAAAPaTaNs/v4ks16ZbWFlZmZ555hktXbpU/fo1//vcFRUVKiwsdD22vPbCfswUAAAAAIC9a5MDdNM0VVZWpvnz52vJkiUaOHDgXucvLy9XdXW169Hz66P3U7YAAAAA2gPu4o5Ma5PXeJeWlmru3Ll66qmnVFBQoKqqKklSYWGh8vPzG80fDAYVDAZd07i8HQAAAACQTdrkKHXmzJmSpJKSEtf02bNn6/LLL9//CQEAAABo97iLOzKtTQ7QTdNs7RQAAAAAAEirNjlABwAAAID9LWq2yVt4oQ1hgA4AAAAASYi1zXtsow2hhQEAAAAAkAU67Bn0wG7re+xmws8ZmD7v32/374pay9p3hs/ZYU2P9PQWK7DbyiVQa8ett5b358QkSbEU7klh+u1y+p11WK8jed6C+eqt+aN2ffnqrenRPO/1FavNkSTl1FrL1h1oxfT6yxJOWfwhO6eINb2w8y5J0i6jMOlYfrs8Tj3FItZ/AnlhSZIR85Zbrt9qE4Fddr3Z7cqIeosjST67fLG8mDtHZ+81ktwGdhn8tVZhjVgnO9mYtziSDLsxmlHrudMWq2Bf2W3V68bs/LG1zUJ2oZz25SUnR7ytfpVrpVLk5CxPufmsTa/cPCuZ+uieX4gI7Laew52Ty8lZd8Bn1ZNh11swaK2k3mM5/XV2GZr4E6vnPsyuD8Mur7rEXHFMI7n6MvyN12tErAS7BuskSZ8nXU5rnaHdVl+heiuOETUbvu3NjhpJUs7OYklSNJjaTX6C1dY23BG22qq/0M4t4rGTkOQL232XXS9Ru+6DW/12jkkGcppDrpXbAX5rf+rkc/Z1z6nJF7FyCoSs5+q6HNe6kmXutJYzg1YSznEu/r6HeE7/6ezjMbsvNCN2kCTbV3ydu6xtGLX7rfjtbVJoGj77eK2Y9fxp2Op4nCthky2ns61ya9xlMZyyeegrcqvt/ShmJeGz6yl+dW6SOTlrNL+wGmTel9aCtckfYhvxRd3PEScpj6eNAjX2f2qtTjmat+e93J3Wc7jA2wZ1tkE0z0om6Lc+XHj+jBKynr/a3eDXhUJ2o/XY5+dssQqTu8M6br9Z3dc9g8du37V6+//BL63+whdNcmhgH6f9tVY9fbrbbhBOPaVw/G4ruEkcMo0z6AAAAAAAZIE2OUCfOXOmjj32WHXt2lVdu3bV8OHD9e9//7u10wIAAADQjkVNX0YfQJtsBf369dNdd92ltWvXas2aNTrttNN0zjnn6L///W9rpwYAAAAAQEra5HfQx48f73p95513aubMmVq9erWOPvroVsoKAAAAQHsW4zvoyLA2OUBvKBqN6p///Kdqa2s1fPjw1k4HAAAAAICUtNkB+ltvvaXhw4errq5OXbp00fz583XUUUe1dloAAAAA2qlo2/yGMNqQNjtAP/zww1VZWanq6mo9/vjjmjhxopYvX97kID0UCikUCrmmxaIR+fxttvgAAAAAgHamzf4JKDc3V4MGDdKwYcNUUVGhIUOG6L777mty3oqKChUWFroem99avJ8zBgAAANCWcRd3ZFq7aQWxWKzRWXJHeXm5qqurXY/ex5y+nzMEAAAAAKB5bfIa7/Lyco0bN079+/fXzp07NXfuXC1btkzPP/98k/MHg0EFg0HXNC5vBwAAAOBFrP2c30SWapOj1K1bt+qyyy7T5s2bVVhYqGOPPVbPP/+8vvWtb7V2agAAAAAApKRNDtAffPDB1k4BAAAAQAcTNfkddGQW12gAAAAAAJAF2uQZdAAAAADY3/gddGQaLQwAAAAAgCzQYc+gR3Ot749EQn5JUk0ouLfZ9x6rkxUjaodI+ScMTevJiNpxg1aOsVgK33VpZhnTb8fM8RjPDrd1d4H10s4xpa/h1FsVFO5i2DnZsTzWWyzgXs55HY76U0jKjpFrxww79eRLKbdd9VYFO/XkMFuyx9l17eRi+k1vi9uzRwqCrtdOu/OyLY2EVcfs/am5drcvZsBazmlfzp8OU2lfRsx+jlgL14Xtxu7ESky+uZx8CStvKhmP+e2oz7P+E7bblR3Tcznt+eu77mlQdSGvO3VCyEhCf+MxKTO6Z/66sJ2X3f5r6oOeQsbbeMT6j8/OLdLJqS9vbV+S1LWLpD37pGGX0/R5ixUNWjmF6qz6zunkjueFU85w2O6z7H3aF7HXlZdsIPspZgXcWNfDimPXUyrHJOf44+xPMbvNeg3ltCsFrFxiAaev8J5TvP80nBztl1GP+5Ezn11vMXt5I5V2JScX53hmbct3dve21+UtZiTPZz+7+9Td9v7tabd0QtjbwDlkeD7+23whu57i+1BqcSQpkvB5Kf4ZzOsmcLqrgs7Wc4NjbLzdJ7sNnD7B3h3ru1gB6sO5HpOyw9m51O1u8PkyklqfH+vkNHbraXNN15TiROy+xrUfO/tTvvVeLMnPKU49Oe1hW11n1/Lt+WvaMX6rHBlGCwMAAAAAIAu0+QH6XXfdJcMwNHny5NZOBQAAAEA7FpUvow+gTbeCV199Vf/v//0/HXvssa2dCgAAAAAALdJmB+g1NTW65JJL9MADD+jAAw9s7XQAAAAAtHNR08joA2izA/TS0lKdddZZGj16dGunAgAAAKADiMmX0QfQJu/i/thjj+m1117Tq6++2tqpAAAAAACQFm1ugP7xxx/rxz/+sRYtWqS8vOR+gyYUCikUCrmmxaIR+fxtrvgAAAAAWkmUn1lDhrW5FrZ27Vpt3bpVX//61xUIBBQIBLR8+XL94Q9/UCAQUDQabbRMRUWFCgsLXY+qyhdaIXsAAAAAAJrW5k4hn3766Xrrrbdc0yZNmqQjjjhCN910k/x+f6NlysvLNXXqVNe0U6f8v4zmCQAAAKB9iYkbuSGz2twAvaCgQIMHD3ZN69y5s7p169ZouiMYDCoYDLqmcXk7AAAAACCbMEoFAAAAgCTwHXRkWrsYoC9btqy1UwAAAAAAoEXaxQAdAAAAADIt2vbusY02hhYGAAAAAEAW4Aw6AAAAACQhZnIXd2RWhx2gB+pMSZIvEJMkdc6tlyR9mUIs/y7rt9eNmFWdPtP9vmEmLtE0I2Y/2z/l7qu349s5Nv6F971ITMIW2G2XO+yxc7FzOyC428rRDp9s2VzshQJ19suo4YplJhnT2Yayf+7CZ1dQl9yQJGmrl5Scuo+4Qsrnj7neT1ZejhUoZL92tmE8fgpMe5s65fRa987sgZ1WVkY035rgSyGes/3tevI726KZdrcvgR1WTkXBXZKkj1IsY8PcnFxy7W0RdrZhkgdWZ5t3yg1LkqobJmMmPCeZ0wG51v4jvzUhJ2AVNOyxnE47Cuza0zADdixP/YQrqP1k52Ya7v1rn4v79xTCKZfTtpx9MuntGV+1ewFfuMnJydn2lbVsrNh+tlfl8TqygN3fB3Lsfj/eZ3irL2vl1lOOHcvpluP9jdd24bMW7J27XZLks3Py2n9Jks9ulP56O0aK+7bD2e2cuKlcv+dz6toJYbcH02MfFp/N7y6b00+nUF3x45G/3lr60Dz7COQUPMnkAnXW8mbALlS8H7PaSL2HzRD/LBGwF0r4jOGZUxR7+ZbcJyvgHBzt1PICEdc6kk7JOVZs32FP6d7ovWT7/MTjl7NNg3b/9YW31OLtNSdnz4E/nMrxVpJvp3Vs9IUPkCQVBK0PUNUe4zhtvOFiTv8QqLF2KF8kqGTEEj4HHpBrf6hryedDAJI68AAdAAAAALzgO+jItDbZwm677TYZhuF6HHHEEa2dFgAAAAAAKWuzZ9CPPvpovfDCC/HXgUCbLQoAAACANiDG76Ajw9rsqDYQCKhXr16tnQYAAAAAAGnRZv8E9N5776lPnz465JBDdMkll2jTpk2tnRIAAACAdiwqI6MPoE0O0E866STNmTNHCxcu1MyZM7Vx40Z985vf1M6dO1s7NQAAAAAAUtImL3EfN25c/P/HHnusTjrpJB188MH6xz/+oSuvvLLR/KFQSKFQyDUtFo3I52+TxQcAAADQCvgOOjKtXbSwAw44QIcddpg2bNjQ5PsVFRUqLCx0PT5bt3g/ZwkAAAAAQPPaxQC9pqZG77//vnr37t3k++Xl5aqurnY9+gw+fT9nCQAAAKAt4zvoyLQ2eY33DTfcoPHjx+vggw/WZ599pmnTpsnv9+uiiy5qcv5gMKhgMOiaxuXtAAAAAIBs0iZHqZ988okuuugiffHFF+rRo4dOOeUUrV69Wj169Gjt1AAAAAC0U3wHHZnWJgfojz32WGunAAAAAABAWrXJAToAAAAA7G9RzqAjw2hhAAAAAJCEmIyMPlIxY8YMDRgwQHl5eTrppJP0yiuv7HX+6dOn6/DDD1d+fr4OOuggTZkyRXV1dSmtG+nHAB0AAAAA2qB58+Zp6tSpmjZtml577TUNGTJEY8eO1datW5ucf+7cubr55ps1bdo0vf3223rwwQc1b948/exnP9vPmaM5HfcSd9N68gdikiTDsCYYMe9/uYrlGPay1utovlKKFbO3RqSz9ZxbY0+POvGTj2caTg6JudrTTU+pxf+UE7ADRvKdgN7ry8yzkzL9Vi6JMyQb0ilj1A5n57gzlGdNTyE3X9iJaS0bi6b2NyyfXcF2ERWNWHFyUojltAsnJ6e8vrCdoz+5OGauvdENu16cNhB1XidfX/H2Y/8nku8sm9pffsMHWNusKLfWSsUpUwrb0Ekh2slqZ6ZdLg/Fc+UQjdltoEFTiOZ5T8sVO2DVWyhsb1yP5TTtxfyhaHxabo71/3qPsYyYlUssx+4D49O99TsNuxSnzsxcu79w6jDJWE778uXE7NzcfUZK7cJnLRO1f9AjFvDaCdrL2f19p/x6K2y9Pd1OzR9OKawVa7fPlaNXpl0vYbuB7Iyk3lCdY0ckz4rps4+Vntuq3QaMsFP/7uW99NNGQjMI7Lan+01PuTldoPKsfcbnd/oK+/0UmkY013rO2WnF+lqwypWz177RqXfns4lzTPHSkcVndT7fOOVzuo0ky+ksFz4wYuW2I8c1Pdk4TeXmHLcDfudA7i1mzNlX8u223uAzj7NNvObkrNsXcde9588Udpyu+aH4pF2+zgkrSzJUJ6ugzme4fp2rJUmf6CBvOe1FNM8+sCS7Pe0iOJ9ROgXsPtHelKmeCW4Lsu0S93vvvVdXX321Jk2aJEmaNWuWnn32WT300EO6+eabG83/0ksvacSIEbr44oslSQMGDNBFF12kl19+eb/mjeZlVwtL0qeffqrvf//76tatm/Lz83XMMcdozZo1rZ0WAAAAAOwX9fX1Wrt2rUaPHh2f5vP5NHr0aK1atarJZU4++WStXbs2fhn8Bx98oOeee05nnnnmfskZ+9bmzqB/9dVXGjFihEaNGqV///vf6tGjh9577z0deOCBrZ0aAAAAgHYs5vWSPI9CoZBCoZBrWjAYVDDY+PKqbdu2KRqNqmfPnq7pPXv21DvvvNNk/Isvvljbtm3TKaecItM0FYlEdO2113KJexZpc2fQ7777bh100EGaPXu2TjzxRA0cOFBjxozRoYce2tqpAQAAAEDKKioqVFhY6HpUVFSkLf6yZcv061//Wn/+85/12muv6cknn9Szzz6rO+64I23rQMu0uTPoTz/9tMaOHavvfe97Wr58ufr27asf/vCHuvrqq1s7NQAAAADtWDTD5zdvLS/X1KlTXdOaOnsuSd27d5ff79eWLVtc07ds2aJevXo1ucwtt9yiSy+9VFdddZUk6ZhjjlFtba2uueYa/fznP5fP1+bO37Y7bW4LfPDBB5o5c6a+9rWv6fnnn9d1112n66+/Xg8//HBrpwYAAAAAKQsGg+ratavr0dwAPTc3V8OGDdPixYvj02KxmBYvXqzhw4c3ucyuXbsaDcL9fuvum6aZ2g1UkV5t7gx6LBbT8ccfr1//+teSpKFDh2rdunWaNWuWJk6c2OQyTX2XIxaNyOdvc8UHAAAA0Eoy/R10r6ZOnaqJEyfq+OOP14knnqjp06ertrY2flf3yy67TH379o1fJj9+/Hjde++9Gjp0qE466SRt2LBBt9xyi8aPHx8fqKN1tbkRau/evXXUUUe5ph155JF64oknml2moqJCt99+u2tan2PGqN+QsRnJEQAAAAAy7YILLtDnn3+uW2+9VVVVVTruuOO0cOHC+I3jNm3a5Dpj/otf/EKGYegXv/iFPv30U/Xo0UPjx4/XnXfe2VpFQII2N0AfMWKE1q9f75r27rvv6uCDD252mfImvstx+rWzMpIfAAAAgPYploXfEC4rK1NZWVmT7y1btsz1OhAIaNq0aZo2bdp+yAypaHMD9ClTpujkk0/Wr3/9a51//vl65ZVXdP/99+v+++9vdpmmfpqAy9sBAAAAANmkzY1STzjhBM2fP1/l5eX65S9/qYEDB2r69Om65JJLWjs1AAAAAO1YNMu+g472p80N0CXp29/+tr797W+3dhoAAAAAAKRNmxygAwAAAMD+lm13cUf7k313OQAAAAAAoAPiDDoAAAAAJCFmcn4TmdVhB+h1B1qXp4TrrCr4ancn6w3D9Bxrd3GOJCli3yg+YofyGivm3GjeZ+VW38WOV2fFD3iIZ0Savvym7kCrUwl3avLt5nOzW8pHOw+0Xuc4K/IWx5VLkZVLJM967fWKIacsUbveonacL3d635axHMMVIz693m89e9xTvqrNt+LlWjlEQ3acYLOLNM9vlyNq2LlaL+P15UuynHZ97OrX2VreKVO9z/V+Mky//WzntKvYjhHyHkuSdg6wKv6dHdZvdsa3Q7JlayCaa7iWra3LtXK1y2smGTMWsOJsr8m3F2ywDmc7Jpmes+6NO4sk7ameUK2VW8BjOSOdrNzquuXGp9XuisoV3CNnMdNu8/EwScZz2rgk7fJbeRl2+9i6s8B6I8lymnbjNmPuTiHSWZ7iNBQ+4iArht09mPZ+ZES8xdlVbG3MXXa76uTESaHaYznWQiG7j/eFnX7IW7B4fdnbbs0O62dH66JWXK/9lyTVd7Vj2rt0LGL9x+exoKbdf/nC1vL1dlNQzH72EC+SZ/eBAbtfzXPaiT2D13Zh11s07HRodpgUjmv1XZ2+z6rzlTWHWyGdz/FJlrOuyMolVGiXLWoFqNkd9BRH2tNPxOy6d9pB/DiUZDnjXYFdlvh+6OyfKdWXXT47p+pdVj/r9BnJxnQ+P9Qd0t2K52/wnv0ZKtk+3/nc5MQI2du02um/PLb9cGe7/kO5jd/0GKv2kEJJe/afL0NWR7anfSUXJ2L3EQ3beMxOr7av1TASPwc1x/ksErM/52yt6+LOKYV+GoClww7QAQAAAMCLaEvOTgFJaJPXaAwYMECGYTR6lJaWtnZqAAAAAACkpE2eQX/11VcVjUbjr9etW6dvfetb+t73vteKWQEAAABoz7iLOzKtTQ7Qe/To4Xp911136dBDD9Wpp57aShkBAAAAANAybXKA3lB9fb0eeeQRTZ06VYbBX7QAAAAAZAZ3cUemtfkWtmDBAm3fvl2XX355a6cCAAAAAEDK2vwZ9AcffFDjxo1Tnz59mp0nFAopFAq5psWiEfn8bb74AAAAAPaTGHdxR4a16TPoH330kV544QVdddVVe52voqJChYWFrsfWNS/spywBAAAAtAdR08joA2jTA/TZs2eruLhYZ5111l7nKy8vV3V1tetRfPzo/ZQlAAAAAAD71mav8Y7FYpo9e7YmTpyoQGDvxQgGgwoGg65pXN4OAAAAwAtuEodMa7Mt7IUXXtCmTZt0xRVXtHYqAAAAAAC0WJs9jTxmzBiZptnaaQAAAADoIGJ8TxwZ1mbPoAMAAAAA0J602TPoAAAAALA/8TNryDTOoAMAAAAAkAU67Bn03J3W99d9gZj12h9NOVZOrRXDMP2SJH+dNT0a9PYXNl+99exPePblOLnleAjW9Pfzc2qt6bEcb7kZdgqdc6ykfBHrddTvKYwVK2z9XSiwy8ol3Cm1v0Q6ZfFFrOUNO6dO+VaONeqcdCxf2H62Yyhq52TXoxHzllterhVwd70Vx8yJuXL0xMklmJCLU23JfhfKbhK5OyJ2Ltbub/pNb3EaxHJyCFbbEwIpxJLUaYtVX93zaiVJH9ttX/ne24az30TsdhHwWxUWsduwkeStK5x67twpJEn6Uvnx95z2YibZ/p39pzDX6hw+sacHgnaDMIONF9oLp4/xRfYUJhCwVhL2FKkJdp8ow9vO7fSlkuSz9xunGXQOWhtlt8d2Ydpt34g/O294bxc5W3dKkgK7re0Yy7X3Tb+3e5nk7rTK6fdZzzEP3XIip+9y6jySY+WSW2v1kVFvzUKG3c8UB62y1kasAF77L0nyh9yvzRS/c2k4faDdJuJxUwjn7HeN2oMj2RwTOgHDee1r8u2k+EN2+eqsyu6ZU23n5KzEY5yQ4VouYH9GqfewHZy6DjuHM+f447SHZPtCZ756+9htddOKHOD090mnFOd83nH47X7a9HmrMKe/z/1yt71Y7p73dtl5dko2KXd5fM7nHqf/SjJMfP2hJibG7HJ53J/ytlk55OyyjtsR5y7iHuveaOqzod0egl/Znw36JtepOWf4nLbqU0J7aMff0+Y76Mg0zqADAAAAAJAF2twAPRqN6pZbbtHAgQOVn5+vQw89VHfccQd3dAcAAACQUTHTl9EH0OYucb/77rs1c+ZMPfzwwzr66KO1Zs0aTZo0SYWFhbr++utbOz0AAAAAAFLS5gboL730ks455xydddZZkqQBAwbo73//u1555ZVWzgwAAABAe8Z30JFpbe46ipNPPlmLFy/Wu+++K0l64403tHLlSo0bN66VMwMAAAAAIHVt7gz6zTffrB07duiII46Q3+9XNBrVnXfeqUsuuaS1UwMAAADQjvE76Mi0NjdA/8c//qFHH31Uc+fO1dFHH63KykpNnjxZffr00cSJE5tcJhQKKRRy/95FLBqRz9/mig8AAAAAaKfa3CXuN954o26++WZdeOGFOuaYY3TppZdqypQpqqioaHaZiooKFRYWuh6b31y8H7MGAAAA0NbFTCOjD6DNDdB37doln8+dtt/vVywWa3aZ8vJyVVdXux69jz0906kCAAAAAJC0NneN9/jx43XnnXeqf//+Ovroo/X666/r3nvv1RVXXNHsMsFgUMFg0DWNy9sBAAAAeMFZbmRamxul/vGPf9Qtt9yiH/7wh9q6dav69OmjH/zgB7r11ltbOzUAAAAAAFLW5gboBQUFmj59uqZPn97aqQAAAADoQDiDjkxrc99BBwAAAACgPWpzZ9ABAAAAoDVwBh2ZxgAdAAAAAJIQEwN0ZFaHHaAb9q+ymfZfwUwz/bFleAtq2LkYEffiRgv6gfgf+YyE3FKUlk4pmvA6IcdkxbehUza7vnJ8iSvY/+LbzM7R8KWhgSXE2LNtvcU27eTii8Ubmoc4zsqdRVvYrnz1VoBIzO+K67Vs9kLWU8x69qUUY4+Ar3Hh4m3Pn2JQOyWf3/pPzGuOTr/V4EtKZkv/oh/vb1KsryZWbyTGTDq2074S22pCYC9i1jKG3T2kug2d5aMxq/IDzjZoQTPbUz/udXgP5H4ZaElfmJCLGUutfRlRu60GrDK2qHtOqONGTT7JdmEm9M/OvuPzx5qO6yE302ctHHYaVoq7ZbwoiUXy0PaNxIIk7kdJih9jw3Y9RRJX5C1eU8uk+hnM2Y+Neiuphn1iyvuRk5MdK+VjiL1YJNogqRQ/HzoV5JSpLtLCj+9G4/97Po47RbCXizgV1pJ+GoCkDjxABwAAAAAvuMQdmdYmbxK3c+dOTZ48WQcffLDy8/N18skn69VXX23ttAAAAAAASFmbPIN+1VVXad26dfrb3/6mPn366JFHHtHo0aP1v//9T3379m3t9AAAAAC0Q5xBR6a1uTPou3fv1hNPPKHf/OY3GjlypAYNGqTbbrtNgwYN0syZM1s7PQAAAAAAUtLmzqBHIhFFo1Hl5eW5pufn52vlypWtlBUAAACA9o4z6Mi0NncGvaCgQMOHD9cdd9yhzz77TNFoVI888ohWrVqlzZs3t3Z6AAAAAACkpM0N0CXpb3/7m0zTVN++fRUMBvWHP/xBF110kXy+posTCoW0Y8cO1yMWTfydEAAAAABoXsw0MvoA2uQA/dBDD9Xy5ctVU1Ojjz/+WK+88orC4bAOOeSQJuevqKhQYWGh67H5rcX7OWsAAAAAAJrXJgfojs6dO6t379766quv9Pzzz+ucc85pcr7y8nJVV1e7Hr2POX0/ZwsAAACgLTNNI6MPoM3dJE6Snn/+eZmmqcMPP1wbNmzQjTfeqCOOOEKTJk1qcv5gMKhgMOia5vO3yaIDAAAAANqpNjlKra6uVnl5uT755BMVFRXpvPPO05133qmcnJzWTg0AAABAOxUTZ7mRWW1ygH7++efr/PPPb+00AAAAAABImzY5QAcAAACA/Y07rSPT2vRN4gAAAAAAaC84gw4AAAAASeBO68i0jjtAt/ctM2r9x7nhQyr7nBE1E17LFcswlRQjZj37IolvmCnn5ojnEnO/TpZTBueyHl+99Tqal2ThGsaKZqhjs1MJ+K1CplJfpnNNiVNfPu/lkyTTXsypNyPZRtCEZreZ1/LFnJzs9mSX1UjDdTSN2qzn5a3k6qJWl+SUOSVOnYdb2M7sOM62VINN6Ozj8nhfSp/TDux9wHnttbhOGNO3p4zOvplqP+Hsl/HiGu7nfWpQPz6fXaKEcnoWM9zPqe9GUszKydl28TbmMWZ8E0asHcdrP+8Sb2N23eeYrhy9xnH47WSCLd0xJfnCTpL2k9djh73tTLsvNZyUUomXUNdmwL0Oz8J25xdI8cDYMDWnPdkhqqOd7Nfejt9OHCNh06U0IEhsk0Yzz0nG8dUbTeaWkoR+NRZzDkjewsT350jjXrRFxxE16ANb2LdGoy0/yBphu/+yy1QXSfGmyE31U84xM+qtE4t/vrGfIzF3nwggdR13gA4AAAAAHvAddGRa1n0HfcWKFRo/frz69OkjwzC0YMEC1/umaerWW29V7969lZ+fr9GjR+u9995rnWQBAAAAAEiTrBug19bWasiQIZoxY0aT7//mN7/RH/7wB82aNUsvv/yyOnfurLFjx6qurm4/ZwoAAACgIzFNI6MPIOsucR83bpzGjRvX5HumaWr69On6xS9+oXPOOUeS9Ne//lU9e/bUggULdOGFF+7PVAEAAAAASJusO4O+Nxs3blRVVZVGjx4dn1ZYWKiTTjpJq1atasXMAAAAALR3MdPI6ANoUwP0qqoqSVLPnj1d03v27Bl/DwAAAACAtijrLnHPhFAopFAo5JoWi0bk83eI4gMAAABIA7MlP/kJJKFNnUHv1auXJGnLli2u6Vu2bIm/15SKigoVFha6HpvfWpzRXAEAAAC0LzEZGX0AbWqAPnDgQPXq1UuLF+8ZXO/YsUMvv/yyhg8f3uxy5eXlqq6udj16H3P6/kgZAAAAAICkZN013jU1NdqwYUP89caNG1VZWamioiL1799fkydP1q9+9St97Wtf08CBA3XLLbeoT58+mjBhQrMxg8GggsGgaxqXtwMAAADwgp9CQ6Zl3Sh1zZo1GjVqVPz11KlTJUkTJ07UnDlz9NOf/lS1tbW65pprtH37dp1yyilauHCh8vLyWitlAAAAAABaLOsG6CUlJTL3cvcFwzD0y1/+Ur/85S/3Y1YAAAAAOjp+Cg2Z1qa+gw4AAAAAQHuVdWfQAQAAACAb8TNryDTOoAMAAAAAkAU67Bl0I2o/+60/gxXkhiRJ21OIlbsjIknyRazqTPmrKTHryRe2n+utZ3/AeiPqJVZiDvZrf13MXoc/pdy659VKkj6p97Z4U7kEQlbd+yLWBMNeh5nkn4389vKGXeE+azOoa7BOkrS1BSmaOVYygZyoK7dkBQPWciF7OWcbGp42otzr9tl/snX+cmt4+xOuM3ugus6eYt9Y0e+xcA1j2vtPTq1T0NRiBb7aJUnqmmPllko9xXOyy2nErHaRm2M1jLCTWpI7qNOecgONk/Fb3YWiwUZvNc3O6YDc3a7XeUFrZ69JMkw8N7uPCNTuyc3vS6Gf0J79zYnptDMz3r6Sqy+nLUhSvl2uXfakLnb/uiXppOznhPbkb0G/Y361XZLki/S2XqfYTwd2WTXss+vJH97b3Mnx2+Wsj/cTdv+c5C4eb/N2Gzgkz937ee2/JMlXb+/b9kb0BVLbt02nPdntI1DnJOs9VrycCY3c87ZMmN8XsMtob4dUShqvJ7uj6R7Y6U4uyf46sNta3nQO0Xb9BXPtfsxDTs529/nd6zYiHoI0XM6ud6dvTPZY3RSnHfjCVv10Dlo7906P2zLeb33xlZ1bj8bvJdtAEjaVs02DOal1PD6n/n0N6j/Ffsf/+XYrVrjAyslvbYR4s0qyr/Dbn23MBo3cqaec7VY/7YvmJhUrHsJed1HQOr59lFwqbRp3cUemcQYdAAAAAIAskHUD9BUrVmj8+PHq06ePDMPQggULXO8/+eSTGjNmjLp16ybDMFRZWdkqeQIAAADoWEzTyOgDyLoBem1trYYMGaIZM2Y0+/4pp5yiu+++ez9nBgAAAABA5mTdd9DHjRuncePGNfv+pZdeKkn68MMP91NGAAAAAMDvoCPzsu4MOgAAAAAAHVHWnUEHAAAAgGzE76Aj0zrEAD0UCikUCrmmxaIR+fwdovgAAAAAgDagQ1ziXlFRocLCQtfjs3WLWzstAAAAAG0Id3FHpnWIAXp5ebmqq6tdjz6DT2/ttAAAAAAAiMu6a7xramq0YcOG+OuNGzeqsrJSRUVF6t+/v7788ktt2rRJn332mSRp/fr1kqRevXqpV69eTcYMBoMKBoOuaVzeDgAAAMALznIj07LuDPqaNWs0dOhQDR06VJI0depUDR06VLfeeqsk6emnn9bQoUN11llnSZIuvPBCDR06VLNmzWq1nAEAAAAAaKmsO41cUlIicy+3R7z88st1+eWX77+EAAAAAEASN3FHpmXdGXQAAAAAADqirDuDDgAAAADZiO+gI9M4gw4AAAAAQBbosGfQY3bJfb70fZPEiFnPZkKtJv2HNvvPJaY/YXoqKTrLGO7nWI7hnu6Rzwns5JpCHDPHnVy8eIk574OzDRPrpz6aWIFJ5OSs045ltLBdOLdRiNmpmNEW/C3MSS5m15e/6beTDRN/HY9jeIojSUa8vuxlW/inPjPX2pgxueOl1L7S9WdHexvWRRp3k57zcjah8x87x6jdLrzGc7adL7qnnRpGim3WXs7023Ufa/mZgXg/kdjmkgxtJPzH9Ds7lPe2Gg+Vk2PHkuvZK6eenPo2IqnFaYrhlK+FR+aYXUE5RjT1XOwqj+9PKTaL+LZzXqehK2zumJh0u3CWd+rbaV4t6adtTr3VmTnuCV7jNNOuvLR9Z16nfE6bT7WrcJaP5aS2fENODs7nJr8vllpOTn3Yv9TjavJe26zpfo5/pkvxbKnT1n0Ny+ZsE68h/VblO597cvyp79vNJWAGPLZ/pywJixnu7rp94kvoyLAOO0AHAAAAAC+4xB2ZlnWXuK9YsULjx49Xnz59ZBiGFixYEH8vHA7rpptu0jHHHKPOnTurT58+uuyyy+K/iQ4AAAAAHcmMGTM0YMAA5eXl6aSTTtIrr7yy1/m3b9+u0tJS9e7dW8FgUIcddpiee+65/ZQt9iXrBui1tbUaMmSIZsyY0ei9Xbt26bXXXtMtt9yi1157TU8++aTWr1+vs88+uxUyBQAAANCRmGZmH17NmzdPU6dO1bRp0/Taa69pyJAhGjt2rLZu3drk/PX19frWt76lDz/8UI8//rjWr1+vBx54QH379m1hzSBdsu4S93HjxmncuHFNvldYWKhFixa5pv3pT3/SiSeeqE2bNql///77I0UAAAAAaHX33nuvrr76ak2aNEmSNGvWLD377LN66KGHdPPNNzea/6GHHtKXX36pl156STn2/VkGDBiwP1PGPmTdGXSvqqurZRiGDjjggNZOBQAAAEA7ZppGRh+hUEg7duxwPUKhUJO51NfXa+3atRo9enR8ms/n0+jRo7Vq1aoml3n66ac1fPhwlZaWqmfPnho8eLB+/etfKxpt4c0HkTYZG6Bv3749U6Hj6urqdNNNN+miiy5S165dM74+AAAAAMiUiooKFRYWuh4VFRVNzrtt2zZFo1H17NnTNb1nz56qqqpqcpkPPvhAjz/+uKLRqJ577jndcsstuueee/SrX/0q7WVBatIyQL/77rs1b968+Ovzzz9f3bp1U9++ffXGG2+kYxWNhMNhnX/++TJNUzNnztzrvE39JSoWTeNv4wAAAABo/0wjo4/y8nJVV1e7HuXl5WlLPxaLqbi4WPfff7+GDRumCy64QD//+c81a9astK0DLZOWAfqsWbN00EEHSZIWLVqkRYsW6d///rfGjRunG2+8MR2rcHEG5x999JEWLVq0z7PnTf0lavObi9OeFwAAAACkKhgMqmvXrq5HMBhsct7u3bvL7/dry5YtrulbtmxRr169mlymd+/eOuyww+T3++PTjjzySFVVVam+vj59BUHK0jJAr6qqig/Qn3nmGZ1//vkaM2aMfvrTn+rVV19NxyrinMH5e++9pxdeeEHdunXb5zJN/SWq97GnpzUvAAAAAO1bNt3FPTc3V8OGDdPixXtOPMZiMS1evFjDhw9vcpkRI0Zow4YNisVi8Wnvvvuuevfurdzc3JTqBOmVlgH6gQceqI8//liStHDhwviNCkzT9HzDgZqaGlVWVqqyslKStHHjRlVWVmrTpk0Kh8P67ne/qzVr1ujRRx9VNBpVVVXVPv/i09Rfonz+rLuBPQAAAAAkberUqXrggQf08MMP6+2339Z1112n2tra+F3dL7vsMtcl8tddd52+/PJL/fjHP9a7776rZ599Vr/+9a9VWlraWkVAgrSMUs8991xdfPHF+trXvqYvvvgi/jNpr7/+ugYNGuQp1po1azRq1Kj466lTp0qSJk6cqNtuu01PP/20JOm4445zLbd06VKVlJSkXggAAAAA2JsUfqs8ky644AJ9/vnnuvXWW1VVVaXjjjtOCxcujN84btOmTfL59pyTPeigg/T8889rypQpOvbYY9W3b1/9+Mc/1k033dRaRUCCtAzQf//732vAgAH6+OOP9Zvf/EZdunSRJG3evFk//OEPPcUqKSmRuZfrO/b2HgAAAAB0JGVlZSorK2vyvWXLljWaNnz4cK1evTrDWSFVaRmg5+Tk6IYbbmg0fcqUKekIDwAAAACtzjSN1k4B7Vzavoj93nvvaenSpdq6davrpgOSdOutt6ZrNQAAAAAAtEtpGaA/8MADuu6669S9e3f16tVLhrHnL0uGYTBABwAAAND28W1bZFhaBui/+tWvdOedd7apmwtEc60/IkQj1k0TautT/1mB+sIcSVLM/jnBWE5qcUyfk5sdx36OhK3Ani6oSbz8xu5MInnW9JjXLW/nti3U2ZVjS0SCVi6mk4vHK4acsjjLOWXaEcrznowdw7S3oRm124dd9w1+KjIpdeEcV7xo2KpAM5U9zrA2nmHn5LST+DY2kjxS2LNHCt31Y0ZSv1TLqaf6LnZSMY852eqL8iVJO+qt3FKqJycnn/Ns5bA7lOOanmxuZsBwLd9wn3L2Ta9tdltdZzuW9RSqT62gTluPdNrTMCNRj43UZsScdmUlZUadCvRWuFhkzw1oakPuDsLzPumsOiGHWAv6HbO4SFKDvstpBx63YbizvS+b7v7LdHdHyXH2Safu7H0xfgxJMpjTop1t92Goh/v9FH6vJer0z07faO/rnnsLO4Bh/6BLJN8J6D0npz9VQhP12N004rRdww6Uys/bhDs79WUtvbW+a0q5OO0rXk/2/plKX5F4PHPqK5ZaVyEzYNVPLNfZpqkfOyJ2l+C09fhnsFjT8zfHWT7W80ArxwZla7Sv70vCTuxsg/oUPx869RyNNWhRKbbVaA+rPTl94O6IfVxzNkGSm8L5zGv49iQSy7HLeYAVPNnPh/H2ZT9vr0/hsxeAJqXlZ9a++uorfe9730tHKK1YsULjx49Xnz59ZBiGFixY4Hr/tttu0xFHHKHOnTvrwAMP1OjRo/Xyyy+nZd0AAAAA0BzTNDL6ANIyQP/e976n//znP+kIpdraWg0ZMkQzZsxo8v3DDjtMf/rTn/TWW29p5cqVGjBggMaMGaPPP/88LesHAAAAAKA1pOUS90GDBumWW27R6tWrdcwxxygnx32N9/XXX590rHHjxsV/R70pF198sev1vffeqwcffFBvvvmmTj/9dG+JAwAAAECy+A46MiwtA/T7779fXbp00fLly7V8+XLXe4ZheBqge1FfX6/7779fhYWFGjJkSEbWAQAAAADA/pCWAfrGjRvTESZpzzzzjC688ELt2rVLvXv31qJFi9S9e/f9mgMAAACAjobviSOz0vIddEd9fb3Wr1+vSCSSzrCNjBo1SpWVlXrppZd0xhln6Pzzz9fWrVubnT8UCmnHjh2uRyya2RwBAAAAAPAiLQP0Xbt26corr1SnTp109NFHa9OmTZKkH/3oR7rrrrvSsQqXzp07a9CgQfrGN76hBx98UIFAQA8++GCz81dUVKiwsND12PL6C2nPCwAAAEA7Zmb4gQ4vLQP08vJyvfHGG1q2bJny8vb8DuLo0aM1b968dKxir2KxmEKh0F7zq66udj16Dh2d8bwAAAAAtCMM0JFhafkO+oIFCzRv3jx94xvfkGHs+V7G0Ucfrffff99TrJqaGm3YsCH+euPGjaqsrFRRUZG6deumO++8U2effbZ69+6tbdu2acaMGfr000/3+jvswWBQwWDQNc3nT0vRAQAAAABIi7SMUj///HMVFxc3ml5bW+sasCdjzZo1GjVqVPz11KlTJUkTJ07UrFmz9M477+jhhx/Wtm3b1K1bN51wwgl68cUXdfTRR7esEAAAAACwNyY3iUNmpWWAfvzxx+vZZ5/Vj370I0mKD8r/8pe/aPjw4Z5ilZSUyDSbv77jySefTD1RAAAAAACyVFoG6L/+9a81btw4/e9//1MkEtF9992n//3vf3rppZca/S46AAAAALRFezmPCKRFWm4Sd8opp6iyslKRSETHHHOM/vOf/6i4uFirVq3SsGHD0rEKAAAAAADatbScQV+3bp0GDx6sBx54oNF7CxYs0IQJE9KxGgAAAABoPZxBR4alZYA+duxYrVy5UgMHDnRNf+KJJ3TZZZeptrY2HatJq0CdtXcFcqKSpM659ZKkr1LY6fx1MUmSYfqtCXYMI9aym0gYEevZ54+54ibFZybkYj0HQtaEcNhjbnacA3J3u3NMoYym3wrmr7eefZHU6snZhk4Ohp1jji/mep0Mp358Yee1HdOuRyPqLbe8HCtQyKl/u8zONvXCdK5zsQu0J0dnjuTqz7SX99dYAYyo9ZOIzn0cPW1Lp1wBKwlfxOdOxeMNVPx1VsX0zt8hSfogmkJOcpaxU8ixksy19/HdKR5Q84NWfe3y7wkQ3ybJxrTnKwrusv7jtAefuw0ny2mPgdo9DTOYY9Wh53I68zvtwO5v4pOTjOcLxBuk/Ha5zBxrWl7AbnPJljO+38Rcufns/SeldvFFtb1sUZPrSpbT7+TZ7cJX76zAc0rxdQfsuos45TLd7yfNrvfiHGs/qovlWKnFml2i+VBO/2w3MV/Abqtec/K7F4jn4nQZHuL5EvrPgP3rqqbX/chZaZ5VOKftxvvCFPoKX9haKMfeJ4/t9LEkaV7CsWRfArvttpBnfZ5w9oFce/8Oe2j78eNajtNPy/WcbPty6iOa4/5ckfg5wwvnOObIt4+Z21P82OSrqbNz6Rqf5nk7Jmwjv72v56f4+dCp58559fFpIV++lZvHPsxXazV2I9JFktSzU40k6XOPfYXfbusNL9GOH0922/tDJCepWM6u7NRz9zzrs/5HTrfdws/AQEeWlkvcr7rqKo0ePVpVVVXxafPmzdNll12mOXPmpGMVAAAAANC6TCOzD3R4aRmg33777TrzzDM1evRoffnll5o7d64mTZqkv/71r3v9ffKmrFixQuPHj1efPn1kGIYWLFjQ7LzXXnutDMPQ9OnTW1YAAAAAAABaWVoG6JL0xz/+UUOGDNE3vvENXX311fr73/+u8847z3Oc2tpaDRkyRDNmzNjrfPPnz9fq1avVp0+fVFMGAAAAgKQZZmYfQMrfQX/66acbTTv33HP14osv6qKLLpJhGPF5zj777KTjjhs3TuPGjdvrPJ9++ql+9KMf6fnnn9dZZ53lLXEAAAAAALJQygP0vd2Z/aGHHtJDDz0kSTIMQ9Goxzts7UUsFtOll16qG2+8UUcffXTa4gIAAADAXnGWGxmW8gA9FkvhtrBpcPfddysQCOj6669vlfUDAAAAAJAJafmZtf1l7dq1uu+++/Taa6/JMJK/y2EoFFIoFHJNi0Uj8vnbVPEBAAAAtCbutI4MS9tN4pYvX67x48dr0KBBGjRokM4++2y9+OKL6QovSXrxxRe1detW9e/fX4FAQIFAQB999JF+8pOfaMCAAc0uV1FRocLCQtdj81uL05obAAAAAAAtkZYB+iOPPKLRo0erU6dOuv7663X99dcrPz9fp59+uubOnZuOVUiSLr30Ur355puqrKyMP/r06aMbb7xRzz//fLPLlZeXq7q62vXofczpacsLAAAAQAdgZviBDi8t13jfeeed+s1vfqMpU6bEp11//fW69957dccdd+jiiy9OOlZNTY02bNgQf71x40ZVVlaqqKhI/fv3V7du3Vzz5+TkqFevXjr88MObjRkMBhUMBl3TuLwdAAAAAJBN0nIG/YMPPtD48eMbTT/77LO1ceNGT7HWrFmjoUOHaujQoZKkqVOnaujQobr11lvTkSoAAAAApIYz6MiwtJxGPuigg7R48WINGjTINf2FF17QQQcd5ClWSUmJTDP51vnhhx96ig8AAAAAQDZq0QD9iiuu0H333aef/OQnuv7661VZWamTTz5ZkvR///d/mjNnju677760JAoAAAAArYqz3MiwFg3QH374Yd1111267rrr1KtXL91zzz36xz/+IUk68sgjNW/ePJ1zzjlpSRQAAAAAgPasRQP0hpeif+c739F3vvOdFicEAAAAAFmJ30FHhrX4O+g7d+5UXl7eXufp2rVrS1eTdrEc6zkS9kuSautzU44V7mzday9m16bp3HrP8HYNjGnYO7zPnWMs4ms4OclgTU+OBA1X7KTD2Sv/MtTJeu233/BYRkkyolYOkTw7F797HcmK5hqu5Zw4u8Pem7XT15rOonaOTt17za0u7K5gM2bn6m9q7iSZ7vLGjw8et0Gk0N3WzVgKcZxcYu5tadr15jmnTlbFf1Hf2YrTgvYV349s9fY+Lo/HU6eewxGngTZ400h43hd7Pmf/ceovarcvz32F31o+3GVPg6qPtKRxKV4+Z5t6rfr4tpcUsvdBw45VW2//ikayQQ2nPflci0WcQ00q7aLIOg7F92VnB/J5ixXuZC3n1Lfh5JTKJY92CtGor8npXtus7Pr+uK5IkhQ2U+u/JClm968xu3+IRazXnltZvH+xnqJO95NCfcX7voQ+P74tk2wX8b7Tnj1+jPXHXLl64fSB/k5WUh/Vd08pVqSTlUvU6VNj1utQvX1M8dD2TZ/7OOZwPqskm1t8jc5xzOd+3ZL6MuPH7ZyElSWZm7189ICEzyZqUM5kOeWzyxOx9/VIONc1Pelw9vrr6hskkuox8oB8Kwc71Jd1+e6ckswtGrb71AbzO3UW6RJw5b0viZ9n4p8PU/wMDGCPFg/QDzvssGbfM01ThmEoGo22dDUAAAAA0Kr42wMyrcUD9Mcff1xFRUXpyEWStGLFCv32t7/V2rVrtXnzZs2fP18TJkyIv3/55Zfr4Ycfdi0zduxYLVy4MG05AAAAAEAjDNCRYS0eoI8YMULFxcXpyEWSVFtbqyFDhuiKK67Queee2+Q8Z5xxhmbPnh1/HQwG07Z+AAAAAABaQ1p+Bz2dxo0bp3Hjxu11nmAwqF69eu2njAAAAAAAyLwUbh2zx8EHHyy/v4U3JkrBsmXLVFxcrMMPP1zXXXedvvjii/2eAwAAAAAA6dSiM+gbN25MVx5JO+OMM3Tuuedq4MCBev/99/Wzn/1M48aN06pVq1rljwUAAAAAOgZuEodMy7pL3PflwgsvjP//mGOO0bHHHqtDDz1Uy5Yt0+mnn97kMqFQSKFQyDUtFo3I529zxQcAAAAAtFMtusQ9GxxyyCHq3r27NmzY0Ow8FRUVKiwsdD2q3li8H7MEAAAA0OaZRmYf6PDa/AD9k08+0RdffKHevXs3O095ebmqq6tdj15Dmj7bDgAAAABAa8i6a7xrampcZ8M3btyoyspKFRUVqaioSLfffrvOO+889erVS++//75++tOfatCgQRo7dmyzMYPBYKOfYuPydgAAAACe8B10ZFjaRqmLFy/W4sWLtXXrVsViMdd7Dz30UNJx1qxZo1GjRsVfT506VZI0ceJEzZw5U2+++aYefvhhbd++XX369NGYMWN0xx138FvoAAAAAIA2LS0D9Ntvv12//OUvdfzxx6t3794yjNS/P1FSUiLTbP5PU88//3zKsQEAAAAgZZxBR4alZYA+a9YszZkzR5deemk6wgEAAAAA0OGkZYBeX1+vk08+OR2hAAAAACAr8TvoyLS03MX9qquu0ty5c9MRCgAAAACADiktZ9Dr6up0//3364UXXtCxxx6rnJwc1/v33ntvOlaTXhn465fz04Wp/mUtvlxsr7N5SybhZUv/6hezA7UoTjrKt5ccnFsgePkpyaz+a6iTWzYmaafU0tQMu03E0vD7n4n7kdnCmEYm6t2pN8P10vPyalC0lpaz2VWlEraFVRZfZ2L7SkMRW9xW7eXNmJVMWluHHdxoYR8ZTUNFOdvA55Q31faVGCgdEtpDyts0saG1QLxd2CFT3QYZ2Y0Ti5fiOpx2mZEusaWfm5oMmlpMp35Mn+tlymKxlp8Li/cJ9nPUzMCvJMecRpzq4h3o97uz8OMY2pe0DNDffPNNHXfccZKkdevWud5ryQ3jAAAAAADoKNIyQF+6dGk6wkiSVqxYod/+9rdau3atNm/erPnz52vChAmued5++23ddNNNWr58uSKRiI466ig98cQT6t+/f9ryAAAAAAAXzqAjwzJwjUzL1NbWasiQIZoxY0aT77///vs65ZRTdMQRR2jZsmV68803dcsttygvL28/ZwoAAAAAQPqkfAb93HPP1Zw5c9S1a1ede+65e533ySefTDruuHHjNG7cuGbf//nPf64zzzxTv/nNb+LTDj300KTjAwAAAEAqsvGWQGhfUj6DXlhYGP9+eWFh4V4f6RKLxfTss8/qsMMO09ixY1VcXKyTTjpJCxYsSNs6AAAAAABoDSmfQZ89e3aT/8+krVu3qqamRnfddZd+9atf6e6779bChQt17rnnaunSpTr11FP3Sx4AAAAAOqCOdMd6tIq03CROkiKRiJYtW6b3339fF198sQoKCvTZZ5+pa9eu6tKlS1rWEYtZvy9xzjnnaMqUKZKk4447Ti+99JJmzZrV7AA9FAopFAq5Y0Uj8vnTVnwAAAAAAFokLTeJ++ijj3TMMcfonHPOUWlpqT7//HNJ0t13360bbrghHauQJHXv3l2BQEBHHXWUa/qRRx6pTZs2NbtcRUVFo8vuq95YnLa8AAAAAHQAZoYf6PDSMkD/8Y9/rOOPP15fffWV8vPz49O/853vaPHi9A2Ec3NzdcIJJ2j9+vWu6e+++64OPvjgZpcrLy9XdXW169FryOlpywsAAAAAgJZKyzXeL774ol566SXl5ua6pg8YMECffvqpp1g1NTXasGFD/PXGjRtVWVmpoqIi9e/fXzfeeKMuuOACjRw5UqNGjdLChQv1r3/9S8uWLWs2ZjAYVDAYdE3j8nYAAAAAXnAXd2RaWkapsVhM0Wi00fRPPvlEBQUFnmKtWbNGo0aNir+eOnWqJGnixImaM2eOvvOd72jWrFmqqKjQ9ddfr8MPP1xPPPGETjnllJYVAgAAAAD2hgE6MiwtA/QxY8Zo+vTpuv/++yVJhmGopqZG06ZN05lnnukpVklJiUxz7y3/iiuu0BVXXJFyvgAAAAAAZJu0DNDvuecejR07VkcddZTq6up08cUX67333lP37t3197//PR2rAAAAAIBWxSXuyLS0DND79eunN954Q/PmzdMbb7yhmpoaXXnllbrkkktcN40DAAAAAABNS8sAfcWKFTr55JN1ySWX6JJLLolPj0QiWrFihUaOHJmO1QAAAABA6+EMOjIsLQP0UaNGafPmzSouLnZNr66u1qhRo5q8gVxrC4SsZ8Nn7WX5uWFJ0lcpxMqpjUmSfBG/HTS1nAwrjPz19rOdoz/HesNTf+Breu7ALmu6r4u3JJ3cuuZaSfnC9htmCoW1U3O2gS/qnp6swG7TlYMvYr3sErQC13iJ58xrl9PZhr6ANcEpf7Jyc6xk6u2yBXKs/xgp7ApGzErGtH8UMZUYVgA7l+p6O671ywaGP/UjjWHn5GwLZ3/y2i5yv6qTJHUJJLSvVDjFsXNwtkVdwvR9ceo5316+usFiTn6xZHtQe90H5Na5Juc47STJMPH12209sGtPw/T7rP97bKpxTjuT3R5Me1saydZXoEEufrvPshftbPcbXyabi5nwH2cfb0G7ML6otmLUd7Ne29vX9Phjo06d++z68bXk8GYXz6mvsHMIibjfT5qdU9/gdtdkr/2XJPnrrVjOccjnT61lJWxCBZxdIIVDh7PNnPI4z2Yzx7tkk3P6Lae/T+XDd8DuYAJ1VoyegR0ppZRjH6OjuaYrR6cfizS5VNOcfsIXcDaCe3qy5XS2oRFJ2A8Nd1wvnPoyolbMYMDZuN7iGHYu/i9r7Vy6xN/ze/2c4vR5zmexkPU6L8fqob/0WE6nb/CnuO80FLDL5w91kiQF/dZGjPeVSebmtPGGszvtIWenVWG+aE5SseIfl+xyOp8PAbRcWgbopmnKMBp3gF988YU6d+6cjlUAAAAAQOviDDoyrEUD9HPPPVeSddf2yy+/3PVb49FoVG+++aZOPvlkTzFXrFih3/72t1q7dq02b96s+fPna8KECfH3m/pDgCT95je/0Y033ui9EAAAAAAAZIEWDdALCwslWWfQCwoKXDeEy83N1Te+8Q1dffXVnmLW1tZqyJAhuuKKK+J/AGho8+bNrtf//ve/deWVV+q8885LoQQAAAAAkBzu4o5Ma9EAffbs2ZKkAQMG6IYbbkjL5ezjxo3TuHHjmn2/V69ertdPPfWURo0apUMOOaTF6wYAAAAAoLWk5Tvo06ZNkyR9/vnnWr9+vSTp8MMPV48ePdIRvllbtmzRs88+q4cffjij6wEAAAAAINM83r+2abt27dIVV1yh3r17a+TIkRo5cqT69OmjK6+8Urt27UrHKpr08MMPq6CgoMlL4QEAAAAAaEvSMkCfMmWKli9frn/961/avn27tm/frqeeekrLly/XT37yk3SsokkPPfSQLrnkEuXl5e11vlAopB07drgesaiXHysBAAAA0OGZGX6gw0vLAP2JJ57Qgw8+qHHjxqlr167q2rWrzjzzTD3wwAN6/PHH07GKRl588UWtX79eV1111T7nraioUGFhoevx2X8XZyQvAAAAAABSkbZL3Hv27NloenFxccYucX/wwQc1bNgwDRkyZJ/zlpeXq7q62vXoc/TpGckLAAAAQPtkmJl9AGkZoA8fPlzTpk1TXV1dfNru3bt1++23a/jw4Z5i1dTUqLKyUpWVlZKkjRs3qrKyUps2bYrPs2PHDv3zn/9M6uy5JAWDwfiZfefh86fl/ngAAAAAAKRFWkap06dP1xlnnKF+/frFz2i/8cYbysvL0/PPP+8p1po1azRq1Kj466lTp0qSJk6cqDlz5kiSHnvsMZmmqYsuuigd6QMAAADAvnGWGxmWlgH6Mccco/fee0+PPvqo3nnnHUnSRRddpEsuuUT5+fmeYpWUlMg0997yr7nmGl1zzTUp5wsAAAAAQLZp8QA9HA7riCOO0DPPPKOrr746HTkBAAAAQPbhDDoyrMXfQc/JyXF99xwAAAAAAHiXlpvElZaW6u6771Ykwm+LAwAAAGifuIs7Mi0t30F/9dVXtXjxYv3nP//RMccco86dO7vef/LJJ9OxmswwDUmSrwXXqxixhGVTDRVz4iXEt/fWdOyzjXJNVsJiRtT5j/d4RsywYzrLGntdV7I5OXJ80abf8MIul2HsY75mxNtTM0VMSWJde41pL27ErAZmOsunITdfuIWtM2LlFHOSaUE4I15O+9lwT086Tnz5hG2pxvtosjn5nAXt135fagVtav2m2cIN6bFMe+Ozy+Xk6fNa+fHZ7TLFEqenwP4Dcrx9pBjLKVPM6cdSbF9NcurNYxeWuO4crwH2Fts+ZqTcvuJ9oFO2xvtTquIppZibaW/DeFfoNLcUwhkJ5yfCpj+lnJz6TsdhLH4YSvyIkuKmdJpVOppXvM0mtgOPucX7wvr65t9LUbyvcD6DpXjMjaXSoBKF3f1XJNay82tN9VdGpGU7ZaylxyAAcWkZoB9wwAE677zz0hEKAAAAALITZ7mRYWkZoM+ePTsdYSRJK1as0G9/+1utXbtWmzdv1vz58zVhwoT4+zU1Nbr55pu1YMECffHFFxo4cKCuv/56XXvttWnLAQAAAAAScRk6Mq1F18jEYjHdfffdGjFihE444QTdfPPN2r17d4sSqq2t1ZAhQzRjxowm3586daoWLlyoRx55RG+//bYmT56ssrIyPf300y1aLwAAAAAAralFA/Q777xTP/vZz9SlSxf17dtX9913n0pLS1uU0Lhx4/SrX/1K3/nOd5p8/6WXXtLEiRNVUlKiAQMG6JprrtGQIUP0yiuvtGi9AAAAALBXZoYf6PBaNED/61//qj//+c96/vnntWDBAv3rX//So48+qlgsjXcbSnDyySfr6aef1qeffirTNLV06VK9++67GjNmTMbWCQAAAADZaMaMGRowYIDy8vJ00kknJX3i8rHHHpNhGK6vE6P1tWiAvmnTJp155pnx16NHj5ZhGPrss89anFhz/vjHP+qoo45Sv379lJubqzPOOEMzZszQyJEjM7ZOAAAAAMi2M+jz5s3T1KlTNW3aNL322msaMmSIxo4dq61bt+51uQ8//FA33HCDvvnNb3pfKTKqRQP0SCSivLw817ScnByFw+EWJbU3f/zjH7V69Wo9/fTTWrt2re655x6VlpbqhRdeaHaZUCikHTt2uB6xKL/ZDgAAAKDtuvfee3X11Vdr0qRJOuqoozRr1ix16tRJDz30ULPLRKNRXXLJJbr99tt1yCGH7MdskYwW3cXdNE1dfvnlCgaD8Wl1dXW69tprXb+Fnq7fQd+9e7d+9rOfaf78+TrrrLMkSccee6wqKyv1u9/9TqNHj25yuYqKCt1+++2uaX2PGaN+x45NS14AAAAA2r9M38U9FAopFAq5pgWDQdd4y1FfX6+1a9eqvLw8Ps3n82n06NFatWpVs+v45S9/qeLiYl155ZV68cUX05c80qJFZ9AnTpyo4uJiFRYWxh/f//731adPH9e0dAmHwwqHw/L53Gn7/f69fu+9vLxc1dXVrkefo09PW14AAAAA0FIVFRWucVRhYaEqKiqanHfbtm2KRqPq2bOna3rPnj1VVVXV5DIrV67Ugw8+qAceeCDtuSM9WnQGPZ2/f+6oqanRhg0b4q83btyoyspKFRUVqX///jr11FN14403Kj8/XwcffLCWL1+uv/71r7r33nubjdnUX518/rT8BDwAAACAjiLDZ9DLy8s1depU17Smzp6nYufOnbr00kv1wAMPqHv37mmJifTLulHqmjVrNGrUqPhrp4FOnDhRc+bM0WOPPaby8nJdcskl+vLLL3XwwQfrzjvv1LXXXttaKQMAAABAizV3OXtTunfvLr/fry1btrimb9myRb169Wo0//vvv68PP/xQ48ePj09zrkIOBAJav369Dj300BZkj3TIugF6SUmJTLP5P0316tUrI2fuAQAAAGCvsui3ynNzczVs2DAtXrw4/lNpsVhMixcvVllZWaP5jzjiCL311luuab/4xS+0c+dO3XfffTrooIP2R9rYh6wboAMAAAAA9m3q1KmaOHGijj/+eJ144omaPn26amtrNWnSJEnSZZddpr59+6qiokJ5eXkaPHiwa/kDDjhAkhpNR+thgA4AAAAAScj0Xdy9uuCCC/T555/r1ltvVVVVlY477jgtXLgwfuO4TZs2NbrBNrIbA3QAAAAAaKPKysqavKRdkpYtW7bXZefMmZP+hNAiHXaAHs2xnp2vu9fW56YcK9zFL0mKObVppBbHtJeP2veF8Ns/gRgJW/E9/e2rmb/uRTr5XOtKOpyVgnbUB105psK06ycatP4Ts2PH6y3J+ovkWzOadsU49b8jlOdaT1IS1x2z/hMNW8EDHv/wGIoEXLlFI3Ycf3NLNM/0WxvTsHNK+ccR7cXDhXnu+NEUG6wk0/51w3AXu13t5ecO9yZclC9J2hWx9kOv7dOVk10/Tr2FI37X9GT/9B3LserF2ZYNxdt/klXnrHt7fb5ruVA4tYJG7e4q3HlPY4h5avBNsOvLdNqZHS/ZsGZkTy5hv93Q7WVr6z12GPH90H529vEcb2Fcuh0oqUHfH28P3sJE8t07oNPvOPXkKZzTF0btmGG7T8xxv78vZkK/9WnoACucmdD2PXDaf9iw20Gshe3L6VPzWtDfxPdte0ILU4rHdXKz23AqYcOdnaWsGNXRTinl4uzT4U52PDs3px/zwqmneB9vuKcn3b6c51zrf07bSK3RW8LO8dvef+qcvtDjISRm94WxogIrXoNqisb3oyRPd9qzOe0sYrfV3eHUPh/G67+lfbOkSI+ukvYce8L2B6d46CRXEXPaeIM+welvwoVWOWNJNrV4+ex116RYT21Slp1BR/vD9Q4AAAAAAGSBrBugr1ixQuPHj1efPn1kGIYWLFjgen/Lli26/PLL1adPH3Xq1ElnnHGG3nvvvdZJFgAAAECHYZiZfQBZN0Cvra3VkCFDNGPGjEbvmaapCRMm6IMPPtBTTz2l119/XQcffLBGjx6t2traVsgWAAAAAID0yLrvoI8bN07jxo1r8r333ntPq1ev1rp163T00UdLkmbOnKlevXrp73//u6666qr9mSoAAACAjoSz3MiwrDuDvjehkHXXtLy8PTe58vl8CgaDWrlyZWulBQAAAABAi7WpAfoRRxyh/v37q7y8XF999ZXq6+t1991365NPPtHmzZtbOz0AAAAA7ZmZ4Qc6vDY1QM/JydGTTz6pd999V0VFRerUqZOWLl2qcePGyedrviihUEg7duxwPWLRyH7MHAAAAACAvWtTA3RJGjZsmCorK7V9+3Zt3rxZCxcu1BdffKFDDjmk2WUqKipUWFjoemx+c/F+zBoAAABAW2dk+AG0uQG6o7CwUD169NB7772nNWvW6Jxzzml23vLyclVXV7sevY89fT9mCwAAAADA3mXdXdxramq0YcOG+OuNGzeqsrJSRUVF6t+/v/75z3+qR48e6t+/v9566y39+Mc/1oQJEzRmzJhmYwaDQQWDQdc0nz/rig4AAAAgm/E9cWRY1o1S16xZo1GjRsVfT506VZI0ceJEzZkzR5s3b9bUqVO1ZcsW9e7dW5dddpluueWW1koXAAAAQAdhMEBHhmXdAL2kpESm2XzLv/7663X99dfvx4wAAAAAAMi8rBugAwAAAEBW4gw6MqzN3iQOAAAAAID2hDPoAAAAAJAMzqAjwzrsAN0XsZ/9sRbHMhN/tLCFP2JoJlzXYKQSL2GZ+A0tWtipBHxWfZl+J14KyRkx97Ip1peRsOmcMvrsQqZyE494zBY2C58vfb13vK7jExKek62/TLQJO2Z8U6ba9u02n+cPu+K1iF1vvhbezSXXH7X+0yCnVPMLOA3Mn5720bBoTjlTbbrxbei0XbPxOpJZXpIMe6E97SLF8joxDa+NvQmRxttRkvf279SLz13GlqTm1I8Ra1mf6CwXbwuxlu9Iicej1APZTy2pLyeGnVO8v06xvzXs/dDZlvHPA2nYPbv461JbMKGMjpT2ocS6TtyNkgzprDqaeCwy3H1FS6TaR8TbQriJ/TvV5p9wbIzvn15TtJuTz9egV27hvu3I8UVTDNSExH0zSYn10dJjLYA9OuwAHQAAAAC84G8RyLSs+w56RUWFTjjhBBUUFKi4uFgTJkzQ+vXrXfPU1dWptLRU3bp1U5cuXXTeeedpy5YtrZQxAAAAAAAtl3UD9OXLl6u0tFSrV6/WokWLFA6HNWbMGNXW1sbnmTJliv71r3/pn//8p5YvX67PPvtM5557bitmDQAAAKDdMzP8QIeXdZe4L1y40PV6zpw5Ki4u1tq1azVy5EhVV1frwQcf1Ny5c3XaaadJkmbPnq0jjzxSq1ev1je+8Y3WSBsAAAAAgBbJujPoiaqrqyVJRUVFkqS1a9cqHA5r9OjR8XmOOOII9e/fX6tWrWqVHAEAAAC0f4aZ2QeQ1QP0WCymyZMna8SIERo8eLAkqaqqSrm5uTrggANc8/bs2VNVVVWtkCUAAAAAAC2XdZe4N1RaWqp169Zp5cqVLYoTCoUUCoVc02LRiHz+rC4+AAAAgGzCWW5kWNaeQS8rK9MzzzyjpUuXql+/fvHpvXr1Un19vbZv3+6af8uWLerVq1eTsSoqKlRYWOh6fLZucSbTBwAAAADAk6wboJumqbKyMs2fP19LlizRwIEDXe8PGzZMOTk5Wrx4zwB7/fr12rRpk4YPH95kzPLyclVXV7sefQafntFyAAAAAGhf+A46Mi3rrvEuLS3V3Llz9dRTT6mgoCD+vfLCwkLl5+ersLBQV155paZOnaqioiJ17dpVP/rRjzR8+PBm7+AeDAYVDAZd07i8HQAAAACQTbJulDpz5kxJUklJiWv67Nmzdfnll0uSfv/738vn8+m8885TKBTS2LFj9ec//3k/ZwoAAACgQ+EsNzIs6wboprnvVp+Xl6cZM2ZoxowZ+yEjAAAAAAAyL+sG6AAAAACQlTiDjgzLupvEAQAAAADQEXEGPQ3SdsdFM+E57StoOV86/mxoGi2PkWlpTtFIwzZMV7UZSXyNxHvMFgaIpTFec/tRitKx7RppaUh7+WzdldKell3QlrWLhIVTjNUoB6ew6WwmaYrlS0PbTXfzz6LDWfuXqW2XhrjpagfxONEmAnpdR7qP+04a6eio01X3TaViT0t1m3TEfbojlhn7F2fQAQAAAADIAlk3QK+oqNAJJ5yggoICFRcXa8KECVq/fr1rnvvvv18lJSXq2rWrDMPQ9u3bWydZAAAAAB2HmeEHOrysG6AvX75cpaWlWr16tRYtWqRwOKwxY8aotrY2Ps+uXbt0xhln6Gc/+1krZgoAAAAAQPpk3XfQFy5c6Ho9Z84cFRcXa+3atRo5cqQkafLkyZKkZcuW7efsAAAAAHRUmbiXD9BQ1g3QE1VXV0uSioqKWjkTAAAAAB0a43NkWNZd4t5QLBbT5MmTNWLECA0ePLi10wEAAAAAIGOy+gx6aWmp1q1bp5UrV7YoTigUUigUck2LRSPy+bO6+AAAAACyCD+zhkzL2jPoZWVleuaZZ7R06VL169evRbEqKipUWFjoeny2bnGaMgUAAAAAoOWyboBumqbKyso0f/58LVmyRAMHDmxxzPLyclVXV7sefQafnoZsAQAAAHQY/MwaMizrrvEuLS3V3Llz9dRTT6mgoEBVVVWSpMLCQuXn50uSqqqqVFVVpQ0bNkiS3nrrLRUUFKh///5N3kwuGAwqGAy6pnF5OwAAAAAgm2TdGfSZM2equrpaJSUl6t27d/wxb968+DyzZs3S0KFDdfXVV0uSRo4cqaFDh+rpp59urbQBAAAAtHOGmdkHkHWnkc0kflvwtttu02233Zb5ZAAAAAAA2E+yboAOAAAAAFmJs9zIsKy7xB0AAAAAgI6IM+gAAAAAkAS+J45M67ADdNO+diAWbflFBEbMfrZ3WDOWYhx7eV/EnpDQAZiGh2DNLJtqp+IsF7Erzhe2XkfzvAc0EusnXR2dHcfvs/7jqb6cEM4yTn2lWGHOrRTS0Ykb0cQJCc/JSqj3xLKmxCmnnWMSt5Boki9iJVcXzXHFbQkj0pKC7ckhGvO5XkuSzy5vNNUeNGbl5rMbiNcuI14yY08ZY6k0+IYxo9byzjZ0wiUdtkH9OPuNEzNlMSeJloVpyGmrLd03oxGrXeS0ZF936tquZDNg15vXBpGwbr+dTE6jzsM7X8RJ0n7yuEkNexuaPqdN2G/EUojn9Mv2sjG//TrFdmaGrW1o+K2AZvxA6T1WfJvZy9ZE81LKyYkT/xxgM1M6oCUG38frfYQxwtYCzR6TvDDdz6nW/Z792d1OpZbvR43e9pibEy5+DEliHc0x7GOk08/Up3rwaWr98eN4yzrFiF3OFh6KAKgDD9ABAAAAwBPOoCPDsu476BUVFTrhhBNUUFCg4uJiTZgwQevXr4+//+WXX+pHP/qRDj/8cOXn56t///66/vrrVV1d3YpZAwAAAADQMlk3QF++fLlKS0u1evVqLVq0SOFwWGPGjFFtba0k6bPPPtNnn32m3/3ud1q3bp3mzJmjhQsX6sorr2zlzAEAAAC0Z/wOOjIt6y5xX7hwoev1nDlzVFxcrLVr12rkyJEaPHiwnnjiifj7hx56qO688059//vfVyQSUSCQdUUCAAAAAGCfsn4061y6XlRUtNd5unbtyuAcAAAAQOakekdcIElZd4l7Q7FYTJMnT9aIESM0ePDgJufZtm2b7rjjDl1zzTX7OTsAAAAAANInq085l5aWat26dVq5cmWT7+/YsUNnnXWWjjrqKN12223NxgmFQgqFQq5psWhEPn9WFx8AAABAFuF74si0rD2DXlZWpmeeeUZLly5Vv379Gr2/c+dOnXHGGSooKND8+fOVk5PTbKyKigoVFha6HpvfWpzJ9AEAAAAA8CTrBuimaaqsrEzz58/XkiVLNHDgwEbz7NixQ2PGjFFubq6efvpp5eXl7TVmeXm5qqurXY/ex5yeqSIAAAAAaI/MDD/Q4WXdNd6lpaWaO3eunnrqKRUUFKiqqkqSVFhYqPz8/PjgfNeuXXrkkUe0Y8cO7dixQ5LUo0cP+f3+RjGDwaCCwaBrGpe3AwAAAACySdaNUmfOnClJKikpcU2fPXu2Lr/8cr322mt6+eWXJUmDBg1yzbNx40YNGDBgf6QJAAAAoIMxYq2dAdq7rBugm/v46YKSkpJ9zgMAAAAAaccwBBmWdd9BBwAAAACgI8q6M+gAAAAAkI34mTVkGmfQAQAAAADIAh32DLovYv35y/BZz51z6yVJX6UQK6cmYsWKWneQT/Uva0bUevbX2zmG7dcB624UMS9xjaYnB3ZZsYxI47vd75V9Q4wDcuskSZ+E7OkF3sK4cglZBfJF7WSd8iVZzkCdvQ1Na3mftRlUmLdbkrTNS30lzuu3JvhzrY3i9YYgwYC1XMjepgF7Gzrb2Iv4uu226mwLrw3NmT1QXWdPsX6e0Ah4b7BOLGfZ3Bq7fP7UGn/gy1pJUpeA1bCcbZkKp76c59wcK1jYqTezmZ0jgZNDXk648Xt2+4/mesupKLjL+o/9Oi9oxa5JLsye9dt9RGDXngbl98Uahk6e4c7RaWem4d6/9hkmsGfNwVyr8nbbzaFr0Gpznyebk9OM7Pbk5OZrvCmSZm770gpZ39u9Do+cOjfsavE5myDJemrI2Y98fquA9fb+FC9nkjnG90e7DRwU/ML9fgo3NPLVW0Fzau1t4GwLj/UW79bt5QNOo3DanYd4ift2fB3eq961cqdsfns7pNBNK2eXHaPeilEU8LpXWwK77RxynQZm97FOP+YhVny/SeiXDad/9dq+wvax1k7CdE7xpLAvOcdvX8SKmWeXb4fHben0hfrC+vRmxHrseS++HyUZNKFNOp9ROuXUN7PAPnJz6r9hI0+xrfq3Wb9W5A91kSTl2B1PPHSS28Bn99MNb+Xk1FPOdvv4m+SBzYnhtLMD7H7+o+RSadu4FxYyjDPoAAAAAABkgawboFdUVOiEE05QQUGBiouLNWHCBK1fv941zw9+8AMdeuihys/PV48ePXTOOefonXfeaaWMAQAAAHQEhpnZB5B1A/Tly5ertLRUq1ev1qJFixQOhzVmzBjV1tbG5xk2bJhmz56tt99+W88//7xM09SYMWMUjaZyYRoAAAAAAK0v676DvnDhQtfrOXPmqLi4WGvXrtXIkSMlSddcc038/QEDBuhXv/qVhgwZog8//FCHHnrofs0XAAAAQAfBWW5kWNadQU9UXV0tSSoqKmry/draWs2ePVsDBw7UQQcdtD9TAwAAAAAgbbJ6gB6LxTR58mSNGDFCgwcPdr335z//WV26dFGXLl3073//W4sWLVJubpK3VAYAAAAAj/gOOjItqwfopaWlWrdunR577LFG711yySV6/fXXtXz5ch122GE6//zzVVdX10QUKRQKaceOHa5HLNqC33ECAAAAACDNsnaAXlZWpmeeeUZLly5Vv379Gr1fWFior33taxo5cqQef/xxvfPOO5o/f36TsSoqKlRYWOh6fPbfxZkuAgAAAID2xDQz+0CHl3UDdNM0VVZWpvnz52vJkiUaOHBgUsuYpqlQKNTk++Xl5aqurnY9+hx9erpTBwAAAAAgZVl3F/fS0lLNnTtXTz31lAoKClRVVSXJOmOen5+vDz74QPPmzdOYMWPUo0cPffLJJ7rrrruUn5+vM888s8mYwWBQwWDQNc3nz7qiAwAAAMhifE8cmZZ1Z9Bnzpyp6upqlZSUqHfv3vHHvHnzJEl5eXl68cUXdeaZZ2rQoEG64IILVFBQoJdeeknFxcWtnD0AAAAAAKnJutPI5j6+e9GnTx8999xz+ykbAAAAALBxBh0ZlnVn0AEAAAAA6Iiy7gw6AAAAAGQjvoOOTOMMOgAAAAAAWaDDnkH3h6w/f/l81rPRgj+HGRE7Rgv/ombE7P/Yz4b9ffyAPypJqvcQyzSanu6Lppik/aecTgH7p+yaiZ8Mp558YXtCiikZdlmcshpWNalLTr1renLB3LnFJ6dYztxARNKeMgZyrNfRFGLF7L3U8LWwndnLGYltoCUN164fX3280brWlXSYiLV8Z6d9xfYy875i2cs62z8/x9oItSkWs1veLknSpgZtwSmm16rL99t7sR0rz24nNR5zilezf09STl/mVXw/ccrkN5t+fx98DebrErS243Y7VJ4/4imnxHpN3MdTEauxatlnp2Kk2Mbi/Y6dY8xvPZvx/TP5TiNeLrvATt2nXE47Xq9AtSSpNhbcy8z7CBVLeE6xn4iXxa4ff73peu0pltPuW3D8sQLJFcgfsArp91vPqVS/EbPbhb0jFPp3pZZaxF0vTr0H/Ck0WCdUwrYzUzw142xLZx9qybHDyaHRtkzoj5IV+XybHfew+DSft26ncS52DgFfap2F0dT6U227MavynTLl+FPrJJzjRLRhHk4/FLZjJln38b7BXizgTEjxWNSmxDpAGdGqOIMOAAAAAEAWyLoBekVFhU444QQVFBSouLhYEyZM0Pr165uc1zRNjRs3ToZhaMGCBfs3UQAAAAAdi5nhBzq8rBugL1++XKWlpVq9erUWLVqkcDisMWPGqLa2ttG806dPl5HqNcgAAAAAAGSRrPsO+sKFC12v58yZo+LiYq1du1YjR46MT6+srNQ999yjNWvWqHfv3vs7TQAAAAAdDHdxR6Zl3QA9UXW1dbOboqKi+LRdu3bp4osv1owZM9SrV6/WSg0AAABAR2IyQkdmZd0l7g3FYjFNnjxZI0aM0ODBg+PTp0yZopNPPlnnnHNOK2YHAAAAAED6ZPUZ9NLSUq1bt04rV66MT3v66ae1ZMkSvf7660nHCYVCCoVCrmmxaEQ+f1YXHwAAAEAW4RJ3ZFrWnkEvKyvTM888o6VLl6pfv37x6UuWLNH777+vAw44QIFAQIGANcg+77zzVFJS0mSsiooKFRYWuh6frF+yP4oBAAAAABkzY8YMDRgwQHl5eTrppJP0yiuvNDvvAw88oG9+85s68MADdeCBB2r06NF7nR/7X9YN0E3TVFlZmebPn68lS5Zo4MCBrvdvvvlmvfnmm6qsrIw/JOn3v/+9Zs+e3WTM8vJyVVdXux79Dj8t00UBAAAA0J5k2c+szZs3T1OnTtW0adP02muvaciQIRo7dqy2bt3a5PzLli3TRRddpKVLl2rVqlU66KCDNGbMGH366afeV46MyLprvEtLSzV37lw99dRTKigoUFVVlSSpsLBQ+fn56tWrV5M3huvfv3+jwbwjGAwqGAy6pnF5OwAAAIC27N5779XVV1+tSZMmSZJmzZqlZ599Vg899JBuvvnmRvM/+uijrtd/+ctf9MQTT2jx4sW67LLL9kvO2LusO4M+c+ZMVVdXq6SkRL17944/5s2b19qpAQAAAOjADNPM6MOL+vp6rV27VqNHj45P8/l8Gj16tFatWpVUjF27dikcDrt+MQutK+tOI5sp/HRBKssAAAAAQDZp6ubWTV0NLEnbtm1TNBpVz549XdN79uypd955J6n13XTTTerTp49rkI/WlXVn0AEAAAAgK8Uy+2jq5tYVFRUZKcpdd92lxx57TPPnz1deXl5G1gHvsu4MOgAAAAB0ROXl5Zo6daprWlNnzyWpe/fu8vv92rJli2v6li1bmrxnV0O/+93vdNddd+mFF17Qscce27KkkVacQQcAAACAJGT6O+jBYFBdu3Z1PZoboOfm5mrYsGFavHhxfFosFtPixYs1fPjwZsvwm9/8RnfccYcWLlyo448/Pu11hJbpsGfQYzmG9RyznmvrrYZvGt5jhQusajTtP3fEnz3GcpaL5VjP0VwrQCTq9xzPcL6Wn7BMJOhzrcNrbjXhPNfyqdSXs1DU7mtiTiv0GCsatBdIqLeacG4KSblSk+x2EY1YwX0e/5QVCrvbRCRq/8efQlJ+05WcmZCL120QKbQvYUooq6c48WXtmPlOozfd7ycpWpgvSdoZsXIz7XpKpX3F68detj4SSCknp13WRRt3k857idui2ZzsdX9V39mVS52dm9dyOm093HlPArFUtqOa6CucAB4DxaJ7cnHav/NzMTs89q9mQvuKryOQ8L4H/qIDrRgt6bskRfOtxhmLRq3nxO4mhbhRu+5Mu7ymx34ivsmi1n+qIoWSpK8inV3ve8rJOf7Y3YWzfQ2vsZxtaLcF59gb7888xIslHmPtZ8PdRSbNsHNzyhbzx/Yy994l9oFbwoWu5JLNLZpnxYnk2fVjFzoc8f45IN6PRt11HW9fye6Pzn+cIjpdohMwlfZlH7+dXOoS+oxkYzp9gr+w0BVP8v45J96OfO4ca1P8TOHkEm3QNyrFthrr1tWKZe+XuyMeC2eL2u2oYf3GP3N2tfrpmMeRgVOW7fV5TU5H5k2dOlUTJ07U8ccfrxNPPFHTp09XbW1t/K7ul112mfr27Ru/TP7uu+/Wrbfeqrlz52rAgAHxX8zq0qWLunTp0mrlwB4ddoAOAAAAAJ5k2b2pL7jgAn3++ee69dZbVVVVpeOOO04LFy6M3zhu06ZN8jU40zRz5kzV19fru9/9rivOtGnTdNttt+3P1NGMrLvEvaKiQieccIIKCgpUXFysCRMmaP369a55SkpKZBiG63Httde2UsYAAAAA0DrKysr00UcfKRQK6eWXX9ZJJ50Uf2/ZsmWaM2dO/PWHH34o0zQbPRicZ4+sG6AvX75cpaWlWr16tRYtWqRwOKwxY8aotrbWNd/VV1+tzZs3xx+/+c1vWiljAAAAAB2CaWb2gQ4v6y5xX7hwoev1nDlzVFxcrLVr12rkyJHx6Z06ddrn3QkBAAAAAGgrsu4MeqLq6mpJUlFRkWv6o48+qu7du2vw4MEqLy/Xrl27WiM9AAAAAB2EYWb2AWTdGfSGYrGYJk+erBEjRmjw4MHx6RdffLEOPvhg9enTR2+++aZuuukmrV+/Xk8++WQrZgsAAAAAQOqyeoBeWlqqdevWaeXKla7p11xzTfz/xxxzjHr37q3TTz9d77//vg499NBGcUKhkEKhkGtaLBqRz5/VxQcAAACQTfieODIsay9xLysr0zPPPKOlS5eqX79+e53XuVPhhg0bmny/oqJChYWFrsen/1uc9pwBAAAAAEhV1g3QTdNUWVmZ5s+fryVLlmjgwIH7XKayslKS1Lt37ybfLy8vV3V1tevR96jT05k2AAAAgHbOiGX2AWTdNd6lpaWaO3eunnrqKRUUFKiqqkqSVFhYqPz8fL3//vuaO3euzjzzTHXr1k1vvvmmpkyZopEjR+rYY49tMmYwGFQwGHRN4/J2AAAAAEA2ybpR6syZMyVJJSUlrumzZ8/W5ZdfrtzcXL3wwguaPn26amtrddBBB+m8887TL37xi1bIFgAAAECHwXfQkWFZN0A399HoDzroIC1fvnw/ZQMAAAAANsbnyLCs+w46AAAAAAAdUdadQQcAAACAbGRwiTsyjDPoAAAAAABkgQ57Bt0Xtv765fNl0V/BnFQM+8l+bRjeczQNdyxHPGaKxQ74ok3GTYURixcwteXtVMyExX2pFk4N6qWFzcKf0K6cIqYS1kz8M1qquaVhm2UkVgMB+/dFWrAJ4/XjtIscfzS1OPbynQJh12vX/z3mGd9/7PaRcltN6CuklvdlTjvz+Vr+Gy/x9m/nl3I5m+m/UmGGI+mJZZ85MZrpY1NhJG47r7kl5JBjpNjmG4ZM06HRiDkHNHtCvO16X4GR8B8nhJlq23fi2G3ebz9HmpvfgwJ/XWopNVOUVD4H7Fk4IXaqbdaX0Lm2QGKIFpVPklkXamKixxjN9OuBFH/3qsndMNWqS/iclGpOTiMwmzyepbYN0tVXtCmcQUeGcQYdAAAAAIAskHUD9IqKCp1wwgkqKChQcXGxJkyYoPXr1zeab9WqVTrttNPUuXNnde3aVSNHjtTu3btbIWMAAAAAHUIsww90eFk3QF++fLlKS0u1evVqLVq0SOFwWGPGjFFtbW18nlWrVumMM87QmDFj9Morr+jVV19VWVmZfL6sKw4AAAAAAEnJuu+gL1y40PV6zpw5Ki4u1tq1azVy5EhJ0pQpU3T99dfr5ptvjs93+OGH79c8AQAAAHQs3MUdmZb1p5yrq6slSUVFRZKkrVu36uWXX1ZxcbFOPvlk9ezZU6eeeqpWrlzZmmkCAAAAANAiWT1Aj8Vimjx5skaMGKHBgwdLkj744ANJ0m233aarr75aCxcu1Ne//nWdfvrpeu+991ozXQAAAADtmWlm9oEOL+sucW+otLRU69atc50dj8Wsuyf84Ac/0KRJkyRJQ4cO1eLFi/XQQw+poqKiUZxQKKRQyP0THLFoRD5/VhcfAAAAANCBZO0Z9LKyMj3zzDNaunSp+vXrF5/eu3dvSdJRRx3lmv/II4/Upk2bmoxVUVGhwsJC1+OT9UsylzwAAACA9ocz6MiwrBugm6apsrIyzZ8/X0uWLNHAgQNd7w8YMEB9+vRp9NNr7777rg4++OAmY5aXl6u6utr16Hf4aRkrAwAAAAAAXmXdNd6lpaWaO3eunnrqKRUUFKiqqkqSVFhYqPz8fBmGoRtvvFHTpk3TkCFDdNxxx+nhhx/WO++8o8cff7zJmMFgUMFg0DWNy9sBAAAAeMJvlSPDsm6UOnPmTElSSUmJa/rs2bN1+eWXS5ImT56suro6TZkyRV9++aWGDBmiRYsW6dBDD93P2QIAAAAAkB5ZN0A3k/zuxc033+z6HXQAAAAAyCR+Bx2ZlnXfQQcAAAAAoCPKujPoAAAAAJCVOIOODOMMOgAAAAAAWYAz6AAAAACQDM6gI8M4gw4AAAAAQBbIugF6RUWFTjjhBBUUFKi4uFgTJkzQ+vXr4+9/+OGHMgyjycc///nPVswcAAAAQLtmmpl9oMPLugH68uXLVVpaqtWrV2vRokUKh8MaM2aMamtrJUkHHXSQNm/e7Hrcfvvt6tKli8aNG9fK2QMAAAAAkJqs+w76woULXa/nzJmj4uJirV27ViNHjpTf71evXr1c88yfP1/nn3++unTpsj9TBQAAANCRxFo7AbR3WTdAT1RdXS1JKioqavL9tWvXqrKyUjNmzNifaQEAAADoYAwuQ0eGZd0l7g3FYjFNnjxZI0aM0ODBg5uc58EHH9SRRx6pk08+eT9nBwAAAABA+mT1GfTS0lKtW7dOK1eubPL93bt3a+7cubrlllv2GicUCikUCrmmxaIR+fxZXXwAAAAA2YQz6MiwrD2DXlZWpmeeeUZLly5Vv379mpzn8ccf165du3TZZZftNVZFRYUKCwtdj0/WL8lE2gAAAAAApCTrBuimaaqsrEzz58/XkiVLNHDgwGbnffDBB3X22WerR48ee41ZXl6u6upq16Pf4aelO3UAAAAA7VnMzOwDHV7WXeNdWlqquXPn6qmnnlJBQYGqqqokSYWFhcrPz4/Pt2HDBq1YsULPPffcPmMGg0EFg0HXNC5vBwAAAABkk6w7gz5z5kxVV1erpKREvXv3jj/mzZvnmu+hhx5Sv379NGbMmFbKFAAAAECHYpqZfaDDy7rTyGaSDfPXv/61fv3rX2c4GwAAAAAA9o+sG6ADAAAAQFbiLDcyLOsucQcAAAAAoCPiDDoAAAAAJIMz6MiwDjtAj+UY1nPMaOVMGnBSsfd7035tmt5zNJy+w3l2YiU8exWJ+d1xW8D0OcmkuLydipGwfCzVwqlBvbSwWUQT2lVL+nIjZseIT0gxUDqPJxk6NkVM66KeFmzCeP047SIc9acWx15+VyTH9dr1f495xvcfu32k3FYT+gqp5X1ZvJ3FWn5hVbz92/mlXM6EdtaSdmHkuA93KccyrAXNxD62BczEbec1t4QcwmaKbb5hyDQdGk2fc0CzJ8TbrvcVmAn/cUIYqbZ9J47d5qNpaPuOndG8lJZrrlpS+RywZ+GE2Km2Waee09A2Eo/bLSqfJCMv2MREjzGa6ded45JXTe6GqdZ9woeIVHNyGoHR5PEstW2Qrr4CwB5Zd4l7RUWFTjjhBBUUFKi4uFgTJkzQ+vXrXfNUVVXp0ksvVa9evdS5c2d9/etf1xNPPNFKGQMAAADoEPgddGRY1g3Qly9frtLSUq1evVqLFi1SOBzWmDFjVFtbG5/nsssu0/r16/X000/rrbfe0rnnnqvzzz9fr7/+eitmDgAAAABA6rLuEveFCxe6Xs+ZM0fFxcVau3atRo4cKUl66aWXNHPmTJ144omSpF/84hf6/e9/r7Vr12ro0KH7PWcAAAAAHYAZa+0M0M5l3Rn0RNXV1ZKkoqKi+LSTTz5Z8+bN05dffqlYLKbHHntMdXV1KikpaaUsAQAAAABomaw7g95QLBbT5MmTNWLECA0ePDg+/R//+IcuuOACdevWTYFAQJ06ddL8+fM1aNCgVswWAAAAQLvGXdyRYVk9QC8tLdW6deu0cuVK1/RbbrlF27dv1wsvvKDu3btrwYIFOv/88/Xiiy/qmGOOaRQnFAopFAq5psWiEfn8WV18AAAAAEAHkrUj1LKyMj3zzDNasWKF+vXrF5/+/vvv609/+pPWrVuno48+WpI0ZMgQvfjii5oxY4ZmzZrVKFZFRYVuv/1217S+R39LBx0zNrOFAAAAANB+cKd1ZFjWfQfdNE2VlZVp/vz5WrJkiQYOHOh6f9euXZIkn8+dut/vVyzW9E0bysvLVV1d7Xr0Per0zBQAAAAAAIAUZN0Z9NLSUs2dO1dPPfWUCgoKVFVVJUkqLCxUfn6+jjjiCA0aNEg/+MEP9Lvf/U7dunXTggULtGjRIj3zzDNNxgwGgwoGg65pXN4OAAAAwBO+g44My7oz6DNnzlR1dbVKSkrUu3fv+GPevHmSpJycHD333HPq0aOHxo8fr2OPPVZ//etf9fDDD+vMM89s5ewBAAAAAEhN1p1GNpP4q9TXvvY1PfHEE/shGwAAAACwcQYdGZZ1Z9ABAAAAAOiIsu4MOgAAAABkJc6gI8MYoAMAAABAMpr51SggXTrsAD3cyZAkmTH72f5jmJHCH8WiuVaMmN+e4EstVizHjmffcN4fsp4jISuwz0s8u1xKWCZil9szO872+jzrpd1yUqsvq2OLBJv5hkWSKUaD1oxG1Hods3PaUlPgPSmHk5JdrlgktW+B1IWtjRnNt15HdudKkvJSiGXU2xUSttuqk5K9jY0kG4bTvmK57jKZETtOCtvSjFrL1nf12a/tg5bHYOGuVqPfFcm1c0wpjJWDvR+adr3s3B20X8tbUMMq22c7utoB97zl5Jc0exM6+4+jdlfQU0rx9dtl9NfvWTAa86UUK97Wc6z/mPWpxTEbfF6prrUbvh2ztt5bOfdsq//f3n2HRXFufwA/u0tZem9KswOxE0HQCBIUlCgYry1RrNEYsSUae+9Xb2LXG6MYsXeDCVbU/OyKiiXYKGoULKCIIEX3+/sDdy4rZWdhFRLP53nm0d139+yZZXZmzjsz77z5V1b4RoVeBZZVOysiInr1JjXhb6jhKvGVQeEbdHQLVzyyl5rnIlCupvFmvvIKZ1y53tA0N3oT50aOPRERKd481jgO/e93pJxfJU2/+1eGhW+QFBTOW6656rZJo3jKbeubZU36WrVZ9PKl/M+b5fP1m9zwZiNSnq3k6zfLk+6Lwn+zFG9+68q/7dsb41Iov2+8tXeWX/AmNw2+L2FdqFD93UjKWVu8Nix8Y4Hxm3XE63LuT5DwtQhftkSZnIa5Qbldc3MhIiJpwf/aXilXt2K/NIXq/CjXQ/mvyrerrFwmZLL/zVRBOfc1FQaFwZTzJH0T4O3vUR3lelplGXiTS4GxZvOpXEYVb/ZZlesbYbmQ8VFmxsrrgy3QGWOMMcYYY0wjfIo7e8eq3CBxc+bMoWbNmpGJiQnZ2tpSWFgY3bhxQ+U1iYmJ1KlTJ7KxsSFTU1Pq2rUrPXz4sJIyZowxxhhjjDHGKq7KFejHjh2jIUOG0OnTp+ngwYNUUFBAbdu2pezsbCIiys7OprZt25JEIqHY2Fg6ceIE5efnU4cOHUjB14QwxhhjjDHG3hXg3U7sg1flTnHft2+fyuO1a9eSra0txcXFUatWrejEiROUkpJCFy9eJFPTwmtDf/nlF7KwsKDY2FgKDAysjLQZY4wxxhhjjLEKqXJH0N+WmZlJRESWlpZERJSXl0cSiYT09fWF18jlcpJKpXT8+PFKyZExxhhjjDH2AVDg3U7sg1elC3SFQkEjRoygFi1aUP369YmIqHnz5mRkZERjxoyhnJwcys7OplGjRtHr168pNTW1kjNmjDHGGGOMMcbKp0oX6EOGDKGrV6/S5s2bhedsbGxo27ZtFB0dTcbGxmRmZkbPnj2jpk2bklRa8uzk5eXR8+fPVSbF61fvazYYY4wxxhhj/wCA4p1OjFXZAj0iIoL27t1LR44cIUdHR5W2tm3bUmJiIj169IiePHlCUVFRdP/+fapZs2aJsebMmUNmZmYqU1r84fcxG4wxxhhjjDHGmChVrkAHQBEREbRr1y6KjY2lGjVqlPpaa2trMjc3p9jYWHr06BF17NixxNeNGzeOMjMzVSb7Rp++q1lgjDHGGGOM/RPxNejsHatyo7gPGTKENm7cSHv27CETExNKS0sjIiIzMzMyMDAgIqLIyEhyd3cnGxsbOnXqFA0fPpxGjhxJ9erVKzGmvr6+yqByRERSWZWbdcYYY4wxxhhjH7AqV6WuWLGCiIj8/f1Vno+MjKQ+ffoQEdGNGzdo3LhxlJGRQa6urjRhwgQaOXLke86UMcYYY4wx9kHhe5Wzd6zKFegQsdDPnTuX5s6d+x6yYYwxxhhjjDHG3o8qV6AzxhhjjDHGWJWk4JHW2btV5QaJY4wxxhhjjDHGPkR8BJ0xxhhjjDHGxOBr0Nk79uEW6JJ3EPLN71VRzthQvk8b5zVIUOJDVHC+pW8CVSiOls7bKC0H5XpTosH6s6LfyzulzK0qJvkmpYqmhjfLhFSTP1ppsd76HUkqGBPv4ntXfm/lTU1YJoo8pYXvrsSPEhlW5WUV/MqEz3x7+dLCLFZ4WX3zfom0MBmtLh1vgqOC60jZmy9KUYHs3t5mlHv5EgKVO5USYpJKzHL/TbX4mxG+pzchZeWc4XfyM377+ynnZyiXy3eySqzoflOJQcsXU/n9SLR0FrNUWvFAwjrhzb8ybSVXlFS5EJf37Vy0MqYtH26BzhhjjDHGGGMaAF+Dzt6xKncN+ooVK6hhw4ZkampKpqam5OPjQzExMUJ7bm4uDRkyhKysrMjY2Jg6d+5MDx8+rMSMGWOMMcYYY4yxiqtyBbqjoyPNnTuX4uLi6Pz58xQQEEChoaF07do1IiIaOXIkRUdH07Zt2+jYsWP04MED+vzzzys5a8YYY4wxxtg/HvBuJ/bBq3KnuHfo0EHl8axZs2jFihV0+vRpcnR0pNWrV9PGjRspICCAiIgiIyPJ3d2dTp8+Tc2bN6+MlBljjDHGGGOMsQqrckfQi3r9+jVt3ryZsrOzycfHh+Li4qigoIACAwOF17i5uZGzszOdOnWqEjNljDHGGGOM/eMp8G4n9sGrckfQiYiuXLlCPj4+lJubS8bGxrRr1y7y8PCgS5cukZ6eHpmbm6u83s7OjtLS0ionWcYYY4wxxtiHATxIHHu3qmSBXq9ePbp06RJlZmbS9u3bqXfv3nTs2LFyx8vLy6O8vDyV5xSvX5FUViVnnzHGGGOMMcbYB6hKnuKup6dHtWvXJk9PT5ozZw41atSIFi1aRPb29pSfn0/Pnj1Tef3Dhw/J3t6+1Hhz5swhMzMzlSkt/vA7ngvGGGOMMcbYPwkUeKcTY1WyQH+bQqGgvLw88vT0JF1dXTp8+H/F9Y0bN+ju3bvk4+NT6vvHjRtHmZmZKpN9o0/fR+qMMcYYY4wxxpgoVe4c73HjxlG7du3I2dmZsrKyaOPGjXT06FHav38/mZmZUf/+/enbb78lS0tLMjU1paFDh5KPj0+ZI7jr6+uTvr6+ynN8ejtjjDHGGGNMI3wNOnvHqlyV+ujRIwoPD6fU1FQyMzOjhg0b0v79+6lNmzZERPTjjz+SVCqlzp07U15eHgUFBdHy5csrOWvGGGOMMcYYY6xiqtwp7qtXr6aUlBTKy8ujR48e0aFDh4TinIhILpfTsmXLKCMjg7Kzs2nnzp1lXn/OGGOMMcYYY9pQFa9BX7ZsGbm6upJcLidvb286e/Zsma/ftm0bubm5kVwupwYNGtDvv/9ers9l70aVK9AZY4wxxhhjjKm3ZcsW+vbbb2nKlCl04cIFatSoEQUFBdGjR49KfP3JkyepR48e1L9/f7p48SKFhYVRWFgYXb169T1nzkrDBTpjjDHGGGOMiQHFu5009MMPP9BXX31Fffv2JQ8PD1q5ciUZGhrSmjVrSnz9okWLKDg4mEaPHk3u7u40Y8YMatq0KS1durSi3wzTEi7QGWOMMcYYY6wKyMvLo+fPn6tMeXl5Jb42Pz+f4uLiKDAwUHhOKpVSYGAgnTp1qsT3nDp1SuX1RERBQUGlvp5VAnygcnNzMWXKFOTm5laZWJwT58Q5cU6cE+fEOXFOnBPn9KHkxIqbMmUKiEhlmjJlSomvvX//PogIJ0+eVHl+9OjR8PLyKvE9urq62Lhxo8pzy5Ytg62trVbyZxX3wRbomZmZICJkZmZWmVicE+fEOXFOnBPnxDlxTpwT5/Sh5MSKy83NRWZmpspUWkcIF+j/TFXuNmuMMcYYY4wx9iHS19cnfX19Ua+1trYmmUxGDx8+VHn+4cOHpd7lyt7eXqPXs/ePr0FnjDHGGGOMsb8ZPT098vT0pMOHDwvPKRQKOnz4MPn4+JT4Hh8fH5XXExEdPHiw1Nez94+PoDPGGGOMMcbY39C3335LvXv3po8//pi8vLxo4cKFlJ2dTX379iUiovDwcKpevTrNmTOHiIiGDx9Ofn5+9J///IdCQkJo8+bNdP78efrpp58qczZYER9sga6vr09TpkwRfQrJ+4jFOXFOnBPnxDlxTpwT58Q5cU4fSk6s4rp160aPHz+myZMnU1paGjVu3Jj27dtHdnZ2RER09+5dkkr/d9K0r68vbdy4kSZOnEjjx4+nOnXq0O7du6l+/fqVNQvsLRIAqOwkGGOMMcYYY4yxDx1fg84YY4wxxhhjjFUBXKAzxhhjjDHGGGNVABfojDHGGGOMMcZYFcAFOmOMMcYYY4wxVgVwgc4YY4wxxhhjjFUBH8xt1p48eUJr1qyhU6dOUVpaGhER2dvbk6+vL/Xp04dsbGxExUlNTaUVK1bQ8ePHKTU1laRSKdWsWZPCwsKoT58+JJPJ3uVslGjHjh3Url07MjQ0fO+fLVZycjLdvn2bHBwcKvU2Dunp6XT58mVq1KgRWVpa0pMnT2j16tWUl5dHXbp0IXd390rLraLy8/Np9+7dJS7joaGhpKenV8kZMsZK8+TJE7K2ttZ63GfPntG2bdvo7t275OLiQl26dCEzM7MKxSwoKCBdXd0K5zZt2jQaMmSIVub74cOHlJeXR87OzhWOpW3Z2dkUFxdHrVq1EvX6V69e0bVr11TW4x4eHuX6zpWfXXR/pWnTpiSRSETHePToEV29epU8PT3JzMyMHj58SL/88gspFAoKCQmhBg0aiIoTFxdHnp6eGs8DY4x9cPABOHv2LCwsLFC9enX07t0b33//Pb7//nv07t0bjo6OsLS0xLlz59TGOXfuHMzMzODp6YmWLVtCJpOhV69e6NatG8zNzeHr64vnz5+/hzlSJZFIYGpqiq+++gqnT58ud5zc3Fzk5+cLj2/fvo3x48ejZ8+emDBhApKSkkTFGTx4MLKysgAAOTk56Ny5M6RSKSQSCaRSKVq3bi20l1fr1q2RkpKi0XvOnDkDMzMzSCQSWFhY4Pz586hRowbq1KmDWrVqwcDAAHFxcaLjvX79utTn79y5o1FupUlLS8O0adPUvu7WrVuoWbMm5HI5/Pz80LVrV3Tt2hV+fn6Qy+WoXbs2bt26Jfpzc3JysHr1avTt2xfBwcFo3749IiIicOjQIY3n4eHDhzh8+DCePXsmzNO8efMwZ84cXL58WVSM7du3Izs7W+PPFuPp06f46aefMHHiRKxatUrIszzy8/Nx8+bNCsUoqk+fPrh//77G73v16pXK49OnT+PYsWMqv2+x3vVy/uLFCxw7dkyj95w5cwYLFy7E2LFjMXbsWCxcuBBnzpzR+LMPHz6MadOm4euvv8Y333yDBQsW4ObNmxrHuXTpElavXo3ExEQAwNWrVzF48GAMGjQI+/btExVDKpUiICAAGzZsQG5ursY5KHXq1Anbtm0T8rC2toaNjQ28vb1hZ2cHe3t7/Pnnn6JibdmyBXl5ecLjJUuWwNnZGVKpFFZWVqLWTQCQmZlZbHr27Bl0dXVx5swZ4Tkxnj9/ji+//BLOzs4IDw9HXl4evvnmG2H70qpVK9Gxli1bhk8//RRdunQptm57/PgxatSoISqOOpcuXYJUKlX7utevX2PChAkwNzeHRCJRmczNzTFx4sRSf48lxRo9ejQMDQ0hlUqFbbBEIoGLiwt+/fVXUXGOHDkCIyMjSCQS2Nvb49KlS3B0dESdOnVQr1496OvrY//+/aJiSSQS1KpVC7NmzSrXeq00BQUFOHDgAH7++WccPHiw2PpPU9pej0+dOhWPHz+ucJy0tDSt7VtoW0FBQZXNjbG/ow+iQPf29sbAgQOhUCiKtSkUCgwcOBDNmzdXG6dFixaYOnWq8DgqKgre3t4AgIyMDDRu3BjDhg3TKLfo6GhMmjQJx48fB1C4w9iuXTsEBQXhv//9r6gYEokE06dPR5MmTSCRSPDRRx/hxx9/xJMnTzTKxc/PT9ixO378OPT19dGwYUN069YNTZo0gaGhIU6ePKk2jlQqxcOHDwEA48aNg6OjI2JjY5GdnY3jx4+jVq1aGDt2rKic9uzZU+Ikk8mwdOlS4bEYgYGBGDBgAJ4/f4758+fD0dERAwYMENr79u2LsLAwtXEyMzPRpUsXyOVy2NraYtKkSSo7BGlpaaJ2xsQQu2MXGBiI0NDQEndMMzMzERoairZt24r6zFu3bsHFxQW2trZwcnKCRCJBSEgIvL29IZPJ0KVLFxQUFIiKpa2dO211QgHaK2DmzZuHnJwcAIUF8XfffQc9PT1IpVLo6Oigb9++ogvi+Pj4EiddXV3s2rVLeKzOgwcP0KJFC8hkMrRq1QoZGRkICQkRdszr1q2LBw8eiMrpfS3nYpdxoLCzp2XLlkKR4eXlBS8vL7i4uEAikaBly5bCukddHC8vL+FvJZVK4enpCXt7e8hkMowePVp0/jt27IBMJoOVlRWMjY1x8OBBmJubIzAwEEFBQZDJZNiwYYPaOBKJBMHBwdDT04OFhQUiIiJw8eJF0XkoWVhYICEhAQDQrl07fPHFF0KRnZ+fj/79+4teFxRdl69ZswZyuRyTJ0/Gb7/9hpkzZ8LIyAirVq0SFaekSVlUK/8VIyIiAm5ubli8eDH8/f0RGhqK+vXr4/jx4zh27Bg8PDwwfvx4tXEWLVoEQ0NDDBkyBD179oSenh5mz54ttFfGenz06NGwsbHBypUrkZycjJycHOTk5CA5ORn//e9/YWtri++//17UZ44ZMwbu7u6Ijo7GwYMH0apVK8ybNw8JCQmYNGmS6HVvy5YtMWTIEGRlZWH+/PmoXr06hgwZIrSPGjUKvr6+onKSSCT46quvYGtrCx0dHYSEhGDXrl0aF9QRERGIjo4GANy7dw9ubm6QyWSws7ODTCZDgwYN8Ndff4mKpc31uLY6orTZCQW8n44oTdbjjDH1PogCXS6XCzssJUlISIBcLlcbx8DAQDhCAhT2UOvq6iItLQ0AcODAAVSrVk10XitXroSOjg48PT1hamqKqKgomJiYYMCAARg0aBAMDAywcOFCtXEkEomwE3X+/HkMHjwY5ubm0NfXR5cuXXDgwAFR+ZiamgpHj/z8/DBy5EiV9okTJ6JFixYa5VO/fn1s3LhRpX3Pnj2oW7euqJyK7sCVNondKFhYWAiFV35+PqRSqcpRt7i4OFSvXl1tnGHDhqFu3brYtm0bVq1aBRcXF4SEhAg7wWlpaZBIJKJyKq0wU05btmwRNX8GBga4cuVKqe2XL1+GgYGBqJzatWuHQYMGCR1ac+fORbt27QAAN2/ehKurK6ZMmSIqlrZ27rTVCQVor4ApWrzMnz8fFhYWWLNmDa5du4b169fD1tYW8+bNE5VTWcu5JgVMr1694Ovri19//RXdunWDr68vPvnkE/z111+4c+cOWrRoofL9l0Wby3lZNNmx69y5M3x8fHD9+vVibdevX4evry/+9a9/qY3TrVs3hIWFITMzE7m5uYiIiEB4eDiAwk5SKysrUeteAGjatClmzpwJANi0aRPMzc0xffp0oX3BggVo3Lix2jjK9ebjx4+xYMECeHh4QCqVomnTpli+fLnoHXIDAwPcvn0bAODg4IALFy6otN+4cQNmZmaiYhVdl3t5eeHf//63Svvy5cvRpEkTtXGqV6+OkJAQxMbG4ujRozh69CiOHDkCmUyGyMhI4TkxnJycEBsbCwC4f/8+JBKJUKwBwN69e1GvXj21cTw8PFQ6Tk6cOAEbGxtMmjQJgGYFuoWFRZmTqampqFh2dnZlnnGxb98+2NraisrJwcEBf/zxh/D4r7/+grGxsXB2xvTp0+Hj46M2jqmpqbA8FRQUQEdHR6Xj6ObNmxovTwUFBdi+fTvat28vFNbff/89bty4ISqOnZ2dsL3r2rUrAgMDhSPU6enp+Oyzz0StBwDtrse11RGlrU4o4P11RHGBzph2fRAFuqurK3755ZdS23/55Re4uLiojePi4iIc6QYKj1ZJJBKh9zU5OVlUoa/k4eGBn376CQAQGxsLuVyOZcuWCe2RkZFwd3dXG6foTpTSy5cvsW7dOvj7+0MqlcLV1VVtHCMjI6FwsbOzw6VLl1Tab9++DWNjY1H5PHr0CABgbW2Nq1evqrSnpKSILhaDg4MREhJSbP50dHRw7do1UTGUjIyMkJycLDw2NjZW6XC5c+eOqL+fs7Mzjhw5Ijx+/PgxvLy80LZtW+Tm5mq0wdNWYebg4KCyk/q2X3/9FQ4ODqJyMjQ0VDnNNy8vD7q6ukIxvHv3blHLE6C9nTttdUIB2itgiubUpEmTYme8rF+/Hh999JGonBo1aoSQkBAkJCQgJSUFKSkpSE5Oho6ODg4ePCg8p46DgwNOnToFoHBHVSKRqBwxOXz4MGrWrCkqJ20t59oqXoDC3+zbf6+izp8/L2odZWpqqrJeevHiBXR1dYUiOCoqSlSRB6iuVxQKBXR1dVUu3UhMTBS93nx7PXfy5En069cPJiYmMDQ0RK9evdTG8fb2FrYrTZo0wa5du1TaDxw4AHt7e7VxlDkVXZeXtE0wMTFRGyc9PR1hYWFo3bq1ypHN8qzH9fX1cffuXeGxoaGhSmGXkpICQ0NDtXEMDAxUtgcAcOXKFdjZ2WHs2LEarccNDQ3x3XffYe3atSVO06ZNExXL0NCwzMt+4uPjYWRkJConExOTYgcUdHR0kJqaCgC4du2aqO+p6DY8OzsbUqlUWMcoc7K2thaVU0nL+F9//YXp06ejZs2akEql+OSTT9TGkcvlwiV3jo6OxS5vuXLlSrlyquh6XFsdUdrqhAK01xHVpEmTMic3Nzcu0BnTog9ikLhRo0bRwIEDKS4ujj799FOys7MjosJBZQ4fPkyrVq2iBQsWqI0TFhZGX3/9Nc2fP5/09fVpxowZ5OfnRwYGBkREdOPGDapevbrovJKTkykoKIiIiFq3bk2vX79WGUTG39+fhgwZojZOSYO9yOVy6tWrF/Xq1Ytu375NkZGRauN4e3tTdHQ0ubm5Ua1atSg+Pp4aNWoktF+6dIksLS3FzBpNmjSJDA0NSSqV0oMHD+ijjz4S2tLT08nIyEhUnJiYGPrxxx/p448/puXLl9Nnn30m6n0lcXJyoqSkJHJ1dSUios2bN5ODg4PQnpqaKmqwosePH5OLi4vw2Nramg4dOkRBQUHUvn17+vnnn0XnZGlpSf/+97/p008/LbH92rVr1KFDB7VxBgwYQOHh4TRp0qQSl/GZM2fS0KFDReVkbm5OWVlZwuOcnBx69eqVMMhcw4YNKTU1VVQsPT09ys3NJaLCQewUCoXwmIjo5cuXGg985OnpSZ6envTDDz/Qtm3baM2aNRQcHEzOzs6UnJys9v0NGzak2NhYqlWrFtnb29OdO3eoSZMmQvudO3eE37Q6yt/e3bt3ydfXV6XN19dXVD5ERGfPnqXvv/+eOnfuTOvXr1fJp1q1airLW1mePn0qrIMsLS3J0NBQ5b21a9cW/bfT1nKel5dHgwcPLnUgqTt37tC0adNExdLX16fnz5+X2p6VlUX6+vqi4hRdb0qlUnr9+jW9evWKiAr/dikpKaJyMjExofT0dHJ1daVnz57Rq1evKD09XWhPT08nY2NjtXFKWo/7+PiQj48PLV68mDZv3kxr1qxRG2fSpEkUHh5Ourq6NGzYMBo5ciSlp6eTu7s73bhxg6ZMmUK9evUSNW9ERPv27SMzMzOSy+WUk5Oj0pabmytqsDFLS0vatWsXrVixgry8vGjBggXUo0cP0TkUZWVlRY8fPyYnJyciIgoNDSVzc3Oh/cWLF6KWAWtra7p3756wPSAiql+/PsXGxlJAQAA9ePBAdE6NGzcmJycn6t27d4nt8fHxopZxf39/GjVqFG3YsKHYtujJkyc0ZswY8vf3F5VTgwYNaNOmTTRhwgQiItq6dSsZGxuTvb09EREpFApR31OLFi1o7NixNHbsWFq3bh01bdqUZs6cSVu2bCGJREIzZsygjz/+WFROJS0r1atXp0mTJtGkSZPo8OHDopbxunXr0tmzZ6lGjRpkYmJSbJ2QlZVFCoVCVE5F86roevzy5cvUv39/mjFjBkVFRQnrYolEQl5eXuTh4SEqzqNHj6h27dpEVLj+NzAwoLp16wrt9evXp3v37omKlZycrDJPvr6+FBsbS4GBgVRQUEAjRowQFefPP/+k7t27U40aNUpsT01NpZs3b4qKxRgTobJ7CN6XzZs3w9vbGzo6OsIRSh0dHXh7e2PLli2iYmRlZaFr165CDF9fX5WB0/bv34+tW7eKzsnR0VE4BU3ZS/rbb78J7UePHoWjo6PaOCX1SpfHyZMnYWZmhilTpmDJkiWwtrbGxIkTsWHDBkyePBnm5uaiTvXy8/ODv7+/ML19jeKMGTPg5+enUW4XL16Eh4cHBg4ciOzs7HIdeZk6dSo2bdpUavv48ePx+eefq41Tr149lb+TUlZWFnx8fNCoUSPRPclt27bFjBkzSm2/dOmS6NOI586dCwcHB+Gou/IIvIODg+hT9ACgd+/e8PPzQ0JCApKSkoQxCJSOHj0KJycnUbFCQ0Px2Wef4fjx4xg4cCA+/vhjhISE4MWLF8jOzsa//vUvBAcHq41T9DTEkty6dUv0KX979+6FpaUlIiMjERkZCVdXV/z88884ceIE1qxZAycnJ1HXIEskEsyaNQuLFi2Cg4NDsYHO4uPjYWFhISonpd9//x2Ojo6YPXu2cMRLk+Xc2dlZ5WjSmDFjkJ6eLjy+dOmS6CNL2lrOfX19yzxdXJNTI7/55hu4uLhg586dKqd8Z2ZmYufOnXB1dUVERITaOJ06dULnzp3x4sUL5OfnY8SIEahdu7bQfvr0adFHmXv27Alvb2+sX78eHTp0QFBQEJo3b46EhARcv34dfn5+ok631dZ6HCgcVNHR0bHY2TlyuRwjRowQfc3v22f1KE/lV/r5559FneJe1LVr19CoUSP06NGjXOvx4OBgrFy5stT2yMhIUZfN9OjRAyNGjCix7erVq7CxsRG9XM6aNUtlfJq33b17F3369FEb5+7du6hfvz50dHTQpEkTBAcHIzg4GE2aNIGOjg4aNmyocvZAWQ4dOgR9fX14eXmhVatW0NHRwY8//ii0z58/HwEBAWrj3Lx5E3Xq1IFEIoG7uzv++usvdOzYETo6OtDR0YGNjY3owVW1tYxHRkbC0dERR44cwbp16+Du7o5Dhw7h/v37iI2NRYMGDVTGl1GXk7bX48uXL0e1atWEy/s0Xc6rVaum8p326NFD5Xu7evWq6JycnJxULnVQunbtGuzs7BAeHi5qOff09MTy5ctLbb948SIfQWdMiz6YAl0pPz8fDx48wIMHD8o1ojFQePp4RUchB4AhQ4agTp06mDlzJry8vNC7d2+4ubkhJiYG+/btQ4MGDdCvXz+1cVJSUkocAK88Tp48iebNmxfbMatevbroazJLo8wxMTER9+7d0/j9OTk5GDRoEOrUqQOZTKbxjp062dnZokZPjoiIKHWH+/nz5/D29ha9odq5cyeioqJKbc/IyMDatWtFxVJKSkrCyZMncfLkSdEj7xf18OFDlWXAxcVFZWdh27ZtWLx4sahY2tq502bxAmingHFxcYGrq6swFd35BYCFCxeKGnzybWlpaWjXrh0++eQTjXfsOnbsWObvdOnSpaJ2ygFg6NChWlnOtVW8AIV3mvj666+FQZzkcjnkcjmkUin09PQwePBgUb/hxMRE1KpVCzo6OtDV1YW5uTkOHjwotEdGRooeyDItLQ1t2rSBsbExgoKC8OzZM0RERAgdZXXq1BEuqSjL2rVrKzR6+9tevXqFs2fPYvPmzdi4cSOOHDmi9buMREdHix6lvqi8vDyMHDkSjRs31ngdlZ6ejqdPn5ba/vvvv6tcmlGa+Ph4rFmzptT2K1eulLncviuvX7/G77//jsmTJ2PgwIEYOHAgJk+ejJiYGNEjuCtdunQJ48ePx3fffafRZUAleXu8j0OHDiE6OlqjcUCOHj0qeoBRdf7zn//A0NAQBgYGwvpAOYWFhYneR3tX6/GKdERpqxMK0F5H1LBhwzB8+PBS22/fvg1/f39ROTHG1JMAQGUfxf9QZWdn08iRI+nUqVPk6+tLS5YsocWLF9OECROooKCA/Pz8aMuWLWRra/vec3v8+DElJSWRQqEgBwcHldMAy0tPT4/i4+MrfK/x6Ohoio2NpXHjxlXKd/P06dNip+0XlZWVRRcuXCA/P7/3mldqaiqtWLGCjh8/rnLP27CwMOrTpw/JZDKN4t26dYvy8vLI3d1d4/e+LT09naysrITHhw8fppcvX5KPj4/K86W5c+cOOTs7a3TvXnVev35NFy5cUFnOPT09ycTERCvxT58+Tfr6+iqnq2ti8eLFdOTIEVqyZAk5OjpqJaezZ8+SoaEh1a9fX+1rq+pyTkT0/PlziouLU7lPtKenJ5mamoqOkZOTQ8ePH6f8/Hxq3ry51u9BnpSURDk5OeTm5kY6Oh/E1WSMvVfPnj2jAwcOUHJysrAOb9GiBdWpU0drn1GR9Xh+fj6NHTuWjhw5Qjt37iz19PC3ZWRkkFQqVbl0o6iYmBgyMDAQdbnD5cuXKS4ujvr27Vti+9WrV2nHjh00ZcoUUbkxxt4PLtCroNzcXCooKNCoUHj58iXFxcWRpaVlseuccnNzaevWrRQeHq42TkJCAp0+fZp8fX2pXr16dP36dVq0aBHl5eVRz549KSAgQG2Mb7/9tsTnFy1aRD179hQKsh9++EHEnKnKzs6mrVu30u3bt8nBwYF69OghqsAjIrpw4QJZWFgIG8moqChauXIl3b17l1xcXCgiIoK6d++uNs7QoUOpa9eu9Mknn2ic/7ty/vx5CgwMpNq1a5OBgQGdOnWKvvjiC8rPz6f9+/eTh4cH7du3T/Qype1inzEmztmzZ+nUqVMqnQ8+Pj7k5eWllfhPnz6l6OhoUdsDJYVCQVKptNjzAOjevXvk7OysNgYASklJIScnJ9LR0aH8/HzatWsX5eXlUfv27SvcQRIQEECRkZGix2soSXJysrBtEdOJpZSXl0dSqVQYTyMxMZHWrFkjbFv69+8vqjjbsWMHtWvXjgwNDcs9D+8iVnx8PMXFxZG/vz/VrFmTrl27RsuWLSOFQkGdOnUSxtIRKzY2tti2pWPHjlotrBlj7G+tMg/fs7LdvXsXffv2Vfu6GzduCPcBVt4js+i9jsWO0hkTEwM9PT1YWlpCLpcjJiYGNjY2CAwMREBAAGQyGQ4fPqw2jkQiQePGjVWuQ/f394dEIkGzZs3g7++P1q1bq40DAO7u7sI1tHfv3oWLiwvMzMzQrFkzWFpawtbWVvQpkg0bNhROY121ahUMDAwwbNgwrFixAiNGjICxsTFWr14tav6Up67OnTtXGBW3vJYsWYJevXoJ18crr6mrV68exo0bJ+qUwBYtWqicjhkVFQVvb28AhafJN27cGMOGDROVz7lz52BmZgZPT0+0bNkSMpkMvXr1Qrdu3WBubg5fX1+NTpXNy8vDli1bMGLECHTv3h3du3fHiBEjsHXrVuGWXRWVlpaGadOmafSee/fulXgaZH5+frHrEEvz5MkTxMbGCsvo48ePMXfuXEybNk3UvdTVqVGjhsqI+ppSKBSIjY3FTz/9hOjoaI0u67l3755w6yIA+OOPP/DFF1+gZcuW+PLLL3Hy5ElRcRYsWCBqBHqxoqOjMWnSJOGOGocPH0a7du0QFBRUbATm9xEHKLz8ZvXq1ejbty+Cg4PRvn17REREFLvncFm0dY93dTS55j8zMxNdunSBXC6Hra0tJk2apHL5h9hty/Xr1+Hi4gKpVIratWsjKSkJnp6eMDIygqGhIaytrUUv53v27ClxkslkWLp0qfBYncGDBwu//5ycHHTu3FnlzhmtW7cWfZq0n58ftm3bBgA4fvw49PX10bBhQ2H8DkNDQ1G/F4lEAlNTU3z11Vc4ffq0qM9+17F27NgBmUwGKysrGBsb4+DBgzA3N0dgYCCCgoIgk8lURgkvy8OHD+Hl5SXcY1wqlcLT0xP29vaQyWSixv4o6syZM1i4cCHGjh2LsWPHYuHChcVGdK+ojIyMMu8CVJLSLkV4/fo17ty5IyqGQqFAUlKSsP3Py8vD5s2b8csvv6isl8urdevWFV4vJyUl4cCBA2Xe4pUxVj5coFdhYnekwsLCEBISgsePH+PWrVsICQlBjRo1hA2B2J0oHx8fTJgwAUDh/XwtLCxUBt4aO3Ys2rRpozbOnDlzUKNGjWLFfHkGBCp67fGXX34JX19fPHv2DEDhYFWBgYHo0aOHqFgGBgbCBqlJkybCrYiUNmzYAA8PD1E5HTp0CMOHD4e1tTV0dXXRsWNHREdHa3yN4IwZM2BiYoLOnTvD3t4ec+fOhZWVFWbOnInZs2fDxsYGkydPFjVvb99SR1dXF2lpaQAKb61UrVo1UTlps9i/desWatasCblcDj8/P3Tt2hVdu3aFn58f5HI5ateujVu3bomKVRZNio4HDx6gWbNmkEqlQudD0R1xsb+XM2fOwMzMDBKJBBYWFjh//jxq1KiBOnXqoFatWjAwMBA9eNKiRYtKnGQyGcaNGyc8Vqddu3bC7yM9PR3e3t6QSCTCdYZubm7CbbPU8fLyEm7ts3v3bkilUnTs2BFjxoxBp06doKurW+at/ZQkEglkMhkCAwOxefPmCnXKrFy5Ejo6OvD09ISpqSmioqJgYmKCAQMGYNCgQTAwMBA1Voa24gCFy7iLiwtsbW3h5OQEiUSCkJAQeHt7QyaToUuXLqI62bR1j/fMzMwyp//7v/8T/VsZNmwY6tati23btmHVqlVwcXFBSEiI8DdMS0sTNYhlaGgoOnbsiMuXL2PEiBFwd3dHaGgo8vPzkZubiw4dOqBnz56icirr1pRFb1GpTtGBJ8eNGwdHR0fExsYiOzsbx48fR61atUSPQ2Bqaip0MPj5+WHkyJEq7RMnTkSLFi1Ezdv06dPRpEkTSCQSfPTRR/jxxx81us5b27GaNm0qDA64adMmmJubY/r06UL7ggUL0LhxY1GxunXrhrCwMGRmZiI3NxcREREIDw8HUNhBZmVlJep39746swDu0FLSZocWY0w9LtArUWkrTuX0448/ilqZ29raqtw7VaFQ4Ouvv4azszMSExNFbxRMTU2FYkk5gnTRew4r7xErxtmzZ1G3bl189913wlG7ihboNWvWLDbQzYkTJ0SPKG5lZYXz588DKPzOSrqnr5j7sxfNKT8/H1u2bBGOJFSrVg3jx48XXXTWqlULO3bsAFC4IyCTybB+/XqhfefOnSqjS5fGxcVFOBIIFBahEokEOTk5AIDk5GRR93gHtFvsBwYGIjQ0VGXEbaXMzEyEhoaibdu2auPEx8eXOW3ZskX0TlR4eDi8vb1x7tw5HDx4EJ6envj444+RkZEBQHzRERgYiAEDBuD58+eYP38+HB0dVUYO7tu3L8LCwkTlJJFI4OjoqDJYkaurqzBAo6urK2rUqCEqjnLZHDx4MDw8PIQzTO7duwdPT098/fXXonIyMjIS3uvt7Y25c+eqtC9ZskTUCN4SiQSRkZEIDQ2Frq4urKysMHz48HIddfHw8BA61mJjYyGXy7Fs2TKhPTIyEu7u7u8tDlDYKTJo0CBhEMy5c+eiXbt2AAoHSXR1dcWUKVPUxtHWPd6L3sWhpElsAQsU3hWg6IBrjx8/hpeXF9q2bYvc3FzR2xYbGxtcvHgRQOE95yUSCf7v//5PaD9x4gScnZ1F5RQcHIyQkJBiBZim25eiv5X69esLI24r7dmzB3Xr1hUVy8jICAkJCQAAOzu7ErctYv92ypzOnz+PwYMHw9zcHPr6+ujSpYtGA71pK5aRkZFwv3iFQgFdXV2V/Y3ExERR8wYU7mMo76kOFC4Lurq6wvYhKipK1P29tdWZBXCHVmV0aDHG1OMCvRJpa8VpYmJS4um0Q4YMEW7lJrZALzrasLGxsUqhlpKSIrrIAwqPcIeHh6Nhw4a4cuUKdHV1y1WgK4/4VatWrdhOvSY59ezZE/379wcAdOnSBRMnTlRpnz17Nho0aCAqp5J65+/cuYMpU6YIPd9iGBgYqJzypqurq7IDk5KSAkNDQ7Vxhg8fjvr16yMmJgaxsbFo3bq1yoiq+/btQ61atUTlpO1iv6xC7PLly6I7RUr7rWhadFSrVk3lNEjlDk/jxo2Rnp4uuuiwsLAQfnf5+fmQSqUqcePi4lC9enVROQ0aNAiNGzcu9juuSNFRr169YkdGDh06JKrQBwAzMzPEx8cDKOzQUv5f6fbt26KWzaI5PXz4EPPmzYObmxukUimaNWuGn376SfQlEyX9XoouX8nJyaJy0lYcADA0NFQ5mpWXlwddXV3hSOXu3bvh6uqqNo6VlRWOHj1aavuRI0dgZWWlNo6pqSnmzZuHo0ePljitWrVKo/XT25cQPX/+HD4+PggICEBSUpKoWG9/38bGxirbmrt370JfX19UTgDwww8/wMnJSeUMjvL8VpTbFmtra5X1LlC47hWzbgKAgIAA/Pvf/wZQeFvBt0+J3r59u6gOiJK2LS9fvsS6devg7+8PqVQqalnSZix7e3uhYzsjIwMSiUSl0+bs2bOib0loY2Oj8jfKycmBVCoVLhFKTEwUtRxoqzML4A4tsbTZocUYU48L9EpUrVo17N69u9R2sfeVbNasGdatW1di25AhQ2Bubi4qTsOGDRETEyM8vnLlisqpmX/88YfonfuiNm3aBDs7O0il0nIV6A0aNECTJk1gbGyM7du3q7QfO3ZMdBF0//59uLq6olWrVvj2229hYGCAli1b4quvvkKrVq2gp6dX4n2fS8qprNPnFAqF6KMTNWrUEL7zmzdvQiqVYuvWrUL7b7/9JmonKisrC127doWOjg4kEgl8fX1Vdqz379+vErcs2iz2HRwcyjwN+tdff4WDg4PaOFZWVli9ejVSUlJKnH777TfRO1FGRkbFTg8sKChAWFgYGjZsiMuXL4uKVfTIElC8Q+vOnTsadWjt3LkTTk5OWLJkifBcRYoOW1vbEosOsYVQx44dhSMiQUFBxU6xX7VqFerUqSMqp5J+L3/88Qd69+4NIyMjGBkZicpJ2eEIFP6eJRKJym/26NGjcHR0fG9xgOL3LH769CkkEonQ6ZCUlCTqO9fWPd79/f0xb968UtsvXbok6igeUNjJU9I6MSsrCz4+PmjUqJGo30qtWrVUCozly5erdMrExcWJLvKULl68CA8PDwwcOBDZ2dnl+q0MGjQII0eOhK2tbbF1dlxcHKytrUXFOnnyJMzMzDBlyhQsWbIE1tbWmDhxIjZs2IDJkyfD3Ny8zL+JUtGjlCW5deuWymVn7yNWz5494e3tjfXr16NDhw4ICgpC8+bNkZCQgOvXr8PPz0/00epOnTqhc+fOePHiBfLz8zFixAiVM8ROnz4tajnQVmcWwB1aYmmzQ4sxph4X6JWoQ4cOmDRpUqntYnekZs+eLZxSWZLBgweLirNixQrs3bu31PZx48YJR6A1de/ePezevRsvXrzQ6H1Tp05Vmd6+5+6oUaPQvXt30fGePn2KMWPGwMPDA3K5HHp6enBxccEXX3yBc+fOiYrh6uparmsCSzJx4kTY2NhgwIABqFGjBsaOHQtnZ2esWLECK1euhJOTU7HrGcvy8uXLCl8Hps1if9KkSbCwsMAPP/yA+Ph4pKWlIS0tDfHx8fjhhx9gaWkp6vTftm3bYsaMGaW2a1J0NGjQoFhHD/C/It3Z2VnUTpSbm5vKOAt79+4VzjIACnc2xRZ5Sn/99RcCAgIQHByM1NTUcu1EtW/fHp06dYKFhUWxzpHTp0+Lvkzlzz//hJWVFcLDwzFjxgwYGxujZ8+emDVrFsLDw6Gvr4/IyEi1cdQVCpmZmcXGgyjNkCFDUKdOHcycORNeXl7o3bs33NzcEBMTg3379qFBgwbo16/fe4sDAL1794afnx8SEhKQlJQkDAymdPToUVGX4WjrHu8//fRTmeMVpKWlib6/99ChQ0stvp4/fw5vb29Rv5VBgwZh1apVpbbPmTMH7du3F5VTUTk5ORg0aBDq1KkDmUym0W/Fz89PZRDTt/ObMWMG/Pz8RMc7efIkmjdvXuwMn+rVq4sez0Bd568mtBUrLS0Nbdq0gbGxMYKCgvDs2TNERESoDJZatHgsS2JiImrVqgUdHR3o6urC3NxcGLgVKLy0RMxp0trqzAK4Q0ssbXZoMcbU4wK9Ev3xxx8qR6zf9uLFizJ7idnf3+vXrzFr1ix89tlnmD17NhQKBTZt2gQnJydYWVmhT58+GndqaIs2in2g8JpcBwcHlVMJJRIJHBwcRB1VAgqPLkdFRZXanpGRgbVr14qK9f3335d63XtBQQE6duwoaods6tSpwsj7JRk/fjw+//xzUTkVpVAoMHv2bGFkY012ovr06aMybdmyRaV99OjRCAoKEh3v9u3b6N69O0xMTISCQ1dXF76+vti1a5eoGNosOl68eIGvvvoK9evXx8CBA5GXl4f58+dDT08PEokE/v7+oj5LW3GAwtP2lYWZVCqFi4uLyum327Ztw+LFi0XPY2ZmJmJjY7Fx40Zs3LgRsbGxJY7h8D5kZGQUO1JW1PPnz7WyjUpKSlK584im9uzZgxEjRmhtOQMKi8l79+5p/L5Hjx7h9OnTOHnypMoZNmKkpKQIYxlUlDZjlSQxMbHYWXZiZGdnY//+/YiOji73aOTa6swCCju0yupA4Q6tQtru0GKMlY3vg84Yey+Sk5NV7u0s5r7A78KrV68oJyeHTE1NS22/f/9+he6nTESUk5NDMpmM9PX1y/X+uLg4On78OIWHh5OFhUWFclHKzs4mmUxGcrlco/cBoEePHpFCoSBra2vhfs9VRW5uLhUUFJCJiUmlxbl16xbl5eWRm5sb6ejoVCgPxph6z58/p7i4OJXtiqenZ6nr9nft6dOn9ODBA/roo49KbM/KyqILFy6Qn59fhT4nOTmZ5HI5OTg4lOv9v/76Kx05coTGjRtHtra2FcpFKSkpifT09MjR0VEr8Rj70EkrOwHG2IehRo0a5OPjQz4+PkJxfu/ePerXr1+FY2sSR0dHp8wduNTUVJo2bVqFc0pPT6fBgweX+/2enp40fPhwsrCw0Nr3lJGRQd98843G75NIJGRnZ0cODg5CcV4Zf7vSyOVyMjExqXCsisSpU6cO1a9fv1hxrkmsly9f0vHjx+nPP/8s1pabm0vr1q17r3E4J86pquaUkJBAO3bsIAcHB+rRowc1adKEtm7dSiNGjKDY2FjR+ShjRUZG0vXr14mI6Pr16zR48GDq16+fRrEsLCxIKpWWGuvcuXOii/OyckpOTtaoOH87Vt26denly5c0duxYjeZPGefGjRvFckpJSeHinDFtquQj+IyxD5gm95h9H3G0GYtzer9xtBmrMnK6ceOGcB9nqVSKVq1a4f79+0K72BGgS4pT9PRxsXG0GYtz4py0GSsmJgZ6enqwtLSEXC5HTEwMbGxsEBgYiICAAMhkMpXxQf5usf7pOTHG1OPz8Bhj78yvv/5aZntSUtJ7jcM5cU5VNacxY8ZQ/fr16fz58/Ts2TMaMWIEtWzZko4ePUrOzs6i8ykpTosWLTSOo81YnBPnpM1Y06dPp9GjR9PMmTNp8+bN9MUXX9DgwYNp1qxZREQ0btw4mjt3LgUEBPwtY/3Tc2KMiVDZPQSMsX+usu5fXvQ+5u8rDufEOVXVnGxtbXH58mXhsUKhwNdffw1nZ2ckJiaKPrqorTicE+dUVXMyNTXFrVu3ABQOtKqjo6MyMOOVK1dE362iKsb6p+fEGFOPr0FnjL0zDg4OtHPnTlIoFCVOFy5ceK9xOCfOqarm9PLlS5Xr1yUSCa1YsYI6dOhAfn5+dPPmzfcah3PinKpqTsr3EhFJpVKSy+VkZmYmtJmYmFBmZubfOtY/PSfGWNm4QGeMvTOenp4UFxdXartEIiGIuJGEtuJwTpxTVc3Jzc2Nzp8/X+z5pUuXUmhoKHXs2FFUPtqKwzlxTlU1J1dXV7p165bw+NSpUyqnyN+9e1f0IGpVMdY/PSfGmHpcoDPG3pnRo0eTr69vqe21a9emI0eOvLc4nBPnVFVz6tSpE23atKnEtqVLl1KPHj1EFfraisM5cU5VNafBgwfT69evhcdv3z0hJiZG9LXQVTHWPz0nxph6fB90xhhjjDHGGGOsCuAj6IwxxhhjjDHGWBXABTpjjDHGGGOMMVYFcIHOGGOMMcYYY4xVAVygM8YYY4wxxhhjVQAX6Iwxxv7x+vTpQ2FhYZWdBmOMMcZYmbhAZ4wxpqJPnz4kkUhIIpGQrq4u2dnZUZs2bWjNmjWkUCgqJaepU6cKOZU2McYYY4z93XGBzhhjrJjg4GBKTU2llJQUiomJodatW9Pw4cPps88+o1evXpX6voKCgneSz6hRoyg1NVWYHB0dafr06SrPMcYYY4z93XGBzhhjrBh9fX2yt7en6tWrU9OmTWn8+PG0Z88eiomJobVr1wqvk0gktGLFCurYsSMZGRnRrFmzaO3atWRubq4Sb/fu3cWOcs+cOZNsbW3JxMSEBgwYQGPHjqXGjRuXmI+xsTHZ29sLk0wmIxMTE+Hx48ePKSAggAwMDMjKyooGDhxIL168KHX+zp07RzY2NjRv3jwiInr27BkNGDCAbGxsyNTUlAICAig+Pl54/dSpU6lx48YUFRVFrq6uZGZmRt27d6esrCzhNdu3b6cGDRoIOQQGBlJ2drbIb5wxxhhjjAt0xhhjIgUEBFCjRo1o586dKs9PnTqVOnXqRFeuXKF+/fqJirVhwwaaNWsWzZs3j+Li4sjZ2ZlWrFhRrryys7MpKCiILCws6Ny5c7Rt2zY6dOgQRURElPj62NhYatOmDc2aNYvGjBlDRERdunShR48eUUxMDMXFxVHTpk3p008/pYyMDOF9iYmJtHv3btq7dy/t3buXjh07RnPnziUiotTUVOrRowf169ePEhIS6OjRo/T5558TgHLNE2OMMcY+TDqVnQBjjLG/Dzc3N7p8+bLKc1988QX17dtXozhLliyh/v37C++bPHkyHThwoMyj3qXZuHEj5ebm0rp168jIyIiIiJYuXUodOnSgefPmkZ2dnfDaXbt2UXh4OP3888/UrVs3IiI6fvw4nT17lh49ekT6+vpERLRgwQLavXs3bd++nQYOHEhERAqFgtauXUsmJiZERNSrVy86fPgwzZo1i1JTU+nVq1f0+eefk4uLCxERNWjQQON5YYwxxtiHjY+gM8YYEw1AsVPVP/74Y43j3Lhxg7y8vFSee/uxWAkJCdSoUSOhOCciatGiBSkUCrpx44bw3JkzZ6hLly4UFRUlFOdERPHx8fTixQuysrIiY2NjYUpOTqbExEThda6urkJxTkTk4OBAjx49IiKiRo0a0aeffkoNGjSgLl260KpVq+jp06flmh/GGGOMfbj4CDpjjDHREhISqEaNGirPFS2MiYikUmmxU7vf1eBxmqhVqxZZWVnRmjVrKCQkhHR1dYmI6MWLF+Tg4EBHjx4t9p6i19IrX68kkUiEUe1lMhkdPHiQTp48SQcOHKAlS5bQhAkT6MyZM8W+L8YYY4yx0vARdMYYY6LExsbSlStXqHPnzmW+zsbGhrKyslQGSLt06ZLKa+rVq0fnzp1Tee7tx2K5u7tTfHy8yuedOHGCpFIp1atXT3jO2tqaYmNj6fbt29S1a1eh06Bp06aUlpZGOjo6VLt2bZXJ2tpadB4SiYRatGhB06ZNo4sXL5Kenh7t2rWrXPPEGGOMsQ8TF+iMMcaKycvLo7S0NLp//z5duHCBZs+eTaGhofTZZ59ReHh4me/19vYmQ0NDGj9+PCUmJtLGjRtVRn4nIho6dCitXr2afvnlF7p16xbNnDmTLl++XK77mX/55Zckl8upd+/edPXqVTpy5AgNHTqUevXqpXL9ORGRra0txcbG0vXr16lHjx706tUrCgwMJB8fHwoLC6MDBw5QSkoKnTx5kiZMmEDnz58XlcOZM2do9uzZdP78ebp79y7t3LmTHj9+TO7u7hrPD2OMMcY+XFygM8YYK2bfvn3k4OBArq6uFBwcTEeOHKHFixfTnj17SCaTlfleS0tLWr9+Pf3+++/UoEED2rRpE02dOlXlNV9++SWNGzeORo0aRU2bNqXk5GTq06cPyeVyjXM1NDSk/fv3U0ZGBjVr1oz+9a9/0aeffkpLly4t8fX29vbC2QBffvklKRQK+v3336lVq1bUt29fqlu3LnXv3p3u3LlTrMAvjampKf3xxx/Uvn17qlu3Lk2cOJH+85//ULt27TSeH8YYY4x9uCTge8AwxhirAtq0aUP29vYUFRVV2akwxhhjjFUKHiSOMcbYe5eTk0MrV66koKAgkslktGnTJjp06BAdPHiwslNjjDHGGKs0fASdMcbYe/fy5Uvq0KEDXbx4kXJzc6levXo0ceJE+vzzzys7NcYYY4yxSsMFOmOMMcYYY4wxVgXwIHGMMcYYY4wxxlgVwAU6Y4wxxhhjjDFWBXCBzhhjjDHGGGOMVQFcoDPGGGOMMcYYY1UAF+iMMcYYY4wxxlgVwAU6Y4wxxhhjjDFWBXCBzhhjjDHGGGOMVQFcoDPGGGOMMcYYY1UAF+iMMcYYY4wxxlgVwAU6Y4wxxhhjjDFWBXCBzhhjjDHGGGOMVQFcoDPGGGOMMcYYY1UAF+iMMcYYY4wxxlgVwAU6Y4wxxhhjjDFWBfw/kqka5/QAPUMAAAAASUVORK5CYII=", "text/plain": [ "" ] }, - "metadata": {}, - "output_type": "display_data" + "metadata": {} }, { + "output_type": "display_data", "data": { "image/svg+xml": [ "\n", @@ -2133,45 +1837,30 @@ "" ] }, - "metadata": {}, - "output_type": "display_data" + "metadata": {} } ], - "source": [ - "from IPython.display import Image, SVG\n", - "\n", - "attention_plot = Image(att_path)\n", - "molecular_img = SVG(mol_path)\n", - "display(attention_plot)\n", - "display(molecular_img)" - ] + "id": "4mHWCbJmGMgG", + "execution_count": null }, { - "cell_type": "markdown", - "id": "1999d67d6d14b263", - "metadata": { - "id": "1999d67d6d14b263" - }, + "metadata": {}, "source": [ "The output images are saved in the `visualization` directory. The attention map shows how much each drug token attends to each protein token, while the molecule image highlights the atoms based on their attention values." - ] + ], + "cell_type": "markdown", + "id": "1999d67d6d14b263" }, { - "cell_type": "markdown", - "id": "eeb308c3", - "metadata": { - "id": "eeb308c3" - }, + "metadata": {}, "source": [ "## Extension Tasks" - ] + ], + "cell_type": "markdown", + "id": "eeb308c3" }, { - "cell_type": "markdown", - "id": "aa2a83d8", - "metadata": { - "id": "aa2a83d8" - }, + "metadata": {}, "source": [ "### Task 1\n", "\n", @@ -2184,14 +1873,12 @@ "Reload the dataset and re-run training and testing.\n", "\n", "> Tip: See if the model struggles more or less with the new dataset. It can reveal how generalisable DrugBAN is.\n" - ] + ], + "cell_type": "markdown", + "id": "aa2a83d8" }, { - "cell_type": "markdown", - "id": "c94f174c", - "metadata": { - "id": "c94f174c" - }, + "metadata": {}, "source": [ "### Task 2\n", "\n", @@ -2203,1734 +1890,9 @@ "cfg.merge_from_file(\"configs/non_DA_cross_domain.yaml\")\n", "```\n", ">Tip: Compare the results with and without domain adaptation to see how it affects model performance." - ] - } - ], - "metadata": { - "accelerator": "GPU", - "colab": { - "gpuType": "T4", - "provenance": [] - }, - "kernelspec": { - "display_name": "Python 3", - "name": "python3" - }, - "widgets": { - "application/vnd.jupyter.widget-state+json": { - "07d966a71f604f63b00707e6d3a0bfe6": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HBoxModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HBoxModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HBoxView", - "box_style": "", - "children": [ - "IPY_MODEL_37f06572693f4972b468c3997fd0687c", - "IPY_MODEL_a0cec4295099427bb8e97229bd49d620", - "IPY_MODEL_931700f44cb0491ca187cbc58dc67476" - ], - "layout": "IPY_MODEL_dd7ad2a05e22470ab00e2118ff994147" - } - }, - "12c8f4410cc74d4f822c779c724bce94": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HTMLModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_79368bfcf5cc4067a59a46c242a77e2d", - "placeholder": "​", - "style": "IPY_MODEL_8632ffdd1b5844f183cc28e824cd117e", - "value": " 2/2 [00:01<00:00,  1.07it/s]" - } - }, - "16cdd1f5e14e405490575177c52f4408": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "DescriptionStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "1aa1b891fe6b447586d3e87ee62768a2": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "1ed13da64943461ab42ab18495a6246b": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HTMLModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_8f16c1ee593d406c88c9bfcf32fdd3df", - "placeholder": "​", - "style": "IPY_MODEL_16cdd1f5e14e405490575177c52f4408", - "value": " 305/305 [01:07<00:00,  4.52it/s, v_num=2]" - } - }, - "2439bd152f3f40a190e80cb5156a8be7": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "2f468dcdec6d4c6db229fcab061fd0d8": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HTMLModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_301141b0856941cba6a0a1f496267c4e", - "placeholder": "​", - "style": "IPY_MODEL_b4f64adc9af34262bb35ea42f0f58899", - "value": "Validation DataLoader 0: 100%" - } - }, - "2fc478f35c3e48f2b0c13bd9a73f3dc6": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "ProgressStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "ProgressStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "bar_color": null, - "description_width": "" - } - }, - "301141b0856941cba6a0a1f496267c4e": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "304a1cadd7c048a2a34568301fb1c4dd": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "32de4bd89e034c35adc198247950d4bf": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "FloatProgressModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "FloatProgressModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "ProgressView", - "bar_style": "", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_941d8002127f499c9868abffea2a2429", - "max": 2, - "min": 0, - "orientation": "horizontal", - "style": "IPY_MODEL_2fc478f35c3e48f2b0c13bd9a73f3dc6", - "value": 2 - } - }, - "33a0e7cd7f7d49ffa57e962ca2bf0b66": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": "2", - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "37f06572693f4972b468c3997fd0687c": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HTMLModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_1aa1b891fe6b447586d3e87ee62768a2", - "placeholder": "​", - "style": "IPY_MODEL_8618e343c4f1435ebf099bd70605606b", - "value": "Validation DataLoader 0: 100%" - } - }, - "3c34cab162424e51b4dcf0008c39c7a0": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": "2", - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "4b0bcd88167b469da36e8433a4d47377": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "FloatProgressModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "FloatProgressModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "ProgressView", - "bar_style": "success", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_a80aad59381c42edaa2adb89d781ddd6", - "max": 305, - "min": 0, - "orientation": "horizontal", - "style": "IPY_MODEL_b71c219972474c2c89ed04fa48ec637d", - "value": 305 - } - }, - "530bec1df42241a9be6a34c561be4a36": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "FloatProgressModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "FloatProgressModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "ProgressView", - "bar_style": "", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_3c34cab162424e51b4dcf0008c39c7a0", - "max": 29, - "min": 0, - "orientation": "horizontal", - "style": "IPY_MODEL_b2e51980648f49f2963ba66745b82b57", - "value": 29 - } - }, - "5436ad6633f7491898361cb877e5c96a": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "DescriptionStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "635b3ed7aa264fa6907187152359a63f": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "64151477b0f4444b852363f72540296e": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "ProgressStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "ProgressStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "bar_color": null, - "description_width": "" - } - }, - "65bb75f11b1f4064b715a166bed1e215": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "6692877437d14f56bdf0bc4e8793bb4b": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": "inline-flex", - "flex": null, - "flex_flow": "row wrap", - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": "100%" - } - }, - "700bde0d67f64f9d8680c483d232fa5b": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HTMLModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_b9b44b6940884c359ebb60834280b401", - "placeholder": "​", - "style": "IPY_MODEL_807b512710c848eabb6816cf8e9ecc75", - "value": " 29/29 [00:02<00:00, 12.12it/s]" - } - }, - "79368bfcf5cc4067a59a46c242a77e2d": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "7d4fd3d9c5ed4cc6af20be7384a758ac": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": "inline-flex", - "flex": null, - "flex_flow": "row wrap", - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": "100%" - } - }, - "807b512710c848eabb6816cf8e9ecc75": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "DescriptionStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "8618e343c4f1435ebf099bd70605606b": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "DescriptionStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "8632ffdd1b5844f183cc28e824cd117e": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "DescriptionStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "8f16c1ee593d406c88c9bfcf32fdd3df": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "931700f44cb0491ca187cbc58dc67476": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HTMLModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_304a1cadd7c048a2a34568301fb1c4dd", - "placeholder": "​", - "style": "IPY_MODEL_e809f1b951f740ddabc3a3e56d4dd903", - "value": " 29/29 [00:02<00:00, 12.61it/s]" - } - }, - "941d8002127f499c9868abffea2a2429": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": "2", - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "98992d70c9d74d3fa794edf0dc9333b1": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HTMLModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_65bb75f11b1f4064b715a166bed1e215", - "placeholder": "​", - "style": "IPY_MODEL_a893676474004fb0a24023403f5acf46", - "value": "Sanity Checking DataLoader 0: 100%" - } - }, - "9a8deecaaef543fbb18d0f50b83b5abc": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HBoxModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HBoxModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HBoxView", - "box_style": "", - "children": [ - "IPY_MODEL_b113d7ba3f50417fb93de1118b4e4dec", - "IPY_MODEL_4b0bcd88167b469da36e8433a4d47377", - "IPY_MODEL_1ed13da64943461ab42ab18495a6246b" - ], - "layout": "IPY_MODEL_7d4fd3d9c5ed4cc6af20be7384a758ac" - } - }, - "9d5ebfa060ac49d6bd7a8c4837b4fc29": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "DescriptionStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "a0cec4295099427bb8e97229bd49d620": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "FloatProgressModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "FloatProgressModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "ProgressView", - "bar_style": "", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_33a0e7cd7f7d49ffa57e962ca2bf0b66", - "max": 29, - "min": 0, - "orientation": "horizontal", - "style": "IPY_MODEL_f223529cf08b4235b8e74ad1049c5a60", - "value": 29 - } - }, - "a3aa520f98b042f29f1fca44f24b61e0": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": "2", - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "a80aad59381c42edaa2adb89d781ddd6": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": "2", - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "a893676474004fb0a24023403f5acf46": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "DescriptionStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "abf0b363155c4b1fb8de818ae41606eb": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": "inline-flex", - "flex": null, - "flex_flow": "row wrap", - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": "hidden", - "width": "100%" - } - }, - "af4b74bd9f13458c8dd9b74ad2004783": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "DescriptionStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "b113d7ba3f50417fb93de1118b4e4dec": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HTMLModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_635b3ed7aa264fa6907187152359a63f", - "placeholder": "​", - "style": "IPY_MODEL_9d5ebfa060ac49d6bd7a8c4837b4fc29", - "value": "Epoch 1: 100%" - } - }, - "b2e51980648f49f2963ba66745b82b57": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "ProgressStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "ProgressStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "bar_color": null, - "description_width": "" - } - }, - "b4f64adc9af34262bb35ea42f0f58899": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "DescriptionStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "b71c219972474c2c89ed04fa48ec637d": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "ProgressStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "ProgressStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "bar_color": null, - "description_width": "" - } - }, - "b9b44b6940884c359ebb60834280b401": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "bc415b5a5635482eb20a65601866febf": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HBoxModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HBoxModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HBoxView", - "box_style": "", - "children": [ - "IPY_MODEL_98992d70c9d74d3fa794edf0dc9333b1", - "IPY_MODEL_32de4bd89e034c35adc198247950d4bf", - "IPY_MODEL_12c8f4410cc74d4f822c779c724bce94" - ], - "layout": "IPY_MODEL_fe6bdb21df9c4415b9899aaa96969502" - } - }, - "be7949a38cfe4129abd0583b3b9c080e": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HTMLModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_e9d78f7f6ffd4d22b7f85f65da6e54c3", - "placeholder": "​", - "style": "IPY_MODEL_5436ad6633f7491898361cb877e5c96a", - "value": " 29/29 [00:02<00:00, 10.86it/s]" - } - }, - "d10203b9631842348ce5ee323f56d8c3": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "FloatProgressModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "FloatProgressModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "ProgressView", - "bar_style": "success", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_a3aa520f98b042f29f1fca44f24b61e0", - "max": 29, - "min": 0, - "orientation": "horizontal", - "style": "IPY_MODEL_64151477b0f4444b852363f72540296e", - "value": 29 - } - }, - "d782f86576c9463eb171f866f007b9b3": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HBoxModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HBoxModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HBoxView", - "box_style": "", - "children": [ - "IPY_MODEL_f7b99f048d0b430b913f03382dcf6cae", - "IPY_MODEL_d10203b9631842348ce5ee323f56d8c3", - "IPY_MODEL_be7949a38cfe4129abd0583b3b9c080e" - ], - "layout": "IPY_MODEL_6692877437d14f56bdf0bc4e8793bb4b" - } - }, - "dd7ad2a05e22470ab00e2118ff994147": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": "inline-flex", - "flex": null, - "flex_flow": "row wrap", - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": "hidden", - "width": "100%" - } - }, - "e809f1b951f740ddabc3a3e56d4dd903": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "DescriptionStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "e9d78f7f6ffd4d22b7f85f65da6e54c3": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "f223529cf08b4235b8e74ad1049c5a60": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "ProgressStyleModel", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "ProgressStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "bar_color": null, - "description_width": "" - } - }, - "f60cfca35cab44e5801bbb2b04f9cc6f": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HBoxModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HBoxModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HBoxView", - "box_style": "", - "children": [ - "IPY_MODEL_2f468dcdec6d4c6db229fcab061fd0d8", - "IPY_MODEL_530bec1df42241a9be6a34c561be4a36", - "IPY_MODEL_700bde0d67f64f9d8680c483d232fa5b" - ], - "layout": "IPY_MODEL_abf0b363155c4b1fb8de818ae41606eb" - } - }, - "f7b99f048d0b430b913f03382dcf6cae": { - "model_module": "@jupyter-widgets/controls", - "model_module_version": "1.5.0", - "model_name": "HTMLModel", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_2439bd152f3f40a190e80cb5156a8be7", - "placeholder": "​", - "style": "IPY_MODEL_af4b74bd9f13458c8dd9b74ad2004783", - "value": "Testing DataLoader 0: 100%" - } - }, - "fe6bdb21df9c4415b9899aaa96969502": { - "model_module": "@jupyter-widgets/base", - "model_module_version": "1.2.0", - "model_name": "LayoutModel", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": "inline-flex", - "flex": null, - "flex_flow": "row wrap", - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": "hidden", - "width": "100%" - } - } - } + ], + "cell_type": "markdown", + "id": "c94f174c" } - }, - "nbformat": 4, - "nbformat_minor": 5 + ] } From f00e029115910c33b5a4f74d8e7a8c0034945b3f Mon Sep 17 00:00:00 2001 From: "L. M. Riza Rizky" <42672299+zaRizk7@users.noreply.github.com> Date: Mon, 14 Jul 2025 19:36:33 +0100 Subject: [PATCH 3/3] add outputs for cancer --- .../tutorial-cancer.ipynb | 1524 ++++++++++------- 1 file changed, 924 insertions(+), 600 deletions(-) diff --git a/tutorials/multiomics-cancer-classification/tutorial-cancer.ipynb b/tutorials/multiomics-cancer-classification/tutorial-cancer.ipynb index 39bf267..33613ce 100644 --- a/tutorials/multiomics-cancer-classification/tutorial-cancer.ipynb +++ b/tutorials/multiomics-cancer-classification/tutorial-cancer.ipynb @@ -1,602 +1,926 @@ { - "cells": [ - { - "cell_type": "markdown", - "id": "dbd2571f", - "metadata": {}, - "source": [ - "# Multiomics Cancer Classification\n", - "\n", - "![](images/mogonet-pykale-api.png)" - ] - }, - { - "cell_type": "markdown", - "id": "e0825580", - "metadata": {}, - "source": [ - "In this tutorial, we demonstrate how to use the standard pipeline in `PyKale` to integrate **patient multiomics data** in **cancer classification**.\n", - "We use **M**ulti-**O**mics **G**raph c**O**nvolutional **NET**works (MOGONET) by **Huang et al. (Nature Communication, 2021)** as an example." - ] - }, - { - "cell_type": "markdown", - "id": "15a944d7", - "metadata": {}, - "source": [ - "This tutorial is about cancer subtypes classification problem, which is a multi-class classification problem. The input is the multiomics data from patient, including mRNA expression data, DNA methylation data, and miRNA expression data. The output will be the subtype of cancers. We have two datasets to work with, **BRCA** and **ROSMAP**. BRCA has five subtypes and ROSMAP has only two." - ] - }, - { - "cell_type": "markdown", - "id": "b419011e", - "metadata": {}, - "source": [ - "## Step 0: Environment Preparation\n", - "\n", - "As a starting point, we will install the required packages and load a set of helper functions to assist throughout this tutorial. To keep the output clean and focused on interpretation, we will also suppress warnings.\n", - "\n", - "To prepare the helper functions and necessary materials, we download them from the GitHub repository." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "551867b5", - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "\n", - "!rm -rf /content/mmai-tutorials\n", - "!git clone https://github.com/pykale/mmai-tutorials.git\n", - "\n", - "%cd /content/mmai-tutorials/tutorials/multiomics-cancer-classification\n", - "\n", - "print(\"Changed working directory to:\", os.getcwd())" - ] - }, - { - "cell_type": "markdown", - "id": "e014d91d", - "metadata": {}, - "source": [ - "### Package Installation" - ] - }, - { - "cell_type": "markdown", - "id": "41ce5cef", - "metadata": {}, - "source": [ - "The main package required for this tutorial is `PyKale`.\n", - "\n", - "`PyKale` is an open-source interdisciplinary machine learning library developed at the University of Sheffield, with a focus on applications in biomedical and scientific domains.\n", - "\n", - "Then, we install `PyG` (PyTorch Geometric) and related packages.\n", - "\n", - "[**WARNING**] Please **do not** re-run this session after installation completed. Runing this installation multiple times will trigger issues related to `PyG`. If you want to re-run this installation, please click the `Runtime` on the top menu and choose `Disconnect and delete runtime` before installing.\n", - "\n", - "[Estimated running time] 3 mins" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "6050d5b4", - "metadata": {}, - "outputs": [], - "source": [ - "!pip install --quiet \\\n", - " git+https://github.com/pykale/pykale@main \\\n", - " yacs==0.1.8 \\\n", - " torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric \\\n", - " -f https://data.pyg.org/whl/torch-2.6.0+cu124.html \\\n", - " && echo \"pykale,yacs and wfdb installed successfully ✅\" \\\n", - " || echo \"Failed to install pykale,yacs ❌\"" - ] - }, - { - "cell_type": "markdown", - "id": "2027e726", - "metadata": {}, - "source": [ - "We then hide the warnings messages to get a clear output." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1c9c4856", - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import warnings\n", - "\n", - "warnings.filterwarnings(\"ignore\")\n", - "os.environ[\"PYTHONWARNINGS\"] = \"ignore\"" - ] - }, - { - "cell_type": "markdown", - "id": "6b32af98", - "metadata": {}, - "source": [ - "### Configuration\n", - "\n", - "To minimize the footprint of the notebook when specifying configurations, we provide a [`config.py`](https://github.com/pykale/mmai-tutorial/blob/main/tutorials/multiomics-cancer-classification/config.py) file that defines default parameters. These can be customized by supplying a `.yaml` configuration file, such as [`configs/BRCA.yaml`](https://github.com/pykale/mmai-tutorial/blob/main/tutorials/multiomics-cancer-classification/configs/BRCA.yaml) as an example.\n", - "\n", - "First, we load the configuration from [`configs/BRCA.yaml`](https://github.com/pykale/mmai-tutorial/blob/main/tutorials/multiomics-cancer-classification/configs/BRCA.yaml)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "20700eaf", - "metadata": {}, - "outputs": [], - "source": [ - "from config import get_cfg_defaults\n", - "\n", - "cfg = get_cfg_defaults()\n", - "cfg.merge_from_file(\"configs/BRCA.yaml\")" - ] - }, - { - "cell_type": "markdown", - "id": "71add965", - "metadata": {}, - "source": [ - "Besides, we also provide a configuration file for another dataset **ROSMAP**, named [`configs/ROSMAP.yaml`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/multiomics-cancer-classification/configs/ROSMAP.yaml). Users can try with this dataset later." - ] - }, - { - "cell_type": "markdown", - "id": "66a1eb4b", - "metadata": {}, - "source": [ - "In this tutorial, we list the hyperparameters we would like users to play with outside the `.yaml` file:\n", - "- `cfg.SOLVER.MAX_EPOCHS_PRETRAIN`: Number of epochs in pre-training stage.\n", - "- `cfg.SOLVER.MAX_EPOCHS`: Number of epochs in training stage.\n", - "- `cfg.DATASET.NUM_MODALITIES`: Number of modalities in the pipeline.\n", - " - `1`: mRNA expression.\n", - " - `2`: mRNA expression + DNA methylation.\n", - " - `3`: mRNA expression + DNA methylation + miRNA expression.\n", - "\n", - "[**NOTE**] Because this tutorial aims to demonmstrate `PyKale` pipeline, we only set `cfg.SOLVER.MAX_EPOCHS_PRETRAIN=100` and `cfg.SOLVER.MAX_EPOCHS=500` to reduce the training time.\n", - "If users are interested, please increase them to get more accurate predictions." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f1f8bb7c", - "metadata": {}, - "outputs": [], - "source": [ - "cfg.SOLVER.MAX_EPOCHS_PRETRAIN = 100\n", - "cfg.SOLVER.MAX_EPOCHS = 500\n", - "cfg.DATASET.NUM_MODALITIES = 3" - ] - }, - { - "cell_type": "markdown", - "id": "3bdf97a1", - "metadata": {}, - "source": [ - "Print hyperparameters:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f85914b1", - "metadata": {}, - "outputs": [], - "source": [ - "print(cfg)" - ] - }, - { - "cell_type": "markdown", - "id": "317fcb9b", - "metadata": {}, - "source": [ - "## Step 1: Data Loading and Preparation\n", - "\n", - "We use two multiomics benchmarks in this tutorial, BRCA and ROSMAP, which have been provided by the authors of MOGONET paper in [their repository](https://github.com/txWang/MOGONET).\n", - "\n", - "If users are interested in more details regarding **data organization, downloading, loading, and pre-processing**, please refer to the [Data page](https://pykale.github.io/mmai-tutorials/tutorials/multiomics-cancer-classification/extend-reading/data.html) of the tutorial." - ] - }, - { - "cell_type": "markdown", - "id": "8bf5c0c0", - "metadata": {}, - "source": [ - "Delete the potential existing data and download new version:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "2ecd6082", - "metadata": {}, - "outputs": [], - "source": [ - "!rm -rf dataset/" - ] - }, - { - "cell_type": "markdown", - "id": "868bcf23", - "metadata": {}, - "source": [ - "To load data, we first define a list the names of data files:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1352ea41", - "metadata": {}, - "outputs": [], - "source": [ - "file_names = []\n", - "for modality in range(1, cfg.DATASET.NUM_MODALITIES + 1):\n", - " file_names.append(f\"{modality}_tr.csv\")\n", - " file_names.append(f\"{modality}_lbl_tr.csv\")\n", - " file_names.append(f\"{modality}_te.csv\")\n", - " file_names.append(f\"{modality}_lbl_te.csv\")\n", - " file_names.append(f\"{modality}_feat_name.csv\")" - ] - }, - { - "cell_type": "markdown", - "id": "ef417d0c", - "metadata": {}, - "source": [ - "Then, we download, load, and pre-process the data by `PyKale`.\n", - "\n", - "[Estimated running time] 20s" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "9041fabd", - "metadata": {}, - "outputs": [], - "source": [ - "import torch\n", - "from kale.loaddata.multiomics_datasets import SparseMultiomicsDataset\n", - "from kale.prepdata.tabular_transform import ToOneHotEncoding, ToTensor\n", - "\n", - "multiomics_data = SparseMultiomicsDataset(\n", - " root=cfg.DATASET.ROOT,\n", - " raw_file_names=file_names,\n", - " num_modalities=cfg.DATASET.NUM_MODALITIES,\n", - " num_classes=cfg.DATASET.NUM_CLASSES,\n", - " edge_per_node=cfg.MODEL.EDGE_PER_NODE,\n", - " url=cfg.DATASET.URL,\n", - " random_split=cfg.DATASET.RANDOM_SPLIT,\n", - " equal_weight=cfg.MODEL.EQUAL_WEIGHT,\n", - " pre_transform=ToTensor(dtype=torch.float),\n", - " target_pre_transform=ToOneHotEncoding(dtype=torch.float),\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "c8819b69", - "metadata": {}, - "source": [ - "Inspect the dataset:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "676ebd93", - "metadata": {}, - "outputs": [], - "source": [ - "print(multiomics_data)" - ] - }, - { - "cell_type": "markdown", - "id": "910ca35a", - "metadata": {}, - "source": [ - "## Step 2: Model Definition" - ] - }, - { - "cell_type": "markdown", - "id": "007e4533", - "metadata": {}, - "source": [ - "If users are interested in more details regarding the model, please refer to the [Helper Function & Model Definition](https://pykale.github.io/mmai-tutorials/tutorials/multiomics-cancer-classification/extend-reading/helper-functions.html) of the tutorial.\n", - "\n", - "To initialize the model, we firstly call `MogonetModel` from [`model.py`](https://github.com/pykale/mmai-tutorials/blob/main/tutorials/multiomics-cancer-classification/model.py)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1537ce26", - "metadata": {}, - "outputs": [], - "source": [ - "from model import MogonetModel\n", - "\n", - "mogonet_model = MogonetModel(cfg, dataset=multiomics_data)" - ] - }, - { - "cell_type": "markdown", - "id": "3bcb4126", - "metadata": {}, - "source": [ - "Visualize the model architecture:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "da221bd6", - "metadata": {}, - "outputs": [], - "source": [ - "print(mogonet_model)" - ] - }, - { - "cell_type": "markdown", - "id": "38d9195c", - "metadata": {}, - "source": [ - "## Step 3: Model Training" - ] - }, - { - "cell_type": "markdown", - "id": "a7f6ad5c", - "metadata": {}, - "source": [ - "### Pretrain Unimodal Encoders\n", - "\n", - "Before training the multiomics model, we first pretrain encoders for each modality independently. This step helps each GCN encoder learn a good representation of its respective modality before integration.\n", - "\n", - "We can define the trainer of pretraining stage by:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "7383c5c1", - "metadata": {}, - "outputs": [], - "source": [ - "import pytorch_lightning as pl\n", - "\n", - "network = mogonet_model.get_model(pretrain=True)\n", - "trainer_pretrain = pl.Trainer(\n", - " max_epochs=cfg.SOLVER.MAX_EPOCHS_PRETRAIN,\n", - " default_root_dir=cfg.OUTPUT.OUT_DIR,\n", - " accelerator=\"auto\",\n", - " devices=\"auto\",\n", - " enable_model_summary=False,\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "b0c71889", - "metadata": {}, - "source": [ - "We pretrain the model by:\n", - "\n", - "\n", - "[Estimated running time] 15s for 100 epochs" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "2b42b719", - "metadata": {}, - "outputs": [], - "source": [ - "trainer_pretrain.fit(network)" - ] - }, - { - "cell_type": "markdown", - "id": "0b03d93e", - "metadata": {}, - "source": [ - "### Train the Multimodal Model\n", - "After pretraining the unimodal pathways, we now train the full MOGONET model by enabling the VCDN. In this stage, all modality-specific encoders and VCDN are trained.\n", - "\n", - "We define the trainer of multimodal training by:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "e94b710d", - "metadata": {}, - "outputs": [], - "source": [ - "network = mogonet_model.get_model(pretrain=False)\n", - "trainer = pl.Trainer(\n", - " max_epochs=cfg.SOLVER.MAX_EPOCHS,\n", - " default_root_dir=cfg.OUTPUT.OUT_DIR,\n", - " accelerator=\"auto\",\n", - " devices=\"auto\",\n", - " enable_model_summary=False,\n", - " log_every_n_steps=1,\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "31b76385", - "metadata": {}, - "source": [ - "We start the multimodal training by:\n", - "\n", - "\n", - "[Estimated running time] 1 min for 500 epochs" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "b3e66c8f", - "metadata": {}, - "outputs": [], - "source": [ - "trainer.fit(network)" - ] - }, - { - "cell_type": "markdown", - "id": "d41dc02a", - "metadata": {}, - "source": [ - "## Step 4: Evaluation\n", - "Once training is complete, we evaluate the model on the test set using `trainer.test()`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "019e2e7b", - "metadata": {}, - "outputs": [], - "source": [ - "trainer.test(network)" - ] - }, - { - "cell_type": "markdown", - "id": "719c655c", - "metadata": {}, - "source": [ - "## Step 5: Interpretation Study\n", - "We use `kale.interpret` to perform interpretation, where a function that systematically masks input features and observes the effect on performance—highlighting which features are most important for classification is provided. Please refer to [Interpret Study page](https://pykale.github.io/mmai-tutorials/tutorials/multiomics-cancer-classification/extend-reading/interpretation-study.html) for more details.\n", - "\n", - "Because the interpretation study needs us to mask one feature and observe the performance drop, we firstly define the trainer for the interpretation experiments.\n", - "\n", - "[**NOTE**] The final results may be different from what they should be because we only train the model for a few epochs to reduce waiting time in this tutorial." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "f061dd93", - "metadata": {}, - "outputs": [], - "source": [ - "from kale.interpret.model_weights import select_top_features_by_masking\n", - "import pytorch_lightning as pl\n", - "\n", - "trainer_biomarker = pl.Trainer(\n", - " max_epochs=cfg.SOLVER.MAX_EPOCHS,\n", - " accelerator=\"auto\",\n", - " devices=\"auto\",\n", - " enable_progress_bar=False,\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "38a31ccf", - "metadata": {}, - "source": [ - "Then, we start the experiment." - ] - }, - { - "cell_type": "markdown", - "id": "4a754a08", - "metadata": {}, - "source": [ - "To supress the verbose messages in the following experiments:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "e428229c", - "metadata": {}, - "outputs": [], - "source": [ - "import logging\n", - "\n", - "logging.getLogger(\"pytorch_lightning\").setLevel(logging.ERROR)" - ] - }, - { - "cell_type": "markdown", - "id": "8565e576", - "metadata": {}, - "source": [ - "Run the interpretation experiments:\n", - "\n", - "[Estimated running time] Because the following block will train the model for 2,503 times for BRCA dataset, the following block may take about 6 minutes." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "2dd9e5e3", - "metadata": {}, - "outputs": [], - "source": [ - "f1_key = \"F1\" if multiomics_data.num_classes == 2 else \"F1 macro\"\n", - "df_featimp_top = select_top_features_by_masking(\n", - " trainer=trainer_biomarker,\n", - " model=network,\n", - " dataset=multiomics_data,\n", - " metric=f1_key,\n", - " num_top_feats=30,\n", - " verbose=False,\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "964300ae", - "metadata": {}, - "source": [ - "Print the most important features:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "c984bdb1", - "metadata": {}, - "outputs": [], - "source": [ - "print(\"{:>4}\\t{:<20}\\t{:>5}\\t{}\".format(\"Rank\", \"Feature name\", \"Omics\", \"Importance\"))\n", - "for rank, row in enumerate(df_featimp_top.itertuples(index=False), 1):\n", - " print(f\"{rank:>4}\\t{row.feat_name:<20}\\t{row.omics:>5}\\t{row.imp:.4f}\")" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "mmai-cancer-tutorial", - "language": "python", - "name": "python3" - } - }, - "nbformat": 4, - "nbformat_minor": 5 + "nbformat": 4, + "nbformat_minor": 5, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "cells": [ + { + "metadata": {}, + "source": [ + "# Multiomics Cancer Classification\n", + "\n", + "![](https://github.com/pykale/mmai-tutorials/blob/main/tutorials/multiomics-cancer-classification/images/mogonet-pykale-api.png?raw=1)" + ], + "cell_type": "markdown", + "id": "dbd2571f" + }, + { + "metadata": {}, + "source": [ + "In this tutorial, we will use a [**M**ulti-**O**mics **G**raph c**O**nvolutional **NET**works (MOGONET) by **Wang et al. (Nature Communication, 2021)**](https://www.nature.com/articles/s41467-021-23774-w) [1] pipeline implemented in `PyKale` [2] to integrate **patient multiomics data** for **cancer subtypes classification**.\n", + "\n", + "We will work with multiomics data from two datasets: [**BRCA** of TCGA](https://www.cancerimagingarchive.net/collection/tcga-brca/) [3] and [**ROSMAP**](https://www.synapse.org/Synapse:syn3219045) [4,5]. The BRCA dataset has five subtypes, while the ROSMAP dataset has only two. Three omics modalities will be used: mRNA expression, DNA methylation, and miRNA expression.\n", + "\n", + "The multimodal approach used in this tutorial involves **interaction**, where a cross-omics tensor is constructed for the probability interaction across three omics modalities.\n", + "\n", + "The main tasks of this tutorial are:\n", + "\n", + "- Load BRCA or ROSMAP dataset.\n", + "- Define a MOGONET model.\n", + "- Train and evaluate the MOGONET model on the multiomics data.\n", + "- Obtain the feature importance and visualize the interpretation of the model." + ], + "cell_type": "markdown", + "id": "e0825580" + }, + { + "metadata": {}, + "source": [ + "## Step 0: Environment Preparation\n", + "\n", + "As a starting point, we will install the required packages and load a set of helper functions to assist throughout this tutorial. To keep the output clean and focused on interpretation, we will also suppress warnings.\n", + "\n", + "To prepare the helper functions and necessary materials, we download them from the GitHub repository." + ], + "cell_type": "markdown", + "id": "b419011e" + }, + { + "metadata": {}, + "source": [ + "import os\n", + "\n", + "!rm -rf /content/mmai-tutorials\n", + "!git clone https://github.com/pykale/mmai-tutorials.git\n", + "\n", + "%cd /content/mmai-tutorials/tutorials/multiomics-cancer-classification\n", + "\n", + "print(\"Changed working directory to:\", os.getcwd())" + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "fatal: destination path 'mmai-tutorials' already exists and is not an empty directory.\n", + "/content/mmai-tutorials/tutorials/multiomics-cancer-classification\n", + "Changed working directory to: /content/mmai-tutorials/tutorials/multiomics-cancer-classification\n" + ] + } + ], + "id": "551867b5", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "### Package Installation" + ], + "cell_type": "markdown", + "id": "e014d91d" + }, + { + "metadata": {}, + "source": [ + "The main package required for this tutorial is `PyKale`.\n", + "\n", + "`PyKale` is an open-source interdisciplinary machine learning library developed at the University of Sheffield, with a focus on applications in biomedical and scientific domains.\n", + "\n", + "Then, we install `PyG` (PyTorch Geometric) and related packages.\n", + "\n", + "[**WARNING**] Please **do not** re-run this session after installation completed. Runing this installation multiple times will trigger issues related to `PyG`. If you want to re-run this installation, please click the `Runtime` on the top menu and choose `Disconnect and delete runtime` before installing.\n", + "\n", + "[Estimated running time] 3 mins" + ], + "cell_type": "markdown", + "id": "41ce5cef" + }, + { + "metadata": {}, + "source": [ + "%pip install --quiet \\\n", + " \"pykale[example]@git+https://github.com/pykale/pykale@main\" \\\n", + " gdown==5.2.0 torch-geometric==2.6.0 torch_sparse torch_scatter \\\n", + " -f https://data.pyg.org/whl/torch-2.6.0+cu124.html \\\n", + " && echo \"pykale, gdown, nilearn, and yacs installed successfully \u2705\" \\\n", + " || echo \"Failed to install pykale, gdown, nilearn, and yacs \u274c\"" + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + " Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n", + " Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n", + " Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n", + "pykale, gdown, nilearn, and yacs installed successfully \u2705\n" + ] + } + ], + "id": "6050d5b4", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "We then hide the warnings messages to get a clear output." + ], + "cell_type": "markdown", + "id": "2027e726" + }, + { + "metadata": {}, + "source": [ + "import os\n", + "import warnings\n", + "\n", + "warnings.filterwarnings(\"ignore\")\n", + "os.environ[\"PYTHONWARNINGS\"] = \"ignore\"" + ], + "cell_type": "code", + "outputs": [], + "id": "1c9c4856", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "### Configuration\n", + "\n", + "To minimize the footprint of the notebook when specifying configurations, we provide a [`config.py`](https://github.com/pykale/mmai-tutorial/blob/main/tutorials/multiomics-cancer-classification/config.py) file that defines default parameters. These can be customized by supplying a `.yaml` configuration file, such as [`configs/BRCA.yaml`](https://github.com/pykale/mmai-tutorial/blob/main/tutorials/multiomics-cancer-classification/configs/BRCA.yaml) as an example.\n", + "\n", + "First, we load the configuration from [`configs/BRCA.yaml`](https://github.com/pykale/mmai-tutorial/blob/main/tutorials/multiomics-cancer-classification/configs/BRCA.yaml)." + ], + "cell_type": "markdown", + "id": "6b32af98" + }, + { + "metadata": {}, + "source": [ + "from config import get_cfg_defaults\n", + "\n", + "cfg = get_cfg_defaults()\n", + "cfg.merge_from_file(\"configs/BRCA.yaml\")" + ], + "cell_type": "code", + "outputs": [], + "id": "20700eaf", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "Besides, we also provide a configuration file for another dataset **ROSMAP**, named [`configs/ROSMAP.yaml`](https://github.com/pykale/embc-mmai25/blob/main/tutorials/multiomics-cancer-classification/configs/ROSMAP.yaml). Users can try with this dataset later." + ], + "cell_type": "markdown", + "id": "71add965" + }, + { + "metadata": {}, + "source": [ + "In this tutorial, we list the hyperparameters we would like users to play with outside the `.yaml` file:\n", + "- `cfg.SOLVER.MAX_EPOCHS_PRETRAIN`: Number of epochs in pre-training stage.\n", + "- `cfg.SOLVER.MAX_EPOCHS`: Number of epochs in training stage.\n", + "- `cfg.DATASET.NUM_MODALITIES`: Number of modalities in the pipeline.\n", + " - `1`: mRNA expression.\n", + " - `2`: mRNA expression + DNA methylation.\n", + " - `3`: mRNA expression + DNA methylation + miRNA expression.\n", + "\n", + "[**NOTE**] Because this tutorial aims to demonmstrate `PyKale` pipeline, we only set `cfg.SOLVER.MAX_EPOCHS_PRETRAIN=100` and `cfg.SOLVER.MAX_EPOCHS=500` to reduce the training time.\n", + "If users are interested, please increase them to get more accurate predictions." + ], + "cell_type": "markdown", + "id": "66a1eb4b" + }, + { + "metadata": {}, + "source": [ + "cfg.SOLVER.MAX_EPOCHS_PRETRAIN = 100\n", + "cfg.SOLVER.MAX_EPOCHS = 500\n", + "cfg.DATASET.NUM_MODALITIES = 3" + ], + "cell_type": "code", + "outputs": [], + "id": "f1f8bb7c", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "Print hyperparameters:" + ], + "cell_type": "markdown", + "id": "3bdf97a1" + }, + { + "metadata": {}, + "source": [ + "print(cfg)" + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "DATASET:\n", + " NAME: TCGA_BRCA\n", + " NUM_CLASSES: 5\n", + " NUM_MODALITIES: 3\n", + " RANDOM_SPLIT: False\n", + " ROOT: dataset/\n", + " URL: https://github.com/pykale/data/raw/main/multiomics/TCGA_BRCA.zip\n", + "MODEL:\n", + " EDGE_PER_NODE: 10\n", + " EQUAL_WEIGHT: False\n", + " GCN_DROPOUT_RATE: 0.5\n", + " GCN_HIDDEN_DIM: [400, 400, 200]\n", + " GCN_LR: 0.0005\n", + " GCN_LR_PRETRAIN: 0.001\n", + " VCDN_LR: 0.001\n", + "OUTPUT:\n", + " OUT_DIR: ./outputs\n", + "SOLVER:\n", + " MAX_EPOCHS: 500\n", + " MAX_EPOCHS_PRETRAIN: 100\n", + " SEED: 2023\n" + ] + } + ], + "id": "f85914b1", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "## Step 1: Data Loading and Preparation\n", + "\n", + "We use two multiomics benchmarks in this tutorial, BRCA and ROSMAP, which have been provided by the authors of MOGONET paper in [their repository](https://github.com/txWang/MOGONET).\n", + "\n", + "If users are interested in more details regarding **data organization, downloading, loading, and pre-processing**, please refer to the [Data page](https://pykale.github.io/mmai-tutorials/tutorials/multiomics-cancer-classification/extend-reading/data.html) of the tutorial." + ], + "cell_type": "markdown", + "id": "317fcb9b" + }, + { + "metadata": {}, + "source": [ + "Delete the potential existing data and download new version:" + ], + "cell_type": "markdown", + "id": "8bf5c0c0" + }, + { + "metadata": {}, + "source": [ + "!rm -rf dataset/" + ], + "cell_type": "code", + "outputs": [], + "id": "2ecd6082", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "To load data, we first define a list the names of data files:" + ], + "cell_type": "markdown", + "id": "868bcf23" + }, + { + "metadata": {}, + "source": [ + "file_names = []\n", + "for modality in range(1, cfg.DATASET.NUM_MODALITIES + 1):\n", + " file_names.append(f\"{modality}_tr.csv\")\n", + " file_names.append(f\"{modality}_lbl_tr.csv\")\n", + " file_names.append(f\"{modality}_te.csv\")\n", + " file_names.append(f\"{modality}_lbl_te.csv\")\n", + " file_names.append(f\"{modality}_feat_name.csv\")" + ], + "cell_type": "code", + "outputs": [], + "id": "1352ea41", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "Then, we download, load, and pre-process the data by `PyKale`.\n", + "\n", + "[Estimated running time] 20s" + ], + "cell_type": "markdown", + "id": "ef417d0c" + }, + { + "metadata": {}, + "source": [ + "import torch\n", + "from kale.loaddata.multiomics_datasets import SparseMultiomicsDataset\n", + "from kale.prepdata.tabular_transform import ToOneHotEncoding, ToTensor\n", + "\n", + "multiomics_data = SparseMultiomicsDataset(\n", + " root=cfg.DATASET.ROOT,\n", + " raw_file_names=file_names,\n", + " num_modalities=cfg.DATASET.NUM_MODALITIES,\n", + " num_classes=cfg.DATASET.NUM_CLASSES,\n", + " edge_per_node=cfg.MODEL.EDGE_PER_NODE,\n", + " url=cfg.DATASET.URL,\n", + " random_split=cfg.DATASET.RANDOM_SPLIT,\n", + " equal_weight=cfg.MODEL.EQUAL_WEIGHT,\n", + " pre_transform=ToTensor(dtype=torch.float),\n", + " target_pre_transform=ToOneHotEncoding(dtype=torch.float),\n", + ")" + ], + "cell_type": "code", + "outputs": [], + "id": "9041fabd", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "Inspect the dataset:" + ], + "cell_type": "markdown", + "id": "c8819b69" + }, + { + "metadata": {}, + "source": [ + "print(multiomics_data)" + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "\n", + "Dataset info:\n", + " number of modalities: 3\n", + " number of classes: 5\n", + "\n", + " modality | total samples | num train | num test | num features\n", + " -----------------------------------------------------------------\n", + " 1 | 875 | 612 | 263 | 1000 \n", + " 2 | 875 | 612 | 263 | 1000 \n", + " 3 | 875 | 612 | 263 | 503 \n", + " -----------------------------------------------------------------\n", + "\n", + "\n" + ] + } + ], + "id": "676ebd93", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "## Step 2: Model Definition" + ], + "cell_type": "markdown", + "id": "910ca35a" + }, + { + "metadata": {}, + "source": [ + "If users are interested in more details regarding the model, please refer to the [Helper Function & Model Definition](https://pykale.github.io/mmai-tutorials/tutorials/multiomics-cancer-classification/extend-reading/helper-functions.html) of the tutorial.\n", + "\n", + "To initialize the model, we firstly call `MogonetModel` from [`model.py`](https://github.com/pykale/mmai-tutorials/blob/main/tutorials/multiomics-cancer-classification/model.py)." + ], + "cell_type": "markdown", + "id": "007e4533" + }, + { + "metadata": {}, + "source": [ + "from model import MogonetModel\n", + "\n", + "mogonet_model = MogonetModel(cfg, dataset=multiomics_data)" + ], + "cell_type": "code", + "outputs": [], + "id": "1537ce26", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "Visualize the model architecture:" + ], + "cell_type": "markdown", + "id": "3bcb4126" + }, + { + "metadata": {}, + "source": [ + "print(mogonet_model)" + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "\n", + "Model info:\n", + " Unimodal encoder:\n", + " (1) MogonetGCN(\n", + " (conv1): MogonetGCNConv(1000, 400)\n", + " (conv2): MogonetGCNConv(400, 400)\n", + " (conv3): MogonetGCNConv(400, 200)\n", + ") (2) MogonetGCN(\n", + " (conv1): MogonetGCNConv(1000, 400)\n", + " (conv2): MogonetGCNConv(400, 400)\n", + " (conv3): MogonetGCNConv(400, 200)\n", + ") (3) MogonetGCN(\n", + " (conv1): MogonetGCNConv(503, 400)\n", + " (conv2): MogonetGCNConv(400, 400)\n", + " (conv3): MogonetGCNConv(400, 200)\n", + ")\n", + "\n", + " Unimodal decoder:\n", + " (1) LinearClassifier(\n", + " (fc): Linear(in_features=200, out_features=5, bias=True)\n", + ") (2) LinearClassifier(\n", + " (fc): Linear(in_features=200, out_features=5, bias=True)\n", + ") (3) LinearClassifier(\n", + " (fc): Linear(in_features=200, out_features=5, bias=True)\n", + ")\n", + "\n", + " Multimodal decoder:\n", + " VCDN(\n", + " (model): Sequential(\n", + " (0): Linear(in_features=125, out_features=125, bias=True)\n", + " (1): LeakyReLU(negative_slope=0.25)\n", + " (2): Linear(in_features=125, out_features=5, bias=True)\n", + " )\n", + ")\n" + ] + } + ], + "id": "da221bd6", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "## Step 3: Model Training" + ], + "cell_type": "markdown", + "id": "38d9195c" + }, + { + "metadata": {}, + "source": [ + "### Pretrain Unimodal Encoders\n", + "\n", + "Before training the multiomics model, we first pretrain encoders for each modality independently. This step helps each GCN encoder learn a good representation of its respective modality before integration.\n", + "\n", + "We can define the trainer of pretraining stage by:" + ], + "cell_type": "markdown", + "id": "a7f6ad5c" + }, + { + "metadata": {}, + "source": [ + "import pytorch_lightning as pl\n", + "\n", + "network = mogonet_model.get_model(pretrain=True)\n", + "trainer_pretrain = pl.Trainer(\n", + " max_epochs=cfg.SOLVER.MAX_EPOCHS_PRETRAIN,\n", + " default_root_dir=cfg.OUTPUT.OUT_DIR,\n", + " accelerator=\"auto\",\n", + " devices=\"auto\",\n", + " enable_model_summary=False,\n", + ")" + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "INFO:pytorch_lightning.utilities.rank_zero:\ud83d\udca1 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.\n", + "INFO:pytorch_lightning.utilities.rank_zero:GPU available: True (cuda), used: True\n", + "INFO:pytorch_lightning.utilities.rank_zero:TPU available: False, using: 0 TPU cores\n", + "INFO:pytorch_lightning.utilities.rank_zero:HPU available: False, using: 0 HPUs\n" + ] + } + ], + "id": "7383c5c1", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "We pretrain the model by:\n", + "\n", + "\n", + "[Estimated running time] 15s for 100 epochs" + ], + "cell_type": "markdown", + "id": "b0c71889" + }, + { + "metadata": {}, + "source": [ + "trainer_pretrain.fit(network)" + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "INFO:pytorch_lightning.accelerators.cuda:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n" + ] + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "Training: | | 0/? [00:00\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n", + "\u2503 Test metric \u2503 DataLoader 0 \u2503\n", + "\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n", + "\u2502 Accuracy \u2502 0.8019999861717224 \u2502\n", + "\u2502 F1 macro \u2502 0.6880000233650208 \u2502\n", + "\u2502 F1 weighted \u2502 0.7699999809265137 \u2502\n", + "\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n", + "\n" + ] + }, + "metadata": {} + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "[{'Accuracy': 0.8019999861717224,\n", + " 'F1 weighted': 0.7699999809265137,\n", + " 'F1 macro': 0.6880000233650208}]" + ] + }, + "metadata": {}, + "execution_count": 17 + } + ], + "id": "019e2e7b", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "## Step 5: Interpretation Study\n", + "We use `kale.interpret` to perform interpretation, where a function that systematically masks input features and observes the effect on performance\u2014highlighting which features are most important for classification is provided. Please refer to [Interpret Study page](https://pykale.github.io/mmai-tutorials/tutorials/multiomics-cancer-classification/extend-reading/interpretation-study.html) for more details.\n", + "\n", + "Because the interpretation study needs us to mask one feature and observe the performance drop, we firstly define the trainer for the interpretation experiments.\n", + "\n", + "[**NOTE**] The final results may be different from what they should be because we only train the model for a few epochs to reduce waiting time in this tutorial." + ], + "cell_type": "markdown", + "id": "719c655c" + }, + { + "metadata": {}, + "source": [ + "from kale.interpret.model_weights import select_top_features_by_masking\n", + "import pytorch_lightning as pl\n", + "\n", + "trainer_biomarker = pl.Trainer(\n", + " max_epochs=cfg.SOLVER.MAX_EPOCHS,\n", + " accelerator=\"auto\",\n", + " devices=\"auto\",\n", + " enable_progress_bar=False,\n", + ")" + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "INFO:pytorch_lightning.utilities.rank_zero:\ud83d\udca1 Tip: For seamless cloud uploads and versioning, try installing [litmodels](https://pypi.org/project/litmodels/) to enable LitModelCheckpoint, which syncs automatically with the Lightning model registry.\n", + "INFO:pytorch_lightning.utilities.rank_zero:GPU available: True (cuda), used: True\n", + "INFO:pytorch_lightning.utilities.rank_zero:TPU available: False, using: 0 TPU cores\n", + "INFO:pytorch_lightning.utilities.rank_zero:HPU available: False, using: 0 HPUs\n" + ] + } + ], + "id": "f061dd93", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "Then, we start the experiment." + ], + "cell_type": "markdown", + "id": "38a31ccf" + }, + { + "metadata": {}, + "source": [ + "To supress the verbose messages in the following experiments:" + ], + "cell_type": "markdown", + "id": "4a754a08" + }, + { + "metadata": {}, + "source": [ + "import logging\n", + "\n", + "logging.getLogger(\"pytorch_lightning\").setLevel(logging.ERROR)" + ], + "cell_type": "code", + "outputs": [], + "id": "e428229c", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "Run the interpretation experiments:\n", + "\n", + "[Estimated running time] Because the following block will train the model for 2,503 times for BRCA dataset, the following block may take about 6 minutes." + ], + "cell_type": "markdown", + "id": "8565e576" + }, + { + "metadata": {}, + "source": [ + "f1_key = \"F1\" if multiomics_data.num_classes == 2 else \"F1 macro\"\n", + "df_featimp_top = select_top_features_by_masking(\n", + " trainer=trainer_biomarker,\n", + " model=network,\n", + " dataset=multiomics_data,\n", + " metric=f1_key,\n", + " num_top_feats=30,\n", + " verbose=False,\n", + ")" + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [] + } + ], + "id": "2dd9e5e3", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "Print the most important features:" + ], + "cell_type": "markdown", + "id": "964300ae" + }, + { + "metadata": {}, + "source": [ + "print(\"{:>4}\\t{:<20}\\t{:>5}\\t{}\".format(\"Rank\", \"Feature name\", \"Omics\", \"Importance\"))\n", + "for rank, row in enumerate(df_featimp_top.itertuples(index=False), 1):\n", + " print(f\"{rank:>4}\\t{row.feat_name:<20}\\t{row.omics:>5}\\t{row.imp:.4f}\")" + ], + "cell_type": "code", + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Rank\tFeature name \tOmics\tImportance\n", + " 1\tMSLN|10232 \t 0\t21.0000\n", + " 2\thsa-mir-9-2 \t 2\t17.6050\n", + " 3\thsa-mir-9-1 \t 2\t16.0960\n", + " 4\thsa-mir-203 \t 2\t15.0900\n", + " 5\tABCC11|85320 \t 0\t13.0000\n", + " 6\tTMEM207 \t 1\t13.0000\n", + " 7\tHOXD11 \t 1\t13.0000\n", + " 8\tKRTAP3-1 \t 1\t13.0000\n", + " 9\tOR1J4 \t 1\t13.0000\n", + " 10\tGPR37L1 \t 1\t13.0000\n", + " 11\thsa-mir-2115 \t 2\t11.5690\n", + " 12\thsa-mir-187 \t 2\t11.5690\n", + " 13\thsa-let-7a-3 \t 2\t9.5570\n", + " 14\thsa-let-7f-2 \t 2\t9.0540\n", + " 15\thsa-mir-205 \t 2\t8.5510\n", + " 16\thsa-mir-551b \t 2\t8.5510\n", + " 17\tANKRD45|339416 \t 0\t8.0000\n", + " 18\tNOTCH1|4851 \t 0\t8.0000\n", + " 19\tMDGA2|161357 \t 0\t8.0000\n", + " 20\tARHGEF4|50649 \t 0\t8.0000\n", + " 21\tCRHR1|1394 \t 0\t8.0000\n", + " 22\tCXCL3|2921 \t 0\t8.0000\n", + " 23\tCSDA|8531 \t 0\t8.0000\n", + " 24\tPI3|5266 \t 0\t8.0000\n", + " 25\tSLC43A3|29015 \t 0\t8.0000\n", + " 26\tTRIML2|205860 \t 0\t8.0000\n", + " 27\tRDH10|157506 \t 0\t8.0000\n", + " 28\tIFFO2|126917 \t 0\t8.0000\n", + " 29\tISL2|64843 \t 0\t8.0000\n", + " 30\tFGFBP1|9982 \t 0\t8.0000\n" + ] + } + ], + "id": "c984bdb1", + "execution_count": null + }, + { + "metadata": {}, + "source": [ + "## References\n", + "\n", + "[1] Wang, T., Shao, W., Huang, Z., Tang, H., Zhang, J., Ding, Z., & Huang, K. (2021). MOGONET integrates multi-omics data using graph convolutional networks allowing patient classification and biomarker identification. Nature communications, 12(1), 3445.\n", + "\n", + "[2] Lu, H., Liu, X., Zhou, S., Turner, R., Bai, P., Koot, R. E., ... & Xu, H. (2022, October). PyKale: Knowledge-aware machine learning from multiple sources in Python. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_ (pp. 4274-4278).\n", + "\n", + "[3] Lingle, W., Erickson, B. J., Zuley, M. L., Jarosz, R., Bonaccio, E., Filippini, J., Net, J. M., Levi, L., Morris, E. A., Figler, G. G., Elnajjar, P., Kirk, S., Lee, Y., Giger, M., & Gruszauskas, N. (2016). The Cancer Genome Atlas Breast Invasive Carcinoma Collection (TCGA-BRCA) (Version 3) [Data set]. The Cancer Imaging Archive.\n", + "\n", + "\n", + "\n", + "[4] Bennett, D. A., Buchman, A. S., Boyle, P. A., Barnes, L. L., Wilson, R. S., & Schneider, J. A. (2018). Religious orders study and rush memory and aging project. Journal of Alzheimer\u2019s disease, 64(s1), S161-S189.\n", + "\n", + "[5] De Jager, P.L.; Ma, Y.; McCabe, C.; Xu, J.; Vardarajan, B.N.; Felsky, D.; Klein, H.U.; White, C.C.; Peters, M.A.; Lodgson, B.; et al. (2018). A multi-omic atlas of the human frontal cortex for aging and Alzheimer\u2019s disease research. Scientific Data 5, 1-13" + ], + "cell_type": "markdown", + "id": "1da8fd92" + } + ] }