tutorials/Creating Federated Dataset for PFL Experiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "81bde9ad-13ac-4851-b958-db7085a38cdc",
   "metadata": {},
   "source": [
    "# Creating Federated Dataset for PFL Experiment\n",
    "\n",
    "Federated dataset has more non-IID characteristics than traditional ML dataset, as users with different contexts independently generate their own data. In realistic scenarios, datasets for different users often come from different distributions and may possibly have conflicting labels. Because of this and the distributed nature of federated learning, the non-IID property of the federated datasets may sometimes make federated learning more challenging.\n",
    "\n",
    "This tutorial covers how to create federated dataset in different ways for running PFL experiment:\n",
    "\n",
    "1. creating `pfl` `Dataset` object\n",
    "2. creating `FederatedDataset` with user IDs\n",
    "3. creating IID `ArtificialFederatedDataset`\n",
    "4. creating non-IID `ArtificialFederatedDataset`\n",
    "5. customizing federated dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07eedfc5-27a1-4aed-b774-ce8d24eb1add",
   "metadata": {},
   "source": [
    "### 0. Setup environment\n",
    "To use this notebook, follow these prerequisite steps:\n",
    "1. Install `pfl` with `pip install 'pfl[pytorch]'`.\n",
    "2. Install jupyter notebook with `pip install jupyter notebook`.\n",
    "3. Start the notebook with `jupyter notebook`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9de0aba5-414c-4df3-8d17-176a2856092c",
   "metadata": {},
   "source": [
    "### 1. Creating a `pfl` Dataset Object\n",
    "A `pfl.data.Dataset` object is the basic representation for a single user's dataset. Alternatively, this can also be used for building a central dataset for evaluation purpose (to be used with `pfl.callback.CentralEvaluationCallback`).   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b27e01f4-3d34-42b0-ad02-795af03d356e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch from `Dataset`: \n",
      "x shape=(5, 4), x=[[-0.93396889  1.38144275 -0.25651903 -0.07058005]\n",
      " [ 1.47505537  1.60093462  2.18438982  0.21201945]\n",
      " [-1.25481843 -0.74810518 -1.5695337  -0.16404775]\n",
      " [ 2.02581709  0.76572641  2.32598324  0.25309372]\n",
      " [-0.61718078 -1.44501622 -1.27858726 -0.10931645]]\n",
      "y shape=(5,), y=[1 1 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "import sklearn.datasets\n",
    "from pfl.data.dataset import Dataset\n",
    "\n",
    "# Create synthetic user data\n",
    "user_id = 'pfl-user-0'\n",
    "user_features, user_labels = sklearn.datasets.make_classification(n_samples=10, n_features=4)\n",
    "\n",
    "# Create a pfl Dataset object\n",
    "user_dataset = Dataset(raw_data=[user_features, user_labels], user_id=user_id)\n",
    "\n",
    "# Get a batch of the user dataset using Dataset.iter\n",
    "batch_x, batch_y = next(user_dataset.iter(batch_size=5))\n",
    "print(f\"Batch from `Dataset`: \\n\"\n",
    "      f\"x shape={batch_x.shape}, x={batch_x}\\n\"\n",
    "      f\"y shape={batch_y.shape}, y={batch_y}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "114f507b-c096-40e5-904b-55485378d3fe",
   "metadata": {},
   "source": [
    "#### 1.1 Creating a PyTorch `pfl` Dataset Object\n",
    "The above example showed how to create the `pfl` dataset in numpy format. `pfl`  supports data that are processed in PyTorch or TensorFlow tensors. We will focus on PyTorch for the purpose of this tutorial. The TensorFlow dataset implementation can be found at [tensorflow.py](https://github.com/apple/pfl-research/blob/main/pfl/data/tensorflow.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5b2f45d9-dd20-4313-90e6-342b2cdbc0a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['PFL_PYTORCH_DEVICE'] = 'cpu'\n",
    "\n",
    "import torch\n",
    "\n",
    "from pfl.internal.ops.selector import set_framework_module\n",
    "from pfl.internal.ops import pytorch_ops, numpy_ops\n",
    "from pfl.data.pytorch import PyTorchTensorDataset, PyTorchDataDataset\n",
    "\n",
    "# Tell pfl to use pytorch as the backend ML framework\n",
    "set_framework_module(pytorch_ops, old_module=numpy_ops)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5da0039d-ebc4-4fef-b2e6-d6eae9512cbf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch from `PyTorchTensorDataset`: \n",
      "x shape=torch.Size([5, 4]), x=tensor([[-0.9340,  1.3814, -0.2565, -0.0706],\n",
      "        [ 1.4751,  1.6009,  2.1844,  0.2120],\n",
      "        [-1.2548, -0.7481, -1.5695, -0.1640],\n",
      "        [ 2.0258,  0.7657,  2.3260,  0.2531],\n",
      "        [-0.6172, -1.4450, -1.2786, -0.1093]], dtype=torch.float64)\n",
      "y shape=torch.Size([5]), y=tensor([1, 1, 0, 0, 0])\n"
     ]
    }
   ],
   "source": [
    "# Option 1: Create a PyTorch pfl Dataset object based on PyTorch tensors\n",
    "user_features, user_labels = torch.as_tensor(user_features), torch.as_tensor(user_labels)\n",
    "user_dataset = PyTorchTensorDataset(tensors=[user_features, user_labels], user_id=user_id)\n",
    "\n",
    "# Get a batch of the user dataset using Dataset.iter\n",
    "batch_x, batch_y = next(user_dataset.iter(batch_size=5))\n",
    "print(f\"Batch from `PyTorchTensorDataset`: \\n\"\n",
    "      f\"x shape={batch_x.shape}, x={batch_x}\\n\"\n",
    "      f\"y shape={batch_y.shape}, y={batch_y}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b04168ad-6349-4019-a957-e80d9249c053",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Batch from `PyTorchDataDataset`: \n",
      "x shape=torch.Size([5, 4]), x=tensor([[-0.9340,  1.3814, -0.2565, -0.0706],\n",
      "        [ 1.4751,  1.6009,  2.1844,  0.2120],\n",
      "        [-1.2548, -0.7481, -1.5695, -0.1640],\n",
      "        [ 2.0258,  0.7657,  2.3260,  0.2531],\n",
      "        [-0.6172, -1.4450, -1.2786, -0.1093]], dtype=torch.float64)\n",
      "y shape=torch.Size([5]), y=tensor([1, 1, 0, 0, 0])\n"
     ]
    }
   ],
   "source": [
    "# Option 2: Create a PyTorch pfl Dataset object based on torch.utils.data.Dataset\n",
    "pytorch_data = torch.utils.data.TensorDataset(user_features, user_labels)\n",
    "user_dataset = PyTorchDataDataset(raw_data=pytorch_data, user_id=user_id)\n",
    "\n",
    "# Get a batch of the user dataset using Dataset.iter\n",
    "batch_x, batch_y = next(user_dataset.iter(batch_size=5))\n",
    "print(f\"\\nBatch from `PyTorchDataDataset`: \\n\"\n",
    "      f\"x shape={batch_x.shape}, x={batch_x}\\n\"\n",
    "      f\"y shape={batch_y.shape}, y={batch_y}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c00a623e-4622-486a-96c5-97d9531e6e66",
   "metadata": {},
   "source": [
    "### 2. Creating Federated Dataset with Real User Partition\n",
    "Previous section showed how to create a single user dataset.  This section covers how to create a federated dataset which is a joint of multiple users' dataset. If the original dataset already have an attribute for user identifier, such as [StackOverflow](https://www.kaggle.com/datasets/stackoverflow/stackoverflow), [LEAF](https://leaf.cmu.edu) ans [FLAIR](https://github.com/apple/ml-flair), we can use the existing user parition in the dataset for more accurate simulation of non-IID characteristics in the real federated learning setting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9aab7f23-8391-4483-94e6-26df8751869d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FederatedDataset with real user partition: \n",
      "\tReal user with user_id=13 has size of 14.\n",
      "\tReal user with user_id=11 has size of 12.\n",
      "\tReal user with user_id=4 has size of 5.\n",
      "\tReal user with user_id=8 has size of 9.\n",
      "\tReal user with user_id=10 has size of 11.\n"
     ]
    }
   ],
   "source": [
    "from pfl.data import get_user_sampler, FederatedDataset\n",
    "\n",
    "# Create a dataset partitioned by user\n",
    "user_id_to_data = {}\n",
    "n_users = 20\n",
    "for i in range(n_users):\n",
    "    user_id_to_data[i] = sklearn.datasets.make_classification(n_samples=i+1, n_features=4)\n",
    "user_ids = list(user_id_to_data.keys())\n",
    "\n",
    "# Create user sampler to sample user uniformly at random\n",
    "user_sampler = get_user_sampler(sample_type=\"random\", user_ids=user_ids)\n",
    "federated_dataset = FederatedDataset.from_slices(data=user_id_to_data, user_sampler=user_sampler)\n",
    "\n",
    "# Iterate federated dataset to get artificial user dataset\n",
    "print(\"FederatedDataset with real user partition: \")\n",
    "for i in range(5):\n",
    "    user_dataset, seed = next(federated_dataset)\n",
    "    print(f\"\\tReal user with user_id={user_dataset.user_id} has size of {len(user_dataset)}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b03d2a10-830b-43ba-b75a-9d31fd161dbd",
   "metadata": {},
   "source": [
    "#### 2.1 Creating a `PyTorchFederatedDataset` Object\n",
    "In certain scenario, it might be beneficial to use `PyTorchFederatedDataset` for speeding up data loading. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "abcc9809-c324-458c-b146-826ece3cf0f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PyTorchFederatedDataset with real user partition: \n",
      "\tReal user 0 has size of 13.\n",
      "\tReal user 1 has size of 18.\n",
      "\tReal user 2 has size of 7.\n",
      "\tReal user 3 has size of 16.\n",
      "\tReal user 4 has size of 7.\n"
     ]
    }
   ],
   "source": [
    "from pfl.data.pytorch import PyTorchFederatedDataset\n",
    "\n",
    "class PyTorchDataset(torch.utils.data.Dataset):\n",
    "    def __init__(self, user_id_to_data):\n",
    "        self._user_id_to_data = user_id_to_data\n",
    "\n",
    "    def __getitem__(self, i):\n",
    "        return [torch.as_tensor(x) for x in self._user_id_to_data[i]]\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self._user_id_to_data)\n",
    "\n",
    "\n",
    "pytorch_dataset = PyTorchDataset(user_id_to_data)\n",
    "federated_dataset = PyTorchFederatedDataset(dataset=pytorch_dataset, user_sampler=user_sampler)\n",
    "# Iterate federated dataset to get artificial user dataset\n",
    "print(\"PyTorchFederatedDataset with real user partition: \")\n",
    "for i in range(5):\n",
    "    user_dataset, seed = next(federated_dataset)\n",
    "    print(f\"\\tReal user {i} has size of {len(user_dataset)}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63bbbac5-6907-4710-adaf-10d472d434e1",
   "metadata": {},
   "source": [
    "#### 2.2 Creating a `FederatedDataset` with `torch.utils.data.Dataset`\n",
    "If a user dataset is too large to fit into memory, `pfl` also supports `torch.utils.data.Dataset` for creating federated dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6b69e7d1-a089-462c-9448-1dae917e7d8f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FederatedDataset with torch.utils.data.Dataset and real user partition: \n",
      "\tReal user 0 has size of 5.\n",
      "\tReal user 1 has size of 7.\n",
      "\tReal user 2 has size of 5.\n",
      "\tReal user 3 has size of 11.\n",
      "\tReal user 4 has size of 12.\n"
     ]
    }
   ],
   "source": [
    "def make_dataset_fn(user_id):\n",
    "    # Create a `torch.utils.data.Dataset` for a single user\n",
    "    user_data = [torch.as_tensor(data) for data in user_id_to_data[user_id]]\n",
    "    pytorch_data = torch.utils.data.TensorDataset(*user_data)\n",
    "    return PyTorchDataDataset(raw_data=pytorch_data, user_id=user_id)\n",
    "\n",
    "federated_dataset = FederatedDataset(make_dataset_fn=make_dataset_fn, user_sampler=user_sampler)\n",
    "print(\"FederatedDataset with torch.utils.data.Dataset and real user partition: \")\n",
    "for i in range(5):\n",
    "    user_dataset, seed = next(federated_dataset)\n",
    "    print(f\"\\tReal user {i} has size of {len(user_dataset)}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "261ffc5d-3467-4880-b616-8954f209ddfa",
   "metadata": {},
   "source": [
    "### 3. Creating IID Artificial Federated Dataset\n",
    "The previous section assumes that a dataset has user IDs, which is often not the case for many existing ML datasets (e.g. CIFAR10). For those datasets, `pfl` supports converting them to Artificial Federated Dataset, meaning that there is no user ID associated with the dataset and the user partition will be artificial, which is useful for experimenting existing ML datasets. This section covers how to create an Artificial Federated Dataset assuming the data distribution between users is IID. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8142003a-ca94-4cd7-87cc-f2843806e0c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "User dataset size is a constant: \n",
      "\tArtificial user 0 has size of 10\n",
      "\tArtificial user 1 has size of 10\n",
      "\tArtificial user 2 has size of 10\n",
      "\tArtificial user 3 has size of 10\n",
      "\tArtificial user 4 has size of 10\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from pfl.data import ArtificialFederatedDataset, get_data_sampler\n",
    "\n",
    "\n",
    "n_samples = 1000\n",
    "features, labels = sklearn.datasets.make_classification(n_samples=n_samples, n_features=8, n_informative=4, n_classes=5)\n",
    "# Create data sampler to sample each artificial user dataset as a random subset of the original dataset\n",
    "data_sampler = get_data_sampler(sample_type=\"random\", max_bound=n_samples)\n",
    "\n",
    "# Option 1: Create an artificial federated dataset where each user dataset has constant size of 10\n",
    "sample_dataset_len = lambda: 10\n",
    "federated_dataset = ArtificialFederatedDataset.from_slices(\n",
    "    data=[features, labels], \n",
    "    data_sampler=data_sampler,\n",
    "    sample_dataset_len=sample_dataset_len,\n",
    ")\n",
    "# Iterate federated dataset to get artificial user dataset\n",
    "print(\"User dataset size is a constant: \")\n",
    "for i in range(5):\n",
    "    user_dataset, seed = next(federated_dataset)\n",
    "    print(f\"\\tArtificial user {i} has size of {len(user_dataset)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "851dcde1-fb72-4d82-90aa-26db3e4c3d06",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "User dataset size follows a Poisson distribution: \n",
      "\tArtificial user 0 has size of 12\n",
      "\tArtificial user 1 has size of 7\n",
      "\tArtificial user 2 has size of 7\n",
      "\tArtificial user 3 has size of 13\n",
      "\tArtificial user 4 has size of 12\n"
     ]
    }
   ],
   "source": [
    "# Option 2: Create an artificial federated dataset where each user dataset size follows Poisson distribution\n",
    "sample_dataset_len = lambda: max(1, np.random.poisson(10))\n",
    "federated_dataset = ArtificialFederatedDataset.from_slices(\n",
    "    data=[features, labels], \n",
    "    data_sampler=data_sampler,\n",
    "    sample_dataset_len=sample_dataset_len,\n",
    ")\n",
    "# Iterate federated dataset to get artificial user dataset\n",
    "print(\"\\nUser dataset size follows a Poisson distribution: \")\n",
    "for i in range(5):\n",
    "    user_dataset, seed = next(federated_dataset)\n",
    "    print(f\"\\tArtificial user {i} has size of {len(user_dataset)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0b7eeaa4-7c53-4f67-a849-15d2a822aec8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# For visualization in the notebook\n",
    "!pip -q install matplotlib PyQt5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "42036245-3f62-48a8-b004-782e591456cc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAHHCAYAAABZbpmkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA33UlEQVR4nO3deVyU5f7/8TeILCkMboAYCpLlUu5LamUqiWt6tNTyFNqeSy5t2kmRrEgr82im1cmtxcrvSbNMS0HbXDK1RTOz0tIUKBNwCSS4fn/0c44ToIIDM1y+no/H/Xg0133d13yuuWfi7b3M+BhjjAAAACzl6+kCAAAAyhJhBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHqMCuvvpqXX311Z4uAxbw8fHR5MmTPV0GUCYIO0A5WrBggXx8fJxLYGCgLr74Yo0cOVLp6emeLq/cRUdHq3fv3kWu+/zzz+Xj46MFCxaUb1HFKGrfRUZGKj4+XjNnztSRI0dKPfb69es1efJkZWZmuq/g/++TTz5Rjx49VKdOHQUGBqpu3brq06ePXnvtNbc/F+Ct/DxdAHA+euSRRxQTE6OcnBx98sknmjNnjt577z1t375dF1xwwVmP88EHH5RhlSjKyX2Xl5entLQ0rVu3TmPGjNH06dO1fPlyNW3atMRjrl+/XklJSRo6dKhCQ0PdVuuSJUs0aNAgNW/eXKNHj1a1atW0Z88effTRR3rxxRd14403Ovv+8ccf8vPjTwLsxDsb8IAePXqodevWkqTbbrtNNWrU0PTp0/X222/rhhtuOOtx/P39y6rE89KxY8dUpUqV0/Y5dd9J0oQJE5SamqrevXvr2muv1c6dOxUUFFTWpZ6VyZMnq3Hjxtq4cWOh90pGRobL48DAwPIsDShXnMYCvECXLl0kSXv27JEk/fnnn5oyZYpiY2MVEBCg6OhoPfTQQ8rNzXXZrqhrdmbNmqUmTZroggsuULVq1dS6dWuXUxZHjhzRmDFjFB0drYCAAIWFhemaa67R1q1bXcZZsmSJWrVqpaCgINWsWVP//Oc/9csvv7j0GTp0qKpWrapffvlF/fr1U9WqVVWrVi3dd999ys/Pd9fL45SWlqZhw4bpwgsvVEBAgGrXrq2+fftq7969Lv1WrlypK6+8UlWqVFFwcLB69eqlHTt2FFn7Dz/8oJ49eyo4OFhDhgwpVV1dunTRxIkT9dNPP+mVV15xtn/11VcaOnSo6tevr8DAQEVEROiWW27RoUOHnH0mT56s+++/X5IUExPjPE12ck7z589Xly5dFBYWpoCAADVu3Fhz5sw5q7p++OEHtWnTpshQHBYW5vL41Gt29u7d63LK7u/LqTZt2qTu3bvL4XDoggsuUKdOnfTpp5+eVX1AeeHIDuAFfvjhB0lSjRo1JP11tGfhwoW67rrrdO+992rTpk1KTk7Wzp07tXTp0mLHefHFF3XPPffouuuu0+jRo5WTk6OvvvpKmzZtcp6yuOuuu/R///d/GjlypBo3bqxDhw7pk08+0c6dO9WyZUtJf12fMmzYMLVp00bJyclKT0/Xv//9b3366afatm2by6mW/Px8xcfHq127dnrqqae0Zs0aPf3004qNjdXdd9/t1tdpwIAB2rFjh0aNGqXo6GhlZGRo9erV+vnnnxUdHS1Jevnll5WQkKD4+HhNnTpVx48f15w5c3TFFVdo27Ztzn7SX6EyPj5eV1xxhZ566qkSnUL8u5tuukkPPfSQPvjgA91+++2SpNWrV+vHH3/UsGHDFBERoR07duiFF17Qjh07tHHjRvn4+Kh///767rvvtHjxYj3zzDOqWbOmJKlWrVqSpDlz5qhJkya69tpr5efnp3feeUfDhw9XQUGBRowYcdqa6tWrp5SUFO3fv18XXnjhWc+lVq1aevnll13a8vLyNHbsWJfglJqaqh49eqhVq1ZKTEyUr6+vM5x9/PHHatu27Vk/J1CmDIByM3/+fCPJrFmzxvz6669m37595vXXXzc1atQwQUFBZv/+/eaLL74wksxtt93msu19991nJJnU1FRnW6dOnUynTp2cj/v27WuaNGly2hocDocZMWJEsetPnDhhwsLCzKWXXmr++OMPZ/u7775rJJlJkyY52xISEowk88gjj7iM0aJFC9OqVavT1mGMMfXq1TO9evUqct3mzZuNJDN//nxjjDGHDx82ksyTTz5Z7HhHjhwxoaGh5vbbb3dpT0tLMw6Hw6X9ZO3jx48/Y53G/G/fbd68udg+DofDtGjRwvn4+PHjhfosXrzYSDIfffSRs+3JJ580ksyePXsK9S9qjPj4eFO/fv0z1vzSSy8ZScbf39907tzZTJw40Xz88ccmPz+/UF9JJjExsdixhg8fbipVquR8/xUUFJgGDRqY+Ph4U1BQ4FJvTEyMueaaa85YH1BeOI0FeEBcXJxq1aqlqKgoDR48WFWrVtXSpUtVp04dvffee5KkcePGuWxz7733SpJWrFhR7LihoaHav3+/Nm/efNo+mzZt0oEDB4pc//nnnysjI0PDhw93uY6jV69eatiwYZHPf9ddd7k8vvLKK/Xjjz8WW0NpBAUFyd/fX+vWrdPhw4eL7LN69WplZmbqhhtu0G+//eZcKlWqpHbt2mnt2rWFtnHn0aeqVau63JV16rU7OTk5+u2333T55ZdLUqHThsU5dYysrCz99ttv6tSpk3788UdlZWWddttbbrlFq1at0tVXX61PPvlEU6ZM0ZVXXqkGDRpo/fr1Zz2vRYsW6bnnntO0adPUuXNnSdIXX3yh3bt368Ybb9ShQ4ecr/WxY8fUtWtXffTRRyooKDjr5wDKEqexAA+YPXu2Lr74Yvn5+Sk8PFyXXHKJfH3/+rfHTz/9JF9fX1100UUu20RERCg0NFQ//fRTseM++OCDWrNmjdq2bauLLrpI3bp104033qiOHTs6+0ybNk0JCQmKiopSq1at1LNnT918882qX7++8/kl6ZJLLik0fsOGDfXJJ5+4tAUGBjpPuZxUrVq1YgNJSZ28RiQgIEBTp07Vvffeq/DwcF1++eXq3bu3br75ZkVEREiSdu/eLel/10D9XUhIiMtjPz+/Ep3eOZOjR4+6XAvz+++/KykpSa+//nqhC4LPFFRO+vTTT5WYmKgNGzbo+PHjhcZwOByn3T4+Pl7x8fE6fvy4tmzZojfeeENz585V79699e233xa6dufvvvjiC91111264YYbXAL4ydc6ISGh2G2zsrJUrVq1M00RKHOEHcAD2rZt63JHT1H+fiHo2WjUqJF27dqld999V6tWrdJ///tfPffcc5o0aZKSkpIkSQMHDtSVV16ppUuX6oMPPtCTTz6pqVOn6q233lKPHj1K/JyVKlUq8TYnBQYG6o8//ihy3ck/7KceXRozZoz69OmjZcuW6f3339fEiROVnJys1NRUtWjRwnkk4eWXX3YGoFP9/dbqgIAAZ8g8V/v371dWVpZLSB04cKDWr1+v+++/X82bN1fVqlVVUFCg7t27n9VRjx9++EFdu3ZVw4YNNX36dEVFRcnf31/vvfeennnmmRIdObngggt05ZVX6sorr1TNmjWVlJSklStXnjasHD58WAMGDNDFF1+s//znPy7rTj73k08+qebNmxe5fdWqVc+6PqAsEXYAL1OvXj0VFBRo9+7datSokbM9PT1dmZmZqlev3mm3r1KligYNGqRBgwbpxIkT6t+/vx577DFNmDDBGRxq166t4cOHa/jw4crIyFDLli312GOPqUePHs7xd+3aVegIya5du874/CWd6zfffFPkul27djn7nCo2Nlb33nuv7r33Xu3evVvNmzfX008/rVdeeUWxsbGS/rrTKC4uzm11no2TF/TGx8dL+isopKSkKCkpSZMmTXL2O3lE5FTFBdt33nlHubm5Wr58uerWretsL+p0XEmcDNoHDx4stk9BQYGGDBmizMxMrVmzptDF2ydf65CQkHJ/rYGS4podwMv07NlTkjRjxgyX9unTp0v669qZ4px6S7P01/fwNG7cWMYY5eXlKT8/v9Dpk7CwMEVGRjpva2/durXCwsI0d+5cl1vdV65cqZ07d572+UuqZ8+e2r9/v5YtW+bSnpubq//85z8KCwtz3iF2/Phx5eTkuPSLjY1VcHCws874+HiFhITo8ccfV15eXqHn+/XXX91W+6lSU1M1ZcoUxcTEOG9fP3nEyxjj0vfv+1WS87t9/v4NykWNkZWVpfnz559VXSkpKUW2n7wurKhTlSclJSXp/fff1+LFixUTE1NofatWrRQbG6unnnpKR48eLbS+rF5roDQ4sgN4mWbNmikhIUEvvPCCMjMz1alTJ3322WdauHCh+vXr57xAtCjdunVTRESEOnbsqPDwcO3cuVPPPvusevXqpeDgYGVmZurCCy/Uddddp2bNmqlq1apas2aNNm/erKefflqSVLlyZU2dOlXDhg1Tp06ddMMNNzhvPY+OjtbYsWPdNtc77rhD8+bN0/XXX69bbrlFLVq00KFDh/TGG29o+/btWrRokfNW5++++05du3bVwIED1bhxY/n5+Wnp0qVKT0/X4MGDJf11lGHOnDm66aab1LJlSw0ePFi1atXSzz//rBUrVqhjx4569tlnz6nmlStX6ttvv9Wff/6p9PR0paamavXq1apXr56WL1/uPHoWEhKiq666StOmTVNeXp7q1KmjDz74wPldSqdq1aqVJOlf//qXBg8erMqVK6tPnz7q1q2b/P391adPH9155506evSoXnzxRYWFhZ32qMxJffv2VUxMjPr06aPY2FgdO3ZMa9as0TvvvKM2bdqoT58+RW739ddfa8qUKbrqqquUkZHh8t1BkvTPf/5Tvr6++s9//qMePXqoSZMmGjZsmOrUqaNffvlFa9euVUhIiN55552SvrxA2fDw3WDAeeVsbl82xpi8vDyTlJRkYmJiTOXKlU1UVJSZMGGCycnJcen391vPn3/+eXPVVVeZGjVqmICAABMbG2vuv/9+k5WVZYwxJjc319x///2mWbNmJjg42FSpUsU0a9bMPPfcc4VqeOONN0yLFi1MQECAqV69uhkyZIjZv3+/S5+EhARTpUqVQtsmJiaas/3fy+HDh83YsWOdcw0JCTGdO3c2K1eudOn322+/mREjRpiGDRuaKlWqGIfDYdq1a2fefPPNQmOuXbvWxMfHG4fDYQIDA01sbKwZOnSo+fzzz89Ye3FO7ruTi7+/v4mIiDDXXHON+fe//22ys7MLbbN//37zj3/8w4SGhhqHw2Guv/56c+DAgSJv854yZYqpU6eO8fX1dbkNffny5aZp06YmMDDQREdHm6lTp5p58+YVe6v6qRYvXmwGDx5sYmNjTVBQkAkMDDSNGzc2//rXvwrVe2pNa9eudZnr35dTbdu2zfTv39/5nqtXr54ZOHCgSUlJOevXFihrPsb87RgrAACARbhmBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAanypoP76WvQDBw4oODi4VL9HBAAAyp8xRkeOHFFkZORpf+eOsCPpwIEDioqK8nQZAACgFPbt26cLL7yw2PWEHUnBwcGS/nqxQkJCPFwNAAA4G9nZ2YqKinL+HS8OYUf/+8XhkJAQwg4AABXMmS5B4QJlAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNX8PF0AAJS16PErPF1Cie19openSwCswZEdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1fw8XQAAoLDo8Ss8XUKJ7X2il6dLAIrEkR0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAq3k07Hz00Ufq06ePIiMj5ePjo2XLlrmsN8Zo0qRJql27toKCghQXF6fdu3e79Pn99981ZMgQhYSEKDQ0VLfeequOHj1ajrMAAADezM+TT37s2DE1a9ZMt9xyi/r3719o/bRp0zRz5kwtXLhQMTExmjhxouLj4/XNN98oMDBQkjRkyBAdPHhQq1evVl5enoYNG6Y77rhDr732WnlPByiR6PErPF1Cqex9openSwCAEvFo2OnRo4d69OhR5DpjjGbMmKGHH35Yffv2lSQtWrRI4eHhWrZsmQYPHqydO3dq1apV2rx5s1q3bi1JmjVrlnr27KmnnnpKkZGR5TYXAADgnbz2mp09e/YoLS1NcXFxzjaHw6F27dppw4YNkqQNGzYoNDTUGXQkKS4uTr6+vtq0aVOxY+fm5io7O9tlAQAAdvLasJOWliZJCg8Pd2kPDw93rktLS1NYWJjLej8/P1WvXt3ZpyjJyclyOBzOJSoqys3VAwAAb+G1YacsTZgwQVlZWc5l3759ni4JAACUEa8NOxEREZKk9PR0l/b09HTnuoiICGVkZLis//PPP/X77787+xQlICBAISEhLgsAALCT14admJgYRUREKCUlxdmWnZ2tTZs2qX379pKk9u3bKzMzU1u2bHH2SU1NVUFBgdq1a1fuNQMAAO/j0buxjh49qu+//975eM+ePfriiy9UvXp11a1bV2PGjNGjjz6qBg0aOG89j4yMVL9+/SRJjRo1Uvfu3XX77bdr7ty5ysvL08iRIzV48GDuxAIAAJI8HHY+//xzde7c2fl43LhxkqSEhAQtWLBADzzwgI4dO6Y77rhDmZmZuuKKK7Rq1Srnd+xI0quvvqqRI0eqa9eu8vX11YABAzRz5sxynwsAAPBOPsYY4+kiPC07O1sOh0NZWVlcv4Nyw5cKlp+K+lpXNBXxvYGK7Wz/fnvtNTsAAADuQNgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABW8+qwk5+fr4kTJyomJkZBQUGKjY3VlClTZIxx9jHGaNKkSapdu7aCgoIUFxen3bt3e7BqAADgTbw67EydOlVz5szRs88+q507d2rq1KmaNm2aZs2a5ewzbdo0zZw5U3PnztWmTZtUpUoVxcfHKycnx4OVAwAAb+Hn6QJOZ/369erbt6969eolSYqOjtbixYv12WefSfrrqM6MGTP08MMPq2/fvpKkRYsWKTw8XMuWLdPgwYM9VjsAAPAOXn1kp0OHDkpJSdF3330nSfryyy/1ySefqEePHpKkPXv2KC0tTXFxcc5tHA6H2rVrpw0bNnikZgAA4F28+sjO+PHjlZ2drYYNG6pSpUrKz8/XY489piFDhkiS0tLSJEnh4eEu24WHhzvXFSU3N1e5ubnOx9nZ2WVQPQAA8AZefWTnzTff1KuvvqrXXntNW7du1cKFC/XUU09p4cKF5zRucnKyHA6Hc4mKinJTxQAAwNt4ddi5//77NX78eA0ePFiXXXaZbrrpJo0dO1bJycmSpIiICElSenq6y3bp6enOdUWZMGGCsrKynMu+ffvKbhIAAMCjvDrsHD9+XL6+riVWqlRJBQUFkqSYmBhFREQoJSXFuT47O1ubNm1S+/btix03ICBAISEhLgsAALCTV1+z06dPHz322GOqW7eumjRpom3btmn69Om65ZZbJEk+Pj4aM2aMHn30UTVo0EAxMTGaOHGiIiMj1a9fP88WDwAAvIJXh51Zs2Zp4sSJGj58uDIyMhQZGak777xTkyZNcvZ54IEHdOzYMd1xxx3KzMzUFVdcoVWrVikwMNCDlQMAAG/hY079OuLzVHZ2thwOh7KysjilhXITPX6Fp0solb1P9PJ0CSVWUV/riqYivjdQsZ3t32+vvmYHAADgXBF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALCan6cLAFCxRI9f4ekSAKBESnVkp379+jp06FCh9szMTNWvX/+ciwIAAHCXUoWdvXv3Kj8/v1B7bm6ufvnll3MuCgAAwF1KdBpr+fLlzv9+//335XA4nI/z8/OVkpKi6OhotxUHAABwrkoUdvr16ydJ8vHxUUJCgsu6ypUrKzo6Wk8//bTbigMAADhXJQo7BQUFkqSYmBht3rxZNWvWLJOiAAAA3KVUd2Pt2bPH3XUAAACUiVLfep6SkqKUlBRlZGQ4j/icNG/evHMuDAAAwB1KFXaSkpL0yCOPqHXr1qpdu7Z8fHzcXRcAAIBblCrszJ07VwsWLNBNN93k7noAAADcqlTfs3PixAl16NDB3bUAAAC4XanCzm233abXXnvN3bUAAAC4XalOY+Xk5OiFF17QmjVr1LRpU1WuXNll/fTp091SHAAAwLkqVdj56quv1Lx5c0nS9u3bXdZxsTIAAPAmpQo7a9eudXcdxfrll1/04IMPauXKlTp+/LguuugizZ8/X61bt5YkGWOUmJioF198UZmZmerYsaPmzJmjBg0alFuNAADAe5Xqmp3ycvjwYXXs2FGVK1fWypUr9c033+jpp59WtWrVnH2mTZummTNnau7cudq0aZOqVKmi+Ph45eTkeLByAADgLUp1ZKdz586nPV2Vmppa6oJONXXqVEVFRWn+/PnOtpiYGOd/G2M0Y8YMPfzww+rbt68kadGiRQoPD9eyZcs0ePBgt9QBAAAqrlId2WnevLmaNWvmXBo3bqwTJ05o69atuuyyy9xW3PLly9W6dWtdf/31CgsLU4sWLfTiiy861+/Zs0dpaWmKi4tztjkcDrVr104bNmwodtzc3FxlZ2e7LAAAwE6lOrLzzDPPFNk+efJkHT169JwKOtWPP/6oOXPmaNy4cXrooYe0efNm3XPPPfL391dCQoLS0tIkSeHh4S7bhYeHO9cVJTk5WUlJSW6rEwAAeC+3XrPzz3/+062/i1VQUKCWLVvq8ccfV4sWLXTHHXfo9ttv19y5c89p3AkTJigrK8u57Nu3z00VAwAAb+PWsLNhwwYFBga6bbzatWurcePGLm2NGjXSzz//LEmKiIiQJKWnp7v0SU9Pd64rSkBAgEJCQlwWAABgp1Kdxurfv7/LY2OMDh48qM8//1wTJ050S2GS1LFjR+3atcul7bvvvlO9evUk/XWxckREhFJSUpzf+5Odna1Nmzbp7rvvdlsdAACg4ipV2HE4HC6PfX19dckll+iRRx5Rt27d3FKYJI0dO1YdOnTQ448/roEDB+qzzz7TCy+8oBdeeEHSX19gOGbMGD366KNq0KCBYmJiNHHiREVGRqpfv35uqwMAAFRcpQo7p94KXpbatGmjpUuXasKECXrkkUcUExOjGTNmaMiQIc4+DzzwgI4dO6Y77rhDmZmZuuKKK7Rq1Sq3nk4DAAAVl48xxpR24y1btmjnzp2SpCZNmqhFixZuK6w8ZWdny+FwKCsri+t3UG6ix6/wdAmAW+19openS8B55mz/fpfqyE5GRoYGDx6sdevWKTQ0VJKUmZmpzp076/XXX1etWrVKVTQAAIC7lepurFGjRunIkSPasWOHfv/9d/3+++/avn27srOzdc8997i7RgAAgFIr1ZGdVatWac2aNWrUqJGzrXHjxpo9e7ZbL1AGAAA4V6U6slNQUKDKlSsXaq9cubIKCgrOuSgAAAB3KVXY6dKli0aPHq0DBw4423755ReNHTtWXbt2dVtxAAAA56pUYefZZ59Vdna2oqOjFRsbq9jYWMXExCg7O1uzZs1yd40AAAClVqprdqKiorR161atWbNG3377raS/fsbh1F8fBwAA8AYlOrKTmpqqxo0bKzs7Wz4+Prrmmms0atQojRo1Sm3atFGTJk308ccfl1WtAAAAJVaisDNjxgzdfvvtRX5xj8Ph0J133qnp06e7rTgAAIBzVaKw8+WXX6p79+7Fru/WrZu2bNlyzkUBAAC4S4nCTnp6epG3nJ/k5+enX3/99ZyLAgAAcJcShZ06depo+/btxa7/6quvVLt27XMuCgAAwF1KFHZ69uypiRMnKicnp9C6P/74Q4mJierdu7fbigMAADhXJbr1/OGHH9Zbb72liy++WCNHjtQll1wiSfr22281e/Zs5efn61//+leZFAoAAFAaJQo74eHhWr9+ve6++25NmDBBxhhJko+Pj+Lj4zV79myFh4eXSaEAAAClUeIvFaxXr57ee+89HT58WN9//72MMWrQoIGqVatWFvUBAACck1J9g7IkVatWTW3atHFnLQAAAG5Xqt/GAgAAqCgIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGp+ni4AcIfo8Ss8XQIAwEtxZAcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYLUKFXaeeOIJ+fj4aMyYMc62nJwcjRgxQjVq1FDVqlU1YMAApaene65IAADgVSpM2Nm8ebOef/55NW3a1KV97Nixeuedd7RkyRJ9+OGHOnDggPr37++hKgEAgLepEGHn6NGjGjJkiF588UVVq1bN2Z6VlaWXXnpJ06dPV5cuXdSqVSvNnz9f69ev18aNGz1YMQAA8BYVIuyMGDFCvXr1UlxcnEv7li1blJeX59LesGFD1a1bVxs2bCh2vNzcXGVnZ7ssAADATl7/21ivv/66tm7dqs2bNxdal5aWJn9/f4WGhrq0h4eHKy0trdgxk5OTlZSU5O5SAQCAF/LqIzv79u3T6NGj9eqrryowMNBt406YMEFZWVnOZd++fW4bGwAAeBevDjtbtmxRRkaGWrZsKT8/P/n5+enDDz/UzJkz5efnp/DwcJ04cUKZmZku26WnpysiIqLYcQMCAhQSEuKyAAAAO3n1aayuXbvq66+/dmkbNmyYGjZsqAcffFBRUVGqXLmyUlJSNGDAAEnSrl279PPPP6t9+/aeKBkAAHgZrw47wcHBuvTSS13aqlSpoho1ajjbb731Vo0bN07Vq1dXSEiIRo0apfbt2+vyyy/3RMkAAMDLeHXYORvPPPOMfH19NWDAAOXm5io+Pl7PPfecp8sCAABewscYYzxdhKdlZ2fL4XAoKyuL63cqqOjxKzxdAnDe2/tEL0+XgPPM2f799uoLlAEAAM4VYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKzm5+kCAAB2iB6/wtMllNjeJ3p5ugSUA47sAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArObVYSc5OVlt2rRRcHCwwsLC1K9fP+3atculT05OjkaMGKEaNWqoatWqGjBggNLT0z1UMQAA8DZeHXY+/PBDjRgxQhs3btTq1auVl5enbt266dixY84+Y8eO1TvvvKMlS5boww8/1IEDB9S/f38PVg0AALyJn6cLOJ1Vq1a5PF6wYIHCwsK0ZcsWXXXVVcrKytJLL72k1157TV26dJEkzZ8/X40aNdLGjRt1+eWXe6JsAADgRbz6yM7fZWVlSZKqV68uSdqyZYvy8vIUFxfn7NOwYUPVrVtXGzZsKHac3NxcZWdnuywAAMBOXn1k51QFBQUaM2aMOnbsqEsvvVSSlJaWJn9/f4WGhrr0DQ8PV1paWrFjJScnKykpqSzLBQBUANHjV3i6hBLb+0QvT5dQ4VSYIzsjRozQ9u3b9frrr5/zWBMmTFBWVpZz2bdvnxsqBAAA3qhCHNkZOXKk3n33XX300Ue68MILne0RERE6ceKEMjMzXY7upKenKyIiotjxAgICFBAQUJYlAwAAL+HVR3aMMRo5cqSWLl2q1NRUxcTEuKxv1aqVKleurJSUFGfbrl279PPPP6t9+/blXS4AAPBCXn1kZ8SIEXrttdf09ttvKzg42HkdjsPhUFBQkBwOh2699VaNGzdO1atXV0hIiEaNGqX27dtzJxYAAJDk5WFnzpw5kqSrr77apX3+/PkaOnSoJOmZZ56Rr6+vBgwYoNzcXMXHx+u5554r50oBAIC38uqwY4w5Y5/AwEDNnj1bs2fPLoeKAABARePV1+wAAACcK8IOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqXv2lgvCM6PErPF0CAABuw5EdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1P08XAAAAzl70+BWeLqHE9j7Ry6PPz5EdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVrAk7s2fPVnR0tAIDA9WuXTt99tlnni4JAAB4ASvCzhtvvKFx48YpMTFRW7duVbNmzRQfH6+MjAxPlwYAADzMxxhjPF3EuWrXrp3atGmjZ599VpJUUFCgqKgojRo1SuPHjz/j9tnZ2XI4HMrKylJISIhba6uI33QJAIA7ldU3KJ/t3+8Kf2TnxIkT2rJli+Li4pxtvr6+iouL04YNGzxYGQAA8AYV/rexfvvtN+Xn5ys8PNylPTw8XN9++22R2+Tm5io3N9f5OCsrS9JfCdHdCnKPu31MAAAqkrL4+3rquGc6SVXhw05pJCcnKykpqVB7VFSUB6oBAMBujhllO/6RI0fkcDiKXV/hw07NmjVVqVIlpaenu7Snp6crIiKiyG0mTJigcePGOR8XFBTo999/V40aNeTj41Om9XqD7OxsRUVFad++fW6/Rsmbna/zlpj7+Tj383XeEnM/n+ZujNGRI0cUGRl52n4VPuz4+/urVatWSklJUb9+/ST9FV5SUlI0cuTIIrcJCAhQQECAS1toaGgZV+p9QkJCzosPw9+dr/OWmPv5OPfzdd4Scz9f5n66IzonVfiwI0njxo1TQkKCWrdurbZt22rGjBk6duyYhg0b5unSAACAh1kRdgYNGqRff/1VkyZNUlpampo3b65Vq1YVumgZAACcf6wIO5I0cuTIYk9bwVVAQIASExMLncqz3fk6b4m5n49zP1/nLTH383Xup2PFlwoCAAAUp8J/qSAAAMDpEHYAAIDVCDsAAMBqhB0AAGA1wo5lkpOT1aZNGwUHByssLEz9+vXTrl27TrvNggUL5OPj47IEBgaWU8XuM3ny5ELzaNiw4Wm3WbJkiRo2bKjAwEBddtlleu+998qpWveJjo4uNG8fHx+NGDGiyP4VeX9/9NFH6tOnjyIjI+Xj46Nly5a5rDfGaNKkSapdu7aCgoIUFxen3bt3n3Hc2bNnKzo6WoGBgWrXrp0+++yzMppB6Zxu3nl5eXrwwQd12WWXqUqVKoqMjNTNN9+sAwcOnHbM0nxePOFM+3zo0KGF5tG9e/czjuvt+1w689yL+tz7+PjoySefLHbMirLf3Y2wY5kPP/xQI0aM0MaNG7V69Wrl5eWpW7duOnbs2Gm3CwkJ0cGDB53LTz/9VE4Vu1eTJk1c5vHJJ58U23f9+vW64YYbdOutt2rbtm3q16+f+vXrp+3bt5djxedu8+bNLnNevXq1JOn6668vdpuKur+PHTumZs2aafbs2UWunzZtmmbOnKm5c+dq06ZNqlKliuLj45WTk1PsmG+88YbGjRunxMREbd26Vc2aNVN8fLwyMjLKaholdrp5Hz9+XFu3btXEiRO1detWvfXWW9q1a5euvfbaM45bks+Lp5xpn0tS9+7dXeaxePHi045ZEfa5dOa5nzrngwcPat68efLx8dGAAQNOO25F2O9uZ2C1jIwMI8l8+OGHxfaZP3++cTgc5VdUGUlMTDTNmjU76/4DBw40vXr1cmlr166dufPOO91cWfkaPXq0iY2NNQUFBUWut2V/SzJLly51Pi4oKDARERHmySefdLZlZmaagIAAs3jx4mLHadu2rRkxYoTzcX5+vomMjDTJycllUve5+vu8i/LZZ58ZSeann34qtk9JPy/eoKi5JyQkmL59+5ZonIq2z405u/3et29f06VLl9P2qYj73R04smO5rKwsSVL16tVP2+/o0aOqV6+eoqKi1LdvX+3YsaM8ynO73bt3KzIyUvXr19eQIUP0888/F9t3w4YNiouLc2mLj4/Xhg0byrrMMnPixAm98soruuWWW077o7a27O9T7dmzR2lpaS771OFwqF27dsXu0xMnTmjLli0u2/j6+iouLq5Cvw+ysrLk4+Nzxt/8K8nnxZutW7dOYWFhuuSSS3T33Xfr0KFDxfa1dZ+np6drxYoVuvXWW8/Y15b9XhKEHYsVFBRozJgx6tixoy699NJi+11yySWaN2+e3n77bb3yyisqKChQhw4dtH///nKs9ty1a9dOCxYs0KpVqzRnzhzt2bNHV155pY4cOVJk/7S0tEI/KRIeHq60tLTyKLdMLFu2TJmZmRo6dGixfWzZ3393cr+VZJ/+9ttvys/Pt+p9kJOTowcffFA33HDDaX8IsqSfF2/VvXt3LVq0SCkpKZo6dao+/PBD9ejRQ/n5+UX2t3GfS9LChQsVHBys/v37n7afLfu9pKz5uQgUNmLECG3fvv2M52Pbt2+v9u3bOx936NBBjRo10vPPP68pU6aUdZlu06NHD+d/N23aVO3atVO9evX05ptvntW/dmzw0ksvqUePHoqMjCy2jy37G4Xl5eVp4MCBMsZozpw5p+1ry+dl8ODBzv++7LLL1LRpU8XGxmrdunXq2rWrBysrX/PmzdOQIUPOeLOBLfu9pDiyY6mRI0fq3Xff1dq1a3XhhReWaNvKlSurRYsW+v7778uouvIRGhqqiy++uNh5REREKD093aUtPT1dERER5VGe2/30009as2aNbrvtthJtZ8v+PrnfSrJPa9asqUqVKlnxPjgZdH766SetXr36tEd1inKmz0tFUb9+fdWsWbPYedi0z0/6+OOPtWvXrhJ/9iV79vuZEHYsY4zRyJEjtXTpUqWmpiomJqbEY+Tn5+vrr79W7dq1y6DC8nP06FH98MMPxc6jffv2SklJcWlbvXq1y1GPimT+/PkKCwtTr169SrSdLfs7JiZGERERLvs0OztbmzZtKnaf+vv7q1WrVi7bFBQUKCUlpUK9D04Gnd27d2vNmjWqUaNGicc40+eloti/f78OHTpU7Dxs2eeneumll9SqVSs1a9asxNvast/PyNNXSMO97r77buNwOMy6devMwYMHncvx48edfW666SYzfvx45+OkpCTz/vvvmx9++MFs2bLFDB482AQGBpodO3Z4Ygqldu+995p169aZPXv2mE8//dTExcWZmjVrmoyMDGNM4Xl/+umnxs/Pzzz11FNm586dJjEx0VSuXNl8/fXXnppCqeXn55u6deuaBx98sNA6m/b3kSNHzLZt28y2bduMJDN9+nSzbds2511HTzzxhAkNDTVvv/22+eqrr0zfvn1NTEyM+eOPP5xjdOnSxcyaNcv5+PXXXzcBAQFmwYIF5ptvvjF33HGHCQ0NNWlpaeU+v+Kcbt4nTpww1157rbnwwgvNF1984fK5z83NdY7x93mf6fPiLU439yNHjpj77rvPbNiwwezZs8esWbPGtGzZ0jRo0MDk5OQ4x6iI+9yYM7/fjTEmKyvLXHDBBWbOnDlFjlFR97u7EXYsI6nIZf78+c4+nTp1MgkJCc7HY8aMMXXr1jX+/v4mPDzc9OzZ02zdurX8iz9HgwYNMrVr1zb+/v6mTp06ZtCgQeb77793rv/7vI0x5s033zQXX3yx8ff3N02aNDErVqwo56rd4/333zeSzK5duwqts2l/r127tsj398n5FRQUmIkTJ5rw8HATEBBgunbtWug1qVevnklMTHRpmzVrlvM1adu2rdm4cWM5zejsnG7ee/bsKfZzv3btWucYf5/3mT4v3uJ0cz9+/Ljp1q2bqVWrlqlcubKpV6+euf322wuFloq4z4058/vdGGOef/55ExQUZDIzM4sco6Lud3fzMcaYMj10BAAA4EFcswMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwCKcPXVV2vMmDGeLgOAGxB2AJS5oUOHysfHRz4+PqpcubLCw8N1zTXXaN68eSooKCjRWAsWLFBoaGjZFHqKt956i1+BByxB2AFQLrp3766DBw9q7969WrlypTp37qzRo0erd+/e+vPPPz1dXiHVq1dXcHCwp8sA4AaEHQDlIiAgQBEREapTp45atmyphx56SG+//bZWrlypBQsWOPtNnz5dl112mapUqaKoqCgNHz5cR48elSStW7dOw4YNU1ZWlvNI0eTJkyVJL7/8slq3bq3g4GBFREToxhtvVEZGxmlreu6559SgQQMFBgYqPDxc1113nXPdqaex1q1b53y+U5ehQ4c6+7/99ttq2bKlAgMDVb9+fSUlJXlliAPOR4QdAB7TpUsXNWvWTG+99ZazzdfXVzNnztSOHTu0cOFCpaam6oEHHpAkdejQQTNmzFBISIgOHjyogwcP6r777pMk5eXlacqUKfryyy+1bNky7d271yWM/N3nn3+ue+65R4888oh27dqlVatW6aqrriqyb4cOHZzPd/DgQaWmpiowMNDZ/+OPP9bNN9+s0aNH65tvvtHzzz+vBQsW6LHHHnPTKwXgnHj6l0gB2C8hIcH07du3yHWDBg0yjRo1KnbbJUuWmBo1ajgfz58/3zgcjjM+5+bNm40kc+TIkSLX//e//zUhISEmOzu7yPWdOnUyo0ePLtT+22+/mfr165vhw4c727p27Woef/xxl34vv/yyqV279hnrBFD2/DwdtgCc34wx8vHxcT5es2aNkpOT9e233yo7O1t//vmncnJydPz4cV1wwQXFjrNlyxZNnjxZX375pQ4fPuy88Pnnn39W48aNC/W/5pprVK9ePdWvX1/du3dX9+7d9Y9//OO0z5GXl6cBAwaoXr16+ve//+1s//LLL/Xpp5+6HMnJz88/q7oBlD1OYwHwqJ07dyomJkaStHfvXvXu3VtNmzbVf//7X23ZskWzZ8+WJJ04caLYMY4dO6b4+HiFhITo1Vdf1ebNm7V06dLTbhccHKytW7dq8eLFql27tiZNmqRmzZopMzOz2Oe5++67tW/fPi1ZskR+fv/7t+LRo0eVlJSkL774wrl8/fXX2r17twIDA0v6kgBwM47sAPCY1NRUff311xo7dqykv47OFBQU6Omnn5av71//FnvzzTddtvH391d+fr5L27fffqtDhw7piSeeUFRUlKS/rsk5Ez8/P8XFxSkuLk6JiYkKDQ1Vamqq+vfvX6jv9OnT9eabb2r9+vWqUaOGy7qWLVtq165duuiii85+8gDKDWEHQLnIzc1VWlqa8vPzlZ6erlWrVik5OVm9e/fWzTffLEm66KKLlJeXp1mzZqlPnz769NNPNXfuXJdxoqOjdfToUaWkpKhZs2a64IILVLduXfn7+2vWrFm66667tH379jN+R867776rH3/8UVdddZWqVaum9957TwUFBbrkkksK9V2zZo0eeOABzZ49WzVr1lRaWpokKSgoSA6HQ5MmTVLv3r1Vt25dXXfddfL19dWXX36p7du369FHH3XTKwig1Dx90RAA+yUkJBhJRpLx8/MztWrVMnFxcWbevHkmPz/fpe/06dNN7dq1TVBQkImPjzeLFi0ykszhw4edfe666y5To0YNI8kkJiYaY4x57bXXTHR0tAkICDDt27c3y5cvN5LMtm3biqzp448/Np06dTLVqlUzQUFBpmnTpuaNN95wrj/1AuXExERn/acuCQkJzv6rVq0yHTp0MEFBQSYkJMS0bdvWvPDCC+54+QCcIx9jjPFU0AIAAChrXKAMAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNX+H8/+42s0E9cLAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Visualize the size of user dataset with Poisson distribution\n",
    "n = 500\n",
    "plt.hist([len(next(federated_dataset)[0]) for _ in range(n)])\n",
    "plt.title('Poisson User Data Size')\n",
    "plt.xlabel('Data size')\n",
    "plt.ylabel('Count')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6a557b6-89c6-414e-9c53-26eb859d0498",
   "metadata": {},
   "source": [
    "### 4. Creating non-IID Artificial Federated Dataset\n",
    "As mentioned before, in the real federated setting, the federated data has many non-IID characteristics in its distribution. This section shows how to artificially simulate the case for a classification dataset where the user label distribution is non-IID and follows a Dirichlet distribution (a common assumption in federated learning research). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23eefd21-b0a2-40f2-aa87-ab6793fad10a",
   "metadata": {},
   "source": [
    "#### 4.1 Sampling non-IID user dataset dynamically with `ArtificialFederatedDataset`\n",
    "The first option is to change the `data_sampler` in a way so that the user label distribution follows a Dirichlet distribution. The user dataset is sampled on the fly and there is no fixed user partition like as in [Section 3.2](#3.2-Partitioning-the-dataset-into-fixed-artificial-users-with-FederatedDataset) below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4be5ebb8-cd2c-4751-9741-31450bf7c941",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ArtificialFederatedDataset with Dirichlet label distribution: \n",
      "\tArtificial user 0 has size of 20 with unique labels=[0 1]\n",
      "\tArtificial user 1 has size of 20 with unique labels=[1 4]\n",
      "\tArtificial user 2 has size of 20 with unique labels=[0 1]\n",
      "\tArtificial user 3 has size of 20 with unique labels=[0]\n",
      "\tArtificial user 4 has size of 20 with unique labels=[1 4]\n"
     ]
    }
   ],
   "source": [
    "# Create data sampler to sample each artificial user has label distribution follows a Dirichlet distribution.\n",
    "dirichlet_alpha = [0.1] * 5\n",
    "data_sampler = get_data_sampler(sample_type=\"dirichlet\", labels=labels, alpha=dirichlet_alpha)\n",
    "\n",
    "# Create an artificial federated dataset with a Dirichlet data sampler\n",
    "sample_dataset_len = lambda: 20\n",
    "federated_dataset = ArtificialFederatedDataset.from_slices(\n",
    "    data=[features, labels], \n",
    "    data_sampler=data_sampler,\n",
    "    sample_dataset_len=sample_dataset_len,\n",
    ")\n",
    "# Iterate federated dataset to get artificial user dataset\n",
    "print(\"ArtificialFederatedDataset with Dirichlet label distribution: \")\n",
    "for i in range(5):\n",
    "    user_dataset, seed = next(federated_dataset)\n",
    "    print(f\"\\tArtificial user {i} has size of {len(user_dataset)} with unique labels={np.unique(user_dataset.raw_data[1])}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f7ec6ba8-b68c-451f-b87a-5a08f106c5c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAHHCAYAAABZbpmkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA2RklEQVR4nO3deVhWdf7/8dcNCrgBUgqSqLijuY2mUSamJJpapjNq4xTOaLaAS15tNu4t1EypZZY1M2lZ5tKMNmkuhFsalmsuoanhLmqpoKao3J/fH/Pl/nULKOCtN3x6Pq7rXJfnnM855/05H+/Ll2e5b4cxxggAAMBSPt4uAAAA4Hoi7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsABYYO3asHA5Hods7HA6NHTu2SMeYPn26HA6H1q9ff9W27du3V/v27Yu0fxRN7pj/9NNPHttn//79VatWLY/tDygpCDtACZMbKnKngIAAhYeHKy4uTm+++aZOnz7t7RKvq5dfflnz588vVNsVK1bI4XDo008/zXd9YmJikULg9da+fXvdeuut3i4D+M0h7AAl1Pjx4zVjxgy98847Gjx4sCRp2LBhatKkibZs2eLWduTIkTp37lyh933u3DmNHDnSo/V6SlHCDgAURhlvFwAgf126dFGrVq1c8yNGjNCyZcvUrVs33XfffUpLS1O5cuUkSWXKlFGZMlf+ODudTl24cEEBAQEKCAi4rrX/Vv36HAMoObiyA5QiHTp00KhRo7Rv3z599NFHruX5PbPjcDiUmJiojz/+WI0bN5a/v78WL17sWnf5MzuHDh3SgAEDFB4eLn9/f0VGRurxxx/XhQsX3NplZ2dr+PDhqlKliipUqKAHHnhAx48fv2rt2dnZGjNmjOrWrSt/f39FRETomWeeUXZ2tlvNZ8+e1QcffOC6jde/f/8inqUrS05OVtu2bRUcHKyKFSuqQYMGev7554tca269BZ3j4tqyZYv69++v2rVrKyAgQGFhYfrLX/6in3/+Od/2P/30k3r37q3AwEDddNNNGjp0qM6fP5+n3UcffaSWLVuqXLlyCgkJUd++fXXgwIGr1jNr1iy1bNlSlSpVUmBgoJo0aaI33njjmvoI3Ghc2QFKmYceekjPP/+8li5dqkceeeSKbZctW6Y5c+YoMTFRN998c4EPnx4+fFitW7fWqVOnNGjQIDVs2FCHDh3Sp59+ql9++UV+fn6utoMHD1blypU1ZswY7d27V5MmTVJiYqJmz55dYB1Op1P33XefVq9erUGDBikqKkpbt27VxIkT9cMPP7huW82YMUMDBw5U69atNWjQIElSnTp1inaCrmD79u3q1q2bmjZtqvHjx8vf31+7d+/WmjVrilxrrsKe48JKTk7Wjz/+qD//+c8KCwvT9u3b9d5772n79u1au3ZtnlDbu3dv1apVS0lJSVq7dq3efPNNnTx5Uh9++KGrzUsvvaRRo0apd+/eGjhwoI4fP67JkyerXbt22rRpk4KDgwus5cEHH1THjh316quvSpLS0tK0Zs0aDR069Jr6CdxQBkCJMm3aNCPJrFu3rsA2QUFBpkWLFq75MWPGmMs/zpKMj4+P2b59e57tJZkxY8a45h9++GHj4+OT7zGdTqdbXbGxsa5lxhjz5JNPGl9fX3Pq1CnXspiYGBMTE+OanzFjhvHx8TFfffWV276nTp1qJJk1a9a4llWoUMHEx8cX2PdfW758uZFk5s6dm+/6hIQEt/MyceJEI8kcP368wH0WpdYrneP8xMTEmMaNG1+xzS+//JJn2SeffGIkmVWrVrmW5Y75fffd59b2iSeeMJLMd999Z4wxZu/evcbX19e89NJLbu22bt1qypQp47Y8Pj7e1KxZ0zU/dOhQExgYaC5dulSo/gElFbexgFKoYsWKhXorKyYmRo0aNbpiG6fTqfnz56t79+5uzwjluvxKwqBBg9yW3XXXXcrJydG+ffsKPMbcuXMVFRWlhg0b6qeffnJNHTp0kCQtX778qn3xhNwrGJ999pmcTqdHai3MOS6K3OewJOn8+fP66aefdPvtt0uSNm7cmKd9QkKC23zuw+xffPGFJOk///mPnE6nevfu7dafsLAw1atX74rnPjg4WGfPnlVycvI19wvwJsIOUAqdOXNGlSpVumq7yMjIq7Y5fvy4srKyCv1KdI0aNdzmK1euLEk6efJkgdvs2rVL27dvV5UqVdym+vXrS5KOHTtWqGNfqz59+ujOO+/UwIEDFRoaqr59+2rOnDluwaeotRbmHBfFiRMnNHToUIWGhqpcuXKqUqWK6xiZmZl52terV89tvk6dOvLx8dHevXtd/THGqF69enn6lJaWdsVz/8QTT6h+/frq0qWLqlevrr/85S/X/EwS4A08swOUMgcPHlRmZqbq1q171ba/vkrgKb6+vvkuN8YUuI3T6VSTJk00YcKEfNdHREQUq5bct54Keu3+l19+cXszqly5clq1apWWL1+uhQsXavHixZo9e7Y6dOigpUuXytfXt8i1evoc9+7dW19//bWefvppNW/eXBUrVpTT6VTnzp0LvBr1a5dfiXM6nXI4HFq0aFG+Y1exYsUC91W1alVt3rxZS5Ys0aJFi7Ro0SJNmzZNDz/8sD744IOidw7wEsIOUMrMmDFDkhQXF+eR/VWpUkWBgYHatm2bR/aXnzp16ui7775Tx44dr/olf0X5EsCaNWtKknbu3Jnv+p07d7ra5PLx8VHHjh3VsWNHTZgwQS+//LL++te/avny5YqNjS1SrZ528uRJpaSkaNy4cRo9erRr+a5duwrcZteuXW5Xl3bv3i2n0+l6ULpOnToyxigyMtJ1daoo/Pz81L17d3Xv3l1Op1NPPPGE3n33XY0aNapQgRsoCbiNBZQiy5Yt0wsvvKDIyEj169fPI/v08fFRjx499Pnnn+f7UxBXumJTWL1799ahQ4f0j3/8I8+6c+fO6ezZs675ChUq6NSpU4Xab7Vq1dS8eXN99NFHebbZsGGD1q5dqy5duriWnThxIs8+mjdvLkmu18qLUqun5V55ufycT5o0qcBtpkyZ4jY/efJkSXL1u2fPnvL19dW4cePy7NcYU+Ar7ZLyrPPx8VHTpk0lKc9r+EBJxpUdoIRatGiRduzYoUuXLuno0aNatmyZkpOTVbNmTf33v//16BfXvfzyy1q6dKliYmJcr1sfOXJEc+fO1erVqwt8NbmwHnroIc2ZM0ePPfaYli9frjvvvFM5OTnasWOH5syZoyVLlrgejm7ZsqW+/PJLTZgwQeHh4YqMjFSbNm0K3PeECRMUFxen5s2bq3///goPD1daWpree+89VatWTSNGjHC1HT9+vFatWqWuXbuqZs2aOnbsmN5++21Vr15dbdu2LXKtxXH8+HG9+OKLeZbnBth27drpb3/7my5evKhbbrlFS5cuVXp6eoH7S09P13333afOnTsrNTVVH330kf74xz+qWbNmkv53ZefFF1/UiBEjtHfvXvXo0UOVKlVSenq65s2bp0GDBumpp57Kd98DBw7UiRMn1KFDB1WvXl379u3T5MmT1bx5c0VFRRX7HAA3nBffBAOQj9xXvHMnPz8/ExYWZu655x7zxhtvmKysrDzbFPTqeUJCQr7H0GWvnhtjzL59+8zDDz9sqlSpYvz9/U3t2rVNQkKCyc7Odqvr8tfTc1//Xr58uWvZ5a+eG2PMhQsXzKuvvmoaN25s/P39TeXKlU3Lli3NuHHjTGZmpqvdjh07TLt27Uy5cuWMpEK9hr527VrTrVs3U7lyZVOmTBlzyy23mIEDB5qDBw+6tUtJSTH333+/CQ8PN35+fiY8PNw8+OCD5ocffihWrVc6x/mJiYlxG9tfTx07djTGGHPw4EHzwAMPmODgYBMUFGT+8Ic/mMOHD+cZs9wx//77783vf/97U6lSJVO5cmWTmJhozp07l+fY//73v03btm1NhQoVTIUKFUzDhg1NQkKC2blzp6vN5a+ef/rpp6ZTp06matWqxs/Pz9SoUcM8+uij5siRI4XuM1ASOIzxwDVqAACAEopndgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArMaXCup/vx1z+PBhVapU6YZ/PTwAACgeY4xOnz6t8PBw+fgUfP2GsCPp8OHDxf4hQgAA4F0HDhxQ9erVC1xP2JFUqVIlSf87WYGBgV6uBgAAFEZWVpYiIiJc/44XhLCj//8ry4GBgYQdAABKmas9gsIDygAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrlfF2AUBpVeu5hd4u4Tdr7ytdvV0CgFKEKzsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACs5tWwk5SUpNtuu02VKlVS1apV1aNHD+3cudOtzfnz55WQkKCbbrpJFStWVK9evXT06FG3Nvv371fXrl1Vvnx5Va1aVU8//bQuXbp0I7sCAABKKK+GnZUrVyohIUFr165VcnKyLl68qE6dOuns2bOuNk8++aQ+//xzzZ07VytXrtThw4fVs2dP1/qcnBx17dpVFy5c0Ndff60PPvhA06dP1+jRo73RJQAAUMI4jDHG20XkOn78uKpWraqVK1eqXbt2yszMVJUqVTRz5kz9/ve/lyTt2LFDUVFRSk1N1e23365FixapW7duOnz4sEJDQyVJU6dO1bPPPqvjx4/Lz8/vqsfNyspSUFCQMjMzFRgYeF37CHvUem6ht0v4zdr7SldvlwCgBCjsv98l6pmdzMxMSVJISIgkacOGDbp48aJiY2NdbRo2bKgaNWooNTVVkpSamqomTZq4go4kxcXFKSsrS9u3b8/3ONnZ2crKynKbAACAnUpM2HE6nRo2bJjuvPNO3XrrrZKkjIwM+fn5KTg42K1taGioMjIyXG1+HXRy1+euy09SUpKCgoJcU0REhId7AwAASooSE3YSEhK0bds2zZo167ofa8SIEcrMzHRNBw4cuO7HBAAA3lHG2wVIUmJiohYsWKBVq1apevXqruVhYWG6cOGCTp065XZ15+jRowoLC3O1+fbbb932l/u2Vm6by/n7+8vf39/DvQAAACWRV6/sGGOUmJioefPmadmyZYqMjHRb37JlS5UtW1YpKSmuZTt37tT+/fsVHR0tSYqOjtbWrVt17NgxV5vk5GQFBgaqUaNGN6YjAACgxPLqlZ2EhATNnDlTn332mSpVquR6xiYoKEjlypVTUFCQBgwYoOHDhyskJESBgYEaPHiwoqOjdfvtt0uSOnXqpEaNGumhhx7S3/72N2VkZGjkyJFKSEjg6g0AAPBu2HnnnXckSe3bt3dbPm3aNPXv31+SNHHiRPn4+KhXr17Kzs5WXFyc3n77bVdbX19fLViwQI8//riio6NVoUIFxcfHa/z48TeqGwAAoAQrUd+z4y18zw6Kg+/Z8R6+ZweAVEq/ZwcAAMDTCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACs5tWws2rVKnXv3l3h4eFyOByaP3++2/r+/fvL4XC4TZ07d3Zrc+LECfXr10+BgYEKDg7WgAEDdObMmRvYCwAAUJJ5NeycPXtWzZo105QpUwps07lzZx05csQ1ffLJJ27r+/Xrp+3btys5OVkLFizQqlWrNGjQoOtdOgAAKCXKePPgXbp0UZcuXa7Yxt/fX2FhYfmuS0tL0+LFi7Vu3Tq1atVKkjR58mTde++9eu211xQeHu7xmgEAQOlS4p/ZWbFihapWraoGDRro8ccf188//+xal5qaquDgYFfQkaTY2Fj5+Pjom2++KXCf2dnZysrKcpsAAICdSnTY6dy5sz788EOlpKTo1Vdf1cqVK9WlSxfl5ORIkjIyMlS1alW3bcqUKaOQkBBlZGQUuN+kpCQFBQW5poiIiOvaDwAA4D1evY11NX379nX9uUmTJmratKnq1KmjFStWqGPHjsXe74gRIzR8+HDXfFZWFoEHAABLlegrO5erXbu2br75Zu3evVuSFBYWpmPHjrm1uXTpkk6cOFHgcz7S/54DCgwMdJsAAICdSlXYOXjwoH7++WdVq1ZNkhQdHa1Tp05pw4YNrjbLli2T0+lUmzZtvFUmAAAoQbx6G+vMmTOuqzSSlJ6ers2bNyskJEQhISEaN26cevXqpbCwMO3Zs0fPPPOM6tatq7i4OElSVFSUOnfurEceeURTp07VxYsXlZiYqL59+/ImFgAAkOTlKzvr169XixYt1KJFC0nS8OHD1aJFC40ePVq+vr7asmWL7rvvPtWvX18DBgxQy5Yt9dVXX8nf39+1j48//lgNGzZUx44dde+996pt27Z67733vNUlAABQwnj1yk779u1ljClw/ZIlS666j5CQEM2cOdOTZQEAAIuUqmd2AAAAioqwAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVvPqr578FtZ5b6O0SfrP2vtLV2yUAAEoAruwAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwWrHCTu3atfXzzz/nWX7q1CnVrl37mosCAADwlGKFnb179yonJyfP8uzsbB06dOiaiwIAAPCUMkVp/N///tf15yVLligoKMg1n5OTo5SUFNWqVctjxQEAAFyrIoWdHj16SJIcDofi4+Pd1pUtW1a1atXS66+/7rHiAAAArlWRwo7T6ZQkRUZGat26dbr55puvS1EAAACeUqSwkys9Pd3TdQAAAFwXxQo7kpSSkqKUlBQdO3bMdcUn1/vvv3/NhQGAt9R6bqG3S/hN2vtKV2+XAEsVK+yMGzdO48ePV6tWrVStWjU5HA5P1wUAAOARxQo7U6dO1fTp0/XQQw95uh4AAACPKtb37Fy4cEF33HGHp2sBAADwuGKFnYEDB2rmzJmergUAAMDjinUb6/z583rvvff05ZdfqmnTpipbtqzb+gkTJnikOAAAgGtVrLCzZcsWNW/eXJK0bds2t3U8rAwAAEqSYoWd5cuXe7oOAACA66JYz+wAAACUFsW6snP33Xdf8XbVsmXLil0QAACAJxUr7OQ+r5Pr4sWL2rx5s7Zt25bnB0IBAAC8qVhhZ+LEifkuHzt2rM6cOXNNBQEAAHiSR5/Z+dOf/sTvYgEAgBLFo2EnNTVVAQEBntwlAADANSnWbayePXu6zRtjdOTIEa1fv16jRo3ySGEAAACeUKywExQU5Dbv4+OjBg0aaPz48erUqZNHCgMAAPCEYoWdadOmeboOAACA66JYYSfXhg0blJaWJklq3LixWrRo4ZGiAAAAPKVYYefYsWPq27evVqxYoeDgYEnSqVOndPfdd2vWrFmqUqWKJ2sEAAAotmK9jTV48GCdPn1a27dv14kTJ3TixAlt27ZNWVlZGjJkiKdrBAAAKLZiXdlZvHixvvzyS0VFRbmWNWrUSFOmTOEBZQAAUKIU68qO0+lU2bJl8ywvW7asnE7nNRcFAADgKcUKOx06dNDQoUN1+PBh17JDhw7pySefVMeOHT1WHAAAwLUqVth56623lJWVpVq1aqlOnTqqU6eOIiMjlZWVpcmTJ3u6RgAAgGIr1jM7ERER2rhxo7788kvt2LFDkhQVFaXY2FiPFgcAAHCtinRlZ9myZWrUqJGysrLkcDh0zz33aPDgwRo8eLBuu+02NW7cWF999dX1qhUAAKDIihR2Jk2apEceeUSBgYF51gUFBenRRx/VhAkTPFYcAADAtSpS2Pnuu+/UuXPnAtd36tRJGzZsuOaiAAAAPKVIYefo0aP5vnKeq0yZMjp+/Pg1FwUAAOApRQo7t9xyi7Zt21bg+i1btqhatWqF3t+qVavUvXt3hYeHy+FwaP78+W7rjTEaPXq0qlWrpnLlyik2Nla7du1ya3PixAn169dPgYGBCg4O1oABA3TmzJmidAsAAFisSGHn3nvv1ahRo3T+/Pk8686dO6cxY8aoW7duhd7f2bNn1axZM02ZMiXf9X/729/05ptvaurUqfrmm29UoUIFxcXFuR2/X79+2r59u5KTk7VgwQKtWrVKgwYNKkq3AACAxYr06vnIkSP1n//8R/Xr11diYqIaNGggSdqxY4emTJminJwc/fWvfy30/rp06aIuXbrku84Yo0mTJmnkyJG6//77JUkffvihQkNDNX/+fPXt21dpaWlavHix1q1bp1atWkmSJk+erHvvvVevvfaawsPDi9I9AABgoSJd2QkNDdXXX3+tW2+9VSNGjNADDzygBx54QM8//7xuvfVWrV69WqGhoR4pLD09XRkZGW7f3RMUFKQ2bdooNTVVkpSamqrg4GBX0JGk2NhY+fj46JtvvvFIHQAAoHQr8pcK1qxZU1988YVOnjyp3bt3yxijevXqqXLlyh4tLCMjQ5LyhKfQ0FDXuoyMDFWtWtVtfZkyZRQSEuJqk5/s7GxlZ2e75rOysjxVNgAAKGGK9Q3KklS5cmXddtttnqzlhklKStK4ceO8XQYAALgBivXbWDdCWFiYpP+97v5rR48eda0LCwvTsWPH3NZfunRJJ06ccLXJz4gRI5SZmemaDhw44OHqAQBASVFiw05kZKTCwsKUkpLiWpaVlaVvvvlG0dHRkqTo6GidOnXK7YsMly1bJqfTqTZt2hS4b39/fwUGBrpNAADATsW+jeUJZ86c0e7du13z6enp2rx5s0JCQlSjRg0NGzZML774ourVq6fIyEiNGjVK4eHh6tGjh6T//fho586d9cgjj2jq1Km6ePGiEhMT1bdvX97EAgAAkrwcdtavX6+7777bNT98+HBJUnx8vKZPn65nnnlGZ8+e1aBBg3Tq1Cm1bdtWixcvVkBAgGubjz/+WImJierYsaN8fHzUq1cvvfnmmze8LwAAoGTyathp3769jDEFrnc4HBo/frzGjx9fYJuQkBDNnDnzepQHAAAsUGKf2QEAAPAEwg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVSnTYGTt2rBwOh9vUsGFD1/rz588rISFBN910kypWrKhevXrp6NGjXqwYAACUNCU67EhS48aNdeTIEde0evVq17onn3xSn3/+uebOnauVK1fq8OHD6tmzpxerBQAAJU0ZbxdwNWXKlFFYWFie5ZmZmfrXv/6lmTNnqkOHDpKkadOmKSoqSmvXrtXtt99+o0sFAAAlUIm/srNr1y6Fh4erdu3a6tevn/bv3y9J2rBhgy5evKjY2FhX24YNG6pGjRpKTU294j6zs7OVlZXlNgEAADuV6LDTpk0bTZ8+XYsXL9Y777yj9PR03XXXXTp9+rQyMjLk5+en4OBgt21CQ0OVkZFxxf0mJSUpKCjINUVERFzHXgAAAG8q0bexunTp4vpz06ZN1aZNG9WsWVNz5sxRuXLlir3fESNGaPjw4a75rKwsAg8AAJYq0Vd2LhccHKz69etr9+7dCgsL04ULF3Tq1Cm3NkePHs33GZ9f8/f3V2BgoNsEAADsVKrCzpkzZ7Rnzx5Vq1ZNLVu2VNmyZZWSkuJav3PnTu3fv1/R0dFerBIAAJQkJfo21lNPPaXu3burZs2aOnz4sMaMGSNfX189+OCDCgoK0oABAzR8+HCFhIQoMDBQgwcPVnR0NG9iAQAAlxIddg4ePKgHH3xQP//8s6pUqaK2bdtq7dq1qlKliiRp4sSJ8vHxUa9evZSdna24uDi9/fbbXq4aAACUJCU67MyaNeuK6wMCAjRlyhRNmTLlBlUEAABKm1L1zA4AAEBREXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNXKeLsAAABuhFrPLfR2Cb9Ze1/p6tXjc2UHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAAAKsRdgAAgNUIOwAAwGqEHQAAYDXCDgAAsBphBwAAWI2wAwAArEbYAQAAViPsAAAAqxF2AACA1Qg7AADAaoQdAABgNcIOAACwGmEHAABYjbADAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFazJuxMmTJFtWrVUkBAgNq0aaNvv/3W2yUBAIASwIqwM3v2bA0fPlxjxozRxo0b1axZM8XFxenYsWPeLg0AAHiZFWFnwoQJeuSRR/TnP/9ZjRo10tSpU1W+fHm9//773i4NAAB4WakPOxcuXNCGDRsUGxvrWubj46PY2FilpqZ6sTIAAFASlPF2Adfqp59+Uk5OjkJDQ92Wh4aGaseOHfluk52drezsbNd8ZmamJCkrK8vj9Tmzf/H4PlE412M8f42x9R7G1k6Mq72u19jm7tcYc8V2pT7sFEdSUpLGjRuXZ3lERIQXqsH1EjTJ2xXgemFs7cS42ut6j+3p06cVFBRU4PpSH3Zuvvlm+fr66ujRo27Ljx49qrCwsHy3GTFihIYPH+6adzqdOnHihG666SY5HI4Cj5WVlaWIiAgdOHBAgYGBnulACfZb6i99tddvqb/01V6/pf4Wpa/GGJ0+fVrh4eFXbFfqw46fn59atmyplJQU9ejRQ9L/wktKSooSExPz3cbf31/+/v5uy4KDgwt9zMDAQOv/sv3ab6m/9NVev6X+0ld7/Zb6W9i+XumKTq5SH3Ykafjw4YqPj1erVq3UunVrTZo0SWfPntWf//xnb5cGAAC8zIqw06dPHx0/flyjR49WRkaGmjdvrsWLF+d5aBkAAPz2WBF2JCkxMbHA21ae4u/vrzFjxuS5BWar31J/6au9fkv9pa/2+i3193r01WGu9r4WAABAKVbqv1QQAADgSgg7AADAaoQdAABgNcIOAACwGmHnMlOmTFGtWrUUEBCgNm3a6Ntvvy2w7fTp0+VwONymgICAG1ht8a1atUrdu3dXeHi4HA6H5s+ff9VtVqxYod/97nfy9/dX3bp1NX369Otep6cUtb8rVqzIM7YOh0MZGRk3puBiSkpK0m233aZKlSqpatWq6tGjh3bu3HnV7ebOnauGDRsqICBATZo00RdffHEDqr12xelvaf3cvvPOO2ratKnri9aio6O1aNGiK25TWse1qH0trWOan1deeUUOh0PDhg27YrvSOraXK0x/PTG+hJ1fmT17toYPH64xY8Zo48aNatasmeLi4nTs2LECtwkMDNSRI0dc0759+25gxcV39uxZNWvWTFOmTClU+/T0dHXt2lV33323Nm/erGHDhmngwIFasmTJda7UM4ra31w7d+50G9+qVatepwo9Y+XKlUpISNDatWuVnJysixcvqlOnTjp79myB23z99dd68MEHNWDAAG3atEk9evRQjx49tG3bthtYefEUp79S6fzcVq9eXa+88oo2bNig9evXq0OHDrr//vu1ffv2fNuX5nEtal+l0jmml1u3bp3effddNW3a9IrtSvPY/lph+yt5YHwNXFq3bm0SEhJc8zk5OSY8PNwkJSXl237atGkmKCjoBlV3/Ugy8+bNu2KbZ555xjRu3NhtWZ8+fUxcXNx1rOz6KEx/ly9fbiSZkydP3pCarpdjx44ZSWblypUFtundu7fp2rWr27I2bdqYRx999HqX53GF6a8tn1tjjKlcubL55z//me86m8bVmCv31YYxPX36tKlXr55JTk42MTExZujQoQW2tWFsi9JfT4wvV3b+z4ULF7RhwwbFxsa6lvn4+Cg2NlapqakFbnfmzBnVrFlTERERV/2fR2mWmprqdm4kKS4u7ornxgbNmzdXtWrVdM8992jNmjXeLqfIMjMzJUkhISEFtrFpbAvTX6n0f25zcnI0a9YsnT17VtHR0fm2sWVcC9NXqfSPaUJCgrp27ZpnzPJjw9gWpb/StY8vYef//PTTT8rJycnzExOhoaEFPqfRoEEDvf/++/rss8/00Ucfyel06o477tDBgwdvRMk3VEZGRr7nJisrS+fOnfNSVddPtWrVNHXqVP373//Wv//9b0VERKh9+/bauHGjt0srNKfTqWHDhunOO+/UrbfeWmC7gsa2pD+fdLnC9rc0f263bt2qihUryt/fX4899pjmzZunRo0a5du2tI9rUfpamsdUkmbNmqWNGzcqKSmpUO1L+9gWtb+eGF9rfi7CG6Kjo93+p3HHHXcoKipK7777rl544QUvVoZr1aBBAzVo0MA1f8cdd2jPnj2aOHGiZsyY4cXKCi8hIUHbtm3T6tWrvV3KDVHY/pbmz22DBg20efNmZWZm6tNPP1V8fLxWrlxZYAgozYrS19I8pgcOHNDQoUOVnJxcah+qLori9NcT40vY+T8333yzfH19dfToUbflR48eVVhYWKH2UbZsWbVo0UK7d+++HiV6VVhYWL7nJjAwUOXKlfNSVTdW69atS01wSExM1IIFC7Rq1SpVr179im0LGtvC/r0vCYrS38uVps+tn5+f6tatK0lq2bKl1q1bpzfeeEPvvvtunralfVyL0tfLlaYx3bBhg44dO6bf/e53rmU5OTlatWqV3nrrLWVnZ8vX19dtm9I8tsXp7+WKM77cxvo/fn5+atmypVJSUlzLnE6nUlJSrnif+NdycnK0detWVatW7XqV6TXR0dFu50aSkpOTC31ubLB58+YSP7bGGCUmJmrevHlatmyZIiMjr7pNaR7b4vT3cqX5c+t0OpWdnZ3vutI8rvm5Ul8vV5rGtGPHjtq6das2b97smlq1aqV+/fpp8+bN+f7DX5rHtjj9vVyxxveaHm+2zKxZs4y/v7+ZPn26+f77782gQYNMcHCwycjIMMYY89BDD5nnnnvO1X7cuHFmyZIlZs+ePWbDhg2mb9++JiAgwGzfvt1bXSi006dPm02bNplNmzYZSWbChAlm06ZNZt++fcYYY5577jnz0EMPudr/+OOPpnz58ubpp582aWlpZsqUKcbX19csXrzYW10okqL2d+LEiWb+/Plm165dZuvWrWbo0KHGx8fHfPnll97qQqE8/vjjJigoyKxYscIcOXLENf3yyy+uNpf/PV6zZo0pU6aMee2110xaWpoZM2aMKVu2rNm6das3ulAkxelvaf3cPvfcc2blypUmPT3dbNmyxTz33HPG4XCYpUuXGmPsGtei9rW0jmlBLn87yaaxzc/V+uuJ8SXsXGby5MmmRo0axs/Pz7Ru3dqsXbvWtS4mJsbEx8e75ocNG+ZqGxoaau69916zceNGL1RddLmvVl8+5fYvPj7exMTE5NmmefPmxs/Pz9SuXdtMmzbthtddXEXt76uvvmrq1KljAgICTEhIiGnfvr1ZtmyZd4ovgvz6KMltrC7/e2yMMXPmzDH169c3fn5+pnHjxmbhwoU3tvBiKk5/S+vn9i9/+YupWbOm8fPzM1WqVDEdO3Z0/eNvjF3jWtS+ltYxLcjl//jbNLb5uVp/PTG+DmOMKfx1IAAAgNKFZ3YAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKxG2AEAAFYj7AAWq1WrliZNmuTtMrxq7Nixat68ubfLcPPee+8pIiJCPj4++Y7P3r175XA4tHnz5kLvs3///urRo8c11bVixQo5HA6dOnXqmvYDlDSEHaCEad++vYYNG5Zn+fTp0xUcHFykfa1bt06DBg3yTGHwiKysLCUmJurZZ5/VoUOHGB/gBuBXzwGLValSxdslWOvixYsqW7Zskbfbv3+/Ll68qK5du5aKH6oEbMCVHaCUyr1t8dprr6latWq66aablJCQoIsXL7raXH4ba9euXWrXrp0CAgLUqFEjJScny+FwaP78+ZLyv42xefNmORwO7d2717Vs9erVuuuuu1SuXDlFRERoyJAhOnv2bIG15t5KmjFjhmrVqqWgoCD17dtXp0+fLrBWSWrevLnGjh3rmnc4HHr33XfVrVs3lS9fXlFRUUpNTdXu3bvVvn17VahQQXfccYf27NmTp4Z3331XERERKl++vHr37q3MzEy39f/85z8VFRWlgIAANWzYUG+//bZrXe5tpdmzZysmJkYBAQH6+OOP8+3r/v37df/996tixYoKDAxU7969dfToUUn/uzrXpEkTSVLt2rXznNeC5OTkaMCAAYqMjFS5cuXUoEEDvfHGG/m2HTdunKpUqaLAwEA99thjunDhgmud0+lUUlKSaz/NmjXTp59+WuBx9+3bp+7du6ty5cqqUKGCGjdurC+++OKq9QIlDWEHKMWWL1+uPXv2aPny5frggw80ffp0TZ8+Pd+2TqdTPXv2lJ+fn7755htNnTpVzz77bJGPuWfPHnXu3Fm9evXSli1bNHv2bK1evVqJiYlX3W7+/PlasGCBFixYoJUrV+qVV14p8vFfeOEFPfzww9q8ebMaNmyoP/7xj3r00Uc1YsQIrV+/XsaYPLXs3r1bc+bM0eeff67Fixdr06ZNeuKJJ1zrP/74Y40ePVovvfSS0tLS9PLLL2vUqFH64IMP3Pbz3HPPaejQoUpLS1NcXFye2pxOp+6//36dOHFCK1euVHJysn788Uf16dNHktSnTx99+eWXkqRvv/1WR44cUURExFX77HQ6Vb16dc2dO1fff/+9Ro8ereeff15z5sxxa5eSkqK0tDStWLFCn3zyif7zn/9o3LhxrvVJSUn68MMPNXXqVG3fvl1PPvmk/vSnP2nlypX5HjchIUHZ2dlatWqVtm7dqldffVUVK1a8ar1AieOJXywF4DmX/wJwrmnTppmgoCDXfHx8vKlZs6a5dOmSa9kf/vAH06dPH9d8zZo1zcSJE40xxixZssSUKVPGHDp0yLV+0aJFRpKZN2+eMeb//zr8yZMnXW02bdpkJJn09HRjjDEDBgwwgwYNcqvtq6++Mj4+PubcuXP59mnMmDGmfPnyJisry7Xs6aefNm3atMm31lzNmjUzY8aMcc1LMiNHjnTNp6amGknmX//6l2vZJ598YgICAtyO7evraw4ePOjWbx8fH3PkyBFjjDF16tQxM2fOdDv2Cy+8YKKjo40xxqSnpxtJZtKkSfn2L9fSpUuNr6+v2b9/v2vZ9u3bjSTz7bffGmPyns/85B5v06ZNBbZJSEgwvXr1cs3Hx8ebkJAQc/bsWdeyd955x1SsWNHk5OSY8+fPm/Lly5uvv/7abT8DBgwwDz74oDEm7/g3adLEjB079op9BkoDntkBSrHGjRvL19fXNV+tWjVt3bo137ZpaWmKiIhQeHi4a1l0dHSRj/ndd99py5YtbrdxjDFyOp1KT09XVFRUvtvVqlVLlSpVcqv12LFjRT5+06ZNXX8ODQ2VJNetodxl58+fV1ZWlgIDAyVJNWrU0C233OJqEx0dLafTqZ07d6pSpUras2ePBgwYoEceecTV5tKlSwoKCnI7dqtWra5YW+45/vXVmkaNGik4OFhpaWm67bbbitzfXFOmTNH777+v/fv369y5c7pw4UKet8yaNWum8uXLu/XzzJkzOnDggM6cOaNffvlF99xzj9s2Fy5cUIsWLfI95pAhQ/T4449r6dKlio2NVa9evdzOP1BaEHaAEiYwMDDP8ySSdOrUqTz/+F7+gKzD4ZDT6Sz2sX18/ndn2xjjWvbrZ4Ak6cyZM3r00Uc1ZMiQPNvXqFGjwH1frVYfHx+34+Z37Mv343A4ClxW2PNw5swZSdI//vEPtWnTxm3dr4OkJFWoUKFQ+/S0WbNm6amnntLrr7+u6OhoVapUSX//+9/1zTffFHofuf1cuHChW/CTJH9//3y3GThwoOLi4rRw4UItXbpUSUlJev311zV48ODidwbwAsIOUMI0aNBAS5cuzbN848aNql+/frH3GxUVpQMHDujIkSOut4DWrl3r1ib37a0jR46ocuXKkpTnu15+97vf6fvvv1fdunWLXUt+qlSpoiNHjrjms7KylJ6e7pF979+/X4cPH3Zd1Vq7dq18fHzUoEEDhYaGKjw8XD/++KP69et3TcfJPccHDhxwXd35/vvvderUKTVq1KjY+12zZo3uuOMOt+eM8nsI+7vvvtO5c+dUrlw5Sf/rZ8WKFRUREaGQkBD5+/tr//79iomJKfSxIyIi9Nhjj+mxxx7TiBEj9I9//IOwg1KHsAOUMI8//rjeeustDRkyRAMHDpS/v78WLlyoTz75RJ9//nmx9xsbG6v69esrPj5ef//735WVlaW//vWvbm3q1q2riIgIjR07Vi+99JJ++OEHvf76625tnn32Wd1+++1KTEzUwIEDVaFCBX3//fdKTk7WW2+9Vez6OnTooOnTp6t79+4KDg7W6NGj81xZKa6AgADFx8frtddeU1ZWloYMGaLevXsrLCxM0v/eYBoyZIiCgoLUuXNnZWdna/369Tp58qSGDx9e6OPExsaqSZMm6tevnyZNmqRLly7piSeeUExMzFVvgV1JvXr19OGHH2rJkiWKjIzUjBkztG7dOkVGRrq1u3DhggYMGKCRI0dq7969GjNmjBITE+Xj46NKlSrpqaee0pNPPimn06m2bdsqMzNTa9asUWBgoOLj4/Mcd9iwYerSpYvq16+vkydPavny5QXepgRKMt7GAkqY2rVra9WqVdqxY4diY2PVpk0bzZkzR3PnzlXnzp2LvV8fHx/NmzdP586dU+vWrTVw4EC99NJLbm3Kli2rTz75RDt27FDTpk316quv6sUXX3Rr07RpU61cuVI//PCD7rrrLrVo0UKjR492exaoOEaMGKGYmBh169ZNXbt2VY8ePVSnTp1r2meuunXrqmfPnrr33nvVqVMnNW3a1O3V8oEDB+qf//ynpk2bpiZNmigmJkbTp0/PEyauxuFw6LPPPlPlypXVrl07xcbGqnbt2po9e/Y11f/oo4+qZ8+e6tOnj9q0aaOff/7Z7SpPro4dO6pevXpq166d+vTpo/vuu8/t1f0XXnhBo0aNUlJSkqKiotS5c2ctXLiwwH7m5OQoISHB1bZ+/fpu5w0oLRzm8pvkAH5THA6H5s2bd80/NQAAJRVXdgAAgNUIOwAAwGo8oAz8xnEnG4DtuLIDAACsRtgBAABWI+wAAACrEXYAAIDVCDsAAMBqhB0AAGA1wg4AALAaYQcAAFiNsAMAAKz2/wCQD4j+hqs/hwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize the number of user unique labels with Dirichlet distribution\n",
    "n = 500\n",
    "unique_labels = [len(np.unique(next(federated_dataset)[0].raw_data[1])) for _ in range(n)]\n",
    "values, counts = np.unique(unique_labels, return_counts=True)\n",
    "plt.bar(values, counts)\n",
    "plt.title('Dirichlet User Labels')\n",
    "plt.xlabel('Unique number of labels')\n",
    "plt.ylabel('Count')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "822505be-d51f-4983-89c9-c0b664d85f91",
   "metadata": {},
   "source": [
    "#### 4.2 Partitioning the dataset into fixed artificial users with `FederatedDataset`\n",
    "An alternative to the above is to partition the original dataset into a fixed set of artificial users where each user's label distribution follows a Dirichlet distribution. The difference of this option is that the user partitions are constructed once using all data and remains the same throughout the training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8b78c088-6b89-4527-a7db-64fb761b8a58",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FederatedDataset with Dirichlet label distribution: \n",
      "\tArtificial user 0 has size of 20 with unique labels=[0]\n",
      "\tArtificial user 1 has size of 20 with unique labels=[3 4]\n",
      "\tArtificial user 2 has size of 20 with unique labels=[0 4]\n",
      "\tArtificial user 3 has size of 20 with unique labels=[2]\n",
      "\tArtificial user 4 has size of 20 with unique labels=[0]\n"
     ]
    }
   ],
   "source": [
    "# Create a federated dataset with fixed user partition\n",
    "# alpha=0.01 is a extreme case where each user likely has only data from 1 class\n",
    "federated_dataset = FederatedDataset.from_slices_with_dirichlet_class_distribution(\n",
    "    data=[features, labels],\n",
    "    labels=labels,\n",
    "    alpha=0.01,\n",
    "    user_dataset_len_sampler=sample_dataset_len)\n",
    "\n",
    "# Iterate federated dataset to get artificial user dataset\n",
    "print(\"FederatedDataset with Dirichlet label distribution: \")\n",
    "for i in range(5):\n",
    "    user_dataset, seed = next(federated_dataset)\n",
    "    print(f\"\\tArtificial user {i} has size of {len(user_dataset)} with unique labels={np.unique(user_dataset.raw_data[1])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6137e7f3-9907-468d-8425-1e691834780d",
   "metadata": {},
   "source": [
    "### 5. Customizing Federated Dataset\n",
    "This section covers customizations to federated dataset for different needs, in case the above general usage does not satisify."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4ce63b0-5a36-46b7-bf95-d5e5c804ad16",
   "metadata": {},
   "source": [
    "#### 5.1 Creating memory-efficient HDF5 Federated Dataset\n",
    "Many ML datasets can get too large to fit into memory and HDF5 is a memory friendly format for processing Numpy arrays. HDF5 can be easily integrated with `FederatedDataset` for large scale PFL experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "648bd9dd-1570-44da-a4f9-9f464a713415",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Customized HDF5 FederatedDataset with real user partition: \n",
      "\tReal user with user_id=17 has size of 18.\n",
      "\tReal user with user_id=6 has size of 7.\n",
      "\tReal user with user_id=0 has size of 1.\n",
      "\tReal user with user_id=8 has size of 9.\n",
      "\tReal user with user_id=17 has size of 18.\n"
     ]
    }
   ],
   "source": [
    "import h5py\n",
    "import tempfile\n",
    "\n",
    "# Create a HDF5 file.\n",
    "temp = tempfile.NamedTemporaryFile()\n",
    "with h5py.File(temp.name, 'w') as h5:\n",
    "    for user_id in user_id_to_data:\n",
    "        features, labels = user_id_to_data[user_id]\n",
    "        h5[f'/{user_id}/features'] = features\n",
    "        h5[f'/{user_id}/labels'] = labels\n",
    "\n",
    "# Customized function for creating a user dataset given the user ID\n",
    "def make_dataset_fn(user_id):\n",
    "    with h5py.File(temp.name, 'r') as h5:\n",
    "        features = np.array(h5[f'/{user_id}/features'])\n",
    "        labels = np.array(h5[f'/{user_id}/labels'])\n",
    "        # Randomly shuffle the user dataset each time when loading\n",
    "        data_order = np.random.permutation(len(features))\n",
    "    return Dataset(raw_data=[features[data_order], labels[data_order]], user_id=user_id)\n",
    "\n",
    "# Create FederatedDataset with the customized make_dataset_fn\n",
    "federated_dataset = FederatedDataset(make_dataset_fn=make_dataset_fn, user_sampler=user_sampler)\n",
    "\n",
    "# Iterate federated dataset to get artificial user dataset\n",
    "print(\"Customized HDF5 FederatedDataset with real user partition: \")\n",
    "for i in range(5):\n",
    "    user_dataset, seed = next(federated_dataset)\n",
    "    print(f\"\\tReal user with user_id={user_dataset.user_id} has size of {len(user_dataset)}.\")\n",
    "\n",
    "# Delete the temp HDF5 file\n",
    "temp.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70397726-252b-4395-8f3b-113ca44d7d93",
   "metadata": {},
   "source": [
    "#### 5.2 Creating `ArtificialFederatedDataset` from HuggingFace datasets \n",
    "HuggingFace [datasets](https://huggingface.co/datasets) provides a collection of ML datasets and it can be integrated with `pfl` easily to create `ArtificialFederatedDataset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c95fa7e5-b1df-4f64-8fc6-079a0be8a353",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install the HuggingFace modules first\n",
    "!pip3 -q install datasets transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9173903f-f631-4e6b-97fb-a349a0b68a81",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/congzhengsong/Repos/nnlm-pfl-deps/miniconda3/envs/pfl-py310/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "2024-01-19 14:51:06.724183: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "You're using a BertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "HuggingFace User dataset size follows a Poisson distribution: \n",
      "\tArtificial user 0 has size of 14\n",
      "\tArtificial user 1 has size of 16\n",
      "\tArtificial user 2 has size of 22\n",
      "\tArtificial user 3 has size of 14\n",
      "\tArtificial user 4 has size of 15\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/congzhengsong/Repos/nnlm-pfl-deps/miniconda3/envs/pfl-py310/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2636: UserWarning: `max_length` is ignored when `padding`=`True` and there is no truncation strategy. To pad to max length, use `padding='max_length'`.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "from transformers import AutoTokenizer, DataCollatorWithPadding\n",
    "\n",
    "\n",
    "# Follow the tutorial from https://huggingface.co/docs/datasets/use_dataset\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n",
    "hf_dataset = load_dataset(\"rotten_tomatoes\", split=\"train\").map(lambda x: tokenizer(x['text']), batched=True)\n",
    "hf_dataset.set_format(type=\"torch\", columns=[\"input_ids\", \"token_type_ids\", \"attention_mask\", \"label\"])\n",
    "data_collator = DataCollatorWithPadding(tokenizer, max_length=64)\n",
    "\n",
    "def make_dataset_fn(indices):\n",
    "    data = data_collator([hf_dataset[int(i)] for i in indices])\n",
    "    return PyTorchTensorDataset(tensors=tuple(data.values()))\n",
    "\n",
    "data_sampler = get_data_sampler(sample_type=\"random\", max_bound=len(hf_dataset))\n",
    "sample_dataset_len = lambda: max(1, np.random.poisson(16))\n",
    "federated_dataset = ArtificialFederatedDataset(\n",
    "    make_dataset_fn=make_dataset_fn,\n",
    "    data_sampler=data_sampler,\n",
    "    sample_dataset_len=sample_dataset_len)\n",
    "\n",
    "# Iterate federated dataset to get artificial user dataset\n",
    "print(\"\\nHuggingFace User dataset size follows a Poisson distribution: \")\n",
    "for i in range(5):\n",
    "    user_dataset, seed = next(federated_dataset)\n",
    "    print(f\"\\tArtificial user {i} has size of {len(user_dataset)}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}