examples/complete-guide.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Complete guide"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This Notebook contains an overview of the basic functionality of the simulator. It introduces the simplest ways to get started with the simulator, and it dives into more advanced concepts that will allow you to get a sense of the flexibility of the system. At the end of this guide, you should be able to configure the pre-loaded simulations with custom parameters and metrics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Main components\n",
    "The following diagram provides a birds-eye view of the simulation dynamics. (Note that the diagram does not include content creators, which are optional.)\n",
    "<img src=\"figures/diagram.jpg\" width=500>\n",
    "\n",
    "A simulation needs the following components:\n",
    "\n",
    "- **Users**: agents who interact with each other and with items.\n",
    "- **Model**: agent that defines the behavior of the sociotechnical system. The model mediates the interactions among users and between users and the system.\n",
    "- **Items**: passive components that are served to the users by the model. (Items may come from a fixed catalog or may be dynamically generated by **creators**.)\n",
    "- **Measurements**: modules built into the models which automatically compute information about the system.\n",
    "\n",
    "## Dynamics\n",
    "The following steps are at the heart of the simulations:\n",
    "1. The **model** presents the **users** with some recommended **items**. The recommendations are generated in accordance with the specific recommender system algorithm the model is using (e.g., content filtering, popularity, etc.). The input to the algorithm is based on the model's _prediction_ or _knowledge_ of user preferences.\n",
    "2. The **users** view the items presented by the **model**, and interact with some **items** according to some _actual_ preferences.\n",
    "3. The **model** updates its system state (such as the prediction of user preferences) based on the interactions of **users** with **items**, and it takes some **measurements**.\n",
    "\n",
    "We will see that this framework is very flexible and it can be a generalization of many classic and new models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick start: instantiate a model and run\n",
    "The fastest way to get started is to choose a model, instantiate it with no parameters, and run it for some time steps. Here we run a simple [content filtering recommendation system](https://elucherini.github.io/t-recs/reference/models.html#module-models.content) (please refer to the [BaseRecommender documentation](https://elucherini.github.io/t-recs/reference/models.html#module-models.recommender) for a complete list of class attributes and methods shared by all models, as that information is currently incomplete in the docs of the other pre-loaded models).\n",
    "\n",
    "Content filters infer information about the _attributes_ of users based on their past interactions and recommend items with similar attributes to those of users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "\n",
    "import trecs\n",
    "from trecs.models import ContentFiltering\n",
    "from trecs.random import Generator\n",
    "from trecs.metrics import MSEMeasurement, InteractionSpread, RecSimilarity, AverageFeatureScoreRange, InteractionSimilarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5/5 [01:40<00:00, 20.06s/it]\n"
     ]
    }
   ],
   "source": [
    "# Create ContentFiltering instance without arguments\n",
    "default_filtering = ContentFiltering()\n",
    "# add an MSE measurement\n",
    "default_filtering.add_metrics(MSEMeasurement())\n",
    "# Run for 5 time steps\n",
    "default_filtering.run(timesteps=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results of the simulation:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mse</th>\n",
       "      <th>timesteps</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.508760</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9.018810</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8.338439</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8.260871</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8.260871</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>8.260871</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        mse  timesteps\n",
       "0  0.508760          0\n",
       "1  9.018810          1\n",
       "2  8.338439          2\n",
       "3  8.260871          3\n",
       "4  8.260871          4\n",
       "5  8.260871          5"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Collect measurements about the simulation\n",
    "results = default_filtering.get_measurements()\n",
    "\n",
    "print(\"Results of the simulation:\")\n",
    "pd.DataFrame(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In what follows, we expand on this minimal example to gain a deeper understanding of what happens under the hood."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models\n",
    "\n",
    "As in the ``Quick Start``, if you want to run a simulation, the smallest piece of information you need is the model you want to run. There are a number of pre-loaded models that work out of the box. We continue to use a generic content filtering recommendation system; please see the docs for a [list of already-implemented models](https://elucherini.github.io/t-recs/reference/models.html#).\n",
    "\n",
    "Recall that content filters infer information about the _attributes_ of users based on their past interactions and recommend items with similar attributes to those of users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Again, we instantiate the model with no arguments\n",
    "default_filtering = ContentFiltering()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell above, we instantiated a content filtering recommender system with default parameters. We print below the default number of users and items in the system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of users in system: 100\n",
      "Number of items in system: 1250\n"
     ]
    }
   ],
   "source": [
    "print(f\"Number of users in system: {default_filtering.num_users}\")\n",
    "print(f\"Number of items in system: {default_filtering.num_items}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model also created a representation for both users and items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In content filtering, the default model representation of users and items are given by:\n",
      "- An all-zeros matrix of users of dimension (100, 1000)\n",
      "- A randomly generated matrix of items of dimension (1000, 1250)\n"
     ]
    }
   ],
   "source": [
    "print(\"In content filtering, the default model representation of users and items are given by:\")\n",
    "print(f\"- An all-zeros matrix of users of dimension {default_filtering.predicted_user_profiles.shape}\")\n",
    "print(f\"- A randomly generated matrix of items of dimension {default_filtering.predicted_item_attributes.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Formally, content filtering supports user profiles of size `|num_users x num_attributes|` and item attributes of size `|num_attributes x num_items|`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set number of users or items\n",
    "We can customize the number of users and items in the system:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of items in the system is now 5000.\n",
      "The number of users in the system is now 500.\n"
     ]
    }
   ],
   "source": [
    "# instantiate with a different number of items and users\n",
    "number_of_items = 5000\n",
    "number_of_users = 500\n",
    "filtering = ContentFiltering(num_items = number_of_items, num_users=number_of_users)\n",
    "print(f\"The number of items in the system is now {filtering.num_items}.\")\n",
    "print(f\"The number of users in the system is now {filtering.num_users}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the representations of items and users are set accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The size of item_attributes is (1000, 5000).\n",
      "The size of user_profiles is (500, 1000).\n"
     ]
    }
   ],
   "source": [
    "print(f\"The size of item_attributes is {filtering.predicted_item_attributes.shape}.\")\n",
    "print(f\"The size of user_profiles is {filtering.predicted_user_profiles.shape}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## User predictions and Items\n",
    "We might also want to define our own representation of users and items. We can do so by defining matrices that satisfy the constraints of the model. The constraints for ContentFiltering (some of which have been mentioned above) are:\n",
    "\n",
    "- User profiles must be of size `|num_users x num_attributes|`.\n",
    "- User profiles are matrices of integers representing the number of interactions of each user with items that have a given attributes. `user_profiles[i, j]` represents the number of interactions user `i` had with items with attribute `j`.\n",
    "- Item attributes must be of size `|num_attributes x num_items|`.\n",
    "- The model doesn't define any constraint on item attributes. If `item_attributes` is binary, then its `[i, j]`th element is 1 if item `j` is attributed attribute `i`; otherwise, it's 0. Item attributes can also be real-valued, representing the probability that each attribute has to describe items.\n",
    "\n",
    "If you're already familiar with Numpy: the model is compatible with `ndarray` and _array_like_ data structures. \n",
    "\n",
    "If you're not familiar with Numpy: the framework provides a random number generator that lets you draw from several distributions (which, in practice, is a thin wrapper around `numpy.random.Generator`). Please refer to the Numpy documentation for a [list of distributions](https://numpy.org/doc/stable/reference/random/generator.html?highlight=generator#distributions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "User representation (num_users x num_attributes):\n",
      "[[1 1 3 0 0 0 1 3 2 3]\n",
      " [2 1 2 0 3 1 3 1 3 0]\n",
      " [2 0 0 2 0 0 3 2 0 1]\n",
      " [3 2 3 1 1 2 2 1 1 3]\n",
      " [1 0 2 1 2 1 3 3 2 2]]\n",
      "\n",
      "Item representation (num_attributes x num_items):\n",
      "[[1 0 0 1 0 0 1 0 0 0 0 0 0 0 0]\n",
      " [0 0 0 1 0 0 1 0 1 1 1 0 0 1 0]\n",
      " [1 0 0 0 0 0 0 0 0 0 1 0 1 0 1]\n",
      " [1 0 0 0 0 0 0 0 0 0 0 0 1 0 0]\n",
      " [1 1 1 0 0 1 0 0 0 0 1 0 1 1 1]\n",
      " [1 0 0 0 0 0 0 0 1 0 0 1 0 0 0]\n",
      " [1 0 0 0 0 0 0 0 0 0 0 0 0 0 1]\n",
      " [0 1 0 0 1 1 1 0 0 1 1 1 0 0 0]\n",
      " [0 1 0 1 0 0 0 0 0 0 0 0 0 0 0]\n",
      " [0 0 0 0 1 1 0 0 0 0 0 0 1 1 1]]\n"
     ]
    }
   ],
   "source": [
    "# Keep the dimensions small for easy visualization\n",
    "number_of_users = 5\n",
    "number_of_attributes = 10\n",
    "number_of_items = 15\n",
    "# We define user_representation using the standard integer generator in Numpy.\n",
    "# We assume a number of interactions with each attribute in the interval [0,4).\n",
    "user_representation = np.random.randint(4, size=(number_of_users, number_of_attributes))\n",
    "\n",
    "# We define item_representation using the Generator that comes with the framework\n",
    "# We assume a binary matrix with a binomial distribution\n",
    "\n",
    "item_representation = Generator().binomial(\n",
    "    n=1, p=.3, size=(number_of_attributes, number_of_items)\n",
    ")\n",
    "# Note that this is equivalent to:\n",
    "# item_representation = np.random.Generator(np.random.MT19937()).binomial(n=1, p=.5, size=(...))\n",
    "\n",
    "print(f\"User representation (num_users x num_attributes):\\n{str(user_representation)}\\n\")\n",
    "print(f\"Item representation (num_attributes x num_items):\\n{str(item_representation)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is user_profiles equivalent to user_representation? yes.\n",
      "Is item_attributes equivalent to item_representation? yes.\n"
     ]
    }
   ],
   "source": [
    "# Initialize with custom representations\n",
    "filtering = ContentFiltering(user_representation=user_representation,\n",
    "                             item_representation=item_representation)\n",
    "\n",
    "# Check if they're equivalent\n",
    "is_user_equivalent = \"yes\" if np.array_equal(user_representation, filtering.predicted_user_profiles) else \"no\"\n",
    "is_item_equivalent = \"yes\" if np.array_equal(item_representation, filtering.predicted_item_attributes) else \"no\"\n",
    "print(\"Is user_profiles equivalent to user_representation? %s.\" % is_user_equivalent)\n",
    "print(\"Is item_attributes equivalent to item_representation? %s.\" % is_item_equivalent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also initialize models with `user_representation` and `item_representation` individually. In this case, the representation that has not been initialized will adapt to the size defined by the user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "After initializing user_profiles, the size of item_attributes (and so the number of attributes in the system) adapts automatically to it.\n",
      "Size of user_profiles, as defined above: (5, 10).\n",
      "Size of item_attributes: (10, 1250).\n",
      "\n",
      "After initializing item_attributes, the size of user_profiles (and so the number of attributes in the system) adapts automatically to it.\n",
      "Size of item_attributes, as defined above: (10, 15).\n",
      "Size of user_profiles: (100, 10).\n"
     ]
    }
   ],
   "source": [
    "# Let's only initialize user_profiles\n",
    "filtering = ContentFiltering(user_representation=user_representation)\n",
    "print(\"After initializing user_profiles, the size of item_attributes (and so the number of attributes in the system) adapts automatically to it.\")\n",
    "print(f\"Size of user_profiles, as defined above: {str(filtering.predicted_user_profiles.shape)}.\")\n",
    "print(f\"Size of item_attributes: {str(filtering.predicted_item_attributes.shape)}.\\n\")\n",
    "\n",
    "# The same happens by only initializing item_attributes\n",
    "filtering = ContentFiltering(item_representation=item_representation)\n",
    "print(\"After initializing item_attributes, the size of user_profiles (and so the number of attributes in the system) adapts automatically to it.\")\n",
    "print(f\"Size of item_attributes, as defined above: {str(filtering.predicted_item_attributes.shape)}.\")\n",
    "print(f\"Size of user_profiles: {str(filtering.predicted_user_profiles.shape)}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run a simulation\n",
    "We can run a simulation for the predefined number of time steps (50), or define our own duration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 50/50 [00:00<00:00, 887.55it/s]\n"
     ]
    }
   ],
   "source": [
    "# let's initialize a model with both user_representation and item_representation defined above\n",
    "filtering = ContentFiltering(user_representation=user_representation,\n",
    "                            item_representation=item_representation)\n",
    "filtering.add_metrics(MSEMeasurement()) # add MSE Measurement\n",
    "# Run the model for the predefined number of timesteps:\n",
    "filtering.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of the simulation, we can examine the results of the measurements. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mse</th>\n",
       "      <th>timesteps</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.327133</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.271610</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.271610</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.258398</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.258398</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        mse  timesteps\n",
       "0  1.327133          0\n",
       "1  0.271610          1\n",
       "2  0.271610          2\n",
       "3  0.258398          3\n",
       "4  0.258398          4"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To get the measurements of all timesteps<=50\n",
    "measurements = filtering.get_measurements()\n",
    "\n",
    "# Measurements can be easily converted to pandas DataFrame objects\n",
    "\n",
    "pd.DataFrame(measurements).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Measurements\n",
    "At each time step of the simulation, measurement modules calculate a quantity based on the system state. An example of such quantity is the mean squared error between the predicted user profiles and the actual user profiles -- that is, how close is the model to predicting the real preferences of the system?\n",
    "\n",
    "It's easy to define new metrics, but in this guide we will use some of the pre-loaded metrics to get a better sense of how they work. For a list of pre-loaded metrics and their descriptions, see the [docs](https://elucherini.github.io/t-recs/reference/metrics.html).\n",
    "\n",
    "### View metrics\n",
    "\n",
    "The metrics monitored are stored in the the model's `metrics` attribute, which contains a list of objects of type [Measurement](https://elucherini.github.io/t-recs/reference/metrics.html#measurement-base-class)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The system is currently monitoring these metrics:\n",
      "[<trecs.metrics.measurement.MSEMeasurement object at 0x7fc8cefc7e50>]\n"
     ]
    }
   ],
   "source": [
    "# The metrics tracked by each model can be examined by printing the `metrics` list.\n",
    "print(\"The system is currently monitoring these metrics:\")\n",
    "print(filtering.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add metrics\n",
    "To **maintain compatibility with pandas**, we suggest to **only add metrics to instances of models that have not been run yet**. This is to avoid having measurements that start at different time steps, resulting in arrays of different length. Feel free to disregard this advice if pandas compatibility is not important to your application.\n",
    "\n",
    "We can instantiate a model, add a new metric, and then run the model.\n",
    "\n",
    "We will add InteractionSpread, which provides a measure of the homogeneity of user interactions in the system as a whole.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "These are the current metrics:\n",
      "[<trecs.metrics.measurement.MSEMeasurement object at 0x7fc8cf023890>, <trecs.metrics.measurement.InteractionSpread object at 0x7fc8cf028350>]\n"
     ]
    }
   ],
   "source": [
    "#change the number of items and users to make metric values more reasonable and dimensions distinguishable\n",
    "number_of_items=100\n",
    "number_of_users=50\n",
    "number_of_attributes=20\n",
    "\n",
    "#change the distribution of item attributes so average feature score range can be added later\n",
    "item_representation = Generator().normal(size=(number_of_attributes, number_of_items))\n",
    "filtering = ContentFiltering(num_users=number_of_users, num_items=number_of_items, \n",
    "                             num_attributes=number_of_attributes,\n",
    "                             item_representation=item_representation,\n",
    "                             record_base_state=True)\n",
    "\n",
    "# This method accepts a variable number of metrics\n",
    "filtering.add_metrics(MSEMeasurement(), InteractionSpread())\n",
    "\n",
    "print(\"These are the current metrics:\")\n",
    "print(filtering.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also add [RecSimilarity](https://elucherini.github.io/t-recs/reference/metrics.html#recsimilarity), which measures the similarity between interaction patterns for pairs of users. In order to measure Jaccard similarity, we must specify which pairs of users we want to compare."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "These are the current metrics:\n",
      "[<trecs.metrics.measurement.MSEMeasurement object at 0x7fc8cf023890>, <trecs.metrics.measurement.InteractionSpread object at 0x7fc8cf028350>, <trecs.metrics.measurement.RecSimilarity object at 0x7fc8cf023d50>]\n"
     ]
    }
   ],
   "source": [
    "js_pairs = [(u1_idx, u2_idx) for u1_idx in range(filtering.num_users) for u2_idx in range(filtering.num_users) if u1_idx != u2_idx] \n",
    "filtering.add_metrics(RecSimilarity(pairs=js_pairs))\n",
    "\n",
    "print(\"These are the current metrics:\")\n",
    "print(filtering.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will add average feature score range, a metric for evaluating within list diversity based on the range of item attribute values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtering.add_metrics(AverageFeatureScoreRange())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5/5 [00:00<00:00, 42.64it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mse</th>\n",
       "      <th>interaction_spread</th>\n",
       "      <th>rec_similarity</th>\n",
       "      <th>afsr</th>\n",
       "      <th>timesteps</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.031274</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.481288</td>\n",
       "      <td>-48.0</td>\n",
       "      <td>0.053801</td>\n",
       "      <td>5.502235</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.363990</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.141115</td>\n",
       "      <td>5.876673</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.331318</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.141019</td>\n",
       "      <td>5.909132</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.329232</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.156319</td>\n",
       "      <td>5.846574</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.330844</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.154776</td>\n",
       "      <td>5.865965</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        mse  interaction_spread  rec_similarity      afsr  timesteps\n",
       "0  1.031274                 NaN             NaN       NaN          0\n",
       "1  1.481288               -48.0        0.053801  5.502235          1\n",
       "2  1.363990                 0.0        0.141115  5.876673          2\n",
       "3  1.331318                 0.5        0.141019  5.909132          3\n",
       "4  1.329232                 0.0        0.156319  5.846574          4\n",
       "5  1.330844                 0.5        0.154776  5.865965          5"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now we run the model\n",
    "filtering.run(timesteps=5)\n",
    "measurements = filtering.get_measurements()\n",
    "pd.DataFrame(measurements)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Measurements at time step 0 can be undefined (`None`, `NaN`, etc.) because it denotes the measurements before the start of the simulation. MSE is defined at the beginning because the system is initialized with random predictions of the user profiles; in contrast, homogeneity is meaningless before the simulation begins because there are no user interactions to consider."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add diagnostics\n",
    "You can also add diagnostics on the data you are using to calculate your metrics. Diagnostic statistics can be calculated for mean squared error across users. The statistics that are calculated include the mean, standard deviation, median, mean, max, skew, kurtosis, Shapiro-Wilk statistic (for testing distribution normality) and its associated p-value for sample sizes less than 5,000. You can also generate either histograms of the metric distribution, Q-Q plots, or both. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/10 [00:00<?, ?it/s]/opt/anaconda3/envs/t_recs/lib/python3.7/site-packages/scipy/stats/morestats.py:1681: UserWarning: p-value may not be accurate for N > 5000.\n",
      "  warnings.warn(\"p-value may not be accurate for N > 5000.\")\n",
      "100%|██████████| 10/10 [03:33<00:00, 21.36s/it]\n"
     ]
    }
   ],
   "source": [
    "default_filtering = ContentFiltering()\n",
    "\n",
    "js_pairs = [(u1_idx, u2_idx) for u1_idx in range(default_filtering.num_users) for u2_idx in range(default_filtering.num_users) if u1_idx != u2_idx] \n",
    "\n",
    "# add an MSE measurement\n",
    "mse, interaction_sim = MSEMeasurement(diagnostics=True), InteractionSimilarity(pairs=js_pairs, diagnostics=True)\n",
    "default_filtering.add_metrics(mse)\n",
    "default_filtering.add_metrics(interaction_sim)\n",
    "\n",
    "default_filtering.run(timesteps=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>median</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>skew</th>\n",
       "      <th>kurtosis</th>\n",
       "      <th>sw_stat</th>\n",
       "      <th>sw_p</th>\n",
       "      <th>n</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.227209</td>\n",
       "      <td>0.305898</td>\n",
       "      <td>0.118263</td>\n",
       "      <td>1.344350e-07</td>\n",
       "      <td>1.307586</td>\n",
       "      <td>1.912966</td>\n",
       "      <td>2.900781</td>\n",
       "      <td>0.720257</td>\n",
       "      <td>1.642023e-12</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8.673687</td>\n",
       "      <td>2.821907</td>\n",
       "      <td>8.227697</td>\n",
       "      <td>3.451776e+00</td>\n",
       "      <td>17.195548</td>\n",
       "      <td>0.632118</td>\n",
       "      <td>0.215346</td>\n",
       "      <td>0.970534</td>\n",
       "      <td>2.429121e-02</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8.034257</td>\n",
       "      <td>2.748252</td>\n",
       "      <td>7.569420</td>\n",
       "      <td>2.669041e+00</td>\n",
       "      <td>17.195548</td>\n",
       "      <td>0.650441</td>\n",
       "      <td>0.359038</td>\n",
       "      <td>0.971244</td>\n",
       "      <td>2.754703e-02</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7.920781</td>\n",
       "      <td>2.824664</td>\n",
       "      <td>7.403255</td>\n",
       "      <td>2.669041e+00</td>\n",
       "      <td>17.195548</td>\n",
       "      <td>0.629440</td>\n",
       "      <td>0.244043</td>\n",
       "      <td>0.971460</td>\n",
       "      <td>2.862137e-02</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7.910181</td>\n",
       "      <td>2.844066</td>\n",
       "      <td>7.403255</td>\n",
       "      <td>2.197389e+00</td>\n",
       "      <td>17.195548</td>\n",
       "      <td>0.590415</td>\n",
       "      <td>0.256858</td>\n",
       "      <td>0.975272</td>\n",
       "      <td>5.666774e-02</td>\n",
       "      <td>100.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       mean       std    median           min        max      skew  kurtosis  \\\n",
       "0  0.227209  0.305898  0.118263  1.344350e-07   1.307586  1.912966  2.900781   \n",
       "1  8.673687  2.821907  8.227697  3.451776e+00  17.195548  0.632118  0.215346   \n",
       "2  8.034257  2.748252  7.569420  2.669041e+00  17.195548  0.650441  0.359038   \n",
       "3  7.920781  2.824664  7.403255  2.669041e+00  17.195548  0.629440  0.244043   \n",
       "4  7.910181  2.844066  7.403255  2.197389e+00  17.195548  0.590415  0.256858   \n",
       "\n",
       "    sw_stat          sw_p      n  \n",
       "0  0.720257  1.642023e-12  100.0  \n",
       "1  0.970534  2.429121e-02  100.0  \n",
       "2  0.971244  2.754703e-02  100.0  \n",
       "3  0.971460  2.862137e-02  100.0  \n",
       "4  0.975272  5.666774e-02  100.0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#This is the weird part. Would be better to get this to function the same as \"get_measurements\"\n",
    "mse_diagnostics = mse.get_diagnostics()\n",
    "isim_diagnostics = interaction_sim.get_diagnostics()\n",
    "\n",
    "mse_diagnostics.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>median</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>skew</th>\n",
       "      <th>kurtosis</th>\n",
       "      <th>sw_stat</th>\n",
       "      <th>sw_p</th>\n",
       "      <th>n</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000808</td>\n",
       "      <td>0.028415</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>35.135464</td>\n",
       "      <td>1232.500809</td>\n",
       "      <td>0.009535</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9900.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.001684</td>\n",
       "      <td>0.026957</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19.960582</td>\n",
       "      <td>512.112593</td>\n",
       "      <td>0.035550</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9900.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.001933</td>\n",
       "      <td>0.027495</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>18.651706</td>\n",
       "      <td>465.444308</td>\n",
       "      <td>0.043203</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9900.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.001886</td>\n",
       "      <td>0.027120</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19.103944</td>\n",
       "      <td>488.358624</td>\n",
       "      <td>0.042546</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9900.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.001886</td>\n",
       "      <td>0.027120</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>19.103944</td>\n",
       "      <td>488.358624</td>\n",
       "      <td>0.042546</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9900.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       mean       std  median  min  max       skew     kurtosis   sw_stat  \\\n",
       "0  0.000808  0.028415     0.0  0.0  1.0  35.135464  1232.500809  0.009535   \n",
       "1  0.001684  0.026957     0.0  0.0  1.0  19.960582   512.112593  0.035550   \n",
       "2  0.001933  0.027495     0.0  0.0  1.0  18.651706   465.444308  0.043203   \n",
       "3  0.001886  0.027120     0.0  0.0  1.0  19.103944   488.358624  0.042546   \n",
       "4  0.001886  0.027120     0.0  0.0  1.0  19.103944   488.358624  0.042546   \n",
       "\n",
       "   sw_p       n  \n",
       "0   NaN  9900.0  \n",
       "1   NaN  9900.0  \n",
       "2   NaN  9900.0  \n",
       "3   NaN  9900.0  \n",
       "4   NaN  9900.0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "isim_diagnostics.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Diagnostics can be particularly helpful if you impose or expect differences differences between subpopulations of users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_of_attributes = 10\n",
    "number_of_maj_users = 150\n",
    "number_of_min_users = 50\n",
    "\n",
    "maj_user_representation = np.random.normal(1, 2, size=(number_of_maj_users, number_of_attributes))\n",
    "min_user_representation = np.random.normal(0.5, 1.25, size=(number_of_min_users, number_of_attributes))\n",
    "actual_user_representation = np.vstack((maj_user_representation, min_user_representation))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([12., 21., 29., 34., 38., 30., 16., 14.,  4.,  2.]),\n",
       " array([-0.33071133, -0.04065503,  0.24940128,  0.53945758,  0.82951388,\n",
       "         1.11957019,  1.40962649,  1.69968279,  1.9897391 ,  2.2797954 ,\n",
       "         2.56985171]),\n",
       " <BarContainer object of 10 artists>)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOGklEQVR4nO3df4xlZ13H8feHsvyINLa417ppuwwCkVQjWxzXEoypRUylCS2xMe0ftRjM4g8iJMS48oeA0aQk0iYqgSy2shoEmrbYSovalCYNiS5O67ZsuyAFl9hm6Q5U+iOami1f/5izMJnO3Xtm7q99pu9XcjPnPufceb5Pzu4nZ849zzmpKiRJ7XnBvAuQJG2OAS5JjTLAJalRBrgkNcoAl6RGvXCWnW3fvr0WFhZm2aUkNe/ee+/9dlUN1rbPNMAXFhZYWlqaZZeS1Lwk31yv3VMoktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUqJnOxJRGWdh7+1z6PXLNJXPpVxqHR+CS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJatTIAE/ykiRfSnJ/kgeTfLBr/0SS/0xysHvtmnq1kqTv63M3wmeAi6rq6STbgC8m+Xy37ver6qbplSdJGmZkgFdVAU93b7d1r5pmUZKk0XqdA09yWpKDwDHgzqo60K360yQPJLkuyYuHfHZPkqUkS8vLy5OpWpLUL8Cr6tmq2gWcA+xO8lPAHwKvBX4WeDnwB0M+u6+qFqtqcTAYTKZqSdLGrkKpqu8CdwMXV9XRWvEM8NfA7inUJ0kaos9VKIMkZ3TLLwXeDHwlyY6uLcBlwKHplSlJWqvPVSg7gP1JTmMl8G+sqs8l+UKSARDgIPBb0ytTkrRWn6tQHgDOX6f9oqlUJEnqxafS6znm9WR4SRvjVHpJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqVJ+HGr8kyZeS3J/kwSQf7NpfmeRAkoeTfCbJi6ZfriTphD5H4M8AF1XV64BdwMVJLgA+BFxXVa8G/ht4x9SqlCQ9x8gArxVPd2+3da8CLgJu6tr3A5dNo0BJ0vp6PdQ4yWnAvcCrgY8AXwe+W1XHu00eAc4e8tk9wB6AnTt3jluvNBXzfJDzkWsumVvfaluvLzGr6tmq2gWcA+wGXtu3g6raV1WLVbU4GAw2V6Uk6Tk2dBVKVX0XuBt4A3BGkhNH8OcAj062NEnSyfS5CmWQ5Ixu+aXAm4HDrAT55d1mVwO3TqlGSdI6+pwD3wHs786DvwC4sao+l+Qh4NNJ/gT4d+D6KdYpSVpjZIBX1QPA+eu0f4OV8+GSpDnodRWK5mOeV0ZIOvU5lV6SGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIa1eehxucmuTvJQ0keTPLurv0DSR5NcrB7vWX65UqSTujzSLXjwHur6r4kpwP3JrmzW3ddVf3Z9MqTJA3T56HGR4Gj3fJTSQ4DZ0+7MEnSyW3oHHiSBVaeUH+ga3pXkgeS3JDkzEkXJ0karneAJ3kZcDPwnqp6Evgo8CpgFytH6B8e8rk9SZaSLC0vL49fsSQJ6BngSbaxEt6frKpbAKrqsap6tqq+B3wc2L3eZ6tqX1UtVtXiYDCYVN2S9LzX5yqUANcDh6vq2lXtO1Zt9jbg0OTLkyQN0+cqlDcCVwFfTnKwa3sfcGWSXUABR4B3TqE+SdIQfa5C+SKQdVbdMflyJEl9ORNTkhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktSoPk/ked5b2Hv7vEuQpOfwCFySGmWAS1Kj+jyV/twkdyd5KMmDSd7dtb88yZ1Jvtb9PHP65UqSTuhzBH4ceG9VnQdcAPxukvOAvcBdVfUa4K7uvSRpRkYGeFUdrar7uuWngMPA2cClwP5us/3AZVOqUZK0jg2dA0+yAJwPHADOqqqj3apvAWcN+cyeJEtJlpaXl8epVZK0Su8AT/Iy4GbgPVX15Op1VVVArfe5qtpXVYtVtTgYDMYqVpL0A70CPMk2VsL7k1V1S9f8WJId3fodwLHplChJWk+fq1ACXA8crqprV626Dbi6W74auHXy5UmShukzE/ONwFXAl5Mc7NreB1wD3JjkHcA3gV+bSoWSpHWNDPCq+iKQIavfNNlyJEl9ORNTkhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktSoPk/kkTRFC3tvn0u/R665ZC79anI8ApekRvV5qPENSY4lObSq7QNJHk1ysHu9ZbplSpLW6nME/gng4nXar6uqXd3rjsmWJUkaZWSAV9U9wOMzqEWStAHjnAN/V5IHulMsZw7bKMmeJEtJlpaXl8foTpK02mYD/KPAq4BdwFHgw8M2rKp9VbVYVYuDwWCT3UmS1tpUgFfVY1X1bFV9D/g4sHuyZUmSRtlUgCfZsert24BDw7aVJE3HyIk8ST4FXAhsT/II8H7gwiS7gAKOAO+cXomSpPWMDPCqunKd5uunUIskaQOcSi89T81rCj84jX9SnEovSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNamYm5jxnjUnSqcgjcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGjUywJPckORYkkOr2l6e5M4kX+t+njndMiVJa/U5Av8EcPGatr3AXVX1GuCu7r0kaYZGBnhV3QM8vqb5UmB/t7wfuGyyZUmSRtnsOfCzqupot/wt4KxhGybZk2QpydLy8vImu5MkrTX2l5hVVUCdZP2+qlqsqsXBYDBud5KkzmYD/LEkOwC6n8cmV5IkqY/NBvhtwNXd8tXArZMpR5LUV5/LCD8F/AvwE0keSfIO4BrgzUm+BvxS916SNEMj7wdeVVcOWfWmCdciSdoAZ2JKUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWrUyEeqnUySI8BTwLPA8apanERRkqTRxgrwzi9W1bcn8HskSRvgKRRJatS4AV7APye5N8meSRQkSepn3FMoP19Vjyb5UeDOJF+pqntWb9AF+x6AnTt3jtmdJOmEsY7Aq+rR7ucx4LPA7nW22VdVi1W1OBgMxulOkrTKpgM8yQ8lOf3EMvDLwKFJFSZJOrlxTqGcBXw2yYnf83dV9Y8TqUqSNNKmA7yqvgG8boK1SJI2YBLXgUvShizsvX0u/R655pK59DstXgcuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVFOpZf0vDGvKfwwnWn8HoFLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktSosQI8ycVJvprk4SR7J1WUJGm0TQd4ktOAjwC/ApwHXJnkvEkVJkk6uXGOwHcDD1fVN6rq/4BPA5dOpixJ0ijjTKU/G/ivVe8fAX5u7UZJ9gB7urdPJ/nqGH2eKrYD3553ERO01cYDW29MW208sPXGdNLx5ENj/e5XrNc49XuhVNU+YN+0+5mlJEtVtTjvOiZlq40Htt6Yttp4YOuNaR7jGecUyqPAuaven9O1SZJmYJwA/zfgNUlemeRFwBXAbZMpS5I0yqZPoVTV8STvAv4JOA24oaoenFhlp7YtdUqIrTce2Hpj2mrjga03ppmPJ1U16z4lSRPgTExJapQBLkmNMsBPYtStApK8OMlnuvUHkizMoczeeozn7UmWkxzsXr85jzr7SnJDkmNJDg1ZnyR/3o33gSSvn3WNG9FjPBcmeWLV/vmjWde4UUnOTXJ3koeSPJjk3ets08x+6jme2e2nqvK1zouVL2a/Dvw48CLgfuC8Ndv8DvCxbvkK4DPzrnvM8bwd+Mt517qBMf0C8Hrg0JD1bwE+DwS4ADgw75rHHM+FwOfmXecGx7QDeH23fDrwH+v8u2tmP/Ucz8z2k0fgw/W5VcClwP5u+SbgTUkywxo3Ysvd+qCq7gEeP8kmlwJ/Uyv+FTgjyY7ZVLdxPcbTnKo6WlX3dctPAYdZmcW9WjP7qed4ZsYAH269WwWs3VHf36aqjgNPAD8yk+o2rs94AH61+zP2piTnrrO+JX3H3JI3JLk/yeeT/OS8i9mI7hTj+cCBNaua3E8nGQ/MaD8Z4FrtH4CFqvpp4E5+8NeFTg33Aa+oqtcBfwH8/XzL6S/Jy4CbgfdU1ZPzrmdcI8Yzs/1kgA/X51YB398myQuBHwa+M5PqNm7keKrqO1X1TPf2r4CfmVFt07KlbvdQVU9W1dPd8h3AtiTb51zWSEm2sRJ2n6yqW9bZpKn9NGo8s9xPBvhwfW4VcBtwdbd8OfCF6r7FOAWNHM+a845vZeX8XstuA369u8rhAuCJqjo676I2K8mPnfiOJcluVv7/nqoHDMDKFSbA9cDhqrp2yGbN7Kc+45nlfpr63QhbVUNuFZDkj4GlqrqNlR35t0keZuXLpyvmV/HJ9RzP7yV5K3CclfG8fW4F95DkU6x84789ySPA+4FtAFX1MeAOVq5weBj4H+A35lNpPz3Gcznw20mOA/8LXHEKHzCc8EbgKuDLSQ52be8DdkKT+6nPeGa2n5xKL0mN8hSKJDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmN+n/gvo4z313wWwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#If plotted without respect to the subgroups, preference means look more or less normally distributed\n",
    "plt.hist(actual_user_representation.mean(axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 2.,  3.,  3.,  8.,  6.,  5., 11.,  5.,  5.,  2.]),\n",
       " array([-0.33071133, -0.17548818, -0.02026503,  0.13495812,  0.29018127,\n",
       "         0.44540442,  0.60062757,  0.75585071,  0.91107386,  1.06629701,\n",
       "         1.22152016]),\n",
       " <BarContainer object of 10 artists>)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOQ0lEQVR4nO3de4xm9V3H8fdH2F4ijRR3ghtgulWJDRpZcFxpMA2CNbTJFhobA3+0oJip2kZIGiOSaFurSU1sSbykdSvY1dQWAtSyhKqEkjRNdHHABRbWWlpphGzZ7Y1LNJilX/+Ys3QyfWafM/Pc9jd9v5Inc87vnGfO95ez88nZ37mlqpAktecHZl2AJGljDHBJapQBLkmNMsAlqVEGuCQ16uRpbmzr1q21ffv2aW5Skpr3wAMPfL2q5la3TzXAt2/fztLS0jQ3KUnNS/LVQe0OoUhSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqOmeiemtF67dk1/m3v3Tn+b0kZ4BC5JjRoa4ElekeT+JA8leTTJ+7v2jyf5ryT7u8+OiVcrSXpJnyGUF4CLq+r5JFuALyT5bLfsd6rqtsmVJ0lay9AAr+W3Hj/fzW7pPr4JWZJmrNcYeJKTkuwHDgP3VNW+btEfJ3k4yY1JXr7GdxeTLCVZOnLkyHiqliT1C/CqerGqdgBnAjuT/BTwe8DrgJ8FTgN+d43v7q6qhapamJv7nueRS5I2aF1XoVTVt4H7gEur6lAtewH4G2DnBOqTJK2hz1Uoc0lO7aZfCbwR+I8k27q2AJcDByZXpiRptT5XoWwD9iQ5ieXAv7Wq7kryuSRzQID9wG9MrkxJ0mp9rkJ5GDhvQPvFE6lIktSLd2JKUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRQwM8ySuS3J/koSSPJnl/1/7aJPuSPJ7kliQvm3y5kqRj+hyBvwBcXFXnAjuAS5NcAPwJcGNV/TjwLeCaiVUpSfoeQwO8lj3fzW7pPgVcDNzWte8BLp9EgZKkwXqNgSc5Kcl+4DBwD/Bl4NtVdbRb5UngjDW+u5hkKcnSkSNHxlCyJAl6BnhVvVhVO4AzgZ3A6/puoKp2V9VCVS3Mzc1trEpJ0vdY11UoVfVt4D7g9cCpSU7uFp0JPDXe0iRJx9PnKpS5JKd2068E3ggcZDnI39atdhXwmQnVKEka4OThq7AN2JPkJJYD/9aquivJY8CnkvwR8O/ATROsU5K0ytAAr6qHgfMGtH+F5fFwSdIMeCemJDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaNTTAk5yV5L4kjyV5NMm1Xfv7kjyVZH/3efPky5UkHTP0rfTAUeA9VfVgklcBDyS5p1t2Y1X96eTKkyStZWiAV9Uh4FA3/VySg8AZky5MknR8fY7AX5JkO3AesA+4EHh3kncASywfpX9rwHcWgUWA+fn5UeuVJm7Xrtlsd+/e2WxX7ep9EjPJKcDtwHVV9SzwEeDHgB0sH6F/aND3qmp3VS1U1cLc3NzoFUuSgJ4BnmQLy+H9iaq6A6Cqnq6qF6vqO8DHgJ2TK1OStFqfq1AC3AQcrKoPr2jftmK1twIHxl+eJGktfcbALwTeDjySZH/XdgNwZZIdQAFPAO+cQH2SpDX0uQrlC0AGLLp7/OVIkvryTkxJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDVqaIAnOSvJfUkeS/Jokmu79tOS3JPkS93PV0++XEnSMX2OwI8C76mqc4ALgHclOQe4Hri3qs4G7u3mJUlTMjTAq+pQVT3YTT8HHATOAC4D9nSr7QEun1CNkqQB1jUGnmQ7cB6wDzi9qg51i74GnL7GdxaTLCVZOnLkyCi1SpJW6B3gSU4Bbgeuq6pnVy6rqgJq0PeqandVLVTVwtzc3EjFSpK+q1eAJ9nCcnh/oqru6JqfTrKtW74NODyZEiVJg/S5CiXATcDBqvrwikV3Ald101cBnxl/eZKktZzcY50LgbcDjyTZ37XdAHwQuDXJNcBXgV+ZSIWSpIGGBnhVfQHIGosvGW85kqS+vBNTkhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqVJ+HWekEsmvXrCuQdKLwCFySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEb1eSv9zUkOJzmwou19SZ5Ksr/7vHmyZUqSVutzBP5x4NIB7TdW1Y7uc/d4y5IkDTM0wKvq88A3p1CLJGkdRhkDf3eSh7shllePrSJJUi8bfRbKR4APANX9/BDwa4NWTLIILALMz89vcHPS5jer59zs3Tub7Wp0GzoCr6qnq+rFqvoO8DFg53HW3V1VC1W1MDc3t9E6JUmrbCjAk2xbMftW4MBa60qSJmPoEEqSTwIXAVuTPAm8F7goyQ6Wh1CeAN45uRIlSYMMDfCqunJA800TqEWStA6+0EH6PufJ03Z5K70kNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1yjfybNCs3mIya79//2gd/8BOX8MijYtH4JLUqKEBnuTmJIeTHFjRdlqSe5J8qfv56smWKUlarc8R+MeBS1e1XQ/cW1VnA/d285KkKRoa4FX1eeCbq5ovA/Z003uAy8dbliRpmI2exDy9qg51018DTl9rxSSLwCLA/Pz8Bje3tu/Xk4mSNPJJzKoqoI6zfHdVLVTVwtzc3KibkyR1NhrgTyfZBtD9PDy+kiRJfWw0wO8EruqmrwI+M55yJEl99bmM8JPAvwA/keTJJNcAHwTemORLwC9285KkKRp6ErOqrlxj0SVjrkWStA7eiSlJjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIa5QsdpsiXIUgaJ4/AJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKW+kbMuqt+ODt+NJm4hG4JDXKAJekRo00hJLkCeA54EXgaFUtjKMoSdJw4xgD/4Wq+voYfo8kaR0cQpGkRo16BF7APycp4K+qavfqFZIsAosA8/PzI25OoxrHlSwtbx+8Ekebx6hH4D9fVecDbwLeleQNq1eoqt1VtVBVC3NzcyNuTpJ0zEgBXlVPdT8PA58Gdo6jKEnScBsO8CQ/mORVx6aBXwIOjKswSdLxjTIGfjrw6STHfs/fV9U/jqUqSdJQGw7wqvoKcO4Ya5EkrYOXEUpSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGfd+8kedEeAaHpO/aNaM/yb2b6FE4HoFLUqMMcElqlAEuSY0ywCWpUamqqW1sYWGhlpaWNvblNc543H//CAVJG+ALIbQRo5w8TfLAoHcOewQuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJatRIAZ7k0iRfTPJ4kuvHVZQkabgNB3iSk4C/BN4EnANcmeSccRUmSTq+UY7AdwKPV9VXqur/gE8Bl42nLEnSMKO80OEM4L9XzD8J/NzqlZIsAovd7PNJvjjCNk8UW4Gvz7qIMdps/YFJ9umuTOTXDuE+OvEdtz8Z7Z/NawY1TvyNPFW1G9g96e1MU5KlQQ+WadVm6w9svj5ttv7A5uvTLPozyhDKU8BZK+bP7NokSVMwSoD/G3B2ktcmeRlwBXDneMqSJA2z4SGUqjqa5N3APwEnATdX1aNjq+zEtqmGhNh8/YHN16fN1h/YfH2aen+m+kIHSdL4eCemJDXKAJekRhngxzHsUQFJXp7klm75viTbZ1Bmbz36c3WSI0n2d59fn0WdfSW5OcnhJAfWWJ4kf9b19+Ek50+7xvXo0Z+LkjyzYv/8wbRrXK8kZyW5L8ljSR5Ncu2AdZrZTz37M739VFV+BnxYPjH7ZeBHgZcBDwHnrFrnt4CPdtNXALfMuu4R+3M18BezrnUdfXoDcD5wYI3lbwY+CwS4ANg365pH7M9FwF2zrnOdfdoGnN9Nvwr4zwH/7prZTz37M7X95BH42vo8KuAyYE83fRtwSTLi/VaTs+kefVBVnwe+eZxVLgP+tpb9K3Bqkm3TqW79evSnOVV1qKoe7KafAw6yfBf3Ss3sp579mRoDfG2DHhWweke9tE5VHQWeAX54KtWtX5/+APxy99/Y25KcNWB5S/r2uSWvT/JQks8m+clZF7Me3RDjecC+VYua3E/H6Q9MaT8Z4FppL7C9qn4auIfv/u9CJ4YHgddU1bnAnwP/MNty+ktyCnA7cF1VPTvrekY1pD9T208G+Nr6PCrgpXWSnAz8EPCNqVS3fkP7U1XfqKoXutm/Bn5mSrVNyqZ63ENVPVtVz3fTdwNbkmydcVlDJdnCcth9oqruGLBKU/tpWH+muZ8M8LX1eVTAncBV3fTbgM9VdxbjBDS0P6vGHd/C8vhey+4E3tFd5XAB8ExVHZp1URuV5EeOnWNJspPlv98T9YABWL7CBLgJOFhVH15jtWb2U5/+THM/TfxphK2qNR4VkOQPgaWqupPlHfl3SR5n+eTTFbOr+Ph69ue3k7wFOMpyf66eWcE9JPkky2f8tyZ5EngvsAWgqj4K3M3yFQ6PA/8D/OpsKu2nR3/eBvxmkqPA/wJXnMAHDMdcCLwdeCTJ/q7tBmAemtxPffoztf3krfSS1CiHUCSpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJatT/A+VxoAnhrxgeAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#when plotting out mean preferences when accounting for group, we can see a clear distinction in preference\n",
    "plt.hist(maj_user_representation.mean(axis=1), alpha=.7, color='b')\n",
    "plt.hist(min_user_representation.mean(axis=1), alpha=0.7, color='r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that we have imposed differences in the average attribute value (and its spread) between minority and majority subpopulations, we can examine how those differences evolve over repeated interactions and retraining"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print majority / minority outcome stats\n",
    "def majority_minority_outcomes(metric, split_index):\n",
    "    split_indices = [split_index]\n",
    "\n",
    "        \n",
    "    maj_mean = metric.last_observation[:split_index].mean()\n",
    "    maj_std = metric.last_observation[:split_index].std()\n",
    "\n",
    "    min_mean = metric.last_observation[split_index:].mean()\n",
    "    min_std = metric.last_observation[split_index:].std()\n",
    "\n",
    "    print(\"Majority group statistics: \", maj_mean, \"(mean), \", maj_std, \"(std)\")\n",
    "    print(\"Minority group statistics: \", min_mean, \"(mean), \", min_std, \"(std)\")\n",
    "    print()\n",
    "    \n",
    "    metric.hist(split_indices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 50/50 [00:00<00:00, 167.34it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Majority group statistics:  0.6436401794021207 (mean),  0.49159291822118334 (std)\n",
      "Minority group statistics:  0.5322918043835236 (mean),  0.46196533931121864 (std)\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD4CAYAAADrRI2NAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWOUlEQVR4nO3dfbQlVXnn8e8P0oiOIGJfsAXaNmqSIY6AXhEVjcBSkfBijDoy0YAhq6MZRbNcCZiZCMZxCWTEIJlF0hleFRVEkJdgRoagxNG007TNaxNFkQhppEVeRRibeeaPUxfutPfeU/elzunb5/tZq9at2lV16rnF4bm7d+3aO1WFJGl0bDPsACRJg2Xil6QRY+KXpBFj4pekEWPil6QR80vDDqCNpUuX1ooVK4YdhiQtKtddd92Pq2ps8/JFkfhXrFjBmjVrhh2GJC0qSe6YqtymHkkaMSZ+SRoxJn5JGjEmfkkaMSZ+SRoxJn5JGjEmfkkaMSZ+SRoxJn5JGjGL4s3d+TjssOFd+/LLh3dtSZqONX5JGjEmfkkaMSZ+SRoxJn5JGjEmfkkaMSZ+SRoxJn5JGjEmfkkaMSZ+SRoxJn5JGjEmfkkaMSZ+SRoxnSX+JNsn+VaS65PcnOQjTfk5SW5Psq5Z9u4qBknSL+pydM7HgAOr6uEkS4CvJ/lys++Pq+qiDq8tSZpGZ4m/qgp4uNlc0izV1fUkSe102safZNsk64B7gKuqanWz62NJbkjyySRPmebclUnWJFmzcePGLsOUpJHSaeKvqseram9gd2DfJC8CPgT8GvAyYGfguGnOXVVV41U1PjY21mWYkjRSBtKrp6ruB64BDq6qDdXzGHA2sO8gYpAk9XTZq2csyU7N+lOB1wG3JlnWlAV4E3BTVzFIkn5R34e7SbYHDgVeDTwH+Bm9ZP13VXXzDKcuA85Nsi29PzAXVtUVSf4hyRgQYB3w7vn9CpKk2Zgx8Td97w8FvgqspveQdnvgV4CTmj8KH6yqGzY/tynbZ4ryA+cftiRprvrV+L9VVSdMs+/UJLsAyxc4JklSh2ZM/FX1d33230PvXwGSpEVixoe7SZ6R5KQktyb5SZJ7k6xvynYaUIySpAXUr1fPhcB9wGuraueqehZwQFN2YdfBSZIWXr/Ev6KqTq6quycKquruqjoZeG63oUmSutAv8d+R5E+S7DpRkGTXJMcBP+w2NElSF/ol/n8PPAv4WpL7ktxHr2vnzsDbOo5NktSBfr167qM3ls6U4+lIkhafNm/uvoHe0Aq7NUV3AZdW1d93GJckqSP93tz9S3pv6Z4H3NkU7w4cm+SNVfX+bsOTJC20fjX+Q6rqVzYvTHIB8B3AxC9Ji0y/h7uPJnnZFOUvAx7tIB5JUsf61fiPBs5IsgNPNvXsATzQ7JMkLTL9evWsBV6e5NlMerg7+YUuSdLi0nay9Xs3T/ZJllbVjzuISZLUoX6DtB2Q5E5gQ5KvJFkxafdXOo1MktSJfg93TwHeUFVLgVXAVUn2a/al08gkSZ3o19Sz3cT0ilV1UZL1wMXNWD3VeXSSpAXXr8b/8+bBLgDNH4GDgBOBF850YpLtk3wryfVJbm6mcSTJ85KsTnJbkguSbDfP30GSNAv9Ev/xwK6TC6rqTuA3gJP6nPsYcGBV7QXsDRzcNBOdDHyyql5Ab1z/Y+YQtyRpjmZM/FX1P6vq+inKH6iqj/U5t6rq4WZzSbMUcCBwUVN+Lr1xgCRJA9Kvxv+EZtyeJ362PGfbJOvozct7FfA94P6q2tQccidPvh+w+bkrk6xJsmbjxo1tLylJ6qN14gde0/z8jbYnVNXjVbU3vYHd9gV+bRbnrqqq8aoaHxsbm0WYkqSZzCbxz1lV3Q9cA7wC2CnJRG+i3ekN8yxJGpDOEn+SsSQ7NetPBV4HrKf3B+AtzWFHAZd2FYMk6Re1HbJhLpYB5ybZlt4fmAur6ooktwCfT/JfgG8DZ3YYgyRpM50l/qq6AdhnivLv02vvlyQNwWyaej7b/Dy/i0AkSYPROvFX1X+d/FOStDi1aupJ8hTgt4EVk8+pqj/vJixJUlfatvFfSm/WrevoDcUgSVqk2ib+3avq4E4jkSQNRNs2/m8k+XedRiJJGoi2Nf79gaOT3E6vqSf0xmF7cWeRSZI60Tbxv7HTKCRJA9Mq8VfVHV0HIkkajIEM0iZJ2nKY+CVpxJj4JWnEzNjGn+QhetMl/sIuer16duwkKklSZ2ZM/FW1w6ACkSQNxqyGZU6yC7D9xHZV/cuCRyRJ6lSrNv4khyf5LnA78DXgB8CXO4xLktSRtg93PwrsB3ynqp4HHAT8U2dRSZI60zbx/7yq7gW2SbJNVV0DjHcYlySpI20T//1Jng5cC5yf5DTgpzOdkGSPJNckuSXJzUne35SfmOSuJOua5ZD5/QqSpNlo+3D3COBnwB8BvwM8A/hIn3M2AR+sqrVJdgCuS3JVs++TzuQlScPRtsb/4ar6v1W1qarOrapPAcfNdEJVbaiqtc36Q8B6YLf5hStJmq+2if91U5S1HrEzyQpgH2B1U/TeJDckOSvJM9t+jiRp/mZM/Enek+RG4FebRD2x3A7c0OYCzbOBLwIfqKoHgTOA5wN7AxuAT0xz3soka5Ks2bhxY/vfSJI0o35t/J+l11//48Dxk8ofqqqf9PvwJEvoJf3zq+pigKr60aT9fwtcMdW5VbUKWAUwPj4+1bARkqQ56DdkwwP0Jlk/MslewKubXf8IzJj4kwQ4E1hfVadOKl9WVRuazd8Cbppj7JKkOWj75u6xwPnALs3ymSTv63Paq4B3Agdu1nXzlCQ3JrkBOIBeTyFJ0oC07c75+8DLq+qnAElOBr4JnD7dCVX1dXqjeG7uytkGKUlaOG179QR4fNL240yd1CVJW7i2Nf6zgdVJLmm23wSc1UlEkqROtZ1s/dQkXwX2b4reVVXf7iwqSVJnWiX+JJ+uqncCa6cokyQtIm3b+H998kaSbYGXLnw4kqSu9Xtz90PNvLsvTvJgszwE3ANcOpAIJUkLasbEX1Ufb+bd/Yuq2rFZdqiqZ1XVhwYUoyRpAfWr8a8AmC7Jp2f3DuKSJHWk38Pdv0iyDb1mneuAjfQmW38BvbduDwJOAO7sMkhJ0sLpN1bPW5PsSW/yld8DlgGP0Btb/0rgY1X1aOdRSpIWTN/unFV1C/CfBhCLJGkA2r65K/V12GHDjmDwLr982BFIs9e2H78kaSth4pekEdN2PP6r25RJkrZ8M7bxJ9keeBqwtJkUfWIo5h2B3TqOTZLUgX4Pd/8A+ADwHHr9+CcS/4PAX3UXliSpK/368Z8GnJbkfVU17WxbkqTFo+14/KcneSWwYvI5VXVeR3FJkjrSejx+4PnAOp6cgrGAaRN/kj2a/bs2x66qqtOS7AxcQO+PyA+At1XVfXMLX5I0W21f4BoH9qyqmsVnbwI+WFVrk+wAXJfkKuBo4OqqOinJ8cDxwHGzCVqSNHdt+/HfBDx7Nh9cVRuqam2z/hC98X12A44Azm0OO5fe/L2SpAFpW+NfCtyS5FvAYxOFVXV4m5Ob4Z33AVYDu1bVhmbX3fSagqY6ZyWwEmD58uUtwxSM5tAJktprm/hPnOsFkjwd+CLwgap6MMkT+6qqkkzZfFRVq4BVAOPj47NpYpIkzaBtr56vzeXDkyyhl/TPr6qLm+IfJVlWVRuSLKM3jaMkaUDaDtnw0KQ5dx9N8niSB/ucE+BMYH1VnTpp12XAUc36UTh3ryQNVNsa/w4T601CPwLYr89prwLeCdyYZF1T9qfAScCFSY4B7gDeNsuYJUnzMOvx+JsunV9KcgK9rpjTHfd1nhziYXMHzfa6i5EPWSVtidq+wPXmSZvb0OvX75SLkrQIta3xT667bqL3xu0RCx6NJKlzbdv439V1IJKkwWjbq2f3JJckuadZvphk966DkyQtvLZDNpxNrxvmc5rl8qZMkrTItE38Y1V1dlVtapZzgLEO45IkdaRt4r83yTuSbNss7wDu7TIwSVI32ib+36P3otXdwAbgLYAPfCVpEWrbq+cOoNVInJKkLVvbXj3nJtlp0vYzk5zVWVSSpM60fYHrxVV1/8RGVd2XZJ9uQlpYf/aabsdN+Oi1l3f6+ZK00Nq28W+T5JkTG828ubMe50eSNHxtk/cngG8m+UKz/VbgY92EJEnqUtuHu+clWQMc2BS9uapu6S4sSVJXWjfXNIneZC9Ji1zbNn5J0lbCxC9JI8bEL0kjpu0LXG9O8t0kDzQTrj/UYrL1s5ohnG+aVHZikruSrGuWQ+b7C0iSZqdtjf8U4PCqekZV7VhVO1TVjn3OOQc4eIryT1bV3s1y5WyClSTNX9vE/6OqWj+bD66qa4GfzD4kSVKX2nbnXJPkAuBLwGMThVV18Ryu+d4kvwusAT5YVfdNdVCSlcBKgOXLl8/hMpKkqbSt8e8IPAK8nt7E64cBh87hemcAzwf2pje88yemO7CqVlXVeFWNj40554skLZSBTrZeVT+aWE/yt8AVC/G5kqT2BjrZepJlkzZ/C7hpumMlSd3obLL1JJ8Dvgn8apI7kxwDnJLkxiQ3AAcAfzTnyCVJc9L24e5YVU1O9Ock+cBMJ1TVkVMUn9k2MElSN9om/nubCdY/12wfiZOtSxzW7Tw/M7rcOYA0R062LkkjxsnWJWnEzJj4k/xJVZ2S5HSgNt9fVcd2FpkkqRP9avwTwzSs6ToQSdJgzJj4q2ri8dEjVfWFyfuSvLWzqCRJnWn7cPdDLcskSVu4fm38bwQOAXZL8qlJu3YENnUZmCSpG/3a+P+VXvv+4cB1k8ofwrduJWlR6tfGfz1wfZLPVtXPBxSTJKlDbd/cXZHk48CewPYThVX1y51EJUnqzGwGaTuDXrv+AcB5wGe6CkqS1J22if+pVXU1kKq6o6pOBH6zu7AkSV1p29TzWJJtgO8meS9wF/D07sKSJHWlbY3//cDTgGOBlwLvAI7qKihJUnfa1vgfr6qHgYdxVE5JWtTa1vg/kWR9ko8meVGnEUmSOtUq8VfVAfR682wE/qaZPvE/dxqZJKkTbWv8VNXdVfUp4N3AOuDDMx2f5KxmYvabJpXtnOSqJN9tfj5zroFLkuamVeJP8m+TnJjkRuB04BvA7n1OOwc4eLOy44Grq+qFwNXNtiRpgNo+3D0L+Dzwhqr61zYnVNW1SVZsVnwE8Npm/Vzgq8BxLWOQJC2Avok/ybbA7VV12gJcb9eq2tCs3w3sOsN1VwIrAZYvX74Al5YkQYumnqp6HNgjyXYLeeGqKqaYznHS/lVVNV5V42NjYwt5aUkaaW2bem4H/leSy4CfThRW1amzvN6Pkiyrqg1JlgH3zPJ8SdI8te3V8z3giub4HSYts3UZT77xexRw6Rw+Q5I0D61q/FX1EYAkT6uqR9qck+Rz9B7kLk1yJ3ACcBJwYZJjgDuAt80laEnS3LVK/EleAZxJb2C25Un2Av6gqv5wunOq6shpdh006yglSQumbVPPXwJvAO6FJ2bmek1HMUmSOjSbN3d/uFnR4wsciyRpANr26vlhklcClWQJvWGa13cXliSpK21r/O8G/iOwG71JWPZutiVJi0zbXj0/Bn6n41gkSQPQdpC2U5LsmGRJkquTbEzyjq6DkyQtvLZNPa+vqgeBQ4EfAC8A/riroCRJ3Wmb+CeahH4T+EJVPdBRPJKkjrXt1XNFkluBnwHvSTIGPNpdWJKkrrSdevF44JXAeFX9nN5AbUd0GZgkqRtth2zYHjga2D9JAV8HzugwLklSR9o29ZwHPERv2kWA/wB8GnhrF0FJkrrTNvG/qKr2nLR9TZJbughIktSttr161ibZb2IjycuBNd2EJEnq0ow1/iQ30psecQnwjST/0uxaDtzacWySpA70a+o5dCBRSJIGZsbEX1V3TKw3k6+8utn8x2ZMfi1yf/aawzr9/I9ee3mnny9p9tqO1fN+4Hxgl2b5TJL3dRmYJKkbbXv1HAO8vKp+CpDkZOCbPNm9c1aS/IBe99DHgU1VNT6Xz5EkzV7bxB/+/xm3Hm/K5uOAZrhnSdIAtU38ZwOrk1zSbL+J3uTrkqRFpu1ELKcm+Sqwf1P0rqr69jyuW8BXmuEf/qaqVs3jsyRJs9C2xk9VrQXWLtB196+qu5LsAlyV5NaqunbyAUlWAisBli9fvkCXXXhd94qRpnPYCH71LreT2IJo++bugqqqu5qf9wCXAPtOccyqqhqvqvGxsbFBhyhJW62BJ/4k/ybJDhPrwOuBmwYdhySNqtZNPQtoV+CSJBPX/2xV/f0Q4pCkkTTwxF9V3wf2GvR1JUk9w6jxa4Qs9iEhFvvDe4fM0FSG8nBXkjQ8Jn5JGjEmfkkaMSZ+SRoxPtyVtGgM623lre2NYWv8kjRiTPySNGJM/JI0Ykz8kjRiTPySNGLs1aNFbbEPqbDYLfYhOdoa5twHXfQossYvSSPGxC9JI8bEL0kjxsQvSSPGh7vSVsyH35qKNX5JGjEmfkkaMUNJ/EkOTvLPSW5LcvwwYpCkUTXwxJ9kW+C/AW8E9gSOTLLnoOOQpFE1jBr/vsBtVfX9qvo/wOeBI4YQhySNpGH06tkN+OGk7TuBl29+UJKVwMpm8+Ek/zzH6y0FfjzHc0eB92d63puZDeD+pNuP79aC3J/M7xY8d6rCLbY7Z1WtAlbN93OSrKmq8QUIaavk/Zme92Zm3p+Zbcn3ZxhNPXcBe0za3r0pkyQNwDAS//8GXpjkeUm2A94OXDaEOCRpJA28qaeqNiV5L/A/gG2Bs6rq5g4vOe/moq2c92d63puZeX9mtsXen1TVsGOQJA2Qb+5K0ogx8UvSiNlqEn+/YSCSPCXJBc3+1UlWDCHMoWhxb45OsjHJumb5/WHEOSxJzkpyT5KbptmfJJ9q7t8NSV4y6BiHpcW9eW2SByZ9dz486BiHJckeSa5JckuSm5O8f4pjtszvTlUt+oXeQ+LvAb8MbAdcD+y52TF/CPx1s/524IJhx70F3Zujgb8adqxDvEevAV4C3DTN/kOAL9N7m2g/YPWwY96C7s1rgSuGHeeQ7s0y4CXN+g7Ad6b4f2uL/O5sLTX+NsNAHAGc26xfBByUzPOduMXBITL6qKprgZ/McMgRwHnV80/ATkmWDSa64Wpxb0ZWVW2oqrXN+kPAenojE0y2RX53tpbEP9UwEJv/B3jimKraBDwAPGsg0Q1Xm3sD8NvNP0UvSrLHFPtHWdt7OKpekeT6JF9O8uvDDmYYmqbjfYDVm+3aIr87W0vi1/xcDqyoqhcDV/Hkv4ykftYCz62qvYDTgS8NN5zBS/J04IvAB6rqwWHH08bWkvjbDAPxxDFJfgl4BnDvQKIbrr73pqrurarHms3/Drx0QLEtFg4zMo2qerCqHm7WrwSWJFk65LAGJskSekn//Kq6eIpDtsjvztaS+NsMA3EZcFSz/hbgH6p5+rKV63tvNmtzPJxeW6WedBnwu00Pjf2AB6pqw7CD2hIkefbEs7Ik+9LLKaNQoaL5vc8E1lfVqdMctkV+d7bY0Tlno6YZBiLJnwNrquoyev+BPp3kNnoPq94+vIgHp+W9OTbJ4cAmevfm6KEFPARJPkevd8rSJHcCJwBLAKrqr4Er6fXOuA14BHjXcCIdvBb35i3Ae5JsAn4GvH1EKlQArwLeCdyYZF1T9qfActiyvzsO2SBJI2ZraeqRJLVk4pekEWPil6QRY+KXpBFj4pekEWPil6QRY+KXpBHz/wDI63mz1KtdLQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "filtering = ContentFiltering(actual_user_representation=actual_user_representation, \n",
    "                             num_attributes=number_of_attributes,\n",
    "                             num_items=500)\n",
    "\n",
    "\n",
    "mse = MSEMeasurement(diagnostics=True)\n",
    "filtering.add_metrics(mse)\n",
    "\n",
    "filtering.startup_and_train(50)\n",
    "majority_minority_outcomes(mse, number_of_maj_users)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 450/450 [00:36<00:00, 12.40it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Majority group statistics:  0.35288460401399957 (mean),  0.34844227531793875 (std)\n",
      "Minority group statistics:  0.41705155855110726 (mean),  0.3903800760442445 (std)\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD4CAYAAADrRI2NAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAX6ElEQVR4nO3df5ReVX3v8fcHBCkIkpAQIzEOXqlebEV0KkgpglT5ISHUqsVKV8C0qbYirV0VuFwplusq6vUHUi82S6ChooDxIgmt1jQSqUWCE34FCAoEIkmBxBh+V0rSb/84e/DJMDPPnmdmn+eZnM9rrbOec/b59Z0zz/rOmX322VsRgZmZNcdO3Q7AzMzq5cRvZtYwTvxmZg3jxG9m1jBO/GZmDfOibgeQY9q0adHX19ftMMzMJpVVq1b9LCKmDy2fFIm/r6+PgYGBbodhZjapSFo3XLmreszMGsaJ38ysYZz4zcwaxonfzKxhnPjNzBrGid/MrGGc+M3MGsaJ38ysYZz4zcwaZlK8uTsec+Z057xLl3bnvGZm7fiO38ysYZz4zcwapmjil7S3pMWS7pG0RtJbJE2VtEzSvelzSskYzMxse6Xv+C8EvhMRrwUOAtYAZwHLI+IAYHlaNjOzmhRL/JJeChwBXAIQEf8ZEY8Bc4FFabNFwEmlYjAzsxcqece/P7AJuEzSrZK+ImkPYEZEPJy2eQSYMdzOkhZIGpA0sGnTpoJhmpk1S8nE/yLgjcDFEXEw8DRDqnUiIoAYbueIWBgR/RHRP336CwaQMTOzDpVM/OuB9RGxMi0vpvpD8KikmQDpc2PBGMzMbIhiiT8iHgEekvSaVHQ0cDewBJiXyuYB15aKwczMXqj0m7unA1dI2hVYC5xG9cfmaknzgXXAewvHYGZmLYom/oi4DegfZtXRJc9rZmYj85u7ZmYN48RvZtYwTvxmZg3jxG9m1jBO/GZmDePEb2bWME78ZmYN48RvZtYwbV/gkrQbcALwW8DLgf8A7gT+MSLuKhuemZlNtFETv6RPUCX9FcBKqg7VdgN+Fbgg/VH4i4i4o3CcZmY2Qdrd8d8cEX81wrrPSdoXmD3BMZmZWUGjJv6I+Mc26zfibpXNzCaVUR/uSnqppAvSYOk/l7Q5DZp+gaS9a4rRzMwmULtWPVcDW4AjI2JqROwDHJXKri4dnJmZTbx2ib8vIj6VBlUBqgFWIuJTwCvLhmZmZiW0S/zrJH1M0vMDokuaIelM4KGyoZmZWQntEv/vAfsA35e0RdIWqqadU/HIWWZmk1K7Vj1bgDPTZGZmO4CcN3ePAU4C9ktFG4BrI+I7BeMyM7NC2r25+wWqt3QvB9an4lnARyQdFxFnlA3PzMwmWrs7/uMj4leHFkq6CvgJ4MRvZjbJtHu4+wtJvzFM+W8AvygQj5mZFdbujv9U4GJJe/LLqp5XAI+ndWZmNsm0a9VzC3CIpJfR8nC39YUuMzObXNq26kk2D032kqZFxM8KxGRmZgW166TtKEnrgYclfVdSX8vq77Y7uKQHJa2WdJukgVQ2VdIySfemzynj+gnMzGxM2j3c/TRwTERMAxYCyyQdmtYp8xxHRcQbIqI/LZ8FLI+IA4DladnMzGrSLvHvOji8YkQspnqRa5Gkk4Do8JxzgUVpflE6ppmZ1aRd4n8uPdgFIP0ROBo4Dzgg4/gBfFfSKkkLUtmMiHg4zT8CzBhuR0kLJA1IGti0aVPGqczMLEe7h7tnUSXm1m6Z10t6K/DhjOMfHhEb0hCNyyTd07oyIkLSsP85RMRCquol+vv7O/3vwszMhmjXnPNfRih/HPhku4NHxIb0uVHSNcCbgUclzYyIhyXNxEM3mpnVql1Vz/NSvz3Pf2Zsv0d68QtJewDvAO4ElgDz0mbzgGvzwzUzs/HKbccPcET6fGvm9jOAayQNnudrEfEdST8CrpY0H1iH+/U3M6vVWBL/mETEWuCgYco3Uz0gNjOzLsiu6jEzsx2DE7+ZWcM48ZuZNcxYEv/X0ucVJQIxM7N6ZCf+iPi/rZ9mZjY5ZbXqkfRi4HeBvtZ9IuKvy4RlZmal5DbnvJZq1K1VwLPlwjEzs9JyE/+siDi2aCRmZlaL3Dr+GyX9etFIzMysFrl3/IcDp0p6gKqqR1Sda76+WGRmZlZEbuI/rmgUZmZWm6zEHxHrSgdiZmb18Ju7ZmYN48RvZtYwTvxmZg0zah2/pCepBkx/wSqqVj17FYnKzMyKaTfm7p51BWJmZvUY0whckvYFdhtcjoifTnhEO4g5c7p37qVLu3duM+t9WXX8kk6UdC/wAPB94EHg2wXjMjOzQnIf7p4PHAr8JCL2pxoz96ZiUZmZWTG5if+5NEj6TpJ2iojrgf6CcZmZWSG5dfyPSXoJcANwhaSNwNPlwjIzs1Jy7/jnAs8Afw58B7gfOKFUUGZmVk5u4j83Iv4rIrZGxKKI+CJwZsnAzMysjNzE//ZhyrJ67JS0s6RbJV2XlveXtFLSfZKukrRrbrBmZjZ+oyZ+SR+StBp4jaQ7WqYHgDsyz3EGsKZl+VPA5yPi1cAWYH4ngZuZWWfa3fF/DZgDLEmfg9ObIuKUdgeXNAt4J/CVtCzgbcDitMki4KROAjczs86067LhcapB1t8n6SDgt9KqfwV+nnH8LwAfAwa7ftgHeCwitqbl9cB+w+0oaQGwAGD27NkZpzIzsxy5b+5+BLgC2DdNX5V0ept9TgA2RsSqTgKLiIUR0R8R/dOnT+/kEGZmNozcdvx/CBwSEU8DSPoU8EPgolH2+U3gREnHU/XvsxdwIbC3pBelu/5ZwIZOgzczs7HLbdUjYFvL8rZUNqKIODsiZkVEH3Ay8L2IeD9wPfDutNk84NoxRWxmZuOSe8d/GbBS0jVp+STg0g7PeSZwpaT/A9wKXNLhcczMrAO5g61/TtIK4PBUdFpE3Jp7kohYAaxI82uBN48pSjMzmzBZiV/SP0TEHwC3DFNmZmaTSG4d/+taFyTtDLxp4sMxM7PS2r25e3Yad/f1kp5I05PARvxQ1sxsUho18UfE36Rxdz8TEXulac+I2Ccizq4pRjMzm0Dt7vj7oGqaOcJ6pW4ZzMxskmj3cPczknaiqtZZBWyiehnr1cBRVEMw/hVV1wtmZjYJtOur5z2SDgTeD3wAmEk1IMsa4J+AT0bEL4pHaWZmE6Ztc86IuBs4p4ZYzMysBrnNOc3MbAfhxG9m1jBO/GZmDZPbH//ynDIzM+t9oz7clbQbsDswTdIUftkV816MMHKWmZn1tnatev4Y+DPg5VTt+AcT/xPA35YLy8zMSmnXjv9C4EJJp0fEaKNtmZnZJJHbH/9Fkg4D+lr3iYjLC8VlZmaFZPfHD/wP4DZ+OQRjAE78ZmaTTO7Qi/3AgRERJYMxM7Pyctvx3wm8rGQgZmZWj9w7/mnA3ZJuBp4dLIyIE4tEZWZmxeQm/vNKBmFmZvXJbdXz/dKBmJlZPXJb9TxJ1YoHYFdgF+DpiNirVGBmZlZG7h3/noPzkgTMBQ4tFZSZmZUz5t45o/It4JjRtpO0m6SbJd0u6S5Jn0jl+0taKek+SVdJ2rWz0M3MrBO5VT3valnciapdf7shF58F3hYRT0naBfiBpG8DHwU+HxFXSvoyMB+4eOyhm5lZJ3Jb9cxpmd8KPEhV3TOi9LLXU2lxlzQF8Dbg91P5IqoWQ078ZmY1ya3jP62Tg0vamapXz1cDXwLuBx6LiK1pk/W4e2czs1rlDsQyS9I1kjam6ZuSZrXbLyK2RcQbgFnAm4HX5gYmaYGkAUkDmzZtyt3NzMzayH24exmwhKpf/pcDS1NZloh4DLgeeAuwt6TB/zRmARtG2GdhRPRHRP/06dNzT2VmZm3kJv7pEXFZRGxN098Do2ZjSdMl7Z3mfwV4O7CG6g/Au9Nm84BrOwnczMw6k5v4N0s6RdLOaToF2Nxmn5nA9ZLuAH4ELIuI64AzgY9Kug/YB7ik0+DNzGzsclv1fAC4CPg8VcucG4FRH/hGxB3AwcOUr6Wq7zczsy7IbdWzDnBPnGZmO4DcVj2LBuvr0/IUSZcWi8rMzIrJreN/fWqZA0BEbGGYahwzM+t9uYl/J0lTBhckTSX/+YCZmfWQ3OT9WeCHkr6Rlt8DfLJMSGZmVlLuw93LJQ1Q9bMD8K6IuLtcWGZmVkp2dU1K9E72ZmaT3Jj74zczs8nNid/MrGGc+M3MGib3Ba53SbpX0uOSnpD0pKQnSgdnZmYTL/fh7qeBORGxpmQwZmZWXm5Vz6NO+mZmO4bcO/4BSVcB36IaRB2AiPj/JYIyM7NychP/XsAzwDtaygJw4jczm2SKDrZuZma9p+hg62Zm1ntqGWzdzMx6R7HB1s3MrDeVHGzdzMx6UG7i/wDwXuAR4GHg3bQZbN3MzHqTB1s3M2uYURO/pI9FxKclXUTVbn87EfGRYpGZmVkR7e74B7tpGCgdiJmZ1WPUxB8RS9PsMxHxjdZ1kt5TLCozMysm9+Hu2Zllz5P0CknXS7pb0l2SzkjlUyUtS908L5M0ZaxBm5lZ59rV8R8HHA/sJ+mLLav2Ara2OfZW4C8i4hZJewKrJC0DTgWWR8QFks4CzgLO7PQHMDOzsWlXx//vVPX7JwKrWsqfBP58tB0j4mGqpp9ExJOS1gD7AXOBI9Nmi4AVOPGbmdWmXR3/7cDtkr4WEc91ehJJfcDBwEpgRvqjANV7ATNG2GcBsABg9uzZnZ7azMyGyK3j75O0ONXXrx2ccnaU9BLgm8CfRcR2wzVGRDBMM9G0bmFE9EdE//Tp7h3CzGyijKWTtoup6u2PAi4HvtpuJ0m7UCX9K1oGbXlU0sy0fiawcaxBm5lZ53IT/69ExHJAEbEuIs4D3jnaDpIEXAKsiYjPtaxaAsxL8/OAa8cWspmZjUfuCFzPStoJuFfSh4ENwEva7PObwB8AqyXdlsr+F3ABcLWk+cA6qj6AzMysJrmJ/wxgd+AjwPlU1T3zRtshIn4AaITVR+cGaGZmEys38W+LiKeAp3CvnGZmk1pu4v+spJcBi4GrIuLOgjFNuI8fMaf2c55/w9L2GxUyp/4fF4Cl3fuRzWwMsh7uRsRRVNU7m4C/k7Ra0v8uGpmZmRWR26qHiHgkIr4IfBC4DTi3VFBmZlZOVuKX9D8lnSdpNXARcCMwq2hkZmZWRG4d/6XAlcAxEfHvBeMxM7PC2iZ+STsDD0TEhTXEY2ZmhbWt6omIbcArJO1aQzxmZlZYblXPA8C/SVoCPD1YOKQrBjMzmwRyE//9adoJ2LNcOGZmVlpW4o+ITwBI2j0inikbkpmZlZTbnPMtku4G7knLB0n6f0UjMzOzInJf4PoCcAywGZ4fmeuIQjGZmVlBY3lz96EhRdsmOBYzM6tB7sPdhyQdBkQaVesMYE25sMzMrJTcO/4PAn8K7Ec1CMsb0rKZmU0yua16fga8v3AsZmZWg9xWPZ+WtJekXSQtl7RJ0imlgzMzs4mXW9Xzjoh4AjgBeBB4NfCXpYIyM7NychP/YJXQO4FvRMTjheIxM7PCclv1XCfpHuA/gA9Jmg78olxYZmZWSu7Qi2cBhwH9EfEcVUdtc0sGZmZmZWTd8UvaDTgVOFxSAD8ALi4Yl5mZFZJb1XM58CTVsIsAvw/8A/CeEkGZmVk5uYn/1yLiwJbl61OnbSOSdClVK6CNEfFrqWwqcBXQR9U66L0RsWWsQU8GHz9iTu3nPP+GpbWfs1fMqf9yA7C0uZfcJrHcVj23SDp0cEHSIcBAm33+Hjh2SNlZwPKIOABYnpbNzKxGo97xS1oNBLALcKOkn6ZVs0ldNI8kIm6Q1DekeC5wZJpfBKwAzhxTxGZmNi7tqnpOmODzzYiIh9P8I8CMkTaUtABYADB79uwJDsPMrLlGreqJiHWDE7A3MCdNe6eyjkVEUP03MdL6hRHRHxH906dPH8+pzMysRW5fPWcAVwD7pumrkk7v4HyPSpqZjjkT2NjBMczMbBxyH+7OBw6JiHMj4lzgUOCPOjjfEmBemp8HXNvBMczMbBxyE7/YfsStbals5B2krwM/BF4jab2k+cAFwNsl3Qv8dlo2M7Ma5bbjvwxYKematHwScMloO0TE+0ZYdXTmOc3MrIDcgVg+J2kFcHgqOi0ibi0WlZmZFZN7x09E3ALcUjAWG6e63xZu8pvCZpNZbh2/mZntIJz4zcwaxonfzKxhnPjNzBrGid/MrGGc+M3MGia7OaeZvVC3BoABDwJjnfMdv5lZwzjxm5k1jBO/mVnDOPGbmTWME7+ZWcO4VY91bGincDd/pkuBAB8/otyx3Rmd7Wh8x29m1jBO/GZmDePEb2bWMK7jN2ujVwe46dZbw35jePLzHb+ZWcM48ZuZNYyresysGVZ0oW7syN6sF/Mdv5lZwzjxm5k1jKt6zGxMujkGwXiUfLt7JOd/dnz7l2pB1ZU7fknHSvqxpPskndWNGMzMmqr2xC9pZ+BLwHHAgcD7JB1YdxxmZk3VjTv+NwP3RcTaiPhP4EpgbhfiMDNrpG7U8e8HPNSyvB44ZOhGkhYAC9LiU5J+PMbzTAN+BnDddR1EWY/nY+xRvR4f7JAxqlggI9gBr+ELFc4DI8Q3vt+lxv9VeOVwhT37cDciFgILO91f0kBE9E9gSBOu12Ps9fjAMU6EXo8Pej/GXo9vqG5U9WwAXtGyPCuVmZlZDbqR+H8EHCBpf0m7AicDS7oQh5lZI9Ve1RMRWyV9GPhnYGfg0oi4q8CpOq4mqlGvx9jr8YFjnAi9Hh/0foy9Ht92FBHdjsHMzGrkLhvMzBrGid/MrGEmZeJv1+WDpBdLuiqtXympr2Xd2an8x5KO6VJ8H5V0t6Q7JC2X9MqWddsk3ZamYg+9M2I8VdKmllj+sGXdPEn3pmleF2P8fEt8P5H0WMu64tdR0qWSNkq6c4T1kvTFFP8dkt7Ysq74NcyI7/0prtWSbpR0UMu6B1P5bZIGSsSXGeORkh5v+V2e27KueNcvGfH9ZUtsd6bv3dS0rpZr2JGImFQT1QPh+4FXAbsCtwMHDtnmT4Avp/mTgavS/IFp+xcD+6fj7NyF+I4Cdk/zHxqMLy0/1SPX8FTgb4fZdyqwNn1OSfNTuhHjkO1Pp2ooUOd1PAJ4I3DnCOuPB75N9RbPocDKmq9hu/gOGzwvVRcqK1vWPQhM64FreCRw3Xi/H6XiG7LtHOB7dV/DTqbJeMef0+XDXGBRml8MHC1JqfzKiHg2Ih4A7kvHqzW+iLg+Ip5JizdRvctQp/F0m3EMsCwifh4RW4BlwLE9EOP7gK8XiGNEEXED8PNRNpkLXB6Vm4C9Jc2kpmvYLr6IuDGdH7rzPcy5hiOppeuXMcZX+3ewU5Mx8Q/X5cN+I20TEVuBx4F9MvetI75W86nuCgftJmlA0k2STprg2Ablxvi7qSpgsaTBl+7quIZjOk+qKtsf+F5LcR3XsZ2Rfoa6ruFYDP0eBvBdSatUdZ/STW+RdLukb0t6XSrrqWsoaXeqP97fbCnupWu4nZ7tsqEJJJ0C9ANvbSl+ZURskPQq4HuSVkfE/V0Ibynw9Yh4VtIfU/0H9bYuxJHjZGBxRGxrKeuV69jzJB1FlfgPbyk+PF2/fYFlku5Jd791u4Xqd/mUpOOBbwEHdCGOduYA/xYRrf8d9Mo1fIHJeMef0+XD89tIehHwUmBz5r51xIek3wbOAU6MiGcHyyNiQ/pcC6wADp7g+LJijIjNLXF9BXhT7r51xdjiZIb8i13TdWxnpJ+hZ7otkfR6qt/v3IjYPFjecv02Atcw8VWiWSLiiYh4Ks3/E7CLpGn00DVMRvsOdvUaDqvbDxnGOlH9l7KW6l/7wYc6rxuyzZ+y/cPdq9P869j+4e5aJv7hbk58B1M9mDpgSPkU4MVpfhpwL2UeWOXEOLNl/neAm9L8VOCBFOuUND+1GzGm7V5L9RBNdV/HdPw+Rn4w+U62f7h7c53XMCO+2VTPuQ4bUr4HsGfL/I3AsSXiy4jxZYO/W6rE+dN0PbO+H6XjS+tfSvUcYI9uXcMx/0zdDqDDX8TxwE9S8jwnlf011d0zwG7AN9KX+mbgVS37npP2+zFwXJfi+xfgUeC2NC1J5YcBq9OXeDUwv4vX8G+Au1Is1wOvbdn3A+na3gec1q0Y0/J5wAVD9qvlOlLd4T0MPEdVxzwf+CDwwbReVIMO3Z/i6K/zGmbE9xVgS8v3cCCVvypdu9vTd+Ccgr/jdjF+uOV7eBMtf6SG+37UHV/a5lSqRiOt+9V2DTuZ3GWDmVnDTMY6fjMzGwcnfjOzhnHiNzNrGCd+M7OGceI3M2sYJ34zs4Zx4jcza5j/BsS0G9T3y3E7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "filtering.run(450)\n",
    "majority_minority_outcomes(mse, number_of_maj_users)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## System state\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some applications might require keeping a history of the system's internal state for future processing. This is useful, for example, to study the evolution of predicted user profiles. The framework provides an interface to store and access all the states of each component over time through the `SystemStateModule` interface. Some components are tracked by default, others are added into the individual models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note that we instantiated filtering with the option record_base_state=True\n",
    "system_state = filtering.get_system_state()\n",
    "print(\"These are the system state components being monitored:\")\n",
    "print(system_state.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"There are as many states as the timesteps for which we ran the system + the initial state.\")\n",
    "print(\"For example, the history of predicted_user_profiles has length:\", (len(system_state['predicted_users'])))\n",
    "# the last states correspond to the current state of the components\n",
    "print(\"Furthermore, the last state is in the history of a component corresponds to its current state.\")\n",
    "print(\"Is this true for predicted_user_profiles?\", np.array_equal(system_state['predicted_users'][5], filtering.predicted_user_profiles))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start tracking a new component, you can use the [add_state_variable()](https://elucherini.github.io/t-recs/reference/components.html#base.base_components.SystemStateModule.add_state_variable). Note that state variables can only be monitored if they must inherit from the `BaseComponent` class. Creating new state variables is outside of the scope of this guide, so please refer to the [advanced-models](advanced-models.ipynb) and the [advanced-metrics](advanced-metrics.ipynb) notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## \"Real\" users, \"real\" items\n",
    "Most of what we've seen so far about users refers to the predictions that the system makes about users' preferences and the item attributes. This framework allows for modeling system predictions as well as \"real\" users and items. These \"real\" user and item attributes can be passed into the simulation model; while they remain invisible by the recommender system algorithm, they dictate which items users interact with, and this interaction data is what the recommender system uses to continually retrain.\n",
    "\n",
    "By default, the preference ordering over items for a particular user is defined by the dot product between the item profile and the user profile. You can"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# consider a toy example where all users have a very strong preference for items with the attribute at index 0,\n",
    "# and only the first item has the attribute at index 0\n",
    "\n",
    "# each row is a user\n",
    "real_users = np.array([\n",
    "    [1, 0, 0, 0, 0.7],\n",
    "    [1, 0, 0.1, 0, 0],\n",
    "    [1, 0, 0, 0.2, 0],\n",
    "    [1, 0, 0.5, 0, 0],\n",
    "])\n",
    "\n",
    "# each column is an item\n",
    "real_items = np.array([\n",
    "    [1.0, 0.0, 0.0, 0.0, 0.0],\n",
    "    [0.0, 0.1, 0.0, 0.2, 0.0],\n",
    "    [0.0, 0.0, 0.2, 0.0, 0.0],\n",
    "    [0.0, 0.0, 0.0, 0.0, 0.3],\n",
    "    [0.1, 0.1, 0.2, 0.3, 0.0],\n",
    "])\n",
    "\n",
    "# instantiate content filtering model; note that the system's internal model of\n",
    "# each user's preferences over items will essentially be random\n",
    "# we set the number of items shown to users at each iteration to be 5\n",
    "filtering = ContentFiltering(actual_user_representation=real_users,\n",
    "                             actual_item_representation=real_items,\n",
    "                             num_items_per_iter=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now when we run the model, we can look at the most recent recommendations\n",
    "# and the items the users actually interacted with\n",
    "filtering.run(timesteps=1)\n",
    "print(\"Items shown to each user during the first timestep (row = one user)\")\n",
    "print(filtering.items_shown)\n",
    "\n",
    "print()\n",
    "print(\"Items each user chose to interact with (item at index i represents the item user i chose)\")\n",
    "print(filtering.interactions)\n",
    "\n",
    "print()\n",
    "filtering.run(timesteps=1)\n",
    "print(\"Items shown to each user during the second timestep (row = one user).\")\n",
    "print(\"Note that the the first item shown is item 0, indicating the recommender system has learned the users' preferences.\")\n",
    "print(filtering.items_shown)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### User interactions with items\n",
    "The Users class also determines how users interact with items. The default behavior is defined in [get_user_feedback()](https://elucherini.github.io/t-recs/reference/components.html#components.users.Users.get_user_feedback). In short, when the system presents items to users, users internally evaluate the items and choose the one item that maximizes the dot product between their own preferences and the item attributes. This default behavior can be overriden to provide a custom model of user-item interaction (for an example, see [DNUsers](https://elucherini.github.io/t-recs/reference/components.html#components.users.DNUsers)).\n",
    "\n",
    "## Model parameters about user interactions\n",
    "Models also provide a few initialization parameters that can be used to tweak the behavior of the model in regards to user interactions. Specifically, models determine the number of items to present users at each iteraction through parameter `num_items_per_iter`. The default is 10 items per user per iteration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other model parameters\n",
    "Please refer to [the docs](https://elucherini.github.io/algo-segregation/reference/models.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}