ipynb/jotto.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"right\" style=\"text-align: right;\"><i>Peter Norvig<br>April 2020</i></div>\n",
    "\n",
    "# Jotto\n",
    "\n",
    "[Jotto](https://en.wikipedia.org/wiki/Jotto) is a word game in which a **guesser** tries to guess a secret **target** word, which is chosen from a list of permissible words, in as few guesses as possible. Each guess must be one of the permissible words, and the **reply** to each guess is the number of letters in common between the guess word and the target word, regardless of the positions of the letters.   \n",
    "\n",
    "Here is an example Jotto game, where I show the guesses, the replies, the number of remaining targets that are **consistent** with all the replies seen so far, and finally the letters that matched (this is as an aid to you, the reader; these matching letters are not known to the guesser). In this game, the guesser gets to the target word, \"wonky\", in 7 guesses. \n",
    "\n",
    "    Guess 1: stoma, Reply: 1, Consistent targets: 1118    (Matched: \"o\")\n",
    "    Guess 2: bairn, Reply: 1, Consistent targets: 441     (Matched: \"n\")\n",
    "    Guess 3: swipe, Reply: 1, Consistent targets: 197     (Matched: \"w\")\n",
    "    Guess 4: lurks, Reply: 1, Consistent targets: 87      (Matched: \"k\")\n",
    "    Guess 5: rowdy, Reply: 3, Consistent targets: 14      (Matched: \"owy\")\n",
    "    Guess 6: roved, Reply: 1, Consistent targets: 2       (Matched: \"o\")\n",
    "    Guess 7: wonky, Reply: 5, Consistent targets: 1       (Matched: \"wonky\")\n",
    "\n",
    "\n",
    "There are several variants of the game; here are four key questions and my answers:\n",
    "\n",
    "- Q: How many letters can each word be? <br>A: **Only five-letter words are allowed in the word list**.\n",
    "- Q: Can a guess be a word that is not in the word list? <br>A: **No. (If the guesser proposes a non-word, the reply is 0.)**\n",
    "- Q: What is the reply for a word that has the same letter twice, like the `s` in `stars`? <br>A: **Only words with no repeated letters are allowed in the word list**.\n",
    "- Q: What if the reply is `5`, but the guess is not the target?<br>A: **No two words in the word list are allowed to have the same set of five letters**.<br>**For example, only one of** `{'apers', 'pares', 'parse', 'pears', 'reaps', 'spare', 'spear'}` **is allowed.**\n",
    "\n",
    "# Choosers and Guessers\n",
    "\n",
    "Typically, the **guesser** plays against a **chooser**, who chooses the target word. This introduces a game-theoretic aspect: the chooser chooses a word that should be difficult for the guesser, but knowing that, the guesser modifies their strategy, and knowing that, the chooser further modifies their choice, etc. To avoid these complications, I will instead use a scenario where every target word is equally likely, and to evaluate a guesser we have it play one game with each target word, and aggregate the scores from all those games. (This is appropriate when the guesser is a program, but would be extremely tedious for a human guesser.)\n",
    "\n",
    "First some imports and (if necessary) the download of a file of words, `sgb-words.txt`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import random\n",
    "from typing      import List, Tuple, Dict, Union\n",
    "from statistics  import mean, stdev\n",
    "from collections import defaultdict, Counter\n",
    "from math        import log\n",
    "\n",
    "! [ -e sgb-words.txt ] || curl -O https://norvig.com/ngrams/sgb-words.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make a Jotto word list from `sgb-words.txt` by putting all the words that have five distinct letters into a dict keyed by the set of letters in each word, and then keeping only one word for each letter set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def jotto_words(words) -> List[str]:\n",
    "    \"\"\"Build a list of permissible Jotto words from an iterable of words.\"\"\"\n",
    "    lettersets = {frozenset(w): w \n",
    "                  for w in words if len(w) == 5 == len(frozenset(w))}\n",
    "    return list(lettersets.values())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2845"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wordlist = jotto_words(open('sgb-words.txt').read().split())\n",
    "len(wordlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see there are 2,845 permissible target words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Playing a Game\n",
    "\n",
    "We will define the function `play_jotto` to play a game and optionally produce the output shown above in the example game (without the \"(Matched: ...)\" part), and finally return as a score the number of guesses made. To avoid a possible infinite loop with a very poor guesser, we limit the number of guesses to the number of words in the word list.\n",
    "\n",
    "The first argument to `play_jotto` is a `guesser`. I choose to implement a guesser as a `callable` (e.g., a function) that is passed the current state of the game and returns a guess word. So how do we represent the state of the game? `play_jotto` is keeping track of the `targets` that are still consistent given previous guesses and replies, so we might as well pass that in. In addition, we pass in the reply from the previous guess. Note that a guess need not be one of the consistent targets, but it must be one of the words in the word list.  The second argument is a `chooser`: a function that selects a word from a wordlist. By default, it chooses randomly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_jotto(guesser, chooser=random.choice, wordlist=wordlist, verbose=False) -> int:\n",
    "    \"\"\"How many guesses does it take for `guesser` to guess the Jotto target word,\n",
    "    which is selected by `chooser` from the words in `wordlist`?\"\"\"\n",
    "    targets = wordlist # The targets that are consistent with all replies\n",
    "    target  = chooser(targets)\n",
    "    reply   = None\n",
    "    for i in range(1, len(wordlist) + 1):\n",
    "        guess = guesser(reply, targets)\n",
    "        reply = reply_for(target, guess) if guess in wordlist else -1\n",
    "        targets = [w for w in targets if reply_for(guess, w) == reply]\n",
    "        if verbose: \n",
    "            print(f'Guess {i}: {guess}, Reply: {reply}, Consistent targets: {len(targets)}')\n",
    "        if guess == target or i == len(wordlist):  \n",
    "            return i\n",
    "        \n",
    "Reply = int # A reply to a guess is an integer (from 0 to 5)\n",
    "                         \n",
    "def reply_for(target, guess) -> Reply: \n",
    "    \"The number of letters in common between the target and guess\"\n",
    "    return len(set(target).intersection(guess))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Guesser\n",
    "\n",
    "One simple guesser function is `random_guesser`: from the remaining consistent  targets, pick one at random. That sounds hopelessly naive, but it is actually a decent strategy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Guess 1: delis, Reply: 3, Consistent targets: 410\n",
      "Guess 2: coeds, Reply: 2, Consistent targets: 210\n",
      "Guess 3: dirts, Reply: 2, Consistent targets: 97\n",
      "Guess 4: veldt, Reply: 2, Consistent targets: 44\n",
      "Guess 5: adieu, Reply: 2, Consistent targets: 17\n",
      "Guess 6: child, Reply: 5, Consistent targets: 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def random_guesser(_, targets) -> str: return random.choice(targets)\n",
    "\n",
    "play_jotto(random_guesser, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating Guessers\n",
    "\n",
    "That was just one sample game. How well will this guesser do averaged over all target words? The function `play_jottos` plays a game against every target and collects the scores. Then, `show` displays a histogram and the mean, standard deviation, and worst case number of guesses. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def play_jottos(guesser, targets=wordlist) -> List[int]:\n",
    "    \"\"\"Scores for this guesser on all targets.\"\"\"\n",
    "    return [play_jotto(guesser, lambda _: target, wordlist, verbose=False)\n",
    "            for target in targets]\n",
    "    \n",
    "def show(scores: List[int]):\n",
    "    \"\"\"Show a histogram and statistics for these scores.\"\"\"\n",
    "    bins = range(min(scores), max(scores) + 2)\n",
    "    plt.hist(scores, align='left', rwidth=0.9, bins=bins,\n",
    "             weights=[100 / len(scores) for _ in scores])\n",
    "    plt.xticks(bins[:-1])\n",
    "    plt.xlabel('Number of guesses to find target'); plt.ylabel('% of targets')\n",
    "    print(f'mean = {mean(scores):.2f} ± {stdev(scores):.2f} guesses; worst = {max(scores)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean = 7.32 ± 1.66 guesses; worst = 16\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZF0lEQVR4nO3deZhcVZnH8e+PhCWArOlghGAHJu5LwIhglEEWHxaHRVFEUVQ0juwqM0ZwFETHoCKK44ggm7L4oCwiIIssKi5AglmNGpeAYEgaF5ZRkZB3/jinTNGp7q6uqludzvl9nqeeuvfWve99a+m3bp+69xxFBGZmVo71RjoBMzPrLhd+M7PCuPCbmRXGhd/MrDAu/GZmhRk70gk0Y/z48dHb2zvSaZiZjSpz5sx5OCJ6+i8fFYW/t7eX2bNnj3QaZmajiqT7Gi13U4+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhRsWVu2YAvTOvbzvG0lkHdCATs9HNR/xmZoVx4TczK4wLv5lZYVz4zcwK48JvZlYYF34zs8K48JuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRWmssIvaZKk2yUtlrRI0gl5+amSHpQ0N9/2ryoHMzNbU5UDsawEPhgR90p6BjBH0i35sbMi4rMV7tvMzAZQWeGPiGXAsjz9mKTFwLZV7c/MzJrTlTZ+Sb3ATsBdedGxkuZLukDSlgNsM0PSbEmz+/r6upGmmVkRKi/8kjYFrgROjIhHgS8DOwJTSf8RnNlou4g4NyKmRcS0np6eqtM0MytGpYVf0vqkon9pRFwFEBHLI+KpiFgFnAfsUmUOZmb2dFWe1SPgfGBxRHyubvnEutUOARZWlYOZma2pyrN6pgNvAxZImpuXnQwcLmkqEMBS4L0V5mBmZv1UeVbPnYAaPHRDVfs0a1bvzOvbjrF01gEdyMSs+3zlrplZYVz4zcwK48JvZlYYF34zs8K48JuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhXPjNzArjwm9mVhgXfjOzwrjwm5kVxoXfzKwwLvxmZoVx4TczK4wLv5lZYVz4zcwK48JvZlYYF34zs8K48JuZFaaywi9pkqTbJS2WtEjSCXn5VpJukbQk329ZVQ5mZramKo/4VwIfjIjnA7sCx0h6ATATuDUipgC35nkzM+uSygp/RCyLiHvz9GPAYmBb4CDg4rzaxcDBVeVgZmZr6kobv6ReYCfgLmCbiFgG6csBmDDANjMkzZY0u6+vrxtpmpkVofLCL2lT4ErgxIh4tNntIuLciJgWEdN6enqqS9DMrDCVFn5J65OK/qURcVVevFzSxPz4RGBFlTmYmdnTVXlWj4DzgcUR8bm6h64FjszTRwLfrioHMzNb09gKY08H3gYskDQ3LzsZmAVcIeko4H7gjRXmYGZm/VRW+CPiTkADPLxXVfs1M7PB+cpdM7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhhiz8knaUtGGe3kPS8ZK2qD41MzOrQjNH/FcCT0n6F9KVuJOByyrNyszMKtNM4V8VESuBQ4DPR8T7gYnVpmVmZlVppvA/KelwUr861+Vl61eXkpmZVamZwv9OYDfgkxHxO0mTgUuqTcvMzKrSTF89+0TE8bWZXPz/VmFOZmZWoWYK/5HAF/ote0eDZWZr6J15fdsxls46oAOZmFnNgIU/t+u/BZgs6dq6h54B/LHqxMzMrBqDHfH/GFgGjAfOrFv+GDC/yqTMzKw6Axb+iLgPuA/YTdKzgSkR8T1J44BxpC8AMzMbZZq5cvc9wLeAr+RF2wHXVJmUmZlVp5nTOY8hDaP4KEBELAEmVJmUmZlVp5nC/0RE/KM2I2ksENWlZGZmVWqm8H9f0snAOEn7AN8EvlNtWmZmVpVmCv9MoA9YALwXuAH4SJVJmZlZdYa8gCsiVgHn5ZuZmY1yQxZ+SQtYs03/EWA28ImI8MVcZmajSDNdNnwXeIrVffC/Od8/ClwE/Fvn0zIzs6o0U/inR8T0uvkFkn4UEdMlHVFVYmZmVo1mftzdVNIrajOSdgE2zbMrK8nKzMwq08wR/1HAhZJqxf4x4ChJmwCfqiwzMzOrxKCFX9J6wA4R8WJJmwOKiL/UrXJFpdmZmVnHDdrUk0/lPDZPP9Kv6A9K0gWSVkhaWLfsVEkPSpqbb/u3nLmZmbWkmTb+WySdJGmSpK1qtya2uwjYt8HysyJiar7dMKxszcysbc208b8r3x9TtyyAHQbbKCJ+IKm3tbTMzKwqzVy5O7nD+zxW0ttJF4B9MCL+3GglSTOAGQDbb799h1MwMytXM009SHqRpDdJenvt1uL+vgzsCEwlje515kArRsS5ETEtIqb19PS0uDszM+uvmS4bPgbsAbyA1EHbfsCdwNeGu7OIWF4X9zzguuHGMDOz9jRzxH8osBfwUES8E3gpsGErO5M0sW72EGDhQOuamVk1mvlx928RsUrSSkmbASsY4oddAEmXk/5TGC/pAeBjwB6SppJ+HF5K6ubZzMy6qJnCP1vSFqRumecAjwN3D7VRRBzeYPH5w0vPzMw6rZmzeo7Ok+dIuhHYLCLmV5uWmZlVZcg2fkm31qYjYmlEzK9fZmZmo8uAR/ySNgI2JrXRbwkoP7QZ8Kwu5GZmZhUYrKnnvcCJpCI/h9WF/1HgSxXnZWZmFRmw8EfEF4AvSDouIr7YxZzMzKxCQ7bxu+ibma1bmuqywczM1h0DFn5J0/N9S1fpmpnZ2mmwI/6z8/1PupGImZl1x2Bn9Twp6UJgW0ln938wIo6vLi0zM6vKYIX/dcDewJ6k0znNzGwdMNjpnA8D35C0OCLmdTEnMzOrUDNn9fxR0tV54PTlkq6UtF3lmZmZWSWaKfwXAteSruDdFvhOXmZmZqNQM4V/QkRcGBEr8+0iwGMhmpmNUs0U/j5JR0gak29HAH+sOjEzM6tGM4X/XcCbgIdIA6QfmpeZmdko1MxALPcDB3YhFzMz6wL31WNmVhgXfjOzwrjwm5kVpunCL2lXSbdJ+pGkg6tMyszMqjPYmLvPjIiH6hZ9gPQjr4AfA9dUnJuZmVVgsLN6zpE0B/hMRPwd+AvwFmAVadxdMzMbhQZs6omIg4G5wHWS3kYaeH0VsDHgph4zs1Fq0PP4I+I7km4AjgauAj4ZET/sSmZmo0TvzOs7EmfprAM6EsdsKIMNvXigpDuB24CFwJuBQyRdLmnHbiVoZmadNdgR/yeA3YBxwA0RsQvwAUlTgE+SvgjMzGyUGazwP0Iq7uOAFbWFEbEEF30zs1FrsPP4DyH9kLuSdDbPsEi6IA/esrBu2VaSbpG0JN9vOfyUzcysHYOd1fNwRHwxIs6JiFZO37wI2LffspnArRExBbg1z5uZWRdV1mVDRPwA+FO/xQcBF+fpi/FpoWZmXdftvnq2iYhlAPl+wkArSpohabak2X19fV1L0MxsXbfWdtIWEedGxLSImNbT45Eezcw6pduFf7mkiQD5fsUQ65uZWYd1u/BfCxyZp48Evt3l/ZuZFa+ywi/pcuAnwHMlPSDpKGAWsI+kJcA+ed7MzLpoyDF3WxURhw/w0F5V7dPMzIa21v64a2Zm1XDhNzMrjAu/mVlhXPjNzArjwm9mVhgXfjOzwrjwm5kVxoXfzKwwlV3AZaNXJwYP98DhZmsvH/GbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhXPjNzArjwm9mVhgXfjOzwrjwm5kVxoXfzKwwLvxmZoVx4TczK4wLv5lZYVz4zcwK48JvZlYYF34zs8K48JuZFcaF38ysMC78ZmaFGZExdyUtBR4DngJWRsS0kcjDzKxEIznY+msi4uER3L+ZWZHc1GNmVpiRKvwB3CxpjqQZjVaQNEPSbEmz+/r6upyemdm6a6QK//SI2BnYDzhG0u79V4iIcyNiWkRM6+np6X6GZmbrqBEp/BHxh3y/Arga2GUk8jAzK1HXf9yVtAmwXkQ8lqdfC3y823mYrY16Z17fkThLZx3QkTi2bhqJs3q2Aa6WVNv/ZRFx4wjkYWZWpK4X/oj4LfDSbu/XzMwSn85pZlYYF34zs8K48JuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAjOeaudVAnuvN1V75mZfARv5lZYVz4zcwK48JvZlYYF34zs8K48JuZFcaF38ysMD6d02wd5tN8rREf8ZuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MCuPTOc2sKT41dN3hI34zs8L4iH+EdOLoCXwEZaOT/3sYWSNyxC9pX0m/lPRrSTNHIgczs1J1/Yhf0hjgS8A+wAPAPZKujYifdzuX4fJRutnax/89DN9INPXsAvw6In4LIOkbwEHAWl/4zWzdVsqXiCKiuzuUDgX2jYh35/m3Aa+IiGP7rTcDmJFnnwv8ssK0xgMPO9aoj9XpeI61bsTqdLy1NVYjz46Inv4LR+KIXw2WrfHtExHnAudWnw5Imh0R0xxrdMfqdDzHWjdidTre2hprOEbix90HgEl189sBfxiBPMzMijQShf8eYIqkyZI2AN4MXDsCeZiZFanrTT0RsVLSscBNwBjggohY1O08+ulkk5JjjVysTsdzrHUjVqfjra2xmtb1H3fNzGxkucsGM7PCuPCbmRWm6MIv6QJJKyQtbDPOJEm3S1osaZGkE9qMt5GkuyXNy/FOaydejjlG0s8kXddmnKWSFkiaK2l2m7G2kPQtSb/Ir91uLcZ5bs6ndntU0olt5PX+/LovlHS5pI3aiHVCjrOolZwafUYlbSXpFklL8v2WbcR6Y85tlaSmTyscINZn8ns5X9LVkrZoI9bpOc5cSTdLelarseoeO0lSSBrfTKxBcjtV0oN1n7f928lN0nG5C5tFkj7dbG5tiYhib8DuwM7AwjbjTAR2ztPPAH4FvKCNeAI2zdPrA3cBu7aZ4weAy4Dr2oyzFBjfodf/YuDdeXoDYIsOxBwDPES6cKWV7bcFfgeMy/NXAO9oMdaLgIXAxqQTKb4HTBlmjDU+o8CngZl5eiZwRhuxnk+6QPIOYFqbeb0WGJunz2gzr83qpo8Hzmk1Vl4+iXRCyX3D+fwOkNupwEktfB4axXpN/lxsmOcntPJZG+6t6CP+iPgB8KcOxFkWEffm6ceAxaQC0mq8iIjH8+z6+dbyr/CStgMOAL7aaoxOk7QZ6Q/hfICI+EdE/KUDofcCfhMR97URYywwTtJYUtFu9TqT5wM/jYi/RsRK4PvAIcMJMMBn9CDSlyb5/uBWY0XE4ogY9lXxA8S6OT9PgJ+SrtFpNdajdbOb0OTnf5C/6bOA/2w2ThPxhm2AWO8DZkXEE3mdFZ3Y11CKLvxVkNQL7EQ6Sm8nzhhJc4EVwC0R0U68z5M+9KvaySkL4GZJc3K3Gq3aAegDLsxNUF+VtEkH8nszcHmrG0fEg8BngfuBZcAjEXFzi+EWArtL2lrSxsD+PP3ixVZtExHLcr7LgAkdiNlp7wK+204ASZ+U9HvgrcBH24hzIPBgRMxrJ59+js1NURc029Q2gOcAr5Z0l6TvS3p5pxIcjAt/B0naFLgSOLHfEcuwRcRTETGVdNS0i6QXtZjT64AVETGnnXzqTI+InYH9gGMk7d5inLGkf3u/HBE7Af9HarZoWb4g8EDgm23E2JJ0RD0ZeBawiaQjWokVEYtJTR63ADcC84CVg260DpB0Cul5XtpOnIg4JSIm5TjHDrX+ALlsDJxCG18cDXwZ2BGYSjo4OLONWGOBLYFdgf8ArpDUqFubjnLh7xBJ65OK/qURcVWn4ubmjzuAfVsMMR04UNJS4BvAnpIuaSOfP+T7FcDVpN5WW/EA8EDdfzLfIn0RtGM/4N6IWN5GjL2B30VEX0Q8CVwFvLLVYBFxfkTsHBG7k/7NX9JGbjXLJU0EyPddaR5ohqQjgdcBb43caN0BlwFvaHHbHUlf4vPy38B2wL2SntlqMhGxPB+YrQLOo/W/AUh/B1fl5t27Sf+VN/3jc6tc+Dsgf0OfDyyOiM91IF5P7YwISeNIxegXrcSKiA9HxHYR0UtqBrktIlo6gpW0iaRn1KZJP+a1dEZURDwE/F7Sc/OivWi/a+7DaaOZJ7sf2FXSxvl93Yv0m01LJE3I99sDr+9AfpC6ODkyTx8JfLsDMdsmaV/gQ8CBEfHXNmNNqZs9kNY//wsiYkJE9Oa/gQdIJ2I81EZuE+tmD6HFv4HsGmDPHPc5pJMcquytM+nGL8hr6430R7gMeJL0gTiqxTivIrV9zwfm5tv+beT1EuBnOd5C4KMder570MZZPaR2+Xn5tgg4pc18pgKz8/O8BtiyjVgbA38ENu/A63QaqdAsBL5OPuOixVg/JH2hzQP2amH7NT6jwNbAraT/Hm4Ftmoj1iF5+glgOXBTG7F+Dfy+7m+g2TNxGsW6Mr/+84HvANu2Gqvf40sZ3lk9jXL7OrAg53YtMLGNWBsAl+Tnei+wZ7uf32Zu7rLBzKwwbuoxMyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPAXLPdUeGbd/EmSTu1Q7IskHdqJWEPs541KPXveXvW+qiRparO9PPbb7vLcdcD7JX1c0t7D3H5po94qJZ083Fxa0erztva48JftCeD1w+mmthskjRnG6kcBR0fEa6rKp0umkvryaVq++vSVEfGSiDgrIj4aEd/rUD7DLvzDfN9qhv28rX0u/GVbSRrz8/39H+h/xC7p8Xy/R+5M6gpJv5I0S9JblcYPWCBpx7owe0v6YV7vdXn7MUr9tt+Tj1TfWxf3dkmXkS6O6Z/P4Tn+Qkln5GUfJV08d46kz/Rbfz1J/5v7OL9O0g2151N/lCtpmqQ78vQmudOte3LHcQfl5S/Mz29uznlKXvd6pTETFko6LK/7svz6zJF0U13XCsdL+nne/hv9ct0A+DhwWN7HYUp97l+T1/+ppJc0eP9uBibkbV5d/57l53iapHvz6/a8vHxrpf7tfybpK6QuwPu/1rNIvZPOlXRpXnZNfk6LVNc5n6TH838adwG7SdpfqU/+OyWdrTz+Q6PXttHzbvAcrQrduErMt7XzBjwObEa6mnFz4CTg1PzYRcCh9evm+z2Av5DGINgQeBA4LT92AvD5uu1vJB1cTCFdqbgRMAP4SF5nQ9KVu5Nz3P8DJjfI81mkrhR6SJ1a3QYcnB+7gwb9yAOHAjfk/T8T+HPt+VB39SYwDbgjT/83cESe3oI0rsImwBdJfc9AutJyHKnvmPPq9rc5qfvsHwM9edlhwAV5+g+s7nN9jXEHgHcA/1M3/0XgY3l6T2Bug216eXrf7v98z/JzPC5PHw18NU+fTb4SnNRVd9DgStba+103v1W+H0e6ynTrPB/Am/L0RqQrdyfn+cvJV4oP8to+7Xn71p2bj/gLF6kX0a+RBrto1j2RxiB4AvgN6cgT0pF6b916V0TEqohYAvwWeB6pf5+3K3U5fRep+4Favyx3R8TvGuzv5aTi3Bepv/dLSX35D+ZVwDfz/h8CmvkN4LXAzJzbHaRCtj3wE+BkSR8iDfDyt/xc95Z0hqRXR8QjpAFNXgTckmN8hNV90s8HLlXq6bOZHjpfReoagIi4Ddha0uZNbFev1lngHFa/L7uTugggIq4nfSE243hJ80j97E9i9Xv2FKl7BUjv72/r3sP6fokGem1tBIwd6QRsrfB5Uj8hF9YtW0luCpQk0pFuzRN106vq5lfx9M9U//5AgtS0cFxE3FT/gKQ9SEf8jbTSTe1g2/zzuZEKUP02b4g1ByZZnJsyDgBukvTuiLhN0stI7dOfknQzqbfSRRHRaAjJA0hF90DgvyS9MFYPWtJs/sPtX6X2vjzF4O/LoPJ7szewW0T8NTeN1V63v0fEU7VVBwtDg9dW0iuGk4t1ho/4jYj4E2mIwaPqFi8FXpanDyI1YwzXG3Nb+46kDt5+SRr+7n1K3Vgj6TkaegCWu4B/lTRe6QfEw0mjWQ3mTuANef/bkJqSapay+rnVd/d7E3Bc/qJD0k75fgfSkezZpE65XqI0BuxfI+IS0sAtO+fn16M8drCk9fPvA+sBkyLidtKAOFsAm/bL9zHSsJ01PyANQFIrvA9Hm2M8NIi7H6kv+EaerL1HpGasP+ei/zxS3/GN/ALYQWkwIkhNXTUNX1vWfN7WBS78VnMmT+8H/DxSsb0beAUDH40P5pekAv1d4N8j4u+k4R9/TuoTfSHwFYb4zzPSKFMfJjXXzCP1uT9UV8RXkn5XqO3jLuCR/NhpwBck/ZB0NFxzOukLbn7O7fS8/DBgYW6meB6paezFwN152SnAJyLiH6TfFs7IzSJzSX35jwEukbSA1OvqWbHmMJO3Ay+o+5HzVGCapPnALFZ3w9yu00ijgt1Lan65f4D1ziW9DpeSfqsZm3M5ndTcs4bcBHY0cKOkO0m9fdZe84Fe2/7P27rAvXPaOkvSphHxuKStgbtJo4e13A+7Da3uNRfwJWBJRJw10nnZ07mN39Zl1ykNaLMBcLqLfle8R2kUrg1I/918ZYTzsQZ8xG9mVhi38ZuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWH+HxHHGJZCy8f/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show(play_jottos(random_guesser))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The average is a bit more than 7 guesses; 2/3 of the time it will take from 6 to 9 guesses, and the worst case is 18 guesses. Can we improve the average and/or worst case?\n",
    "\n",
    "#  Guessers that Partition Targets\n",
    "\n",
    "A key idea in guessing is to reduce the number of consistent targets. We can think of a guess as **partitioning** the consistent targets into different **branches** of a tree, each branch corresponding to a different reply:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def partition(guess, targets) ->  Dict[Reply, List[str]]:\n",
    "    \"\"\"A partition of targets by the possible replies to guess: {reply: [word, ...]}.\"\"\"\n",
    "    branches = defaultdict(list)\n",
    "    for target in targets:\n",
    "        branches[reply_for(target, guess)].append(target)\n",
    "    return branches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It would take too much space to show a partition of all 2,845 words, so here's a partition of just the first 22 words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(list,\n",
       "            {4: ['their', 'might'],\n",
       "             1: ['about', 'sword', 'resay', 'nuder', 'house'],\n",
       "             0: ['would', 'cloud', 'place', 'sound', 'fondu'],\n",
       "             3: ['throe', 'write', 'rifts', 'think', 'grate'],\n",
       "             2: ['water', 'after', 'ethos', 'while'],\n",
       "             5: ['girth']})"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "partition('girth', wordlist[:22])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that after guesssing `'girth'` we will be left with no more than five consistent targets, no matter what the reply.\n",
    "\n",
    "To decide which guess is best, we don't need to know the identity of all the words in each branch, just the *count* of how many there are. The function `partition_counts` computes that,  returning a list of counts, one for each possible reply:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def partition_counts(guess, targets) -> List[int]: \n",
    "    \"A partition: {reply: number_of_targets_with_reply, ...}.\"\n",
    "    counts = Counter(reply_for(guess, target) for target in targets)\n",
    "    return [counts[i] for i in all_possible_replies]\n",
    "\n",
    "all_possible_replies = range(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are the partition counts for two possible first guesses, `ouija` and `coder`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[175, 1848, 755, 65, 1, 1]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "partition_counts('ouija', wordlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that for the guess `'ouija'`,  1848 consistent targets (about 2/3 of the word list) are bunched into one branch, meaning that 2/3 of the time you will be left with 1848 targets after guessing `'ouija'`. (*Note:* Why is this so? Because `'ouija'` has all the vowels except `'e'`, and about 2/3 of the words have exactly one of these vowels, so the reply for those words is 1. Only one other word, `'audio'`, has these four vowels. Only 56 words have a `'j'`, so that is not a major factor.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[433, 1030, 1014, 327, 40, 1]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "partition_counts('coder', wordlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the guess `'coder'`, the targets are more spread out over multiple branches. After guessing `'coder'` you will never be left with more than 1030 consistent targets. That suggests that `'coder'` is a better first guess, and it suggests a general strategy: **guess the word that best partitions the consistent targets into small branches.**\n",
    "\n",
    "What's a good metric for measuring how **small** the branches are? What we really want to know is how many additional guesses it will take to handle each branch, but since we don't know that, we can use one of the following proxy metrics:\n",
    "\n",
    "- A simple metric is to **minimize the maximum number in the partition counts**. That is, we should deduce that `coder`, with its `max` partition count of 1030, is a better guess than `ouija`, with its `max` of 1848. \n",
    "\n",
    "- A more sophisticated metric is to **minimize the expected value of the partition counts**: in probability theory the expected value or **expectation** is the weighted average of a random variable. Here it means the sum, over all branches, of the size of the branch multiplied by the probability of ending up in the branch.\n",
    "\n",
    "- Information theory provides a suggestion to **maximize the entropy in the partition counts**. Entropy is similar to expectation, except that it weights each branch size by its base 2 logarithm (whereas expectation weights it by its actual size). Note that we want to minimize *negative* entropy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def expectation(counts) -> float:\n",
    "    \"Given a list of counts, give the expected count.\"\n",
    "    def P(x, scale=1/sum(counts)): return scale * x\n",
    "    return sum(P(x) * x for x in counts)\n",
    "\n",
    "def entropy(counts) -> float: \n",
    "    \"\"\"Information theoretic entropy of a list of counts.\"\"\"\n",
    "    def P(x, scale=1/sum(counts)): return scale * x\n",
    "    return - sum(P(x) * log(P(x), 2) for x in counts if x)\n",
    "\n",
    "def negative_entropy(counts) -> float: return - entropy(counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the top 10 guesses according to each of the three metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def top(metric, n=10, words=wordlist) -> str: \n",
    "    \"\"\"The top n words according to metric(partition_counts(w, words)).\"\"\"\n",
    "    return ' '.join(sorted(words, key=lambda w: metric(partition_counts(w, words)))[:n])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'wader cadre armed diner coder padre rayed raved delta drone'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top(max) # minimize the maximum number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'raved debar roved orbed wader armed fader dater alder cadre'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top(expectation) # minimize the expectation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'debar alder raved dater cadre armed garde wader lased padre'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top(negative_entropy) # maximize the entropy (by minimizing negative entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three metrics yield a lot of overlap; `wader`, `raved`, `armed` and `cadre` appear in the top 10 for all three. Every word has both a `'d'` and an `'e'`, and 28 out of 30 have an `'r'`. Which metric is best? We have the tools to answer that: we could use `play_jottos` to get scores for guessers that use each metric. But that would take a long time. It takes about 5 seconds to compute the `top` word for just the first guess; I don't want to repeat that computation 3 × 2,845 times (and that's just for choosing the first word)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Guesser Trees \n",
    "\n",
    "I can speed up computation by precomputing a **guesser tree**: a tree that keeps partitioning target words into branches until every branch holds a single word (we call that a leaf). Think of the tree as a universal strategy that we can  apply to every target word with no need for additional computation&mdash;just branch-following in the tree.  I define a guesser tree as either:\n",
    "- A **leaf**, which is a string, such as `'coder'`, indicating that this is the sole remaining consistent target. Every word in the word list should appear as a leaf in exactly one place in the guesser tree.\n",
    "- An **interior node**, which is a tuple of a guess and a dict of branches, `(guess, {reply: subtree, ...})`, where the reply is an integer, 0 to 5, and each subtree covers all the target words that are consistent with that reply."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "Leaf = str\n",
    "Node = Tuple[str, Dict[Reply, 'Tree']]\n",
    "Tree = Union[Leaf, Node]    # A Tree is a Node or a Leaf\n",
    "GUESS, BRANCHES = 0, 1      # Indexes into a Node tuple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `make_tree(metric, targets)` builds a tree that covers all the targets and that, at every node, guesses a word that  minimizes the `metric` applied to the `partition_counts` of the guess."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_tree(metric, targets=wordlist) -> Tree:\n",
    "    \"\"\"Make a tree that guesses to minimize metric(partition_counts(guess, targets)).\"\"\"\n",
    "    if len(targets) == 1:\n",
    "        return targets[0]\n",
    "    else:\n",
    "        guess = min(targets, key=lambda guess: metric(partition_counts(guess, targets))) \n",
    "        branches = partition(guess, targets)\n",
    "        return (guess, {reply: make_tree(metric, branches[reply]) \n",
    "                        for reply in sorted(branches)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a tree that covers five words by always making the guess that minimizes the **maximum** number in the partition counts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('bites',\n",
       " {1: ('purge', {1: ('sulky', {1: 'hayed', 5: 'sulky'}), 5: 'purge'}),\n",
       "  2: 'patsy',\n",
       "  5: 'bites'})"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words5 = ['purge', 'bites', 'sulky', 'patsy', 'hayed']\n",
    "tree5  = make_tree(max, words5)\n",
    "tree5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tree says that the first guess is `'bites'`, and if the reply is `1` there is a complex subtree to consider (starting with the guess `'purge'`), but if the reply is `2` the target can only be `'patsy'` and of course if the reply is `5` then `'bites'` was the target."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Turning a Tree into a Callable Guesser\n",
    "\n",
    "Now that we've made a tree, we need to use it as a component of a guesser that `play_jotto` can call upon. A `TreeGuesser` is an object that stores a tree in the `self.root` and `self.tree` attributes, and while the game is being played, it follows branches in the tree, choosing the guess indicated at each node. This is done within the `__call__` method, because `play_jotto` expects a callable.  Note that on the first turn of a new game the reply will be `None`, in which case we reset `self.tree` to be the root tree. For subsequent turns, we follow the branch indicated by the reply. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TreeGuesser:\n",
    "    \"\"\"Given a guesser tree, use it to create a callable that can play_jotto.\"\"\"\n",
    "    def __init__(self, tree): self.root = self.tree = tree\n",
    "        \n",
    "    def __call__(self, reply, _) -> str:\n",
    "        self.tree = self.root if reply is None else self.tree[BRANCHES][reply]\n",
    "        return self.tree if isinstance(self.tree, str) else self.tree[GUESS]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's how we play a game with the five-word list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Guess 1: bites, Reply: 1, Consistent targets: 3\n",
      "Guess 2: purge, Reply: 1, Consistent targets: 2\n",
      "Guess 3: sulky, Reply: 1, Consistent targets: 1\n",
      "Guess 4: hayed, Reply: 5, Consistent targets: 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "play_jotto(TreeGuesser(tree5), wordlist=words5, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here we build a tree over the whole word list and use it to play a game:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Guess 1: wader, Reply: 3, Consistent targets: 319\n",
      "Guess 2: sword, Reply: 1, Consistent targets: 131\n",
      "Guess 3: paled, Reply: 3, Consistent targets: 56\n",
      "Guess 4: cadet, Reply: 4, Consistent targets: 15\n",
      "Guess 5: hated, Reply: 5, Consistent targets: 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree    = make_tree(max)\n",
    "guesser = TreeGuesser(tree)\n",
    "\n",
    "play_jotto(guesser, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making Inconsistent Guesses\n",
    "\n",
    "So far,  we have always guessed one of the consistent targets. That seems reasonable; why waste a guess on a word that could not possibly be the target? But it turns out that in some cases it *is* a good strategy to guess such a word.\n",
    "\n",
    "I will redefine `make_tree` so that it is passed both the list of consistent target words and the complete word list. It also has a flag, `inconsistent`. When this flag is true, any word in the wordlist can be considered as a guess; when false, only consistent targets are considered, as before. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_tree(metric, targets=wordlist, wordlist=wordlist, inconsistent=False) -> Tree:\n",
    "    \"\"\"Make a tree that guesses to minimize metric(partition_counts(guess, targets)).\"\"\"\n",
    "    if len(targets) == 1:\n",
    "        return targets[0]\n",
    "    else:\n",
    "        words = wordlist if (inconsistent and len(targets) > 3) else targets\n",
    "        guess = min(words, key=lambda guess: metric(partition_counts(guess, targets))) \n",
    "        branches = partition(guess, targets)\n",
    "        return (guess, {reply: make_tree(metric, branches[reply], wordlist, inconsistent) \n",
    "                        for reply in sorted(branches)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see that by default, `make_tree` behaves just as it did before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('bites',\n",
       " {1: ('purge', {1: ('sulky', {1: 'hayed', 5: 'sulky'}), 5: 'purge'}),\n",
       "  2: 'patsy',\n",
       "  5: 'bites'})"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "make_tree(max, words5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But when allowed `make_tree` to guess an inconsistent word, it comes up with a different tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('dashy', {0: 'purge', 1: 'bites', 2: 'sulky', 3: 'patsy', 4: 'hayed'})"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree5b = make_tree(max, words5, inconsistent=True)\n",
    "tree5b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tree guesses an inconsistent word, `dashy`, with the first guess. There is no chance that this is the target, but it sets us up so that we will always be able to get the target on the second guess. That means that both the average and the worst case number of guesses is 2 for `tree5b`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 2, 2, 2, 2]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "play_jottos(TreeGuesser(tree5b), words5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 2)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean(_), max(_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In contrast, `tree5`  has an average score of 2.4, and can take up to four guesses in the worst case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 1, 3, 2, 4]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "play_jottos(TreeGuesser(tree5), words5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2.4, 4)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean(_), max(_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the tree with an inconsistent guess is better both in the average  and in the worst case.\n",
    "\n",
    "# Comparing Metrics\n",
    "\n",
    "Now we will compare six guessers derived from `make_tree`:\n",
    "- Minimizing either `max`, `expectation`, or `negative_entropy`.\n",
    "- For each of the above, guessing either consistent targets only or allowing inconsistent targets.\n",
    "\n",
    "The function `show_metric` makes the appropriate `TreeGuesser` and calls `play_jottos` and `show` to display results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_metric(metric, inconsistent):\n",
    "    \"\"\"Show statistics and histogram for a guesser that minimizes `metric` over partition counts.\"\"\"\n",
    "    tree = make_tree(metric, inconsistent=inconsistent)\n",
    "    guesser = TreeGuesser(tree)\n",
    "    show(play_jottos(guesser))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing Metrics with Consistent Guesses Only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean = 7.15 ± 1.81 guesses; worst = 18\n",
      "CPU times: user 12.6 s, sys: 29.5 ms, total: 12.6 s\n",
      "Wall time: 12.6 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAaXklEQVR4nO3debxdVX338c/XJEAYA+SCEdLnBhqnUg14G6FRigyW6WGoKFLxiRWNjzIIlqeNYBWKtqGKCNaqQSbLJMogAjKUQaVq4AYzYaRUjBgIyUVlqook+T1/rHXN4eYO5+x797lJ1vf9ep3X2Xufvdb+neH+zrrr7L2WIgIzMyvHy0Y7ADMzay8nfjOzwjjxm5kVxonfzKwwTvxmZoUZO9oBNGPixInR2dk52mGYmW1U5s+f/1REdPTdvlEk/s7OTrq7u0c7DDOzjYqkn/e33V09ZmaFceI3MyuME7+ZWWGc+M3MCuPEb2ZWGCd+M7PCOPGbmRXGid/MrDBO/GZmhdkorty1TU/n7Fsql10257ARjMSsPG7xm5kVxonfzKwwTvxmZoVx4jczK4wTv5lZYZz4zcwK48RvZlYYJ34zs8I48ZuZFcaJ38ysME78ZmaFceI3MyuME7+ZWWFqS/yStpB0v6SFkh6SdHbePkXSPEmPSPqapM3qisHMzNZXZ4v/BWD/iHg9MA04WNLewLnA+RExFfg1cEKNMZiZWR+1Jf5Ins+r4/ItgP2Bb+TtlwNH1RWDmZmtr9Y+fkljJC0AVgF3Aj8Fno6I1XmX5cAuA5SdJalbUndPT0+dYZqZFaXWxB8RayJiGrArMB14TX+7DVB2bkR0RURXR0dHnWGamRWlLWf1RMTTwL3A3sAESb1TPu4KPNGOGMzMLKnzrJ4OSRPy8njgQGApcA9wTN5tJvDNumIwM7P11TnZ+iTgckljSF8w10bEzZJ+DFwj6ZPAj4CLa4zBzMz6qC3xR8QiYM9+tj9K6u83M7NR4Ct3zcwK48RvZlYYJ34zs8I48ZuZFcaJ38ysME78ZmaFceI3MyuME7+ZWWGc+M3MCuPEb2ZWGCd+M7PCOPGbmRXGid/MrDBO/GZmhXHiNzMrjBO/mVlhnPjNzArjxG9mVhgnfjOzwjjxm5kVxonfzKwwTvxmZoWpLfFLmizpHklLJT0k6cN5+1mSHpe0IN8OrSsGMzNb39ga614N/G1EPChpG2C+pDvzY+dHxGdqPLaZmQ2gtsQfESuAFXn5OUlLgV3qOp6ZmTWnLX38kjqBPYF5edNJkhZJukTS9gOUmSWpW1J3T09PO8I0MytC7Ylf0tbAdcCpEfEs8EVgd2Aa6T+C8/orFxFzI6IrIro6OjrqDtPMrBi1Jn5J40hJ/8qIuB4gIlZGxJqIWAtcBEyvMwYzM3upOs/qEXAxsDQiPtuwfVLDbkcDS+qKwczM1lfnWT0zgHcDiyUtyNvOAI6TNA0IYBnwgRpjMDOzPuo8q+c+QP08dGtdxzQzs6H5yl0zs8I48ZuZFcaJ38ysME78ZmaFceI3MyuME7+ZWWGc+M3MCuPEb2ZWGCd+M7PCOPGbmRXGid/MrDBO/GZmhXHiNzMrjBO/mVlh6hyP3zZRnbNvqVx22ZzDRjASM6tiyBa/pN0lbZ6X95N0iqQJ9YdmZmZ1aKar5zpgjaQ/Jk2lOAW4qtaozMysNs0k/rURsZo0P+7nIuI0YNIQZczMbAPVTOJ/UdJxwEzg5rxtXH0hmZlZnZpJ/H8D7AN8KiJ+JmkKcEW9YZmZWV2aOavnoIg4pXclJ//f1hiTmZnVqJkW/8x+tr1nhOMwM7M2GbDFn/v1/xqYIummhoe2AX5Zd2BmZlaPwbp6vg+sACYC5zVsfw5YNFTFkiYDXwVeDqwF5kbEBZJ2AL4GdALLgHdExK+rBG9mZq0bsKsnIn4eEfdGxD6kBD0uIr4DLAXGN1H3auBvI+I1wN7AiZJeC8wG7oqIqcBded3MzNqkmSt33w98A/hy3rQrcONQ5SJiRUQ8mJefI31h7AIcCVyed7scOKr1sM3MrKpmftw9EZgBPAsQEY8AO7VyEEmdwJ7APGDniFiR61oxUF2SZknqltTd09PTyuHMzGwQzST+FyLi970rksYC0ewBJG1NGvbh1Ih4ttlyETE3Iroioqujo6PZYmZmNoRmEv93JJ0BjJd0EPB14FvNVC5pHCnpXxkR1+fNKyVNyo9PAla1HraZmVXVTOKfDfQAi4EPALcCHxuqkCSRBnVbGhGfbXjoJtZdGzAT+GYrAZuZ2fAMeeVuRKwFLsq3VswA3g0slrQgbzsDmANcK+kE4DHg7S3Wa2ZmwzBk4pe0mPX79J8BuoFPRkS/F3NFxH2ABqj2gFaCNDOzkdPMWD3fBtawbgz+d+b7Z4HLgP898mGZmVldmkn8MyJiRsP6Ykn/GREzJB1fV2BmZlaPZhL/1pLeGBHzACRNB7bOj62uLTKzIXjuX7Nqmkn8JwCX5vPxIY3Vc4KkrYB/ri0yMzOrxaCJX9LLgN0i4k8lbQcoIp5u2OXaWqMzM7MRN+h5/PlUzpPy8jN9kr6ZmW2EmrmA605Jp0uaLGmH3lvtkZmZWS2a6eN/b74/sWFbALuNfDhmZla3Zq7cndKOQMzMrD2aafEjaQ/gtcAWvdsi4qt1BWVmZvVpZsiGTwD7kRL/rcAhwH2kaRXNzGwj08yPu8eQxtZ5MiL+Bng9sHmtUZmZWW2aSfy/zad1rpa0LWn8fP+wa2a2kWqmj79b0gTSsMzzgeeB+2uNyszMatPMWT0fyotfknQbsG1ELKo3LDMzq8uQXT2S7updjohlEbGocZuZmW1cBmzxS9oC2BKYKGl71k2qsi3wijbEZmZmNRisq+cDwKmkJD+fdYn/WeALNcdlZmY1GTDxR8QFwAWSTo6Iz7cxJjMzq9GQffxO+mZmm5ZmzuM3M7NNyICJX9KMfO+rdM3MNiGDtfgvzPc/aEcgZmbWHoOd1fOipEuBXSRd2PfBiDhlsIolXQIcDqyKiD3ytrOA9wM9ebczIuLWKoGbmVk1gyX+w4EDgf1Jp3O26jLgX1l/FM/zI+IzFeozM7MRMNjpnE8B10haGhELW604Ir4rqXMYsZmZWQ2aOavnl5JukLRK0kpJ10nadRjHPEnSIkmX5CuC+yVplqRuSd09PT0D7WZmZi1qJvFfCtxEuoJ3F+BbeVsVXwR2B6YBK4DzBtoxIuZGRFdEdHV0dFQ8nJmZ9dVM4t8pIi6NiNX5dhlQKRNHxMqIWJPH978ImF6lHjMzq66ZxN8j6XhJY/LteOCXVQ4maVLD6tHAkir1mJlZdc1MxPJe0tk55wMBfD9vG5Skq0lz9U6UtBz4BLCfpGm5nmWkgeDMzKyNmpmI5THgiFYrjojj+tl8cav1mJnZyPJYPWZmhXHiNzMrjBO/mVlhmk78kvaWdLek/5R0VJ1BmZlZfQabc/flEfFkw6aPkH7kFenMnhtrjs3MzGow2Fk9X5I0H/h0RPwOeBr4a2Atad5dMzPbCA3Y1RMRRwELgJslvZs08fpaYEvAXT1mZhupQfv4I+JbwF8CE4DrgYcj4sKI8KhpZmYbqcGmXjxC0n3A3aShFd4JHC3pakm7tytAMzMbWYP18X8S2AcYD9waEdOBj0iaCnyK9EVgZmYbmcES/zOk5D4eWNW7MSIewUnfzGyjNVgf/9GkH3JXk87mMTOzTcBQUy9+vo2xmJlZG3jIBjOzwjjxm5kVxonfzKwwTvxmZoVx4jczK4wTv5lZYZz4zcwK48RvZlYYJ34zs8LUlvglXSJplaQlDdt2kHSnpEfy/fZ1Hd/MzPpXZ4v/MuDgPttmA3dFxFTgrrxuZmZtVFvij4jvAr/qs/lI4PK8fDmeycvMrO3a3ce/c0SsAMj3Ow20o6RZkroldff0eMIvM7ORssH+uBsRcyOiKyK6Ojo6RjscM7NNRrsT/0pJkwDy/aoh9jczsxHW7sR/EzAzL88Evtnm45uZFa/O0zmvBn4AvErSckknAHOAgyQ9AhyU183MrI0Gm3N3WCLiuAEeOqCuY5qZ2dA22B93zcysHrW1+G3D1Dn7lspll805bAQjMbPR4ha/mVlhnPjNzArjxG9mVhgnfjOzwjjxm5kVxonfzKwwTvxmZoVx4jczK4wTv5lZYZz4zcwK48RvZlYYJ34zs8I48ZuZFcaJ38ysME78ZmaFceI3MyuME7+ZWWGc+M3MCuOpF61onorSSuQWv5lZYZz4zcwKMypdPZKWAc8Ba4DVEdE1GnGYmZVoNPv43xIRT43i8c3MiuSuHjOzwoxW4g/gDknzJc3qbwdJsyR1S+ru6elpc3hmZpuu0Ur8MyJiL+AQ4ERJ+/bdISLmRkRXRHR1dHS0P0Izs03UqCT+iHgi368CbgCmj0YcZmYlanvil7SVpG16l4G3AkvaHYeZWalG46yenYEbJPUe/6qIuG0U4jAzK1LbE39EPAq8vt3HNTOzxKdzmpkVxonfzKwwTvxmZoVx4jczK4wTv5lZYTwRi9kwDWcyF/CELtZ+bvGbmRXGid/MrDDu6tmIeH5YMxsJbvGbmRXGid/MrDBO/GZmhXHiNzMrjBO/mVlhfFaP2QbAF4FZO7nFb2ZWGCd+M7PCOPGbmRXGffxmmwj/TmDNcovfzKwwTvxmZoVxV0+beIA1M9tQOPGb2R/4d4IyjEril3QwcAEwBvhKRMwZjTjMbOT5y2PD1/bEL2kM8AXgIGA58ICkmyLix+2Oxcw2TP7yqNdotPinA/8dEY8CSLoGOBLYYBO/++fNNj4j8eWxodQx0hQRI17poAeUjgEOjoj35fV3A2+MiJP67DcLmJVXXwU8XFNIE4GnXIfrcB2uYyOuYyD/KyI6+m4cjRa/+tm23rdPRMwF5tYejNQdEV2uw3W4DtexsdbRqtE4j385MLlhfVfgiVGIw8ysSKOR+B8ApkqaImkz4J3ATaMQh5lZkdre1RMRqyWdBNxOOp3zkoh4qN1xNBiJ7iTX4Tpch+sYzTpa0vYfd83MbHR5rB4zs8I48ZuZFabYxC/pEkmrJC2pWH6ypHskLZX0kKQPV6hjC0n3S1qY6zi7Siy5rjGSfiTp5orll0laLGmBpO6KdUyQ9A1JP8mvyz4tln9VPn7v7VlJp1aI47T8ei6RdLWkLSrU8eFc/qFWYujvcyVpB0l3Snok329foY6351jWShry1L8B6vh0fm8WSbpB0oQKdZyTyy+QdIekV7RaR8Njp0sKSRMrxHGWpMcbPiuHVolD0smSHs6v7b9UiONrDTEsk7SgQh3TJP2w929P0vTB6hgREVHkDdgX2AtYUrH8JGCvvLwN8F/Aa1usQ8DWeXkcMA/Yu2I8HwGuAm6uWH4ZMHGYr+nlwPvy8mbAhGHUNQZ4knQBSivldgF+BozP69cC72mxjj2AJcCWpBMg/gOYWvVzBfwLMDsvzwbOrVDHa0gXMt4LdFWM463A2Lx8bsU4tm1YPgX4Uqt15O2TSSd4/Hyoz90AcZwFnN7Ce9pfHW/J7+3meX2nKs+l4fHzgI9XiOMO4JC8fChwbyuf1yq3Ylv8EfFd4FfDKL8iIh7My88BS0lJp5U6IiKez6vj8q3lX9sl7QocBnyl1bIjRdK2pA/1xQAR8fuIeHoYVR4A/DQifl6h7FhgvKSxpOTd6nUirwF+GBG/iYjVwHeAo5spOMDn6kjSlyL5/qhW64iIpRHR9NXrA9RxR34+AD8kXUPTah3PNqxuxRCf10H+zs4H/m6o8kPU0bQB6vggMCciXsj7rKoahyQB7wCurlBHANvm5e1ow3VNxSb+kSSpE9iT1GJvteyY/O/hKuDOiGi5DuBzpD+itRXK9grgDknzlYbLaNVuQA9wae5y+oqkrYYRzzsZ4o+oPxHxOPAZ4DFgBfBMRNzRYjVLgH0l7ShpS1IrbPIQZQazc0SsyPGtAHYaRl0j5b3At6sUlPQpSb8A3gV8vEL5I4DHI2JhleM3OCl3O10yVPfZAF4JvFnSPEnfkfRnw4jlzcDKiHikQtlTgU/n1/QzwEeHEUdTnPiHSdLWwHXAqX1aQ02JiDURMY3U+pouaY8Wj384sCoi5rd67D5mRMRewCHAiZL2bbH8WNK/sF+MiD2B/yF1a7RM6cK+I4CvVyi7PamFPQV4BbCVpONbqSMilpK6Qu4EbgMWAqsHLbQRkXQm6flcWaV8RJwZEZNz+ZOG2r/PsbcEzqTCF0YfXwR2B6aRvuDPq1DHWGB7YG/g/wHX5pZ7FcdRoaGSfRA4Lb+mp5H/a66TE/8wSBpHSvpXRsT1w6krd4vcCxzcYtEZwBGSlgHXAPtLuqLC8Z/I96uAG0ijqLZiObC84T+Wb5C+CKo4BHgwIlZWKHsg8LOI6ImIF4HrgT9vtZKIuDgi9oqIfUn/mldpyfVaKWkSQL4ftEuhTpJmAocD74rcqTwMVwFva7HM7qQv5YX5M7sr8KCkl7dSSUSszI2mtcBFtP55hfSZvT53ud5P+o950B+a+5O7FP8K+FqFGABmkj6nkBo7tf+468RfUW4ZXAwsjYjPVqyjo/fMCknjSUnrJ63UEREfjYhdI6KT1D1yd0S01MKVtJWkbXqXST8CtnS2U0Q8CfxC0qvypgOoPtT2cFpPjwF7S9oyv0cHkH5/aYmknfL9H5H+qKvGA2lIkpl5eSbwzWHUVZnSBEh/DxwREb+pWMfUhtUjaP3zujgidoqIzvyZXU46SeLJFuOY1LB6NC1+XrMbgf1zfa8knZBQZZTMA4GfRMTyCmUh9en/RV7en+E1MppT96/HG+qN9Ie8AniR9OE7ocXybyL1iy8CFuTboS3W8TrgR7mOJQxxRkAT9e1HhbN6SP3zC/PtIeDMisefBnTn53MjsH2FOrYEfglsN4zX4WxSQloC/Dv5rI0W6/ge6YtrIXDAcD5XwI7AXaQ/6LuAHSrUcXRefgFYCdxeoY7/Bn7R8Hkd6oyc/uq4Lr+ui4BvAbu0Wkefx5cx9Fk9/cXx78DiHMdNwKQKdWwGXJGfz4PA/lWeC3AZ8H+H8fl4EzA/f9bmAW+o+tlv9uYhG8zMCuOuHjOzwjjxm5kVxonfzKwwTvxmZoVx4jczK4wTf8HyyIjnNayfLumsEar7MknHjERdQxzn7Uojgd5T97HqlEdoHHSEyQHKXZ2HLThN0j9KOrDF8sv6Gx1T0hmtxlJF1edtw+PEX7YXgL8aaljcdpM0poXdTwA+FBFvqSueNplGGhOoaflq1z+PiNdFxPkR8fGI+I8RiqflxN/i+9ar5edtw+fEX7bVpPk+T+v7QN8Wu6Tn8/1+eUCrayX9l6Q5kt6lNK/AYkm7N1RzoKTv5f0Oz+XHKI0L/0BuqX6god57JF1FujCnbzzH5fqXSDo3b/s46eKXL0n6dJ/9Xybp35TGWb9Z0q29z6exlSupS9K9eXmrPODXA3mguSPz9j/Jz29Bjnlq3vcWpbkUlkg6Nu/7hvz6zJd0e8NQDadI+nEuf02fWDcD/hE4Nh/jWKUx/G/M+/9Q0uv6ef/uAHbKZd7c+J7l53i2pAfz6/bqvH1HpXH0fyTpy6Shwfu+1nNIo5sukHRl3nZjfk4PqWEQP0nP5/805gH7SDpUacz/+yRdqDw/RH+vbX/Pu5/naHWo+wox3zbcG/A8aTjYZaThYE8HzsqPXQYc07hvvt8PeJo0H8HmwOPA2fmxDwOfayh/G6lxMZV0leIWwCzgY3mfzUlX+k7J9f4PMKWfOF9BGoqhgzSw1t3AUfmxe+lnfHrgGODWfPyXA7/ufT40XC0KdJHHPwf+CTg+L08gzbGwFfB50tg2kK72HE8ao+aihuNtRxpW+/tAR952LHBJXn6CdeO+rzdPAfAe4F8b1j8PfCIv7w8s6KdMJy8d1/0P71l+jifn5Q8BX8nLF5KvECcN5R30c+Vs7/vdsL5Dvh9PutJ1x7wewDvy8hakK4On5PWryVeSD/LavuR5+9aem1v8hYs0ouhXSZNqNOuBSPMRvAD8lNTyhNRS72zY79qIWBtpqNpHgVeTxgH6P0pDUc8jDWfQO/7L/RHxs36O92ek5NwTaTz5K0lj/w/mTcDX8/GfBJr5DeCtwOwc272kRPZHwA+AMyT9PWlimN/m53qgpHMlvTkiniFNlLIHcGeu42OsG/N+EXCl0kihzYz0+SbSsARExN3AjpK2a6Jco96Bv+az7n3ZlzRMARFxC+kLsRmnSFpIGsd/MuveszWkYRwgvb+PNryHjeMbDfTa2igYO9oB2Abhc6SxSi5t2Laa3BUoSaSWbq8XGpbXNqyv5aWfqb7jgQSpa+HkiLi98QFJ+5Fa/P2pMlTuYGX+8NxICaixzNti/QlPluaujMOA2yW9LyLulvQGUv/0P0u6gzSq6UMR0d+Uk4eRku4RwD9I+pNYNylKs/G3Or5K7/uyhsHfl0Hl9+ZAYJ+I+E3uGut93X4XEWt6dx2sGvp5bSW9sZVYbGS4xW9ExK9IUxSe0LB5GfCGvHwkqRujVW/Pfe27kwaCe5g03d4HlYa0RtIrNfSELfOAv5A0UekHxONIs2IN5j7gbfn4O5O6knotY91zaxxW+Hbg5PxFh6Q98/1upJbshaQBwV6nNNfsbyLiCtLkGXvl59ehPNewpHH594GXAZMj4h7ShDkTgK37xPscaQrPXt8lTXTSm3ifigrzPfSjsd5DSOPR9+fF3veI1I3165z0X00av74/PwF2U5qYCFJXV69+X1vWf97WBk781us8XjoW+UWkZHs/8EYGbo0P5mFSgv42afTC35Gmh/wxaQz2JcCXGeI/z0izVn2U1F2zkDRW/1BDG19H+l2h9xjzgGfyY2cDF0j6Hqk13Osc0hfcohzbOXn7scCS3E3xalLX2J8C9+dtZwKfjIjfk35bODd3iywgzQUwBrhC0mLSaKznx/rTUt4DvLbhR86zgC5Ji4A5rBvWebjOJs0u9iCp++WxAfabS3odriT9VjM2x3IOqbtnPbkL7EPAbZLuI40i2vuaD/Ta9n3e1gYendM2WZK2jojnJe0I3E+aZaylcd+tNQ2vuYAvAI9ExPmjHZe9lPv4bVN2s9JEN5sB5zjpt8X7lWb52oz0382XRzke64db/GZmhXEfv5lZYZz4zcwK48RvZlYYJ34zs8I48ZuZFeb/A8G/ffi5yxS8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(max, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean = 7.14 ± 1.82 guesses; worst = 17\n",
      "CPU times: user 13 s, sys: 59.7 ms, total: 13.1 s\n",
      "Wall time: 13.2 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAaC0lEQVR4nO3de7xcVX338c/XhEu4BsgBI5cm8OCtVgOmCI1S5OLDxYdLRZGqjRUbnyI3laem0CoUfRpqEcXHR+VOy00qFxEQSIGoVAVOIDeMNBUjAiE5qNyqIkl+/WOt0wwnc86Z2XP2nJys7/v1mtfsvWfvtX97Zs5v1ll777UUEZiZWTleMdoBmJlZdznxm5kVxonfzKwwTvxmZoVx4jczK8z40Q6gFZMmTYopU6aMdhhmZmPK/Pnzn46InoHLx0TinzJlCr29vaMdhpnZmCLpZ82Wu6nHzKwwTvxmZoVx4jczK4wTv5lZYZz4zcwK48RvZlYYJ34zs8I48ZuZFcaJ38ysMGPizl0rx5TZt3a0/fI5R4xQJGYbL9f4zcwK48RvZlYYJ34zs8I48ZuZFcaJ38ysME78ZmaFceI3MyuME7+ZWWGc+M3MCuPEb2ZWmNoSv6TNJd0vaaGkhyWdnZdPlXSfpGWSvi5p07piMDOz9dVZ438RODAi3gRMAw6VtC9wLnB+ROwJ/Ao4ocYYzMxsgNoSfyQv5NlN8iOAA4Fv5OVXAEfXFYOZma2v1jZ+SeMkLQBWAXOBnwDPRMTqvMrjwM6DbDtLUq+k3r6+vjrDNDMrSq2JPyLWRMQ0YBdgH+B1zVYbZNsLI2J6REzv6empM0wzs6J05aqeiHgGmAfsC0yU1D8OwC7Ak92IwczMkjqv6umRNDFPTwAOBpYC9wDH5tVmAt+sKwYzM1tfnSNwTQaukDSO9ANzXUTcIulHwLWSPgM8BFxSYwxmZjZAbYk/IhYBezVZ/iipvd/MzEaB79w1MyuME7+ZWWGc+M3MCuPEb2ZWGCd+M7PCOPGbmRXGid/MrDBO/GZmhXHiNzMrjBO/mVlhnPjNzArjxG9mVhgnfjOzwjjxm5kVxonfzKwwTvxmZoVx4jczK4wTv5lZYZz4zcwK48RvZlYYJ34zs8I48ZuZFcaJ38ysMLUlfkm7SrpH0lJJD0s6NS8/S9ITkhbkx+F1xWBmZusbX2PZq4FPRMSDkrYG5kuam187PyL+scZ9m5nZIGpL/BGxAliRp5+XtBTYua79mZlZa7rSxi9pCrAXcF9edJKkRZIulbTdINvMktQrqbevr68bYZqZFaH2xC9pK+B64LSIeA74CrAHMI30H8F5zbaLiAsjYnpETO/p6ak7TDOzYtSa+CVtQkr6V0XEDQARsTIi1kTEWuAiYJ86YzAzs5er86oeAZcASyPi8w3LJzesdgywpK4YzMxsfXVe1TMD+ACwWNKCvOwM4HhJ04AAlgMfqTEGMzMboM6reu4F1OSl2+rap5mZDc937pqZFcaJ38ysME78ZmaFceI3MyuME7+ZWWGc+M3MCuPEb2ZWGCd+M7PC1HnnrhVkyuxbO9p++ZwjRigSMxuOa/xmZoVx4jczK4wTv5lZYZz4zcwK48RvZlYYJ34zs8I48ZuZFcaJ38ysMMMmfkl7SNosTx8g6RRJE+sPzczM6tBKjf96YI2k/0EaPH0qcHWtUZmZWW1aSfxrI2I1cAzwhYj4GDC53rDMzKwurST+lyQdD8wEbsnLNqkvJDMzq1Mrif/Pgf2Az0bETyVNBa6sNywzM6tLK71zHhIRp/TP5OT/mxpjMjOzGrVS45/ZZNkHh9tI0q6S7pG0VNLDkk7Ny7eXNFfSsvy8XZsxm5lZBwat8ed2/T8Fpkq6ueGlrYFftFD2auATEfGgpK2B+ZLmkn407oqIOZJmA7OBT1Y9ADMza89QTT3fB1YAk4DzGpY/DywaruCIWJG3JyKel7QU2Bk4Cjggr3YFMA8nfjOzrhm0qScifhYR8yJiP2A5sElEfAdYCkxoZyeSpgB7AfcBO+Ufhf4fhx0H2WaWpF5JvX19fe3szszMhtDKnbt/AXwD+FpetAtwU6s7kLQV6Saw0yLiuVa3i4gLI2J6REzv6elpdTMzMxtGKyd3PwrMAJ4DiIhlDFJLH0jSJqSkf1VE3JAXr5Q0Ob8+GVjVbtBmZlZdK4n/xYj4Xf+MpPFADLeRJJG6eFgaEZ9veOlm1l0pNBP4ZuvhmplZp1q5jv87ks4AJkg6BDgR+FYL280APgAslrQgLzsDmANcJ+kE4DHg3e2HbWZmVbWS+GcDJwCLgY8AtwEXD7dRRNwLaJCXD2o1QDMzG1nDJv6IWAtclB9mZjbGDZv4JS1m/Tb9Z4Fe4DMR0crNXGZmtoFopann28Aa1vXB/978/BxwOfC/Rj4sMzOrSyuJf0ZEzGiYXyzp3yJihqT31xWYmZnVo5XLObeS9Jb+GUn7AFvl2dW1RGVmZrVppcZ/AnBZvgMXUl89J0jaEvj72iIzM7NaDJn4Jb0C2D0i/kDStoAi4pmGVa6rNTozMxtxQzb15Es5T8rTzw5I+mZmNga10sY/V9LpeWCV7fsftUdmZma1aKWN/0P5+aMNywLYfeTDMTOzurVy5+7UbgRiZmbd0UqNH0lvAF4PbN6/LCL+qa6gzMysPq102fBp0lCJryd10HYYcC/gxG9mNga1UuM/FngT8FBE/LmknWihd06z0TRl9q0dbb98zhEjFInZhqeVq3p+ky/rXC1pG9KIWT6xa2Y2RrVS4++VNJHULfN84AXg/lqjMjOz2rRyVc+JefKrkm4HtomIRfWGZWZmdRm2qUfSXf3TEbE8IhY1LjMzs7Fl0Bq/pM2BLYBJkrZj3TCK2wCv6kJsZmZWg6Gaej4CnEZK8vNZl/ifA75cc1xmZlaTQRN/RHwR+KKkkyPiS12MyczMajRsG7+TvpnZxqWV6/jNzGwjMmjilzQjP29WpWBJl0paJWlJw7KzJD0haUF+HF6lbDMzq26oGv8F+fkHFcu+HDi0yfLzI2JaftxWsWwzM6toqKt6XpJ0GbCzpAsGvhgRpwxVcER8V9KUzsIzM7ORNlTifydwMHAg6XLOkXKSpD8DeoFPRMSvRrBsMzMbxlCXcz4NXCtpaUQsHKH9fQU4hzSC1znAeawb4etlJM0CZgHstttuI7R7MzNr5aqeX0i6MZ+oXSnpekm7VNlZRKyMiDW5t8+LgH2GWPfCiJgeEdN7enqq7M7MzJpoJfFfBtxMuoN3Z+BbeVnbJE1umD0GWDLYumZmVo9WumXeMSIaE/3lkk4bbiNJ15BG7pok6XHg08ABkqaRmnqWk7qFMDOzLmol8fdJej9wTZ4/HvjFcBtFxPFNFl/SRmxmZlaDVpp6PgS8B3gKWEEairHpCVkzM9vwtTIQy2PAkV2IxczMusB99ZiZFcaJ38ysME78ZmaFaTnxS9pX0t2S/k3S0XUGZWZm9RlqzN1XRsRTDYs+TjrJK+D7wE01x2ZmZjUY6qqer0qaD3wuIn4LPAP8KbCWNO6umZmNQYM29UTE0cAC4BZJHyANvL4W2AJwU4+Z2Rg1ZBt/RHwL+J/AROAG4JGIuCAi+roRnJmZjbyhhl48UtK9wN2kztTeCxwj6RpJe3QrQDMzG1lDtfF/BtgPmADcFhH7AB+XtCfwWdIPgZmZjTFDJf5nScl9ArCqf2FELMNJ38xszBqqjf8Y0onc1aSreczMbCMw3NCLX+piLGZm1gXussHMrDBO/GZmhXHiNzMrjBO/mVlhnPjNzArjxG9mVhgnfjOzwjjxm5kVprbEL+lSSaskLWlYtr2kuZKW5eft6tq/mZk1V2eN/3Lg0AHLZgN3RcSewF153szMuqi2xB8R3wV+OWDxUcAVefoKPKCLmVnXdbuNf6eIWAGQn3fs8v7NzIq3wZ7clTRLUq+k3r4+D/hlZjZSup34V0qaDJCfVw22YkRcGBHTI2J6T09P1wI0M9vYdTvx3wzMzNMzgW92ef9mZsWr83LOa4AfAK+R9LikE4A5wCGSlgGH5HkzM+uioYZe7EhEHD/ISwfVtU8zMxveBnty18zM6lFbjd/Ghimzb+1o++VzjhihSMysW1zjNzMrjBO/mVlhnPjNzArjxG9mVhgnfjOzwjjxm5kVxonfzKwwTvxmZoVx4jczK4wTv5lZYZz4zcwK48RvZlYYJ34zs8I48ZuZFcaJ38ysME78ZmaF8UAsZkPwQDW2MXKN38ysME78ZmaFceI3MyuME7+ZWWGc+M3MCjMqV/VIWg48D6wBVkfE9NGIw8ysRKN5OefbI+LpUdy/mVmR3NRjZlaY0Ur8Adwpab6kWc1WkDRLUq+k3r6+vi6HZ2a28RqtxD8jIvYGDgM+Kmn/gStExIURMT0ipvf09HQ/QjOzjdSoJP6IeDI/rwJuBPYZjTjMzErU9cQvaUtJW/dPA+8AlnQ7DjOzUo3GVT07ATdK6t//1RFx+yjEYWZWpK4n/oh4FHhTt/drZmaJL+c0MyuME7+ZWWGc+M3MCuMRuMy6wCN52YbENX4zs8I48ZuZFcaJ38ysME78ZmaF8cndMconC82sKtf4zcwK48RvZlYYJ34zs8K4jd9sDPG5HRsJrvGbmRXGid/MrDBO/GZmhXHiNzMrjBO/mVlhfFWPWYF8dVDZXOM3MyuME7+ZWWHc1NNl/hfbNib+Po9NrvGbmRVmVGr8kg4FvgiMAy6OiDmjEYeZbRg6/c8B/N9DO7qe+CWNA74MHAI8Djwg6eaI+FG3YzGzjYt/QFozGjX+fYD/iIhHASRdCxwFbNCJ322ZZuUYqR+QDfWHSBEx4oUOuUPpWODQiPhwnv8A8JaIOGnAerOAWXn2NcAjNYU0CXja5bgcl+NyxnA5g/m9iOgZuHA0avxqsmy9X5+IuBC4sPZgpN6ImO5yXI7LcTljtZx2jcZVPY8DuzbM7wI8OQpxmJkVaTQS/wPAnpKmStoUeC9w8yjEYWZWpK439UTEakknAXeQLue8NCIe7nYcDUaqOcnluByX43JGq5y2dP3krpmZjS7fuWtmVhgnfjOzwhSb+CVdKmmVpCUdlrOrpHskLZX0sKRTK5azuaT7JS3M5ZzdYVzjJD0k6ZYOylguabGkBZJ6OyhnoqRvSPpxfp/2q1DGa3Ic/Y/nJJ1WMZ6P5fd4iaRrJG1esZxTcxkPtxNLs++epO0lzZW0LD9vV7Gcd+d41kpq6TLBQcr5XP68Fkm6UdLEiuWck8tYIOlOSa+qUk7Da6dLCkmTKsZzlqQnGr5Hh1eNR9LJkh7J7/c/VIzn6w2xLJe0YLhyRkREFPkA9gf2BpZ0WM5kYO88vTXw78DrK5QjYKs8vQlwH7BvB3F9HLgauKWDMpYDk0bgvb4C+HCe3hSY2GF544CnSDentLvtzsBPgQl5/jrggxXKeQOwBNiCdJHEvwJ7Vv3uAf8AzM7Ts4FzK5bzOtINj/OA6R3E8w5gfJ4+t4N4tmmYPgX4apVy8vJdSReF/KyV7+Ug8ZwFnN7mZ92snLfnz3yzPL9j1eNqeP084FPtfherPIqt8UfEd4FfjkA5KyLiwTz9PLCUlFzaLSci4oU8u0l+VDrzLmkX4Ajg4irbjyRJ25C+8JcARMTvIuKZDos9CPhJRPys4vbjgQmSxpMSd5X7SF4H/DAifh0Rq4HvAMe0suEg372jSD+Q5Oejq5QTEUsjoq273Acp5858XAA/JN1vU6Wc5xpmt6SF7/QQf5vnA3/VShnDlNOWQcr5S2BORLyY11nVSTySBLwHuKazaFtTbOKvg6QpwF6k2nqV7cflf/VWAXMjolI5wBdIfyBrK27fL4A7Jc1X6kKjit2BPuCy3PR0saQtO4zrvVT8A4mIJ4B/BB4DVgDPRsSdFYpaAuwvaQdJWwCH8/IbE9u1U0SsyDGuAHbsoKyR9iHg21U3lvRZST8H3gd8qmIZRwJPRMTCqnE0OCk3P13aSpPaIF4NvE3SfZK+I+kPO4zpbcDKiFjWYTktceIfIZK2Aq4HThtQy2lZRKyJiGmk2tU+kt5QIY53AqsiYn6VGAaYERF7A4cBH5W0f4UyxpP+vf1KROwF/CepKaMSpZv+jgT+peL225Fq11OBVwFbSnp/u+VExFJSE8hc4HZgIbB6yI3GIElnko7rqqplRMSZEbFrLuOk4dZvEsMWwJlU/NEY4CvAHsA00g//eRXLGQ9sB+wL/B/gulxrr+p4ulTbByf+ESFpE1LSvyoibui0vNwUMg84tMLmM4AjJS0HrgUOlHRlxTiezM+rgBtJPau263Hg8Yb/Xr5B+iGo6jDgwYhYWXH7g4GfRkRfRLwE3AD8UZWCIuKSiNg7IvYn/QvfSW1tpaTJAPl52KaDukmaCbwTeF/kRugOXQ28q8J2e5B+qBfm7/UuwIOSXtluQRGxMlew1gIXUe07Del7fUNuor2f9N/1sCecm8lNjn8CfL1iLG1z4u9Q/pW/BFgaEZ/voJye/isnJE0gJagft1tORPx1ROwSEVNITSJ3R0TbNVpJW0raun+adLKv7SugIuIp4OeSXpMXHURnXXB3WjN6DNhX0hb5szuIdF6mbZJ2zM+7kf5wO4nrZmBmnp4JfLODsjqmNFjSJ4EjI+LXHZSzZ8PskVT7Ti+OiB0jYkr+Xj9OuqDiqQrxTG6YPYYK3+nsJuDAXOarSRctVO1l82DgxxHxeMXt29eNM8gb4oP0R7oCeIn0RTqhYjlvJbWFLwIW5MfhFcp5I/BQLmcJI3B2HziAilf1kNrmF+bHw8CZHcQxDejNx3YTsF3FcrYAfgFs2+H7cjYpAS0B/pl8ZUaFcr5H+hFbCBzUyXcP2AG4i/Rfw13A9hXLOSZPvwisBO6oWM5/AD9v+E63cjVOs3Kuz+/zIuBbwM5Vyhnw+nJau6qnWTz/DCzO8dwMTK5YzqbAlfnYHgQOrHpcwOXA/+7kO93uw102mJkVxk09ZmaFceI3MyuME7+ZWWGc+M3MCuPEb2ZWGCf+guVeDs9rmD9d0lkjVPblko4dibKG2c+7lXr8vKfufdVJ0rRWeopsst01ufuBj0n6O0kHt7n98mY9XUo6o91Yqqh63NYZJ/6yvQj8SStd3HaTpHFtrH4CcGJEvL2ueLpkGqm/n5blO1f/KCLeGBHnR8SnIuJfRyiethN/m59bv7aP2zrnxF+21aQxPz828IWBNXZJL+TnA3KnVNdJ+ndJcyS9T2ksgcWS9mgo5mBJ38vrvTNvP06pr/cHck31Iw3l3iPpatINNgPjOT6Xv0TSuXnZp0g30H1V0ucGrP8KSf9fqa/0WyTd1n88jbVcSdMlzcvTW+aOux7IHcodlZf/fj6+BTnmPfO6tyqNn7BE0nF53Tfn92e+pDsaumE4RdKP8vbXDoh1U+DvgOPyPo5T6p//prz+DyW9scnndyewY97mbY2fWT7GsyU9mN+31+blOyj1i/+QpK+RugMf+F7PIfVeukDSVXnZTfmYHlZDh32SXsj/adwH7CfpcKV+/O+VdIHyeBDN3ttmx93kGK0O3bxbzI8N6wG8AGxDuhNyW+B04Kz82uXAsY3r5ucDgGdI4xBsBjwBnJ1fOxX4QsP2t5MqF3uS7lTcHJgF/E1eZzPSHb1Tc7n/CUxtEuerSF0t9JA6x7obODq/No8m/c4DxwK35f2/EvhV//HQcOcnMB2Yl6f/L/D+PD2RNLbClsCXSP3VQLpjcwKpz5mLGva3Lakr7e8DPXnZccClefpJ1vXdvt54BMAHgf/XMP8l4NN5+kBgQZNtpvDyPuL/+zPLx3hynj4RuDhPX0C+K5zUdXfQ5C7Y/s+7YX77/DyBdLfqDnk+gPfk6c1Jd/tOzfPXkO8cH+K9fdlx+9Gdh2v8hYvUk+g/kQbJaNUDkcYheBH4CanmCammPqVhvesiYm2krmYfBV5L6vPnz5S6n76P1FVBf38u90fET5vs7w9JybkvUh/xV5H6+B/KW4F/yft/CmjlHMA7gNk5tnmkRLYb8APgDEmfJA3+8pt8rAdLOlfS2yLiWdIAKG8A5uYy/oZ1/dgvAq5S6gm0lV4830rqXoCIuBvYQdK2LWzXqL/DwPms+1z2J3U1QETcSvpBbMUpkhaS+ubflXWf2RpStwyQPt9HGz7Dxr6LBntvbRSMH+0AbIPwBVJ/I5c1LFtNbgqUJFJNt9+LDdNrG+bX8vLv1MD+QILUtHByRNzR+IKkA0g1/maqdHc71Db/fWykBNS4zbti/YFMluamjCOAOyR9OCLulvRmUvv030u6k9SD6cMR0WxoySNISfdI4G8l/X6sG+ik1fjb7V+l/3NZw9Cfy5DyZ3MwsF9E/Do3jfW/b7+NiDX9qw5VDE3eW0lvaScWGxmu8RsR8UvSEIQnNCxeDrw5Tx9FasZo17tzW/sepE7fHiENnfeXSl1ZI+nVGn5glvuAP5Y0SekE4vGkEa+Gci/wrrz/nUhNSf2Ws+7YGrsJvgM4Of/QIWmv/Lw7qSZ7AaljrzcqjR3764i4kjSwy975+HqUxxSWtEk+P/AKYNeIuIc0QM5EYKsB8T5PGrqz33dJA5f0J96no+I4DwM0lnsYqU/5Zl7q/4xIzVi/ykn/taQ+6Jv5MbC70oBEkJq6+jV9b1n/uK0LnPit33m8vD/xi0jJ9n7gLQxeGx/KI6QE/W1S74O/JQ0H+SNSf+pLgK8xzH+ekUak+mtSc81CUn/8w3VbfD3pvEL/Pu4Dns2vnQ18UdL3SLXhfueQfuAW5djOycuPA5bkZorXkprG/gC4Py87E/hMRPyOdG7h3NwssoDU1/844EpJi0k9sJ4f6w8/eQ/w+oaTnGcB0yUtAuawrsvmTp1NGjnsQVLzy2ODrHch6X24inSuZnyO5RxSc896chPYicDtku4l9Q7a/54P9t4OPG7rAvfOaRstSVtFxAuSdgDuJ40o1nYf7ta6hvdcwJeBZRFx/mjHZS/nNn7bmN2iNLjNpsA5Tvpd8RdKI3dtSvrv5mujHI814Rq/mVlh3MZvZlYYJ34zs8I48ZuZFcaJ38ysME78ZmaF+S/gHXKzRGTfZgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(expectation, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean = 7.09 ± 1.79 guesses; worst = 19\n",
      "CPU times: user 13.6 s, sys: 87.6 ms, total: 13.6 s\n",
      "Wall time: 14.6 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAawElEQVR4nO3debhcVZnv8e+PhCGMAXLACOEmcBG1bQ2QRugoIoOXwcvQokirjQ12vCqjctsotkKjt0MrotheMchkM4kyyCSQZhBpNZBgJgyIYlQgJAdlbBVJ8vYfa5WpnJw6p3bl7DpJ1u/zPPXUrl317v1W1a63dq3aey1FBGZmVo4NhjsBMzPrLhd+M7PCuPCbmRXGhd/MrDAu/GZmhRk53Am0Y8yYMTF+/PjhTsPMbJ0ye/bspyOip+/8daLwjx8/nlmzZg13GmZm6xRJv+pvvpt6zMwK48JvZlYYF34zs8K48JuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MCrNOnLlr67bxU2/pKG7RtMOGOBMzA+/xm5kVx4XfzKwwLvxmZoVx4TczK4wLv5lZYVz4zcwK48JvZlYYF34zs8K48JuZFcaF38ysMC78ZmaFqa3wS9pE0v2S5kp6SNJZef4ESTMlPSrpW5I2qisHMzNbXZ17/C8B+0fEG4CJwMGS9gbOAc6LiF2BZ4ATaszBzMz6qK3wR/JivrlhvgSwP/CdPP8y4Mi6cjAzs9XV2sYvaYSkOcBSYAbwC+DZiFiWH/I4sEOdOZiZ2apqLfwRsTwiJgI7AnsBr+nvYf3FSpoiaZakWb29vXWmaWZWlK4c1RMRzwL3AHsDoyU1BoDZEXiyRcz0iJgUEZN6enq6kaaZWRHqPKqnR9LoPD0KOBBYCNwNHJ0fdhzw3bpyMDOz1dU59OJY4DJJI0hfMNdExM2SfgpcLemzwE+Ai2rMwczM+qit8EfEPGD3fuY/RmrvNzOzYeAzd83MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhXPjNzArjwm9mVhgXfjOzwrjwm5kVxoXfzKwwLvxmZoVx4TczK4wLv5lZYVz4zcwK48JvZlYYF34zs8K48JuZFcaF38ysMC78ZmaFceE3MytMbYVf0jhJd0taKOkhSafk+WdKekLSnHw5tK4czMxsdSNrXPYy4GMR8aCkLYDZkmbk+86LiC/UuG4zM2uhtsIfEYuBxXn6BUkLgR3qWp+ZmbWnK238ksYDuwMz86wTJc2TdLGkrVvETJE0S9Ks3t7ebqRpZlaE2gu/pM2Ba4FTI+J54GvALsBE0i+Cc/uLi4jpETEpIib19PTUnaaZWTFqLfySNiQV/Ssi4jqAiFgSEcsjYgVwIbBXnTmYmdmq6jyqR8BFwMKI+GLT/LFNDzsKWFBXDmZmtro6j+qZDLwPmC9pTp73SeBYSROBABYBH6wxBzMz66POo3ruA9TPXbfWtU4zMxucz9w1MyuMC7+ZWWFc+M3MClPnn7u2nhg/9ZaO4hZNO2yIMzGzoeA9fjOzwrjwm5kVxoXfzKwwLvxmZoVx4TczK4wLv5lZYVz4zcwK48JvZlYYF34zs8K48JuZFcaF38ysMC78ZmaFGbTwS9pF0sZ5ej9JJ0saXX9qZmZWh3b2+K8Flkv6n6QxdCcAV9aalZmZ1aadwr8iIpaRBkb/UkScBowdJMbMzNZS7RT+lyUdCxwH3JznbVhfSmZmVqd2Cv/fA/sAn4uIX0qaAFxeb1pmZlaXdkbgOigiTm7cyMX/DzXmZGZmNWpnj/+4fua9f4jzMDOzLmm5x5/b9f8WmCDpxqa7tgB+O9iCJY0Dvgm8AlgBTI+IL0vaBvgWMB5YBLwrIp7p9AmYmVk1AzX1/BBYDIwBzm2a/wIwr41lLwM+FhEPStoCmC1pBunXwp0RMU3SVGAq8PFOkjczs+paNvVExK8i4p6I2Ie0Z75hRHwfWAiMGmzBEbE4Ih7M0y/kuB2AI4DL8sMuA45co2dgZmaVtHPm7j8A3wG+nmftCNxQZSWSxgO7AzOB7SNiMaQvB2C7FjFTJM2SNKu3t7fK6szMbADt/Ln7EWAy8DxARDxKi2LdH0mbk87+PTUinm83LiKmR8SkiJjU09PTbpiZmQ2incL/UkT8qXFD0kgg2lm4pA1JRf+KiLguz14iaWy+fyywtFrKZma2Jtop/N+X9ElglKSDgG8DNw0WJEmkvn0WRsQXm+66kZWHiB4HfLdaymZmtibaKfxTgV5gPvBB4FbgU23ETQbeB+wvaU6+HApMAw6S9ChwUL5tZmZdMuiZuxGxArgwX9oWEfcBanH3AVWWZWZmQ2fQwi9pPqu36T8HzAI+GxGDnsxlZmZrj3b66vkesJyVffC/O18/D1wK/O+hT8vMzOrSTuGfHBGTm27Pl/SfETFZ0nvrSszMzOrRzp+7m0t6Y+OGpL2AzfPNZbVkZWZmtWlnj/8E4JJ8IhakvnpOkLQZ8C+1ZWZmZrUYsPBL2gDYOSL+UtJWgCLi2aaHXFNrdmZmNuQGbOrJh3KemKef61P0zcxsHdROG/8MSadLGidpm8al9szMzKwW7bTxH5+vP9I0L4Cdhz4dMzOrWztn7k7oRiJmZtYd7ezxI+l1wGuBTRrzIuKbdSVlZmb1aafLhs8A+5EK/63AIcB9pPF0zcxsHdPOn7tHkzpVeyoi/h54A7BxrVmZmVlt2mnq+UNErJC0TNKWpIFT/Meudc34qbd0FLdo2mFDnInZ+qGdwj9L0mhSt8yzgReB+2vNyszMatPOUT0fzpMXSLoN2DIi5tWblpmZ1WXQNn5JdzamI2JRRMxrnmdmZuuWlnv8kjYBNgXGSNqalaNpbQm8sgu5mZlZDQZq6vkgcCqpyM9mZeF/HvhqzXmZmVlNWhb+iPgy8GVJJ0XEV7qYk5mZ1WjQNn4XfTOz9Us7J3CZmdl6pGXhlzQ5X/ssXTOz9chAe/zn5+sfdbJgSRdLWippQdO8MyU9IWlOvhzaybLNzKxzAx3V87KkS4AdJJ3f986IOHmQZV8K/Burd+Z2XkR8oVKWZmY2ZAYq/G8HDgT2Jx3OWUlE3CtpfGdpmZlZXQY6nPNp4GpJCyNi7hCu80RJfwfMAj4WEc/09yBJU4ApADvttNMQrt7MrGztHNXzW0nX5/b6JZKulbRjh+v7GrALMBFYDJzb6oERMT0iJkXEpJ6eng5XZ2ZmfbVT+C8BbiSdwbsDcFOeV1lELImI5RGxgtTb516dLMfMzDrXTuHfLiIuiYhl+XIp0NEuuKSxTTePAha0eqyZmdWjnf74eyW9F7gq3z4W+O1gQZKuIg3ZOEbS48BngP0kTQQCWETqD8jMzLqoncJ/POmwzPNIBfuHed6AIuLYfmZfVCk7MzMbcu0MxPJr4PAu5GJmZl3gvnrMzArjwm9mVhgXfjOzwrRd+CXtLekuSf8p6cg6kzIzs/oMNObuKyLiqaZZHyX9ySvSkT031JybmZnVYKCjei6QNBv4fET8EXgW+FtgBWncXTMzWwe1bOqJiCOBOcDNkt5HGnh9BbAp4KYeM7N11IBt/BFxE/C/gNHAdcAjEXF+RPR2IzkzMxt6Aw29eLik+4C7SH3qvBs4StJVknbpVoJmZja0Bmrj/yywDzAKuDUi9gI+KmlX4HOkLwIzM1vHDFT4nyMV91HA0sbMiHgUF30zs3XWQG38R5H+yF1GOprHzMzWA4MNvfiVLuZiZmZd4C4bzMwK48JvZlYYF34zs8K48JuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MClNb4Zd0saSlkhY0zdtG0gxJj+brretav5mZ9a/OPf5LgYP7zJsK3BkRuwJ35ttmZtZFtRX+iLgX+F2f2UcAl+Xpy/BIXmZmXdftNv7tI2IxQL7ertUDJU2RNEvSrN5eD/hlZjZU1to/dyNiekRMiohJPT09w52Omdl6o9uFf4mksQD5eukgjzczsyHW7cJ/I3Bcnj4O+G6X129mVrw6D+e8CvgRsJukxyWdAEwDDpL0KHBQvm1mZl000Ji7ayQijm1x1wF1rdPMzAZXW+G3tcf4qbd0FLdo2mFDnImZrQ3W2qN6zMysHi78ZmaFceE3MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhXPjNzArjwm9mVhgXfjOzwrjwm5kVxoXfzKwwHoHL1nsegcxsVd7jNzMrjAu/mVlhXPjNzArjwm9mVphh+XNX0iLgBWA5sCwiJg1HHmZmJRrOo3reGhFPD+P6zcyK5KYeM7PCDFfhD+AOSbMlTenvAZKmSJolaVZvb2+X0zMzW38NV+GfHBF7AIcAH5G0b98HRMT0iJgUEZN6enq6n6GZ2XpqWAp/RDyZr5cC1wN7DUceZmYl6nrhl7SZpC0a08DbgAXdzsPMrFTDcVTP9sD1khrrvzIibhuGPMzMitT1wh8RjwFv6PZ6zcws8eGcZmaFceE3MyuMC7+ZWWFc+M3MCuPCb2ZWGA+9aDaIToduBA/faGsn7/GbmRXGhd/MrDBu6lnLuZnBzIaa9/jNzArjwm9mVhgXfjOzwrjwm5kVxn/umtXMf9Db2sZ7/GZmhXHhNzMrjAu/mVlhXPjNzArjwm9mVhgXfjOzwvhwTrO1nA8HtaHmwl8zf2jNbG3jwm+2nvPOh/U1LG38kg6W9Iikn0uaOhw5mJmVqut7/JJGAF8FDgIeBx6QdGNE/LTbuZhZezr91dD4xTDc8baq4Wjq2Qv4eUQ8BiDpauAIYK0s/P6ZbLbu8xfHqhQR3V2hdDRwcER8IN9+H/DGiDixz+OmAFPyzd2AR2pIZwzw9DAvw/GO9zbo+DXdBlr5HxHR03fmcOzxq595q337RMR0YHqtiUizImLScC7D8Y73Nuj4Nd0GqhqOP3cfB8Y13d4ReHIY8jAzK9JwFP4HgF0lTZC0EfBu4MZhyMPMrEhdb+qJiGWSTgRuB0YAF0fEQ93OIxuKpqQ1XYbjHb+mhjsHxw9vfGVd/3PXzMyGlztpMzMrjAu/mVlhiiz8ki6WtFTSgg7jx0m6W9JCSQ9JOqVi/CaS7pc0N8ef1WEeIyT9RNLNHcQukjRf0hxJszqIHy3pO5Iezq/DPhXjd8vrblyel3RqxWWcll+/BZKukrRJxfhTcuxD7ay7v+1G0jaSZkh6NF9vXTH+nXn9KyQNeEhfi/jP5/dgnqTrJY2uGH92jp0j6Q5Jr6yaQ9N9p0sKSWMq5nCmpCeatoVDq65f0km5G5iHJP1rxfV/q2ndiyTNqRg/UdKPG58lSXtVjH+DpB/lz+NNkrZsFT9kIqK4C7AvsAewoMP4scAeeXoL4GfAayvEC9g8T28IzAT27iCPjwJXAjd3ELsIGLMGr+FlwAfy9EbA6DVY1gjgKdLJJu3G7AD8EhiVb18DvL9C/OuABcCmpIMc/gPYtep2A/wrMDVPTwXOqRj/GtIJivcAkzpY/9uAkXn6nA7Wv2XT9MnABVVzyPPHkQ7Y+NVA21WLHM4ETm/zfesv/q35/ds4396uav5N958LfLri+u8ADsnThwL3VIx/AHhLnj4eOLvd7bjTS5F7/BFxL/C7NYhfHBEP5ukXgIWkQtRufETEi/nmhvlS6V92STsChwHfqBI3FPIeyb7ARQAR8aeIeHYNFnkA8IuI+FXFuJHAKEkjSQW8yvkgrwF+HBG/j4hlwPeBowYKaLHdHEH6EiRfH1klPiIWRkRbZ6W3iL8j5w/wY9J5MVXin2+6uRmDbIcDfHbOA/5xDeLb0iL+Q8C0iHgpP2ZpJ+uXJOBdwFUV4wNo7KVvxQDbYYv43YB78/QM4B2t4odKkYV/KEkaD+xO2muvEjci/6RcCsyIiErxwJdIH7QVFeMaArhD0myl7jGq2BnoBS7JTU3fkLRZh3lAOpej5YetPxHxBPAF4NfAYuC5iLijwiIWAPtK2lbSpqQ9tXGDxPRn+4hYnHNaDGzXwTKGyvHA96oGSfqcpN8A7wE+3UH84cATETG3amyTE3OT08UDNZe18CrgzZJmSvq+pL/qMIc3A0si4tGKcacCn8+v4ReAT1SMXwAcnqffSWfbYSUu/GtA0ubAtcCpffacBhURyyNiImkPbS9Jr6uw3rcDSyNidqWEVzU5IvYADgE+ImnfCrEjST9XvxYRuwP/RWrmqEzpJL7DgW9XjNuatLc9AXglsJmk97YbHxELSU0jM4DbgLnAsgGD1mKSziDlf0XV2Ig4IyLG5dgTB3t8n/VuCpxBB18YTb4G7AJMJH2Jn1sxfiSwNbA38H+Ba/Lee1XHUnEHJPsQcFp+DU8j/xKu4HjSZ3A2qen4Tx3kUIkLf4ckbUgq+ldExHWdLic3kdwDHFwhbDJwuKRFwNXA/pIur7jeJ/P1UuB6Uq+p7XoceLzpV8p3SF8EnTgEeDAillSMOxD4ZUT0RsTLwHXAX1dZQERcFBF7RMS+pJ/fVff0AJZIGguQr1s2M9RF0nHA24H3RG4o7tCVVG9m2IX05Ts3b487Ag9KekW7C4iIJXlHaAVwIdW2RUjb43W5CfV+0q/gln8w9yc3F/4N8K2K6wY4jrT9QdqBqZR/RDwcEW+LiD1JXzy/6CCHSlz4O5D3Ji4CFkbEFzuI72kcfSFpFKmIPdxufER8IiJ2jIjxpGaSuyKi7b1dSZtJ2qIxTfqDsO0jnCLiKeA3knbLsw6g8261O93L+jWwt6RN8/txAOm/lrZJ2i5f70T60HeSx42kDz75+rsdLKNjkg4GPg4cHhG/7yB+16abh1NhOwSIiPkRsV1EjM/b4+OkAx+eqpDD2KabR1FhW8xuAPbPy3oV6WCDqr1dHgg8HBGPV4yD1Kb/ljy9PxV3IJq2ww2ATwEXdJBDNXX/e7w2Xkgf8MXAy6QN9YSK8W8itZHPA+bky6EV4l8P/CTHL2CAowjaWNZ+VDyqh9RGPzdfHgLO6GC9E4FZ+TncAGzdwTI2BX4LbNXhcz+LVKgWAP9OPqqjQvwPSF9Yc4EDOtlugG2BO0kf9juBbSrGH5WnXwKWALdXjP858Jum7bDlUTkt4q/Nr9884CZghzX57DDI0WItcvh3YH7O4UZgbMX4jYDL8/N4ENi/av7ApcD/6XAbeBMwO29HM4E9K8afQjoy8GfANHKPCnVe3GWDmVlh3NRjZlYYF34zs8K48JuZFcaF38ysMC78ZmaFceEvWO5J8dym26dLOnOIln2ppKOHYlmDrOedSr2D3l33uuqUe3hs2SvlAHFX5a4OTpP0z5IOrBi/qL/eNCV9smounej0eduaceEv20vA3wzUje5wkDSiwsNPAD4cEW+tK58umUjqL6ht+ezYv46I10fEeRHx6Yj4jyHKp3Lhr/i+NVR+3rbmXPjLtow03udpfe/ou8cu6cV8vV/uCOsaST+TNE3Se5TGF5gvaZemxRwo6Qf5cW/P8SOU+pB/IO+pfrBpuXdLupJ0Mk/ffI7Ny18g6Zw879Okk2cukPT5Po/fQNL/V+qf/WZJtzaeT/NerqRJku7J05vlTsIeyJ3PHZHn/0V+fnNyzrvmx96iNKbCAknH5MfumV+f2ZJub+rO4WRJP83xV/fJdSPgn4Fj8jqOUern/4b8+B9Len0/798dwHY55s3N71l+jmdJejC/bq/O87dV6nf/J5K+TuoivO9rPY3U6+kcSVfkeTfk5/SQmjr1k/Ri/qUxE9hH0qFK4wPcJ+l85bEi+ntt+3ve/TxHq0PdZ4j5svZegBdJ3ckuInUnezpwZr7vUuDo5sfm6/2AZ0ljEmwMPAGcle87BfhSU/xtpJ2LXUlnKW4CTAE+lR+zMens3wl5uf8FTOgnz1eSumjoIXXIdRdwZL7vHvrpxx44Grg1r/8VwDON50PT2aXAJHL/6cD/A96bp0eTzqTcDPgKqR8cSGeJjiL1aXNh0/q2InWv/UOgJ887Brg4Tz/Jyv7iVxu7AHg/8G9Nt78CfCZP7w/M6SdmPKv26/7n9yw/x5Py9IeBb+Tp88lnipO69Q76OdO28X433d4mX48inSG7bb4dwLvy9Caks4gn5NtXkc8qH+C1XeV5+9Kdi/f4CxepV9FvkgbhaNcDkcYkeInUoVSjO+T5pGLUcE1ErIjUze1jwKtJ/QL9nVKX1DNJXR40+ou5PyJ+2c/6/opUnHsj9T1/BWk8gIG8Cfh2Xv9TQDv/AbwNmJpzu4dUyHYCfgR8UtLHSYPF/CE/1wMlnSPpzRHxHKlf9dcBM/IyPsXK/vHnAVco9SDaTi+gbyJ1ZUBE3AVsK2mrNuKaNToOm83K92VfUvcGRMQtpC/EdpwsaS6pz/9xrHzPlpO6fYD0/j7W9B42933U6rW1YTByuBOwtcKXSH2cXNI0bxm5KVCSSHu6DS81Ta9our2CVbepvv2BBKlp4aSIuL35Dkn7kfb4+9NJF7sDxfz5uZEKUHPMO2L1gVEW5qaMw4DbJX0gIu6StCepffpfJN1B6uX0oYjobxjKw0hF93DgnyT9RawcQKXd/Kv2r9J4X5Yz8PsyoPzeHAjsExG/z01jjdftjxGxvPHQgRZDP6+tpDdWycWGhvf4jYj4HWnowhOaZi8C9szTR5CaMap6Z25r34XUMdwjpOH5PqTUrTWSXqXBB3GZCbxF0hilPxCPJY2YNZD7gHfk9W9PakpqWMTK59bcDfHtwEn5iw5Ju+frnUl7sueTOhF7vdLYtL+PiMtJg2/skZ9fj/L4w5I2zP8PbACMi4i7SYPnjAY275PvC6S+2BvuJQ2M0ii8T0fFMR9aaF7uIaR+7PvzcuM9IjVjPZOL/qtJ/d7352FgZ6XBiSA1dTX0+9qy+vO2LnDht4ZzWbUP8wtJxfZ+4I203hsfyCOkAv09Us+HfyQNFflTUp/tC4CvM8gvz0gjW32C1Fwzl9R//2DdH19L+l+hsY6ZwHP5vrOAL0v6AWlvuOFs0hfcvJzb2Xn+McCC3EzxalLT2F8C9+d5ZwCfjYg/kf5bOCc3i8whjREwArhc0nxSr6znxepDVd4NvLbpT84zgUmS5pF6bDyOoXEWaeSxB0nNL79u8bjppNfhCtJ/NSNzLmeTmntWk5vAPgzcJuk+Um+jjde81Wvb93lbF7h3TltvSdo8Il6UtC1wP2nUsbb7ibfqml5zAV8FHo2I84Y7L1uV2/htfXaz0oA3GwFnu+h3xT8ojQi2EenXzdeHOR/rh/f4zcwK4zZ+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrzH8DJidrc7m3ukUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(negative_entropy, False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing Metrics with Inconsistent Guesses Allowed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean = 7.05 ± 0.98 guesses; worst = 10\n",
      "CPU times: user 38.2 s, sys: 134 ms, total: 38.4 s\n",
      "Wall time: 38.5 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAWsUlEQVR4nO3de7gkdX3n8fdHLoIgosygCMZBl/USo6izCMEYAphFcREjKkRcokTcFUVjfBJ0XSPRbOBxjYproigCWRFF8YJAQMJFQ1RwuINI8DIaIjCjcpFV0YHv/lF15HDmzJmeyVSdmfm9X8/TT1dVV/X3V90zn67zq+5fpaqQJLXjQfPdAEnSuAx+SWqMwS9JjTH4JakxBr8kNWbT+W7AJBYsWFCLFi2a72ZI0gbl8ssv/1FVLZy5fIMI/kWLFrFkyZL5boYkbVCSfH+25Xb1SFJjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYzaIX+5KWrVFR589Wq2lx+4/Wi0NxyN+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaszgwZ9kkyRXJjmrn985yaVJbkryqSSbD90GSdL9xjjifwNww7T544D3VtUuwO3A4SO0QZLUGzT4k+wE7A98tJ8PsDfwmX6VU4ADh2yDJOmBhj7ifx/wZ8B9/fx2wB1VtaKfvxnYcbYNkxyRZEmSJcuXLx+4mZLUjsGCP8kLgGVVdfn0xbOsWrNtX1UnVNXiqlq8cOHCQdooSS3adMDn3hM4IMnzgS2Abej+Atg2yab9Uf9OwA8HbIMkaYbBjvir6i1VtVNVLQIOBi6sqpcDFwEH9asdBnxhqDZIklY2H9/j/3PgTUm+Tdfnf+I8tEGSmjVkV8+vVdXFwMX99HeB3caoK0lamb/claTGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JasxgwZ9kiySXJbk6yfVJjumX75zk0iQ3JflUks2HaoMkaWVDHvHfA+xdVU8DdgX2S7I7cBzw3qraBbgdOHzANkiSZhgs+Ktzdz+7WX8rYG/gM/3yU4ADh2qDJGllg/bxJ9kkyVXAMuB84DvAHVW1ol/lZmDHIdsgSXqgQYO/qu6tql2BnYDdgCfNttps2yY5IsmSJEuWL18+ZDMlqSmrDf4kj0/y4H56ryRHJdl2TYpU1R3AxcDuwLZJNu0f2gn44Sq2OaGqFlfV4oULF65JOUnSHCY54j8DuDfJfwBOBHYGPrG6jZIsnPqASLIlsC9wA3ARcFC/2mHAF9ai3ZKktbTp6lfhvqpakeRFwPuq6gNJrpxgux2AU5JsQvcBc3pVnZXkm8Ank7wLuJLuw0SSNJJJgv9XSQ6hOzr/L/2yzVa3UVVdAzx9luXfpevvlyTNg0m6el4J7AH8VVV9L8nOwMeHbZYkaSiTHPE/t6qOmprpw//nA7ZJkjSgSY74D5tl2R+t43ZIkkayyiP+vl//D4Gdk5w57aGHAj8eumGSpGHM1dXzVeAWYAHwnmnLfwpcM2SjJEnDWWXwV9X3ge8DeyR5LLBLVf1j/538Lek+ACRJG5hJfrn7arpB1T7cL9oJ+PyQjZIkDWeSk7tHAnsCdwFU1U3A9kM2SpI0nEmC/56q+uXUTD/OzqwDq0mS1n+TBP+Xk7wV2DLJc4FPA18ctlmSpKFMEvxHA8uBa4HXAOcAbxuyUZKk4az2l7tVdR/wkf4mSdrArTb4k1zLyn36dwJLgHdVlT/mkqQNyCRj9fwDcC/3j8F/cH9/F3Ay94/YKakhi44+e9R6S4/df9R6G7NJgn/Pqtpz2vy1Sf65qvZMcuhQDZMkDWOSk7tbJ3nW1EyS3YCt+9kVs28iSVpfTXLEfzhwUpKpsP8pcHiSrYC/HqxlkqRBzBn8SR4EPK6qfivJw4D0F06fcvqgrZMkrXNzdvX0X+V8XT9954zQlyRtgCbp4z8/yZuTPCbJI6Zug7dMkjSISfr4X9XfHzltWQGPW/fNkSQNbZJf7u48RkMkSeOY5IifJE8BngxsMbWsqv5+qEZJkoYzyZANfwHsRRf85wDPAy4BDH5J2gBNcnL3IGAf4NaqeiXwNODBg7ZKkjSYSYL/5/3XOlck2QZYhid2JWmDNUkf/5Ik29INy3w5cDdw2aCtkiQNZpJv9by2n/xQknOBbarqmmGbJUkaymq7epJcMDVdVUur6prpyyRJG5ZVHvEn2QJ4CLAgycOB9A9tAzx6hLZJkgYwV1fPa4A30oX85dwf/HcBHxy4XZKkgawy+Kvq/cD7k7y+qj4wYpskSQNabR+/oS9JG5dJvscvSdqIrDL4k+zZ3/srXUnaiMx1xH98f/+1MRoiSRrHXN/q+VWSk4Adkxw/88GqOmq4ZkmShjJX8L8A2BfYm+7rnJKkjcBcX+f8EfDJJDdU1dVr+sRJHkM3dPOjgPuAE6rq/f1lGz8FLAKWAi+tqtvXou2SpLUwybd6fpzkc0mWJbktyRlJdppguxXAn1bVk4DdgSOTPBk4GrigqnYBLujnJUkjmST4TwLOpPsF747AF/tlc6qqW6rqin76p8AN/fYvBE7pVzsFOHDNmy1JWluTBP/2VXVSVa3obycDC9ekSJJFwNOBS4FHVtUt0H04ANuvYpsjkixJsmT58uVrUk6SNIdJgn95kkOTbNLfDgV+PGmBJFsDZwBvrKq7Jt2uqk6oqsVVtXjhwjX6nJEkzWGS4H8V8FLgVuAWuksxvmqSJ0+yGV3on1pVn+0X35Zkh/7xHeiu6CVJGskkF2L5AXDAmj5xkgAnAjdU1d9Me+hM4DDg2P7+C2v63JKktTfJpRfX1p7AK4Brk1zVL3srXeCfnuRw4AfASwZsgyRphsGCv6ou4f4x/GfaZ6i6kqS5OTqnJDVm4uBPsnuSC5P8cxK/ey9JG6i5rrn7qKq6ddqiN9Gd5A3wVeDzA7dN2mAsOvrs0WotPXb/0Wpp4zRXH/+HklwOvLuqfgHcAfwh3bg7E38fX5K0flllV09VHQhcBZyV5BV0F16/D3gIDrMgSRusOfv4q+qLwH8GtgU+C9xYVcdXlWMoSNIGaq5LLx6Q5BLgQuA64GDgRUlOS/L4sRooSVq35urjfxewB7AlcE5V7Qa8KckuwF/RfRBIkjYwcwX/nXThviXTxtOpqpsw9CVpgzVXH/+L6E7krqD7No8kaSOwuksvfmDEtkiSRuCQDZLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhozWPAn+ViSZUmum7bsEUnOT3JTf//woepLkmY35BH/ycB+M5YdDVxQVbsAF/TzkqQRDRb8VfUV4CczFr8QOKWfPgU4cKj6kqTZjd3H/8iqugWgv99+VSsmOSLJkiRLli9fPloDJWljt96e3K2qE6pqcVUtXrhw4Xw3R5I2GmMH/21JdgDo75eNXF+Smjd28J8JHNZPHwZ8YeT6ktS8Ib/OeRrwNeAJSW5OcjhwLPDcJDcBz+3nJUkj2nSoJ66qQ1bx0D5D1ZQkrd56e3JXkjQMg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1JjBrsAljW3R0WePWm/psfuPWk9aVwx+SRscP+T/fezqkaTGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGC+9qHXKS+JJ6795OeJPsl+SG5N8O8nR89EGSWrV6MGfZBPgg8DzgCcDhyR58tjtkKRWzUdXz27At6vquwBJPgm8EPjmPLRlUGN2e9jlIY1jY/h/naoa5IlXWTA5CNivqv64n38F8Kyqet2M9Y4AjuhnnwDcOGpDYQHwo5FrznftFve51dot7nOLtR9bVQtnLpyPI/7MsmylT5+qOgE4YfjmzC7Jkqpa3FLtFve51dot7nPLtWeaj5O7NwOPmTa/E/DDeWiHJDVpPoL/G8AuSXZOsjlwMHDmPLRDkpo0eldPVa1I8jrgPGAT4GNVdf3Y7ZjAvHUzzWPtFve51dot7nPLtR9g9JO7kqT55ZANktQYg1+SGmPwz5DkY0mWJblu5LqPSXJRkhuSXJ/kDSPW3iLJZUmu7msfM1btvv4mSa5MctbIdZcmuTbJVUmWjFx72ySfSfKt/j3fY6S6T+j3d+p2V5I3jlG7r/8n/b+x65KclmSLkeq+oa95/dD7O1uGJHlEkvOT3NTfP3zINqyOwb+yk4H95qHuCuBPq+pJwO7AkSMOZXEPsHdVPQ3YFdgvye4j1QZ4A3DDiPWm+72q2nUevl/9fuDcqnoi8DRG2v+qurHf312BZwI/Az43Ru0kOwJHAYur6il0X+44eIS6TwFeTTdqwNOAFyTZZcCSJ7NyhhwNXFBVuwAX9PPzxuCfoaq+AvxkHureUlVX9NM/pQuCHUeqXVV1dz+7WX8b5ax/kp2A/YGPjlFvfZBkG+A5wIkAVfXLqrpjHpqyD/Cdqvr+iDU3BbZMsinwEMb5Dc+TgK9X1c+qagXwZeBFQxVbRYa8EDilnz4FOHCo+pMw+NdDSRYBTwcuHbHmJkmuApYB51fVWLXfB/wZcN9I9aYr4EtJLu+HCBnL44DlwEl9F9dHk2w1Yv0pBwOnjVWsqv4N+N/AD4BbgDur6ksjlL4OeE6S7ZI8BHg+D/wR6RgeWVW3QHeQB2w/cv0HMPjXM0m2Bs4A3lhVd41Vt6ru7f/83wnYrf/zeFBJXgAsq6rLh661CntW1TPoRoo9MslzRqq7KfAM4O+q6unA/2PkP/37H08eAHx6xJoPpzvy3Rl4NLBVkkOHrltVNwDHAecD5wJX03WtNsvgX48k2Ywu9E+tqs/ORxv6LoeLGec8x57AAUmWAp8E9k7y8RHqAlBVP+zvl9H1c+82UumbgZun/VX1GboPgjE9D7iiqm4bsea+wPeqanlV/Qr4LPDbYxSuqhOr6hlV9Ry6bpibxqg7zW1JdgDo75eNXP8BDP71RJLQ9fneUFV/M3LthUm27ae3pPsP+q2h61bVW6pqp6paRNftcGFVDX4ECJBkqyQPnZoGfp+uS2BwVXUr8K9JntAv2ofxhyU/hBG7eXo/AHZP8pD+3/s+jHRSO8n2/f1vAH/A+Pt+JnBYP30Y8IWR6z+Al16cIclpwF7AgiQ3A39RVSeOUHpP4BXAtX1fO8Bbq+qcEWrvAJzSXyTnQcDpVTXqVyvnwSOBz3X5w6bAJ6rq3BHrvx44te9y+S7wyrEK9/3czwVeM1ZNgKq6NMlngCvoulquZLxhDM5Ish3wK+DIqrp9qEKzZQhwLHB6ksPpPgBfMlT9SThkgyQ1xq4eSWqMwS9JjTH4JakxBr8kNcbgl6TGGPyNSlJJ3jNt/s1J3rGOnvvkJAeti+daTZ2X9CNbXjR0rSEl2TXJ89diu9OSXNOPePmXSfZdw+2XJlkwy/K3rmlb1sba7rf+/Qz+dt0D/MFs//HnU/9bgkkdDry2qn5vqPaMZFe68WMmluRRwG9X1VOr6r1V9faq+sd11J41Dv41fN+mrPF+a90w+Nu1gu7HM38y84GZR+xJ7u7v90ry5SSnJ/mXJMcmeXm6sfyvTfL4aU+zb5J/6td7Qb/9JkneneQb/ZHqa6Y970VJPgFcO0t7Dumf/7okx/XL3g48G/hQknfPWP9BSf62H3v9rCTnTO3P9KPcJIuTXNxPb5VuHPVv9AOnvbBf/pv9/l3Vt3mXft2z012/4LokL+vXfWb/+lye5LxpP9E/Ksk3++0/OaOtmwN/Cbysr/GydGO3f75f/+tJnjrL+/clYPt+m9+Z/p71+3hMkiv61+2J/fLtknyp378PA5nltT6WbvTMq5Kc2i/7fL9P12faYHZJ7u7/0rgU2CPJ89NdX+CSJMenv77CbK/tbPs9yz5qKFXlrcEbcDewDbAUeBjwZuAd/WMnAwdNX7e/3wu4g+6Xvg8G/g04pn/sDcD7pm1/Lt2BxS50Y9NsARwBvK1f58HAEroBu/aiG6hs51na+Wi6XzoupPuF7YXAgf1jF9ON7T5zm4OAc/r6jwJun9qffn8X9NOLgYv76f8FHNpPbwv8C7AV8AHg5f3yzYEtgRcDH5lW72F0Q1l/FVjYL3sZ8LF++ofAg6eee5b2/hHwf6bNf4DuF+MAewNXzbLNIuC6afO/fs/6fXx9P/1a4KP99PHA2/vp/elGJ10w27+NGfOP6O+3pBvWYrt+voCX9tNbAP869R7SDYlw1mpe2wfst7fxbh7xN6y60T//nu7iGJP6RnXXDrgH+A7dkSd0R+qLpq13elXdV1U30Q1J8ES68XD+a7ohKS4FtqP7YAC4rKq+N0u9/0QXzsurG0v9VLqx7OfybODTff1bgUnOAfw+cHTftovpguw3gK8Bb03y58Bjq+rn/b7um+S4JL9TVXcCTwCeApzfP8fb6EY6BbiGbniGQ5lsVMhnA/8XoKouBLZL8rAJtptuapC/y7n/fXkO8PH+ec+m+0CcxFFJrga+Tjec8dR7di/doILQvb/fnfYeTh8LZ1WvreaJY/XofXRjp5w0bdkK+m7AJKE70p1yz7Tp+6bN38cD/z3NHAuk6LoWXl9V501/IMledEf8s1mpO2ICc23z632jC6Dp27y4qm6csf4NfVfG/sB5Sf64qi5M8ky6/um/TvIlutE9r6+q2S6huD9d6B4A/M8kv9l/iK1J+9d0bJWp9+Ve5n5f5tS/N/sCe1TVz/qusanX7RdVde/UqnM9DbO8tkmetSZt0brjEX/jquonwOl0J0qnLKW7LB9046dvthZP/ZK+r/3xdBceuRE4D/jv6YafJsl/zOovQHIp8LtJFqQ7gXgI3RWU5nIJ8OK+/iPpupKmLOX+fXvxtOXnAa/vP+hI8vT+/nF0R7LH042w+NQkjwZ+VlUfp7uwyDP6/VuY/tq5STbrzw88CHhMVV1Ed8GZbYGtZ7T3p8BDp81/BXh5/zx7AT+qdXNthunP+zxgVdd9/dXUe0TXjXV7H/pPpLss6Gy+BTwu3UWEoOvqmjLra8vK+62RGPwCeA8w/ds9H6EL28uAZ7Hqo/G53EgX0P8A/Leq+gXd5RW/CVyR7kLUH2Y1f3VWd7Wit9B111xNN4b86oa0PYPuvMJUjUuBO/vHjgHen+Sf6I6Gp7yT7gPumr5t7+yXvwy4ru+meCJd19hvAZf1y/4H8K6q+iXduYXj+m6Rq+jGmt8E+HiSa+lGo3xvrXyZxYuAJ087yfkOYHGSa+hGdTyMdeMYuitRXUHX/fKDVax3At3rcCrduZpN+7a8k667ZyV9F9hrgXOTXALcxv2v+ape25n7rZE4Oqc2Skm2rqq70w3Fexnd1bZune92bcymveYBPgjcVFXvne92aWX28WtjdVa6i8tsDrzT0B/Fq5McRveaX0n315bWQx7xS1Jj7OOXpMYY/JLUGINfkhpj8EtSYwx+SWrM/wcjPEwbRHjHlAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(max, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean = 6.84 ± 0.95 guesses; worst = 10\n",
      "CPU times: user 39.9 s, sys: 183 ms, total: 40.1 s\n",
      "Wall time: 40.5 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAWsElEQVR4nO3de7gkdX3n8fdH7oKIMoMiGAfdWS8xijqLEIwhgFkUFzFihIg7USLuioIanwRd10g0CTyuUTFmFUUgK2JQvCAQkHDRsCo4IFeR4AUNkct4AWRVdOC7f1QdOZw5c6ZnnKozM7/363n66arqrv7+qnvm03V+1fWrVBWSpHY8aL4bIEkal8EvSY0x+CWpMQa/JDXG4Jekxmw63w2YxIIFC2rRokXz3QxJ2qBcfvnlP6iqhTOXbxDBv2jRIpYtWzbfzZCkDUqS78623K4eSWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqzAZx5q6kVVt09Nmj1brp2P1Hq6XhuMcvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1JjBgz/JJkm+luSsfn6XJJcmuTHJPybZfOg2SJLuN8Ye/1HA9dPmjwPeXVWLgR8Dh43QBklSb9DgT7IzsD/w4X4+wN7AJ/unnAIcOGQbJEkPNPQe/3uAPwPu6+e3B+6oqhX9/M3ATrOtmOTwJMuSLFu+fPnAzZSkdgwW/EmeD9xeVZdPXzzLU2u29avqhKpaUlVLFi5cOEgbJalFmw742nsCByR5HrAlsC3dXwDbJdm03+vfGfj+gG2QJM0w2B5/Vb2pqnauqkXAwcCFVfVS4CLgoP5pS4HPDtUGSdLK5uN3/H8OvCHJN+n6/E+chzZIUrOG7Or5laq6GLi4n/42sNsYdSVJK/PMXUlqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1ZrDgT7JlksuSXJXkuiTH9Mt3SXJpkhuT/GOSzYdqgyRpZUPu8d8D7F1VTwV2BfZLsjtwHPDuqloM/Bg4bMA2SJJmWG3wJ3lcki366b2SHJlku9WtV527+9nN+lsBewOf7JefAhy4Vi2XJK2VSfb4zwDuTfIfgBOBXYCPTfLiSTZJciVwO3A+8C3gjqpa0T/lZmCnVax7eJJlSZYtX758knKSpAlMEvz39UH9QuA9VfV6YMdJXryq7q2qXYGdgd2AJ872tFWse0JVLamqJQsXLpyknCRpApME/y+THAIsBc7ql222JkWq6g7gYmB3YLskm/YP7Qx8f01eS5L065kk+F8O7AH8VVV9J8kuwEdXt1KShVPHApJsBewLXA9cBBzUP20p8Nm1abgkae1suvqn8JyqOnJqpg//n02w3o7AKUk2ofuCOb2qzkrydeDjSd4BfI3uuIEkaSSTBP9S4L0zlv3xLMseoKquBp42y/Jv0/X3S5LmwSqDv+/X/yNglyRnTnvoIcAPh26YJGkYc+3xfwm4BVgAvGva8p8AVw/ZKEnScFYZ/FX1XeC7wB5JHgMsrqp/7g/UbkX3BSBJ2sBMcubuK+nOtP1gv2hn4DNDNkqSNJxJfs55BLAncBdAVd0I7DBkoyRJw5kk+O+pql9MzfQnX816tq0kaf03SfB/Icmbga2SPAf4BPC5YZslSRrKJMF/NLAcuAZ4FXAO8JYhGyVJGs5qT+CqqvuAD/U3SdIGbrXBn+QaVu7TvxNYBryjqjyZS5I2IJMM2fBPwL3cPwb/wf39XcDJwH9Z982SJA1lkuDfs6r2nDZ/TZL/W1V7Jjl0qIZJkoYxycHdbZI8c2omyW7ANv3sitlXkSStrybZ4z8MOCnJVNj/BDgsydbA3wzWMknSIOYM/iQPAh5bVb+V5KFA+qtpTTl90NZJG4hFR589Wq2bjt1/tFraOM3Z1dP/lPM1/fSdM0JfkrQBmqSP//wkb0zy6CQPn7oN3jJJ0iAm6eN/RX9/xLRlBTx23TdHkjS0Sc7c3WWMhkiSxjHJHj9Jngw8CdhyallV/cNQjZIkDWeSIRv+AtiLLvjPAZ4LXAIY/JK0AZrk4O5BwD7ArVX1cuCpwBaDtkqSNJhJgv9n/c86VyTZFrgdD+xK0gZrkj7+ZUm2oxuW+XLgbuCyQVslSRrMJL/qeXU/+YEk5wLbVtXVwzZLkjSU1Xb1JLlgarqqbqqqq6cvkyRtWFa5x59kS+DBwIIkDwPSP7Qt8KgR2iZJGsBcXT2vAl5HF/KXc3/w3wW8f+B2SZIGssrgr6r3Au9N8tqqet+IbZIkDWi1ffyGviRtXCb5Hb8kaSOyyuBPsmd/71m6krQRmWuP//j+/stjNESSNI65ftXzyyQnATslOX7mg1V15HDNkiQNZa7gfz6wL7A33c85JUkbgbl+zvkD4ONJrq+qq0ZskyRpQJP8queHST6d5PYktyU5I8nOg7dMkjSISYL/JOBMujN4dwI+1y+bU39x9ouSXJ/kuiRH9csfnuT8JDf29w/7dTZAkrRmJgn+HarqpKpa0d9OBhZOsN4K4E+r6onA7sARSZ4EHA1cUFWLgQv6eUnSSCYJ/uVJDk2ySX87FPjh6laqqluq6op++ifA9XR/MbwAOKV/2inAgWvXdEnS2pgk+F8B/CFwK3AL3aUYX7EmRZIsAp4GXAo8oqpuge7LAdhhFescnmRZkmXLly9fk3KSpDlMciGW7wEHrG2BJNsAZwCvq6q7kqxulam6JwAnACxZsqTWtr4k6YEmufTiWkuyGV3on1pVn+oX35Zkx6q6JcmOdNfwlbSBWXT02aPWu+nY/UettzEbbJC2dLv2JwLXV9XfTnvoTGBpP70U+OxQbZAkrWzIPf49gZcB1yS5sl/2ZuBY4PQkhwHfA148YBskSTNMHPxJdgf+GtgCeGdVfWau51fVJdx/1a6Z9pm4hZKkdWqua+4+sqpunbboDXQHeQN8CZgz+CVJ66e59vg/kORyur37nwN3AH8E3Ed33V1J0gZolQd3q+pA4ErgrCQvo7vw+n3Ag/GkK0naYM35q56q+hzwn4HtgE8BN1TV8VXlGVWStIGa69KLByS5BLgQuBY4GHhhktOSPG6sBkqS1q25+vjfAewBbAWcU1W7AW9Ishj4K7ovAknSBmau4L+TLty3YtrZtVV1I4a+JG2w5urjfyHdgdwVdL/mkSRtBFZ36cX3jdgWSdIIBhurR5K0fjL4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNWaw4E/ykSS3J7l22rKHJzk/yY39/cOGqi9Jmt2Qe/wnA/vNWHY0cEFVLQYu6OclSSMaLPir6ovAj2YsfgFwSj99CnDgUPUlSbMbu4//EVV1C0B/v8Oqnpjk8CTLkixbvnz5aA2UpI3dentwt6pOqKolVbVk4cKF890cSdpojB38tyXZEaC/v33k+pLUvLGD/0xgaT+9FPjsyPUlqXlD/pzzNODLwOOT3JzkMOBY4DlJbgSe089Lkka06VAvXFWHrOKhfYaqKUlavfX24K4kaRgGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqzGDj8UtjW3T02aPWu+nY/UetJ60r7vFLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEO0iZpg+OAfL8e9/glqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMv+PXOuXvq6X137zs8SfZL8kNSb6Z5Oj5aIMktWr04E+yCfB+4LnAk4BDkjxp7HZIUqvmo6tnN+CbVfVtgCQfB14AfH0e2jKoMbs9ZnZ5zGdtSeu3VNW4BZODgP2q6k/6+ZcBz6yq18x43uHA4f3s44EbRm0oLAB+MHLN+a7d4ja3WrvFbW6x9mOqauHMhfOxx59Zlq307VNVJwAnDN+c2SVZVlVLWqrd4ja3WrvFbW659kzzcXD3ZuDR0+Z3Br4/D+2QpCbNR/B/FVicZJckmwMHA2fOQzskqUmjd/VU1YokrwHOAzYBPlJV143djgnMWzfTPNZucZtbrd3iNrdc+wFGP7grSZpfDtkgSY0x+CWpMQb/DEk+kuT2JNeOXPfRSS5Kcn2S65IcNWLtLZNcluSqvvYxY9Xu62+S5GtJzhq57k1JrklyZZJlI9feLsknk3yj/8z3GKnu4/vtnbrdleR1Y9Tu67++/zd2bZLTkmw5Ut2j+prXDb29s2VIkocnOT/Jjf39w4Zsw+oY/Cs7GdhvHuquAP60qp4I7A4cMeJQFvcAe1fVU4Fdgf2S7D5SbYCjgOtHrDfd71XVrvPw++r3AudW1ROApzLS9lfVDf327go8A/gp8OkxaifZCTgSWFJVT6b7ccfBI9R9MvBKulEDngo8P8niAUuezMoZcjRwQVUtBi7o5+eNwT9DVX0R+NE81L2lqq7op39CFwQ7jVS7qurufnaz/jbKUf8kOwP7Ax8eo976IMm2wLOBEwGq6hdVdcc8NGUf4FtV9d0Ra24KbJVkU+DBjHMOzxOBr1TVT6tqBfAF4IVDFVtFhrwAOKWfPgU4cKj6kzD410NJFgFPAy4dseYmSa4EbgfOr6qxar8H+DPgvpHqTVfA55Nc3g8RMpbHAsuBk/ourg8n2XrE+lMOBk4bq1hV/Tvwv4DvAbcAd1bV50cofS3w7CTbJ3kw8DweeBLpGB5RVbdAt5MH7DBy/Qcw+NczSbYBzgBeV1V3jVW3qu7t//zfGdit//N4UEmeD9xeVZcPXWsV9qyqp9ONFHtEkmePVHdT4OnA/66qpwH/j5H/9O9PnjwA+MSINR9Gt+e7C/AoYOskhw5dt6quB44DzgfOBa6i61ptlsG/HkmyGV3on1pVn5qPNvRdDhczznGOPYEDktwEfBzYO8lHR6gLQFV9v7+/na6fe7eRSt8M3Dztr6pP0n0RjOm5wBVVdduINfcFvlNVy6vql8CngN8eo3BVnVhVT6+qZ9N1w9w4Rt1pbkuyI0B/f/vI9R/A4F9PJAldn+/1VfW3I9demGS7fnoruv+g3xi6blW9qap2rqpFdN0OF1bV4HuAAEm2TvKQqWng9+m6BAZXVbcC/5bk8f2ifRh/WPJDGLGbp/c9YPckD+7/ve/DSAe1k+zQ3/8G8AeMv+1nAkv76aXAZ0eu/wBeenGGJKcBewELktwM/EVVnThC6T2BlwHX9H3tAG+uqnNGqL0jcEp/kZwHAadX1ag/rZwHjwA+3eUPmwIfq6pzR6z/WuDUvsvl28DLxyrc93M/B3jVWDUBqurSJJ8ErqDravka4w1jcEaS7YFfAkdU1Y+HKjRbhgDHAqcnOYzuC/DFQ9WfhEM2SFJj7OqRpMYY/JLUGINfkhpj8EtSYwx+SWqMwd+oJJXkXdPm35jkbevotU9OctC6eK3V1HlxP7LlRUPXGlKSXZM8by3WOy3J1f2Il3+ZZN81XP+mJAtmWf7mNW3L2ljb7davz+Bv1z3AH8z2H38+9ecSTOow4NVV9XtDtWcku9KNHzOxJI8EfruqnlJV766qt1bVP6+j9qxx8K/h5zZljbdb64bB364VdCfPvH7mAzP32JPc3d/vleQLSU5P8q9Jjk3y0nRj+V+T5HHTXmbfJP/SP+/5/fqbJHlnkq/2e6qvmva6FyX5GHDNLO05pH/9a5Mc1y97K/As4ANJ3jnj+Q9K8vf92OtnJTlnanum7+UmWZLk4n5663TjqH+1HzjtBf3y3+y378q+zYv7556d7voF1yZ5Sf/cZ/Tvz+VJzpt2iv6RSb7er//xGW3dHPhL4CV9jZekG7v9M/3zv5LkKbN8fp8HdujX+Z3pn1m/jcckuaJ/357QL98+yef77fsgkFne62PpRs+8Msmp/bLP9Nt0XaYNZpfk7v4vjUuBPZI8L931BS5Jcnz66yvM9t7Ott2zbKOGUlXeGrwBdwPbAjcBDwXeCLytf+xk4KDpz+3v9wLuoDvTdwvg34Fj+seOAt4zbf1z6XYsFtONTbMlcDjwlv45WwDL6Abs2otuoLJdZmnno+jOdFxId4bthcCB/WMX043tPnOdg4Bz+vqPBH48tT399i7op5cAF/fTfw0c2k9vB/wrsDXwPuCl/fLNga2AFwEfmlbvoXRDWX8JWNgvewnwkX76+8AWU689S3v/GPi7afPvoztjHGBv4MpZ1lkEXDtt/lefWb+Nr+2nXw18uJ8+HnhrP70/3eikC2b7tzFj/uH9/VZ0w1ps388X8If99JbAv019hnRDIpy1mvf2Advtbbybe/wNq270z3+guzjGpL5a3bUD7gG+RbfnCd2e+qJpzzu9qu6rqhvphiR4At14OP813ZAUlwLb030xAFxWVd+Zpd5/ogvn5dWNpX4q3Vj2c3kW8Im+/q3AJMcAfh84um/bxXRB9hvAl4E3J/lz4DFV9bN+W/dNclyS36mqO4HHA08Gzu9f4y10I50CXE03PMOhTDYq5LOA/wNQVRcC2yd56ATrTTc1yN/l3P+5PBv4aP+6Z9N9IU7iyCRXAV+hG8546jO7l25QQeg+329P+wynj4WzqvdW88SxevQeurFTTpq2bAV9N2CS0O3pTrln2vR90+bv44H/nmaOBVJ0XQuvrarzpj+QZC+6Pf7ZrNQdMYG51vnVttEF0PR1XlRVN8x4/vV9V8b+wHlJ/qSqLkzyDLr+6b9J8nm60T2vq6rZLqG4P13oHgD8zyS/2X+JrUn713RslanP5V7m/lzm1H82+wJ7VNVP+66xqfft51V179RT53oZZnlvkzxzTdqidcc9/sZV1Y+A0+kOlE65ie6yfNCNn77ZWrz0i/u+9sfRXXjkBuA84L+nG36aJP8xq78AyaXA7yZZkO4A4iF0V1CayyXAi/r6j6DrSppyE/dv24umLT8PeG3/RUeSp/X3j6Xbkz2eboTFpyR5FPDTqvoo3YVFnt5v38L0185Nsll/fOBBwKOr6iK6C85sB2wzo70/AR4ybf6LwEv719kL+EGtm2szTH/d5wKruu7rL6c+I7purB/3of8EusuCzuYbwGPTXUQIuq6uKbO+t6y83RqJwS+AdwHTf93zIbqwvQx4JqveG5/LDXQB/U/Af6uqn9NdXvHrwBXpLkT9QVbzV2d1Vyt6E113zVV0Y8ivbkjbM+iOK0zVuBS4s3/sGOC9Sf6Fbm94ytvpvuCu7tv29n75S4Br+26KJ9B1jf0WcFm/7H8A76iqX9AdWziu7xa5km6s+U2Ajya5hm40ynfXypdZvAh40rSDnG8DliS5mm5Ux6WsG8fQXYnqCrrul++t4nkn0L0Pp9Idq9m0b8vb6bp7VtJ3gb0aODfJJcBt3P+er+q9nbndGomjc2qjlGSbqro73VC8l9FdbevW+W7Xxmzaex7g/cCNVfXu+W6XVmYfvzZWZ6W7uMzmwNsN/VG8MslSuvf8a3R/bWk95B6/JDXGPn5JaozBL0mNMfglqTEGvyQ1xuCXpMb8f6nfSCtsoNDTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(expectation, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mean = 6.81 ± 1.00 guesses; worst = 10\n",
      "CPU times: user 41.2 s, sys: 242 ms, total: 41.5 s\n",
      "Wall time: 42.1 s\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZh0lEQVR4nO3de7wdZX3v8c+XJJBwM0I2GIm4A01BSyXoboTG0hjAIuHFpWKBIie11NAjVy+nBupRqNqGl0UUj0eNAkkPCI1cBAMCKSRSjhrYgZCLgUYx0kggG+WWoyJJfueP59llsbP23ishMyvJ832/Xuu1ZmbNrN9v1kp+a/YzM8+jiMDMzMqxU7sTMDOzernwm5kVxoXfzKwwLvxmZoVx4TczK8zQdifQilGjRkVnZ2e70zAz264sWrTomYjo6Lt8uyj8nZ2ddHd3tzsNM7PtiqSfN1vuph4zs8K48JuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCbBd37ppZ/zqn315brFUzptQWy6rjI34zs8K48JuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCVF74JQ2R9LCkuXl+rKSFklZK+ldJO1edg5mZvaKOI/4LgBUN85cBV0TEOOBZ4KwacjAzs6zSwi9pDDAF+GaeFzAZuDGvMhs4qcoczMzs1ao+4v8i8HfAxjy/N/BcRKzP86uB/ZptKGmapG5J3T09PRWnaWZWjsoKv6TjgbURsahxcZNVo9n2ETEzIroioqujo6OSHM3MSlRlt8wTgRMkHQcMB/Yk/QUwUtLQfNQ/BniywhzMzKyPyo74I+KiiBgTEZ3AacC9EXEGMB84Ja82Fbi1qhzMzGxT7biO/xPARyX9hNTmf1UbcjAzK1YtI3BFxAJgQZ5+HJhQR1wzM9uU79w1MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhXPjNzArjwm9mVhgXfjOzwrjwm5kVxoXfzKwwVY65O1zSA5IekbRc0qV5+SxJP5O0OD/GV5WDmZltqsqBWF4CJkfEOknDgPslfS+/9j8i4sYKY5uZWT8qK/wREcC6PDssP6KqeGZm1ppK2/glDZG0GFgLzIuIhfmlz0laIukKSbtUmYOZmb1apYU/IjZExHhgDDBB0iHARcDBwB8Be5EGX9+EpGmSuiV19/T0VJmmmVlRarmqJyKeIw22fmxErInkJeAa+hl4PSJmRkRXRHR1dHTUkaaZWRGqvKqnQ9LIPD0COBp4VNLovEzAScCyqnIwM7NNVXlVz2hgtqQhpB+YORExV9K9kjoAAYuBv60wBzMz66PKq3qWAIc1WT65qphmZjY437lrZlYYF34zs8K48JuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhXPjNzApT5dCLwyU9IOkRScslXZqXj5W0UNJKSf8qaeeqcjAzs01VecT/EjA5Ig4FxgPHSjocuAy4IiLGAc8CZ1WYg5mZ9VFZ4Y9kXZ4dlh8BTAZuzMtnkwZcNzOzmlTaxi9piKTFwFpgHvBT4LmIWJ9XWQ3s18+20yR1S+ru6empMk0zs6JUWvgjYkNEjAfGABOAtzRbrZ9tZ0ZEV0R0dXR0VJmmmVlRarmqJyKeAxYAhwMjJQ3NL40BnqwjBzMzS6q8qqdD0sg8PQI4GlgBzAdOyatNBW6tKgczM9vU0MFX2WKjgdmShpB+YOZExFxJPwZukPRZ4GHgqgpzMDOzPior/BGxBDisyfLHSe39ZmbWBr5z18ysMIMWfkkHStolT0+SdH5v272ZmW1/WjnivwnYIOn3SO3xY4FvVZqVmZlVppXCvzHfcHUy8MWI+AjpxK2ZmW2HWin8L0s6nXTp5dy8bFh1KZmZWZVaKfwfBI4APhcRP5M0Fri22rTMzKwqrVzOeUxEnN87k4v/byrMyczMKtTKEf/UJsv+aivnYWZmNen3iD+36/8lMFbSbQ0v7QH8surEzMysGgM19fwAWAOMAi5vWP4isKTKpMzMrDr9Fv6I+Dnwc+AISW8GxkXEv+UO10aQfgDMDOicfnut8VbNmFJrPNuxtHLn7odII2Z9PS8aA3ynyqTMzKw6rZzcPQeYCLwAEBErgX2qTMrMzKrTSuF/KSJ+1zuTB1FpOmqWmZlt+1op/N+XdDEwQtIxwLeB71ablpmZVaWVwj8d6AGWAmcDdwCfHGwjSW+SNF/SCknLJV2Ql18i6ReSFufHca9lB8zMbPMMeuduRGwEvpEfm2M98LGIeEjSHsAiSfPya1dExD9v5vuZmdlWMGjhl7SUTdv0nwe6gc9GRNObuSJiDek+ACLiRUkrgP1eW7pmZvZatdLU8z3gduCM/PgucB/wFDCrlSCSOknDMC7Mi86VtETS1ZJe38820yR1S+ru6elpJYyZmbWglcI/MSIuioil+fH3wKSIuAzoHGxjSbuTBnO5MCJeAL4KHAiMJ/1FcHmz7SJiZkR0RURXR0dHi7tjZmaDaaXw7y7pnb0zkiYAu+fZ9QNtKGkYqehfFxE3A0TE0xGxoeHcgQdeNzOrUSvdMp8FXJOP3CF11XCWpN2Af+pvI0kiDdW4IiK+0LB8dG7/hzSq17ItytzMzLbIgIVf0k7AARHxh5JeByginmtYZc4Am08EzgSWSlqcl10MnC5pPOmE8SrSJaJmZlaTAQt/RGyUdC4wJyKe35w3joj7ATV56Y7NeR8zM9u6Wmnjnyfp4/mGrL16H5VnZmZmlWiljf+v8/M5DcsCOGDrp2NmZlVr5c7dsXUkYmZm9WjliB9JhwBvBYb3LouIf6kqKTMzq04rXTZ8GphEKvx3AO8F7gdc+M3MtkOtnNw9BTgKeCoiPggcCuxSaVZmZlaZVgr/b/Jdtusl7QmsxSd2zcy2W6208XdLGknqXmERsA54oNKszMysMq1c1fPhPPk1SXcCe0bEkmrTMjOzqgza1CPpnt7piFgVEUsal5mZ2fal3yN+ScOBXYFRuc/83u4X9gTeWENuZmZWgYGaes4GLiQV+UW8UvhfAL5ScV5mZlaRfgt/RHwJ+JKk8yLiyzXmZGZmFRq0jd9F38xsx9LKdfxmZrYD6bfwS5qYn7foLt3cjfN8SSskLZd0QV6+l6R5klbm56aDrZuZWTUGOuK/Mj//cAvfez3wsYh4C3A4cI6ktwLTgXsiYhxwT543M7OaDHRVz8uSrgH2k3Rl3xcj4vyB3jiPq7smT78oaQWwH3AiqdM3gNnAAuATm525mZltkYEK//HA0cBk0uWcW0xSJ3AYsBDYt3ew9YhYI2mffraZBkwD2H///V9LeDMzazDQ5ZzPADdIWhERj2xpAEm7AzcBF0bEC1KzYXibxp8JzATo6uqKLY1vZmav1spVPb+UdIuktZKelnSTpDGtvLmkYaSif11E3JwXPy1pdH59NKm3TzMzq0krhf8a4DbSHbz7Ad/NywakdGh/FbAiIr7Q8NJtwNQ8PRW4dXMSNjOz16aVbpn3iYjGQj9L0oUtbDcROBNYKmlxXnYxMAOYI+ks4Ang/ZuTsJltGzqn315rvFUzptQab0fWSuHvkfQB4Po8fzrwy8E2ioj7eaV/n76Oai09MzPb2lpp6vlr4C+Ap0iXZ56Sl5mZ2XaolYFYngBOqCEXMzOrgfvqMTMrjAu/mVlhXPjNzArTcuGXdLikeyX9X0knVZmUmZlVZ6Axd98QEU81LPoo6SSvgB8A36k4NzMzq8BAV/V8TdIi4PMR8VvgOeAvgY2kcXfNzGw71G9TT0ScBCwG5ko6kzTw+kZgV8BNPWZm26kB2/gj4rvAnwEjgZuBxyLiyojoqSM5MzPb+gYaevEESfcD9wLLgNOAkyVdL+nAuhI0M7Ota6A2/s8CRwAjgDsiYgLwUUnjgM+RfgjMzGw7M1Dhf55U3EfQ0Gd+RKzERd/MbLs1UBv/yaQTuetJV/OYmdkOYLChF79cYy5mZlaDyrpskHR1Hq5xWcOySyT9QtLi/DiuqvhmZtZclX31zAKObbL8iogYnx93VBjfzMyaqKzwR8R9wK+qen8zM9sy7eid81xJS3JT0Ov7W0nSNEndkrp7eny/mJnZ1lJ34f8qcCAwnjSM4+X9rRgRMyOiKyK6Ojo66srPzGyHV2vhj4inI2JDRGwEvgFMqDO+mZnVXPgljW6YPZnUFYSZmdVo0MHWt5Sk64FJwChJq4FPA5MkjQcCWAWcXVV8MzNrrrLCHxGnN1l8VVXxzMysNR5z18ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrjAu/mVlhXPjNzArjwm9mVhgXfjOzwlRW+PNg6mslLWtYtpekeZJW5ud+B1s3M7NqVHnEPws4ts+y6cA9ETEOuCfPm5lZjSor/BFxH/CrPotPBGbn6dnASVXFNzOz5upu4983ItYA5Od9+ltR0jRJ3ZK6e3p6akvQzGxHt82e3I2ImRHRFRFdHR0d7U7HzGyHUXfhf1rSaID8vLbm+GZmxau78N8GTM3TU4Fba45vZla8Ki/nvB74IXCQpNWSzgJmAMdIWgkck+fNzKxGQ6t644g4vZ+XjqoqppmZDW6bPblrZmbVcOE3MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGhd/MrDAu/GZmhXHhNzMrTGVdNpjVrXP67bXGWzVjSq3xzLYWH/GbmRXGhd/MrDAu/GZmhXHhNzMrTFtO7kpaBbwIbADWR0RXO/IwMytRO6/qeXdEPNPG+GZmRXJTj5lZYdpV+AO4W9IiSdPalIOZWZHa1dQzMSKelLQPME/SoxFxX+MK+QdhGsD+++/fjhzNzHZIbTnij4gn8/Na4BZgQpN1ZkZEV0R0dXR01J2imdkOq/bCL2k3SXv0TgPvAZbVnYeZWana0dSzL3CLpN7434qIO9uQh5lZkWov/BHxOHBo3XHNzCzx5ZxmZoVx4TczK4wLv5lZYVz4zcwK4xG4zGy749HWXhsf8ZuZFcaF38ysMC78ZmaFceE3MyuMC7+ZWWFc+M3MCuPCb2ZWGBd+M7PCuPCbmRXGd+7aVuU7Ks22fT7iNzMrTFsKv6RjJT0m6SeSprcjBzOzUtXe1CNpCPAV4BhgNfCgpNsi4sd152Jmtrl2hObMdrTxTwB+kodgRNINwInADlf42/kPpM7Ybmc3274oIuoNKJ0CHBsRf5PnzwTeGRHn9llvGjAtzx4EPFZrojAKeKbmmO2OXeI+lxq7xH0uMfabI6Kj78J2HPGrybJNfn0iYiYws/p0mpPUHRFdJcUucZ9LjV3iPpccu692nNxdDbypYX4M8GQb8jAzK1I7Cv+DwDhJYyXtDJwG3NaGPMzMilR7U09ErJd0LnAXMAS4OiKW151HC9rWzNTG2CXuc6mxS9znkmO/Su0nd83MrL18566ZWWFc+M3MCuPC34ekqyWtlbSs5rhvkjRf0gpJyyVdUGPs4ZIekPRIjn1pXbEbchgi6WFJc2uOu0rSUkmLJXXXGHekpBslPZq/8yNqintQ3tfexwuSLqwjdo7/kfxvbJmk6yUNrynuBTnm8jr2t1kdkbSXpHmSVubn11edR39c+Dc1Czi2DXHXAx+LiLcAhwPnSHprTbFfAiZHxKHAeOBYSYfXFLvXBcCKmmP2endEjK/5GusvAXdGxMHAodS07xHxWN7X8cA7gF8Dt9QRW9J+wPlAV0QcQrq447Qa4h4CfIjUa8ChwPGSxlUcdhab1pHpwD0RMQ64J8+3hQt/HxFxH/CrNsRdExEP5ekXSYVgv5piR0Ssy7PD8qO2s/6SxgBTgG/WFbOdJO0JHAlcBRARv4uI59qQylHATyPi5zXGHAqMkDQU2JV67uF5C/CjiPh1RKwHvg+cXGXAfurIicDsPD0bOKnKHAbiwr8NktQJHAYsrDHmEEmLgbXAvIioLTbwReDvgI01xuwVwN2SFuVuQupwANADXJObt74pabeaYjc6Dbi+rmAR8Qvgn4EngDXA8xFxdw2hlwFHStpb0q7Acbz6JtK67BsRayAd6AH7tCEHwIV/myNpd+Am4MKIeKGuuBGxIf/5PwaYkP88rpyk44G1EbGojnhNTIyItwPvJTWvHVlDzKHA24GvRsRhwP+j5j/7882TJwDfrjHm60lHvWOBNwK7SfpA1XEjYgVwGTAPuBN4hNS0WiwX/m2IpGGkon9dRNzcjhxyk8MC6jvPMRE4QdIq4AZgsqRra4pNRDyZn9eS2ron1BB2NbC64a+qG0k/BHV6L/BQRDxdY8yjgZ9FRE9EvAzcDPxxHYEj4qqIeHtEHElqgllZR9w+npY0GiA/r21DDoAL/zZDkkhtvisi4gs1x+6QNDJPjyD9B320jtgRcVFEjImITlLTw70RUflRIICk3STt0TsNvIfULFCpiHgK+E9JB+VFR1F/t+SnU2MzT/YEcLikXfO/96Oo6aS2pH3y8/7An1P/vkPqmmZqnp4K3NqGHACPubsJSdcDk4BRklYDn46Iq2oIPRE4E1ia29oBLo6IO2qIPRqYnQfJ2QmYExG1XlbZJvsCt6QaxFDgWxFxZ02xzwOuy00ujwMfrCkuuZ37GODsumICRMRCSTcCD5GaWh6mvm4MbpK0N/AycE5EPFtlsGZ1BJgBzJF0FulH8P1V5jBgfu6ywcysLG7qMTMrjAu/mVlhXPjNzArjwm9mVhgXfjOzwrjwF0xSSLq8Yf7jki7ZSu89S9IpW+O9Bonz/ty75fyqY1VJ0nhJx23BdtdLWpJ7vfwHSUdv5varJI1qsvzizc1lS2zpfttr48JftpeAP2/2H7+d8v0ErToL+HBEvLuqfGoyntSHTMskvQH444h4W0RcERGfioh/20r5bHbh38zvrddm77e9di78ZVtPuoHmI31f6HvELmldfp4k6fuS5kj6D0kzJJ2h1J//UkkHNrzN0ZL+Pa93fN5+iKTPS3owH6me3fC+8yV9C1jaJJ/T8/svk3RZXvYp4F3A1yR9vs/6O0n637n/9bmS7ujdn8ajXEldkhbk6d2U+lF/MHeedmJe/gd5/xbnnMfldW9XGsNgmaRT87rvyJ/PIkl3Ndyif76kH+ftb+iT687APwCn5hinKvXd/p28/o8kva3J93c3sE/e5k8av7O8j5dKeih/bgfn5XtLujvv39cBNfmsZ5B60Fws6bq87Dt5n5aroTM7SevyXxoLgSMkHac0xsD9kq5UHl+h2WfbbL+b7KNVISL8KPQBrAP2BFYBrwM+DlySX5sFnNK4bn6eBDxHutt3F+AXwKX5tQuALzZsfyfp4GIcqX+a4cA04JN5nV2AblKnXZNInZWNbZLnG0l3OnaQ7rC9Fzgpv7aA1L97321OAe7I8d8APNu7P3l/R+XpLmBBnv5H4AN5eiTwH8BuwJeBM/LynYERwPuAbzTEex2pO+sfAB152anA1Xn6SWCX3vduku9fAf+rYf7LpLvGASYDi5ts0wksa5j/r+8s7+N5efrDwDfz9JXAp/L0FFLvpKOa/dvoM79Xfh5B6tZi7zwfwF/k6eHAf/Z+h6RuEeYO8tm+ar/9qOfhI/7CReoB9F9IA2S06sFI4we8BPyUdOQJ6Ui9s2G9ORGxMSJWkrolOJjUH85/U+qWYiGwN+mHAeCBiPhZk3h/RCrOPZH6U7+O1J/9QN4FfDvHfwpo5RzAe4DpObcFpEK2P/BD4GJJnwDeHBG/yft6tKTLJP1JRDwPHAQcAszL7/FJUm+nAEtIXTR8gNZ6hnwX8H8AIuJeYG9Jr2thu0a9Hf0t4pXv5Ujg2vy+t5N+EFtxvqRHgB+RujTu/c42kDoWhPT9Pt7wHTb2h9PfZ2tt4L56DFJ/+A8B1zQsW09uCpQk0pFur5capjc2zG/k1f+m+vYHEqSmhfMi4q7GFyRNIh3xN7NJc0QLBtrmv/aNVIAat3lfRDzWZ/0VuSljCnCXpL+JiHslvYPUPv1Pku4m9e65PCKaDaM4hVR0TwD+p6Q/yD9im5P/5vav0vu9bGDg72VA+bs5GjgiIn6dm8Z6P7ffRsSG3lUHehuafLaS3rk5udjW4SN+IyJ+BcwhnSjttYo0NB+kPtSHbcFbvz+3tR9IGnzkMeAu4L8rdUGNpN/X4IOQLAT+VNIopROIp5NGURrI/cD7cvx9SU1JvVbxyr69r2H5XcB5+YcOSYfl5wNIR7JXknpYfJukNwK/johrSYOLvD3vX4fy+LmShuXzAzsBb4qI+aQBZ0YCu/fJ90Vgj4b5+4Az8vtMAp6JrTM+Q+P7vhfob9zXl3u/I1Iz1rO56B9MGhq0mUeBA5QGEoLU1NWr6WfLpvttNXDht16XA41X93yDVGwfAN5J/0fjA3mMVKC/B/xtRPyWNLzij4GHlAai/jqD/OUZabSii0jNNY+Q+pEfrEvbm0jnFXpjLASez69dCnxJ0r+TjoZ7fYb0A7ck5/aZvPxUYFlupjiY1DT2h8ADednfA5+NiN+Rzi1clptFFpP6mx8CXCtpKalHyiti06EW5wNvbTjJeQnQJWkJqVfHqWwdl5JGo3qI1PzyRD/rzSR9DteRztUMzbl8htTcs4ncBPZh4E5J9wNP88pn3t9n23e/rQbundN2WJJ2j4h1St3xPkAabeupdue1I2v4zAV8BVgZEVe0Oy97Nbfx245srtIAMzsDn3HRr8WHJE0lfeYPk/7asm2Mj/jNzArjNn4zs8K48JuZFcaF38ysMC78ZmaFceE3MyvM/wdcrWs7N81NHQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%time show_metric(negative_entropy, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Jotto Summary\n",
    "\n",
    "Here's a table of results:\n",
    "\n",
    "|<br>Algorithm|Consistent<br>Mean (Max)|Inconsistent<br>Mean (Max)|\n",
    "|--|--|--|\n",
    "|baseline random guesser|7.38 (18)|  |\n",
    "|min(max(partition_counts))|7.15 (18)|7.05 (10)|\n",
    "|min(expectation(partition_counts))|7.14 (17)|6.84 (10)|\n",
    "|max(entropy(partition_counts))|7.09 (19)|6.81 (10)|\n",
    "\n",
    "So we started out with a mean of 7.38 and a worst score of 18 with the random guesser, and were able to improve by half a guess to a mean of 6.81 and a worst score of 10 using  maximization of entropy over partition counts, with inconsistent guesses allowed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next Steps\n",
    "\n",
    "There are many directions you could take this if you are interested:\n",
    "- Try different length words.\n",
    "- Allow words with repeated letters, like `stars`.\n",
    "- Have each reply consist of two numbers: the number of letters in common with the target, and the number of letters that are in the exact correct position.\n",
    "- Apply the same techniques to related games, like Mastermind.\n",
    "- Examine the strategy that a chooser would use, and how the guesser responds to that. Is there an equilibrium?\n",
    "- Our `make_tree` function is **greedy** in that it guesses the word that minimizes some metric of the current situation without looking ahead to future branches in the tree. Can you get better performance by doing some **look-ahead**? Perhaps with a beam search?\n",
    "- Can you improve a tree by editing it? Given a tree, look for interior nodes that have a worse-than-expected average score, and see if the node can be replaced with something better (covering the same target words).\n",
    "- Research what other computer scientists have done with [Jotto](https://arxiv.org/abs/1107.3342) or [Mastermind](http://serkangur.freeservers.com/).\n",
    "- What else can you explore?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}