notebooks/Building a simple agent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/microsoft/TextWorld/blob/main/notebooks/Building%20a%20simple%20agent.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a simple agent with TextWorld\n",
    "\n",
    "This tutorial outlines the steps to build an agent that learns how to play __choice-based__ text-based games generated with TextWorld."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prerequisite\n",
    "Install TextWorld as described in the [README.md](https://github.com/microsoft/TextWorld#readme). Most of the time, a simple `pip install` should work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install textworld\n",
    "!pip install gym==0.21"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and [PyTorch](https://pytorch.org/) (tested with both v1.8.2 and v.1.9.0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install torch torchvision torchaudio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Optional]** Download all data beforehand. Otherwise, they are going to be generated as needed (slower)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://aka.ms/textworld/notebooks/data.zip\n",
    "!unzip -nq data.zip && rm -f data.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning challenges\n",
    "Training an agent such that it can learn how to play text-based games is not trivial. Among other challenges, we have to deal with\n",
    "\n",
    "1. a combinatorial action space (that grows w.r.t. vocabulary)\n",
    "2. a really sparse reward signal.\n",
    "\n",
    "To ease the learning process, we will be requesting additional information alongside the game's narrative (as covered in [Playing TextWorld generated games with OpenAI Gym](Playing%20TextWorld%20generated%20games%20with%20OpenAI%20Gym.ipynb#Interact-with-the-game)). More specifically, we will request the following information:\n",
    "\n",
    "- __Description__:\n",
    "For every game state, we will get the output of the `look` command which describes the current location;\n",
    "\n",
    "- __Inventory__:\n",
    "For every game state, we will get the output of the `inventory` command which describes the player's inventory;\n",
    "\n",
    "- __Admissible commands__:\n",
    "For every game state, we will get the list of commands guaranteed to be understood by the game interpreter;\n",
    "\n",
    "- __Intermediate reward__:\n",
    "For every game state, we will get an intermediate reward which can either be:\n",
    "  - __-1__: last action needs to be undone before resuming the quest\n",
    "  -  __0__: last action didn't affect the quest\n",
    "  -  __1__: last action brought us closer to completing the quest\n",
    "\n",
    "- __Entities__:\n",
    "For every game, we will get a list of entity names that the agent can interact with.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple test games\n",
    "We can use TextWorld to generate a few simple games with the following handcrafted world\n",
    "```\n",
    "                     Bathroom\n",
    "                        +\n",
    "                        |\n",
    "                        +\n",
    "    Bedroom +-(d1)-+ Kitchen +--(d2)--+ Backyard\n",
    "      (P)               +                  +\n",
    "                        |                  |\n",
    "                        +                  +\n",
    "                   Living Room           Garden\n",
    "```\n",
    "where the goal is always to retrieve a hidden food item and put it on the stove which is located in the kitchen. One can lose the game if it eats the food item instead of putting it on the stove!\n",
    "\n",
    "Using `tw-make tw-simple ...`, we are going to generate the following 7 games:\n",
    "\n",
    "| gamefile | description |\n",
    "| :------- | :---------- |\n",
    "| `games/rewardsDense_goalDetailed.z8` | dense reward + detailed instructions |\n",
    "| `games/rewardsBalanced_goalDetailed.z8` | balanced rewards + detailed instructions |\n",
    "| `games/rewardsSparse_goalDetailed.z8` | sparse rewards + detailed instructions |\n",
    "| | |\n",
    "| `games/rewardsDense_goalBrief.z8` | dense rewards + no instructions but the goal is mentionned |\n",
    "| `games/rewardsBalanced_goalBrief.z8` | balanced rewards + no instructions but the goal is mentionned |\n",
    "| `games/rewardsSparse_goalBrief.z8` | sparse rewards + no instructions but the goal is mentionned |\n",
    "| | |\n",
    "| `games/rewardsSparse_goalNone.z8` | sparse rewards + no instructions/goal<br>_Hint: there's an hidden note in the game that describes the goal!_ |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can skip this if you already downloaded the games in the prequisite section.\n",
    "\n",
    "# Same as !make_games.sh\n",
    "!tw-make tw-simple --rewards dense    --goal detailed --seed 18 --test --silent -f --output games/tw-rewardsDense_goalDetailed.z8\n",
    "!tw-make tw-simple --rewards balanced --goal detailed --seed 18 --test --silent -f --output games/tw-rewardsBalanced_goalDetailed.z8\n",
    "!tw-make tw-simple --rewards sparse   --goal detailed --seed 18 --test --silent -f --output games/tw-rewardsSparse_goalDetailed.z8\n",
    "!tw-make tw-simple --rewards dense    --goal brief    --seed 18 --test --silent -f --output games/tw-rewardsDense_goalBrief.z8\n",
    "!tw-make tw-simple --rewards balanced --goal brief    --seed 18 --test --silent -f --output games/tw-rewardsBalanced_goalBrief.z8\n",
    "!tw-make tw-simple --rewards sparse   --goal brief    --seed 18 --test --silent -f --output games/tw-rewardsSparse_goalBrief.z8\n",
    "!tw-make tw-simple --rewards sparse   --goal none     --seed 18 --test --silent -f --output games/tw-rewardsSparse_goalNone.z8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the random baseline\n",
    "Let's start with building an agent that simply selects an admissible command at random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Mapping, Any\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "import textworld.gym\n",
    "\n",
    "\n",
    "class RandomAgent(textworld.gym.Agent):\n",
    "    \"\"\" Agent that randomly selects a command from the admissible ones. \"\"\"\n",
    "    def __init__(self, seed=1234):\n",
    "        self.seed = seed\n",
    "        self.rng = np.random.RandomState(self.seed)\n",
    "\n",
    "    @property\n",
    "    def infos_to_request(self) -> textworld.EnvInfos:\n",
    "        return textworld.EnvInfos(admissible_commands=True)\n",
    "\n",
    "    def act(self, obs: str, score: int, done: bool, infos: Mapping[str, Any]) -> str:\n",
    "        return self.rng.choice(infos[\"admissible_commands\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Play function\n",
    "Let's write a simple play function that we will use to evaluate our agent on a given game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from glob import glob\n",
    "\n",
    "import textworld.gym\n",
    "\n",
    "import torch\n",
    "\n",
    "\n",
    "def play(agent, path, max_step=100, nb_episodes=10, verbose=True):\n",
    "    torch.manual_seed(20211021)  # For reproducibility when using action sampling.\n",
    "\n",
    "    infos_to_request = agent.infos_to_request\n",
    "    infos_to_request.max_score = True  # Needed to normalize the scores.\n",
    "\n",
    "    gamefiles = [path]\n",
    "    if os.path.isdir(path):\n",
    "        gamefiles = glob(os.path.join(path, \"*.z8\"))\n",
    "\n",
    "    env_id = textworld.gym.register_games(gamefiles,\n",
    "                                          request_infos=infos_to_request,\n",
    "                                          max_episode_steps=max_step)\n",
    "    env = textworld.gym.make(env_id)  # Create a Gym environment to play the text game.\n",
    "    if verbose:\n",
    "        if os.path.isdir(path):\n",
    "            print(os.path.dirname(path), end=\"\")\n",
    "        else:\n",
    "            print(os.path.basename(path), end=\"\")\n",
    "\n",
    "    # Collect some statistics: nb_steps, final reward.\n",
    "    avg_moves, avg_scores, avg_norm_scores = [], [], []\n",
    "    for no_episode in range(nb_episodes):\n",
    "        obs, infos = env.reset()  # Start new episode.\n",
    "\n",
    "        score = 0\n",
    "        done = False\n",
    "        nb_moves = 0\n",
    "        while not done:\n",
    "            command = agent.act(obs, score, done, infos)\n",
    "            obs, score, done, infos = env.step(command)\n",
    "            nb_moves += 1\n",
    "\n",
    "        agent.act(obs, score, done, infos)  # Let the agent know the game is done.\n",
    "\n",
    "        if verbose:\n",
    "            print(\".\", end=\"\")\n",
    "        avg_moves.append(nb_moves)\n",
    "        avg_scores.append(score)\n",
    "        avg_norm_scores.append(score / infos[\"max_score\"])\n",
    "\n",
    "    env.close()\n",
    "    if verbose:\n",
    "        if os.path.isdir(path):\n",
    "            msg = \"  \\tavg. steps: {:5.1f}; avg. normalized score: {:4.1f} / {}.\"\n",
    "            print(msg.format(np.mean(avg_moves), np.mean(avg_norm_scores), 1))\n",
    "        else:\n",
    "            msg = \"  \\tavg. steps: {:5.1f}; avg. score: {:4.1f} / {}.\"\n",
    "            print(msg.format(np.mean(avg_moves), np.mean(avg_scores), infos[\"max_score\"]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate the random agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tw-rewardsDense_goalDetailed.z8..........  \tavg. steps: 100.0; avg. score:  4.2 / 10.\n",
      "tw-rewardsBalanced_goalDetailed.z8..........  \tavg. steps: 100.0; avg. score:  0.7 / 4.\n",
      "tw-rewardsSparse_goalDetailed.z8..........  \tavg. steps: 100.0; avg. score:  0.0 / 1.\n"
     ]
    }
   ],
   "source": [
    "# We report the score and steps averaged over 10 playthroughs.\n",
    "play(RandomAgent(), \"./games/tw-rewardsDense_goalDetailed.z8\")    # Dense rewards\n",
    "play(RandomAgent(), \"./games/tw-rewardsBalanced_goalDetailed.z8\") # Balanced rewards\n",
    "play(RandomAgent(), \"./games/tw-rewardsSparse_goalDetailed.z8\")   # Sparse rewards"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural agent\n",
    "\n",
    "Now, let's create an agent that can learn to play text-based games. The agent will be trained to select a command from the list of admissible commands given the current game's narrative, inventory, and room description. Here is an overview of the architecture used for the agent: \n",
    "\n",
    "<div>\n",
    "  <img src=\"https://raw.githubusercontent.com/MarcCote/TextWorld/msr_summit_2021/notebooks/figs/neural_agent.png\" width=\"500\"/>\n",
    "</div>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code\n",
    "Here's the implementation of that learning agent built with [PyTorch](https://pytorch.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from typing import List, Mapping, Any, Optional\n",
    "from collections import defaultdict\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "import textworld\n",
    "import textworld.gym\n",
    "from textworld import EnvInfos\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch import optim\n",
    "import torch.nn.functional as F\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "\n",
    "class CommandScorer(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size):\n",
    "        super(CommandScorer, self).__init__()\n",
    "        torch.manual_seed(42)  # For reproducibility\n",
    "        self.embedding    = nn.Embedding(input_size, hidden_size)\n",
    "        self.encoder_gru  = nn.GRU(hidden_size, hidden_size)\n",
    "        self.cmd_encoder_gru  = nn.GRU(hidden_size, hidden_size)\n",
    "        self.state_gru    = nn.GRU(hidden_size, hidden_size)\n",
    "        self.hidden_size  = hidden_size\n",
    "        self.state_hidden = torch.zeros(1, 1, hidden_size, device=device)\n",
    "        self.critic       = nn.Linear(hidden_size, 1)\n",
    "        self.att_cmd      = nn.Linear(hidden_size * 2, 1)\n",
    "\n",
    "    def forward(self, obs, commands, **kwargs):\n",
    "        input_length = obs.size(0)\n",
    "        batch_size = obs.size(1)\n",
    "        nb_cmds = commands.size(1)\n",
    "\n",
    "        embedded = self.embedding(obs)\n",
    "        encoder_output, encoder_hidden = self.encoder_gru(embedded)\n",
    "        state_output, state_hidden = self.state_gru(encoder_hidden, self.state_hidden)\n",
    "        self.state_hidden = state_hidden\n",
    "        value = self.critic(state_output)\n",
    "\n",
    "        # Attention network over the commands.\n",
    "        cmds_embedding = self.embedding.forward(commands)\n",
    "        _, cmds_encoding_last_states = self.cmd_encoder_gru.forward(cmds_embedding)  # 1 x cmds x hidden\n",
    "\n",
    "        # Same observed state for all commands.\n",
    "        cmd_selector_input = torch.stack([state_hidden] * nb_cmds, 2)  # 1 x batch x cmds x hidden\n",
    "\n",
    "        # Same command choices for the whole batch.\n",
    "        cmds_encoding_last_states = torch.stack([cmds_encoding_last_states] * batch_size, 1)  # 1 x batch x cmds x hidden\n",
    "\n",
    "        # Concatenate the observed state and command encodings.\n",
    "        cmd_selector_input = torch.cat([cmd_selector_input, cmds_encoding_last_states], dim=-1)\n",
    "\n",
    "        # Compute one score per command.\n",
    "        scores = F.relu(self.att_cmd(cmd_selector_input)).squeeze(-1)  # 1 x Batch x cmds\n",
    "\n",
    "        probs = F.softmax(scores, dim=2)  # 1 x Batch x cmds\n",
    "        index = probs[0].multinomial(num_samples=1).unsqueeze(0) # 1 x batch x indx\n",
    "        return scores, index, value\n",
    "\n",
    "    def reset_hidden(self, batch_size):\n",
    "        self.state_hidden = torch.zeros(1, batch_size, self.hidden_size, device=device)\n",
    "\n",
    "\n",
    "class NeuralAgent:\n",
    "    \"\"\" Simple Neural Agent for playing TextWorld games. \"\"\"\n",
    "    MAX_VOCAB_SIZE = 1000\n",
    "    UPDATE_FREQUENCY = 10\n",
    "    LOG_FREQUENCY = 1000\n",
    "    GAMMA = 0.9\n",
    "\n",
    "    def __init__(self) -> None:\n",
    "        self._initialized = False\n",
    "        self._epsiode_has_started = False\n",
    "        self.id2word = [\"<PAD>\", \"<UNK>\"]\n",
    "        self.word2id = {w: i for i, w in enumerate(self.id2word)}\n",
    "\n",
    "        self.model = CommandScorer(input_size=self.MAX_VOCAB_SIZE, hidden_size=128)\n",
    "        self.optimizer = optim.Adam(self.model.parameters(), 0.00003)\n",
    "\n",
    "        self.mode = \"test\"\n",
    "\n",
    "    def train(self):\n",
    "        self.mode = \"train\"\n",
    "        self.stats = {\"max\": defaultdict(list), \"mean\": defaultdict(list)}\n",
    "        self.transitions = []\n",
    "        self.model.reset_hidden(1)\n",
    "        self.last_score = 0\n",
    "        self.no_train_step = 0\n",
    "\n",
    "    def test(self):\n",
    "        self.mode = \"test\"\n",
    "        self.model.reset_hidden(1)\n",
    "\n",
    "    @property\n",
    "    def infos_to_request(self) -> EnvInfos:\n",
    "        return EnvInfos(description=True, inventory=True, admissible_commands=True,\n",
    "                        won=True, lost=True)\n",
    "\n",
    "    def _get_word_id(self, word):\n",
    "        if word not in self.word2id:\n",
    "            if len(self.word2id) >= self.MAX_VOCAB_SIZE:\n",
    "                return self.word2id[\"<UNK>\"]\n",
    "\n",
    "            self.id2word.append(word)\n",
    "            self.word2id[word] = len(self.word2id)\n",
    "\n",
    "        return self.word2id[word]\n",
    "\n",
    "    def _tokenize(self, text):\n",
    "        # Simple tokenizer: strip out all non-alphabetic characters.\n",
    "        text = re.sub(\"[^a-zA-Z0-9\\- ]\", \" \", text)\n",
    "        word_ids = list(map(self._get_word_id, text.split()))\n",
    "        return word_ids\n",
    "\n",
    "    def _process(self, texts):\n",
    "        texts = list(map(self._tokenize, texts))\n",
    "        max_len = max(len(l) for l in texts)\n",
    "        padded = np.ones((len(texts), max_len)) * self.word2id[\"<PAD>\"]\n",
    "\n",
    "        for i, text in enumerate(texts):\n",
    "            padded[i, :len(text)] = text\n",
    "\n",
    "        padded_tensor = torch.from_numpy(padded).type(torch.long).to(device)\n",
    "        padded_tensor = padded_tensor.permute(1, 0) # Batch x Seq => Seq x Batch\n",
    "        return padded_tensor\n",
    "\n",
    "    def _discount_rewards(self, last_values):\n",
    "        returns, advantages = [], []\n",
    "        R = last_values.data\n",
    "        for t in reversed(range(len(self.transitions))):\n",
    "            rewards, _, _, values = self.transitions[t]\n",
    "            R = rewards + self.GAMMA * R\n",
    "            adv = R - values\n",
    "            returns.append(R)\n",
    "            advantages.append(adv)\n",
    "\n",
    "        return returns[::-1], advantages[::-1]\n",
    "\n",
    "    def act(self, obs: str, score: int, done: bool, infos: Mapping[str, Any]) -> Optional[str]:\n",
    "\n",
    "        # Build agent's observation: feedback + look + inventory.\n",
    "        input_ = \"{}\\n{}\\n{}\".format(obs, infos[\"description\"], infos[\"inventory\"])\n",
    "\n",
    "        # Tokenize and pad the input and the commands to chose from.\n",
    "        input_tensor = self._process([input_])\n",
    "        commands_tensor = self._process(infos[\"admissible_commands\"])\n",
    "\n",
    "        # Get our next action and value prediction.\n",
    "        outputs, indexes, values = self.model(input_tensor, commands_tensor)\n",
    "        action = infos[\"admissible_commands\"][indexes[0]]\n",
    "\n",
    "        if self.mode == \"test\":\n",
    "            if done:\n",
    "                self.model.reset_hidden(1)\n",
    "            return action\n",
    "\n",
    "        self.no_train_step += 1\n",
    "\n",
    "        if self.transitions:\n",
    "            reward = score - self.last_score  # Reward is the gain/loss in score.\n",
    "            self.last_score = score\n",
    "            if infos[\"won\"]:\n",
    "                reward += 100\n",
    "            if infos[\"lost\"]:\n",
    "                reward -= 100\n",
    "\n",
    "            self.transitions[-1][0] = reward  # Update reward information.\n",
    "\n",
    "        self.stats[\"max\"][\"score\"].append(score)\n",
    "        if self.no_train_step % self.UPDATE_FREQUENCY == 0:\n",
    "            # Update model\n",
    "            returns, advantages = self._discount_rewards(values)\n",
    "\n",
    "            loss = 0\n",
    "            for transition, ret, advantage in zip(self.transitions, returns, advantages):\n",
    "                reward, indexes_, outputs_, values_ = transition\n",
    "\n",
    "                advantage        = advantage.detach() # Block gradients flow here.\n",
    "                probs            = F.softmax(outputs_, dim=2)\n",
    "                log_probs        = torch.log(probs)\n",
    "                log_action_probs = log_probs.gather(2, indexes_)\n",
    "                policy_loss      = (-log_action_probs * advantage).sum()\n",
    "                value_loss       = (.5 * (values_ - ret) ** 2.).sum()\n",
    "                entropy     = (-probs * log_probs).sum()\n",
    "                loss += policy_loss + 0.5 * value_loss - 0.1 * entropy\n",
    "\n",
    "                self.stats[\"mean\"][\"reward\"].append(reward)\n",
    "                self.stats[\"mean\"][\"policy\"].append(policy_loss.item())\n",
    "                self.stats[\"mean\"][\"value\"].append(value_loss.item())\n",
    "                self.stats[\"mean\"][\"entropy\"].append(entropy.item())\n",
    "                self.stats[\"mean\"][\"confidence\"].append(torch.exp(log_action_probs).item())\n",
    "\n",
    "            if self.no_train_step % self.LOG_FREQUENCY == 0:\n",
    "                msg = \"{:6d}. \".format(self.no_train_step)\n",
    "                msg += \"  \".join(\"{}: {: 3.3f}\".format(k, np.mean(v)) for k, v in self.stats[\"mean\"].items())\n",
    "                msg += \"  \" + \"  \".join(\"{}: {:2d}\".format(k, np.max(v)) for k, v in self.stats[\"max\"].items())\n",
    "                msg += \"  vocab: {:3d}\".format(len(self.id2word))\n",
    "                print(msg)\n",
    "                self.stats = {\"max\": defaultdict(list), \"mean\": defaultdict(list)}\n",
    "\n",
    "            loss.backward()\n",
    "            nn.utils.clip_grad_norm_(self.model.parameters(), 40)\n",
    "            self.optimizer.step()\n",
    "            self.optimizer.zero_grad()\n",
    "\n",
    "            self.transitions = []\n",
    "            self.model.reset_hidden(1)\n",
    "        else:\n",
    "            # Keep information about transitions for Truncated Backpropagation Through Time.\n",
    "            self.transitions.append([None, indexes, outputs, values])  # Reward will be set on the next call\n",
    "\n",
    "        if done:\n",
    "            self.last_score = 0  # Will be starting a new episode. Reset the last score.\n",
    "\n",
    "        return action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the neural agent\n",
    "Let's first evaluate the agent before training to get a sense of its initial performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tw-rewardsDense_goalDetailed.z8..........  \tavg. steps: 100.0; avg. score:  4.3 / 10.\n"
     ]
    }
   ],
   "source": [
    "agent = NeuralAgent()\n",
    "play(agent, \"./games/tw-rewardsDense_goalDetailed.z8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unsurprisingly, the result is not much different from what the random agent can achieve since our neural agent is initialized to a random policy.\n",
    "\n",
    "Let's train the agent for a few episodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training\n",
      "  1000. reward:  0.042  policy:  0.258  value:  0.074  entropy:  2.337  confidence:  0.099  score:  8  vocab: 258\n",
      "  2000. reward: -0.058  policy: -1.414  value:  24.997  entropy:  2.385  confidence:  0.095  score:  9  vocab: 316\n",
      "  3000. reward:  0.052  policy:  0.146  value:  0.097  entropy:  2.429  confidence:  0.092  score:  9  vocab: 318\n",
      "  4000. reward:  0.042  policy: -0.040  value:  0.081  entropy:  2.397  confidence:  0.095  score:  5  vocab: 318\n",
      "  5000. reward:  0.053  policy:  0.084  value:  0.104  entropy:  2.480  confidence:  0.089  score:  7  vocab: 319\n",
      "  6000. reward:  0.048  policy: -0.015  value:  0.083  entropy:  2.402  confidence:  0.095  score:  5  vocab: 319\n",
      "  7000. reward:  0.053  policy:  0.030  value:  0.086  entropy:  2.426  confidence:  0.096  score:  8  vocab: 319\n",
      "  8000. reward: -0.049  policy: -0.944  value:  18.233  entropy:  2.421  confidence:  0.098  score:  9  vocab: 320\n",
      "  9000. reward:  0.054  policy:  0.062  value:  0.093  entropy:  2.477  confidence:  0.093  score:  6  vocab: 320\n",
      " 10000. reward: -0.056  policy: -1.178  value:  20.702  entropy:  2.374  confidence:  0.103  score:  9  vocab: 321\n",
      " 11000. reward:  0.054  policy:  0.010  value:  0.105  entropy:  2.428  confidence:  0.097  score:  6  vocab: 321\n",
      " 12000. reward:  0.066  policy:  0.114  value:  0.131  entropy:  2.443  confidence:  0.096  score:  9  vocab: 321\n",
      " 13000. reward:  0.062  policy: -0.002  value:  0.107  entropy:  2.487  confidence:  0.097  score:  7  vocab: 321\n",
      " 14000. reward:  0.054  policy: -0.051  value:  0.091  entropy:  2.382  confidence:  0.102  score:  7  vocab: 321\n",
      " 15000. reward:  0.062  policy:  0.042  value:  0.121  entropy:  2.476  confidence:  0.098  score:  9  vocab: 321\n",
      " 16000. reward:  0.052  policy: -0.046  value:  0.088  entropy:  2.414  confidence:  0.099  score:  7  vocab: 321\n",
      " 17000. reward:  0.063  policy:  0.039  value:  0.112  entropy:  2.420  confidence:  0.101  score:  7  vocab: 321\n",
      " 18000. reward:  0.057  policy: -0.036  value:  0.094  entropy:  2.432  confidence:  0.098  score:  7  vocab: 321\n",
      " 19000. reward:  0.060  policy: -0.008  value:  0.091  entropy:  2.438  confidence:  0.102  score:  7  vocab: 321\n",
      " 20000. reward:  0.063  policy: -0.018  value:  0.102  entropy:  2.435  confidence:  0.101  score:  7  vocab: 321\n",
      " 21000. reward:  0.067  policy:  0.062  value:  0.145  entropy:  2.450  confidence:  0.099  score:  9  vocab: 321\n",
      " 22000. reward: -0.148  policy: -2.970  value:  46.054  entropy:  2.391  confidence:  0.106  score:  9  vocab: 323\n",
      " 23000. reward:  0.059  policy: -0.053  value:  0.098  entropy:  2.413  confidence:  0.106  score:  9  vocab: 323\n",
      " 24000. reward: -0.131  policy: -2.155  value:  39.827  entropy:  2.487  confidence:  0.103  score:  9  vocab: 325\n",
      " 25000. reward: -0.241  policy: -2.584  value:  44.997  entropy:  2.457  confidence:  0.105  score:  9  vocab: 328\n",
      " 26000. reward: -0.224  policy: -1.829  value:  35.522  entropy:  2.426  confidence:  0.112  score:  9  vocab: 331\n",
      " 27000. reward: -0.126  policy: -1.487  value:  28.636  entropy:  2.363  confidence:  0.126  score:  9  vocab: 332\n",
      " 28000. reward: -0.550  policy: -5.302  value:  92.847  entropy:  2.322  confidence:  0.132  score:  9  vocab: 338\n",
      " 29000. reward: -0.548  policy: -7.017  value:  121.968  entropy:  2.291  confidence:  0.136  score:  9  vocab: 342\n",
      " 30000. reward: -0.338  policy: -5.695  value:  79.081  entropy:  2.238  confidence:  0.144  score:  9  vocab: 344\n",
      " 31000. reward: -0.336  policy: -4.484  value:  68.651  entropy:  2.314  confidence:  0.139  score:  9  vocab: 346\n",
      " 32000. reward: -0.227  policy: -2.834  value:  41.036  entropy:  2.256  confidence:  0.152  score:  9  vocab: 348\n",
      " 33000. reward: -0.008  policy: -1.326  value:  23.433  entropy:  2.277  confidence:  0.151  score:  9  vocab: 349\n",
      " 34000. reward: -0.436  policy: -5.410  value:  85.167  entropy:  2.263  confidence:  0.151  score:  9  vocab: 352\n",
      " 35000. reward: -0.434  policy: -5.897  value:  89.690  entropy:  2.184  confidence:  0.163  score:  9  vocab: 353\n",
      " 36000. reward: -0.877  policy: -8.919  value:  143.631  entropy:  2.283  confidence:  0.158  score:  9  vocab: 359\n",
      " 37000. reward: -0.122  policy: -2.909  value:  46.651  entropy:  2.308  confidence:  0.151  score:  9  vocab: 360\n",
      " 38000. reward: -0.217  policy: -4.576  value:  70.621  entropy:  2.187  confidence:  0.171  score:  9  vocab: 360\n",
      " 39000. reward: -0.231  policy: -2.571  value:  48.777  entropy:  2.187  confidence:  0.162  score:  9  vocab: 363\n",
      " 40000. reward: -0.544  policy: -4.282  value:  81.480  entropy:  2.217  confidence:  0.159  score:  9  vocab: 365\n",
      " 41000. reward: -0.222  policy: -4.207  value:  65.609  entropy:  2.292  confidence:  0.149  score:  9  vocab: 367\n",
      " 42000. reward: -0.427  policy: -4.491  value:  66.247  entropy:  2.296  confidence:  0.157  score:  9  vocab: 369\n",
      " 43000. reward: -0.537  policy: -6.994  value:  108.595  entropy:  2.268  confidence:  0.163  score:  9  vocab: 375\n",
      " 44000. reward: -0.217  policy: -2.053  value:  40.536  entropy:  2.207  confidence:  0.167  score:  9  vocab: 376\n",
      " 45000. reward: -0.224  policy: -4.271  value:  70.424  entropy:  2.214  confidence:  0.166  score:  9  vocab: 378\n",
      " 46000. reward: -0.537  policy: -3.577  value:  62.061  entropy:  2.218  confidence:  0.166  score:  9  vocab: 383\n",
      "Trained in 2556.39 secs\n"
     ]
    }
   ],
   "source": [
    "# You can skip this if you already downloaded the data in the prequisite section.\n",
    "\n",
    "from time import time\n",
    "agent = NeuralAgent()\n",
    "\n",
    "print(\"Training\")\n",
    "agent.train()  # Tell the agent it should update its parameters.\n",
    "starttime = time()\n",
    "play(agent, \"./games/tw-rewardsDense_goalDetailed.z8\", nb_episodes=500, verbose=False)  # Dense rewards game.\n",
    "\n",
    "print(\"Trained in {:.2f} secs\".format(time() - starttime))\n",
    "\n",
    "# Save the trained agent.\n",
    "import os\n",
    "os.makedirs('checkpoints', exist_ok=True)\n",
    "torch.save(agent, 'checkpoints/agent_trained_on_single_game.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the agent trained on a single game"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tw-rewardsDense_goalDetailed.z8..........  \tavg. steps:  85.0; avg. score:  8.8 / 10.\n"
     ]
    }
   ],
   "source": [
    "# We report the score and steps averaged over 10 playthroughs.\n",
    "agent = torch.load('checkpoints/agent_trained_on_single_game.pt')\n",
    "agent.test()\n",
    "play(agent, \"./games/tw-rewardsDense_goalDetailed.z8\")  # Dense rewards game."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, since we trained on that single simple game, it's not surprinsing the agent can achieve a high score on it. It would be more interesting to evaluate the generalization capability of the agent.\n",
    "\n",
    "To do so, we are going to test the agent on another game drawn from the same game distribution (i.e. same world but the goal is to pick another food item). Let's generate `games/another_game.z8` with the same rewards density (`--rewards dense`) and the same goal description (`--goal detailed`), but using `--seed 1` and without the `--test` flag (to make sure the game is not part of the test set since `games/rewardsDense_goalDetailed.z8` is)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tw-make tw-simple --rewards dense --goal detailed --seed 1 --output games/tw-another_game.z8 -v -f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tw-another_game.z8..........  \tavg. steps: 100.0; avg. score:  3.9 / 8.\n",
      "tw-another_game.z8..........  \tavg. steps: 100.0; avg. score:  5.6 / 8.\n"
     ]
    }
   ],
   "source": [
    "# We report the score and steps averaged over 10 playthroughs.\n",
    "play(RandomAgent(), \"./games/tw-another_game.z8\")\n",
    "play(agent, \"./games/tw-another_game.z8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see the trained agent does a bit better than the random agent. In order to improve the agent's generalization capability, we should train it on many different games drawn from the game distribution.\n",
    "\n",
    "One could use the following command to easily generate 100 training games:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# You can skip this if you already downloaded the data in the prequisite section.\n",
    "\n",
    "! seq 1 100 | xargs -n1 -P4 tw-make tw-simple --rewards dense --goal detailed --format z8 --output training_games/ --seed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we train our agent on that set of training games."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training on 100 games\n",
      "  1000. reward: -0.067  policy: -0.285  value:  10.002  entropy:  2.348  confidence:  0.097  score:  9  vocab: 513\n",
      "  2000. reward: -0.059  policy: -1.836  value:  23.834  entropy:  2.400  confidence:  0.095  score:  9  vocab: 595\n",
      "  3000. reward:  0.043  policy:  0.107  value:  0.073  entropy:  2.357  confidence:  0.097  score:  6  vocab: 616\n",
      "  4000. reward:  0.048  policy:  0.110  value:  0.094  entropy:  2.457  confidence:  0.090  score:  6  vocab: 644\n",
      "  5000. reward:  0.048  policy:  0.008  value:  0.084  entropy:  2.382  confidence:  0.096  score:  9  vocab: 655\n",
      "  6000. reward: -0.048  policy: -2.254  value:  42.841  entropy:  2.441  confidence:  0.093  score:  9  vocab: 670\n",
      "  7000. reward:  0.041  policy: -0.017  value:  0.062  entropy:  2.378  confidence:  0.095  score:  5  vocab: 686\n",
      "  8000. reward:  0.047  policy:  0.051  value:  0.073  entropy:  2.412  confidence:  0.095  score:  6  vocab: 689\n",
      "  9000. reward: -0.051  policy: -0.222  value:  5.534  entropy:  2.449  confidence:  0.093  score:  7  vocab: 694\n",
      " 10000. reward:  0.049  policy:  0.016  value:  0.089  entropy:  2.470  confidence:  0.094  score:  7  vocab: 704\n",
      " 11000. reward: -0.049  policy: -1.591  value:  20.716  entropy:  2.421  confidence:  0.095  score:  7  vocab: 709\n",
      " 12000. reward:  0.054  policy:  0.072  value:  0.110  entropy:  2.436  confidence:  0.095  score:  6  vocab: 709\n",
      " 13000. reward:  0.058  policy:  0.030  value:  0.107  entropy:  2.494  confidence:  0.091  score:  8  vocab: 710\n",
      " 14000. reward: -0.048  policy: -1.504  value:  23.623  entropy:  2.444  confidence:  0.097  score:  9  vocab: 712\n",
      " 15000. reward: -0.052  policy: -1.158  value:  16.649  entropy:  2.467  confidence:  0.095  score:  7  vocab: 713\n",
      " 16000. reward: -0.049  policy: -0.268  value:  5.564  entropy:  2.406  confidence:  0.099  score:  7  vocab: 715\n",
      " 17000. reward:  0.161  policy:  0.926  value:  14.505  entropy:  2.431  confidence:  0.097  score:  7  vocab: 716\n",
      " 18000. reward: -0.041  policy: -1.291  value:  22.410  entropy:  2.522  confidence:  0.091  score:  9  vocab: 720\n",
      " 19000. reward: -0.042  policy: -1.627  value:  22.006  entropy:  2.411  confidence:  0.101  score:  9  vocab: 720\n",
      " 20000. reward: -0.047  policy: -1.133  value:  22.659  entropy:  2.395  confidence:  0.104  score:  9  vocab: 724\n",
      " 21000. reward:  0.179  policy:  0.365  value:  6.029  entropy:  2.422  confidence:  0.100  score:  8  vocab: 725\n",
      " 22000. reward:  0.173  policy:  0.997  value:  14.421  entropy:  2.470  confidence:  0.099  score:  8  vocab: 725\n",
      " 23000. reward: -0.037  policy: -1.756  value:  24.899  entropy:  2.533  confidence:  0.093  score:  8  vocab: 726\n",
      " 24000. reward: -0.039  policy: -0.616  value:  9.761  entropy:  2.471  confidence:  0.106  score:  8  vocab: 727\n",
      " 25000. reward: -0.032  policy: -1.013  value:  18.614  entropy:  2.466  confidence:  0.102  score:  9  vocab: 729\n",
      " 26000. reward: -0.037  policy: -0.770  value:  13.584  entropy:  2.474  confidence:  0.100  score:  9  vocab: 730\n",
      " 27000. reward:  0.084  policy: -1.364  value:  30.589  entropy:  2.439  confidence:  0.112  score:  9  vocab: 734\n",
      " 28000. reward: -0.031  policy: -0.712  value:  13.120  entropy:  2.444  confidence:  0.113  score:  9  vocab: 736\n",
      " 29000. reward: -0.143  policy: -3.264  value:  45.924  entropy:  2.459  confidence:  0.115  score:  9  vocab: 737\n",
      " 30000. reward: -0.031  policy: -0.849  value:  59.370  entropy:  2.407  confidence:  0.122  score:  9  vocab: 740\n",
      " 31000. reward:  0.076  policy: -0.083  value:  0.157  entropy:  2.394  confidence:  0.127  score: 10  vocab: 742\n",
      " 32000. reward: -0.119  policy: -2.216  value:  73.379  entropy:  2.425  confidence:  0.126  score:  9  vocab: 746\n",
      " 33000. reward:  0.079  policy:  0.131  value:  45.571  entropy:  2.375  confidence:  0.130  score:  9  vocab: 747\n",
      " 34000. reward: -0.129  policy: -1.131  value:  21.589  entropy:  2.426  confidence:  0.130  score:  9  vocab: 748\n",
      " 35000. reward: -0.248  policy: -2.199  value:  40.637  entropy:  2.392  confidence:  0.130  score:  9  vocab: 754\n",
      " 36000. reward: -0.126  policy: -4.269  value:  77.857  entropy:  2.384  confidence:  0.126  score:  9  vocab: 756\n",
      " 37000. reward: -0.132  policy: -2.362  value:  59.195  entropy:  2.377  confidence:  0.130  score:  9  vocab: 759\n",
      " 38000. reward: -0.138  policy: -2.624  value:  39.265  entropy:  2.468  confidence:  0.120  score:  9  vocab: 761\n",
      " 39000. reward: -0.022  policy: -0.115  value:  49.352  entropy:  2.474  confidence:  0.119  score:  9  vocab: 765\n",
      " 40000. reward: -0.028  policy: -1.081  value:  18.047  entropy:  2.433  confidence:  0.120  score:  9  vocab: 765\n",
      " 41000. reward:  0.082  policy:  0.796  value:  43.794  entropy:  2.540  confidence:  0.105  score:  9  vocab: 766\n",
      " 42000. reward: -0.463  policy: -6.601  value:  102.364  entropy:  2.499  confidence:  0.116  score:  9  vocab: 769\n",
      " 43000. reward: -0.451  policy: -4.854  value:  85.497  entropy:  2.428  confidence:  0.124  score:  9  vocab: 772\n",
      " 44000. reward: -0.247  policy: -1.431  value:  26.682  entropy:  2.407  confidence:  0.132  score:  9  vocab: 773\n",
      " 45000. reward:  0.071  policy:  0.044  value:  0.135  entropy:  2.473  confidence:  0.117  score:  8  vocab: 773\n",
      " 46000. reward:  0.291  policy:  3.715  value:  48.858  entropy:  2.561  confidence:  0.104  score:  8  vocab: 774\n",
      " 47000. reward: -0.036  policy: -1.167  value:  18.583  entropy:  2.478  confidence:  0.113  score:  9  vocab: 775\n",
      "Trained in 2731.61 secs\n"
     ]
    }
   ],
   "source": [
    "# You can skip this if you already downloaded the data in the prequisite section.\n",
    "\n",
    "from time import time\n",
    "agent = NeuralAgent()\n",
    "\n",
    "print(\"Training on 100 games\")\n",
    "agent.train()  # Tell the agent it should update its parameters.\n",
    "starttime = time()\n",
    "play(agent, \"./training_games/\", nb_episodes=100 * 5, verbose=False)  # Each game will be seen 5 times.\n",
    "print(\"Trained in {:.2f} secs\".format(time() - starttime))\n",
    "\n",
    "# Save the trained agent.\n",
    "import os\n",
    "os.makedirs('checkpoints', exist_ok=True)\n",
    "torch.save(agent, 'checkpoints/agent_trained_on_multiple_games.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the agent trained on 100 games."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tw-rewardsDense_goalDetailed.z8..........  \tavg. steps:  91.3; avg. score:  7.9 / 10.\n",
      "tw-another_game.z8..........  \tavg. steps:  97.8; avg. score:  6.0 / 8.\n"
     ]
    }
   ],
   "source": [
    "agent = torch.load('checkpoints/agent_trained_on_multiple_games.pt')\n",
    "agent.test()\n",
    "play(agent, \"./games/tw-rewardsDense_goalDetailed.z8\")  # Averaged over 10 playthroughs.\n",
    "play(agent, \"./games/tw-another_game.z8\")               # Averaged over 10 playthroughs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare it to the agent trained on a single game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tw-rewardsDense_goalDetailed.z8..........  \tavg. steps:  85.0; avg. score:  8.8 / 10.\n",
      "tw-another_game.z8..........  \tavg. steps: 100.0; avg. score:  5.6 / 8.\n"
     ]
    }
   ],
   "source": [
    "agent = torch.load('checkpoints/agent_trained_on_single_game.pt')\n",
    "agent.test()\n",
    "play(agent, \"./games/tw-rewardsDense_goalDetailed.z8\")  # Averaged over 10 playthroughs.\n",
    "play(agent, \"./games/tw-another_game.z8\")               # Averaged over 10 playthroughs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating the agent on a test distribution\n",
    "We will generate 20 test games and evaluate the agent on them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# You can skip this if you already downloaded the games in the prequisite section.\n",
    "\n",
    "! seq 1 20 | xargs -n1 -P4 tw-make tw-simple --rewards dense --goal detailed --test --format z8 --output testing_games/ --seed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./testing_games........................................................................................................................................................................................................  \tavg. steps:  99.6; avg. normalized score:  0.5 / 1.\n",
      "./testing_games........................................................................................................................................................................................................  \tavg. steps:  92.5; avg. normalized score:  0.8 / 1.\n"
     ]
    }
   ],
   "source": [
    "agent = torch.load('checkpoints/agent_trained_on_multiple_games.pt')\n",
    "agent.test()\n",
    "play(RandomAgent(), \"./testing_games/\", nb_episodes=20 * 10)\n",
    "play(agent, \"./testing_games/\", nb_episodes=20 * 10)  # Averaged over 10 playthroughs for each test game."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While not being perfect, the agent manage to score more points on average compared to the random agent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "Here are a few possible directions one can take to improve the agent's performance.\n",
    "- Adding more training games\n",
    "- Changing the agent architecture\n",
    "- Leveraging already trained word embeddings\n",
    "- Playing more games at once (see [`textworld.gym.make_batch`](https://textworld.readthedocs.io/en/latest/textworld.gym.html#textworld.gym.utils.make_batch))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Papers about RL applied to text-based games\n",
    "* [Language Understanding for Text-based games using Deep Reinforcement Learning][narasimhan_et_al_2015]\n",
    "* [Learning How Not to Act in Text-based Games][haroush_et_al_2017]\n",
    "* [Deep Reinforcement Learning with a Natural Language Action Space][he_et_al_2015]\n",
    "* [What can you do with a rock? Affordance extraction via word embeddings][fulda_et_al_2017]\n",
    "* [Text-based adventures of the Golovin AI Agent][kostka_et_al_2017]\n",
    "* [Using reinforcement learning to learn how to play text-based games][zelinka_2018]\n",
    "\n",
    "[narasimhan_et_al_2015]: https://arxiv.org/abs/1506.08941\n",
    "[haroush_et_al_2017]: https://openreview.net/pdf?id=B1-tVX1Pz\n",
    "[he_et_al_2015]: https://arxiv.org/abs/1511.04636\n",
    "[fulda_et_al_2017]: https://arxiv.org/abs/1703.03429\n",
    "[kostka_et_al_2017]: https://arxiv.org/abs/1705.05637\n",
    "[zelinka_2018]: https://arxiv.org/abs/1801.01999"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}