predictive-text-and-text-generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Text generation with Markovify\n",
    "\n",
    "By [Allison Parrish](http://www.decontextualize.com/)\n",
    "\n",
    "This notebook is a tour of how to generate text with Markov chains! Markov chains are a simple example of *predictive text generation*, a term I use to refer to methods of text generation that make use of statistical model that, given a certain stretch of text, *predicts* which bit of text should come next, based on probabilities learned from an existing corpus of text.\n",
    "\n",
    "The code is written in Python, but you don't really need to know Python in order to use the notebook. Everything's pre-written for you, so you can just execute the cells, making small changes to the code as needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with text files\n",
    "\n",
    "Before we get started, we'll first need some text! Grab two [plain text files from Project Gutenberg](http://www.gutenberg.org/) (or from another source of your choice) and save them to the same directory as this notebook. (I suggest working with two files because we'll be running some code explicitly to \"compare\" two texts. Also, I think seeing two different outputs from the text generation methods discussed in this notebook will help you better understand how those methods work.) The code in the following cell loads into Python variables the contents of *two plain text files*, assigned to variables `text_a` and `text_b`. You'll need to replace the filenames with the names of the files that you downloaded, keeping the quotation marks (`\"`) intact."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "text_a = open(\"1342-0.txt\").read()\n",
    "text_b = open(\"84-0.txt\").read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These variables are *strings*, which are essentially just long lists of the characters that occur in the text, in the order that they occur. The code in the following cell shows the first two hundred characters of text A:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "﻿The Project Gutenberg EBook of Pride and Prejudice, by Jane Austen\n",
      "\n",
      "This eBook is for the use of anyone anywhere at no cost and with\n",
      "almost no restrictions whatsoever.  You may copy it, give it away \n"
     ]
    }
   ],
   "source": [
    "print(text_a[:200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can change `text_a` to `text_b` to see the output from your second text, or change `200` to a number of your choosing.\n",
    "\n",
    "The `random.sample()` function gives us a random sampling of the contents of a variable (as long as that variable is a sequence of things, like a string or a list). So, for example, to see twenty random characters from text B:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['t',\n",
       " 'd',\n",
       " 'i',\n",
       " 'h',\n",
       " ' ',\n",
       " 's',\n",
       " 's',\n",
       " 'r',\n",
       " 'e',\n",
       " 'r',\n",
       " 'i',\n",
       " 'r',\n",
       " 'd',\n",
       " 's',\n",
       " 'g',\n",
       " 'v',\n",
       " 'u',\n",
       " 'd',\n",
       " 'i',\n",
       " 't']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import random\n",
    "random.sample(text_b, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This isn't incredibly helpful on its own, but you'll notice that the characters it drew (probably) more or less follow the expected letter distribution for English (i.e., lots of `e`s and `n`s and `t`s)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perhaps more interesting would be to see a randomly-sampled list of *words*. To do this, we'll make separate variables for the words in the text, using a Python function called `.split()`, which takes a string and turns it into a list of words contained in that string. The following cell makes two new variables that contain the words from both texts respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "a_words = text_a.split()\n",
    "b_words = text_b.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, ten random words from both text A and text B:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['listened',\n",
       " 'a',\n",
       " 'see',\n",
       " 'absence',\n",
       " 'listening',\n",
       " 'she',\n",
       " 'as',\n",
       " 'exhibit.”',\n",
       " 'Fitzwilliam,',\n",
       " 'state']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.sample(a_words, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['saw',\n",
       " 'shepherd.',\n",
       " 'only',\n",
       " 'rustic,',\n",
       " 'being',\n",
       " 'resolved',\n",
       " 'near',\n",
       " 'would',\n",
       " 'feel',\n",
       " 'with']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.sample(b_words, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code in the following cell uses Python's `Counter` object to count the *most common* letters in the first of these texts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(' ', 113941),\n",
       " ('e', 70344),\n",
       " ('t', 47283),\n",
       " ('a', 42156),\n",
       " ('o', 41138),\n",
       " ('n', 38430),\n",
       " ('i', 36273),\n",
       " ('h', 33883),\n",
       " ('r', 33293),\n",
       " ('s', 33292),\n",
       " ('d', 22247),\n",
       " ('l', 21282)]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "Counter(text_a).most_common(12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specifying the `a_words` variable gives the most frequent *words* instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 4205),\n",
       " ('to', 4121),\n",
       " ('of', 3662),\n",
       " ('and', 3309),\n",
       " ('a', 1945),\n",
       " ('her', 1858),\n",
       " ('in', 1813),\n",
       " ('was', 1795),\n",
       " ('I', 1740),\n",
       " ('that', 1419),\n",
       " ('not', 1356),\n",
       " ('she', 1306)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(a_words).most_common(12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare these to the most common words in text B:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 4056),\n",
       " ('and', 2971),\n",
       " ('of', 2741),\n",
       " ('I', 2719),\n",
       " ('to', 2142),\n",
       " ('my', 1631),\n",
       " ('a', 1394),\n",
       " ('in', 1125),\n",
       " ('was', 993),\n",
       " ('that', 987),\n",
       " ('with', 700),\n",
       " ('had', 679)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Counter(b_words).most_common(12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Markov models\n",
    "\n",
    "I won't go into the precise details of how to implement a Markov chain text generator in this notebook. (I have written [a tutorial on this topic](https://github.com/aparrish/rwet/blob/master/ngrams-and-markov-chains.ipynb) elsewhere, however!) But I think it's helpful to understand the fundamentals of how Markov chain text generation works. The next question we’re going to try to answer is this: Given a stretch of text (say a string of characters, or run of words), what is most the most likely bit of text to come next?\n",
    "\n",
    "One way to answer this question is with an n-gram based Markov model. What's an n-gram? I'm glad you asked!\n",
    "\n",
    "### N-grams\n",
    "\n",
    "An n-gram is simply a sequence of units drawn from a longer sequence; in the case of text, the unit in question is usually a character or a word. For convenience, we'll call the unit of the n-gram is called its level; the length of the n-gram is called its order. For example, the following is a list of all unique character-level order-2 n-grams in the word condescendences:\n",
    "\n",
    "    co\n",
    "    on\n",
    "    nd\n",
    "    de\n",
    "    es\n",
    "    sc\n",
    "    ce\n",
    "    en\n",
    "    nc\n",
    "\n",
    "And the following is an excerpt from the list of all unique word-level order-5 n-grams in The Road Not Taken:\n",
    "\n",
    "    Two roads diverged in a\n",
    "    roads diverged in a yellow\n",
    "    diverged in a yellow wood,\n",
    "    in a yellow wood, And\n",
    "    a yellow wood, And sorry\n",
    "    yellow wood, And sorry I\n",
    "\n",
    "N-grams are used frequently in natural language processing and are a basic tool text analysis. Their applications range from programs that correct spelling to creative visualizations to compression algorithms to stylometrics to generative text.\n",
    "\n",
    "### What comes next?\n",
    "\n",
    "A Markov model for text begins with a list of n-grams. But in addition to making this list, we also keep track of what unit of text (word, character, etc.) *follows* each of those n-grams.\n",
    "\n",
    "Let’s do a quick example by hand. This is the same character-level order-2 n-gram analysis of the (very brief) text “condescendences” as above, but this time keeping track of all characters that follow each n-gram:\n",
    "\n",
    "| n-grams |\tnext? |\n",
    "| ------- | ----- |\n",
    "|co| n|\n",
    "|on| d|\n",
    "|nd| e, e|\n",
    "|de| s, n|\n",
    "|es| c, (end of text)|\n",
    "|sc| e|\n",
    "|ce| n, s|\n",
    "|en| d, c|\n",
    "|nc| e|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this table, we can determine that while the n-gram `co` is followed by n 100% of the time, and while the n-gram `on` is followed by `d` 100% of the time, the n-gram `de` is followed by `s` 50% of the time, and `n` the rest of the time. Likewise, the n-gram `es` is followed by `c` 50% of the time, and followed by the end of the text the other 50% of the time.\n",
    "\n",
    "Exercise: Imagine (or even better, write out) what this table might look like if you were analyzing words instead of characters, with a source text of your choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Markov chains: Generating text from a Markov model\n",
    "\n",
    "The Markov models we created above don't just give us interesting statistical probabilities. It also allows us generate a *new* text with those probabilities by *chaining together predictions*. Here’s how we’ll do it, starting with the order 2 character-level Markov model of `condescendences`: (1) start with the initial n-gram (`co`)—those are the first two characters of our output. (2) Now, look at the last *n* characters of output, where *n* is the order of the n-grams in our table, and find those characters in the “n-grams” column. (3) Choose randomly among the possibilities in the corresponding “next” column, and append that letter to the output. (Sometimes, as with `co`, there’s only one possibility). (4) If you chose “end of text,” then the algorithm is over. Otherwise, repeat the process starting with (2). Here’s a record of the algorithm in action:\n",
    "\n",
    "    co\n",
    "    con\n",
    "    cond\n",
    "    conde\n",
    "    conden\n",
    "    condend\n",
    "    condendes\n",
    "    condendesc\n",
    "    condendesce\n",
    "    condendesces\n",
    "    \n",
    "As you can see, we’ve come up with a word that looks like the original word, and could even be passed off as a genuine English word (if you squint at it). From a statistical standpoint, the output of our algorithm is nearly indistinguishable from the input. This kind of algorithm—moving from one state to the next, according to a list of probabilities—is known as a Markov chain generator.\n",
    "\n",
    "### Generating with Markovify\n",
    "\n",
    "Fortunately, with the invention of digital computers, you don't have to perform this algorithm by hand! In fact, Markov chain text generation has been a pastime of poets and programmers going back [all the way to 1983](https://www.jstor.org/stable/24969024), so it should be no surprise that there are many implementations of the idea in Python that you can download and install. The one we're going to use is [Markovify](https://github.com/jsvine/markovify), a Markov chain text generation library originally developed for BuzzFeed, apparently. It comes with a lot of extra niceties that will make our lives easier, but underneath the hood, it implements an algorithm very similar to the one we just did by hand above.\n",
    "\n",
    "To install Markovify on your computer, run the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: markovify in /Users/allison/opt/miniconda3/envs/predictive-text-and-text-generation/lib/python3.9/site-packages (0.8.3)\r\n",
      "Requirement already satisfied: unidecode in /Users/allison/opt/miniconda3/envs/predictive-text-and-text-generation/lib/python3.9/site-packages (from markovify) (1.1.1)\r\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install markovify"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then run this cell to make the library available in your notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import markovify"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code in the following cell creates a new text generator, using the text in the variable specified to build the Markov model, which is then assigned to the variable `generator_a`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "generator_a = markovify.Text(text_a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then call the `.make_sentence()` method to generate a sentence from the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "“I want to see his sister; and, oh! how ardently did she say?”\n"
     ]
    }
   ],
   "source": [
    "print(generator_a.make_sentence())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.make_short_sentence()` method allows you to specify a maximum length for the generated sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As long as possible.\n"
     ]
    }
   ],
   "source": [
    "print(generator_a.make_short_sentence(50))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, Markovify tries to generate a sentence that is significantly different from any existing sentence in the input text. As a consequence, sometimes the `.make_sentence()` or `.make_short_sentence()` methods will return `None`, which means that in ten tries it wasn't able to generate such a sentence. You can work around this by increasing the number of times it tries to generate a sufficiently unique sentence using the `tries` parameter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elizabeth was forced to rejoice.\n"
     ]
    }
   ],
   "source": [
    "print(generator_a.make_short_sentence(40, tries=100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or by disabling the check altogether with `test_output=False` (note that this means the generator will occasionally return stretches of text that are present in the source text):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mrs. Bennet, “that is rather singular.”\n"
     ]
    }
   ],
   "source": [
    "print(generator_a.make_short_sentence(40, test_output=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changing the order\n",
    "\n",
    "When you create the model, you can specify the order of the model using the `state_size` parameter. It defaults to 2. Let's make two model with different orders and compare:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "gen_a_1 = markovify.Text(text_a, state_size=1)\n",
    "gen_a_4 = markovify.Text(text_a, state_size=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "order 1\n",
      "“_My_ overhearings were miserable.\n",
      "\n",
      "order 4\n",
      "As they walked across the hall towards the river, Elizabeth turned back to look again; her uncle and aunt had already lost three days of happiness, and immediately wrote as follows: “I would have thanked you before, my dear aunt, as I ought to have done, for your long, kind, satisfactory, detail of particulars; but to say the truth, I was too cross to write.\n"
     ]
    }
   ],
   "source": [
    "print(\"order 1\")\n",
    "print(gen_a_1.make_sentence(test_output=False))\n",
    "print()\n",
    "print(\"order 4\")\n",
    "print(gen_a_4.make_sentence(test_output=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In general, the higher the order, the more the sentences will seem \"coherent\" (i.e., more closely resembling the source text). Lower order models will produce more variation. Deciding on the order is usually a matter of taste and trial-and-error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changing the level\n",
    "\n",
    "Markovify, by default, works with *words* as the individual unit. It doesn't come out-of-the-box with support for character-level models. The following code defines a new kind of Markovify generator that implements character-level models. Execute it before continuing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "class SentencesByChar(markovify.Text):\n",
    "    def word_split(self, sentence):\n",
    "        return list(sentence)\n",
    "    def word_join(self, words):\n",
    "        return \"\".join(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any of the parameters you passed to `markovify.Text` you can also pass to `SentencesByChar`. The `state_size` parameter still controls the order of the model, but now the n-grams are characters, not words.\n",
    "\n",
    "The following cell implements a character-level Markov text generator for the word \"condescendences\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "con_model = SentencesByChar(\"condescendences\", state_size=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the cell below to see the output—it'll be a lot like what we implemented by hand earlier!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'condescendescences'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "con_model.make_sentence()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, you can use a character-level model on any text of your choice. So, for example, the following cell creates a character-level order-7 Markov chain text generator from text A:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "gen_a_char = SentencesByChar(text_a, state_size=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the cell below prints out a random sentence from this generator. (The `.replace()` is to get rid of any newline characters in the output.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "As I must acknowledged, however, they can get.\n"
     ]
    }
   ],
   "source": [
    "print(gen_a_char.make_sentence(test_output=False).replace(\"\\n\", \" \"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining models\n",
    "\n",
    "Markovify has a handy feature that allows you to *combine* models, creating a new model that draws on probabilities from both of the source models. You can use this to create hybrid output that mixes the style and content of two (or more!) different source texts. To do this, you need to create the models independently, and then call `.combine()` to combine them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "generator_a = markovify.Text(text_a)\n",
    "generator_b = markovify.Text(text_b)\n",
    "combo = markovify.combine([generator_a, generator_b], [0.5, 0.5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The bit of code `[0.5, 0.5]` controls the \"weights\" of the models, i.e., how much to emphasize the probabilities of any model. You can change this to suit your tastes. (E.g., if you want mostly text A with but a *soupçon* of text B, you would write `[0.9, 0.1]`. Try it!) \n",
    "\n",
    "Then you can create sentences using the combined model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We all know that his religion and taught to look on the subject, but that the arguments with which I feared to meet her again.\n"
     ]
    }
   ],
   "source": [
    "print(combo.make_sentence())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bringing it all together\n",
    "\n",
    "I've pre-written some code below to make it easy for you to experiment and produce output from Markovify. Just make adjustments to the values assigned to the variables in the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# change to \"word\" for a word-level model\n",
    "level = \"char\"\n",
    "# controls the length of the n-gram\n",
    "order = 7\n",
    "# controls the number of lines to output\n",
    "output_n = 14\n",
    "# weights between the models; text A first, text B second.\n",
    "# if you want to completely exclude one model, set its corresponding value to 0\n",
    "weights = [0.5, 0.5]\n",
    "# limit sentence output to this number of characters\n",
    "length_limit = 280"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(The lines beginning with `#` are \"comments\"—they don't do anything, they're just there to explain what's happening in the code.)\n",
    "\n",
    "After making your changes above, run the cell below to generate text according to your parameters. Repeat as necessary until you get something you really like!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "He was really instructed and repeatedly turned to Charlotte's first began.\n",
      "\n",
      "My ancestors had manifested that would yet had been brought to provoke Darcy should think it a fainted.\n",
      "\n",
      "Women fainted.\n",
      "\n",
      "The completed; and its extreme profundity, until I almost as far enough to hold this explaining.\n",
      "\n",
      "“So much mistaken in.\n",
      "\n",
      "Your tempered, miserable elopement with extremest agitation for many years after breakfast a few stars, it will be in this likely to have them before the dead at my first shock of procure.\n",
      "\n",
      "A military being perpetual fretting between the occasion requiring after _that_, I am a married, but a conversation with almost to London.\n",
      "\n",
      "“I do not give us a ball.\n",
      "\n",
      "In vain did Elizabeth ventured on me.”\n",
      "\n",
      "In a few grey hairs covered them in the lane, who married.\n",
      "\n",
      "Three years old.\n",
      "\n",
      "“What a persuade him to the probably been soon to what promises fairly; and the church.\n",
      "\n",
      "They soon cease to return Mr. Bingley's continued:  “Lizzy, let me be at home, therefore, to admire enough to endure; when a selfish pursuit?\n",
      "\n",
      "She could reflections, all out of its propositions of my truth of its usual querulous serene constructed her eyes.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model_cls = markovify.Text if level == \"word\" else SentencesByChar\n",
    "gen_a = model_cls(text_a, state_size=order)\n",
    "gen_b = model_cls(text_b, state_size=order)\n",
    "gen_combo = markovify.combine([gen_a, gen_b], weights)\n",
    "for i in range(output_n):\n",
    "    out = gen_combo.make_short_sentence(length_limit, test_output=False)\n",
    "    out = out.replace(\"\\n\", \" \")\n",
    "    print(out)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating with non-prose text\n",
    "\n",
    "Markovify assumes you're feeding it prose, i.e., a text file that can be parsed into sentences by separating on sentence-ending punctuation. But often you're *not* working with text like this. For example, let's generate some sonnets. First, download [this plaintext version of Shakespeare's sonnets](https://raw.githubusercontent.com/aparrish/plaintext-example-files/master/sonnets.txt) and keep it in the same directory as this notebook. We'll define the sonnet-generating task as consisting of (a) training a Markov chain on lines of poetry and then (b) generating a sequence of fourteen lines of poetry. Since the *line* is the unit now and not the *sentence*, we need to use Markovify's `NewlineText` class instead of `Text`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "sonnets_text = open(\"sonnets.txt\").read()\n",
    "sonnets_model = markovify.NewlineText(sonnets_text, state_size=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we can generate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"For thy light's flame to make time's waste:\""
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sonnets_model.make_sentence()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now make a sonnet, sorta:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thy pity like a face?\n",
      "Or bends with all those.\n",
      "Although thou art forced to thee, when he may not so of self-doing crime.\n",
      "Could make you see aright?\n",
      "For thee, as thou wilt look,\n",
      "And therefore love, when first my verse alone kingdoms of self-doing crime.\n",
      "Save breed, to say you a league is she might the gaudy spring,\n",
      "The fairest creatures broke away,\n",
      "Oaths of yours must strive\n",
      "Therefore I fear,\n",
      "Deserves the subject that thy years told:\n",
      "Then happy me! but yet, like a seeting bath, which methinks I should example where I am to my love, that sorrow, which gives thee with winter hath taught it thy beauty herself is so long,\n",
      "I better in thee to be forgot,\n",
      "To give back again is took,\n"
     ]
    }
   ],
   "source": [
    "for i in range(14):\n",
    "    print(sonnets_model.make_sentence())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doing this with a character-level model is a bit more tricky. I've written code in the cell below that defines a new class, `LinesByCharacter` that works like `NewlineText` but operates character-by-character instead of word-by-word:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LinesByChar(markovify.NewlineText):\n",
    "    def word_split(self, sentence):\n",
    "        return list(sentence)\n",
    "    def word_join(self, words):\n",
    "        return \"\".join(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create a character model with the sonnets, line by line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "sonnets_char_model = LinesByChar(sonnets_text, state_size=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And generate new sonnets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "One of hatred when in the be so blunt back nights in her proceeds:\n",
      "And from thy fair, and see:\n",
      "And faint of and the voice thy swear is best once with mask'd the deathese powerfumes invention, like a false with from her repair,\n",
      "Which hall the night,\n",
      "The leaven,\n",
      "For take a hours, by title morn\n",
      "The worth thee, form death desire is black lines thy deeds;\n",
      "And all upon their riot ease child me not confound age's mine eye away,\n",
      "My her.\n",
      "Since canopy disgrace image infect.\n",
      "Which now unfair witherein I do shall reness, where bore to temperfect thou away, year, when of a world's flatter stain,\n",
      "It is pleast,\n",
      "Lo! in their breat true my mother plead where fresh, who art compounds,\n",
      "Doubting lack of bloody tyrannot action but drown vision laughts, are of your critied be.\n"
     ]
    }
   ],
   "source": [
    "for i in range(14):\n",
    "    print(sonnets_char_model.make_sentence())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### New moods\n",
    "\n",
    "Character-level Markov chains are especially suitable, in my experience, for generating shorter texts, like individual words or names. Let's generate names of new moods using this technique. First, download [this JSON file of moods](https://raw.githubusercontent.com/dariusk/corpora/9bb62927951f79bec2454f29d71b6e9b28d874b1/data/humans/moods.json) from [Corpora Project](https://github.com/dariusk/corpora/) and save to the same directory as this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then load the JSON file and grab just the list of words naming moods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "mood_data = json.loads(open(\"./moods.json\").read())\n",
    "moods = mood_data['moods']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easiest way to use this is to make one big string with the moods joined together with newlines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "moods_text = \"\\n\".join(moods)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then use `LinesByChar` to make the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "moods_char_model = LinesByChar(moods_text, state_size=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And voila, new moods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cowarlessive\n",
      "toughted\n",
      "lyricapative\n",
      "convictorted\n",
      "embarratived\n",
      "teasane\n",
      "feistative\n",
      "powerloaded\n",
      "bothetic\n",
      "rejuvenaccepted\n",
      "teassioned\n",
      "innovatired\n",
      "chieved\n",
      "alievoless\n",
      "incomperational\n",
      "greterpressived\n",
      "imped\n",
      "soreborn\n",
      "lethant\n",
      "jadequashamed\n",
      "intemplifeless\n",
      "deterrientalkatirical\n",
      "enconfidead\n",
      "strappallous\n"
     ]
    }
   ],
   "source": [
    "for i in range(24):\n",
    "    print(moods_char_model.make_sentence())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further reading\n",
    "\n",
    "* Hayes, Brian. “Computer recreations.” Scientific American, vol. 249, no. 5, 1983, pp. 18–31. JSTOR, http://www.jstor.org/stable/24969024. (Original column from Scientific American that described how Markov chain text generation works—very readable! I can send a PDF, hit me up.)\n",
    "* [A Travesty Generator for Micros](https://elmcip.net/critical-writing/travesty-generator-micros) is a follow-up to Hayes' article that has some more theory and an actual Pascal listing (which is now mostly of only historical interest).\n",
    "* [This notebook](https://github.com/aparrish/rwet/blob/master/ngrams-and-markov-chains.ipynb) shows how to implement a Markov chain generator from scratch in Python, if you're interested in such things!\n",
    "* Lillian-Yvonne Bertram's [Travesty Generator](http://www.noemipress.org/catalog/poetry/travesty-generator/) is a striking example of Markov chains put to poetic use."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}