notebooks/GrammarFuzzer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": true,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Efficient Grammar Fuzzing\n",
    "\n",
    "In the [chapter on grammars](Grammars.ipynb), we have seen how to use _grammars_ for very effective and efficient testing.  In this chapter, we refine the previous string-based algorithm into a tree-based algorithm, which is much faster and allows for much more control over the production of fuzz inputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": true,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The algorithm in this chapter serves as a foundation for several more techniques; this chapter thus is a \"hub\" in the book."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "**Prerequisites**\n",
    "\n",
    "* You should know how grammar-based fuzzing works, e.g. from the [chapter on grammars](Grammars.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## An Insufficient Algorithm\n",
    "\n",
    "In the [previous chapter](Grammars.ipynb), we have introduced the `simple_grammar_fuzzer()` function which takes a grammar and automatically produces a syntactically valid string from it.  However, `simple_grammar_fuzzer()` is just what its name suggests – simple.  To illustrate the problem, let us get back to the `expr_grammar` we created from `EXPR_GRAMMAR_BNF` in the [chapter on grammars](Grammars.ipynb):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import fuzzingbook_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fuzzingbook_utils import unicode_escape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Grammars import EXPR_EBNF_GRAMMAR, convert_ebnf_grammar, simple_grammar_fuzzer, is_valid_grammar, exp_string, exp_opts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "expr_grammar = convert_ebnf_grammar(EXPR_EBNF_GRAMMAR)\n",
    "expr_grammar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "`expr_grammar` has an interesting property.  If we feed it into `simple_grammar_fuzzer()`, the function gets stuck in an infinite expansion:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from ExpectError import ExpectTimeout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "with ExpectTimeout(1):\n",
    "    simple_grammar_fuzzer(grammar=expr_grammar, max_nonterminals=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Why is that so?  The problem is in this rule:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "expr_grammar['<factor>']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Here, any choice except for `(expr)` increases the number of symbols, even if only temporary.  Since we place a hard limit on the number of symbols to expand, the only choice left for expanding `<factor>` is `(<expr>)`, which leads to an infinite addition of parentheses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The problem of potentially infinite expansion is only one of the problems with `simple_grammar_fuzzer()`.  More problems include:\n",
    "\n",
    "1. *It is inefficient*.  With each iteration, this fuzzer would go search the string produced so far for symbols to expand.  This becomes inefficient as the production string grows.\n",
    "\n",
    "2. *It is hard to control.*  Even while limiting the number of symbols, it is still possible to obtain very long strings – and even infinitely long ones, as discussed above.\n",
    "\n",
    "Let us illustrate both problems by plotting the time required for strings of different lengths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Grammars import simple_grammar_fuzzer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Grammars import START_SYMBOL, EXPR_GRAMMAR, URL_GRAMMAR, CGI_GRAMMAR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Grammars import RE_NONTERMINAL, nonterminals, is_nonterminal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Timer import Timer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "trials = 50\n",
    "xs = []\n",
    "ys = []\n",
    "for i in range(trials):\n",
    "    with Timer() as t:\n",
    "        s = simple_grammar_fuzzer(EXPR_GRAMMAR, max_nonterminals=15)\n",
    "    xs.append(len(s))\n",
    "    ys.append(t.elapsed_time())\n",
    "    print(i, end=\" \")\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "average_time = sum(ys) / trials\n",
    "print(\"Average time:\", average_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "plt.scatter(xs, ys)\n",
    "plt.title('Time required for generating an output')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We see that (1) the effort increases quadratically over time, and (2) we can easily produce outputs that are tens of thousands of characters long."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To address these problems, we need a _smarter algorithm_ – one that is more efficient, that gets us better control over expansions, and that is able to foresee in `expr_grammar` that the `(expr)` alternative yields a potentially infinite expansion, in contrast to the other two."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Derivation Trees\n",
    "\n",
    "To both obtain a more efficient algorithm _and_ exercise better control over expansions, we will use a special representation for the strings that our grammar produces.  The general idea is to use a *tree* structure that will be subsequently expanded – a so-called *derivation tree*.  This representation allows us to always keep track of our expansion status – answering questions such as which elements have been expanded into which others, and which symbols still need to be expanded.  Furthermore, adding new elements to a tree is far more efficient than replacing strings again and again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Like other trees used in programming, a derivation tree (also known as *parse tree* or *concrete syntax tree*) consists of *nodes* which have other nodes (called *child nodes*) as their *children*.  The tree starts with one node that has no parent; this is called the *root node*; a node without children is called a *leaf*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The grammar expansion process with derivation trees is illustrated in the following steps, using the arithmetic grammar [from\n",
    "the chapter on grammars](Grammars.ipynb).  We start with a single node as root of the tree, representing the *start symbol* – in our case `<start>`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ipub": {
     "ignore": true
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "(We use `dot` as a drawing program; you don't need to look at the code, just at its results.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "ipub": {
     "ignore": true
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from graphviz import Digraph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "ipub": {
     "ignore": true
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tree = Digraph(\"root\")\n",
    "tree.attr('node', shape='plain')\n",
    "tree.node(r\"\\<start\\>\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To expand the tree, we traverse it, searching for a nonterminal symbol $S$ without children.  $S$ thus is a symbol that still has to be expanded.  We then chose an expansion for $S$ from the grammar.  Then, we add the expansion as a new child of $S$.  For our start symbol `<start>`, the only expansion is `<expr>`, so we add it as a child."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "ipub": {
     "ignore": true
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tree.edge(r\"\\<start\\>\", r\"\\<expr\\>\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To construct the produced string from a derivation tree, we traverse the tree in order and collect the symbols at the leaves of the tree.  In the case above, we obtain the string `\"<expr>\"`.\n",
    "\n",
    "To further expand the tree, we choose another symbol to expand, and add its expansion as new children.  This would get us the `<expr>` symbol, which gets expanded into `<expr> + <term>`, adding three children."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "ipub": {
     "ignore": true
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tree.edge(r\"\\<expr\\>\", r\"\\<expr\\> \")\n",
    "tree.edge(r\"\\<expr\\>\", r\"+\")\n",
    "tree.edge(r\"\\<expr\\>\", r\"\\<term\\>\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We repeat the expansion until there are no symbols left to expand:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "ipub": {
     "ignore": true
    },
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "tree.edge(r\"\\<expr\\> \", r\"\\<term\\> \")\n",
    "tree.edge(r\"\\<term\\> \", r\"\\<factor\\> \")\n",
    "tree.edge(r\"\\<factor\\> \", r\"\\<integer\\> \")\n",
    "tree.edge(r\"\\<integer\\> \", r\"\\<digit\\> \")\n",
    "tree.edge(r\"\\<digit\\> \", r\"2 \")\n",
    "\n",
    "tree.edge(r\"\\<term\\>\", r\"\\<factor\\>\")\n",
    "tree.edge(r\"\\<factor\\>\", r\"\\<integer\\>\")\n",
    "tree.edge(r\"\\<integer\\>\", r\"\\<digit\\>\")\n",
    "tree.edge(r\"\\<digit\\>\", r\"2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We now have a representation for the string `2 + 2`.  In contrast to the string alone, though, the derivation tree records _the entire structure_ (and production history, or _derivation_ history) of the produced string.  It also allows for simple comparison and manipulation – say, replacing one subtree (substructure) against another."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Representing Derivation Trees\n",
    "\n",
    "To represent a derivation tree in Python, we use the following format.  A node is a pair\n",
    "\n",
    "```python\n",
    "(SYMBOL_NAME, CHILDREN)\n",
    "```\n",
    "\n",
    "where `SYMBOL_NAME` is a string representing the node (i.e. `\"<start>\"` or `\"+\"`) and `CHILDREN` is a list of children nodes.\n",
    "\n",
    "`CHILDREN` can take some special values:\n",
    "\n",
    "1. `None` as a placeholder for future expansion.  This means that the node is a *nonterminal symbol* that should be expanded further.\n",
    "2. `[]` (i.e., the empty list) to indicate _no_ children.  This means that the node is a *terminal symbol* that can no longer be expanded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let us take a very simple derivation tree, representing the intermediate step `<expr> + <term>`, above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "derivation_tree = (\"<start>\",\n",
    "                   [(\"<expr>\",\n",
    "                     [(\"<expr>\", None),\n",
    "                      (\" + \", []),\n",
    "                         (\"<term>\", None)]\n",
    "                     )])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To better understand the structure of this tree, let us introduce a function that visualizes this tree.  We use the `dot` drawing program from the `graphviz` package algorithmically, traversing the above structure.  (Unless you're deeply interested in tree visualization, you can directly skip to the example below.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from graphviz import Digraph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def dot_escape(s):\n",
    "    \"\"\"Return s in a form suitable for dot\"\"\"\n",
    "    s = re.sub(r'([^a-zA-Z0-9\" ])', r\"\\\\\\1\", s)\n",
    "    return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "assert dot_escape(\"hello\") == \"hello\"\n",
    "assert dot_escape(\"<hello>, world\") == \"\\\\<hello\\\\>\\\\, world\"\n",
    "assert dot_escape(\"\\\\n\") == \"\\\\\\\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "While we are interested at present in visualizing a `derivation_tree`, it is in our interest to generalize the visualization procedure. In particular, it would be helpful if our method `display_tree()` can display *any* tree    like data structure. To enable this, we define a helper method `extract_node()` that extract the current symbol and  children from a given data structure. The default implementation simply extracts the symbol, children, and           annotation from any `derivation_tree` node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def extract_node(node, id):\n",
    "    symbol, children, *annotation = node\n",
    "    return symbol, children, ''.join(str(a) for a in annotation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "While visualizing a tree, it is often useful to display certain nodes differently. For example, it is sometimes useful to distinguish between non-processed nodes and processed nodes. We define a helper procedure `default_node_attr()` that provides the basic display, which can be customized by the user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def default_node_attr(dot, nid, symbol, ann):\n",
    "    dot.node(repr(nid), dot_escape(unicode_escape(symbol)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to nodes, the edges may also require modifications. We define `default_edge_attr()` as a helper        procedure that can be customized by the user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def default_edge_attr(dot, start_node, stop_node):\n",
    "    dot.edge(repr(start_node), repr(stop_node))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While visualizing a tree, one may sometimes wish to change the appearance of the tree. For example, it is      sometimes easier to view the tree if it was laid out left to right rather than top to bottom. We define another      helper procedure `default_graph_attr()` for that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def default_graph_attr(dot):\n",
    "    dot.attr('node', shape='plain')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we define a method `display_tree()` that accepts these four functions `extract_node()`,               `default_edge_attr()`, `default_node_attr()` and `default_graph_attr()` and uses them to display the tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_tree(derivation_tree,\n",
    "                 log=False,\n",
    "                 extract_node=extract_node,\n",
    "                 node_attr=default_node_attr,\n",
    "                 edge_attr=default_edge_attr,\n",
    "                 graph_attr=default_graph_attr):\n",
    "    counter = 0\n",
    "\n",
    "    def traverse_tree(dot, tree, id=0):\n",
    "        (symbol, children, annotation) = extract_node(tree, id)\n",
    "        node_attr(dot, id, symbol, annotation)\n",
    "\n",
    "        if children:\n",
    "            for child in children:\n",
    "                nonlocal counter\n",
    "                counter += 1\n",
    "                child_id = counter\n",
    "                edge_attr(dot, id, child_id)\n",
    "                traverse_tree(dot, child, child_id)\n",
    "\n",
    "    dot = Digraph(comment=\"Derivation Tree\")\n",
    "    graph_attr(dot)\n",
    "    traverse_tree(dot, derivation_tree)\n",
    "    if log:\n",
    "        print(dot)\n",
    "    display(dot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is what our tree visualizes into:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say we want to customize our output, where we want to annotate certain nodes and edges. Here is a method       `display_annotated_tree()` that displays an annotated tree structure, and lays the graph out left to right."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_annotated_tree(tree, a_nodes, a_edges, log=False):\n",
    "    def graph_attr(dot):\n",
    "        dot.attr('node', shape='plain')\n",
    "        dot.graph_attr['rankdir'] = 'LR'\n",
    "\n",
    "    def annotate_node(dot, nid, symbol, ann):\n",
    "        if nid in a_nodes:\n",
    "            dot.node(repr(nid), \"%s (%s)\" % (dot_escape(unicode_escape(symbol)), a_nodes[nid]))\n",
    "        else:\n",
    "            dot.node(repr(nid), dot_escape(unicode_escape(symbol)))\n",
    "\n",
    "    def annotate_edge(dot, start_node, stop_node):\n",
    "        if (start_node, stop_node) in a_edges:\n",
    "            dot.edge(repr(start_node), repr(stop_node),\n",
    "                     a_edges[(start_node, stop_node)])\n",
    "        else:\n",
    "            dot.edge(repr(start_node), repr(stop_node))\n",
    "\n",
    "    display_tree(tree, log=log,\n",
    "                 node_attr=annotate_node,\n",
    "                 edge_attr=annotate_edge,\n",
    "                 graph_attr=graph_attr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_annotated_tree(derivation_tree, {3: 'plus'}, {(1, 3): 'op'}, log=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to see all the leaf nodes in a tree, the following `all_terminals()` function comes in handy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def all_terminals(tree):\n",
    "    (symbol, children) = tree\n",
    "    if children is None:\n",
    "        # This is a nonterminal symbol not expanded yet\n",
    "        return symbol\n",
    "\n",
    "    if len(children) == 0:\n",
    "        # This is a terminal symbol\n",
    "        return symbol\n",
    "\n",
    "    # This is an expanded symbol:\n",
    "    # Concatenate all terminal symbols from all children\n",
    "    return ''.join([all_terminals(c) for c in children])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "all_terminals(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `all_terminals()` function returns the string representation of all leaf nodes. However, some of these leaf nodes may be due to nonterminals deriving empty strings. For these, we want to return the empty string. Hence, we define a new function `tree_to_string()` to retrieve the original string back from a tree like structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tree_to_string(tree):\n",
    "    symbol, children, *_ = tree\n",
    "    if children:\n",
    "        return ''.join(tree_to_string(c) for c in children)\n",
    "    else:\n",
    "        return '' if is_nonterminal(symbol) else symbol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_to_string(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Expanding a Node"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let us now develop an algorithm that takes a tree with unexpanded symbols (say, `derivation_tree`, above), and expands all these symbols one after the other.  As with earlier fuzzers, we create a special subclass of `Fuzzer` – in this case, `GrammarFuzzer`.  A `GrammarFuzzer` gets a grammar and a start symbol; the other parameters will be used later to further control creation and to support debugging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from Fuzzer import Fuzzer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(Fuzzer):\n",
    "    def __init__(self, grammar, start_symbol=START_SYMBOL,\n",
    "                 min_nonterminals=0, max_nonterminals=10, disp=False, log=False):\n",
    "        self.grammar = grammar\n",
    "        self.start_symbol = start_symbol\n",
    "        self.min_nonterminals = min_nonterminals\n",
    "        self.max_nonterminals = max_nonterminals\n",
    "        self.disp = disp\n",
    "        self.log = log\n",
    "        self.check_grammar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In the following, we will add further methods to `GrammarFuzzer`, using the hack already introduced for [the `MutationFuzzer` class](MutationFuzzer.ipynb).  The construct\n",
    "\n",
    "```python\n",
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def new_method(self, args):\n",
    "        pass\n",
    "```\n",
    "\n",
    "allows us to add a new method `new_method()` to the `GrammarFuzzer` class. (Actually, we get a new `GrammarFuzzer` class that extends the old one, but for all our purposes, this does not matter.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this hack, let us define a helper method `check_grammar()` that checks the given grammar for consistency:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def check_grammar(self):\n",
    "        assert self.start_symbol in self.grammar\n",
    "        assert is_valid_grammar(\n",
    "            self.grammar,\n",
    "            start_symbol=self.start_symbol,\n",
    "            supported_opts=self.supported_opts())\n",
    "\n",
    "    def supported_opts(self):\n",
    "        return set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let us now define a helper method `init_tree()` that constructs a tree with just the start symbol:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def init_tree(self):\n",
    "        return (self.start_symbol, None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f = GrammarFuzzer(EXPR_GRAMMAR)\n",
    "display_tree(f.init_tree())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Next, we will need a helper function `expansion_to_children()` that takes an expansion string and decomposes it into a list of derivation trees – one for each symbol (terminal or nonterminal) in the string.  It uses the `re.split()` method to split an expansion string into a list of children nodes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def expansion_to_children(expansion):\n",
    "    # print(\"Converting \" + repr(expansion))\n",
    "    # strings contains all substrings -- both terminals and nonterminals such\n",
    "    # that ''.join(strings) == expansion\n",
    "\n",
    "    expansion = exp_string(expansion)\n",
    "    assert isinstance(expansion, str)\n",
    "\n",
    "    if expansion == \"\":  # Special case: epsilon expansion\n",
    "        return [(\"\", [])]\n",
    "\n",
    "    strings = re.split(RE_NONTERMINAL, expansion)\n",
    "    return [(s, None) if is_nonterminal(s) else (s, [])\n",
    "            for s in strings if len(s) > 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "expansion_to_children(\"<term> + <expr>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The case of an *epsilon expansion*, i.e. expanding into an empty string as in `<symbol> ::=` needs special treatment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "expansion_to_children(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Just like `nonterminals()` in the [chapter on Grammars](Grammars.ipynb), we provide for future extensions, allowing the expansion to be a tuple with extra data (which will be ignored)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "expansion_to_children((\"+<term>\", [\"extra_data\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We realize this helper as a method in `GrammarFuzzer` such that it can be overloaded by subclasses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def expansion_to_children(self, expansion):\n",
    "        return expansion_to_children(expansion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "With this, we can now take some unexpanded node in the tree, choose a random expansion, and return the new tree.  This is what the method `expand_node_randomly()` does, using a helper function `choose_node_expansion()` to randomly pick an index from an array of possible children.  (`choose_node_expansion()` can be overloaded in subclasses.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def choose_node_expansion(self, node, possible_children):\n",
    "        \"\"\"Return index of expansion in `possible_children` to be selected.  Defaults to random.\"\"\"\n",
    "        return random.randrange(0, len(possible_children))\n",
    "\n",
    "    def expand_node_randomly(self, node):\n",
    "        (symbol, children) = node\n",
    "        assert children is None\n",
    "\n",
    "        if self.log:\n",
    "            print(\"Expanding\", all_terminals(node), \"randomly\")\n",
    "\n",
    "        # Fetch the possible expansions from grammar...\n",
    "        expansions = self.grammar[symbol]\n",
    "        possible_children = [self.expansion_to_children(\n",
    "            expansion) for expansion in expansions]\n",
    "\n",
    "        # ... and select a random expansion\n",
    "        index = self.choose_node_expansion(node, possible_children)\n",
    "        chosen_children = possible_children[index]\n",
    "\n",
    "        # Process children (for subclasses)\n",
    "        chosen_children = self.process_chosen_children(chosen_children,\n",
    "                                                       expansions[index])\n",
    "\n",
    "        # Return with new children\n",
    "        return (symbol, chosen_children)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The generic `expand_node()` method can later be used to select different expansion strategies; as of now, it only uses `expand_node_randomly()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def expand_node(self, node):\n",
    "        return self.expand_node_randomly(node)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The helper function `process_chosen_children()` does nothing; it can be overloaded by subclasses to process the children once chosen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def process_chosen_children(self, chosen_children, expansion):\n",
    "        \"\"\"Process children after selection.  By default, does nothing.\"\"\"\n",
    "        return chosen_children"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "This is how `expand_node_randomly()` works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f = GrammarFuzzer(EXPR_GRAMMAR, log=True)\n",
    "\n",
    "print(\"Before:\")\n",
    "tree = (\"<integer>\", None)\n",
    "display_tree(tree)\n",
    "\n",
    "print(\"After:\")\n",
    "tree = f.expand_node_randomly(tree)\n",
    "display_tree(tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Expanding a Tree\n",
    "\n",
    "Let us now apply the above node expansion to some node in the tree.  To this end, we first need to search the tree for unexpanded nodes.  `possible_expansions()` counts how many unexpanded symbols there are in a tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def possible_expansions(self, node):\n",
    "        (symbol, children) = node\n",
    "        if children is None:\n",
    "            return 1\n",
    "\n",
    "        return sum(self.possible_expansions(c) for c in children)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f = GrammarFuzzer(EXPR_GRAMMAR)\n",
    "print(f.possible_expansions(derivation_tree))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The method `any_possible_expansions()` returns True if the tree has any unexpanded nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def any_possible_expansions(self, node):\n",
    "        (symbol, children) = node\n",
    "        if children is None:\n",
    "            return True\n",
    "\n",
    "        return any(self.any_possible_expansions(c) for c in children)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f = GrammarFuzzer(EXPR_GRAMMAR)\n",
    "f.any_possible_expansions(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here comes `expand_tree_once()`, the core method of our tree expansion algorithm.  It first checks whether it is currently being applied on a nonterminal symbol without expansion; if so, it invokes `expand_node()` on it, as discussed above.  \n",
    "\n",
    "If the node is already expanded (i.e. has children), it checks the subset of children which still have unexpanded symbols, randomly selects one of them, and applies itself recursively on that child.\n",
    "\n",
    "The `expand_tree_once()` method replaces the child _in place_, meaning that it actually mutates the tree being passed as an argument rather than returning a new tree.  This in-place mutation is what makes this function particularly efficient.  Again, we use a helper method (`choose_tree_expansion()`) to return the chosen index from a list of children that can be expanded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def choose_tree_expansion(self, tree, children):\n",
    "        \"\"\"Return index of subtree in `children` to be selected for expansion.  Defaults to random.\"\"\"\n",
    "        return random.randrange(0, len(children))\n",
    "\n",
    "    def expand_tree_once(self, tree):\n",
    "        \"\"\"Choose an unexpanded symbol in tree; expand it.  Can be overloaded in subclasses.\"\"\"\n",
    "        (symbol, children) = tree\n",
    "        if children is None:\n",
    "            # Expand this node\n",
    "            return self.expand_node(tree)\n",
    "\n",
    "        # Find all children with possible expansions\n",
    "        expandable_children = [\n",
    "            c for c in children if self.any_possible_expansions(c)]\n",
    "\n",
    "        # `index_map` translates an index in `expandable_children`\n",
    "        # back into the original index in `children`\n",
    "        index_map = [i for (i, c) in enumerate(children)\n",
    "                     if c in expandable_children]\n",
    "\n",
    "        # Select a random child\n",
    "        child_to_be_expanded = \\\n",
    "            self.choose_tree_expansion(tree, expandable_children)\n",
    "\n",
    "        # Expand in place\n",
    "        children[index_map[child_to_be_expanded]] = \\\n",
    "            self.expand_tree_once(expandable_children[child_to_be_expanded])\n",
    "\n",
    "        return tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let's put it to use, expanding our derivation tree from above twice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "derivation_tree = (\"<start>\",\n",
    "                   [(\"<expr>\",\n",
    "                     [(\"<expr>\", None),\n",
    "                      (\" + \", []),\n",
    "                         (\"<term>\", None)]\n",
    "                     )])\n",
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "f = GrammarFuzzer(EXPR_GRAMMAR, log=True)\n",
    "derivation_tree = f.expand_tree_once(derivation_tree)\n",
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "derivation_tree = f.expand_tree_once(derivation_tree)\n",
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We see that with each step, one more symbol is expanded.  Now all it takes is to apply this again and again, expanding the tree further and further."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Closing the Expansion\n",
    "\n",
    "With `expand_tree_once()`, we can keep on expanding the tree – but how do we actually stop?  The key idea here, introduced by Luke in \\cite{Luke2000}, is that after inflating the derivation tree to some maximum size, we _only want to apply expansions that increase the size of the tree by a minimum_.  For `<factor>`, for instance, we would prefer an expansion into `<integer>`, as this will not introduce further recursion (and potential size inflation); for `<integer>`, likewise, an expansion into `<digit>` is preferred, as it will less increase tree size than `<digit><integer>`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To identify the _cost_ of expanding a symbol, we introduce two functions that mutually rely on each other:\n",
    "\n",
    "* `symbol_cost()` returns the minimum cost of all expansions of a symbol, using `expansion_cost()` to compute the cost for each expansion.\n",
    "* `expansion_cost()` returns the sum of all expansions in `expansions`.  If a nonterminal is encountered again during traversal, the cost of the expansion is $\\infty$, indicating (potentially infinite) recursion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def symbol_cost(self, symbol, seen=set()):\n",
    "        expansions = self.grammar[symbol]\n",
    "        return min(self.expansion_cost(e, seen | {symbol}) for e in expansions)\n",
    "\n",
    "    def expansion_cost(self, expansion, seen=set()):\n",
    "        symbols = nonterminals(expansion)\n",
    "        if len(symbols) == 0:\n",
    "            return 1  # no symbol\n",
    "\n",
    "        if any(s in seen for s in symbols):\n",
    "            return float('inf')\n",
    "\n",
    "        # the value of a expansion is the sum of all expandable variables\n",
    "        # inside + 1\n",
    "        return sum(self.symbol_cost(s, seen) for s in symbols) + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here's two examples: The minimum cost of expanding a digit is 1, since we have to choose from one of its expansions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f = GrammarFuzzer(EXPR_GRAMMAR)\n",
    "assert f.symbol_cost(\"<digit>\") == 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The minimum cost of expanding `<expr>`, though, is five, as this is the minimum number of expansions required.  (`<expr>` $\\rightarrow$ `<term>` $\\rightarrow$ `<factor>` $\\rightarrow$ `<integer>` $\\rightarrow$ `<digit>` $\\rightarrow$ 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "assert f.symbol_cost(\"<expr>\") == 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here's now a variant of `expand_node()` that takes the above cost into account.  It determines the minimum cost `cost` across all children and then chooses a child from the list using the `choose` function, which by default is the minimum cost.  If multiple children all have the same minimum cost, it chooses randomly between these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def expand_node_by_cost(self, node, choose=min):\n",
    "        (symbol, children) = node\n",
    "        assert children is None\n",
    "\n",
    "        # Fetch the possible expansions from grammar...\n",
    "        expansions = self.grammar[symbol]\n",
    "\n",
    "        possible_children_with_cost = [(self.expansion_to_children(expansion),\n",
    "                                        self.expansion_cost(\n",
    "                                            expansion, {symbol}),\n",
    "                                        expansion)\n",
    "                                       for expansion in expansions]\n",
    "\n",
    "        costs = [cost for (child, cost, expansion)\n",
    "                 in possible_children_with_cost]\n",
    "        chosen_cost = choose(costs)\n",
    "        children_with_chosen_cost = [child for (child, child_cost, _) in possible_children_with_cost\n",
    "                                     if child_cost == chosen_cost]\n",
    "        expansion_with_chosen_cost = [expansion for (_, child_cost, expansion) in possible_children_with_cost\n",
    "                                      if child_cost == chosen_cost]\n",
    "\n",
    "        index = self.choose_node_expansion(node, children_with_chosen_cost)\n",
    "\n",
    "        chosen_children = children_with_chosen_cost[index]\n",
    "        chosen_expansion = expansion_with_chosen_cost[index]\n",
    "        chosen_children = self.process_chosen_children(\n",
    "            chosen_children, chosen_expansion)\n",
    "\n",
    "        # Return with a new list\n",
    "        return (symbol, chosen_children)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The shortcut `expand_node_min_cost()` passes `min()` as the `choose` function, which makes it expand nodes at minimum cost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def expand_node_min_cost(self, node):\n",
    "        if self.log:\n",
    "            print(\"Expanding\", all_terminals(node), \"at minimum cost\")\n",
    "\n",
    "        return self.expand_node_by_cost(node, min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can now apply this function to close the expansion of our derivation tree, using `expand_tree_once()` with the above `expand_node_min_cost()` as expansion function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def expand_node(self, node):\n",
    "        return self.expand_node_min_cost(node)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "f = GrammarFuzzer(EXPR_GRAMMAR, log=True)\n",
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "if f.any_possible_expansions(derivation_tree):\n",
    "    derivation_tree = f.expand_tree_once(derivation_tree)\n",
    "    display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "if f.any_possible_expansions(derivation_tree):\n",
    "    derivation_tree = f.expand_tree_once(derivation_tree)\n",
    "    display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "if f.any_possible_expansions(derivation_tree):\n",
    "    derivation_tree = f.expand_tree_once(derivation_tree)\n",
    "    display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We keep on expanding until all nonterminals are expanded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "while f.any_possible_expansions(derivation_tree):\n",
    "    derivation_tree = f.expand_tree_once(derivation_tree)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here is the final tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We see that in each step, `expand_node_min_cost()` chooses an expansion that does not increase the number of symbols, eventually closing all open expansions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Node Inflation\n",
    "\n",
    "Especially at the beginning of an expansion, we may be interested in getting _as many nodes as possible_ – that is, we'd like to prefer expansions that give us _more_ nonterminals to expand.  This is actually the exact opposite of what `expand_node_min_cost()` gives us, and we can implement a method `expand_node_max_cost()` that will always choose among the nodes with the _highest_ cost:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def expand_node_max_cost(self, node):\n",
    "        if self.log:\n",
    "            print(\"Expanding\", all_terminals(node), \"at maximum cost\")\n",
    "\n",
    "        return self.expand_node_by_cost(node, max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To illustrate `expand_node_max_cost()`, we can again redefine `expand_node()` to use it, and then use `expand_tree_once()` to show a few expansion steps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def expand_node(self, node):\n",
    "        return self.expand_node_max_cost(node)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "derivation_tree = (\"<start>\",\n",
    "                   [(\"<expr>\",\n",
    "                     [(\"<expr>\", None),\n",
    "                      (\" + \", []),\n",
    "                         (\"<term>\", None)]\n",
    "                     )])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f = GrammarFuzzer(EXPR_GRAMMAR, log=True)\n",
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "if f.any_possible_expansions(derivation_tree):\n",
    "    derivation_tree = f.expand_tree_once(derivation_tree)\n",
    "    display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "if f.any_possible_expansions(derivation_tree):\n",
    "    derivation_tree = f.expand_tree_once(derivation_tree)\n",
    "    display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "if f.any_possible_expansions(derivation_tree):\n",
    "    derivation_tree = f.expand_tree_once(derivation_tree)\n",
    "    display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We see that with each step, the number of nonterminals increases.  Obviously, we have to put a limit on this number."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Three Expansion Phases\n",
    "\n",
    "We can now put all three phases together in a single function `expand_tree()` which will work as follows:\n",
    "\n",
    "1. **Max cost expansion.** Expand the tree using expansions with maximum cost until we have at least `min_nonterminals` nonterminals.  This phase can be easily skipped by setting `min_nonterminals` to zero.\n",
    "2. **Random expansion.**  Keep on expanding the tree randomly until we reach `max_nonterminals` nonterminals.\n",
    "3. **Min cost expansion.** Close the expansion with minimum cost.\n",
    "\n",
    "We implement these three phases by having `expand_node` reference the expansion method to apply.  This is controlled by setting `expand_node` (the method reference) to first `expand_node_max_cost` (i.e., calling `expand_node()` invokes `expand_node_max_cost()`), then `expand_node_randomly`, and finally `expand_node_min_cost`.  In the first two phases, we also set a maximum limit of `min_nonterminals` and `max_nonterminals`, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def log_tree(self, tree):\n",
    "        \"\"\"Output a tree if self.log is set; if self.display is also set, show the tree structure\"\"\"\n",
    "        if self.log:\n",
    "            print(\"Tree:\", all_terminals(tree))\n",
    "            if self.disp:\n",
    "                display_tree(tree)\n",
    "            # print(self.possible_expansions(tree), \"possible expansion(s) left\")\n",
    "\n",
    "    def expand_tree_with_strategy(self, tree, expand_node_method, limit=None):\n",
    "        \"\"\"Expand tree using `expand_node_method` as node expansion function\n",
    "        until the number of possible expansions reaches `limit`.\"\"\"\n",
    "        self.expand_node = expand_node_method\n",
    "        while ((limit is None\n",
    "                or self.possible_expansions(tree) < limit)\n",
    "               and self.any_possible_expansions(tree)):\n",
    "            tree = self.expand_tree_once(tree)\n",
    "            self.log_tree(tree)\n",
    "        return tree\n",
    "\n",
    "    def expand_tree(self, tree):\n",
    "        \"\"\"Expand `tree` in a three-phase strategy until all expansions are complete.\"\"\"\n",
    "        self.log_tree(tree)\n",
    "        tree = self.expand_tree_with_strategy(\n",
    "            tree, self.expand_node_max_cost, self.min_nonterminals)\n",
    "        tree = self.expand_tree_with_strategy(\n",
    "            tree, self.expand_node_randomly, self.max_nonterminals)\n",
    "        tree = self.expand_tree_with_strategy(\n",
    "            tree, self.expand_node_min_cost)\n",
    "\n",
    "        assert self.possible_expansions(tree) == 0\n",
    "\n",
    "        return tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let us try this out on our example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "derivation_tree = (\"<start>\",\n",
    "                   [(\"<expr>\",\n",
    "                     [(\"<expr>\", None),\n",
    "                      (\" + \", []),\n",
    "                         (\"<term>\", None)]\n",
    "                     )])\n",
    "\n",
    "f = GrammarFuzzer(\n",
    "    EXPR_GRAMMAR,\n",
    "    min_nonterminals=3,\n",
    "    max_nonterminals=5,\n",
    "    log=True)\n",
    "derivation_tree = f.expand_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "display_tree(derivation_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "all_terminals(derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Putting it all Together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Based on this, we can now define a function `fuzz()` that – like `simple_grammar_fuzzer()` – simply takes a grammar and produces a string from it.  It thus no longer exposes the complexity of derivation trees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class GrammarFuzzer(GrammarFuzzer):\n",
    "    def fuzz_tree(self):\n",
    "        # Create an initial derivation tree\n",
    "        tree = self.init_tree()\n",
    "        # print(tree)\n",
    "\n",
    "        # Expand all nonterminals\n",
    "        tree = self.expand_tree(tree)\n",
    "        if self.log:\n",
    "            print(repr(all_terminals(tree)))\n",
    "        if self.disp:\n",
    "            display_tree(tree)\n",
    "        return tree\n",
    "\n",
    "    def fuzz(self):\n",
    "        self.derivation_tree = self.fuzz_tree()\n",
    "        return all_terminals(self.derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We can now apply this on all our defined grammars (and visualize the derivation tree along)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f = GrammarFuzzer(EXPR_GRAMMAR)\n",
    "f.fuzz()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "After calling `fuzz()`, the produced derivation tree is accessible in the `derivation_tree` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "display_tree(f.derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Let us try out the grammar fuzzer (and its trees) on other grammar formats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f = GrammarFuzzer(URL_GRAMMAR)\n",
    "f.fuzz()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "display_tree(f.derivation_tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "f = GrammarFuzzer(CGI_GRAMMAR, min_nonterminals=3, max_nonterminals=5)\n",
    "f.fuzz()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "display_tree(f.derivation_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "How do we stack up against `simple_grammar_fuzzer()`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "trials = 50\n",
    "xs = []\n",
    "ys = []\n",
    "f = GrammarFuzzer(EXPR_GRAMMAR, max_nonterminals=20)\n",
    "for i in range(trials):\n",
    "    with Timer() as t:\n",
    "        s = f.fuzz()\n",
    "    xs.append(len(s))\n",
    "    ys.append(t.elapsed_time())\n",
    "    print(i, end=\" \")\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "average_time = sum(ys) / trials\n",
    "print(\"Average time:\", average_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "plt.scatter(xs, ys)\n",
    "plt.title('Time required for generating an output')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Our test generation is much faster, but also our inputs are much smaller.  We see that with derivation trees, we can get much better control over grammar production."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Finally, how does `GrammarFuzzer` work with `expr_grammar`, where `simple_grammar_fuzzer()` failed?  It works without any issue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "f = GrammarFuzzer(expr_grammar, max_nonterminals=10)\n",
    "f.fuzz()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "With `GrammarFuzzer`, we now have a solid foundation on which to build further fuzzers and illustrate more exciting concepts from the world of generating software tests.  Many of these do not even require writing a grammar – instead, they _infer_ a grammar from the domain at hand, and thus allow to use grammar-based fuzzing even without writing a grammar.  Stay tuned!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": true,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Lessons Learned\n",
    "\n",
    "* _Derivation trees_ are important for expressing input structure\n",
    "* _Grammar fuzzing based on derivation trees_ \n",
    "    1. is much more efficient than string-based grammar fuzzing,\n",
    "    2. gives much better control over input generation, and\n",
    "    3. effectively avoids running into infinite expansions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Next Steps\n",
    "\n",
    "Congratulations!  You have reached one of the central \"hubs\" of the book.  From here, there is a wide range of techniques that build on grammar fuzzing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Extending Grammars\n",
    "\n",
    "First, we have a number of techniques that all _extend_ grammars in some form:\n",
    "\n",
    "* [Parsing and recombining inputs](Parser.ipynb) allows to make use of existing inputs, again using derivation trees\n",
    "* [Covering grammar expansions](GrammarCoverageFuzzer.ipynb) allows for _combinatorial_ coverage\n",
    "* [Assigning _probabilities_ to individual expansions](ProbabilisticGrammarFuzzer.ipynb) gives additional control over expansions\n",
    "* [Assigning _constraints_ to individual expansions](GeneratorGrammarFuzzer.ipynb) allows to express _semantic constraints_ on individual rules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Applying Grammars\n",
    "\n",
    "Second, we can _apply_ grammars in a variety of contexts that all involve some form of learning it automatically:\n",
    "\n",
    "* [Fuzzing APIs](APIFuzzer.ipynb), learning a grammar from APIs\n",
    "* [Fuzzing graphical user interfaces](WebFuzzer.ipynb), learning a grammar from user interfaces for subsequent fuzzing\n",
    "* [Mining grammars](GrammarMiner.ipynb), learning a grammar for arbitrary input formats\n",
    "\n",
    "Keep on expanding!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background\n",
    "\n",
    "Derivation trees (then frequently called _parse trees_) are a standard data structure into which *parsers* decompose inputs.  The *Dragon Book* (also known as *Compilers: Principles, Techniques, and Tools*) \\cite{Aho2006} discusses parsing into derivation trees as part of compiling programs.  We also use derivation trees [when parsing and recombining inputs](Parser.ipynb).\n",
    "\n",
    "The key idea in this chapter, namely expanding until a limit of symbols is reached, and then always choosing the shortest path, stems from Luke \\cite{Luke2000}."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise 1: Caching Method Results\n",
    "\n",
    "Tracking `GrammarFuzzer` reveals that some methods are called again and again, always with the same values.  \n",
    "\n",
    "Set up a class `FasterGrammarFuzzer` with a _cache_ that checks whether the method has been called before, and if so, return the previously computed \"memoized\" value.  Do this for `expansion_to_children()`.  Compare the number of invocations before and after the optimization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "fragment"
    },
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "**Important**: For `expansion_to_children()`, make sure that each list returned is an individual copy.  If you return the same (cached) list, this will interfere with the in-place modification of `GrammarFuzzer`.  Use the Python `copy.deepcopy()` function for this purpose."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "**Solution.** Let us demonstrate this for `expansion_to_children()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "import copy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "class FasterGrammarFuzzer(GrammarFuzzer):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self._expansion_cache = {}\n",
    "        self._expansion_invocations = 0\n",
    "        self._expansion_invocations_cached = 0\n",
    "\n",
    "    def expansion_to_children(self, expansion):\n",
    "        self._expansion_invocations += 1\n",
    "        if expansion in self._expansion_cache:\n",
    "            self._expansion_invocations_cached += 1\n",
    "            cached_result = copy.deepcopy(self._expansion_cache[expansion])\n",
    "            return cached_result\n",
    "\n",
    "        result = super().expansion_to_children(expansion)\n",
    "        self._expansion_cache[expansion] = result\n",
    "        return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "f = FasterGrammarFuzzer(EXPR_GRAMMAR, min_nonterminals=3, max_nonterminals=5)\n",
    "f.fuzz()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "f._expansion_invocations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "f._expansion_invocations_cached"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "print(\"%.2f%% of invocations can be cached\" %\n",
    "      (f._expansion_invocations_cached * 100 / f._expansion_invocations))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "### Exercise 2: Grammar Pre-Compilation\n",
    "\n",
    "Some methods such as `symbol_cost()` or `expansion_cost()` return a value that is dependent on the grammar only.  Set up a class `EvenFasterGrammarFuzzer()` that pre-computes these values once upon initialization, such that later invocations of `symbol_cost()` or `expansion_cost()` need only look up these values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "**Solution.** Here's a possible solution, using a hack to substitute the `symbol_cost()` and `expansion_cost()` functions once the pre-computed values are set up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "class EvenFasterGrammarFuzzer(GrammarFuzzer):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self._symbol_costs = {}\n",
    "        self._expansion_costs = {}\n",
    "        self.precompute_costs()\n",
    "\n",
    "    def new_symbol_cost(self, symbol, seen=set()):\n",
    "        return self._symbol_costs[symbol]\n",
    "\n",
    "    def new_expansion_cost(self, expansion, seen=set()):\n",
    "        return self._expansion_costs[expansion]\n",
    "\n",
    "    def precompute_costs(self):\n",
    "        for symbol in self.grammar:\n",
    "            self._symbol_costs[symbol] = super().symbol_cost(symbol)\n",
    "            for expansion in self.grammar[symbol]:\n",
    "                self._expansion_costs[expansion] = super(\n",
    "                ).expansion_cost(expansion)\n",
    "\n",
    "        # Make sure we now call the caching methods\n",
    "        self.symbol_cost = self.new_symbol_cost\n",
    "        self.expansion_cost = self.new_expansion_cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "f = EvenFasterGrammarFuzzer(EXPR_GRAMMAR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "source": [
    "Here are the individual costs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "f._symbol_costs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "f._expansion_costs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "solution2": "hidden"
   },
   "outputs": [],
   "source": [
    "f = EvenFasterGrammarFuzzer(EXPR_GRAMMAR)\n",
    "f.fuzz()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "solution": "hidden",
    "solution2": "hidden",
    "solution2_first": true,
    "solution_first": true
   },
   "source": [
    "### Exercise 3: Maintaining Trees to be Expanded\n",
    "\n",
    "In `expand_tree_once()`, the algorithm traverses the tree again and again to find nonterminals that still can be extended.  Speed up the process by keeping a list of nonterminal symbols in the tree that still can be expanded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution": "hidden",
    "solution2": "hidden"
   },
   "source": [
    "**Solution.** Left as exercise for the reader."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise 4: Alternate Random Expansions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We could define `expand_node_randomly()` such that it simply invokes `expand_node_by_cost(node, random.choice)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class ExerciseGrammarFuzzer(GrammarFuzzer):\n",
    "    def expand_node_randomly(self, node):\n",
    "        if self.log:\n",
    "            print(\"Expanding\", all_terminals(node), \"randomly by cost\")\n",
    "\n",
    "        return self.expand_node_by_cost(node, random.choice)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "solution2": "hidden",
    "solution2_first": true
   },
   "source": [
    "What is the difference between the original implementation and this alternative?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "solution": "hidden",
    "solution2": "hidden"
   },
   "source": [
    "**Solution.** The alternative in `ExerciseGrammarFuzzer` has another probability distribution.  In the original `GrammarFuzzer`, all expansions have the same likelihood of being expanded.  In `ExerciseGrammarFuzzer`, first, a cost is chosen (randomly); then, one of the expansions with this cost is chosen (again randomly).  This means that expansions whose cost is unique have a higher chance of being selected."
   ]
  }
 ],
 "metadata": {
  "ipub": {
   "bibliography": "fuzzingbook.bib",
   "toc": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}