432_PCI_Ch06.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "432-PCI-Ch06.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyORuunFNMXFH8fBLXsCrog1",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/cs432-websci-master/public/blob/main/432_PCI_Ch06.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bfXfovkrW2G6"
      },
      "source": [
        "# Document Filtering\n",
        "\n",
        "Ch 6 from *Programming Collective Intelligence*, based on code from\n",
        "* https://github.com/arthur-e/Programming-Collective-Intelligence/tree/master/chapter6\n",
        "* https://go.oreilly.com/old-dominion-university/library/view/programming-collective-intelligence/9780596529321/\n",
        "\n",
        "**Goal:** Classify email as spam or not spam.\n",
        "\n",
        "**Implemented Example:** Classify a given document as \"bad\" or \"good\"."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XChxdkXms_XE"
      },
      "source": [
        "## General Functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OHgwbcheq5QS"
      },
      "source": [
        "import sqlite3 as sqlite   # replaces import stmt from book\n",
        "import re\n",
        "import math"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uoeWBsbiYMrP"
      },
      "source": [
        "`getwords(doc)` - returns a list of unique words found in the given document\n",
        "\n",
        "* breaks up the text into words, by dividing on any character that isn’t a letter\n",
        "* leaves only actual words, converted to lowercase\n",
        "* returns only unique words (so doesn't calculate the count if a word is used multiple times in a document)\n",
        "\n",
        "Note that this reduces the number of features because text is now case insensitive. However, this will completely miss ALL CAPS as potential feature for spam.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BQw0CH-MrJq7"
      },
      "source": [
        "def getwords(doc):\n",
        "  splitter=re.compile('\\W+')  # different than book\n",
        "  #print (doc)\n",
        "  # Split the words by non-alpha characters\n",
        "  words=[s.lower() for s in splitter.split(doc) \n",
        "          if len(s)>2 and len(s)<20]\n",
        "  \n",
        "  # Return the unique set of words only\n",
        "  uniq_words = dict([(w,1) for w in words])\n",
        "\n",
        "  return uniq_words"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lC4TV2qZsyjZ"
      },
      "source": [
        "## Basic Classifier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "POqzY5LXeL-J"
      },
      "source": [
        "`class basic_classifer` - holds what the classifier has learned so far\n",
        "* implemented in pgs. 119-127, no SQL DB involved (this is in the `class classifier` below)\n",
        "\n",
        "Instance variables:\n",
        "* `fc` - stores counts for different features in the different classifications \\\\\n",
        "example: `{'python': {'bad': 0, 'good': 6}, 'the': {'bad': 3, 'good': 3}}`\n",
        "* `cc` - dictionary of how many times every classification has been used, will be used in later probability calculations\n",
        "* `getfeatures()` - extracts the features from the items being classified, we use `getwords()`\n",
        "\n",
        "Helper functions - increment and access the counts (so that we can later store the training data in a file or db)\n",
        "* `incf()` - increase the count of a feature/category pair\n",
        "* `incc()` - increase the count of a category\n",
        "* `fcount()` - num times a feature has appeared in a category\n",
        "* `catcount()` - number of items in a category\n",
        "* `totalcount()` - total number of items\n",
        "* `categories()` - list of all categories\n",
        "\n",
        "Other functions:\n",
        "* `train()` - processes the training data, extracts words, and updates counts\n",
        "* `fprob()` - returns Pr(w|c), probability that a word appears in a category, implements the Multiple Bernoulli method\n",
        "* `weightedprob()` - returns the weighted probability of Pr(w|c), using assumed probabilities"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aHgTtW2pKGGo"
      },
      "source": [
        "class basic_classifier:\n",
        "\n",
        "  def __init__(self,getfeatures,filename=None):\n",
        "    # Counts of feature/category combinations\n",
        "    self.fc={}\n",
        "    # Counts of documents in each category\n",
        "    self.cc={}\n",
        "    self.getfeatures=getfeatures\n",
        "    \n",
        "  # Increase the count of a feature/category pair  \n",
        "  def incf(self,f,cat):\n",
        "    self.fc.setdefault(f, {})\n",
        "    self.fc[f].setdefault(cat, 0)\n",
        "    self.fc[f][cat]+=1\n",
        "  \n",
        "  # Increase the count of a category  \n",
        "  def incc(self,cat):\n",
        "    self.cc.setdefault(cat, 0)\n",
        "    self.cc[cat]+=1  \n",
        "\n",
        "  # The number of times a feature has appeared in a category\n",
        "  def fcount(self,f,cat):\n",
        "    if f in self.fc and cat in self.fc[f]:\n",
        "      return float(self.fc[f][cat])\n",
        "    return 0.0\n",
        "\n",
        "  # The number of items in a category\n",
        "  def catcount(self,cat):\n",
        "    if cat in self.cc:\n",
        "        return float(self.cc[cat])\n",
        "    return 0\n",
        "\n",
        "  # The total number of items\n",
        "  def totalcount(self):\n",
        "    return sum(self.cc.values())\n",
        "\n",
        "  # The list of all categories\n",
        "  def categories(self):\n",
        "    return self.cc.keys()\n",
        "\n",
        "  def train(self,item,cat):\n",
        "    features=self.getfeatures(item)\n",
        "    # Increment the count for every feature with this category\n",
        "    for f in features:\n",
        "      self.incf(f,cat)\n",
        "\n",
        "    # Increment the count for this category\n",
        "    self.incc(cat)\n",
        "\n",
        "  def fprob(self,f,cat):\n",
        "    if self.catcount(cat)==0: return 0\n",
        "\n",
        "    # The total number of times this feature appeared in this \n",
        "    # category divided by the total number of items in this category\n",
        "    return self.fcount(f,cat)/self.catcount(cat)\n",
        "\n",
        "  def weightedprob(self,f,cat,prf,weight=1.0,ap=0.5):\n",
        "    # Calculate current probability\n",
        "    basicprob=prf(f,cat)\n",
        "\n",
        "    # Count the number of times this feature has appeared in\n",
        "    # all categories\n",
        "    totals=sum([self.fcount(f,c) for c in self.categories()])\n",
        "\n",
        "    # Calculate the weighted average\n",
        "    bp=((weight*ap)+(totals*basicprob))/(weight+totals)\n",
        "    return bp"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XvfzQX21si-m"
      },
      "source": [
        "## Training Examples"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o7u6mDIWrWAf"
      },
      "source": [
        "def sampletrain(cl):\n",
        "  cl.train('Nobody owns the water.','good')\n",
        "  cl.train('the quick rabbit jumps fences','good')\n",
        "  cl.train('buy pharmaceuticals now','bad')\n",
        "  cl.train('make quick money at the online casino','bad')\n",
        "  cl.train('the quick brown fox jumps','good')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TE8NtDFDiniL"
      },
      "source": [
        "### Example 1 - simple counts\n",
        "\n",
        "First, instantiate the basic classifier with `getwords()` as the getfeatures function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ik--QY1yim7J"
      },
      "source": [
        "cl = basic_classifier(getwords)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Svir57jSpCWy"
      },
      "source": [
        "Load sample training data and print out data from the classifier"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b_2W5eedpMM8"
      },
      "source": [
        "sampletrain(cl)\n",
        "print(\"\")\n",
        "print(\"Total items:\", cl.totalcount())\n",
        "print(\"Categories:\", cl.categories())\n",
        "for cat in cl.categories():\n",
        "  print(cat, cl.catcount(cat))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1on-vinipvYE"
      },
      "source": [
        "cl.fcount('quick', 'good')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cJ3TrJvZpxJT"
      },
      "source": [
        "cl.fcount('quick', 'bad')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HICNYWVZqHdH"
      },
      "source": [
        "### Example 2 (pg. 122) - simple prob\n",
        "\n",
        "First, reset the classifier by re-instantiating"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "axzBKM2KqyQL"
      },
      "source": [
        "cl = basic_classifier(getwords)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f9CoG2rnq_5F"
      },
      "source": [
        "sampletrain(cl)\n",
        "cl.fprob('quick', 'good')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N-mLjCHcq4BJ"
      },
      "source": [
        "### Example 3 (pg. 122) - simple weightedprob"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B38WcYRvh8vH"
      },
      "source": [
        "cl = basic_classifier(getwords)\n",
        "cl.weightedprob('money', 'bad', cl.fprob)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m3IWpcWQmc1d"
      },
      "source": [
        "cl.train(\"This money is bad.\", \"bad\")\n",
        "cl.weightedprob('money', 'bad', cl.fprob)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pl8fwuMctIE8"
      },
      "source": [
        "### Example 4 - fprob vs. weightedprob"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "obTlbgrXuMFH"
      },
      "source": [
        "cl = basic_classifier(getwords)\n",
        "sampletrain(cl)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QgyFffZ7uVP0"
      },
      "source": [
        "cl.fprob('money', 'good')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wj2dEwy2uRBA"
      },
      "source": [
        "cl.weightedprob('money', 'good', cl.fprob)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qN22-1FPsAfn"
      },
      "source": [
        "### Example 5 (pg. 123) - adding more training data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C6pAqaDqi09a"
      },
      "source": [
        "cl = basic_classifier(getwords)\n",
        "sampletrain(cl)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gc3ujJ1Eu3On"
      },
      "source": [
        "cl.weightedprob('money', 'good', cl.fprob)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u54omZlBpNRE"
      },
      "source": [
        "sampletrain(cl)\n",
        "cl.weightedprob('money', 'good', cl.fprob)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SBB62YFRsfgZ"
      },
      "source": [
        "## Naive Bayes Classifier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K_SEu0QhwDlS"
      },
      "source": [
        "*To use this with the basic classifier (and to change it back later), make the following changes:*\n",
        "* `class naivebayes(classifier)` -> `class naivebayes(basic_classifier)`\n",
        "* `classifier.__init__(self,getfeatures)` -> `basic_classifier.__init__(self,getfeatures)`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0hSyFIKnrRly"
      },
      "source": [
        "class naivebayes(classifier):   # change for basic_classifier\n",
        "\n",
        "  def __init__(self,getfeatures):   \n",
        "    classifier.__init__(self,getfeatures)  # change for basic_classifier\n",
        "    self.thresholds={}\n",
        "  \n",
        "  def docprob(self,item,cat):\n",
        "    features=self.getfeatures(item)   \n",
        "\n",
        "    # Multiply the probabilities of all the features together\n",
        "    p=1\n",
        "    for f in features: p*=self.weightedprob(f,cat,self.fprob)\n",
        "    return p\n",
        "\n",
        "  def prob(self,item,cat):\n",
        "    catprob=self.catcount(cat)/self.totalcount()\n",
        "    docprob=self.docprob(item,cat)\n",
        "    return docprob*catprob\n",
        "  \n",
        "  def setthreshold(self,cat,t):\n",
        "    self.thresholds[cat]=t\n",
        "    \n",
        "  def getthreshold(self,cat):\n",
        "    if cat not in self.thresholds: return 1.0\n",
        "    return self.thresholds[cat]\n",
        "  \n",
        "  def classify(self,item,default=None):\n",
        "    probs={}\n",
        "    # Find the category with the highest probability\n",
        "    max=0.0\n",
        "    for cat in self.categories():\n",
        "      probs[cat]=self.prob(item,cat)\n",
        "      if probs[cat]>max: \n",
        "        max=probs[cat]\n",
        "        best=cat\n",
        "\n",
        "    # Make sure the probability exceeds threshold*next best\n",
        "    for cat in probs:\n",
        "      if cat==best: continue\n",
        "      if probs[cat]*self.getthreshold(best)>probs[best]: return default\n",
        "    return best"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xBXr3UZPw7k-"
      },
      "source": [
        "## Bayesian Examples"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aO0hRvutWP00"
      },
      "source": [
        "### Example 1 (pg. 125) - prob\n",
        "\n",
        "Training dataset: \n",
        "```\n",
        "('Nobody owns the water.','good')\n",
        "('the quick rabbit jumps fences','good')\n",
        "('buy pharmaceuticals now','bad')\n",
        "('make quick money at the online casino','bad')\n",
        "('the quick brown fox jumps','good')\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SAQf5uG0xK6_"
      },
      "source": [
        "cl = naivebayes(getwords)\n",
        "sampletrain(cl)\n",
        "cl.prob('quick rabbit', 'good')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ltYOiPfQx4tz"
      },
      "source": [
        "cl.prob('quick rabbit', 'bad')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pk-7_qV2Wavk"
      },
      "source": [
        "### Example 2 (pg. 127) - using thresholds"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5h-HfU9KZ85l"
      },
      "source": [
        "cl = naivebayes(getwords)\n",
        "sampletrain(cl)\n",
        "cl.classify('quick rabbit', default='unknown')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hvziD6X5aHE7"
      },
      "source": [
        "cl.classify('quick money', default='unknown')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fZltAQUPaKfZ"
      },
      "source": [
        "cl.setthreshold('bad', 3.0)\n",
        "cl.classify('quick money', default='unknown')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MINmeQZpaQjb"
      },
      "source": [
        "for i in range(10): sampletrain(cl)\n",
        "cl.classify('quick money', default='unknown')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Crf8WHNqoX2z"
      },
      "source": [
        "## Classifier w/SQL\n",
        "\n",
        "Uses a SQL database"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "phzg7xZyrLn6"
      },
      "source": [
        "class classifier:\n",
        "  def __init__(self,getfeatures,filename=None):\n",
        "    # Counts of feature/category combinations\n",
        "    self.fc={}\n",
        "    # Counts of documents in each category\n",
        "    self.cc={}\n",
        "    self.getfeatures=getfeatures\n",
        "    \n",
        "  def setdb(self,dbfile):\n",
        "    self.con=sqlite.connect(dbfile)    \n",
        "    self.con.execute('create table if not exists fc(feature,category,count)')\n",
        "    self.con.execute('create table if not exists cc(category,count)')\n",
        "\n",
        "  def incf(self,f,cat):\n",
        "    count=self.fcount(f,cat)\n",
        "    if count==0:\n",
        "      self.con.execute(\"insert into fc values ('%s','%s',1)\" \n",
        "                       % (f,cat))\n",
        "    else:\n",
        "      self.con.execute(\n",
        "        \"update fc set count=%d where feature='%s' and category='%s'\" \n",
        "        % (count+1,f,cat)) \n",
        "  \n",
        "  def fcount(self,f,cat):\n",
        "    res=self.con.execute(\n",
        "      'select count from fc where feature=\"%s\" and category=\"%s\"'\n",
        "      %(f,cat)).fetchone()\n",
        "    if res==None: return 0\n",
        "    else: return float(res[0])\n",
        "\n",
        "  def incc(self,cat):\n",
        "    count=self.catcount(cat)\n",
        "    if count==0:\n",
        "      self.con.execute(\"insert into cc values ('%s',1)\" % (cat))\n",
        "    else:\n",
        "      self.con.execute(\"update cc set count=%d where category='%s'\" \n",
        "                       % (count+1,cat))    \n",
        "\n",
        "  def catcount(self,cat):\n",
        "    res=self.con.execute('select count from cc where category=\"%s\"'\n",
        "                         %(cat)).fetchone()\n",
        "    if res==None: return 0\n",
        "    else: return float(res[0])\n",
        "\n",
        "  def categories(self):\n",
        "    cur=self.con.execute('select category from cc');\n",
        "    return [d[0] for d in cur]\n",
        "\n",
        "  def totalcount(self):\n",
        "    res=self.con.execute('select sum(count) from cc').fetchone();\n",
        "    if res==None: return 0\n",
        "    return res[0]\n",
        "\n",
        "  def train(self,item,cat):\n",
        "    features=self.getfeatures(item)\n",
        "    # Increment the count for every feature with this category\n",
        "    for f in features:\n",
        "      self.incf(f,cat)\n",
        "\n",
        "    # Increment the count for this category\n",
        "    self.incc(cat)\n",
        "    self.con.commit()\n",
        "\n",
        "  def fprob(self,f,cat):\n",
        "    if self.catcount(cat)==0: return 0\n",
        "\n",
        "    # The total number of times this feature appeared in this \n",
        "    # category divided by the total number of items in this category\n",
        "    return self.fcount(f,cat)/self.catcount(cat)\n",
        "\n",
        "  def weightedprob(self,f,cat,prf,weight=1.0,ap=0.5):\n",
        "    # Calculate current probability\n",
        "    basicprob=prf(f,cat)\n",
        "\n",
        "    # Count the number of times this feature has appeared in\n",
        "    # all categories\n",
        "    totals=sum([self.fcount(f,c) for c in self.categories()])\n",
        "\n",
        "    # Calculate the weighted average\n",
        "    bp=((weight*ap)+(totals*basicprob))/(weight+totals)\n",
        "    return bp\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "csq1--iEsU8J"
      },
      "source": [
        "## Examples - Full Bayesian Classifier w/SQL\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "69xRiX9NsMWp"
      },
      "source": [
        "def spamTrain(cl):\n",
        "  cl.train('the the', 'not spam')\n",
        "  cl.train('cheap cheap cheap banking the', 'spam')\n",
        "  cl.train('the', 'not spam')\n",
        "  cl.train('cheap cheap banking banking banking the the', 'spam')\n",
        "  cl.train('cheap cheap cheap cheap cheap buy buy the', 'spam')\n",
        "  cl.train('banking the', 'not spam')\n",
        "  cl.train('buy banking the', 'not spam')\n",
        "  cl.train('the', 'not spam')\n",
        "  cl.train('the', 'not spam')\n",
        "  cl.train('cheap buy dinner the the', 'not spam')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eQlVRcDDT9zY"
      },
      "source": [
        "*Don't forget to adjust `class naivebayes` to use `classifier`*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZaZ1mvK6fjyL"
      },
      "source": [
        "cl = naivebayes(getwords)\n",
        "cl.setdb('test1.db')\n",
        "spamTrain(cl)\n",
        "cl.setthreshold('spam', 3.0)\n",
        "cl.classify('the banking dinner', default='unknown')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EmWGy3lGrIBp"
      },
      "source": [
        "cl2 = naivebayes(getwords)\n",
        "cl2.setdb('test2.db')\n",
        "sampletrain(cl2)\n",
        "cl2.setthreshold('bad', 3.0)\n",
        "cl2.classify('quick money', default='unknown')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p9AWghJOrTW3"
      },
      "source": [
        "cl = naivebayes(getwords)\n",
        "cl.setdb('test1.db')\n",
        "cl.classify('cheap money', default='unknown')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JZRRnVBgrWWm"
      },
      "source": [
        "cl2.classify('online casino now', default='unknown')"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}