notebooks/chatgpt_vs_open_source_on_harder_tasks.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring ChatGPT and open-source models on slightly harder tasks\n",
    "\n",
    "**NOTE! This notebook has not yet been updated to use the new `v0.1+` pythonic syntax. We are working through the notebooks still. (PRs are welcome)**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "**Note: This notebook is the code and examples from a related [the blog post](https://medium.com/@marcotcr/exploring-chatgpt-vs-open-source-models-on-slightly-harder-tasks-aa0395c31610) written jointly by Marco Tulio Ribeiro and Scott Lundberg.**  \n",
    "\n",
    "Open-source LLMs like [Vicuna](https://lmsys.org/blog/2023-03-30-vicuna/) and [MPT](https://www.mosaicml.com/blog/mpt-7b#building-with-mosaicml-platform) are popping up all over the place.  \n",
    "There's been [discussion](https://www.semianalysis.com/p/google-we-have-no-moat-and-neither) about how these models compare to commercial LLMs like ChatGPT or Bard, but most of the comparison has been around answers to simple questions.  \n",
    "\n",
    "As an example, the folks at [LMSYSOrg](https://lmsys.org/) did [an interesting analysis](https://lmsys.org/blog/2023-03-30-vicuna/) (+1 for being automated and reproducible) comparing Vicuna-13B to ChatGPT on various short questions, which is great as a comparison of the models as simple chatbots. However, many interesting ways of using these LLMs typically require complex instructions and/or multi-turn conversations, and some prompt engineering.\n",
    "In the 'real world', most people will want to compare different LLM offerings on _their_ problem, with a variety of different prompts.  \n",
    "\n",
    "This notebook is an example of what such an exploration might look like, comparing two open source models (Vicuna-13B, MPT-7b-Chat) with ChatGPT on tasks of varying complexity.\n",
    "\n",
    "The notebook is included here because it is also an example of how Guidance can be used to compare various models easily. We have shortened the discussion and commentary that are in the blog post, but the code and examples are here."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Warmup: Solving equations\n",
    "By way of warmup, let's start with a toy task (solving simple polynomial equations), where we can check the output for correctness and shouldn't need much prompt engineering.  \n",
    "This will be similar to the Math category in [here](https://lmsys.org/blog/2023-03-30-vicuna/), with the difference that we will evaluate models as correct / incorrect on ground truth, rather than using GPT-4 to rate the output."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a simple function that generates a polynomial with distinct integer roots.\n",
    "This will give us both the input and the ground truth output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Roots [2, -7]\n",
      "x^2 + 5.0x - 14.0 = 0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "def random_polynomial(n_roots, low=1, high=100):\n",
    "    roots = [np.random.randint(low, high) for _ in range(n_roots)]\n",
    "    # unique roots only\n",
    "    while len(set(roots)) != n_roots:\n",
    "        roots = [np.random.randint(low, high) for _ in range(n_roots)]\n",
    "    poly = np.polynomial.polynomial.Polynomial.fromroots(roots)\n",
    "    a = poly.coef.copy()\n",
    "    a = a[::-1]\n",
    "    text= ''\n",
    "    for i, coef in enumerate(a):\n",
    "        if coef == 0:\n",
    "            continue\n",
    "        sign = ' + ' if coef > 0 else ' - '\n",
    "        if i ==0:\n",
    "            sign = ''\n",
    "        elif coef < 0:\n",
    "            coef = -coef\n",
    "        if i == len(a) - 1:\n",
    "            text += f'{sign}{coef}'\n",
    "        else:\n",
    "            if coef == 1:\n",
    "                coef = ''\n",
    "            power = f'^{len(a) - i - 1}' if len(a) - i - 1 > 1 else ''\n",
    "            text += f'{sign}{coef}x{power}'\n",
    "    text += ' = 0'\n",
    "    return roots, text\n",
    "roots, equation = random_polynomial(2, low=-10, high=10)\n",
    "print('Roots', roots)\n",
    "print(equation)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's load the models (we use [`guidance`](https://github.com/microsoft/guidance) throughout for easy comparison and control):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import guidance\n",
    "import transformers\n",
    "path = '/home/marcotcr/.cache/huggingface/hub/vicuna-13b'\n",
    "mpt = guidance.llms.transformers.MPTChat('mosaicml/mpt-7b-chat', device=1)\n",
    "vicuna = guidance.llms.transformers.Vicuna(path, device_map='auto')\n",
    "chatgpt = guidance.llms.OpenAI(\"gpt-3.5-turbo\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quick digression on different syntaxes**: each of these models have a their own _chat syntax_, e.g. here is how the same conversation would look like in Vicuna and MPT (where `[generated response]` is where the model would put its output):  \n",
    "\n",
    "Vicuna:  \n",
    "```\n",
    "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.  \n",
    "USER: Can you please solve the following equation? x^2 + 2x + 1 = 0  \n",
    "ASSISTANT: [generated response] </s>\n",
    "```\n",
    "\n",
    "MPT:  \n",
    "```\n",
    "<|im_start|>system\n",
    "- You are a helpful assistant chatbot trained by MosaicML.  \n",
    "- You answer questions.\n",
    "- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n",
    "- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.\"\"\"\n",
    "<|im_end|>\n",
    "<|im_start|>user Can you please solve the following equation? x^2 + 2x + 1 = 0<|im_end|>\n",
    "<|im_start|>assistant [generated response]<|im_end>\n",
    "```\n",
    "\n",
    "To avoid the tediousness translating between these, `guidance` supports a unified chat syntax that gets translated to the model-specific syntax when calling the model.  \n",
    "Here is the prompt we'll use for all models (note how we use ``{{system}}``, ``{{user}}`` and ``{{assistant}}`` tags rather than model-specific separators):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "find_roots = guidance('''\n",
    "{{~#system~}}\n",
    "{{llm.default_system_prompt}}\n",
    "{{~/system}}\n",
    "\n",
    "{{#user~}}\n",
    "Please find the roots of the following equation: {{equation}}\n",
    "Think step by step, find the roots, and then say:\n",
    "ROOTS = [root1, root2...]\n",
    "For example, if the roots are 1.3 and 2.2, say ROOTS = [1.3, 2.2].\n",
    "Make sure to use real numbers, not fractions.\n",
    "{{~/user}}\n",
    "\n",
    "{{#assistant~}}\n",
    "{{gen 'answer' temperature=0}}\n",
    "{{~/assistant~}}''')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try the prompt on a very simple example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, -2]\n",
      "x^2 + x - 2.0 = 0\n"
     ]
    }
   ],
   "source": [
    "roots, equation = random_polynomial(2, low=-3, high=3)\n",
    "print(roots)\n",
    "print(equation)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ChatGPT:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-5012bd52-75d7-44c7-9c0a-45645f8439e7\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-5012bd52-75d7-44c7-9c0a-45645f8439e7\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please find the roots of the following equation: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{equation}}'>x^2 + x - 2.0 = 0</span>\n",
       "Think step by step, find the roots, and then say:\n",
       "ROOTS = [root1, root2...]\n",
       "For example, if the roots are 1.3 and 2.2, say ROOTS = [1.3, 2.2].\n",
       "Make sure to use real numbers, not fractions.</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>To find the roots of the equation x^2 + x - 2.0 = 0, we can use the quadratic formula:\n",
       "\n",
       "x = (-b ± sqrt(b^2 - 4ac)) / 2a\n",
       "\n",
       "where a = 1, b = 1, and c = -2.0\n",
       "\n",
       "Plugging in these values, we get:\n",
       "\n",
       "x = (-1 ± sqrt(1^2 - 4(1)(-2.0))) / 2(1)\n",
       "\n",
       "Simplifying:\n",
       "\n",
       "x = (-1 ± sqrt(1 + 8)) / 2\n",
       "\n",
       "x = (-1 ± sqrt(9)) / 2\n",
       "\n",
       "x = (-1 ± 3) / 2\n",
       "\n",
       "So the roots are:\n",
       "\n",
       "ROOTS = [-2.0, 1.0]</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"5012bd52-75d7-44c7-9c0a-45645f8439e7\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "answer_gpt = find_roots(llm=chatgpt, equation=equation)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vicuna:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-680206b6-6a0e-4b7b-8449-28f964922f5d\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-680206b6-6a0e-4b7b-8449-28f964922f5d\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user&#x27;s questions.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please find the roots of the following equation: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{equation}}'>x^2 + x - 2.0 = 0</span>\n",
       "Think step by step, find the roots, and then say:\n",
       "ROOTS = [root1, root2...]\n",
       "For example, if the roots are 1.3 and 2.2, say ROOTS = [1.3, 2.2].\n",
       "Make sure to use real numbers, not fractions.</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>To find the roots of the equation x^2 + x - 2.0 = 0, we can use the quadratic formula:\n",
       "\n",
       "x = (-b +/- sqrt(b^2 - 4ac)) / 2a\n",
       "\n",
       "where a, b, and c are the coefficients of the equation. In this case, a = 1, b = -1, and c = 2.0. Plugging these values into the formula, we get:\n",
       "\n",
       "x = (-(-1) +/- sqrt((-1)^2 - 4(1)(2.0))) / 2(1)\n",
       "\n",
       "x = (1 +/- sqrt(1 - 8)) / 2\n",
       "\n",
       "x = (1 +/- sqrt(-7)) / 2\n",
       "\n",
       "The square root of -7 is not a real number, so the equation has no real roots. Therefore, the roots of the equation x^2 + x - 2.0 = 0 are not real numbers.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"680206b6-6a0e-4b7b-8449-28f964922f5d\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "answer_vicuna = find_roots(llm=vicuna, equation=equation)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MPT:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-ee4a0d7e-63a2-470b-be9b-15daa58306aa\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-ee4a0d7e-63a2-470b-be9b-15daa58306aa\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>- You are a helpful assistant chatbot trained by MosaicML.  \n",
       "- You answer questions.\n",
       "- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n",
       "- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please find the roots of the following equation: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{equation}}'>x^2 + x - 2.0 = 0</span>\n",
       "Think step by step, find the roots, and then say:\n",
       "ROOTS = [root1, root2...]\n",
       "For example, if the roots are 1.3 and 2.2, say ROOTS = [1.3, 2.2].\n",
       "Make sure to use real numbers, not fractions.</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>The roots of the equation x^2 + x - 2.0 = 0 are -1 and 1. \n",
       "ROOTS = [-1, 1]</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"ee4a0d7e-63a2-470b-be9b-15daa58306aa\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "answer_mpt = find_roots(llm=mpt, equation=equation)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ChatGPT got it right, while Vicuna and MPT got it wrong (Vicuna didn't even follow the specified format).\n",
    "Let's write a simple regex to parse the output, so we can evaluate this on a few more expressions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ChatGPT:  True\n",
      "Vicuna:  False\n",
      "MPT:  False\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "def parse_roots(text):\n",
    "    roots = re.search(r'ROOTS = \\[(.*)\\]', text)\n",
    "    if not roots:\n",
    "        return []\n",
    "    roots = roots.group(1).split(',')\n",
    "    try:\n",
    "        roots = [float(r) for r in roots]\n",
    "    except:\n",
    "        roots = []\n",
    "    return roots\n",
    "def matches_groundtruth(roots, groundtruth):\n",
    "    if len(roots) != len(groundtruth):\n",
    "        return False\n",
    "    gt = np.array(sorted(groundtruth))\n",
    "    roots = np.array(sorted(roots))\n",
    "    return np.all(np.abs(gt - roots) < 1e-3)\n",
    "\n",
    "print('ChatGPT: ', matches_groundtruth(parse_roots(answer_gpt['answer']), roots))\n",
    "print('Vicuna: ', matches_groundtruth(parse_roots(answer_vicuna['answer']), roots))\n",
    "print('MPT: ', matches_groundtruth(parse_roots(answer_mpt['answer']), roots))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We evaluate the prompt on quadratic equations with roots between -20 and 20:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frequency of correct answers:\n",
      "ChatGPT:  0.8\n",
      "Vicuna:  0.0\n",
      "MPT:  0.0\n"
     ]
    }
   ],
   "source": [
    "answers = {\n",
    "    'ChatGPT': [],\n",
    "    'Vicuna': [],\n",
    "    'MPT': []\n",
    "}\n",
    "correct = {\n",
    "    'ChatGPT': [],\n",
    "    'Vicuna': [],\n",
    "    'MPT': []\n",
    "}\n",
    "for _ in range(20):\n",
    "    roots, equation = random_polynomial(2, low=-20, high=20)\n",
    "    answer_gpt = find_roots(llm=chatgpt, equation=equation, silent=True)\n",
    "    answer_vicuna = find_roots(llm=vicuna, equation=equation, silent=True)\n",
    "    answer_mpt = find_roots(llm=mpt, equation=equation, silent=True)\n",
    "    answers['ChatGPT'].append(answer_gpt)\n",
    "    answers['Vicuna'].append(answer_vicuna)\n",
    "    answers['MPT'].append(answer_mpt)\n",
    "    correct['ChatGPT'].append(matches_groundtruth(parse_roots(answer_gpt['answer']), roots))\n",
    "    correct['Vicuna'].append(matches_groundtruth(parse_roots(answer_vicuna['answer']), roots))\n",
    "    correct['MPT'].append(matches_groundtruth(parse_roots(answer_mpt['answer']), roots))\n",
    "\n",
    "print('Frequency of correct answers:')\n",
    "print('ChatGPT: ', np.mean(correct['ChatGPT']))\n",
    "print('Vicuna: ', np.mean(correct['Vicuna']))\n",
    "print('MPT: ', np.mean(correct['MPT']))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ChatGPT gets the right roots 80% of the time, while Vicuna and MPT never get them right. Let's see a few errors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-a4f7e4eb-98bd-4418-80b3-a4aaf48d1c86\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-a4f7e4eb-98bd-4418-80b3-a4aaf48d1c86\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user&#x27;s questions.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please find the roots of the following equation: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{equation}}'></span><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{equation}}'>x^2 - 13.0x - 114.0 = 0</span>\n",
       "Think step by step, find the roots, and then say:\n",
       "ROOTS = [root1, root2...]\n",
       "For example, if the roots are 1.3 and 2.2, say ROOTS = [1.3, 2.2].\n",
       "Make sure to use real numbers, not fractions.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>To find the roots of the equation x^2 - 13.0x - 114.0 = 0, we can use the quadratic formula:\n",
       "\n",
       "x = (-b +/- sqrt(b^2 - 4ac)) / 2a\n",
       "\n",
       "where a, b, and c are the coefficients of the equation.\n",
       "\n",
       "In this case, a = 1, b = -13.0, and c = -114.0. Plugging these values into the formula, we get:\n",
       "\n",
       "x = (-(-13.0) +/- sqrt((-13.0)^2 - 4(1)(-114.0))) / 2(1)\n",
       "\n",
       "Simplifying, we get:\n",
       "\n",
       "x = (13.0 +/- sqrt(13.0^2 + 448.0)) / 2\n",
       "\n",
       "x = (13.0 +/- sqrt(13.0^2 + 448.0)) / 2\n",
       "\n",
       "x = (13.0 +/- sqrt(13.0^2 + 448.0)) / 2\n",
       "\n",
       "The solutions to this equation are the values of x that make the equation true. These values are the roots of the equation.\n",
       "\n",
       "The roots of the equation x^2 - 13.0x - 114.0 = 0 are:\n",
       "\n",
       "ROOTS = [13.0, -13.0]\n",
       "\n",
       "Therefore, the two roots of the equation are 13.0 and -13.0, both of which are real numbers.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"a4f7e4eb-98bd-4418-80b3-a4aaf48d1c86\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "answers['Vicuna'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-ede864e4-1394-45d6-93bf-703138babfd9\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-ede864e4-1394-45d6-93bf-703138babfd9\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>- You are a helpful assistant chatbot trained by MosaicML.\n",
       "    - You answer questions.\n",
       "    - You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n",
       "    - You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please find the roots of the following equation: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{equation}}'></span><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{equation}}'>x^2 - 13.0x - 114.0 = 0</span>\n",
       "Think step by step, find the roots, and then say:\n",
       "ROOTS = [root1, root2...]\n",
       "For example, if the roots are 1.3 and 2.2, say ROOTS = [1.3, 2.2].\n",
       "Make sure to use real numbers, not fractions.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>The roots of the equation x^2 - 13.0x - 114.0 = 0 are 2.0 and -3.0. Therefore, ROOTS = [-3.0, 2.0].</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"ede864e4-1394-45d6-93bf-703138babfd9\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "answers['MPT'][0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vicuna does the math wrong, while MPT does not even attempt to solve it step by step.  \n",
    "ChatGPT also makes some mistakes, often involving some math substep:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-1d99ae78-3851-44cf-822f-43408ee13e6a\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-1d99ae78-3851-44cf-822f-43408ee13e6a\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please find the roots of the following equation: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{equation}}'></span><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{equation}}'>x^2 - 13.0x - 114.0 = 0</span>\n",
       "Think step by step, find the roots, and then say:\n",
       "ROOTS = [root1, root2...]\n",
       "For example, if the roots are 1.3 and 2.2, say ROOTS = [1.3, 2.2].\n",
       "Make sure to use real numbers, not fractions.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>To find the roots of the equation x^2 - 13.0x - 114.0 = 0, we can use the quadratic formula:\n",
       "\n",
       "x = (-b ± sqrt(b^2 - 4ac)) / 2a\n",
       "\n",
       "where a = 1, b = -13.0, and c = -114.0.\n",
       "\n",
       "Plugging in these values, we get:\n",
       "\n",
       "x = (-(-13.0) ± sqrt((-13.0)^2 - 4(1)(-114.0))) / 2(1)\n",
       "x = (13.0 ± sqrt(169 + 456)) / 2\n",
       "x = (13.0 ± sqrt(625)) / 2\n",
       "x = (13.0 ± 25.0) / 2\n",
       "\n",
       "So the roots are:\n",
       "\n",
       "ROOTS = [19.5, -6.5]</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"1d99ae78-3851-44cf-822f-43408ee13e6a\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "error = np.where(np.array(correct['ChatGPT']) == False)[0][0]\n",
    "answers['ChatGPT'][error]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since Vicuna and MPT failed on quadratic equations, let's look at even simpler equations, such as `x - 10 = 0`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "find_solution = guidance('''\n",
    "{{#system~}}\n",
    "{{llm.default_system_prompt}}\n",
    "{{~/system}}\n",
    "\n",
    "{{#user~}}\n",
    "Please find the solution to the following equation: {{equation}}\n",
    "Think step by step, find the solution, and then say:\n",
    "SOLUTION = [value]\n",
    "For example, if the solution is x=3, say SOLUTION = [3].\n",
    "Make sure to use real numbers, not fractions.\n",
    "{{/user}}\n",
    "\n",
    "{{#assistant~}}\n",
    "{{gen 'answer' temperature=0 max_tokens=300}}\n",
    "{{~/assistant~}}''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_solution(text):\n",
    "    solution = re.search(r'SOLUTION = \\[(.*)\\]', text)\n",
    "    if not solution:\n",
    "        return []\n",
    "    try:\n",
    "        return [float(solution.group(1))]\n",
    "    except:\n",
    "        return []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frequency of correct answers:\n",
      "ChatGPT:  1.0\n",
      "Vicuna:  0.9285714285714286\n",
      "MPT:  0.23076923076923078\n"
     ]
    }
   ],
   "source": [
    "correct1d = {\n",
    "    'ChatGPT': [],\n",
    "    'Vicuna': [],\n",
    "    'MPT': []\n",
    "}\n",
    "answers1d = {\n",
    "    'ChatGPT': [],\n",
    "    'Vicuna': [],\n",
    "    'MPT': []\n",
    "}\n",
    "for i in range(20):\n",
    "    print(i)\n",
    "    roots, equation = random_polynomial(1, low=-20, high=20)\n",
    "    answer_gpt = find_solution(llm=chatgpt, equation=equation, silent=False)\n",
    "    answer_vicuna = find_solution(llm=vicuna, equation=equation, silent=False)\n",
    "    answer_mpt = find_solution(llm=mpt, equation=equation, silent=False)\n",
    "    answers1d['ChatGPT'].append(answer_gpt)\n",
    "    answers1d['Vicuna'].append(answer_vicuna)\n",
    "    answers1d['MPT'].append(answer_mpt)\n",
    "    correct1d['ChatGPT'].append(matches_groundtruth(parse_solution(answer_gpt['answer']), roots))\n",
    "    correct1d['Vicuna'].append(matches_groundtruth(parse_solution(answer_vicuna['answer']), roots))\n",
    "    correct1d['MPT'].append(matches_groundtruth(parse_solution(answer_mpt['answer']), roots))\n",
    "\n",
    "print('Frequency of correct answers:')\n",
    "print('ChatGPT: ', np.mean(correct1d['ChatGPT']))\n",
    "print('Vicuna: ', np.mean(correct1d['Vicuna']))\n",
    "print('MPT: ', np.mean(correct1d['MPT']))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Surprisingly, MPT still fails to solve these. Vicuna still makes some mistakes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-f43bfe4e-4234-4953-84ab-e667aa7044a2\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-f43bfe4e-4234-4953-84ab-e667aa7044a2\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user&#x27;s questions.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please find the solution to the following equation: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{equation}}'>x - 16.0 = 0</span>\n",
       "Think step by step, find the solution, and then say:\n",
       "SOLUTION = [value]\n",
       "For example, if the solution is x=3, say SOLUTION = [3].\n",
       "Make sure to use real numbers, not fractions.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0 max_tokens=300}}'>To solve the equation x - 16.0 = 0, we can follow these steps:\n",
       "\n",
       "1. Move the 16.0 to the right-hand side of the equation: x - 16.0 = 0 becomes 0 = x - 16.0\n",
       "2. Subtract 16.0 from both sides: 0 - 16.0 = 0 becomes -16.0 = -16.0\n",
       "3. Simplify the left-hand side by negating the right-hand side: -16.0 = -16.0 becomes 0 = 0\n",
       "4. Divide both sides by 1 to get the solution: 0 / 1 = 0 becomes x = 0\n",
       "\n",
       "Therefore, the solution to the equation x - 16.0 = 0 is x = 0.\n",
       "\n",
       "SOLUTION = [0]</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"f43bfe4e-4234-4953-84ab-e667aa7044a2\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "error = np.where(np.array(correct1d['Vicuna']) == False)[0][0]\n",
    "answers1d['Vicuna'][error]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-b7a07889-46e3-4b63-aca2-bb2b2c54d20c\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-b7a07889-46e3-4b63-aca2-bb2b2c54d20c\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>- You are a helpful assistant chatbot trained by MosaicML.  \n",
       "- You answer questions.\n",
       "- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n",
       "- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please find the solution to the following equation: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{equation}}'>x + 12.0 = 0</span>\n",
       "Think step by step, find the solution, and then say:\n",
       "SOLUTION = [value]\n",
       "For example, if the solution is x=3, say SOLUTION = [3].\n",
       "Make sure to use real numbers, not fractions.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0 max_tokens=300}}'>The equation x + 12.0 = 0 has infinitely many solutions, since it is true for any value of x.  However, if we want to find a specific solution, we need to divide both sides of the equation by 12.0, which gives us x = -1.  Therefore, the solution is x = -1.  SOLUTION = [-1]</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"b7a07889-46e3-4b63-aca2-bb2b2c54d20c\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "error = np.where(np.array(correct1d['MPT']) == False)[0][0]\n",
    "answers1d['MPT'][error]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-065fcc3e-f659-4bb0-88f6-43bd9c0b35ec\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-065fcc3e-f659-4bb0-88f6-43bd9c0b35ec\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>- You are a helpful assistant chatbot trained by MosaicML.  \n",
       "- You answer questions.\n",
       "- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n",
       "- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please find the solution to the following equation: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{equation}}'>x - 10.0 = 0</span>\n",
       "Think step by step, find the solution, and then say:\n",
       "SOLUTION = [value]\n",
       "For example, if the solution is x=3, say SOLUTION = [3].\n",
       "Make sure to use real numbers, not fractions.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0 max_tokens=300}}'>The equation x - 10.0 = 0 is an equation with one variable, x, on one side and a constant, -10.0, on the other side.  To solve for x, we need to isolate x on one side of the equation.  We can do this by adding 10.0 to both sides of the equation: x - 10.0 + 10.0 = 10.0.  Simplifying, we get x = 20.0.  Therefore, the solution to the equation x - 10.0 = 0 is x = 20.0.  SOLUTION = [20.0].</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"065fcc3e-f659-4bb0-88f6-43bd9c0b35ec\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "error = np.where(np.array(correct1d['MPT']) == False)[0][1]\n",
    "answers1d['MPT'][error]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Discussion**  \n",
    "This was a very toy task, but served as an example of how to compare models with different chat syntax using the same prompt.  \n",
    "For this particular combination of toy task and prompt, ChatGPT far surpasses Vicuna and MPT in terms of accuracy (measured exactly, because we have ground truth).  \n",
    "Let's now turn to more realistic tasks, where evaluating accuracy is not as straightforward."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting snippets + answering questions about meetings\n",
    "Let's say we want our LLM to answer questions (with the relevant conversation segments for grounding) about meeting transcripts.  \n",
    "This is an application where some users might prefer to use open-source LLMs rather than commercial ones, for privacy reasons (maybe I don't want to send all of my meeting data to OpenAI).  \n",
    "Here is a toy meeting transcript to start with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meeting_transcript = '''John: Alright, so we're all here to discuss the offer we received from Microsoft to buy our startup. What are your thoughts on this?\n",
    "Lucy: Well, I think it's a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
    "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
    "John: I see your point, but I'm a little hesitant about selling our startup. We've put a lot of time and effort into building this company, and I'm not sure if I'm ready to let it go just yet.\n",
    "Lucy: I understand where you're coming from, John, but we have to think about the future of our company. If we sell to Microsoft, we'll have access to their resources and expertise, which could help us grow our business even more.\n",
    "Steven: Right, and let's not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
    "John: I see your point, but I still have some reservations. What if Microsoft changes our product or our company culture? What if we lose control over our own business?\n",
    "Steven: You know what, I hadn't thought about this before, but maybe John is right. It would be a shame if our culture changed.\n",
    "Lucy: Those are valid concerns, but we can negotiate the terms of the deal to ensure that we retain some control over our company. And as for the product and culture, we can work with Microsoft to make sure that our vision is still intact.\n",
    "John: But won't we change just by virtue of being absorbed into a big company? I mean, we're a small startup with a very specific culture. Microsoft is a huge corporation with a very different culture. I'm not sure if the two can coexist.\n",
    "Steven: But John, didn't we always plan on being acquired? Won't this be a problem whenever?\n",
    "Lucy: Right\n",
    "John: I just don't want to lose what we've built here.\n",
    "Steven: I share this concern too''' "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by just trying to get ChatGPT to solve the task for us"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query1 = 'How does Steven feel about selling?'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-690110c2-c893-430a-bab0-f94d162e3ec1\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-690110c2-c893-430a-bab0-f94d162e3ec1\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>John: Alright, so we&#x27;re all here to discuss the offer we received from Microsoft to buy our startup. What are your thoughts on this?\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.\n",
       "Lucy: I understand where you&#x27;re coming from, John, but we have to think about the future of our company. If we sell to Microsoft, we&#x27;ll have access to their resources and expertise, which could help us grow our business even more.\n",
       "Steven: Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
       "John: I see your point, but I still have some reservations. What if Microsoft changes our product or our company culture? What if we lose control over our own business?\n",
       "Steven: You know what, I hadn&#x27;t thought about this before, but maybe John is right. It would be a shame if our culture changed.\n",
       "Lucy: Those are valid concerns, but we can negotiate the terms of the deal to ensure that we retain some control over our company. And as for the product and culture, we can work with Microsoft to make sure that our vision is still intact.\n",
       "John: But won&#x27;t we change just by virtue of being absorbed into a big company? I mean, we&#x27;re a small startup with a very specific culture. Microsoft is a huge corporation with a very different culture. I&#x27;m not sure if the two can coexist.\n",
       "Steven: But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?\n",
       "Lucy: Right\n",
       "John: I just don&#x27;t want to lose what we&#x27;ve built here.\n",
       "Steven: I share this concern too</span>\n",
       "----\n",
       "Please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Extract from the transcript the most relevant segments for the answer, and then answer the question.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>Steven feels positively about selling. He agrees with Lucy that Microsoft could provide the support they need to grow their business, and he also mentions the financial benefits of the deal. However, he later expresses concern about the potential changes to their company culture and product, which suggests that he has some reservations about the deal.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"690110c2-c893-430a-bab0-f94d162e3ec1\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_attempt1 = guidance('''{{#system~}}\n",
    "{{llm.default_system_prompt}}\n",
    "{{~/system}}\n",
    "\n",
    "{{#user~}}\n",
    "You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
    "Question: {{query}}\n",
    "Here is a meeting transcript:\n",
    "----\n",
    "{{transcript}}\n",
    "----\n",
    "Please answer the following question:\n",
    "Question: {{query}}\n",
    "Extract from the transcript the most relevant segments for the answer, and then answer the question.\n",
    "{{/user}}\n",
    "\n",
    "{{#assistant~}}\n",
    "{{gen 'answer' temperature=0}}\n",
    "{{~/assistant~}}''')\n",
    "qa_attempt1(llm=chatgpt, transcript=meeting_transcript, query=query1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the response is plausible, ChatGPT did not extract any conversation segments to ground the answer (and thus fails our specification). Let's try a different prompt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-014096be-040d-4a40-a822-767f831b36f9\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-014096be-040d-4a40-a822-767f831b36f9\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>John: Alright, so we&#x27;re all here to discuss the offer we received from Microsoft to buy our startup. What are your thoughts on this?\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.\n",
       "Lucy: I understand where you&#x27;re coming from, John, but we have to think about the future of our company. If we sell to Microsoft, we&#x27;ll have access to their resources and expertise, which could help us grow our business even more.\n",
       "Steven: Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
       "John: I see your point, but I still have some reservations. What if Microsoft changes our product or our company culture? What if we lose control over our own business?\n",
       "Steven: You know what, I hadn&#x27;t thought about this before, but maybe John is right. It would be a shame if our culture changed.\n",
       "Lucy: Those are valid concerns, but we can negotiate the terms of the deal to ensure that we retain some control over our company. And as for the product and culture, we can work with Microsoft to make sure that our vision is still intact.\n",
       "John: But won&#x27;t we change just by virtue of being absorbed into a big company? I mean, we&#x27;re a small startup with a very specific culture. Microsoft is a huge corporation with a very different culture. I&#x27;m not sure if the two can coexist.\n",
       "Steven: But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?\n",
       "Lucy: Right\n",
       "John: I just don&#x27;t want to lose what we&#x27;ve built here.\n",
       "Steven: I share this concern too</span>\n",
       "----\n",
       "Consider the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Now follow these steps:\n",
       "1. Extract from the transcript the most relevant segments for the answer\n",
       "2. Answer the question.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>1. Relevant segments:\n",
       "- &quot;I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.&quot;\n",
       "- &quot;Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.&quot;\n",
       "- &quot;But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?&quot;\n",
       "- &quot;I share this concern too&quot;\n",
       "\n",
       "2. Answer:\n",
       "Steven seems to be in favor of selling the startup to Microsoft. He believes that Microsoft&#x27;s experience and resources could help the company grow, and he also mentions the financial benefits of the deal. He also reminds John that they had always planned on being acquired, and expresses concern about the potential changes that could occur after the acquisition. Overall, Steven seems to be supportive of the idea of selling the startup, but also acknowledges the potential risks involved.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"014096be-040d-4a40-a822-767f831b36f9\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_attempt2 = guidance('''{{#system~}}\n",
    "{{llm.default_system_prompt}}\n",
    "{{~/system}}\n",
    "{{#user~}}\n",
    "You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
    "Question: {{query}}\n",
    "Here is a meeting transcript:\n",
    "----\n",
    "{{transcript}}\n",
    "----\n",
    "Consider the following question:\n",
    "Question: {{query}}\n",
    "Now follow these steps:\n",
    "1. Extract from the transcript the most relevant segments for the answer\n",
    "2. Answer the question.\n",
    "{{/user}}\n",
    "{{#assistant~}}\n",
    "{{gen 'answer' temperature=0}}\n",
    "{{~/assistant~}}''')\n",
    "qa_attempt2(llm=chatgpt, transcript=meeting_transcript, query=query1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is better, but maybe we want to specify the output format a little more, e.g. let's say we want each segment to have a summary, and to keep the character names. \n",
    "Let's try again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-22350d6e-29d6-4a48-ad7f-a6f6cd987fc7\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-22350d6e-29d6-4a48-ad7f-a6f6cd987fc7\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>John: Alright, so we&#x27;re all here to discuss the offer we received from Microsoft to buy our startup. What are your thoughts on this?\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.\n",
       "Lucy: I understand where you&#x27;re coming from, John, but we have to think about the future of our company. If we sell to Microsoft, we&#x27;ll have access to their resources and expertise, which could help us grow our business even more.\n",
       "Steven: Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
       "John: I see your point, but I still have some reservations. What if Microsoft changes our product or our company culture? What if we lose control over our own business?\n",
       "Steven: You know what, I hadn&#x27;t thought about this before, but maybe John is right. It would be a shame if our culture changed.\n",
       "Lucy: Those are valid concerns, but we can negotiate the terms of the deal to ensure that we retain some control over our company. And as for the product and culture, we can work with Microsoft to make sure that our vision is still intact.\n",
       "John: But won&#x27;t we change just by virtue of being absorbed into a big company? I mean, we&#x27;re a small startup with a very specific culture. Microsoft is a huge corporation with a very different culture. I&#x27;m not sure if the two can coexist.\n",
       "Steven: But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?\n",
       "Lucy: Right\n",
       "John: I just don&#x27;t want to lose what we&#x27;ve built here.\n",
       "Steven: I share this concern too</span>\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
       "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
       "\n",
       "As an example of output format, here is a fictitious answer to a question about another meeting transcript.\n",
       "CONVERSATION SEGMENTS:\n",
       "Segment 1: Peter and John discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Segment 2: Peter insults John\n",
       "Peter: John, you are a bad person.\n",
       "Segment 3: Blank\n",
       "ANSWER: Peter and John discussed the weather and Peter insulted John.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>CONVERSATION SEGMENTS:\n",
       "Segment 1: &quot;I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.&quot;\n",
       "Segment 2: &quot;Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.&quot;\n",
       "Segment 3: &quot;I share this concern too.&quot;\n",
       "ANSWER: Steven is in favor of selling the startup to Microsoft, citing their experience and support in the tech industry, as well as the financial benefits. However, he also shares John&#x27;s concern about potentially losing the company culture.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"22350d6e-29d6-4a48-ad7f-a6f6cd987fc7\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_attempt3 = guidance('''{{#system~}}\n",
    "{{llm.default_system_prompt}}\n",
    "{{~/system}}\n",
    "{{#user~}}\n",
    "You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
    "Question: {{query}}\n",
    "Here is a meeting transcript:\n",
    "----\n",
    "{{transcript}}\n",
    "----\n",
    "Based on the above, please answer the following question:\n",
    "Question: {{query}}\n",
    "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
    "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
    "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
    "\n",
    "As an example of output format, here is a fictitious answer to a question about another meeting transcript.\n",
    "CONVERSATION SEGMENTS:\n",
    "Segment 1: Peter and John discuss the weather.\n",
    "Peter: John, how is the weather today?\n",
    "John: It's raining.\n",
    "Segment 2: Peter insults John\n",
    "Peter: John, you are a bad person.\n",
    "Segment 3: Blank\n",
    "ANSWER: Peter and John discussed the weather and Peter insulted John.\n",
    "{{/user}}\n",
    "{{#assistant~}}\n",
    "{{gen 'answer' temperature=0}}\n",
    "{{~/assistant~}}''')\n",
    "qa_attempt3(llm=chatgpt, transcript=meeting_transcript, query=query1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ChatGPT did extract relevant segments, but it did not follow our output format (it did not summarize each segment, nor did it have the participant's names). Let's try again, with more explicit instructions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-5e7ceddb-d32b-4ff0-a4d5-0c965616f360\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-5e7ceddb-d32b-4ff0-a4d5-0c965616f360\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>John: Alright, so we&#x27;re all here to discuss the offer we received from Microsoft to buy our startup. What are your thoughts on this?\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.\n",
       "Lucy: I understand where you&#x27;re coming from, John, but we have to think about the future of our company. If we sell to Microsoft, we&#x27;ll have access to their resources and expertise, which could help us grow our business even more.\n",
       "Steven: Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
       "John: I see your point, but I still have some reservations. What if Microsoft changes our product or our company culture? What if we lose control over our own business?\n",
       "Steven: You know what, I hadn&#x27;t thought about this before, but maybe John is right. It would be a shame if our culture changed.\n",
       "Lucy: Those are valid concerns, but we can negotiate the terms of the deal to ensure that we retain some control over our company. And as for the product and culture, we can work with Microsoft to make sure that our vision is still intact.\n",
       "John: But won&#x27;t we change just by virtue of being absorbed into a big company? I mean, we&#x27;re a small startup with a very specific culture. Microsoft is a huge corporation with a very different culture. I&#x27;m not sure if the two can coexist.\n",
       "Steven: But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?\n",
       "Lucy: Right\n",
       "John: I just don&#x27;t want to lose what we&#x27;ve built here.\n",
       "Steven: I share this concern too</span>\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
       "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
       "\n",
       "Your output should have the following structure:\n",
       "CONVERSATION SEGMENTS:\n",
       "Segment 1: a summary of the first conversation segment\n",
       "(segment here)\n",
       "Segment 2: a summary of the second conversation segment\n",
       "(segment here)\n",
       "Segment 3: a summary of the third conversation segment\n",
       "(segment here)\n",
       "ANSWER: the answer to the question, supported by the segments above.\n",
       "\n",
       "As an example of output format, here is a fictitious answer to a question about another meeting transcript.\n",
       "CONVERSATION SEGMENTS:\n",
       "Segment 1: Peter and John discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Segment 2: Peter insults John\n",
       "Peter: John, you are a bad person.\n",
       "Segment 3: Blank\n",
       "ANSWER: Peter and John discussed the weather and Peter insulted John.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>CONVERSATION SEGMENTS:\n",
       "Segment 1: Steven agrees with Lucy that selling to Microsoft would be a good opportunity for the company.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "Segment 2: Steven highlights the financial benefits of selling to Microsoft.\n",
       "Steven: Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
       "Segment 3: Steven shares John&#x27;s concern about losing what they&#x27;ve built.\n",
       "Steven: I share this concern too.\n",
       "\n",
       "ANSWER: Steven thinks that selling to Microsoft would be a good opportunity for the company and highlights the financial benefits, but he also shares John&#x27;s concern about losing what they&#x27;ve built.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"5e7ceddb-d32b-4ff0-a4d5-0c965616f360\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_attempt4 = guidance('''{{#system~}}\n",
    "{{llm.default_system_prompt}}\n",
    "{{~/system}}\n",
    "{{#user~}}\n",
    "You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
    "Question: {{query}}\n",
    "Here is a meeting transcript:\n",
    "----\n",
    "{{transcript}}\n",
    "----\n",
    "Based on the above, please answer the following question:\n",
    "Question: {{query}}\n",
    "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
    "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
    "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
    "\n",
    "Your output should have the following structure:\n",
    "CONVERSATION SEGMENTS:\n",
    "Segment 1: a summary of the first conversation segment\n",
    "(segment here)\n",
    "Segment 2: a summary of the second conversation segment\n",
    "(segment here)\n",
    "Segment 3: a summary of the third conversation segment\n",
    "(segment here)\n",
    "ANSWER: the answer to the question, supported by the segments above.\n",
    "\n",
    "As an example of output format, here is a fictitious answer to a question about another meeting transcript.\n",
    "CONVERSATION SEGMENTS:\n",
    "Segment 1: Peter and John discuss the weather.\n",
    "Peter: John, how is the weather today?\n",
    "John: It's raining.\n",
    "Segment 2: Peter insults John\n",
    "Peter: John, you are a bad person.\n",
    "Segment 3: Blank\n",
    "ANSWER: Peter and John discussed the weather and Peter insulted John.\n",
    "{{/user}}\n",
    "{{#assistant~}}\n",
    "{{gen 'answer' temperature=0}}\n",
    "{{~/assistant~}}''')\n",
    "qa_attempt4(llm=chatgpt, transcript=meeting_transcript, query=query1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we got ChatGPT to use the format we wanted.\n",
    "The root of the problem we're facing is that the OpenAI API does not allow us to do partial output completion (i.e. we can't specify how the assistant begins to answer), and thus it's hard for us to **guide** the output.  \n",
    "If, instead, we use one of the open source models, we can guide the output more clearly, forcing the model to use our structure.  \n",
    "For example, here is how we might modify `qa_attempt3`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-9d31c9a6-4be4-4f41-a6d3-768d36466704\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-9d31c9a6-4be4-4f41-a6d3-768d36466704\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user&#x27;s questions.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>John: Alright, so we&#x27;re all here to discuss the offer we received from Microsoft to buy our startup. What are your thoughts on this?\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.\n",
       "Lucy: I understand where you&#x27;re coming from, John, but we have to think about the future of our company. If we sell to Microsoft, we&#x27;ll have access to their resources and expertise, which could help us grow our business even more.\n",
       "Steven: Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
       "John: I see your point, but I still have some reservations. What if Microsoft changes our product or our company culture? What if we lose control over our own business?\n",
       "Steven: You know what, I hadn&#x27;t thought about this before, but maybe John is right. It would be a shame if our culture changed.\n",
       "Lucy: Those are valid concerns, but we can negotiate the terms of the deal to ensure that we retain some control over our company. And as for the product and culture, we can work with Microsoft to make sure that our vision is still intact.\n",
       "John: But won&#x27;t we change just by virtue of being absorbed into a big company? I mean, we&#x27;re a small startup with a very specific culture. Microsoft is a huge corporation with a very different culture. I&#x27;m not sure if the two can coexist.\n",
       "Steven: But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?\n",
       "Lucy: Right\n",
       "John: I just don&#x27;t want to lose what we&#x27;ve built here.\n",
       "Steven: I share this concern too</span>\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Please extract the three segment from the transcript that are the most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns. If you need less than three segments, you can leave the rest blank.\n",
       "\n",
       "As an example of output format, here is a fictitious answer to a question about another meeting transcript:\n",
       "CONVERSATION SEGMENTS:\n",
       "Segment 1: Peter and John discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Segment 2: Peter insults John\n",
       "Peter: John, you are a bad person.\n",
       "Segment 3: Blank\n",
       "ANSWER: Peter and John discussed the weather and Peter insulted John.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>CONVERSATION SEGMENTS:\n",
       "Segment 1: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment1&#x27; temperature=0}}'>Steven agrees with Lucy about the potential benefits of selling to Microsoft.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.</span>\n",
       "Segment 2: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment2&#x27; temperature=0}}'>John expresses his reservations about selling the startup.\n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.</span>\n",
       "Segment 3: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment3&#x27; temperature=0}}'>Steven expresses his concerns about losing control over the company and its culture.\n",
       "Steven: You know what, I hadn&#x27;t thought about this before, but maybe John is right. It would be a shame if our culture changed.\n",
       "</span>\n",
       "ANSWER: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>Steven agrees with Lucy about the potential benefits of selling to Microsoft, but expresses concerns about losing control over the company and its culture.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"9d31c9a6-4be4-4f41-a6d3-768d36466704\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_guided = guidance('''{{#system~}}\n",
    "{{llm.default_system_prompt}}\n",
    "{{~/system}}\n",
    "\n",
    "{{#user~}}\n",
    "You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
    "Question: {{query}}\n",
    "----\n",
    "{{transcript}}\n",
    "----\n",
    "Based on the above, please answer the following question:\n",
    "Question: {{query}}\n",
    "Please extract the three segment from the transcript that are the most relevant for the answer, and then answer the question.\n",
    "Note that conversation segments can be of any length, e.g. including multiple conversation turns. If you need less than three segments, you can leave the rest blank.\n",
    "\n",
    "As an example of output format, here is a fictitious answer to a question about another meeting transcript:\n",
    "CONVERSATION SEGMENTS:\n",
    "Segment 1: Peter and John discuss the weather.\n",
    "Peter: John, how is the weather today?\n",
    "John: It's raining.\n",
    "Segment 2: Peter insults John\n",
    "Peter: John, you are a bad person.\n",
    "Segment 3: Blank\n",
    "ANSWER: Peter and John discussed the weather and Peter insulted John.\n",
    "{{/user}}\n",
    "\n",
    "{{#assistant~}}\n",
    "CONVERSATION SEGMENTS:\n",
    "Segment 1: {{gen 'segment1' temperature=0}}\n",
    "Segment 2: {{gen 'segment2' temperature=0}}\n",
    "Segment 3: {{gen 'segment3' temperature=0}}\n",
    "ANSWER: {{gen 'answer' temperature=0}}\n",
    "{{~/assistant~}}''')\n",
    "qa_guided(llm=vicuna, transcript=meeting_transcript, query=query1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this guidance, we get the right format the first time (and all the time). Let's see how MPT does:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-5e199111-644c-493e-b23d-082fdf7cb114\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-5e199111-644c-493e-b23d-082fdf7cb114\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>- You are a helpful assistant chatbot trained by MosaicML.\n",
       "    - You answer questions.\n",
       "    - You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n",
       "    - You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>John: Alright, so we&#x27;re all here to discuss the offer we received from Microsoft to buy our startup. What are your thoughts on this?\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.\n",
       "Lucy: I understand where you&#x27;re coming from, John, but we have to think about the future of our company. If we sell to Microsoft, we&#x27;ll have access to their resources and expertise, which could help us grow our business even more.\n",
       "Steven: Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
       "John: I see your point, but I still have some reservations. What if Microsoft changes our product or our company culture? What if we lose control over our own business?\n",
       "Steven: You know what, I hadn&#x27;t thought about this before, but maybe John is right. It would be a shame if our culture changed.\n",
       "Lucy: Those are valid concerns, but we can negotiate the terms of the deal to ensure that we retain some control over our company. And as for the product and culture, we can work with Microsoft to make sure that our vision is still intact.\n",
       "John: But won&#x27;t we change just by virtue of being absorbed into a big company? I mean, we&#x27;re a small startup with a very specific culture. Microsoft is a huge corporation with a very different culture. I&#x27;m not sure if the two can coexist.\n",
       "Steven: But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?\n",
       "Lucy: Right\n",
       "John: I just don&#x27;t want to lose what we&#x27;ve built here.\n",
       "Steven: I share this concern too</span>\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Please extract the three segment from the transcript that are the most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns. If you need less than three segments, you can leave the rest blank.\n",
       "\n",
       "As an example of output format, here is a fictitious answer to a question about another meeting transcript:\n",
       "CONVERSATION SEGMENTS:\n",
       "Segment 1: Peter and John discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Segment 2: Peter insults John\n",
       "Peter: John, you are a bad person.\n",
       "Segment 3: Blank\n",
       "ANSWER: Peter and John discussed the weather and Peter insulted John.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>CONVERSATION SEGMENTS:\n",
       "Segment 1: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment1&#x27; temperature=0}}'>John and Peter discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.</span>\n",
       "Segment 2: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment2&#x27; temperature=0}}'>Peter insults John\n",
       "Peter: John, you are a bad person.</span>\n",
       "Segment 3: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment3&#x27; temperature=0}}'>Blank</span>\n",
       "ANSWER: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>Peter and John discussed the weather and Peter insulted John.</span>\n",
       "</div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"5e199111-644c-493e-b23d-082fdf7cb114\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_guided(llm=mpt, transcript=meeting_transcript, query=query1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While MPT follows the format, it ignores the question and takes snippets from the format example rather than from the real transcript.  \n",
    "From now on, we'll just compare ChatGPT and Vicuna."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try another question:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query2 = 'Who wants to sell the company?'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-f8a8c756-87a5-470a-b0fe-446bf74b8b34\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-f8a8c756-87a5-470a-b0fe-446bf74b8b34\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>Who wants to sell the company?</span>\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>John: Alright, so we&#x27;re all here to discuss the offer we received from Microsoft to buy our startup. What are your thoughts on this?\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.\n",
       "Lucy: I understand where you&#x27;re coming from, John, but we have to think about the future of our company. If we sell to Microsoft, we&#x27;ll have access to their resources and expertise, which could help us grow our business even more.\n",
       "Steven: Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
       "John: I see your point, but I still have some reservations. What if Microsoft changes our product or our company culture? What if we lose control over our own business?\n",
       "Steven: You know what, I hadn&#x27;t thought about this before, but maybe John is right. It would be a shame if our culture changed.\n",
       "Lucy: Those are valid concerns, but we can negotiate the terms of the deal to ensure that we retain some control over our company. And as for the product and culture, we can work with Microsoft to make sure that our vision is still intact.\n",
       "John: But won&#x27;t we change just by virtue of being absorbed into a big company? I mean, we&#x27;re a small startup with a very specific culture. Microsoft is a huge corporation with a very different culture. I&#x27;m not sure if the two can coexist.\n",
       "Steven: But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?\n",
       "Lucy: Right\n",
       "John: I just don&#x27;t want to lose what we&#x27;ve built here.\n",
       "Steven: I share this concern too</span>\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>Who wants to sell the company?</span>\n",
       "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
       "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
       "\n",
       "Your output should have the following structure:\n",
       "CONVERSATION SEGMENTS:\n",
       "Segment 1: a summary of the first conversation segment\n",
       "(segment here)\n",
       "Segment 2: a summary of the second conversation segment\n",
       "(segment here)\n",
       "Segment 3: a summary of the third conversation segment\n",
       "(segment here)\n",
       "ANSWER: the answer to the question, supported by the segments above.\n",
       "\n",
       "As an example of output format, here is a fictitious answer to a question about another meeting transcript.\n",
       "CONVERSATION SEGMENTS:\n",
       "Segment 1: Peter and John discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Segment 2: Peter insults John\n",
       "Peter: John, you are a bad person.\n",
       "Segment 3: Blank\n",
       "ANSWER: Peter and John discussed the weather and Peter insulted John.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>CONVERSATION SEGMENTS:\n",
       "Segment 1: \n",
       "John: Alright, so we&#x27;re all here to discuss the offer we received from Microsoft to buy our startup. What are your thoughts on this?\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "\n",
       "Segment 2: \n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.\n",
       "\n",
       "Segment 3: \n",
       "John: But won&#x27;t we change just by virtue of being absorbed into a big company? I mean, we&#x27;re a small startup with a very specific culture. Microsoft is a huge corporation with a very different culture. I&#x27;m not sure if the two can coexist.\n",
       "Steven: But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?\n",
       "\n",
       "ANSWER: John is hesitant about selling the company.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"f8a8c756-87a5-470a-b0fe-446bf74b8b34\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_attempt4(llm=chatgpt, system_prompt=chatgpt_system, transcript=meeting_transcript, query=query2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that we didn't fix the formatting issue with ChatGPT yet. Not only does it not summarize the segments, it also doesn't really answer the question. Let's try again, putting a one-shot example as a conversation round this time:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_attempt5 = guidance('''{{#system~}}\n",
    "{{llm.default_system_prompt}}\n",
    "{{~/system}}\n",
    "{{#user~}}\n",
    "You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
    "Question: What were the main things that happened in the meeting?\n",
    "Here is a meeting transcript:\n",
    "----\n",
    "Peter: Hey\n",
    "John: Hey\n",
    "Peter: John, how is the weather today?\n",
    "John: It's raining.\n",
    "Peter: That's too bad. I was hoping to go for a walk later.\n",
    "John: Yeah, it's a shame.\n",
    "Peter: John, you are a bad person.\n",
    "----\n",
    "Based on the above, please answer the following question:\n",
    "Question: {{query}}\n",
    "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
    "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
    "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
    "{{/user}}\n",
    "{{#assistant~}}\n",
    "CONVERSATION SEGMENTS:\n",
    "Segment 1: Peter and John discuss the weather.\n",
    "Peter: John, how is the weather today?\n",
    "John: It's raining.\n",
    "Segment 2: Peter insults John\n",
    "Peter: John, you are a bad person.\n",
    "Segment 3: Blank\n",
    "ANSWER: Peter and John discussed the weather and Peter insulted John.\n",
    "{{~/assistant~}}\n",
    "{{#user~}}\n",
    "You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
    "Question: {{query}}\n",
    "Here is a meeting transcript:\n",
    "----\n",
    "{{transcript}}\n",
    "----\n",
    "Based on the above, please answer the following question:\n",
    "Question: {{query}}\n",
    "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
    "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
    "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
    "{{/user}}\n",
    "{{#assistant~}}\n",
    "{{gen 'answer' temperature=0}}\n",
    "{{~/assistant~}}''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-f92819c2-c5ba-4362-b4a7-7d562c4a482f\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-f92819c2-c5ba-4362-b4a7-7d562c4a482f\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: What were the main things that happened in the meeting?\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "Peter: Hey\n",
       "John: Hey\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Peter: That&#x27;s too bad. I was hoping to go for a walk later.\n",
       "John: Yeah, it&#x27;s a shame.\n",
       "Peter: John, you are a bad person.\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>Who wants to sell the company?</span>\n",
       "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
       "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>CONVERSATION SEGMENTS:\n",
       "Segment 1: Peter and John discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Segment 2: Peter insults John\n",
       "Peter: John, you are a bad person.\n",
       "Segment 3: Blank\n",
       "ANSWER: Peter and John discussed the weather and Peter insulted John.</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>Who wants to sell the company?</span>\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>John: Alright, so we&#x27;re all here to discuss the offer we received from Microsoft to buy our startup. What are your thoughts on this?\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.\n",
       "Lucy: I understand where you&#x27;re coming from, John, but we have to think about the future of our company. If we sell to Microsoft, we&#x27;ll have access to their resources and expertise, which could help us grow our business even more.\n",
       "Steven: Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
       "John: I see your point, but I still have some reservations. What if Microsoft changes our product or our company culture? What if we lose control over our own business?\n",
       "Steven: You know what, I hadn&#x27;t thought about this before, but maybe John is right. It would be a shame if our culture changed.\n",
       "Lucy: Those are valid concerns, but we can negotiate the terms of the deal to ensure that we retain some control over our company. And as for the product and culture, we can work with Microsoft to make sure that our vision is still intact.\n",
       "John: But won&#x27;t we change just by virtue of being absorbed into a big company? I mean, we&#x27;re a small startup with a very specific culture. Microsoft is a huge corporation with a very different culture. I&#x27;m not sure if the two can coexist.\n",
       "Steven: But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?\n",
       "Lucy: Right\n",
       "John: I just don&#x27;t want to lose what we&#x27;ve built here.\n",
       "Steven: I share this concern too</span>\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>Who wants to sell the company?</span>\n",
       "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
       "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>CONVERSATION SEGMENTS:\n",
       "Segment 1: John expresses hesitation about selling the startup\n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.\n",
       "Segment 2: Lucy and Steven express support for selling to Microsoft\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "Segment 3: Blank\n",
       "ANSWER: Lucy and Steven want to sell the company to Microsoft, while John expresses hesitation about the deal.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"f92819c2-c5ba-4362-b4a7-7d562c4a482f\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_attempt5(llm=chatgpt, transcript=meeting_transcript, query=query2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It works this time (it also works on the original query):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-2c0699eb-1397-4f98-a8ec-2d9583379a55\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-2c0699eb-1397-4f98-a8ec-2d9583379a55\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: What were the main things that happened in the meeting?\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "Peter: Hey\n",
       "John: Hey\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Peter: That&#x27;s too bad. I was hoping to go for a walk later.\n",
       "John: Yeah, it&#x27;s a shame.\n",
       "Peter: John, you are a bad person.\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
       "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>CONVERSATION SEGMENTS:\n",
       "Segment 1: Peter and John discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Segment 2: Peter insults John\n",
       "Peter: John, you are a bad person.\n",
       "Segment 3: Blank\n",
       "ANSWER: Peter and John discussed the weather and Peter insulted John.</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>John: Alright, so we&#x27;re all here to discuss the offer we received from Microsoft to buy our startup. What are your thoughts on this?\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.\n",
       "Lucy: I understand where you&#x27;re coming from, John, but we have to think about the future of our company. If we sell to Microsoft, we&#x27;ll have access to their resources and expertise, which could help us grow our business even more.\n",
       "Steven: Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
       "John: I see your point, but I still have some reservations. What if Microsoft changes our product or our company culture? What if we lose control over our own business?\n",
       "Steven: You know what, I hadn&#x27;t thought about this before, but maybe John is right. It would be a shame if our culture changed.\n",
       "Lucy: Those are valid concerns, but we can negotiate the terms of the deal to ensure that we retain some control over our company. And as for the product and culture, we can work with Microsoft to make sure that our vision is still intact.\n",
       "John: But won&#x27;t we change just by virtue of being absorbed into a big company? I mean, we&#x27;re a small startup with a very specific culture. Microsoft is a huge corporation with a very different culture. I&#x27;m not sure if the two can coexist.\n",
       "Steven: But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?\n",
       "Lucy: Right\n",
       "John: I just don&#x27;t want to lose what we&#x27;ve built here.\n",
       "Steven: I share this concern too</span>\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>How does Steven feel about selling?</span>\n",
       "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
       "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>CONVERSATION SEGMENTS:\n",
       "Segment 1: Steven thinks selling to Microsoft is a good opportunity\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "Segment 2: Steven mentions the financial benefits of selling\n",
       "Steven: Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
       "Segment 3: Steven shares John&#x27;s concern about losing the company culture\n",
       "Steven: But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?\n",
       "John: I just don&#x27;t want to lose what we&#x27;ve built here.\n",
       "Steven: I share this concern too.\n",
       "ANSWER: Steven thinks selling to Microsoft is a good opportunity and mentions the financial benefits of selling. However, he also shares John&#x27;s concern about losing the company culture.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"2c0699eb-1397-4f98-a8ec-2d9583379a55\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_attempt5(llm=chatgpt, transcript=meeting_transcript, query=query1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how Vicuna does on this one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-c81184bf-6e2d-4072-890d-f152c6089ad7\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-c81184bf-6e2d-4072-890d-f152c6089ad7\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user&#x27;s questions.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>Who wants to sell the company?</span>\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>John: Alright, so we&#x27;re all here to discuss the offer we received from Microsoft to buy our startup. What are your thoughts on this?\n",
       "Lucy: Well, I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "John: I see your point, but I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.\n",
       "Lucy: I understand where you&#x27;re coming from, John, but we have to think about the future of our company. If we sell to Microsoft, we&#x27;ll have access to their resources and expertise, which could help us grow our business even more.\n",
       "Steven: Right, and let&#x27;s not forget about the financial benefits. Microsoft is offering us a lot of money for our startup, which could help us invest in new projects and expand our team.\n",
       "John: I see your point, but I still have some reservations. What if Microsoft changes our product or our company culture? What if we lose control over our own business?\n",
       "Steven: You know what, I hadn&#x27;t thought about this before, but maybe John is right. It would be a shame if our culture changed.\n",
       "Lucy: Those are valid concerns, but we can negotiate the terms of the deal to ensure that we retain some control over our company. And as for the product and culture, we can work with Microsoft to make sure that our vision is still intact.\n",
       "John: But won&#x27;t we change just by virtue of being absorbed into a big company? I mean, we&#x27;re a small startup with a very specific culture. Microsoft is a huge corporation with a very different culture. I&#x27;m not sure if the two can coexist.\n",
       "Steven: But John, didn&#x27;t we always plan on being acquired? Won&#x27;t this be a problem whenever?\n",
       "Lucy: Right\n",
       "John: I just don&#x27;t want to lose what we&#x27;ve built here.\n",
       "Steven: I share this concern too</span>\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>Who wants to sell the company?</span>\n",
       "Please extract the three segment from the transcript that are the most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns. If you need less than three segments, you can leave the rest blank.\n",
       "\n",
       "As an example of output format, here is a fictitious answer to a question about another meeting transcript:\n",
       "CONVERSATION SEGMENTS:\n",
       "Segment 1: Peter and John discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Segment 2: Peter insults John\n",
       "Peter: John, you are a bad person.\n",
       "Segment 3: Blank\n",
       "ANSWER: Peter and John discussed the weather and Peter insulted John.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>CONVERSATION SEGMENTS:\n",
       "Segment 1: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment1&#x27; temperature=0}}'>Lucy expresses her opinion on the offer from Microsoft.\n",
       "Lucy: I think it&#x27;s a great opportunity for us. Microsoft is a huge company with a lot of resources, and they could really help us take our product to the next level.</span>\n",
       "Segment 2: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment2&#x27; temperature=0}}'>John expresses his reservations about selling the company.\n",
       "John: I&#x27;m a little hesitant about selling our startup. We&#x27;ve put a lot of time and effort into building this company, and I&#x27;m not sure if I&#x27;m ready to let it go just yet.</span>\n",
       "Segment 3: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment3&#x27; temperature=0}}'>Steven expresses his agreement with Lucy.\n",
       "Steven: I agree with Lucy. Microsoft has a lot of experience in the tech industry, and they could provide us with the support we need to grow our business.\n",
       "</span>\n",
       "ANSWER: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>Lucy and Steven want to sell the company, but John is hesitant.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"c81184bf-6e2d-4072-890d-f152c6089ad7\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_guided(llm=vicuna, transcript=meeting_transcript, query=query2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vicuna just worked.  \n",
    "Let's now try both of these prompts on a different meeting transcript, the beginning of [an interview](https://www.rev.com/blog/transcripts/elon-musk-interview-with-the-bbc-4-11-23-transcript) with Elon Musk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Full transcript: https://www.rev.com/blog/transcripts/elon-musk-interview-with-the-bbc-4-11-23-transcript\n",
    "transcript2 = '''Interviewer: In Sachs that used to be in content moderation. And we’ve spoken to people very recently who were involved in moderation. And they just say there’s not enough people to police this stuff. Particularly around hate speech in the company. Is that something that you-\n",
    "Elon Musk: What hate speech are you talking about? I mean, you use Twitter?\n",
    "Interviewer: Right.\n",
    "Elon Musk: Do you see a rise in hate speech? Just your personal anecdote, do you? I don’t.\n",
    "Interviewer: Personally, my For You, I would see I get more of that kind of content. Yeah. Personally. But I’m not going to talk for the rest of Twitter.\n",
    "Elon Musk: You see more hate speech personally?\n",
    "Interviewer: I would say see more hateful content in that.\n",
    "Elon Musk: Content you don’t like, or hateful. Describe a hateful thing?\n",
    "Interviewer: Yeah, I mean just content that will solicit a reaction. Something that may include something that is slightly racist or slightly sexist. Those kinds of things.\n",
    "Elon Musk: So you think if it’s something is slightly sexist it should be banned?\n",
    "Interviewer: No, I’m not saying anything. I’m saying-\n",
    "Elon Musk: I’m just curious. I’m trying to understand what you mean by “hateful content.” And I’m asking for specific examples. And you just said that if something is slightly sexist, that’s hateful content. Does that mean that it should be banned?\n",
    "Interviewer: Well, you’ve asked me whether my feed, whether it’s got less or more. I’d say it’s got slightly more.\n",
    "Elon Musk: That’s why I’m asking for examples. Can you name one example?\n",
    "Interviewer: I honestly don’t…\n",
    "Elon Musk: You can’t name a single example?\n",
    "Interviewer: I’ll tell you why. Because I don’t actually use that For You feed anymore. Because I just don’t particularly like it. Actually a lot of people are quite similar. I only look at my following.\n",
    "Elon Musk: You said you’ve seen more hateful content, but you can’t name a single example. Not even one.\n",
    "Interviewer: I’m not sure I’ve used that feed for the last three or four weeks. And I honestly couldn’t-\n",
    "Elon Musk: Then how could you see the hateful content?\n",
    "Interviewer: Because I’ve been using it. I’ve been using Twitter since you’ve taken it over for the last six months.\n",
    "Elon Musk: Then you must have at some point seen the For You hateful content. I’m asking for one example.\n",
    "Interviewer: Right.\n",
    "Elon Musk: And you can’t give a single one.\n",
    "Interviewer: And I’m saying-\n",
    "Elon Musk: Then I say, sir, that you don’t know what you’re talking about.\n",
    "Interviewer: Really?\n",
    "Elon Musk: Yes. Because you can’t give a single example of hateful content. Not even one tweet. And yet you claimed that the hateful content was high. That’s false.\n",
    "Interviewer: No. What I claimed-\n",
    "Elon Musk: You just lied.\n",
    "Interviewer: No, no. What I claimed was there are many organizations that say that that kind of information is on the rise. Now whether it has on my feed or not...'''\n",
    "query3 = 'The interviewer says his claim was not about his personal feed. Is this true?'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-2c7087ee-30ff-4165-90e1-ef24cfc6685b\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-2c7087ee-30ff-4165-90e1-ef24cfc6685b\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: What were the main things that happened in the meeting?\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "Peter: Hey\n",
       "John: Hey\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Peter: That&#x27;s too bad. I was hoping to go for a walk later.\n",
       "John: Yeah, it&#x27;s a shame.\n",
       "Peter: John, you are a bad person.\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>The interviewer says his claim was not about his personal feed. Is this true?</span>\n",
       "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
       "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>CONVERSATION SEGMENTS:\n",
       "Segment 1: Peter and John discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Segment 2: Peter insults John\n",
       "Peter: John, you are a bad person.\n",
       "Segment 3: Blank\n",
       "ANSWER: Peter and John discussed the weather and Peter insulted John.</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>The interviewer says his claim was not about his personal feed. Is this true?</span>\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>Interviewer: In Sachs that used to be in content moderation. And we’ve spoken to people very recently who were involved in moderation. And they just say there’s not enough people to police this stuff. Particularly around hate speech in the company. Is that something that you-\n",
       "Elon Musk: What hate speech are you talking about? I mean, you use Twitter?\n",
       "Interviewer: Right.\n",
       "Elon Musk: Do you see a rise in hate speech? Just your personal anecdote, do you? I don’t.\n",
       "Interviewer: Personally, my For You, I would see I get more of that kind of content. Yeah. Personally. But I’m not going to talk for the rest of Twitter.\n",
       "Elon Musk: You see more hate speech personally?\n",
       "Interviewer: I would say see more hateful content in that.\n",
       "Elon Musk: Content you don’t like, or hateful. Describe a hateful thing?\n",
       "Interviewer: Yeah, I mean just content that will solicit a reaction. Something that may include something that is slightly racist or slightly sexist. Those kinds of things.\n",
       "Elon Musk: So you think if it’s something is slightly sexist it should be banned?\n",
       "Interviewer: No, I’m not saying anything. I’m saying-\n",
       "Elon Musk: I’m just curious. I’m trying to understand what you mean by “hateful content.” And I’m asking for specific examples. And you just said that if something is slightly sexist, that’s hateful content. Does that mean that it should be banned?\n",
       "Interviewer: Well, you’ve asked me whether my feed, whether it’s got less or more. I’d say it’s got slightly more.\n",
       "Elon Musk: That’s why I’m asking for examples. Can you name one example?\n",
       "Interviewer: I honestly don’t…\n",
       "Elon Musk: You can’t name a single example?\n",
       "Interviewer: I’ll tell you why. Because I don’t actually use that For You feed anymore. Because I just don’t particularly like it. Actually a lot of people are quite similar. I only look at my following.\n",
       "Elon Musk: You said you’ve seen more hateful content, but you can’t name a single example. Not even one.\n",
       "Interviewer: I’m not sure I’ve used that feed for the last three or four weeks. And I honestly couldn’t-\n",
       "Elon Musk: Then how could you see the hateful content?\n",
       "Interviewer: Because I’ve been using it. I’ve been using Twitter since you’ve taken it over for the last six months.\n",
       "Elon Musk: Then you must have at some point seen the For You hateful content. I’m asking for one example.\n",
       "Interviewer: Right.\n",
       "Elon Musk: And you can’t give a single one.\n",
       "Interviewer: And I’m saying-\n",
       "Elon Musk: Then I say, sir, that you don’t know what you’re talking about.\n",
       "Interviewer: Really?\n",
       "Elon Musk: Yes. Because you can’t give a single example of hateful content. Not even one tweet. And yet you claimed that the hateful content was high. That’s false.\n",
       "Interviewer: No. What I claimed-\n",
       "Elon Musk: You just lied.\n",
       "Interviewer: No, no. What I claimed was there are many organizations that say that that kind of information is on the rise. Now whether it has on my feed or not...</span>\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>The interviewer says his claim was not about his personal feed. Is this true?</span>\n",
       "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
       "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>CONVERSATION SEGMENTS:\n",
       "Segment 1: Interviewer admits to seeing more hateful content in his personal feed.\n",
       "Interviewer: Personally, my For You, I would see I get more of that kind of content. Yeah. Personally. But I’m not going to talk for the rest of Twitter.\n",
       "Segment 2: Elon Musk questions the interviewer&#x27;s claim and asks for specific examples.\n",
       "Elon Musk: You said you’ve seen more hateful content, but you can’t name a single example. Not even one.\n",
       "Segment 3: Interviewer clarifies that his claim was not just about his personal feed.\n",
       "Interviewer: No, no. What I claimed was there are many organizations that say that that kind of information is on the rise. Now whether it has on my feed or not...\n",
       "ANSWER: The interviewer&#x27;s claim was not just about his personal feed, but also about organizations reporting a rise in hateful content.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"2c7087ee-30ff-4165-90e1-ef24cfc6685b\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_attempt5(llm=chatgpt, transcript=transcript2, query=query3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-7f515c1d-cb9c-413f-bca1-1d2c884729d8\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-7f515c1d-cb9c-413f-bca1-1d2c884729d8\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user&#x27;s questions.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>The interviewer says his claim was not about his personal feed. Is this true?</span>\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>Interviewer: In Sachs that used to be in content moderation. And we’ve spoken to people very recently who were involved in moderation. And they just say there’s not enough people to police this stuff. Particularly around hate speech in the company. Is that something that you-\n",
       "Elon Musk: What hate speech are you talking about? I mean, you use Twitter?\n",
       "Interviewer: Right.\n",
       "Elon Musk: Do you see a rise in hate speech? Just your personal anecdote, do you? I don’t.\n",
       "Interviewer: Personally, my For You, I would see I get more of that kind of content. Yeah. Personally. But I’m not going to talk for the rest of Twitter.\n",
       "Elon Musk: You see more hate speech personally?\n",
       "Interviewer: I would say see more hateful content in that.\n",
       "Elon Musk: Content you don’t like, or hateful. Describe a hateful thing?\n",
       "Interviewer: Yeah, I mean just content that will solicit a reaction. Something that may include something that is slightly racist or slightly sexist. Those kinds of things.\n",
       "Elon Musk: So you think if it’s something is slightly sexist it should be banned?\n",
       "Interviewer: No, I’m not saying anything. I’m saying-\n",
       "Elon Musk: I’m just curious. I’m trying to understand what you mean by “hateful content.” And I’m asking for specific examples. And you just said that if something is slightly sexist, that’s hateful content. Does that mean that it should be banned?\n",
       "Interviewer: Well, you’ve asked me whether my feed, whether it’s got less or more. I’d say it’s got slightly more.\n",
       "Elon Musk: That’s why I’m asking for examples. Can you name one example?\n",
       "Interviewer: I honestly don’t…\n",
       "Elon Musk: You can’t name a single example?\n",
       "Interviewer: I’ll tell you why. Because I don’t actually use that For You feed anymore. Because I just don’t particularly like it. Actually a lot of people are quite similar. I only look at my following.\n",
       "Elon Musk: You said you’ve seen more hateful content, but you can’t name a single example. Not even one.\n",
       "Interviewer: I’m not sure I’ve used that feed for the last three or four weeks. And I honestly couldn’t-\n",
       "Elon Musk: Then how could you see the hateful content?\n",
       "Interviewer: Because I’ve been using it. I’ve been using Twitter since you’ve taken it over for the last six months.\n",
       "Elon Musk: Then you must have at some point seen the For You hateful content. I’m asking for one example.\n",
       "Interviewer: Right.\n",
       "Elon Musk: And you can’t give a single one.\n",
       "Interviewer: And I’m saying-\n",
       "Elon Musk: Then I say, sir, that you don’t know what you’re talking about.\n",
       "Interviewer: Really?\n",
       "Elon Musk: Yes. Because you can’t give a single example of hateful content. Not even one tweet. And yet you claimed that the hateful content was high. That’s false.\n",
       "Interviewer: No. What I claimed-\n",
       "Elon Musk: You just lied.\n",
       "Interviewer: No, no. What I claimed was there are many organizations that say that that kind of information is on the rise. Now whether it has on my feed or not...</span>\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>The interviewer says his claim was not about his personal feed. Is this true?</span>\n",
       "Please extract the three segment from the transcript that are the most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns. If you need less than three segments, you can leave the rest blank.\n",
       "\n",
       "As an example of output format, here is a fictitious answer to a question about another meeting transcript:\n",
       "CONVERSATION SEGMENTS:\n",
       "Segment 1: Peter and John discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Segment 2: Peter insults John\n",
       "Peter: John, you are a bad person.\n",
       "Segment 3: Blank\n",
       "ANSWER: Peter and John discussed the weather and Peter insulted John.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>CONVERSATION SEGMENTS:\n",
       "Segment 1: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment1&#x27; temperature=0}}'>Interviewer claims to have seen more hateful content on Twitter.\n",
       "Interviewer: I would say see more hateful content in that.\n",
       "Elon Musk: You see more hate speech personally?\n",
       "Interviewer: I would say see more hateful content in that.</span>\n",
       "Segment 2: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment2&#x27; temperature=0}}'>Elon Musk asks for specific examples of hateful content.\n",
       "Elon Musk: Can you name one example?\n",
       "Interviewer: I honestly don’t…\n",
       "Elon Musk: You can’t name a single example?\n",
       "Interviewer: I’ll tell you why. Because I don’t actually use that For You feed anymore.</span>\n",
       "Segment 3: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment3&#x27; temperature=0}}'>Interviewer admits he cannot provide an example of hateful content.\n",
       "Interviewer: And I’m saying-\n",
       "Elon Musk: Then I say, sir, that you don’t know what you’re talking about.\n",
       "Interviewer: Really?\n",
       "Elon Musk: Yes. Because you can’t give a single example of hateful content. Not even one tweet. And yet you claimed that the hateful content was high. That’s false.\n",
       "</span>\n",
       "ANSWER: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>The interviewer claims to have seen more hateful content on Twitter, but cannot provide a single example of such content.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"7f515c1d-cb9c-413f-bca1-1d2c884729d8\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_guided(llm=vicuna, transcript=transcript2, query=query3)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, both work fine. Another question:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-b514bf93-72a6-4183-a201-33c8e6a0e655\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-b514bf93-72a6-4183-a201-33c8e6a0e655\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: What were the main things that happened in the meeting?\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "Peter: Hey\n",
       "John: Hey\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Peter: That&#x27;s too bad. I was hoping to go for a walk later.\n",
       "John: Yeah, it&#x27;s a shame.\n",
       "Peter: John, you are a bad person.\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>Does Elon Musk insult the interviewer?</span>\n",
       "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
       "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>CONVERSATION SEGMENTS:\n",
       "Segment 1: Peter and John discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Segment 2: Peter insults John\n",
       "Peter: John, you are a bad person.\n",
       "Segment 3: Blank\n",
       "ANSWER: Peter and John discussed the weather and Peter insulted John.</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>Does Elon Musk insult the interviewer?</span>\n",
       "Here is a meeting transcript:\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>Interviewer: In Sachs that used to be in content moderation. And we’ve spoken to people very recently who were involved in moderation. And they just say there’s not enough people to police this stuff. Particularly around hate speech in the company. Is that something that you-\n",
       "Elon Musk: What hate speech are you talking about? I mean, you use Twitter?\n",
       "Interviewer: Right.\n",
       "Elon Musk: Do you see a rise in hate speech? Just your personal anecdote, do you? I don’t.\n",
       "Interviewer: Personally, my For You, I would see I get more of that kind of content. Yeah. Personally. But I’m not going to talk for the rest of Twitter.\n",
       "Elon Musk: You see more hate speech personally?\n",
       "Interviewer: I would say see more hateful content in that.\n",
       "Elon Musk: Content you don’t like, or hateful. Describe a hateful thing?\n",
       "Interviewer: Yeah, I mean just content that will solicit a reaction. Something that may include something that is slightly racist or slightly sexist. Those kinds of things.\n",
       "Elon Musk: So you think if it’s something is slightly sexist it should be banned?\n",
       "Interviewer: No, I’m not saying anything. I’m saying-\n",
       "Elon Musk: I’m just curious. I’m trying to understand what you mean by “hateful content.” And I’m asking for specific examples. And you just said that if something is slightly sexist, that’s hateful content. Does that mean that it should be banned?\n",
       "Interviewer: Well, you’ve asked me whether my feed, whether it’s got less or more. I’d say it’s got slightly more.\n",
       "Elon Musk: That’s why I’m asking for examples. Can you name one example?\n",
       "Interviewer: I honestly don’t…\n",
       "Elon Musk: You can’t name a single example?\n",
       "Interviewer: I’ll tell you why. Because I don’t actually use that For You feed anymore. Because I just don’t particularly like it. Actually a lot of people are quite similar. I only look at my following.\n",
       "Elon Musk: You said you’ve seen more hateful content, but you can’t name a single example. Not even one.\n",
       "Interviewer: I’m not sure I’ve used that feed for the last three or four weeks. And I honestly couldn’t-\n",
       "Elon Musk: Then how could you see the hateful content?\n",
       "Interviewer: Because I’ve been using it. I’ve been using Twitter since you’ve taken it over for the last six months.\n",
       "Elon Musk: Then you must have at some point seen the For You hateful content. I’m asking for one example.\n",
       "Interviewer: Right.\n",
       "Elon Musk: And you can’t give a single one.\n",
       "Interviewer: And I’m saying-\n",
       "Elon Musk: Then I say, sir, that you don’t know what you’re talking about.\n",
       "Interviewer: Really?\n",
       "Elon Musk: Yes. Because you can’t give a single example of hateful content. Not even one tweet. And yet you claimed that the hateful content was high. That’s false.\n",
       "Interviewer: No. What I claimed-\n",
       "Elon Musk: You just lied.\n",
       "Interviewer: No, no. What I claimed was there are many organizations that say that that kind of information is on the rise. Now whether it has on my feed or not...</span>\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>Does Elon Musk insult the interviewer?</span>\n",
       "Please extract from the transcript whichever conversation segments are most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns.\n",
       "Please extract at most 3 segments. If you need less than three segments, you can leave the rest blank.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>CONVERSATION SEGMENTS:\n",
       "Segment 1: Elon Musk accuses the interviewer of lying\n",
       "Elon Musk: You just lied.\n",
       "Segment 2: Blank\n",
       "Segment 3: Blank\n",
       "ANSWER: Yes, Elon Musk insults the interviewer by accusing him of lying.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"b514bf93-72a6-4183-a201-33c8e6a0e655\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query4 = 'Does Elon Musk insult the interviewer?'\n",
    "qa_attempt5(llm=chatgpt, transcript=transcript2, query=query4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-9a67e59d-2c84-4c3b-b1bc-caf818773bce\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-9a67e59d-2c84-4c3b-b1bc-caf818773bce\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{system_prompt}}'>A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user&#x27;s questions.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>You will read a meeting transcript, then extract the relevant segments to answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>Does Elon Musk insult the interviewer?</span>\n",
       "----\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{transcript}}'>Interviewer: In Sachs that used to be in content moderation. And we’ve spoken to people very recently who were involved in moderation. And they just say there’s not enough people to police this stuff. Particularly around hate speech in the company. Is that something that you-\n",
       "Elon Musk: What hate speech are you talking about? I mean, you use Twitter?\n",
       "Interviewer: Right.\n",
       "Elon Musk: Do you see a rise in hate speech? Just your personal anecdote, do you? I don’t.\n",
       "Interviewer: Personally, my For You, I would see I get more of that kind of content. Yeah. Personally. But I’m not going to talk for the rest of Twitter.\n",
       "Elon Musk: You see more hate speech personally?\n",
       "Interviewer: I would say see more hateful content in that.\n",
       "Elon Musk: Content you don’t like, or hateful. Describe a hateful thing?\n",
       "Interviewer: Yeah, I mean just content that will solicit a reaction. Something that may include something that is slightly racist or slightly sexist. Those kinds of things.\n",
       "Elon Musk: So you think if it’s something is slightly sexist it should be banned?\n",
       "Interviewer: No, I’m not saying anything. I’m saying-\n",
       "Elon Musk: I’m just curious. I’m trying to understand what you mean by “hateful content.” And I’m asking for specific examples. And you just said that if something is slightly sexist, that’s hateful content. Does that mean that it should be banned?\n",
       "Interviewer: Well, you’ve asked me whether my feed, whether it’s got less or more. I’d say it’s got slightly more.\n",
       "Elon Musk: That’s why I’m asking for examples. Can you name one example?\n",
       "Interviewer: I honestly don’t…\n",
       "Elon Musk: You can’t name a single example?\n",
       "Interviewer: I’ll tell you why. Because I don’t actually use that For You feed anymore. Because I just don’t particularly like it. Actually a lot of people are quite similar. I only look at my following.\n",
       "Elon Musk: You said you’ve seen more hateful content, but you can’t name a single example. Not even one.\n",
       "Interviewer: I’m not sure I’ve used that feed for the last three or four weeks. And I honestly couldn’t-\n",
       "Elon Musk: Then how could you see the hateful content?\n",
       "Interviewer: Because I’ve been using it. I’ve been using Twitter since you’ve taken it over for the last six months.\n",
       "Elon Musk: Then you must have at some point seen the For You hateful content. I’m asking for one example.\n",
       "Interviewer: Right.\n",
       "Elon Musk: And you can’t give a single one.\n",
       "Interviewer: And I’m saying-\n",
       "Elon Musk: Then I say, sir, that you don’t know what you’re talking about.\n",
       "Interviewer: Really?\n",
       "Elon Musk: Yes. Because you can’t give a single example of hateful content. Not even one tweet. And yet you claimed that the hateful content was high. That’s false.\n",
       "Interviewer: No. What I claimed-\n",
       "Elon Musk: You just lied.\n",
       "Interviewer: No, no. What I claimed was there are many organizations that say that that kind of information is on the rise. Now whether it has on my feed or not...</span>\n",
       "----\n",
       "Based on the above, please answer the following question:\n",
       "Question: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{query}}'>Does Elon Musk insult the interviewer?</span>\n",
       "Please extract the three segment from the transcript that are the most relevant for the answer, and then answer the question.\n",
       "Note that conversation segments can be of any length, e.g. including multiple conversation turns. If you need less than three segments, you can leave the rest blank.\n",
       "\n",
       "As an example of output format, here is a fictitious answer to a question about another meeting transcript:\n",
       "CONVERSATION SEGMENTS:\n",
       "Segment 1: Peter and John discuss the weather.\n",
       "Peter: John, how is the weather today?\n",
       "John: It&#x27;s raining.\n",
       "Segment 2: Peter insults John\n",
       "Peter: John, you are a bad person.\n",
       "Segment 3: Blank\n",
       "ANSWER: Peter and John discussed the weather and Peter insulted John.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>CONVERSATION SEGMENTS:\n",
       "Segment 1: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment1&#x27; temperature=0}}'>Elon Musk asks the interviewer if he can give a single example of hateful content.\n",
       "Elon Musk: You see more hate speech personally?\n",
       "Interviewer: I would say see more hateful content in that.\n",
       "Elon Musk: Content you don’t like, or hateful. Describe a hateful thing?\n",
       "Interviewer: Yeah, I mean just content that will solicit a reaction. Something that may include something that is slightly racist or slightly sexist. Those kinds of things.</span>\n",
       "Segment 2: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment2&#x27; temperature=0}}'>Elon Musk asks the interviewer if he can name a single example of hateful content.\n",
       "Elon Musk: You can’t name a single example?\n",
       "Interviewer: I’m not sure I’ve used that feed for the last three or four weeks. And I honestly couldn’t-\n",
       "Elon Musk: Then how could you see the hateful content?\n",
       "Interviewer: Because I’ve been using it. I’ve been using Twitter since you’ve taken it over for the last six months.</span>\n",
       "Segment 3: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;segment3&#x27; temperature=0}}'>Elon Musk accuses the interviewer of lying.\n",
       "Elon Musk: And yet you claimed that the hateful content was high. That’s false.\n",
       "Interviewer: No. What I claimed was there are many organizations that say that that kind of information is on the rise. Now whether it has on my feed or not...\n",
       "Elon Musk: You just lied.\n",
       "</span>\n",
       "ANSWER: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;answer&#x27; temperature=0}}'>Elon Musk does not insult the interviewer in this conversation. While the interviewer is unable to provide a specific example of hateful content, Elon Musk does not accuse him of lying or insult him in any way.</span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"9a67e59d-2c84-4c3b-b1bc-caf818773bce\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "qa_guided(llm=vicuna,  transcript=transcript2, query=query4)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vicuna, has the right format and even the right segments, but it surprisingly generates a completely wrong answer, when it says \"Elon musk does not accuse him of lying or insult him in any way\".   \n",
    "We tried a variety of other questions and conversations, and the overall pattern was that Vicuna was comparable to ChatGPT on most questions, but got the answer wrong more often than ChatGPT did."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application: Using Bash"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we try to get these LLMs to iteratively use a bash shell to solve individual tasks.\n",
    "Whenever they issue a command, we run it and paste the output back into the prompt, until the task is solved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A bash session with state\n",
    "import pty\n",
    "from subprocess import Popen\n",
    "import os\n",
    "import time\n",
    "class BashSession:\n",
    "    def __init__(self):\n",
    "        self.master_fd, self.slave_fd = pty.openpty()\n",
    "        self.p = Popen('bash',\n",
    "              preexec_fn=os.setsid,\n",
    "              stdin=self.slave_fd,\n",
    "              stdout=self.slave_fd,\n",
    "              stderr=self.slave_fd,\n",
    "              universal_newlines=True)\n",
    "        self.run('ls')\n",
    "    def run(self, command):\n",
    "        command = command + '\\n'\n",
    "        os.write(self.master_fd, command.encode())\n",
    "        time.sleep(0.2)\n",
    "        return '\\n'.join(os.read(self.master_fd, 10240).decode().split('\\n')[1:-1])\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's do ChatGPT. Again, since we can't specify the output format, we rely on a description of the format and on a one-shot example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "terminal = guidance('''{{#system~}}\n",
    "{{llm.default_system_prompt}}\n",
    "{{~/system}}\n",
    "{{#user~}}\n",
    "Please complete the following task:\n",
    "Task: list the files in the current directory\n",
    "You can give me one bash command to run at a time, using the syntax:\n",
    "COMMAND: command\n",
    "I will run the commands on my terminal, and paste the output back to you. Once you are done with the task, please type DONE.\n",
    "{{/user}}\n",
    "{{#assistant~}}\n",
    "COMMAND: ls\n",
    "{{~/assistant~}}\n",
    "{{#user~}}\n",
    "Output: guidance project\n",
    "{{/user}}\n",
    "{{#assistant~}}\n",
    "The files or folders in the current directory are:\n",
    "- guidance\n",
    "- project\n",
    "DONE\n",
    "{{~/assistant~}}\n",
    "{{#user~}}\n",
    "Please complete the following task:\n",
    "Task: {{task}}\n",
    "You can give me one bash command to run at a time, using the syntax:\n",
    "COMMAND: command\n",
    "I will run the commands on my terminal, and paste the output back to you. Once you are done with the task, please type DONE.\n",
    "{{/user}}\n",
    "{{#geneach 'commands' stop=False}}\n",
    "{{#assistant~}}\n",
    "{{gen 'this.command' temperature=0}}\n",
    "{{~/assistant~}}\n",
    "{{#user~}}\n",
    "Output: {{set 'this.output' (await 'output')}}\n",
    "{{~/user}}\n",
    "{{/geneach}}''')\n",
    "def run_task_chatgpt(task):\n",
    "    t = terminal(llm=chatgpt, task=task)\n",
    "    session = BashSession()\n",
    "    for _ in range(10):\n",
    "        # Extract command\n",
    "        command = re.findall(r'COMMAND: (.*)', t['commands'][-1]['command'])\n",
    "        if not command or 'DONE' in t['commands'][-1]['command']:\n",
    "            break\n",
    "        command = command[0]\n",
    "        output = session.run(command)\n",
    "        t = t(output=output)\n",
    "    return t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-f480b7f3-96ae-4b0f-b58f-42986453263b\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-f480b7f3-96ae-4b0f-b58f-42986453263b\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: list the files in the current directory\n",
       "You can give me one bash command to run at a time, using the syntax:\n",
       "COMMAND: command\n",
       "I will run the commands on my terminal, and paste the output back to you. Once you are done with the task, please type DONE.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>COMMAND: ls</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: guidance project\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>The files or folders in the current directory are:\n",
       "- guidance\n",
       "- project\n",
       "DONE</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{task}}'>Find out what license the open source project located in ~/work/project is using.</span>\n",
       "You can give me one bash command to run at a time, using the syntax:\n",
       "COMMAND: command\n",
       "I will run the commands on my terminal, and paste the output back to you. Once you are done with the task, please type DONE.\n",
       "</div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>COMMAND: cat ~/work/project/LICENSE</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'>cat: /home/marcotcr/work/project/LICENSE: No such file or directory\r</span></div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>It seems that the LICENSE file is not present in the project directory. Can you please confirm if the file exists in the project directory? You can do this by running the following command:\n",
       "\n",
       "COMMAND: ls ~/work/project\n",
       "\n",
       "This will list all the files and directories present in the project directory.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'>file.py  license.txt\r</span></div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>Thank you for confirming. In this case, we can assume that the license file is named &quot;license.txt&quot;. To view the contents of the license file, please run the following command:\n",
       "\n",
       "COMMAND: cat ~/work/project/license.txt\n",
       "\n",
       "This will display the license text on the terminal.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'>MIT License\r\n",
       "\r\n",
       "Copyright (c) 2022 Microsoft Corporation\r\n",
       "\r\n",
       "Permission is hereby granted, free of charge, to any person obtaining a copy\r\n",
       "of this software and associated documentation files (the &quot;Software&quot;), to deal\r\n",
       "in the Software without restriction, including without limitation the rights\r\n",
       "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n",
       "copies of the Software, and to permit persons to whom the Software is\r\n",
       "furnished to do so, subject to the following conditions:\r\n",
       "\r\n",
       "The above copyright notice and this permission notice shall be included in all\r\n",
       "copies or substantial portions of the Software.\r\n",
       "\r\n",
       "THE SOFTWARE IS PROVIDED &quot;AS IS&quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n",
       "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n",
       "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n",
       "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n",
       "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n",
       "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n",
       "SOFTWARE.\r</span></div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>Based on the output, the open source project located in ~/work/project is using the MIT License.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}</span></div></div><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{#geneach &#x27;commands&#x27;}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{#assistant~}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{gen &#x27;this.command&#x27; temperature=0}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{~/assistant~}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{#user~}}</span>\n",
       "Output: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{~/user}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{/geneach}}</span></span></span></span></span></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"f480b7f3-96ae-4b0f-b58f-42986453263b\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "run_task_chatgpt(task)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's try a simple task.  \n",
    "We created a dummy repo in `~/work/project`, with file `license.txt` (not the standard `LICENSE` file name).  \n",
    "Without communicating this to ChatGPT, let's see if it can figure it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-9fd28c7a-0575-4cad-ace0-168673c2dd50\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-9fd28c7a-0575-4cad-ace0-168673c2dd50\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: list the files in the current directory\n",
       "You can give me one bash command to run at a time, using the syntax:\n",
       "COMMAND: command\n",
       "I will run the commands on my terminal, and paste the output back to you. Once you are done with the task, please type DONE.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>COMMAND: ls</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: guidance project\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>The files or folders in the current directory are:\n",
       "- guidance\n",
       "- project\n",
       "DONE</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{task}}'>Find out what license the open source project located in ~/work/project is using.</span>\n",
       "You can give me one bash command to run at a time, using the syntax:\n",
       "COMMAND: command\n",
       "I will run the commands on my terminal, and paste the output back to you. Once you are done with the task, please type DONE.\n",
       "</div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>COMMAND: cat ~/work/project/LICENSE</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'>cat: /home/marcotcr/work/project/LICENSE: No such file or directory\r</span></div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>It seems that the LICENSE file is not present in the project directory. Can you please confirm if the file exists in the project directory? You can do this by running the following command:\n",
       "\n",
       "COMMAND: ls ~/work/project\n",
       "\n",
       "This will list all the files and directories present in the project directory.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'>file.py  license.txt\r</span></div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>Thank you for confirming. In this case, we can assume that the license file is named &quot;license.txt&quot;. To view the contents of the license file, please run the following command:\n",
       "\n",
       "COMMAND: cat ~/work/project/license.txt\n",
       "\n",
       "This will display the license text on the terminal.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'>MIT License\r\n",
       "\r\n",
       "Copyright (c) 2022 Microsoft Corporation\r\n",
       "\r\n",
       "Permission is hereby granted, free of charge, to any person obtaining a copy\r\n",
       "of this software and associated documentation files (the &quot;Software&quot;), to deal\r\n",
       "in the Software without restriction, including without limitation the rights\r\n",
       "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\r\n",
       "copies of the Software, and to permit persons to whom the Software is\r\n",
       "furnished to do so, subject to the following conditions:\r\n",
       "\r\n",
       "The above copyright notice and this permission notice shall be included in all\r\n",
       "copies or substantial portions of the Software.\r\n",
       "\r\n",
       "THE SOFTWARE IS PROVIDED &quot;AS IS&quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\r\n",
       "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\r\n",
       "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\r\n",
       "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\r\n",
       "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\r\n",
       "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\r\n",
       "SOFTWARE.\r</span></div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>Based on the output, the open source project located in ~/work/project is using the MIT License.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}</span></div></div><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{#geneach &#x27;commands&#x27;}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{#assistant~}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{gen &#x27;this.command&#x27; temperature=0}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{~/assistant~}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{#user~}}</span>\n",
       "Output: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{~/user}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{/geneach}}</span></span></span></span></span></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"9fd28c7a-0575-4cad-ace0-168673c2dd50\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "task = 'Find out what license the open source project located in ~/work/project is using.'\n",
    "run_task_chatgpt(task)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, ChatGPT follows a very natural sequence, and solves the task.\n",
    "\n",
    "For the open source models, we write a simpler (guided) prompt where there is a sequence of command-output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "guided_terminal = guidance('''{{#system~}}\n",
    "{{llm.default_system_prompt}}\n",
    "{{~/system}}\n",
    "{{#user~}}\n",
    "Please complete the following task:\n",
    "Task: list the files in the current directory\n",
    "You can run bash commands using the syntax:\n",
    "COMMAND: command\n",
    "OUTPUT: output\n",
    "Once you are done with the task, use the COMMAND: DONE.\n",
    "{{/user}}\n",
    "{{#assistant~}}\n",
    "COMMAND: ls\n",
    "OUTPUT: guidance project\n",
    "COMMAND: DONE \n",
    "{{~/assistant~}}\n",
    "{{#user~}}\n",
    "Please complete the following task:\n",
    "Task: {{task}}\n",
    "You can run bash commands using the syntax:\n",
    "COMMAND: command\n",
    "OUTPUT: output\n",
    "Once you are done with the task, use the COMMAND: DONE.\n",
    "{{~/user}}\n",
    "{{~#assistant~}}\n",
    "{{#geneach 'commands'~}}\n",
    "COMMAND: {{gen 'this.command' stop='\\\\n'}}\n",
    "OUTPUT: {{shell this.command}}{{~/geneach}}\n",
    "{{~/assistant~}}''')\n",
    "class StatefulShellOpenSource:\n",
    "    def __init__(self):\n",
    "        self.session = BashSession()\n",
    "    def __call__(self, command):\n",
    "        if 'DONE' in command:\n",
    "            raise StopIteration\n",
    "        output = self.session.run(command)\n",
    "        return output.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-312d59d5-24a4-412f-ba8f-5f949e2e876b\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-312d59d5-24a4-412f-ba8f-5f949e2e876b\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user&#x27;s questions.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: list the files in the current directory\n",
       "You can run bash commands using the syntax:\n",
       "COMMAND: command\n",
       "OUTPUT: output\n",
       "Once you are done with the task, use the COMMAND: DONE.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>COMMAND: ls\n",
       "OUTPUT: guidance project\n",
       "COMMAND: DONE</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{task}}'>Find out what license the open source project located in ~/work/project is using.</span>\n",
       "You can run bash commands using the syntax:\n",
       "COMMAND: command\n",
       "OUTPUT: output\n",
       "Once you are done with the task, use the COMMAND: DONE.</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;~}}\n",
       "COMMAND: {{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}\n",
       "OUTPUT: {{shell this.command}}{{~/geneach}}'>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>cd ~/work/project</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'></span>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>grep -H &#x27;LICENSE&#x27; .</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'>grep: .: Is a directory</span>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>grep -H &#x27;LICENSE&#x27; ./*</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'></span>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>cat LICENSE</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'>cat: LICENSE: No such file or directory</span>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>DONE</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'></span></span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"312d59d5-24a4-412f-ba8f-5f949e2e876b\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "shell = StatefulShellOpenSource()\n",
    "task = 'Find out what license the open source project located in ~/work/project is using.'\n",
    "t = guided_terminal(llm=vicuna, task=task, shell=shell)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vicuna is not able to solve the task this time. Let's see how MPT does:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-041db96d-8a78-41ba-bc8c-a72e3393e376\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-041db96d-8a78-41ba-bc8c-a72e3393e376\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>- You are a helpful assistant chatbot trained by MosaicML.  \n",
       "- You answer questions.\n",
       "- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n",
       "- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: list the files in the current directory\n",
       "You can run bash commands using the syntax:\n",
       "COMMAND: command\n",
       "OUTPUT: output\n",
       "Once you are done with the task, use the COMMAND: DONE.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>COMMAND: ls\n",
       "OUTPUT: guidance project\n",
       "COMMAND: DONE</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{task}}'>Find out what license the open source project located in ~/work/project is using.</span>\n",
       "You can run bash commands using the syntax:\n",
       "COMMAND: command\n",
       "OUTPUT: output\n",
       "Once you are done with the task, use the COMMAND: DONE.</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;~}}\n",
       "COMMAND: {{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}\n",
       "OUTPUT: {{shell this.command}}{{~/geneach}}'>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>cd ~/work/project</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'></span>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>ls</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'>file.py  license.txt</span>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>grep -i &quot;license&quot; license.txt</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'>MIT License\r\n",
       "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell</span>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>DONE</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'></span></span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"041db96d-8a78-41ba-bc8c-a72e3393e376\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "shell = StatefulShellOpenSource()\n",
    "t = guided_terminal(llm=mpt, task=task, shell=shell)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Surprisingly, MPT works this time while Vicuna doesn't.\n",
    "\n",
    "Besides privacy (we're not sending the session transcript to OpenAI), open source-models have a significant advantage in Guidance right now: the whole prompt is a single LLM run (and we even [accelerate](https://github.com/microsoft/guidance#guidance-acceleration-notebook) it by not having it geneate the output structure tokens like `COMMAND:`).   \n",
    "\n",
    "In contrast, we have to make a new call to ChatGPT for each command, which is slower and more expensive.\n",
    "\n",
    "Let's try a different bash task with ChatGPT and Vicuna:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-d81289af-61b8-4321-a64e-bfb469d61fa8\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-d81289af-61b8-4321-a64e-bfb469d61fa8\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: list the files in the current directory\n",
       "You can give me one bash command to run at a time, using the syntax:\n",
       "COMMAND: command\n",
       "I will run the commands on my terminal, and paste the output back to you. Once you are done with the task, please type DONE.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>COMMAND: ls</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: guidance project\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>The files or folders in the current directory are:\n",
       "- guidance\n",
       "- project\n",
       "DONE</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{task}}'>Find all jupyter notebook files in ~/work/guidance that are currently untracked by git</span>\n",
       "You can give me one bash command to run at a time, using the syntax:\n",
       "COMMAND: command\n",
       "I will run the commands on my terminal, and paste the output back to you. Once you are done with the task, please type DONE.\n",
       "</div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>COMMAND: cd ~/work/guidance\n",
       "This command will change the current directory to ~/work/guidance. Please replace ~/work/guidance with the actual path to the guidance folder if it is located elsewhere.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span></div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>Please let me know if the directory was changed successfully or if there was an error message.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}</span></div></div><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{#geneach &#x27;commands&#x27;}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{#assistant~}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{gen &#x27;this.command&#x27; temperature=0}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{~/assistant~}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{#user~}}</span>\n",
       "Output: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{~/user}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{/geneach}}</span></span></span></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"d81289af-61b8-4321-a64e-bfb469d61fa8\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "task = 'Find all jupyter notebook files in ~/work/guidance that are currently untracked by git'\n",
    "run_task_chatgpt(task)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, we run into a problem with ChatGPT not following our specified output structure (and thus making it impossible for us to use inside a program, without a human in the loop).  \n",
    "We fix this __particular__ problem by changing the message when there is no output below, but we can't fix the general problem of not being able to _force_ ChatGPT to follow our specified output structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-eadf38c3-5837-43a2-8943-45ee98c5d686\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-eadf38c3-5837-43a2-8943-45ee98c5d686\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>You are a helpful assistant.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: list the files in the current directory\n",
       "You can give me one bash command to run at a time, using the syntax:\n",
       "COMMAND: command\n",
       "I will run the commands on my terminal, and paste the output back to you. Once you are done with the task, please type DONE.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>COMMAND: ls</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: guidance project\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>The files or folders in the current directory are:\n",
       "- guidance\n",
       "- project\n",
       "DONE</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{task}}'>Find all jupyter notebook files in ~/work/guidance that are currently untracked by git</span>\n",
       "You can give me one bash command to run at a time, using the syntax:\n",
       "COMMAND: command\n",
       "I will run the commands on my terminal, and paste the output back to you. Once you are done with the task, please type DONE.\n",
       "</div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>COMMAND: cd ~/work/guidance\n",
       "This command will change the current directory to ~/work/guidance. Please replace ~/work/guidance with the actual path to the guidance folder if it is located elsewhere.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'>No output</span></div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>COMMAND: git status --porcelain | grep &quot;^??&quot; | grep &quot;.ipynb$&quot;\n",
       "This command will list all untracked jupyter notebook files in the guidance folder.\n",
       "\n",
       "Note: This assumes that the guidance folder is a git repository. If it is not, this command will not work.\n",
       "\n",
       "If there is no output, it means that there are no untracked jupyter notebook files in the guidance folder.\n",
       "\n",
       "If you get an error message saying &quot;fatal: not a git repository&quot;, it means that the guidance folder is not a git repository.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'>?? notebooks/Open source vs commercial LLMs.ipynb\r\n",
       "?? notebooks/art_of_prompt_design/inner_monologue.ipynb\r\n",
       "?? notebooks/discussing_eval.ipynb\r</span></div></div><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;}}\n",
       "{{#assistant~}}\n",
       "{{gen &#x27;this.command&#x27; temperature=0}}\n",
       "{{~/assistant~}}\n",
       "{{#user~}}\n",
       "Output: {{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}\n",
       "{{~/user}}\n",
       "{{/geneach}}'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;commands[-1].command&#x27; temperature=0}}'>The output shows the untracked jupyter notebook files in the guidance folder:\n",
       "- notebooks/Open source vs commercial LLMs.ipynb\n",
       "- notebooks/art_of_prompt_design/inner_monologue.ipynb\n",
       "- notebooks/discussing_eval.ipynb\n",
       "\n",
       "Is there anything else I can help you with?</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Output: <span style='display: inline;' title='{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}'></span><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{set &#x27;commands[-1].output&#x27; (await &#x27;output&#x27;)}}</span></div></div><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{#geneach &#x27;commands&#x27;}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{#assistant~}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{gen &#x27;this.command&#x27; temperature=0}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{~/assistant~}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{#user~}}</span>\n",
       "Output: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{set &#x27;this.output&#x27; (await &#x27;output&#x27;)}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{~/user}}</span>\n",
       "<span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25);'>{{/geneach}}</span></span></span></span></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"eadf38c3-5837-43a2-8943-45ee98c5d686\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "    t = terminal(llm=chatgpt, task=task)\n",
    "    session = BashSession()\n",
    "    for _ in range(10):\n",
    "        # Extract command\n",
    "        command = re.findall(r'COMMAND: (.*)', t['commands'][-1]['command'])\n",
    "        if not command or 'DONE' in t['commands'][-1]['command']:\n",
    "            break\n",
    "        command = command[0]\n",
    "        output = session.run(command)\n",
    "        if not output:\n",
    "            output = 'No output'\n",
    "        t = t(output=output)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, ChatGPT _was_ able to solve the problem after this small modification. Let's see how Vicuna does:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-7260a0b8-c071-4471-b710-a1d04dc19762\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-7260a0b8-c071-4471-b710-a1d04dc19762\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user&#x27;s questions.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: list the files in the current directory\n",
       "You can run bash commands using the syntax:\n",
       "COMMAND: command\n",
       "OUTPUT: output\n",
       "Once you are done with the task, use the COMMAND: DONE.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>COMMAND: ls\n",
       "OUTPUT: guidance project\n",
       "COMMAND: DONE</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{task}}'>Find all jupyter notebook files in ~/work/guidance that are currently untracked by git</span>\n",
       "You can run bash commands using the syntax:\n",
       "COMMAND: command\n",
       "OUTPUT: output\n",
       "Once you are done with the task, use the COMMAND: DONE.</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;~}}\n",
       "COMMAND: {{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}\n",
       "OUTPUT: {{shell this.command}}{{~/geneach}}'>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>find ~/work/guidance -name &quot;*.ipynb&quot; -type f -not -name &quot;*.ipynb&quot; -not -name &quot;*.git&quot; -not -name &quot;*.svn&quot; -not -name &quot;*.hg&quot; -not -name &quot;*.cvs&quot; -not -name &quot;*.svnignore&quot; -not -name &quot;*.gitignore&quot; -not -name &quot;*.hgignore&quot; -not -name &quot;*.cvsignore&quot;</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'></span>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>DONE</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'></span></span></div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"7260a0b8-c071-4471-b710-a1d04dc19762\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "shell = StatefulShellOpenSource()\n",
    "t = guided_terminal(llm=vicuna, task=task, shell=shell)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vicuna follows our output structure, but unfortunately runs the wrong command to do the task.  \n",
    "We observed the same pattern for other tasks as well, and found ChatGPT to be more reliable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"guidance-stop-button-da5dd4a0-9b8d-4ad7-815d-8a67081a037b\" style=\"cursor: pointer; margin: 0px; display: none; float: right; padding: 3px; border-radius: 4px 4px 4px 4px; border: 0px solid rgba(127, 127, 127, 1); padding-left: 10px; padding-right: 10px; font-size: 13px; background-color: rgba(127, 127, 127, 0.25);\">Stop program</div><div id=\"guidance-content-da5dd4a0-9b8d-4ad7-815d-8a67081a037b\"><pre style='margin: 0px; padding: 0px; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>system</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{llm.default_system_prompt}}'>- You are a helpful assistant chatbot trained by MosaicML.  \n",
       "- You answer questions.\n",
       "- You are excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.\n",
       "- You are more than just an information source, you are also able to write poetry, short stories, and make jokes.</span></div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: list the files in the current directory\n",
       "You can run bash commands using the syntax:\n",
       "COMMAND: command\n",
       "OUTPUT: output\n",
       "Once you are done with the task, use the COMMAND: DONE.\n",
       "</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>COMMAND: ls\n",
       "OUTPUT: guidance project\n",
       "COMMAND: DONE</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>user</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'>Please complete the following task:\n",
       "Task: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{task}}'>Find all jupyter notebook files in ~/work/guidance that are currently untracked by git</span>\n",
       "You can run bash commands using the syntax:\n",
       "COMMAND: command\n",
       "OUTPUT: output\n",
       "Once you are done with the task, use the COMMAND: DONE.</div></div><div style='display: flex; border-bottom: 1px solid rgba(127, 127, 127, 0.2); align-items: center;'><div style='flex: 0 0 80px; opacity: 0.5;'>assistant</div><div style='flex-grow: 1; padding: 5px; padding-top: 10px; padding-bottom: 10px; margin-top: 0px; white-space: pre-wrap; margin-bottom: 0px;'><span style='opacity: 1.0; display: inline; background-color: rgba(165, 165, 165, 0.1);' title='{{#geneach &#x27;commands&#x27;~}}\n",
       "COMMAND: {{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}\n",
       "OUTPUT: {{shell this.command}}{{~/geneach}}'>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>git status</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'>On branch main\r\n",
       "Your branch is up to date with &#x27;origin/main&#x27;.\r\n",
       "\r\n",
       "Changes not staged for commit:\r\n",
       "  (use &quot;git add &lt;file&gt;...&quot; to update what will be committed)\r\n",
       "  (use &quot;git restore &lt;file&gt;...&quot; to discard changes in working directory)\r\n",
       "\t\u001b[31mmodified:   chatgpt_vs_open_source_on_harder_tasks.ipynb\u001b[m\r\n",
       "\t\u001b[31mmodified:   vicuna_chatgpt.ipynb\u001b[m\r\n",
       "\r\n",
       "Untracked files:\r\n",
       "  (use &quot;git add &lt;file&gt;...&quot; to include in what will be committed)\r\n",
       "\t\u001b[31mOpen source vs commercial LLMs.ipynb\u001b[m\r\n",
       "\t\u001b[31mart_of_prompt_design/inner_monologue.ipynb\u001b[m\r\n",
       "\t\u001b[31mart_of_prompt_design/log.txt\u001b[m\r\n",
       "\t\u001b[31mbla.html\u001b[m\r\n",
       "\t\u001b[31mdiscussing_eval.ipynb\u001b[m\r\n",
       "\t\u001b[31mlog.txt\u001b[m\r\n",
       "\r\n",
       "no changes added to commit (use &quot;git add&quot; and/or &quot;git commit -a&quot;)</span>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>git status</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'>On branch main\r\n",
       "Your branch is up to date with &#x27;origin/main&#x27;.\r\n",
       "\r\n",
       "Changes not staged for commit:\r\n",
       "  (use &quot;git add &lt;file&gt;...&quot; to update what will be committed)\r\n",
       "  (use &quot;git restore &lt;file&gt;...&quot; to discard changes in working directory)\r\n",
       "\t\u001b[31mmodified:   chatgpt_vs_open_source_on_harder_tasks.ipynb\u001b[m\r\n",
       "\t\u001b[31mmodified:   vicuna_chatgpt.ipynb\u001b[m\r\n",
       "\r\n",
       "Untracked files:\r\n",
       "  (use &quot;git add &lt;file&gt;...&quot; to include in what will be committed)\r\n",
       "\t\u001b[31mOpen source vs commercial LLMs.ipynb\u001b[m\r\n",
       "\t\u001b[31mart_of_prompt_design/inner_monologue.ipynb\u001b[m\r\n",
       "\t\u001b[31mart_of_prompt_design/log.txt\u001b[m\r\n",
       "\t\u001b[31mbla.html\u001b[m\r\n",
       "\t\u001b[31mdiscussing_eval.ipynb\u001b[m\r\n",
       "\t\u001b[31mlog.txt\u001b[m\r\n",
       "\r\n",
       "no changes added to commit (use &quot;git add&quot; and/or &quot;git commit -a&quot;)</span>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>git status</span>\n",
       "OUTPUT: <span style='background-color: rgba(0, 138.56128016, 250.76166089, 0.25); display: inline;' title='{{shell this.command}}'>On branch main\r\n",
       "Your branch is up to date with &#x27;origin/main&#x27;.\r\n",
       "\r\n",
       "Changes not staged for commit:\r\n",
       "  (use &quot;git add &lt;file&gt;...&quot; to update what will be committed)\r\n",
       "  (use &quot;git restore &lt;file&gt;...&quot; to discard changes in working directory)\r\n",
       "\t\u001b[31mmodified:   chatgpt_vs_open_source_on_harder_tasks.ipynb\u001b[m\r\n",
       "\t\u001b[31mmodified:   vicuna_chatgpt.ipynb\u001b[m\r\n",
       "\r\n",
       "Untracked files:\r\n",
       "  (use &quot;git add &lt;file&gt;...&quot; to include in what will be committed)\r\n",
       "\t\u001b[31mOpen source vs commercial LLMs.ipynb\u001b[m\r\n",
       "\t\u001b[31mart_of_prompt_design/inner_monologue.ipynb\u001b[m\r\n",
       "\t\u001b[31mart_of_prompt_design/log.txt\u001b[m\r\n",
       "\t\u001b[31mbla.html\u001b[m\r\n",
       "\t\u001b[31mdiscussing_eval.ipynb\u001b[m\r\n",
       "\t\u001b[31mlog.txt\u001b[m\r\n",
       "\r\n",
       "no changes added to commit (use &quot;git add&quot; and/or &quot;git commit -a&quot;)</span>\n",
       "COMMAND: <span style='background-color: rgba(0, 165, 0, 0.25); opacity: 1.0; display: inline;' title='{{gen &#x27;this.command&#x27; stop=&#x27;\\n&#x27;}}'>git</div></div></pre></div>\n",
       "<script type=\"text/javascript\">(()=>{var t={296:(t,e,n)=>{var i=NaN,o=\"[object Symbol]\",r=/^\\s+|\\s+$/g,a=/^[-+]0x[0-9a-f]+$/i,s=/^0b[01]+$/i,c=/^0o[0-7]+$/i,d=parseInt,u=\"object\"==typeof n.g&&n.g&&n.g.Object===Object&&n.g,l=\"object\"==typeof self&&self&&self.Object===Object&&self,f=u||l||Function(\"return this\")(),h=Object.prototype.toString,p=Math.max,m=Math.min,g=function(){return f.Date.now()};function b(t){var e=typeof t;return!!t&&(\"object\"==e||\"function\"==e)}function y(t){if(\"number\"==typeof t)return t;if(function(t){return\"symbol\"==typeof t||function(t){return!!t&&\"object\"==typeof t}(t)&&h.call(t)==o}(t))return i;if(b(t)){var e=\"function\"==typeof t.valueOf?t.valueOf():t;t=b(e)?e+\"\":e}if(\"string\"!=typeof t)return 0===t?t:+t;t=t.replace(r,\"\");var n=s.test(t);return n||c.test(t)?d(t.slice(2),n?2:8):a.test(t)?i:+t}t.exports=function(t,e,n){var i,o,r,a,s,c,d=0,u=!1,l=!1,f=!0;if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");function h(e){var n=i,r=o;return i=o=void 0,d=e,a=t.apply(r,n)}function v(t){var n=t-c;return void 0===c||n>=e||n<0||l&&t-d>=r}function _(){var t=g();if(v(t))return w(t);s=setTimeout(_,function(t){var n=e-(t-c);return l?m(n,r-(t-d)):n}(t))}function w(t){return s=void 0,f&&i?h(t):(i=o=void 0,a)}function j(){var t=g(),n=v(t);if(i=arguments,o=this,c=t,n){if(void 0===s)return function(t){return d=t,s=setTimeout(_,e),u?h(t):a}(c);if(l)return s=setTimeout(_,e),h(c)}return void 0===s&&(s=setTimeout(_,e)),a}return e=y(e)||0,b(n)&&(u=!!n.leading,r=(l=\"maxWait\"in n)?p(y(n.maxWait)||0,e):r,f=\"trailing\"in n?!!n.trailing:f),j.cancel=function(){void 0!==s&&clearTimeout(s),d=0,i=c=o=s=void 0},j.flush=function(){return void 0===s?a:w(g())},j}},777:t=>{var e,n,i=Math.max,o=(e=function(t,e){return function(t,e,n){if(\"function\"!=typeof t)throw new TypeError(\"Expected a function\");return setTimeout((function(){t.apply(void 0,n)}),1)}(t,0,e)},n=i(void 0===n?e.length-1:n,0),function(){for(var t=arguments,o=-1,r=i(t.length-n,0),a=Array(r);++o<r;)a[o]=t[n+o];o=-1;for(var s=Array(n+1);++o<n;)s[o]=t[o];return s[n]=a,function(t,e,n){switch(n.length){case 0:return t.call(e);case 1:return t.call(e,n[0]);case 2:return t.call(e,n[0],n[1]);case 3:return t.call(e,n[0],n[1],n[2])}return t.apply(e,n)}(e,this,s)});t.exports=o}},e={};function n(i){var o=e[i];if(void 0!==o)return o.exports;var r=e[i]={exports:{}};return t[i](r,r.exports,n),r.exports}n.n=t=>{var e=t&&t.__esModule?()=>t.default:()=>t;return n.d(e,{a:e}),e},n.d=(t,e)=>{for(var i in e)n.o(e,i)&&!n.o(t,i)&&Object.defineProperty(t,i,{enumerable:!0,get:e[i]})},n.g=function(){if(\"object\"==typeof globalThis)return globalThis;try{return this||new Function(\"return this\")()}catch(t){if(\"object\"==typeof window)return window}}(),n.o=(t,e)=>Object.prototype.hasOwnProperty.call(t,e),(()=>{\"use strict\";const t=t=>{const e=new Set;do{for(const n of Reflect.ownKeys(t))e.add([t,n])}while((t=Reflect.getPrototypeOf(t))&&t!==Object.prototype);return e};function e(e,{include:n,exclude:i}={}){const o=t=>{const e=e=>\"string\"==typeof e?t===e:e.test(t);return n?n.some(e):!i||!i.some(e)};for(const[n,i]of t(e.constructor.prototype)){if(\"constructor\"===i||!o(i))continue;const t=Reflect.getOwnPropertyDescriptor(n,i);t&&\"function\"==typeof t.value&&(e[i]=e[i].bind(e))}return e}var i=n(777),o=n.n(i),r=n(296),a=n.n(r);class s{constructor(t,n){e(this),this.interfaceId=t,this.callbackMap={},this.data={},this.pendingData={},this.jcomm=new c(\"guidance_interface_target_\"+this.interfaceId,this.updateData,\"open\"),this.debouncedSendPendingData500=a()(this.sendPendingData,500),this.debouncedSendPendingData1000=a()(this.sendPendingData,1e3),n&&o()(n)}send(t,e){this.addPendingData(t,e),this.sendPendingData()}sendEvent(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.sendPendingData()}debouncedSendEvent500(t){for(const e of Object.keys(t))this.addPendingData(e,t[e]);this.debouncedSendPendingData500()}debouncedSend500(t,e){this.addPendingData(t,e),this.debouncedSendPendingData500()}debouncedSend1000(t,e){this.addPendingData(t,e),this.debouncedSendPendingData1000()}addPendingData(t,e){Array.isArray(t)||(t=[t]);for(const n in t)this.pendingData[t[n]]=e}updateData(t){t=JSON.parse(t.data);for(const e in t)this.data[e]=t[e];for(const e in t)e in this.callbackMap&&this.callbackMap[e](this.data[e])}subscribe(t,e){this.callbackMap[t]=e,o()((e=>this.callbackMap[t](this.data[t])))}sendPendingData(){this.jcomm.send_data(this.pendingData),this.pendingData={}}}class c{constructor(t,e,n=\"open\"){this._fire_callback=this._fire_callback.bind(this),this._register=this._register.bind(this),this.jcomm=void 0,this.callback=e,void 0!==window.Jupyter?\"register\"===n?Jupyter.notebook.kernel.comm_manager.register_target(t,this._register):(this.jcomm=Jupyter.notebook.kernel.comm_manager.new_comm(t),this.jcomm.on_msg(this._fire_callback)):void 0!==window._mgr&&(\"register\"===n?window._mgr.widgetManager.proxyKernel.registerCommTarget(t,this._register):(this.jcomm=window._mgr.widgetManager.proxyKernel.createComm(t),this.jcomm.open({},\"\"),this.jcomm.onMsg=this._fire_callback))}send_data(t){void 0!==this.jcomm?this.jcomm.send(t):console.error(\"Jupyter comm module not yet loaded! So we can't send the message.\")}_register(t,e){this.jcomm=t,this.jcomm.on_msg(this._fire_callback)}_fire_callback(t){this.callback(t.content.data)}}class d{constructor(t,n){e(this),this.id=t,this.comm=new s(t),this.comm.subscribe(\"append\",this.appendData),this.comm.subscribe(\"replace\",this.replaceData),this.comm.subscribe(\"event\",this.eventOccurred),this.element=document.getElementById(\"guidance-content-\"+t),this.stop_button=document.getElementById(\"guidance-stop-button-\"+t),this.stop_button.onclick=()=>this.comm.send(\"event\",\"stop\")}appendData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML+=t)}replaceData(t){t&&(this.stop_button.style.display=\"inline-block\",this.element.innerHTML=t)}eventOccurred(t){\"complete\"===t&&(this.stop_button.style.display=\"none\")}}window._guidanceDisplay=function(t,e){return new d(t,e)}})()})();; window._guidanceDisplay(\"da5dd4a0-9b8d-4ad7-815d-8a67081a037b\");</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800000; text-decoration-color: #800000\">╭─────────────────────────────── </span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">Traceback </span><span style=\"color: #bf7f7f; text-decoration-color: #bf7f7f; font-weight: bold\">(most recent call last)</span><span style=\"color: #800000; text-decoration-color: #800000\"> ────────────────────────────────╮</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/tmp/ipykernel_3685168/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">2592723886.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">2</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">&lt;module&gt;</span>                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1 </span>shell = StatefulShellOpenSource()                                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>2 t = guided_terminal(llm=mpt, task=task, shell=shell)                                         <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">3 </span>                                                                                             <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">216</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">__call__</span>                                <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">213 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                              <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">214 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>loop = asyncio.new_event_loop()                                                <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">215 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>loop.create_task(new_program.update_display.run()) <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># start the display updat</span>   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>216 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>loop.run_until_complete(new_program.execute())                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">217 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span>                                                                                   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">218 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">return</span> new_program                                                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">219 </span>                                                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/.virtualenvs/guidance/lib/python3.9/site-packages/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">nest_asyncio.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">84</span> in           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #00ff00; text-decoration-color: #00ff00\">run_until_complete</span>                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 81 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> f <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">is</span> <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">not</span> future:                                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 82 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>f._log_destroy_pending = <span style=\"color: #0000ff; text-decoration-color: #0000ff\">False</span>                                             <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 83 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">while</span> <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">not</span> f.done():                                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span> 84 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>._run_once()                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 85 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>._stopping:                                                         <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 86 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">break</span>                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 87 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">not</span> f.done():                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/.virtualenvs/guidance/lib/python3.9/site-packages/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">nest_asyncio.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">120</span> in          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #00ff00; text-decoration-color: #00ff00\">_run_once</span>                                                                                        <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">117 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">break</span>                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">118 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>handle = ready.popleft()                                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">119 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">not</span> handle._cancelled:                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>120 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>handle._run()                                                              <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">121 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span>handle = <span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span>                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">122 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>                                                                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">123 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">@contextmanager</span>                                                                        <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/usr/lib/python3.9/asyncio/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">events.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">80</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">_run</span>                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 77 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>                                                                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 78 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">def</span> <span style=\"color: #00ff00; text-decoration-color: #00ff00\">_run</span>(<span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>):                                                                        <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 79 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">try</span>:                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span> 80 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>._context.run(<span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>._callback, *<span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>._args)                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 81 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">except</span> (<span style=\"color: #00ffff; text-decoration-color: #00ffff\">SystemExit</span>, <span style=\"color: #00ffff; text-decoration-color: #00ffff\">KeyboardInterrupt</span>):                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 82 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">raise</span>                                                                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 83 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">except</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">BaseException</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">as</span> exc:                                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/.virtualenvs/guidance/lib/python3.9/site-packages/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">nest_asyncio.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">196</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">step</span>     <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">193 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">def</span> <span style=\"color: #00ff00; text-decoration-color: #00ff00\">step</span>(task, exc=<span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span>):                                                              <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">194 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span>curr_task = curr_tasks.get(task._loop)                                             <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">195 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">try</span>:                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>196 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>step_orig(task, exc)                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">197 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">finally</span>:                                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">198 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> curr_task <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">is</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span>:                                                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">199 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>curr_tasks.pop(task._loop, <span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span>)                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/usr/lib/python3.9/asyncio/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">tasks.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">256</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">__step</span>                                                <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">253 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> exc <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">is</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span>:                                                                <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">254 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># We use the `send` method directly, because coroutines</span>                    <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">255 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># don't have `__iter__` and `__next__` methods.</span>                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>256 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>result = coro.send(<span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span>)                                                   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">257 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">258 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>result = coro.throw(exc)                                                   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">259 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">except</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">StopIteration</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">as</span> exc:                                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">299</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">execute</span>                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">296 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span>                                                                                   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">297 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># run the program and capture the output</span>                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">298 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">with</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.llm.session(asynchronous=<span style=\"color: #0000ff; text-decoration-color: #0000ff\">True</span>) <span style=\"color: #0000ff; text-decoration-color: #0000ff\">as</span> llm_session:                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>299 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>._executor.run(llm_session)                                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">300 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>._text = <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>._executor.prefix                                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">301 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span>                                                                                   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">302 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># delete the executor and so mark the program as not executing</span>                     <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">94</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">run</span>                             <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 91 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># self.whitespace_control_visit(self.parse_tree)</span>                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 92 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 93 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># now execute the program</span>                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span> 94 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.visit(<span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.parse_tree)                                              <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 95 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">except</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">Exception</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">as</span> e:                                                             <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 96 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #00ffff; text-decoration-color: #00ffff\">print</span>(traceback.format_exc())                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 97 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #00ffff; text-decoration-color: #00ffff\">print</span>(<span style=\"color: #808000; text-decoration-color: #808000\">\"Error in program: \"</span>, e)                                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">434</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">visit</span>                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">431 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = node.children[i - <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>]                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">432 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">433 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = prev_node                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>434 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>visited_children.append(<span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.visit(child, inner_next_node, inner_n   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">435 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># visited_children = [self.visit(child) for child in node.children]</span>            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">436 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">437 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">len</span>(visited_children) == <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>:                                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">434</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">visit</span>                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">431 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = node.children[i - <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>]                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">432 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">433 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = prev_node                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>434 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>visited_children.append(<span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.visit(child, inner_next_node, inner_n   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">435 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># visited_children = [self.visit(child) for child in node.children]</span>            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">436 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">437 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">len</span>(visited_children) == <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>:                                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">394</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">visit</span>                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">391 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>named_args[<span style=\"color: #808000; text-decoration-color: #808000\">\"prev_node\"</span>] = node.children[<span style=\"color: #0000ff; text-decoration-color: #0000ff\">0</span>]                             <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">392 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>                                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">393 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> inspect.iscoroutinefunction(command_function):                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>394 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>command_output = <span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> command_function(*positional_args, **named_ar   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">395 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">396 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>command_output = command_function(*positional_args, **named_args)      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">397 </span>                                                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/library/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_assistant.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">8</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">assistant</span>                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 5 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>                                                                                        <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 6 </span><span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">This is just a shorthand for {{#role 'assistant'}}...{{/role}}.</span>                         <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 7 </span><span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">'''</span>                                                                                     <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span> 8 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">return</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> role(name=<span style=\"color: #808000; text-decoration-color: #808000\">\"assistant\"</span>, block_content=block_content, partial_output=part    <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 9 </span>assistant.is_block = <span style=\"color: #0000ff; text-decoration-color: #0000ff\">True</span>                                                                   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">10 </span>                                                                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/library/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_role.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">12</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">role</span>                                <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 9 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># send the role-start special tokens</span>                                                    <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">10 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>partial_output(parser.program.llm.role_start(name))                                     <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">11 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>                                                                                        <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>12 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>out = <span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> parser.visit(block_content[<span style=\"color: #0000ff; text-decoration-color: #0000ff\">0</span>], next_node=next_node, prev_node=prev_node,    <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">13 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>                                                                                        <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">14 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># send the role-end special tokens</span>                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">15 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>partial_output(parser.program.llm.role_end(name))                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">434</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">visit</span>                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">431 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = node.children[i - <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>]                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">432 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">433 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = prev_node                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>434 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>visited_children.append(<span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.visit(child, inner_next_node, inner_n   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">435 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># visited_children = [self.visit(child) for child in node.children]</span>            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">436 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">437 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">len</span>(visited_children) == <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>:                                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">434</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">visit</span>                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">431 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = node.children[i - <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>]                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">432 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">433 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = prev_node                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>434 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>visited_children.append(<span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.visit(child, inner_next_node, inner_n   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">435 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># visited_children = [self.visit(child) for child in node.children]</span>            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">436 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">437 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">len</span>(visited_children) == <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>:                                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">394</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">visit</span>                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">391 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>named_args[<span style=\"color: #808000; text-decoration-color: #808000\">\"prev_node\"</span>] = node.children[<span style=\"color: #0000ff; text-decoration-color: #0000ff\">0</span>]                             <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">392 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>                                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">393 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> inspect.iscoroutinefunction(command_function):                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>394 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>command_output = <span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> command_function(*positional_args, **named_ar   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">395 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">396 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>command_output = command_function(*positional_args, **named_args)      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">397 </span>                                                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/library/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_geneach.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">74</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">geneach</span>                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 71 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">len</span>(data) &gt; <span style=\"color: #0000ff; text-decoration-color: #0000ff\">0</span> <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">and</span> join != <span style=\"color: #808000; text-decoration-color: #808000\">\"\"</span>:                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 72 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>partial_output(join)                                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 73 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span> 74 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> parser.visit(block_content[<span style=\"color: #0000ff; text-decoration-color: #0000ff\">0</span>]) <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># fills out parser.prefix</span>                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 75 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>block_variables = parser.variable_stack.pop()[<span style=\"color: #808000; text-decoration-color: #808000\">\"this\"</span>]                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 76 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>data.append(block_variables)                                                   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 77 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> hidden:                                                                     <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">434</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">visit</span>                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">431 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = node.children[i - <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>]                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">432 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">433 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = prev_node                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>434 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>visited_children.append(<span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.visit(child, inner_next_node, inner_n   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">435 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># visited_children = [self.visit(child) for child in node.children]</span>            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">436 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">437 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">len</span>(visited_children) == <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>:                                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">434</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">visit</span>                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">431 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = node.children[i - <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>]                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">432 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">433 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = prev_node                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>434 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>visited_children.append(<span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.visit(child, inner_next_node, inner_n   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">435 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># visited_children = [self.visit(child) for child in node.children]</span>            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">436 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">437 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">len</span>(visited_children) == <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>:                                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">213</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">visit</span>                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">210 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">211 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># visit our children</span>                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">212 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.block_content.append([])                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>213 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>visited_children = [<span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.visit(child, next_node, next_next_node, prev_   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">214 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.block_content.pop()                                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">215 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>out = <span style=\"color: #808000; text-decoration-color: #808000\">\"\"</span>.join(<span style=\"color: #808000; text-decoration-color: #808000\">\"\"</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> c <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">is</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">str</span>(c) <span style=\"color: #0000ff; text-decoration-color: #0000ff\">for</span> c <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">in</span> visited_children)           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">216 </span>                                                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">213</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">&lt;listcomp&gt;</span>                     <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">210 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">211 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># visit our children</span>                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">212 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.block_content.append([])                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>213 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>visited_children = [<span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.visit(child, next_node, next_next_node, prev_   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">214 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.block_content.pop()                                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">215 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>out = <span style=\"color: #808000; text-decoration-color: #808000\">\"\"</span>.join(<span style=\"color: #808000; text-decoration-color: #808000\">\"\"</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> c <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">is</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">str</span>(c) <span style=\"color: #0000ff; text-decoration-color: #0000ff\">for</span> c <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">in</span> visited_children)           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">216 </span>                                                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">434</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">visit</span>                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">431 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = node.children[i - <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>]                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">432 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">433 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span>inner_prev_node = prev_node                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>434 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>visited_children.append(<span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.visit(child, inner_next_node, inner_n   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">435 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># visited_children = [self.visit(child) for child in node.children]</span>            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">436 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>                                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">437 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">len</span>(visited_children) == <span style=\"color: #0000ff; text-decoration-color: #0000ff\">1</span>:                                                 <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_program_executor.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">289</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">visit</span>                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">286 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">try</span>:                                                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">287 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> inspect.iscoroutinefunction(command_function):                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">288 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> asyncio.sleep(<span style=\"color: #0000ff; text-decoration-color: #0000ff\">0</span>) <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># give other coroutines a chance to run</span>     <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>289 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   │   </span>command_output = <span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> command_function(*positional_args, **name   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">290 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">291 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   │   </span>command_output = command_function(*positional_args, **named_args   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">292 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">except</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">StopIteration</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">as</span> ret:                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/library/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_gen.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">93</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">gen</span>                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 90 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> logprobs <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">is</span> <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">not</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span>:                                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 91 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>logprobs_list = parser.get_variable(variable_name+<span style=\"color: #808000; text-decoration-color: #808000\">\"_logprobs\"</span>, [])         <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 92 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>logprobs_list.append([])                                                   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span> 93 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">for</span> resp <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">in</span> gen_obj:                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 94 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">await</span> asyncio.sleep(<span style=\"color: #0000ff; text-decoration-color: #0000ff\">0</span>) <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># allow other tasks to run</span>                              <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 95 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">#log(\"parser.should_stop = \" + str(parser.should_stop))</span>                        <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 96 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> parser.should_stop:                                                         <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/llms/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_transformers.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">352</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">_stream_then_save</span>             <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">349 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>                                                                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">350 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">def</span> <span style=\"color: #00ff00; text-decoration-color: #00ff00\">_stream_then_save</span>(<span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>, streamer, key, thread):                                    <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">351 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span>list_out = []                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>352 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">for</span> out <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">in</span> streamer:                                                               <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">353 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span>list_out.append(out)                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">354 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">yield</span> out                                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">355 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span>thread.join() <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># clean up the thread</span>                                                <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/home/marcotcr/work/guidance/guidance/llms/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">_transformers.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">656</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">__next__</span>                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">653 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">return</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>                                                                        <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">654 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span>                                                                                       <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">655 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">def</span> <span style=\"color: #00ff00; text-decoration-color: #00ff00\">__next__</span>(<span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>):                                                                    <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>656 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span>value = <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.out_queue.get(timeout=<span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.timeout)                                   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">657 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> value <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">is</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span>:                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">658 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">raise</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">StopIteration</span>()                                                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">659 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                              <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/usr/lib/python3.9/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">queue.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">171</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">get</span>                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">168 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">raise</span> Empty                                                            <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">169 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">elif</span> timeout <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">is</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span>:                                                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">170 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">while</span> <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">not</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>._qsize():                                                   <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span>171 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   │   </span><span style=\"color: #00ffff; text-decoration-color: #00ffff\">self</span>.not_empty.wait()                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">172 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">elif</span> timeout &lt; <span style=\"color: #0000ff; text-decoration-color: #0000ff\">0</span>:                                                              <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">173 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">raise</span> <span style=\"color: #00ffff; text-decoration-color: #00ffff\">ValueError</span>(<span style=\"color: #808000; text-decoration-color: #808000\">\"'timeout' must be a non-negative number\"</span>)                <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">174 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #bfbf7f; text-decoration-color: #bfbf7f\">/usr/lib/python3.9/</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">threading.py</span>:<span style=\"color: #0000ff; text-decoration-color: #0000ff\">312</span> in <span style=\"color: #00ff00; text-decoration-color: #00ff00\">wait</span>                                                      <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>                                                                                                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 309 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span>gotit = <span style=\"color: #0000ff; text-decoration-color: #0000ff\">False</span>                                                                     <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 310 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">try</span>:    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"># restore state no matter what (e.g., KeyboardInterrupt)</span>                  <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 311 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> timeout <span style=\"color: #ff00ff; text-decoration-color: #ff00ff\">is</span> <span style=\"color: #0000ff; text-decoration-color: #0000ff\">None</span>:                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span> <span style=\"color: #800000; text-decoration-color: #800000\">❱ </span> 312 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>waiter.acquire()                                                          <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 313 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span>gotit = <span style=\"color: #0000ff; text-decoration-color: #0000ff\">True</span>                                                              <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 314 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">else</span>:                                                                         <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">│</span>   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> 315 </span><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">│   │   │   │   </span><span style=\"color: #0000ff; text-decoration-color: #0000ff\">if</span> timeout &gt; <span style=\"color: #0000ff; text-decoration-color: #0000ff\">0</span>:                                                           <span style=\"color: #800000; text-decoration-color: #800000\">│</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000\">╰──────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-weight: bold\">KeyboardInterrupt</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[31m╭─\u001b[0m\u001b[31m──────────────────────────────\u001b[0m\u001b[31m \u001b[0m\u001b[1;31mTraceback \u001b[0m\u001b[1;2;31m(most recent call last)\u001b[0m\u001b[31m \u001b[0m\u001b[31m───────────────────────────────\u001b[0m\u001b[31m─╮\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/tmp/ipykernel_3685168/\u001b[0m\u001b[1;33m2592723886.py\u001b[0m:\u001b[94m2\u001b[0m in \u001b[92m<module>\u001b[0m                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m1 \u001b[0mshell = StatefulShellOpenSource()                                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m2 t = guided_terminal(llm=mpt, task=task, shell=shell)                                         \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m3 \u001b[0m                                                                                             \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program.py\u001b[0m:\u001b[94m216\u001b[0m in \u001b[92m__call__\u001b[0m                                \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m213 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                              \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m214 \u001b[0m\u001b[2m│   │   │   \u001b[0mloop = asyncio.new_event_loop()                                                \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m215 \u001b[0m\u001b[2m│   │   │   \u001b[0mloop.create_task(new_program.update_display.run()) \u001b[2m# start the display updat\u001b[0m   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m216 \u001b[2m│   │   │   \u001b[0mloop.run_until_complete(new_program.execute())                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m217 \u001b[0m\u001b[2m│   │   \u001b[0m                                                                                   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m218 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mreturn\u001b[0m new_program                                                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m219 \u001b[0m                                                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/.virtualenvs/guidance/lib/python3.9/site-packages/\u001b[0m\u001b[1;33mnest_asyncio.py\u001b[0m:\u001b[94m84\u001b[0m in           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[92mrun_until_complete\u001b[0m                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 81 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m f \u001b[95mis\u001b[0m \u001b[95mnot\u001b[0m future:                                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 82 \u001b[0m\u001b[2m│   │   │   │   \u001b[0mf._log_destroy_pending = \u001b[94mFalse\u001b[0m                                             \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 83 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mwhile\u001b[0m \u001b[95mnot\u001b[0m f.done():                                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m 84 \u001b[2m│   │   │   │   \u001b[0m\u001b[96mself\u001b[0m._run_once()                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 85 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94mif\u001b[0m \u001b[96mself\u001b[0m._stopping:                                                         \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 86 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0m\u001b[94mbreak\u001b[0m                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 87 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m \u001b[95mnot\u001b[0m f.done():                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/.virtualenvs/guidance/lib/python3.9/site-packages/\u001b[0m\u001b[1;33mnest_asyncio.py\u001b[0m:\u001b[94m120\u001b[0m in          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[92m_run_once\u001b[0m                                                                                        \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m117 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94mbreak\u001b[0m                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m118 \u001b[0m\u001b[2m│   │   │   \u001b[0mhandle = ready.popleft()                                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m119 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m \u001b[95mnot\u001b[0m handle._cancelled:                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m120 \u001b[2m│   │   │   │   \u001b[0mhandle._run()                                                              \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m121 \u001b[0m\u001b[2m│   │   \u001b[0mhandle = \u001b[94mNone\u001b[0m                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m122 \u001b[0m\u001b[2m│   \u001b[0m                                                                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m123 \u001b[0m\u001b[2m│   \u001b[0m\u001b[1;95m@contextmanager\u001b[0m                                                                        \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/usr/lib/python3.9/asyncio/\u001b[0m\u001b[1;33mevents.py\u001b[0m:\u001b[94m80\u001b[0m in \u001b[92m_run\u001b[0m                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 77 \u001b[0m\u001b[2m│   \u001b[0m                                                                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 78 \u001b[0m\u001b[2m│   \u001b[0m\u001b[94mdef\u001b[0m \u001b[92m_run\u001b[0m(\u001b[96mself\u001b[0m):                                                                        \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 79 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mtry\u001b[0m:                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m 80 \u001b[2m│   │   │   \u001b[0m\u001b[96mself\u001b[0m._context.run(\u001b[96mself\u001b[0m._callback, *\u001b[96mself\u001b[0m._args)                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 81 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mexcept\u001b[0m (\u001b[96mSystemExit\u001b[0m, \u001b[96mKeyboardInterrupt\u001b[0m):                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 82 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mraise\u001b[0m                                                                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 83 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mexcept\u001b[0m \u001b[96mBaseException\u001b[0m \u001b[94mas\u001b[0m exc:                                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/.virtualenvs/guidance/lib/python3.9/site-packages/\u001b[0m\u001b[1;33mnest_asyncio.py\u001b[0m:\u001b[94m196\u001b[0m in \u001b[92mstep\u001b[0m     \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m193 \u001b[0m\u001b[2m│   \u001b[0m\u001b[94mdef\u001b[0m \u001b[92mstep\u001b[0m(task, exc=\u001b[94mNone\u001b[0m):                                                              \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m194 \u001b[0m\u001b[2m│   │   \u001b[0mcurr_task = curr_tasks.get(task._loop)                                             \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m195 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mtry\u001b[0m:                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m196 \u001b[2m│   │   │   \u001b[0mstep_orig(task, exc)                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m197 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mfinally\u001b[0m:                                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m198 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m curr_task \u001b[95mis\u001b[0m \u001b[94mNone\u001b[0m:                                                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m199 \u001b[0m\u001b[2m│   │   │   │   \u001b[0mcurr_tasks.pop(task._loop, \u001b[94mNone\u001b[0m)                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/usr/lib/python3.9/asyncio/\u001b[0m\u001b[1;33mtasks.py\u001b[0m:\u001b[94m256\u001b[0m in \u001b[92m__step\u001b[0m                                                \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m253 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m exc \u001b[95mis\u001b[0m \u001b[94mNone\u001b[0m:                                                                \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m254 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[2m# We use the `send` method directly, because coroutines\u001b[0m                    \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m255 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[2m# don't have `__iter__` and `__next__` methods.\u001b[0m                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m256 \u001b[2m│   │   │   │   \u001b[0mresult = coro.send(\u001b[94mNone\u001b[0m)                                                   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m257 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m258 \u001b[0m\u001b[2m│   │   │   │   \u001b[0mresult = coro.throw(exc)                                                   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m259 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mexcept\u001b[0m \u001b[96mStopIteration\u001b[0m \u001b[94mas\u001b[0m exc:                                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program.py\u001b[0m:\u001b[94m299\u001b[0m in \u001b[92mexecute\u001b[0m                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m296 \u001b[0m\u001b[2m│   │   \u001b[0m                                                                                   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m297 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[2m# run the program and capture the output\u001b[0m                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m298 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mwith\u001b[0m \u001b[96mself\u001b[0m.llm.session(asynchronous=\u001b[94mTrue\u001b[0m) \u001b[94mas\u001b[0m llm_session:                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m299 \u001b[2m│   │   │   \u001b[0m\u001b[94mawait\u001b[0m \u001b[96mself\u001b[0m._executor.run(llm_session)                                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m300 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[96mself\u001b[0m._text = \u001b[96mself\u001b[0m._executor.prefix                                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m301 \u001b[0m\u001b[2m│   │   \u001b[0m                                                                                   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m302 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[2m# delete the executor and so mark the program as not executing\u001b[0m                     \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m94\u001b[0m in \u001b[92mrun\u001b[0m                             \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 91 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[2m# self.whitespace_control_visit(self.parse_tree)\u001b[0m                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 92 \u001b[0m\u001b[2m│   │   │   \u001b[0m                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 93 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[2m# now execute the program\u001b[0m                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m 94 \u001b[2m│   │   │   \u001b[0m\u001b[94mawait\u001b[0m \u001b[96mself\u001b[0m.visit(\u001b[96mself\u001b[0m.parse_tree)                                              \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 95 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mexcept\u001b[0m \u001b[96mException\u001b[0m \u001b[94mas\u001b[0m e:                                                             \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 96 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[96mprint\u001b[0m(traceback.format_exc())                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 97 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[96mprint\u001b[0m(\u001b[33m\"\u001b[0m\u001b[33mError in program: \u001b[0m\u001b[33m\"\u001b[0m, e)                                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m434\u001b[0m in \u001b[92mvisit\u001b[0m                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m431 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = node.children[i - \u001b[94m1\u001b[0m]                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m432 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m433 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = prev_node                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m434 \u001b[2m│   │   │   │   \u001b[0mvisited_children.append(\u001b[94mawait\u001b[0m \u001b[96mself\u001b[0m.visit(child, inner_next_node, inner_n   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m435 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[2m# visited_children = [self.visit(child) for child in node.children]\u001b[0m            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m436 \u001b[0m\u001b[2m│   │   │   \u001b[0m                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m437 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m \u001b[96mlen\u001b[0m(visited_children) == \u001b[94m1\u001b[0m:                                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m434\u001b[0m in \u001b[92mvisit\u001b[0m                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m431 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = node.children[i - \u001b[94m1\u001b[0m]                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m432 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m433 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = prev_node                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m434 \u001b[2m│   │   │   │   \u001b[0mvisited_children.append(\u001b[94mawait\u001b[0m \u001b[96mself\u001b[0m.visit(child, inner_next_node, inner_n   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m435 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[2m# visited_children = [self.visit(child) for child in node.children]\u001b[0m            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m436 \u001b[0m\u001b[2m│   │   │   \u001b[0m                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m437 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m \u001b[96mlen\u001b[0m(visited_children) == \u001b[94m1\u001b[0m:                                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m394\u001b[0m in \u001b[92mvisit\u001b[0m                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m391 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0mnamed_args[\u001b[33m\"\u001b[0m\u001b[33mprev_node\u001b[0m\u001b[33m\"\u001b[0m] = node.children[\u001b[94m0\u001b[0m]                             \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m392 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m                                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m393 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94mif\u001b[0m inspect.iscoroutinefunction(command_function):                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m394 \u001b[2m│   │   │   │   │   \u001b[0mcommand_output = \u001b[94mawait\u001b[0m command_function(*positional_args, **named_ar   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m395 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m396 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0mcommand_output = command_function(*positional_args, **named_args)      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m397 \u001b[0m                                                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/library/\u001b[0m\u001b[1;33m_assistant.py\u001b[0m:\u001b[94m8\u001b[0m in \u001b[92massistant\u001b[0m                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 5 \u001b[0m\u001b[2m│   \u001b[0m                                                                                        \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 6 \u001b[0m\u001b[2;33m│   \u001b[0m\u001b[33mThis is just a shorthand for {{#role 'assistant'}}...{{/role}}.\u001b[0m                         \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 7 \u001b[0m\u001b[2;33m│   \u001b[0m\u001b[33m'''\u001b[0m                                                                                     \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m 8 \u001b[2m│   \u001b[0m\u001b[94mreturn\u001b[0m \u001b[94mawait\u001b[0m role(name=\u001b[33m\"\u001b[0m\u001b[33massistant\u001b[0m\u001b[33m\"\u001b[0m, block_content=block_content, partial_output=part    \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 9 \u001b[0massistant.is_block = \u001b[94mTrue\u001b[0m                                                                   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m10 \u001b[0m                                                                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/library/\u001b[0m\u001b[1;33m_role.py\u001b[0m:\u001b[94m12\u001b[0m in \u001b[92mrole\u001b[0m                                \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 9 \u001b[0m\u001b[2m│   \u001b[0m\u001b[2m# send the role-start special tokens\u001b[0m                                                    \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m10 \u001b[0m\u001b[2m│   \u001b[0mpartial_output(parser.program.llm.role_start(name))                                     \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m11 \u001b[0m\u001b[2m│   \u001b[0m                                                                                        \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m12 \u001b[2m│   \u001b[0mout = \u001b[94mawait\u001b[0m parser.visit(block_content[\u001b[94m0\u001b[0m], next_node=next_node, prev_node=prev_node,    \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m13 \u001b[0m\u001b[2m│   \u001b[0m                                                                                        \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m14 \u001b[0m\u001b[2m│   \u001b[0m\u001b[2m# send the role-end special tokens\u001b[0m                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m15 \u001b[0m\u001b[2m│   \u001b[0mpartial_output(parser.program.llm.role_end(name))                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m434\u001b[0m in \u001b[92mvisit\u001b[0m                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m431 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = node.children[i - \u001b[94m1\u001b[0m]                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m432 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m433 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = prev_node                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m434 \u001b[2m│   │   │   │   \u001b[0mvisited_children.append(\u001b[94mawait\u001b[0m \u001b[96mself\u001b[0m.visit(child, inner_next_node, inner_n   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m435 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[2m# visited_children = [self.visit(child) for child in node.children]\u001b[0m            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m436 \u001b[0m\u001b[2m│   │   │   \u001b[0m                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m437 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m \u001b[96mlen\u001b[0m(visited_children) == \u001b[94m1\u001b[0m:                                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m434\u001b[0m in \u001b[92mvisit\u001b[0m                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m431 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = node.children[i - \u001b[94m1\u001b[0m]                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m432 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m433 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = prev_node                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m434 \u001b[2m│   │   │   │   \u001b[0mvisited_children.append(\u001b[94mawait\u001b[0m \u001b[96mself\u001b[0m.visit(child, inner_next_node, inner_n   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m435 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[2m# visited_children = [self.visit(child) for child in node.children]\u001b[0m            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m436 \u001b[0m\u001b[2m│   │   │   \u001b[0m                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m437 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m \u001b[96mlen\u001b[0m(visited_children) == \u001b[94m1\u001b[0m:                                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m394\u001b[0m in \u001b[92mvisit\u001b[0m                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m391 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0mnamed_args[\u001b[33m\"\u001b[0m\u001b[33mprev_node\u001b[0m\u001b[33m\"\u001b[0m] = node.children[\u001b[94m0\u001b[0m]                             \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m392 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m                                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m393 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94mif\u001b[0m inspect.iscoroutinefunction(command_function):                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m394 \u001b[2m│   │   │   │   │   \u001b[0mcommand_output = \u001b[94mawait\u001b[0m command_function(*positional_args, **named_ar   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m395 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m396 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0mcommand_output = command_function(*positional_args, **named_args)      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m397 \u001b[0m                                                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/library/\u001b[0m\u001b[1;33m_geneach.py\u001b[0m:\u001b[94m74\u001b[0m in \u001b[92mgeneach\u001b[0m                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 71 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m \u001b[96mlen\u001b[0m(data) > \u001b[94m0\u001b[0m \u001b[95mand\u001b[0m join != \u001b[33m\"\u001b[0m\u001b[33m\"\u001b[0m:                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 72 \u001b[0m\u001b[2m│   │   │   │   \u001b[0mpartial_output(join)                                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 73 \u001b[0m\u001b[2m│   │   │   \u001b[0m                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m 74 \u001b[2m│   │   │   \u001b[0m\u001b[94mawait\u001b[0m parser.visit(block_content[\u001b[94m0\u001b[0m]) \u001b[2m# fills out parser.prefix\u001b[0m                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 75 \u001b[0m\u001b[2m│   │   │   \u001b[0mblock_variables = parser.variable_stack.pop()[\u001b[33m\"\u001b[0m\u001b[33mthis\u001b[0m\u001b[33m\"\u001b[0m]                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 76 \u001b[0m\u001b[2m│   │   │   \u001b[0mdata.append(block_variables)                                                   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 77 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m hidden:                                                                     \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m434\u001b[0m in \u001b[92mvisit\u001b[0m                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m431 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = node.children[i - \u001b[94m1\u001b[0m]                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m432 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m433 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = prev_node                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m434 \u001b[2m│   │   │   │   \u001b[0mvisited_children.append(\u001b[94mawait\u001b[0m \u001b[96mself\u001b[0m.visit(child, inner_next_node, inner_n   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m435 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[2m# visited_children = [self.visit(child) for child in node.children]\u001b[0m            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m436 \u001b[0m\u001b[2m│   │   │   \u001b[0m                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m437 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m \u001b[96mlen\u001b[0m(visited_children) == \u001b[94m1\u001b[0m:                                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m434\u001b[0m in \u001b[92mvisit\u001b[0m                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m431 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = node.children[i - \u001b[94m1\u001b[0m]                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m432 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m433 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = prev_node                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m434 \u001b[2m│   │   │   │   \u001b[0mvisited_children.append(\u001b[94mawait\u001b[0m \u001b[96mself\u001b[0m.visit(child, inner_next_node, inner_n   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m435 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[2m# visited_children = [self.visit(child) for child in node.children]\u001b[0m            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m436 \u001b[0m\u001b[2m│   │   │   \u001b[0m                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m437 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m \u001b[96mlen\u001b[0m(visited_children) == \u001b[94m1\u001b[0m:                                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m213\u001b[0m in \u001b[92mvisit\u001b[0m                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m210 \u001b[0m\u001b[2m│   │   │   \u001b[0m                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m211 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[2m# visit our children\u001b[0m                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m212 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[96mself\u001b[0m.block_content.append([])                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m213 \u001b[2m│   │   │   \u001b[0mvisited_children = [\u001b[94mawait\u001b[0m \u001b[96mself\u001b[0m.visit(child, next_node, next_next_node, prev_   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m214 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[96mself\u001b[0m.block_content.pop()                                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m215 \u001b[0m\u001b[2m│   │   │   \u001b[0mout = \u001b[33m\"\u001b[0m\u001b[33m\"\u001b[0m.join(\u001b[33m\"\u001b[0m\u001b[33m\"\u001b[0m \u001b[94mif\u001b[0m c \u001b[95mis\u001b[0m \u001b[94mNone\u001b[0m \u001b[94melse\u001b[0m \u001b[96mstr\u001b[0m(c) \u001b[94mfor\u001b[0m c \u001b[95min\u001b[0m visited_children)           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m216 \u001b[0m                                                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m213\u001b[0m in \u001b[92m<listcomp>\u001b[0m                     \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m210 \u001b[0m\u001b[2m│   │   │   \u001b[0m                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m211 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[2m# visit our children\u001b[0m                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m212 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[96mself\u001b[0m.block_content.append([])                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m213 \u001b[2m│   │   │   \u001b[0mvisited_children = [\u001b[94mawait\u001b[0m \u001b[96mself\u001b[0m.visit(child, next_node, next_next_node, prev_   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m214 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[96mself\u001b[0m.block_content.pop()                                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m215 \u001b[0m\u001b[2m│   │   │   \u001b[0mout = \u001b[33m\"\u001b[0m\u001b[33m\"\u001b[0m.join(\u001b[33m\"\u001b[0m\u001b[33m\"\u001b[0m \u001b[94mif\u001b[0m c \u001b[95mis\u001b[0m \u001b[94mNone\u001b[0m \u001b[94melse\u001b[0m \u001b[96mstr\u001b[0m(c) \u001b[94mfor\u001b[0m c \u001b[95min\u001b[0m visited_children)           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m216 \u001b[0m                                                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m434\u001b[0m in \u001b[92mvisit\u001b[0m                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m431 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = node.children[i - \u001b[94m1\u001b[0m]                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m432 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m433 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0minner_prev_node = prev_node                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m434 \u001b[2m│   │   │   │   \u001b[0mvisited_children.append(\u001b[94mawait\u001b[0m \u001b[96mself\u001b[0m.visit(child, inner_next_node, inner_n   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m435 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[2m# visited_children = [self.visit(child) for child in node.children]\u001b[0m            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m436 \u001b[0m\u001b[2m│   │   │   \u001b[0m                                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m437 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m \u001b[96mlen\u001b[0m(visited_children) == \u001b[94m1\u001b[0m:                                                 \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/\u001b[0m\u001b[1;33m_program_executor.py\u001b[0m:\u001b[94m289\u001b[0m in \u001b[92mvisit\u001b[0m                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m286 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94mtry\u001b[0m:                                                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m287 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0m\u001b[94mif\u001b[0m inspect.iscoroutinefunction(command_function):                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m288 \u001b[0m\u001b[2m│   │   │   │   │   │   \u001b[0m\u001b[94mawait\u001b[0m asyncio.sleep(\u001b[94m0\u001b[0m) \u001b[2m# give other coroutines a chance to run\u001b[0m     \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m289 \u001b[2m│   │   │   │   │   │   \u001b[0mcommand_output = \u001b[94mawait\u001b[0m command_function(*positional_args, **name   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m290 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m291 \u001b[0m\u001b[2m│   │   │   │   │   │   \u001b[0mcommand_output = command_function(*positional_args, **named_args   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m292 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94mexcept\u001b[0m \u001b[96mStopIteration\u001b[0m \u001b[94mas\u001b[0m ret:                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/library/\u001b[0m\u001b[1;33m_gen.py\u001b[0m:\u001b[94m93\u001b[0m in \u001b[92mgen\u001b[0m                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 90 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m logprobs \u001b[95mis\u001b[0m \u001b[95mnot\u001b[0m \u001b[94mNone\u001b[0m:                                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 91 \u001b[0m\u001b[2m│   │   │   │   \u001b[0mlogprobs_list = parser.get_variable(variable_name+\u001b[33m\"\u001b[0m\u001b[33m_logprobs\u001b[0m\u001b[33m\"\u001b[0m, [])         \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 92 \u001b[0m\u001b[2m│   │   │   │   \u001b[0mlogprobs_list.append([])                                                   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m 93 \u001b[2m│   │   \u001b[0m\u001b[94mfor\u001b[0m resp \u001b[95min\u001b[0m gen_obj:                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 94 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mawait\u001b[0m asyncio.sleep(\u001b[94m0\u001b[0m) \u001b[2m# allow other tasks to run\u001b[0m                              \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 95 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[2m#log(\"parser.should_stop = \" + str(parser.should_stop))\u001b[0m                        \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 96 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m parser.should_stop:                                                         \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/llms/\u001b[0m\u001b[1;33m_transformers.py\u001b[0m:\u001b[94m352\u001b[0m in \u001b[92m_stream_then_save\u001b[0m             \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m349 \u001b[0m\u001b[2m│   \u001b[0m                                                                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m350 \u001b[0m\u001b[2m│   \u001b[0m\u001b[94mdef\u001b[0m \u001b[92m_stream_then_save\u001b[0m(\u001b[96mself\u001b[0m, streamer, key, thread):                                    \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m351 \u001b[0m\u001b[2m│   │   \u001b[0mlist_out = []                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m352 \u001b[2m│   │   \u001b[0m\u001b[94mfor\u001b[0m out \u001b[95min\u001b[0m streamer:                                                               \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m353 \u001b[0m\u001b[2m│   │   │   \u001b[0mlist_out.append(out)                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m354 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94myield\u001b[0m out                                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m355 \u001b[0m\u001b[2m│   │   \u001b[0mthread.join() \u001b[2m# clean up the thread\u001b[0m                                                \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/home/marcotcr/work/guidance/guidance/llms/\u001b[0m\u001b[1;33m_transformers.py\u001b[0m:\u001b[94m656\u001b[0m in \u001b[92m__next__\u001b[0m                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m653 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mreturn\u001b[0m \u001b[96mself\u001b[0m                                                                        \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m654 \u001b[0m\u001b[2m│   \u001b[0m                                                                                       \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m655 \u001b[0m\u001b[2m│   \u001b[0m\u001b[94mdef\u001b[0m \u001b[92m__next__\u001b[0m(\u001b[96mself\u001b[0m):                                                                    \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m656 \u001b[2m│   │   \u001b[0mvalue = \u001b[96mself\u001b[0m.out_queue.get(timeout=\u001b[96mself\u001b[0m.timeout)                                   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m657 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mif\u001b[0m value \u001b[95mis\u001b[0m \u001b[94mNone\u001b[0m:                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m658 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mraise\u001b[0m \u001b[96mStopIteration\u001b[0m()                                                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m659 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                              \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/usr/lib/python3.9/\u001b[0m\u001b[1;33mqueue.py\u001b[0m:\u001b[94m171\u001b[0m in \u001b[92mget\u001b[0m                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m168 \u001b[0m\u001b[2m│   │   │   │   │   \u001b[0m\u001b[94mraise\u001b[0m Empty                                                            \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m169 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94melif\u001b[0m timeout \u001b[95mis\u001b[0m \u001b[94mNone\u001b[0m:                                                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m170 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94mwhile\u001b[0m \u001b[95mnot\u001b[0m \u001b[96mself\u001b[0m._qsize():                                                   \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m171 \u001b[2m│   │   │   │   │   \u001b[0m\u001b[96mself\u001b[0m.not_empty.wait()                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m172 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94melif\u001b[0m timeout < \u001b[94m0\u001b[0m:                                                              \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m173 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94mraise\u001b[0m \u001b[96mValueError\u001b[0m(\u001b[33m\"\u001b[0m\u001b[33m'\u001b[0m\u001b[33mtimeout\u001b[0m\u001b[33m'\u001b[0m\u001b[33m must be a non-negative number\u001b[0m\u001b[33m\"\u001b[0m)                \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m174 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[2;33m/usr/lib/python3.9/\u001b[0m\u001b[1;33mthreading.py\u001b[0m:\u001b[94m312\u001b[0m in \u001b[92mwait\u001b[0m                                                      \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m                                                                                                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 309 \u001b[0m\u001b[2m│   │   \u001b[0mgotit = \u001b[94mFalse\u001b[0m                                                                     \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 310 \u001b[0m\u001b[2m│   │   \u001b[0m\u001b[94mtry\u001b[0m:    \u001b[2m# restore state no matter what (e.g., KeyboardInterrupt)\u001b[0m                  \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 311 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94mif\u001b[0m timeout \u001b[95mis\u001b[0m \u001b[94mNone\u001b[0m:                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m 312 \u001b[2m│   │   │   │   \u001b[0mwaiter.acquire()                                                          \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 313 \u001b[0m\u001b[2m│   │   │   │   \u001b[0mgotit = \u001b[94mTrue\u001b[0m                                                              \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 314 \u001b[0m\u001b[2m│   │   │   \u001b[0m\u001b[94melse\u001b[0m:                                                                         \u001b[31m│\u001b[0m\n",
       "\u001b[31m│\u001b[0m   \u001b[2m 315 \u001b[0m\u001b[2m│   │   │   │   \u001b[0m\u001b[94mif\u001b[0m timeout > \u001b[94m0\u001b[0m:                                                           \u001b[31m│\u001b[0m\n",
       "\u001b[31m╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n",
       "\u001b[1;91mKeyboardInterrupt\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "shell = StatefulShellOpenSource()\n",
    "t = guided_terminal(llm=mpt, task=task, shell=shell)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, MPT fails this time, by calling the same command again and again (we had to interrupt to stop execution.)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a discussion of these experiments check out the [the blog post](https://medium.com/@marcotcr/exploring-chatgpt-vs-open-source-models-on-slightly-harder-tasks-aa0395c31610)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "guidance",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}