learn/generation/canopy/00-canopy-data-prep.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Prep for Canopy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -qU \\\n",
    "    canopy-sdk==0.1.0 \\\n",
    "    datasets==2.14.6"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create JSON File"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Canopy reads local JSON / parquet files that contain the fields `[\"id\", \"text\", \"source\", \"metadata\"]`. We will use the [`jamescalam/ai-arxiv`](https://huggingface.co/datasets/jamescalam/ai-arxiv) dataset. First we download it like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['id', 'title', 'summary', 'source', 'authors', 'categories', 'comment', 'journal_ref', 'primary_category', 'published', 'updated', 'content', 'references'],\n",
       "    num_rows: 423\n",
       "})"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "data = load_dataset(\"jamescalam/ai-arxiv\", split=\"train\")\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '2210.03945',\n",
       " 'title': 'Understanding HTML with Large Language Models',\n",
       " 'summary': 'Large language models (LLMs) have shown exceptional performance on a variety\\nof natural language tasks. Yet, their capabilities for HTML understanding --\\ni.e., parsing the raw HTML of a webpage, with applications to automation of\\nweb-based tasks, crawling, and browser-assisted retrieval -- have not been\\nfully explored. We contribute HTML understanding models (fine-tuned LLMs) and\\nan in-depth analysis of their capabilities under three tasks: (i) Semantic\\nClassification of HTML elements, (ii) Description Generation for HTML inputs,\\nand (iii) Autonomous Web Navigation of HTML pages. While previous work has\\ndeveloped dedicated architectures and training procedures for HTML\\nunderstanding, we show that LLMs pretrained on standard natural language\\ncorpora transfer remarkably well to HTML understanding tasks. For instance,\\nfine-tuned LLMs are 12% more accurate at semantic classification compared to\\nmodels trained exclusively on the task dataset. Moreover, when fine-tuned on\\ndata from the MiniWoB benchmark, LLMs successfully complete 50% more tasks\\nusing 192x less data compared to the previous best supervised model. Out of the\\nLLMs we evaluate, we show evidence that T5-based models are ideal due to their\\nbidirectional encoder-decoder architecture. To promote further research on LLMs\\nfor HTML understanding, we create and open-source a large-scale HTML dataset\\ndistilled and auto-labeled from CommonCrawl.',\n",
       " 'source': 'http://arxiv.org/pdf/2210.03945',\n",
       " 'authors': ['Izzeddin Gur',\n",
       "  'Ofir Nachum',\n",
       "  'Yingjie Miao',\n",
       "  'Mustafa Safdari',\n",
       "  'Austin Huang',\n",
       "  'Aakanksha Chowdhery',\n",
       "  'Sharan Narang',\n",
       "  'Noah Fiedel',\n",
       "  'Aleksandra Faust'],\n",
       " 'categories': ['cs.LG', 'cs.AI'],\n",
       " 'comment': None,\n",
       " 'journal_ref': None,\n",
       " 'primary_category': 'cs.LG',\n",
       " 'published': '20221008',\n",
       " 'updated': '20230519',\n",
       " 'content': 'UNDERSTANDING HTML WITH LARGE LANGUAGE\\nMODELS\\nIzzeddin Gur, Oﬁr Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang\\nAakanksha Chowdhery, Sharan Narang, Noah Fiedel, Aleksandra Faust\\nGoogle Research\\nfizzeddin,ofirnachum,yingjiemiao,msafdari,austinvhuang\\nchowdhery,sharannarang,nfiedel,sandrafaust g@google.com\\nABSTRACT\\nLarge language models (LLMs) have shown exceptional performance on a va-\\nriety of natural language tasks. Yet, their capabilities for HTML understanding\\n– i.e., parsing the raw HTML of a webpage, with applications to automation of\\nweb-based tasks, crawling, and browser-assisted retrieval – have not been fully\\nexplored. We contribute HTML understanding models (ﬁne-tuned LLMs) and an\\nin-depth analysis of their capabilities under three tasks: (i) Semantic Classiﬁca-\\ntionof HTML elements, (ii) Description Generation for HTML inputs, and (iii)\\nAutonomous Web Navigation of HTML pages. While previous work has devel-\\noped dedicated architectures and training procedures for HTML understanding,\\nwe show that LLMs pretrained on standard natural language corpora transfer re-\\nmarkably well to HTML understanding tasks. For instance, ﬁne-tuned LLMs are\\n12% more accurate at semantic classiﬁcation compared to models trained exclu-\\nsively on the task dataset. Moreover, when ﬁne-tuned on data from the MiniWoB\\nbenchmark, LLMs successfully complete 50% more tasks using 192x less data\\ncompared to the previous best supervised model. Out of the LLMs we evalu-\\nate, we show evidence that T5-based models are ideal due to their bidirectional\\nencoder-decoder architecture. To promote further research on LLMs for HTML\\nunderstanding, we create and open-source a large-scale HTML dataset distilled\\nand auto-labeled from CommonCrawl.1\\n1 I NTRODUCTION\\nWeb crawling (Olston et al., 2010), form-ﬁlling (Diaz et al., 2013; Gur et al., 2021), or information\\nretrieving web agents (Nogueira & Cho, 2016) are important for both automating and assisting\\nusers in web-based tasks. These and similar applications rely on models that can search for speciﬁc\\ncontent or controls on a web page as well as navigate a website autonomously. Since a web page in\\nits raw form is represented as an HTML-based text sequence, the success of models for web-based\\ntasks relies on their ability to understand HTML semantics, structure, and embedded interactions.\\nThe predominant approach to web automation and HTML understanding is to train specialized mod-\\nels, i.e., gathering application-speciﬁc datasets and designing neural network (NN) architectures to\\nleverage inductive biases of the HTML’s structure; see, e.g., Liu et al. (2018); Toyama et al. (2021);\\nGur et al. (2021); Humphreys et al. (2022). However, both dataset collection and neural architecture\\ndesign are expensive, time-consuming, and require highly-specialized, domain-speciﬁc knowledge.\\nMeanwhile, in the natural language processing (NLP) literature, large language models (LLMs) have\\nemerged as a solution to the difﬁculties of dataset collection and specialized NN design (Kaplan\\net al., 2020; Bommasani et al., 2021). A popular paradigm in NLP is to take an off-the-shelf LLM\\n– pretrained on a large text corpus via an unsupervised and task-agnostic learning objective – and\\neither ﬁne-tune or prompt the LLM on a small task-speciﬁc dataset. This paradigm has shown\\nexceptional performance on a variety of NLP tasks (Xue et al., 2020; Brown et al., 2020; Austin\\net al., 2021). Whether LLMs can be applied to HTML understanding – especially given the much\\nlarger context and sequence lengths – remains an under-explored question.\\n1See visualizations of the results at https://sites.google.com/view/llm4html/home .\\n1arXiv:2210.03945v2  [cs.LG]  19 May 2023\\n<html> \\n   <body> \\n      <form class= \"login-form\" >\\n   <div> \\n            <label class= \"form-label\" for= ”uName” >\\n               Enter Email Address \\n            </label> \\n      <label class= \"form-label\" for= ”pass” >\\n               Enter Password: \\n            </label> \\n   </div> \\n         <div> \\n <input type= \"email\"  id=\"uName” >\\n<input type= \"password\"  id=\"pass\" >\\n<span class= \"hidden\" >\\n               Please enter your password. \\n            </span> \\n         </div> \\n         <button type= \"submit\" >Sign In </button> \\n       </form> \\n   </body> \\n</html> (a)\\n<div><label class= \"form-label\" for= ”uName” >Email Address </label><label \\nclass= \"form-label\" for= ”pass” >Enter Password: </label></div><div><input \\ntype= \"email\"  id=\"uName” target ><input type= \"password\"  id=\"pass\" ><span \\nclass= \"hidden\" >Please enter your password. </span></div> (b)\\nFigure 1: a) HTML example page with a highlighted salient element, an element of interest (dashed box).\\nAll canonical tasks evaluate a distinct interaction with this element, either by classifying it as one of a set of\\ncategories, generating a text description of its purpose, or applying an action as part of a sequential navigation\\nof a multi-page website. b) LLM architectures overview. Dashed boxes denote sub-modules that are speciﬁc to\\neither encoder-only or encoder-decoder models. For encoder-only models, we add an extra classiﬁcation layer.\\nDecoder-only models (not in the diagram) are similar to encoder-decoder models, the main difference is that\\nthe HTML snippet is fed to the decoder and processed from left-to-right.\\nIn this paper, we investigate whether LLMs can be applied to HTML understanding to produce\\nbetter-performing, more sample-efﬁcient HTML understanding models and without the need for\\ncustom NN architecture design. To that end, we present a suite of three benchmarking tasks for\\nHTML understanding that capture the essence of these applications and require understanding both\\nstructure and content. First, we devise Semantic Classiﬁcation as a task that requires a model to\\nclassify a given HTML element into one of a set of categories, such as address, email, password\\netc., with application to automated form-ﬁlling. Second, we present Description Generation , a\\nlabel-extraction task where a model is given an HTML snippet and is asked to produce a natural\\nlanguage description. For instance for an email ﬁeld, the description might be “Please enter your\\nemail address.” Note that in the majority of web pages, this connection between input elements and\\ndescription content is only implicit in the raw HTML code and inferring such links is a prerequisite\\nfor higher-level navigation objectives. The third task is Autonomous Web Navigation (Shi et al.,\\n2017). A model is presented with an HTML page paired with a natural language command and\\nmust apply appropriate actions on a sequence of HTML pages to satisfy the command. See Figure\\n1a for a simpliﬁed example of these tasks.\\nWith these benchmark tasks in hand, we evaluate the transfer capabilities of a variety of pretrained\\nLLMs (Table 1), varying in architecture (encoder-only, encoder-decoder, or decoder-only), model\\nsize (from 24.6M to 62B parameters), and training data corpora (both including and excluding pre-\\ntraining NLP and HTML corpus). While prior work universally pre-parses the HTML as input to the\\nmodel (Gur et al., 2021; Liu et al., 2018; Nakano et al., 2021), ours – to the best of our knowledge – is\\nthe ﬁrst work that uses raw, unprocessed HTML. Our results show that LLMs demonstrate a remark-\\nable level of HTML understanding across all tasks, with up to 192\\x02more sample-efﬁciency than\\nmodels trained from scratch, and achieving a new SoTA for supervised learning on the MiniWoB\\nbenchmark suite (Shi et al., 2017). The encoder-decoder architectures with bi-directional attention\\nshow the best performance across the board even when their pretraining does not include HTML. In\\naddition, we show that the performance scales sub-linearly with the model size.\\nThe broader objective of this research is to advance the integration of LLMs with autonomous web\\nagents. It has only been in the last year that researchers have begun to utilize LLMs outside of\\nNLP and integrate them as core capabilities in autonomy (Lu et al. (2021); Ahn et al. (2022)). In\\nthis context, LLMs are reasoning engines for sequential decision making agents interacting with\\nenvironments.\\nThe present work is the ﬁrst in the research literature to embed an LLM and train it as an agent for\\nautonomous web navigation. This requires new implementations to adapt LLM training for behavior\\n2\\ncloning in addition to designing interfaces for integrating text generation into a perception-compute-\\naction cycle operating in a stateful web environment. Our implementation allows us to answer new\\nquestions regarding trade-offs among various model characteristics.\\nWe believe these contributions expand the scope of language models and connect their unique capa-\\nbilities with autonomous agents for the web. We provide a new perspective on machine learning for\\nHTML understanding and web automation, showing that pretrained LLMs can achieve signiﬁcant\\nperformance on such tasks, reducing the need for specialized architectures and training protocols.\\nTo encourage further research in this direction, we open sourced2model weights for agents used in\\nthe WoB environment and our dataset for description generation.\\n2 R ELATED WORK\\nHTML Understanding Autonomous web navigation has been a popular application for neural net-\\nwork models, and a variety of works propose simulated websites for training web-based agents, with\\napplication to task fulﬁllment (Yao et al., 2022; Gur et al., 2021; Burns et al., 2022; Mazumder &\\nRiva, 2020; Shi et al., 2017; Liu et al., 2018) as well as information retrieval or question-answering\\n(Adolphs et al., 2021; Nogueira & Cho, 2016). Simulated websites provide an easy way to evaluate\\nmodels online, and for this reason we use the existing MiniWoB benchmark (Shi et al., 2017) for our\\nweb navigation setting. However, it is still important to have a mechanism for evaluating models on\\na wide variety of real-world websites. This was the key motivation for generating our own dataset\\nfor the description generation task, which is distilled and auto-labeled from CommonCrawl and is a\\nkey contribution of our paper.\\nAlongside these benchmarks, many works have developed models for web navigation and related\\nsubtasks (Pasupat et al., 2018; Bommasani et al., 2021; He et al., 2021; Gur et al., 2021; Humphreys\\net al., 2022; Liu et al., 2018; Jia et al., 2019). These works often rely on specialized neural network\\narchitectures that leverage inductive biases of HTML structure, or on preprocessing of HTML to\\nmake it easier to input to a model (Li et al. (2021a;b)). In contrast, our work takes a minimalist\\napproach, providing HTML in text form with minimal processing and using widely-adopted trans-\\nformer networks.\\nLLMs and HTML Works that explore the intersection of LLMs and HTML generally fall into two\\ncategories. The ﬁrst category uses LLMs to assist web navigation (Nakano et al., 2021; Yao et al.,\\n2022), and typically relies on a custom preprocessing to map the context and structure of a web page\\nto natural language, thus severely restricting what HTML pages the model can parse. The second\\ncategory pretrains LLMs on a large corpora of HTML text (Aghajanyan et al., 2021). However,\\nthese works typically restrict the model evaluation to standard NLP tasks, e.g., summarization and\\nquestion/answering as opposed to tasks more relevant to HTML understanding and web automation.\\nOur work can be thought of as the reverse: We keep the pretraining of LLMs unchanged and focus\\non the mechanisms for transferring the pretrained LLMs to HTML-relevant tasks.\\n3 B RIEF BACKGROUND ON HTML ASSEMI-STRUCTURED TEXT DATA\\nHTML is a markup language, used to organize web page structure andcontent . Consider the\\nexample HTML page in Figure 1a. This web page includes two adjacent input elements, one for\\ne-mail and another for password, with their corresponding label s on a separate branch of the page.\\nThese input s and label s are one of many possible elements that serve as HTML building blocks.\\nEach element has a set of attributes – key and value pair – that describe the element’s content, such\\nas style and human-readable text. When rendered in a browser, these attributes will be responsible\\nfor how the element is shown and where it is positioned. In the example in Figure 1a, the ﬁrst\\ninput has three attributes, tag=\"input\" ,type=\"email\" , and id=\"uName\" , that identify\\nthe element as an email input with an identiﬁer (“uName”) that can be accessed programmatically.\\n2https://console.cloud.google.com/storage/browser/gresearch/webllm\\n3\\nModel\\nTask Dataset Size Input Architecture Output Task Output\\nAutonomous Web Navigation MiniWoB Demos (Shi et al., 2017) 12K PageEnc-DecText DictionaryDec\\nSemantic Classiﬁcation Annotated Shopping Webpages (Gur et al., 2021) 28K Snippet All Text Category\\nDescription Generation CommonCrawl (new) 85K SnippetEnc-DecText TextDec\\nTable 1: Task, dataset, and model summary. All models receive raw HTML. Autonomous Web Navigation\\nreceives the entire HTML, while the other tasks receive HTML snippets extracted given salient element.\\n4 C ANONICAL TASKS FOR HTML U NDERSTANDING\\nWe devise three canonical tasks to study HTML understanding capabilities of LLM-based web\\nagents. These tasks require correctly interpreting both structure and content to varying degrees\\nto make predictions, with autonomous navigation being the most challenging capability of the three.\\nAutonomous Web Navigation . This task evaluates how well a model navigates multi-page web-\\nsites as a sequential decision-making problem (Shi et al., 2017; Liu et al., 2018). At the beginning\\nof an episode, the agent is given a natural language instruction, e.g. Enter the username “lyda”\\nand the password “N22t” into the text ﬁelds and press login . The agent applies actions to a se-\\nquence of HTML pages, where each action is of the form function(selector, text) . The\\nfunction is one of click ortype,selector is an integer pointer that uniquely identiﬁes an ele-\\nment, and text is a text to input if the type functionality is activated. An episode terminates when\\neither the page reaches a terminal state (e.g., the ‘sign in’ button is clicked) or the maximum number\\nof steps is reached.\\nSemantic Classiﬁcation .Many HTML understanding applications require a model that can classify\\nHTML elements into standardized categories. For example, in automated form-ﬁlling (Diaz et al.,\\n2013; Gur et al., 2021), it is useful to identify a ‘submit button’ across many websites (e.g., shopping,\\nﬂight booking, utility application) with various button representations (e.g., position, color, or text).\\nThus, we formulate Semantic Classiﬁcation as classifying elements into role categories. Take the\\nexample HTML in Figure 1a which includes two input elements and a submit button . Let’s\\npick the ﬁrst input as an element of interest to be classiﬁed by the system, also called a salient\\nelement . The system should classify this element as username , since it appears on a login page and\\nit has a label with Email Address which is typically associated with the username in form-ﬁlling\\napplications. To solve this, the system can aggregate information from multiple sources in the page\\n– the label that says Enter Email Address , theinput attributes ( type=“email” andid=“uName” ),\\nor even the ordering of other elements in the page such as ‘password’ and ‘sign in’.\\nDescription Generation .Motivated by applications in accessibility-minded web browser con-\\ntrol (Jorgensen & Binsted, 2005), we formulate description generation as an extractive problem\\nwhere the goal is to locate the textual description of an element in the HTML and generate it as\\noutput. For instance, the description of the salient element in Figure 1a is Enter Email Address ;\\nwhen rendered, this label will appear above the ‘email’ input ﬁeld. HTML provides a large\\namount of ﬂexibility, and so in general a descriptive text that appears alongside a speciﬁc element\\nwhen rendered can be very far from that element when looking at the HTML plaintext. Thus, this\\ntask evaluates a model’s ability to understand the structure of HTML as it would appear to a user,\\ndespite not having access to the rendered web page directly.\\n5 D ATASETS\\nEach of our canonical tasks requires a separate dataset, with the description generation task using a\\nnewly contributed, auto-labelled dataset based on CommonCrawl.\\nAutonomous Web Navigation .We use the 12K demonstrations included in the publicly available\\nMiniWoB benchmark (Shi et al., 2017), which encompass 62 website applications ranging from\\nemail forwarding to social media interactions. Each demonstration is a sequence of (instruction,\\nHTML, action) tuples. Every element in a MiniWoB demonstration is accompanied by a reference\\nnumber unique within its respective pages. This number can be used as an element selector, making\\nthe action space uniﬁed across all tasks and time steps. For instance, the action in Figure 1a would be\\n4\\ntype(ref=5, ”username@email.com”) , where 5 refers to the index of the input when counted from\\ntop-to-bottom. As model input, we concatenate the natural language instruction and HTML into a\\nsingle text input sequence. Similarly, we treat the action as a text sequence for the model to predict.\\nSemantic Classiﬁcation .We use a dataset of 28K labelled examples, containing 66 different cat-\\negories, of the form (HTML, element, category) , previously used in the context of environment\\ngeneration (Gur et al., 2021). The dataset consists of HTMLs from real-world shopping websites\\nand categories relevant to form-ﬁlling during payment and checkout on these websites.\\nDescription Generation .For this task, we derive a dataset from CommonCrawl.3CommonCrawl\\ndoes not include renderings or annotations that would reveal what text in the HTML is associated\\nwith which elements. Instead, we infer descriptions of various elements by exploiting a special\\nattribute in the HTML schema known as for. As an example in Figure 1a, the ﬁrst label in\\nthe HTML has a for attribute with value uName , which is the idof the element described by\\nlabel ; in this case, the idis that of the ﬁrst input in the page. This annotation does not affect\\nthe rendering of the page and is typically used for accessibility purposes. We utilize the information\\ngiven by these for attributes to create a large-scale dataset to study description generation. A small\\nsample is available in the supplemental material, while the entire dataset will be available upon\\npublication.\\nSpeciﬁcally, we collected 100 WARC (from April 2019) ﬁles from the CommonCrawl project and\\nextracted all HTML label s that have a for attribute. Removing non-Unicode and alphanumeric\\ntext in HTML label s results in a 400K example datset. We balance the distribution of labels,\\neffectively downsampling the dataset to 85Ksamples. Each example is represented as (HTML,\\nelement, description) , where HTML is the HTML plaintext of the page, element is the element\\nwhose idattribute matches that appearing in the label ’sfor attribute, and description is the text\\ninside the label element (see example in Figure 1a). More details of the dataset can be found in\\nAppendix A.1.\\n6 P RE-PROCESSING\\nIn treating HTML as token sequences, we minimize any HTML tree pre-processing prior to model\\ninput. We thus provide HTML as raw text (i.e., sequences of text tokens) and only apply a snippet\\nextraction pre-processing for pages which are too large to ﬁt into the typical LLMs context windows.\\nSnippet Extraction. Real HTML pages can grow extremely large, reaching thousands of elements,\\nfar beyond the context window of the largest LLM that we studied (1920 tokens in PaLM (Chowdh-\\nery et al., 2022)). LLMs typically truncate such long sequences, which can be detrimental to HTML\\nunderstanding as HTMLs are not linearly structured. We take an element-centric approach and ex-\\ntract HTML snippets (a small portion of HTML code) surrounding a salient element (Figure 5). A\\nsimple heuristic, which controls the tree’s width and depth, guides the process: Start with a salient\\nelement and traverse its ancestors in the HTML tree until a stopping condition is satisﬁed. As we\\ntraverse up, we estimate the height of the tree and the increased number of descendants of the new\\nroot. We stop when either metric violates a pre-deﬁned limit and take the resulting sub-tree as the\\nsnippet. We mark the salient element using a special attribute, called target , to distinguish it from\\nother elements. We perform the snippet extraction for the semantic classiﬁcation and description\\ngeneration datasets, and keep the full HTML pages in MiniWoB because these pages are typically\\nmuch smaller than real-world HTML.\\nHTML un-Parsing. We provide the models with the unparsed plaintext HTML in the form of\\na sequence of tokens. This canonical representation does not require speciﬁc model architectures\\nsuch as hierarchical networks (Liu et al., 2018; Gur et al., 2021) and can be fed into any LLM. We\\ntransform all datasets by converting every HTML page or snippet into a sequence. For MiniWoB,\\nwe additionally concatenate (action history, instruction, HTML) tuples into a single sequence.\\n3http://commoncrawl.org\\n5\\n7 M ODEL TRAINING\\nWe study a variety of transformer-based LLMs (Vaswani et al., 2017) with different sizes and archi-\\ntectures for HTML understanding tasks (Table 1). In the rest of the text, we preﬁx models ﬁne-tuned\\nforAutonomous Web Navigation ,Description Generation , and Semantic Classiﬁcation with WebN-\\n, WebD-, and WebC-, respectively. For instance, WebD–T5-3B is the three billion parameter T5\\nmodel (Raffel et al., 2020) ﬁne-tuned for the Description Generation task. The rest of this section\\nelaborates on training details.\\nEncoder-Decoder and Decoder-only Models. We train encoder-decoder models, i.e., T5 (Raffel\\net al., 2020), and decoder-only models, i.e., LaMDA (Thoppilan et al., 2022) and PaLM (Chowdh-\\nery et al., 2022), with text input and text output (Figure 1b). Inputs are raw HTML pages or snippet\\ntexts; similarly, outputs are categories, natural language descriptions, or actions represented as text.\\nNamely, for Semantic Classiﬁcation we use the textual representation of categories, similar to previ-\\nous classiﬁcation problems in NLP (Raffel et al., 2020). For Autonomous Web Navigation , actions\\nare converted into text by ﬁrst converting them into key and value pairs and then concatenating the\\npairs.\\nMany websites in MiniWoB require multiple interactions, such as click-button-sequence orclick-\\ncheckboxes , where each interaction might cause a subtle change in the website state. For instance,\\nafter clicking on a checkbox in the click-checkboxes website, its value ﬂips from positive to negative\\nor the other way around, which is not always reﬂected in LLMs’ predictions and leads to action\\nrepetitions. We solve this issue by augmenting tuples in the dataset with a sequence of past actions,\\n(action history, instruction, HTML, action) , and allowing LLMs to learn from past experience.\\nEncoder-only Models. We train encoder-only models, i.e., BERT (Devlin et al., 2018), with text\\ninput and categorical output. We keep semantic categories as discrete one-hot classes. To train\\nencoder-only models, we add a new classiﬁcation layer after the ﬁnal encoder layer to produce a\\ndistribution over semantic categories. In addition to the typical BERT models, we study Mobile-\\nBERT (Sun et al., 2020), distilled from BERT-large with inverted bottlenecks, and Albert-XL (Lan\\net al., 2020), with parameter sharing and embedding split.\\n8 R ESULTS\\nWe now present the results of ﬁne-tuned LLMs for HTML understanding. We compare the models’\\nperformance with the existing baselines where possible (autonomous web navigation) and against\\nother LLM architectures and training regimes (all tasks). Sections 8.1, 8.2, and 8.3 evaluate task-\\nspeciﬁc performance, while Section 8.4 assesses the performance across all the tasks.\\nMetrics: For autonomous web navigation we evaluate models’ Success Rate , which is averaged over\\n100 episodes per task. For the other tasks, we use Accuracy to measure exact match between predic-\\ntion and ground truth. In the description generation task, we additionally provide evaluations using\\nalternative ‘soft’ text evaluation metrics, BLEU andROUGE-1 , measuring the similarity between\\npredicted and ground truth text.\\n8.1 A UTONOMOUS WEBNAVIGATION RESULTS\\nForAutonomous Web Navigation we ﬁne-tune two WebN- encoder-decoder architectures (WebN-\\nT5-large and WebN-T5-3B) on 12k demonstrations from human-annotated real websites. We eval-\\nuate the models on MiniWob (Liu et al., 2018) benchmark, and compare with specialized architec-\\ntures trained using supervised learning (SL) on 2.4 million human expert demonstrations CC-Net\\n(SL) (Humphreys et al., 2022), and two RL models bootstrapped with SL, CC-Net (SL) (CC-Net\\n(SL & RL) (Humphreys et al., 2022), and WGE (SL & RL) (Liu et al., 2018)). Additionally, we\\ncompare with the decoder-only architecture (WebN-Lambda-1B) and perform an ablation study on\\nthe impact of including the action history in the input.\\nComparison to SoTA. Since previous works report success on only a subset of websites in Mini-\\nWoB, we evaluate on 48 out of 62 websites that are common across all models. Table 8 in the\\nAppendix reports ﬁne-grained results while Figure 2a presents results averaged over all websites.\\nCompared to CC-Net (SL) which is trained on all 2.4M demonstrations, WebN-T5-3B improves the\\n6\\n2.4M \\nDemos 12K \\nDemos (a) Baseline comparison.Model Name Success (%) Model Size\\nT5-large 18.1 800M\\nLaMDA-1B 15.6 1B\\nT5-3B 11.1 3B\\nWebN-T5-large 46.4 800M\\nWebN-LaMDA-1B 48.8 1B\\nWebN-T5-3B 51.8 3B\\n(b) Pre-training effect.\\nFigure 2: a) WebN–T5* performance compared to the previous SOTA models on MiniWoB benchmark.\\nWebN-T5-3B improves the task success 16% while using 192 times less data, compared to the best supervised\\nlearning (SL) model, CC-Net (SL). LLMs performance is only surpassed by works utilizing RL, requiring or-\\nders of magnitude more online experience interaction with websites. b) LLMs with and without pretraining\\nonAutonomous Web Navigation task. Those with pretraining (denoted by the ‘WebN-’ preﬁx) show a 2.5-4.5x\\nperformance improvement.\\nModel Name Test (%) Dev (%) Model Size Code in training Corpus\\nWebC-MobileBERT 78.1 77.7 24.6 M\\n0%WebC-Albert-XL 83.5 83.1 58.9 M\\nWebC-BERT-smallest 84.4 83.6 38.7 M\\nWebC-BERT-small 84.4 85.2 52.8 M\\nWebC-BERT-medium 85.2 84.5 67 M\\nWebC-BERT-base 83.9 84.8 109.5 M\\nWebC-BERT-large 84.1 85.8 335.2 M\\nWebC-T5-base 86.8 89.9 250 M\\nWebC-T5-large 87.0 89.3 800 M\\nWebC-T5-3B 87.7 90.3 3 B\\nWebC-LaMDA-1B 87.4 87.1 1 B 12.5% Code\\nWebC-PaLM-8B 86.6 89.9 8 B 5% Code (0.875% HTML)\\nWebC-PaLM-62B 88.7 90.5 62 B 5% Code (0.875% HTML)\\nT5-large 76.4 75.2 800 M\\n0% T5-3B 77.2 73.8 3 B\\nPaLM-8B 73.3 70.1 8 B\\nTable 2: LLMs performance on the Semantic Classiﬁcation task. Fine-tuning off-the-shelf pretrained LLMs\\n(model names with preﬁx ‘Web*’) helps LLMs transfer better compared to training the same architecture from\\nscratch on the HTML dataset (model names without preﬁx ‘Web*’), improving the accuracy of PaLM-8B more\\nthan 12%. While WebC-PaLM-62B clearly performed better than all other models, we found WebC-T5-large\\nto be competitive with much larger models such as WebC-LaMDA-1B or WebC-PaLM-8B.\\nsuccess 16% while only training on 12K publicly-available demonstrations, yielding over 192x im-\\nprovement in sample-efﬁciency. We ﬁnd that all choices of LLMs outperform previous SL models.\\nNotably, WebN-T5-3B signiﬁcantly improves on websites requiring multiple-action sequences such\\nasclick checkboxes or websites requiring entering text such as login user (Table 8). We observe that\\nthe performance of LLMs is only surpassed by previous works utilizing RL, which require orders of\\nmagnitude more online experience interaction. Extending our ﬁne-tuned LLMs to an RL setting is\\na promising avenue for future work.\\nAction history ablation. Across all LLMs we consistently observe a decrease in success, on av-\\nerage 6.4%, when past actions are excluded from the inputs (Figure 2a). Action history helps with\\nwebsites that require entering multiple texts, as well as understanding minor changes that could be\\ndifﬁcult to detect (e.g. click checkboxes andmulti layout ).multi layout requires entering 3 different\\ntexts in the website where the layout is randomized at each episode, yet, surprisingly, even the (rel-\\natively smaller) WebN-T5-large model without action history outperforms the CC-Net (SL) model;\\nillustrating that incorporating action history is not the only contributing factor for the better success.\\n7\\nCategories Figure 3: Accuracy per classiﬁcation category of the WebC-T5-3B model on the development dataset.\\nNew Height Test (%) Dev (%)\\ndescendants (%)\\n25 3 87.7 90.3\\n25 4 88.6 89.2\\n50 3 88.4 90.0\\n50 4 89.3 89.2\\n300 5 87.8 88.8\\n500 7 75.8 74.5\\n(a)\\nData SizeAccuracy\\n55606570758085\\n500 1000 1500 2000WebC-PaLM WebC-T5-3B\\nT5-3B (full data / no pretraining) (b)\\nFigure 4: a) Effect of snippet extraction parameters on WebC-T5-3B. Increases above 50% in new descendants\\nand height of 4. Large increases in both parameters lead to large snippets and decrease in accuracy. b) Accu-\\nracy over training data size. Using only 1000 labeled examples (4.4% of all training dataset), WebC-T5-3B\\noutperforms T5-3B (full data without pretraining) which is trained on allavailable labeled data (approximately\\n30k examples), and outperforms WebC-PaLM-8B which is an order of magnitude larger.\\n8.2 S EMANTIC CLASSIFICATION TASK RESULTS\\nTo evaluate the Semantic Classiﬁcation task, we compare the T5 encoder-decoder architecture’s\\nthree size variants (WebC-T5-base, WebC-T5-large, and WebC-T5-3B) ﬁne-tuned on 22K real,\\nhuman-labeled training websites. We compare with a ﬁne-tuned encoder only architectures\\n(WebC-*BERT*), three ﬁne-tuned decoder-only architectures (WebC-LaMDA and PaLM), and both\\nencoder-decoder and decoder-only models trained on human labeled websites from scratch. Results\\nare presented in Table-2, where we ﬁnd that all WebC-LLMs perform well and signiﬁcantly better\\nthan the same architectures without pretraining.\\nAccuracy per category. In Figure 3, we present accuracy distribution of the WebC-T5-3B model\\non the development dataset. The ﬁne-tuned encoder-decoder model performs strongly on a majority\\nof the categories (Figure 3), even on those with very few samples. For instance, the model is 100%\\naccurate on password newwhich has only 56 training examples, because the class is unambiguous.\\nOn the other hand, unsurprisingly, the performance drops when the category is ambiguous, such as\\nin the email category which is frequently mistaken as username .\\nSnippet generation ablation. Two hyper-parameters govern snippet generation: percentage of\\nnew descendants and height of the new root. While small variations of both parameters do not\\nchange the performance, increasing both degrades the performance signiﬁcantly (Table 4a). With\\nnew descendants up to 500% and height up to 7, the performance drops by more than 15%. Note\\nthat snippet generation returns the full-page HTML when both parameters increase indeﬁnitely.\\nData size impact. When varying the ﬁne-tuning training data sizes (1, 5, 10, 20, or 50 samples per\\nclass) in Figure 4b, WebC-T5-3B slightly outperforms WebC-PaLM-8B which is an order of mag-\\nnitude larger. Compared to T5-3B that is trained on all available HTML data without pretraining,\\nWebC-T5-3B achieves better performance while using only 3.4% of labeled data (1000 samples),\\n8\\nTest Dev\\nModel Name Accuracy (%) BLEU ROUGE-1 Accuracy (%) BLEU ROUGE-1\\nWebD-T5-large 83.2 90.2 90.5 84.3 91.7 91.5\\nWebD-LaMDA-1B 83.3 87.5 90.2 84.3 88.6 91.2\\nWebD-T5-3B 84 90.8 90.9 85.2 92.1 91.9\\nClosest Description 57.4 24.4 59.2 60.8 23.9 62.1\\nTable 3: Description generation accuracy of LLMs.\\nthus highlighting the beneﬁt of using standard off-the-shelf pretrained LLMs for HTML understand-\\ning.\\n8.3 D ESCRIPTION GENERATION TASK RESULTS\\nForDescription Generation we split the CommonCrawl dataset based on URL top-level domains to\\ntest LLMs’ capabilities to generalize to unseen HTML. We ﬁne-tune encoder-decoder architectures\\n(WebD–T5*) and decoder-only models (WebD–LaMDA*), with results presented in Table 3. We\\nalso evaluate a strong heuristic baseline which simply ﬁnds the description closest to the salient\\nelement in the HTML text (Closest Description).\\nAccuracy and Similarity Performance We show results of our evaluations in Table 3. All models\\nachieve high scores across all metrics, achieving \\x1984% on the accuracy in terms of exact match and\\na higher non-exact match score based on BLEU and ROUGE-1 ( \\x1991%). This difference indicates\\nthat the models are capable of locating the descriptions, but not always generating the exact output.\\n8.4 HTML U NDERSTANDING LLM SPERFORMANCE ANALYSIS ACROSS TASKS\\nWe now analyze our results in aggregate to derive our main conclusions.\\n8.4.1 P RETRAINING EFFECT : PRETRAINING ON LARGE TEXT CORPORA MATTERS\\nFine-tuned pretrained LLMs outperform LLMs trained on HTML-only data, improving the perfor-\\nmance by more than 34.1% on the Autonomous Web Navigation (Table 2b), and 10% to 12.7% on\\ntheSemantic Classiﬁcation task (Table 2).\\nSince Autonomous Web Navigation is the most difﬁcult task, the improved performance is an en-\\ncouraging evidence of the value of LLMs in HTML understanding tasks. Speciﬁcally, we observe\\nthat LLMs without pretraining are comparable to ﬁne-tuned pretrained models only on websites that\\nrequire simple text matching. In contrast, for websites such as click checkboxes , text matching is\\nharder and we ﬁnd that pretraining is key to good performance. We also found that without pretrain-\\ning, model outputs were frequently in an incorrect format such as invalid dictionaries or invalid refs\\nwith non-integer values. This suggests that the large corpora used for pretraining helps models to\\nlearn general HTML structure.\\n8.4.2 A RCHITECTURE EFFECT : T5- BASED MODELS PERFORM BESTACROSS ALLTASKS\\nEncoder-decoder T5 based models perform better across all three tasks. On the Autonomous Web\\nNavigation task, encoder-decoder (WebN-T5) architectures are better or comparable to WebN-\\nLaMDA-1B (Figure 2a). On the Semantic Classiﬁcation , the smallest encoder-decoder model\\n(WebC-T5-base) performs comparably to much larger decoder-only models (WebC-LaMDA-1B or\\nWebC-PaLM-8B) and the largest encoder-only model (WebC-BERT-large) which has 85M more pa-\\nrameters (Table 2). We also observe that decoder-only PaLM-8B performs worse than much-smaller\\nencoder-decoder T5-large when trained only on HTML data. Finally, on the Description Generation\\nencoder-decoder architecture has higher BLEU score.\\nOne possible explanation for the strong performance of T5-based moels is the encoder-decoder\\narchitecture of these models. Namely, T5 models utilize an encoder with a bidirectional attention\\nmechanism, not present in the LaMDA and PaLM decoders. The bidirectional attention mechanism\\ncan process HTML pages from both ends, potentially overcoming the loss of information when\\ntree-structured HTML pages are converted into a ﬁxed linear text sequences.\\n9\\n8.4.3 M ODEL SIZEEFFECT : SIZE(SUB-LINEARLY ) M ATTERS\\nAcross the tasks it appears that the architecture plays an important role in the model performance.\\nModel size and performance are also positively correlated, although they reach diminishing returns.\\nFor instance, the model performance is roughly O(log log n)with respect to model size on Seman-\\ntic Classiﬁcation (Figure 4b in Appendix). On the Autonomous Web Navigation task, performance\\ngrows slowly with the model size (Table 8), while on the Description Generation it plateaus (Ta-\\nble 3).\\n8.5 D ISCUSSION\\nBi-directional attention vs training corpora: Pretraining on large corpora matters, yielding \\x144.5x\\nperformance improvements. Larger models tend to be better and we credit the bidirectional attention\\nfor T5’s best overall performance across the tasks. PaLM and LaMDA include HTML and other\\ncode in their pretraining corpora, while BERT and T5 architectures did not, showing that pretraining\\non HTML is not necessary for strong performance when ﬁne-tuned for HTML understanding. This\\nstrengthens the hypothesis behind the role of the bidirectional attention, and opens up the possibility\\nto further improve the performance of T5 architectures by pretraining them on corpora with HTML.\\nPractical impact on labeling: When available, the pretrained LLMs need very little new expert\\ndata (200x and 30x reduction on the web navigation and classiﬁcation tasks, respectively). This has\\na big potential impact on practical applications, reducing the data collection time and cost by orders\\nof magnitude.\\nBigger is not always better: When choosing the model size, the expected performance gains (sub-\\nlinear at best and asymptotic at worst) should be considered alongside the model’s training and\\ninference time and cost. For instance, on the classiﬁcation task, the largest model WebC-PaLM-62B\\ntakes several days to ﬁne-tune, and evaluates at 30 Hz, while WebC-T5-large ﬁne-tunes in several\\nhours and evaluates at 700 Hz – an order of magnitude more expensive for a single percent uplift in\\naccuracy. BERT models on the other hand train in minutes. If the application does not require high\\nprecision, these might be a good choice.\\nContext window is a bottleneck: The major bottleneck for the HTML understanding tasks seems to\\nbe the context window length that the current LLMs support, even with models that accept 1000+ to-\\nkens. It remains prohibitive to evaluate web navigation tasks on real websites that are orders of mag-\\nnitude larger than pages in MiniWob. Similarly, we observed that increasing the snippet size leads\\nto major performance degradation. This makes HTML understanding an interesting benchmark for\\nfuture LLM development. For instance, new methods may need to be developed to compress the\\nstate representation of web content for use in LLM context windows.\\n9 C ONCLUSION\\nWe presented canonical tasks and ﬁne-tuned LLMs for HTML understanding. The comprehensive\\nevaluations and analyses over a range of architectures, dataset sizes, and baselines yields practical\\nﬁndings and highlights current limitations of these models. We ﬁnd that a) pretraining is critical for\\nthe performance and can reduce labeled data requirements, improving sample efﬁciency up to 200x;\\nb) model architecture is the second-most important factor, and T5 models with bidirectional attention\\nand encoder-decoder architecture perform the best across the board; c) given a choice, model size\\nshould be evaluated in the context of the model’s training and inference performance, as the model\\nsize sub-linearly correlates with its performance. Finally, the proposed HTML understanding tasks\\nhighlight the relatively short context window that limits current LLMs, suggesting possibilities for\\nfuture research that incorporate or eliminate this constraint.\\nREFERENCES\\nLeonard Adolphs, Benjamin Boerschinger, Christian Buck, Michelle Chen Huebscher, Massimil-\\niano Ciaramita, Lasse Espeholt, Thomas Hofmann, and Yannic Kilcher. Boosting search engines\\nwith interactive agents. arXiv preprint arXiv:2109.00527 , 2021.\\n10\\nArmen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke\\nZettlemoyer. Htlm: Hyper-text pre-training and prompting of language models. arXiv preprint\\narXiv:2107.06955 , 2021.\\nMichael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea\\nFinn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say:\\nGrounding language in robotic affordances. arXiv preprint arXiv:2204.01691 , 2022.\\nJacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan,\\nEllen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language\\nmodels. arXiv preprint arXiv:2108.07732 , 2021.\\nRishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx,\\nMichael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu-\\nnities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.\\nTom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,\\nArvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are\\nfew-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020.\\nAndrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A Plummer.\\nInteractive mobile app navigation with uncertain or under-speciﬁed natural language commands.\\narXiv preprint arXiv:2202.02312 , 2022.\\nAakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam\\nRoberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm:\\nScaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.\\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep\\nbidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.\\nOscar Diaz, Itziar Otaduy, and Gorka Puente. User-driven automation of web form ﬁlling. In\\nInternational Conference on Web Engineering , pp. 171–185. Springer, 2013.\\nIzzeddin Gur, Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and\\nAleksandra Faust. Environment generation for zero-shot compositional reinforcement learning.\\nAdvances in Neural Information Processing Systems , 34:4157–4169, 2021.\\nZecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schu-\\nbiner, Ruby Lee, and Jindong Chen. Actionbert: Leveraging user actions for semantic under-\\nstanding of user interfaces. In Proceedings of the AAAI Conference on Artiﬁcial Intelligence ,\\nvolume 35, pp. 5931–5938, 2021.\\nPeter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair\\nMuldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven\\napproach for learning to control computers. In International Conference on Machine Learning ,\\npp. 9466–9482. PMLR, 2022.\\nSheng Jia, Jamie Ryan Kiros, and Jimmy Ba. DOM-q-NET: Grounded RL on structured lan-\\nguage. In International Conference on Learning Representations , 2019. URL https://\\nopenreview.net/forum?id=HJgd1nAqFX .\\nChuck Jorgensen and Kim Binsted. Web browser control using emg based sub vocal speech recog-\\nnition. In Proceedings of the 38th Annual Hawaii International Conference on System Sciences ,\\npp. 294c–294c. IEEE, 2005.\\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child,\\nScott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language\\nmodels. arXiv preprint arXiv:2001.08361 , 2020.\\nZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori-\\ncut. Albert: A lite bert for self-supervised learning of language representations. In International\\nConference on Learning Representations , 2020.\\n11\\nChenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. Structurallm:\\nStructural pre-training for form understanding. arXiv preprint arXiv:2105.11210 , 2021a.\\nJunlong Li, Yiheng Xu, Lei Cui, and Furu Wei. Markuplm: Pre-training of text and markup language\\nfor visually-rich document understanding. arXiv preprint arXiv:2110.08518 , 2021b.\\nEvan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement\\nlearning on web interfaces using workﬂow-guided exploration. arXiv preprint arXiv:1802.08802 ,\\n2018.\\nKevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal\\ncomputation engines. arXiv preprint arXiv:2103.05247 , 2021.\\nSahisnu Mazumder and Oriana Riva. Flin: A ﬂexible natural language interface for web navigation.\\narXiv preprint arXiv:2010.12844 , 2020.\\nReiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo-\\npher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted\\nquestion-answering with human feedback. arXiv preprint arXiv:2112.09332 , 2021.\\nRodrigo Nogueira and Kyunghyun Cho. End-to-end goal-driven web navigation. Advances in neural\\ninformation processing systems , 29, 2016.\\nChristopher Olston, Marc Najork, et al. Web crawling. Foundations and Trends® in Information\\nRetrieval , 4(3):175–246, 2010.\\nPanupong Pasupat, Tian-Shun Jiang, Evan Zheran Liu, Kelvin Guu, and Percy Liang. Mapping\\nnatural language commands to web elements. arXiv preprint arXiv:1808.09132 , 2018.\\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi\\nZhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a uniﬁed text-to-text\\ntransformer. J. Mach. Learn. Res. , 21(140):1–67, 2020.\\nTianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An\\nopen-domain platform for web-based agents. In International Conference on Machine Learning ,\\npp. 3135–3144. PMLR, 2017.\\nZhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobile-\\nBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th\\nAnnual Meeting of the Association for Computational Linguistics . Association for Computational\\nLinguistics, 2020.\\nRomal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze\\nCheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven\\nZheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin,\\nJames Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi\\nZhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-\\nHellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So-\\nraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson,\\nAlejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna,\\nMatthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil,\\nBlaise Aguera-Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. Lamda: Language\\nmodels for dialog applications. CoRR , 2022.\\nDaniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali\\nAhmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: a reinforcement learn-\\ning platform for android. arXiv preprint arXiv:2105.13231 , 2021.\\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,\\nŁukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa-\\ntion processing systems , 30, 2017.\\n12\\ne xpand \\no ne l e v el \\nup s ali en t \\nel emen t s s nip pe t \\ng ener a t i o n <html> \\n   <body> \\n      <form class= \"login-form\" >\\n         <div> \\n            <label class= \"form-label\" for= ”uName” >\\n               Enter Email Address \\n            </label> \\n      <label class= \"form-label\" for= ”pass” >\\n               Enter Password: \\n            </label> \\n         </div> \\n         <div> \\n  <input type= \"email\"  id=\"uName” >\\n            <input type= \"password\"  id=\"pass\" >\\n            <span class= \"hidden\" >\\n               Please enter your password. \\n            </span> \\n         </div> \\n         <button type= \"submit\" >Sign In </button> \\n       </form> \\n   </body> \\n</html> HTML \\n<input  name= \"uName\" >\\n<input  name= \"pass\" >\\n<button  type= \"submit\" ><input type= \"email\"  id=\"uName” >if e xpand ab l e : \\ne xpand \\n<div> \\n  <input type= \"email\"  id=\"uName” >\\n  <input type= \"password\"  id=\"pass\" >\\n  <span class= \"hidden\" >\\n     Please enter your password. \\n  </span> \\n</div> o t her wis e \\no u t p u t <input type= \"email\"            \\nid=\"uName” target >\\nFigure 5: High-level overview of our pre-processing pipeline for generating snippets from a full HTML web-\\npage. Given the page, we detect salient elements and for each one of them we extract snippets by recursively\\nmoving up in the HTML tree until a validation heuristic fails.\\nThomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,\\nPierric Cistac, Tim Rault, R ´emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface’s\\ntransformers: State-of-the-art natural language processing. CoRR , abs/1910.03771, 2019. URL\\nhttp://arxiv.org/abs/1910.03771 .\\nLinting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya\\nBarua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv\\npreprint arXiv:2010.11934 , 2020.\\nShunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-\\nworld web interaction with grounded language agents. arXiv preprint arXiv:2207.01206 , 2022.\\nA A PPENDIX\\nA.1 D ATASET DETAIL\\nExamining the description distribution, we found the original 400Kdataset to be very skewed; only\\n20 descriptions (such as Email andPassword ) were covering 50% of the dataset. We sub-sampled the\\ndataset so that each unique description has at most 10 data points. We also found that for attributes\\nare almost always deﬁned for HTML label s. This could cause a model to overﬁt and just ﬁnd the\\nlabel element in the HTML and ignore everything else. To avoid this sort of ‘cheating’ we replace\\nthe tags of HTML label s by randomly sampling from fdiv, span, a, label g. These tags\\nare also frequently used to inject text in HTML but they are very rarely used with for attributes.\\nFinally, we removed examples where there are only a single text in the HTML since models can\\ntrivially generate descriptions by ﬁnding the only text in the HTML, which biases model weights\\nand evaluation metrics. After this ﬁnal step, we have a total of 85Klabeled examples.\\nA.1.1 S NIPPET GENERATION\\nIn Figure 5, we give a high-level overview of our snippet generation procedure.\\nA.2 A DDITIONAL RESULTS\\nA.2.1 S EMANTIC CLASSIFICATION\\nError Analysis. We manually examined 50 errors of T5-3B model over the development set (Ta-\\nble 4) and assigned them into one of the 9 error types that we devised. We found that 32% of the\\nerrors are due to lack of information in the HTML snippets, which is mainly the result of lost in-\\nformation during snippet extraction process. Annotation errors or email/username ambiguity make\\nup 30% of the errors. These can’t be improved without revising the annotated data or adding extra\\ninformation to resolve the ambiguity. We also found that the model sometimes picks a more general\\ncategory, or a nearby text misleads the model; the latter usually happens when the HTML snippet is\\nlong where majority of the elements are noise.\\n13\\nError Type Percentage of Examples\\nNot enough information in the HTML snippet 30\\nIncorrect annotation (ex: ”unknown role” instead of ”organization”) 12\\nAnnotation tool translates user selection incorrectly 8\\nEmail/Username ambiguity 10\\nMore general category (ex: ”header” instead of ”cart header”) 8\\nImmediate neighboring text misleads 8\\nIncorrect date formatting (ex: ”mm” instead of ”mmm”) 4\\nNo information in the HTML snippet 2\\nOthers 18\\nTable 4: Types of errors over 50 manually examined examples. 32% of errors are due to lack of information\\nin HTML snippets, 30% of errors are related to annotations or can’t be improved due to ambiguity (email/user-\\nname), and the remaining errors are incorrect predictions by the model.\\nFew-Shot Prompting In Table 5, we present few-shot prompting performance of a 540B PaLM\\nmodel. We probe the model using a prompt template <html> Role: <category> with 1 ex-\\nample per category and generate categories using greedy-decoding. In our preliminary experiments,\\nwe found that few-shot prompting achieves only 45.6 accuracy, much lower than a model ﬁne-tuned\\non the same data (Figure 6). We found two common problems – the model is not able to canonicalize\\npredictions into categories and many of the examples are dropped due to context length.\\nModel Name Test Dev\\nPaLM-540B 64.2 60.3\\n- w/o Example Cleaning 57.9 57.2\\n- w/o Category Rewriting 52.1 50.7\\n- w/o Dictionary Mapping 45.6 45.1\\nTable 5: Few-shot prompting performance with differ-\\nent pre- and post-processing steps.We developed post-processing methods to al-\\nleviate the canonicalization problem and pre-\\nprocessing methods to reduce lengths of ex-\\namples. Adding a dictionary-based mapping\\non predictions – a manually curated paraphrase\\ndictionary – improves the performance to 52.1.\\nWe also tried rewriting predictions by chang-\\ning the order of tokens around ” ” such as\\nname ﬁrsttoﬁrst name which further improved\\nthe performance to 57.9. Finally, we cleaned\\nexamples in the prompt by removing certain el-\\nements such as ”svg”, ”path”, ”img” , and ”iframe” and also removing class attribute from every\\nelement; this pre-processing step gives 64.2.\\nFigure 6: Performance comparison w.r.t. increasing model size. As the model size increases, we\\nobserve an increase in overall accuracy with PaLM-62B model achieving the highest accuracy while\\nbeing 7x larger than PaLM-8B.\\n14\\nA.3 S AMPLE EPISODES FROM MINIWOB\\nSee Table 6 for an example episode of web navigation inferred by a ﬁne-tuned LLM.\\nA.4 D ETAILED MINIWOB R ESULTS\\nSee Table 7 for detailed performance of various models on MiniWob.\\nA.5 R ESOURCE REQUIREMENTS\\nSee Table 8.\\nA.6 S TRUCTURE DEPENDENCE ABLATION STUDY\\nWe conducted an ablation study to examine the sensitivity of model performance to preserving\\nstructural information. To do so, we evaluate the model’s performance on HTML input with criti-\\ncal structure components removed. We kept the order of elements and their attributes ﬁxed while\\ncorrupting the nesting structure by removing closing tags.\\nRemoving closing tags corresponds to a valid traversal (BFS) and keeps the order of elements the\\nsame as the text based input.\\nAs a simple example:\\n<div id=\"form\"><div><input id=\"username\"></div></div>\\nwould be converted into:\\n<div id=\"form\"><div><input id=\"username\">\\nWe evaluated the trained WebN-T5-3B model on the same set of synthetic websites from the\\nMiniWoB benchmark with this aspect of structure removed from the HTML pages. WebN-T5-\\n3B achieves a 45.4% success rate, 6% lower than before, suggesting that WebN-T5-3B is at least\\npartially dependent on the DOM topology.\\nA.7 T ASK-SPECIFIC MODELS\\nAn alternative to LLMs is to adapt bespoke task-speciﬁc architectures tailored towards processing\\nof structured documents and HTML (Li et al. (2021b;a)).\\nStructuralLM (Li et al. (2021a)) is an approach speciﬁcally tailored for document understanding\\n(i.e., combinations of images and text), and thus makes several simplifying assumptions for its model\\nthat limit its applicability to HTML understanding (i.e., trees of elements with a richer structure and\\nfunctionality). It is trained only on the textual content of a document - the markup information is\\nignored. For example, any input ﬁeld or dropdown in a document would be missing from the model\\ninputs. All of the tasks we study require knowledge of this information. For example, in autonomous\\nnavigation the model needs to interact with input elements (e.g. text, checkboxes, dropdowns) such\\nas username and password in the login-user task in MiniWoB. Typically, a “type” action with a\\nreference to an element and a text argument is generated by the model. Without knowing which\\ninput elements are available in the page, it is impossible to generate a reference to any input element.\\nWhile MarkupLM (Li et al. (2021b)) is better tailored for understanding HTML pages, it has similar\\ndrawbacks as StructuralLM in that it focuses solely on text and structure of text while ignoring\\neverything else in the markup. To illustrate our point better, we used the open source implementation\\nof MarkupLM from the HuggingFace library (Wolf et al. (2019)) to process the sample HTML\\nsnippet in Figure-1(b). The MarkupLM ignores all input elements, both username and password,\\nand generates <s>Email AddressEnter Password:Please enter your password. </s>which is the\\ntext input to the MarkupLM Transformer. Classifying this text as username or password is not\\npossible without the additional context on which input element is the salient element (in this context\\nit is the username). See below for the code to reproduce our result.\\n15\\nfrom transformers import MarkupLMProcessor\\nprocessor = MarkupLMProcessor.from_pretrained(f\"microsoft/markuplm-base\")\\nsnippet = ’’’<div><label class=\"form-label\" for=\"uName\">Email Address\\n</label><label class=\"form-label\" for=\"pass\">Enter Password:\\n</label></div><div><input type=\"email\" id=\"uName\" target><input\\ntype=\"password\" id=\"pass\"><span class=\"hidden\">Please enter your password.\\n</span></div>’’’\\nencoding = processor(snippet)\\nprint(processor.batch_decode(encoding[\"input_ids\"]))\\nMarkupLM is also evaluated on NLP-like tasks such as QA or entity classiﬁcation where understand-\\ning page content is paramount, whereas we focus on HTML understanding tasks such as autonomous\\nnavigation where both content and the page’s layout structure need to be understood.\\nWe perform a quantitative evaluation of MarkupLM on our tasks to understand how signiﬁcant\\nthese limitations are. We ﬁne-tune the MarkupLM-base model on the semantic classiﬁcation task,\\nusing the same setup as other WebC models but with the suggested hyperparameters from (Li et al.\\n(2021b)). We use the MarkupLM implementation from the HuggingFace library (Wolf et al. (2019)).\\nOn development and test sets, MarkupLM-base achieves 65% and 66% accuracy, respectively. These\\nresults are more than 16% lower compared to similar size WebC-BERT-base results that we report\\nin our work. This suggests that although domain speciﬁc models may be suitable for processing\\nHTML for NLP tasks, the generality, ﬂexibility, and sample efﬁciency LLMs provide advantages\\nfor autonomous navigation tasks.\\n16\\nTable 6: A sample web page and corresponding episode using the T5-3B model. At each time step,\\nprevious actions, instruction, and HTML are concatenated into a single HTML text. Note that at the\\nbeginning of episode, there is no past actions and we simply concatenate instruction and HTML.\\nAction is generated as a sequence of tokens which is later parsed into a dictionary. The refin the\\naction points to an element that has a refattribute with the same value. For instance, at the beginning\\nof episode, ref: 6 corresponds to an input with ref=6 . At the end of the episode, the model clicks on\\nthe submit button and the episode terminates.\\nWeb page\\nHTML Text Action Text\\nfaction: click, ref: 6 g\\nfaction: click, ref: 10 g\\nfaction: click, ref: 12 g\\nfaction: click, ref: 14 g 17\\nfaction: click, ref: 16 g\\nfaction: click, ref: 17 g\\n18\\nTable 7: Success rate comparison of various models in MiniWoB tasks. Baseline results are borrowed from\\n(Humphreys et al., 2022). Note that these are normalized between 0 and 1.\\nTASK Human CC-Net CC-Net World Workﬂow Learning DOM-Q-Net Workﬂow Learning Aggregated Aggregated\\nWebN-T5-3B WebN-T5-3B (SL & RL) (SL) of guided to (RL) guided to SOTA SOTA\\n(no history) bits exploration navigate exploration navigate (SL & RL) (Augmented)\\n(SL & RL) (SL & RL) the web (Augmented) the web\\n(RL) (Augmented)\\nbisect-angle 0.92 n/a n/a 0.97 0.29 0.8 n/a n/a n/a n/a n/a 0.8 0.8\\nbook-ﬂight 0.87 0 0 0.87 0 0 0 n/a n/a 0 1 0 1\\nchase-circle 0.82 n/a n/a 0.93 0.8 1 n/a n/a n/a n/a n/a 1 1\\nchoose-date-easy 0.99 0.03 0.05 0.99 0.42 n/a n/a n/a n/a n/a n/a n/a n/a\\nchoose-date-medium 0.98 0 0 0.99 0.26 n/a n/a n/a n/a n/a n/a n/a n/a\\nchoose-date 0.97 0 0 0.97 0.12 0 0 n/a 1 0 n/a 1 1\\nchoose-list 0.98 0.26 0.14 0.99 0.19 0.25 0.16 0.26 n/a 0.16 0.26 0.26 0.26\\ncircle-center 0.96 n/a n/a 0.97 0.36 0.98 n/a n/a n/a n/a n/a 0.98 0.98\\nclick-button-sequence 0.94 1 1 1 0.47 0.22 0.99 n/a 1 1 n/a 1 1\\nclick-button 0.98 1 0.96 1 0.78 0.62 1 1 1 1 1 1 1\\nclick-checkboxes-large 0.87 0.22 0 0.71 0 n/a 0.68 n/a n/a 0.84 n/a 0.68 0.84\\nclick-checkboxes-soft 0.73 0.54 0.43 0.95 0.04 n/a 0.51 n/a n/a 0.94 n/a 0.51 0.94\\nclick-checkboxes-transfer 0.98 0.63 0.34 0.99 0.36 n/a 0.64 n/a n/a 0.64 n/a 0.64 0.64\\nclick-checkboxes 0.97 0.96 0.84 0.98 0.32 0.48 0.98 n/a 1 1 n/a 1 1\\nclick-collapsible-2 0.97 0 0.01 0.98 0.17 0.11 0.65 n/a n/a 0.99 n/a 0.65 0.99\\nclick-collapsible 0.99 0 0.01 1 0.81 0.98 1 1 n/a 1 1 1 1\\nclick-color 0.97 0.27 0.23 1 0.82 0.23 1 n/a n/a 1 n/a 1 1\\nclick-dialog-2 0.99 0.24 0.35 1 0.88 0.53 1 n/a n/a 1 n/a 1 1\\nclick-dialog 1 1 1 1 0.95 1 1 1 1 1 1 1 1\\nclick-link 0.99 1 0.96 0.99 0.59 0.31 1 1 1 1 1 1 1\\nclick-menu-2 0.98 n/a n/a 0.83 0.52 0.16 n/a n/a n/a n/a n/a 0.16 0.16\\nclick-menu 0.97 0.37 0.38 0.94 0.22 0.13 n/a n/a n/a n/a n/a 0.13 0.13\\nclick-option 0.99 0.87 0.78 0.99 0.21 0.28 1 n/a 1 1 n/a 1 1\\nclick-pie 0.98 0.51 0.14 0.97 0.15 0.15 0.32 1 n/a 0.32 1 1 1\\nclick-scroll-list 0.91 0 0 0.6 0.01 0.07 n/a n/a n/a n/a n/a 0.07 0.07\\nclick-shades 0.91 0 0 1 0.04 0.27 0.22 n/a n/a 0.99 n/a 0.27 0.99\\nclick-shape 0.88 0.53 0.54 0.95 0.11 0.11 0.64 n/a n/a 0.64 n/a 0.64 0.64\\nclick-tab-2-easy 0.99 n/a n/a 0.99 0.61 n/a n/a n/a n/a n/a n/a n/a n/a\\nclick-tab-2-hard 0.96 0.12 0.13 0.98 0.19 n/a n/a n/a n/a n/a n/a n/a n/a\\nclick-tab-2-medium 0.97 n/a n/a 0.99 0.54 n/a n/a n/a n/a n/a n/a n/a n/a\\nclick-tab-2 0.97 0.18 0.09 0.98 0.27 0.08 0.64 n/a 1 0.98 n/a 1 1\\nclick-tab 0.99 0.74 1 1 0.95 0.97 0.55 1 1 1 1 1 1\\nclick-test-2 0.99 1 1 1 0.95 0.83 1 n/a 1 1 n/a 1 1\\nclick-test-transfer 0.99 n/a n/a 1 0.94 n/a n/a n/a n/a n/a n/a n/a n/a\\nclick-test 1 1 1 1 1 1 1 n/a 1 1 n/a 1 1\\nclick-widget 0.83 1 0.97 1 0.56 0.34 0.93 n/a 1 0.93 n/a 1 1\\ncopy-paste-2 0.94 n/a n/a 0.63 0.01 0 n/a n/a n/a n/a n/a 0 0\\ncopy-paste 0.94 n/a n/a 0.79 0.04 0 n/a n/a n/a n/a n/a 0 0\\ncount-shape 0.82 0.41 0.43 0.85 0.21 0.18 0.59 n/a n/a 0.76 n/a 0.59 0.76\\ncount-sides 0.98 n/a n/a 1 0.74 0.3 n/a n/a n/a n/a n/a 0.3 0.3\\ndrag-box 0.99 n/a n/a 1 0.61 0.31 n/a n/a n/a n/a n/a 0.31 0.31\\ndrag-cube 0.99 n/a n/a 0.79 0.23 0.18 n/a n/a n/a n/a n/a 0.18 0.18\\ndrag-item 0.98 n/a n/a 1 0.61 n/a n/a n/a n/a n/a n/a n/a n/a\\ndrag-items-grid 0.87 n/a n/a 0.98 0.05 0.01 n/a n/a n/a n/a n/a 0.01 0.01\\ndrag-items 0.93 n/a n/a 0.99 0.13 0.41 n/a n/a n/a n/a n/a 0.41 0.41\\ndrag-shapes 0.96 n/a n/a 0.99 0.26 0.92 n/a n/a n/a n/a n/a 0.92 0.92\\ndrag-sort-numbers 0.92 n/a n/a 0.97 0.11 0.66 n/a n/a n/a n/a n/a 0.66 0.66\\nemail-inbox-delete 0.99 n/a n/a 1 0.22 n/a n/a n/a 1 n/a n/a 1 1\\nemail-inbox-forward-nl-turk 0.88 0.33 0.09 1 0 n/a n/a n/a n/a n/a n/a n/a n/a\\nemail-inbox-forward-nl 0.91 0.60 0.09 1 0 n/a n/a n/a n/a n/a n/a n/a n/a\\nemail-inbox-forward 0.96 n/a n/a 1 0.01 n/a n/a n/a n/a n/a n/a n/a n/a\\nemail-inbox-important 0.99 n/a n/a 1 0.3 n/a n/a n/a n/a n/a n/a n/a n/a\\nemail-inbox-nl-turk 0.93 0.23 0.26 1 0.05 n/a 0.77 n/a n/a 0.93 n/a 0.77 0.93\\nemail-inbox-noscroll 0.96 n/a n/a 1 0.13 n/a n/a n/a n/a n/a n/a n/a n/a\\nemail-inbox-reply 0.91 n/a n/a 1 0 n/a n/a n/a n/a n/a n/a n/a n/a\\nemail-inbox-star-reply 0.95 n/a n/a 1 0.11 n/a n/a n/a n/a n/a n/a n/a n/a\\nemail-inbox 0.96 0.38 0.21 1 0.09 0.03 0.43 n/a 0.54 0.99 n/a 0.54 0.99\\nenter-date 0.97 0 0 1 0.02 0.61 0 1 n/a 0.96 1 1 1\\nenter-password 0.96 0.97 0.92 1 0.02 0 0.99 1 1 1 1 1 1\\nenter-text-2 0.91 n/a n/a 0.98 0.04 0 n/a n/a n/a n/a n/a 0 0\\nenter-text-dynamic 0.97 0.98 0.92 1 0.39 1 1 1 1 1 1 1 1\\nenter-text 0.98 0.89 0.99 1 0.35 0 1 n/a 1 1 n/a 1 1\\nenter-time 0.98 0 0.01 0.97 0.04 0.08 0.52 n/a n/a 0.9 n/a 0.52 0.9\\nﬁnd-midpoint 0.94 n/a n/a 0.97 0.35 0.31 n/a n/a n/a n/a n/a 0.31 0.31\\nﬁnd-word 0.96 n/a n/a 0.88 0.05 0 n/a n/a n/a n/a n/a 0 0\\nfocus-text-2 0.99 1 1 1 0.96 0.83 1 n/a 1 1 n/a 1 1\\nfocus-text 1 1 1 1 0.99 0.95 1 n/a 1 1 n/a 1 1\\ngrid-coordinate 0.87 0.49 0.42 1 0.66 0.26 1 n/a n/a 1 n/a 1 1\\nguess-number 0.99 0 0 1 0.21 0.2 0 n/a n/a 0 n/a 0.2 0.2\\nhighlight-text-2 0.97 n/a n/a 1 0.4 0.13 n/a n/a n/a n/a n/a 0.13 0.13\\nhighlight-text 0.97 n/a n/a 1 0.51 0.9 n/a n/a n/a n/a n/a 0.9 0.9\\nidentify-shape 0.98 0.88 0.89 1 0.68 0.36 0.9 n/a n/a 1 n/a 0.9 1\\nlogin-user-popup 0.94 0.72 0.40 1 0.02 n/a n/a n/a n/a n/a n/a n/a n/a\\nlogin-user 0.96 0.82 0.64 1 0 0 0.99 1 1 1 1 1 1\\nmoving-items 0.18 n/a n/a 0.88 0.13 0.78 n/a n/a n/a n/a n/a 0.78 0.78\\nmulti-layouts 0.95 0.83 0.48 1 0 n/a 0.99 n/a n/a 1 n/a 0.99 1\\nmulti-orderings 0.96 0.88 0.64 1 0 n/a 0.05 n/a n/a 1 n/a 0.05 1\\nnavigate-tree 0.98 0.91 0.99 0.99 0.32 0.2 0.99 1 1 0.99 1 1 1\\nnumber-checkboxes 0.96 n/a n/a 0.99 0 0.16 n/a n/a n/a n/a n/a 0.16 0.16\\nread-table-2 0.95 n/a n/a 0.94 0 0 n/a n/a n/a n/a n/a 0 0\\nread-table 0.97 n/a n/a 0.97 0.01 0 n/a n/a n/a n/a n/a 0 0\\nresize-textarea 0.94 n/a n/a 1 0.27 0.11 n/a n/a n/a n/a n/a 0.11 0.11\\nright-angle 0.87 n/a n/a 0.98 0.26 0.38 n/a n/a n/a n/a n/a 0.38 0.38\\nscroll-text-2 0.97 n/a n/a 1 0.88 0.96 n/a n/a n/a n/a n/a 0.96 0.96\\nscroll-text 0.97 n/a n/a 0.96 0.04 0 n/a n/a n/a n/a n/a 0 0\\nsearch-engine 0.97 0.34 0.34 1 0.15 0 0.26 n/a 1 0.99 n/a 1 1\\nsimon-says 0.62 n/a n/a 0 0.02 0.28 n/a n/a n/a n/a n/a 0.28 0.28\\nsimple-algebra 0.86 n/a n/a 0.75 0.03 0.04 n/a n/a n/a n/a n/a 0.04 0.04\\nsimple-arithmetic 0.96 n/a n/a 0.86 0.38 0.07 n/a n/a n/a n/a n/a 0.07 0.07\\nsocial-media-all 0.89 0 0 0.75 0 n/a 0.01 n/a n/a 0.01 1 0.01 1\\nsocial-media-some 0.91 0.02 0 0.85 0.01 n/a 0.01 n/a n/a 0.42 n/a 0.01 0.42\\nsocial-media 0.96 0.21 0.24 0.9 0.03 0.23 0.39 n/a 1 1 n/a 1 1\\nterminal 0.88 n/a n/a -0.01 0 0 n/a n/a n/a n/a n/a 0 0\\ntext-editor 0.88 n/a n/a 0.98 0.11 0.01 n/a n/a n/a n/a n/a 0.01 0.01\\ntext-transform 0.86 n/a n/a 0.6 0.19 0 n/a n/a n/a n/a n/a 0 0\\ntic-tac-toe 0.71 0.48 0.40 0.83 0.32 0.34 0.37 n/a n/a 0.47 n/a 0.37 0.47\\nunicode-test 0.99 n/a n/a 1 0.86 n/a n/a n/a n/a n/a n/a n/a n/a\\nuse-autocomplete 0.98 0.22 0.15 1 0.07 0 0.78 n/a n/a 0.98 n/a 0.78 0.98\\nuse-colorwheel-2 0.94 n/a n/a 0.95 0.38 1 n/a n/a n/a n/a n/a 1 1\\nuse-colorwheel 0.9 n/a n/a 0.98 0.68 1 n/a n/a n/a n/a n/a 1 1\\nuse-slider-2 0.97 n/a n/a 0.95 0.03 0.15 n/a n/a n/a n/a n/a 0.15 0.15\\nuse-slider 0.98 n/a n/a 0.91 0.18 0.51 n/a n/a n/a n/a n/a 0.51 0.51\\nuse-spinner 0.98 0.07 0.05 1 0.47 0.17 0.04 n/a n/a 0.04 n/a 0.17 0.17\\nvisual-addition 0.97 n/a n/a 0.99 0.36 0.01 n/a n/a n/a n/a n/a 0.01 0.01\\n19\\nTable 8: Resource requirements and running time of LLMs.\\nModel Name Model Size TPU version Batch size Input sequence length Examples per sec (training) Examples per sec (inference)\\nPaLM 62B TPU v4 8 1920 9.313 30.51\\nPaLM 8B TPU v4 32 1920 64.4 184.3\\nT5 3B TPU v4 128 512 163.8 734.5\\nLaMDA 1B TPU v2 128 512 363.1 1416\\n20',\n",
       " 'references': [{'id': '2107.06955'},\n",
       "  {'id': '2204.02311'},\n",
       "  {'id': '2204.01691'},\n",
       "  {'id': '2010.11934'},\n",
       "  {'id': '1802.08802'},\n",
       "  {'id': '2110.08518'},\n",
       "  {'id': '2105.11210'},\n",
       "  {'id': '2210.03945'},\n",
       "  {'id': '2109.00527'},\n",
       "  {'id': '2001.08361'},\n",
       "  {'id': '2112.09332'},\n",
       "  {'id': '1810.04805'},\n",
       "  {'id': '2202.02312'},\n",
       "  {'id': '2105.13231'},\n",
       "  {'id': '2108.07732'},\n",
       "  {'id': '2010.12844'},\n",
       "  {'id': '2108.07258'},\n",
       "  {'id': '2207.01206'},\n",
       "  {'id': '1808.09132'},\n",
       "  {'id': '2103.05247'}]}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "66352"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data[0][\"content\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we must format it into the format we need (`[\"id\", \"text\", \"source\", \"metadata\"]`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['id', 'source', 'text', 'metadata'],\n",
       "    num_rows: 423\n",
       "})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = data.map(lambda x: {\n",
    "    \"id\": x[\"id\"],\n",
    "    \"text\": x[\"content\"],\n",
    "    \"source\": x[\"source\"],\n",
    "    \"metadata\": {\n",
    "        \"title\": x[\"title\"],\n",
    "        \"primary_category\": x[\"primary_category\"],\n",
    "        \"published\": x[\"published\"],\n",
    "        \"updated\": x[\"updated\"],\n",
    "    }\n",
    "})\n",
    "# drop uneeded columns\n",
    "data = data.remove_columns([\n",
    "    \"title\", \"summary\", \"content\",\n",
    "    \"authors\", \"categories\", \"comment\",\n",
    "    \"journal_ref\", \"primary_category\",\n",
    "    \"published\", \"updated\", \"references\"\n",
    "])\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8034f6184e6e4b2f8aff0002637524ec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "38070885"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.to_json(\"ai_arxiv.jsonl\", orient=\"records\", lines=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Jump into Canopy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From here we can switch across to Canopy CLI (or other method) and run:\n",
    "\n",
    "```\n",
    "canopy\n",
    "canopy upsert ./ai_arxiv.jsonl\n",
    "```\n",
    "\n",
    "Then we begin chatting by first starting the Canopy Server:\n",
    "\n",
    "```\n",
    "canopy start\n",
    "```\n",
    "\n",
    "Then begin chatting with:\n",
    "\n",
    "```\n",
    "canopy chat\n",
    "```\n",
    "\n",
    "_(we can also add the `--no-rag` flag to see how our RAG vs. non-RAG results compare!)_"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}