index.rss

<?xml version="1.0"?>
<!-- name="generator" content="blosxom/2.0" -->
<!DOCTYPE rss PUBLIC "-//Netscape Communications//DTD RSS 0.91//EN" "http://my.netscape.com/publish/formats/rss-0.91.dtd">

<rss version="0.91">
  <channel>
    <title>Ocean of Awareness   </title>
    <link>http://jeffreykegler.github.io/Ocean-of-Awareness-blog</link>
    <description>Ocean of Awareness.</description>
    <language>en</language>

  <item>
    <title>Infinite Lookahead and Ruby Slippers</title>
    <link>http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2019/06/vgap.html</link>
    <description>&lt;html&gt;
  &lt;head&gt;
  &lt;/head&gt;
  &lt;body&gt;
    &lt;!--
      marpa_r2_html_fmt --no-added-tag-comment --no-ws-ok-after-start-tag
      --&gt;
    &lt;h2&gt;About this post&lt;/h2&gt;
    &lt;p&gt;This post presents a practical, compact example which
    demonstrates a use case for both infinite lookahead
    and Ruby Slippers parsing.
    While the example itself is very simple,
    this post may not be a good first tutorial --
    it focuses on Marpa implementation strategy,
    instead of basics.
    &lt;/p&gt;
    &lt;h2&gt;About Urbit&lt;/h2&gt;
    &lt;p&gt;The example described in this post is one part of
    &lt;tt&gt;hoonlint&lt;/tt&gt;.
    &lt;tt&gt;hoonlint&lt;/tt&gt;, currently under development,
    will be a &quot;lint&quot; program for a language called Hoon.
    &lt;/p&gt;
    &lt;p&gt;
    Hoon is part of
    &lt;a href=&quot;https://urbit.org/&quot;&gt;the Urbit project&lt;/a&gt;.
    Urbit is an effort to return control of the Internet
    experience to the individual user.
    (The Urbit community has, generously, been supporting my work on Hoon.)
    &lt;/p&gt;
    &lt;p&gt;
    The original Internet and its predecessors were cosy places.
    Users controlled their experience.
    Authority was so light you
    could forget it was there,
    but so adequate to its task that you could forget why it
    was necessary.
    What we old timers do remember of the early Internet was the feeling of entering
    into a &quot;brave new world&quot;.
    &lt;/p&gt;
    &lt;p&gt;
The Internet grew beyond our imaginings,
and our pure wonder of decades ago now seems ridiculous.
But the price has been a shift
of power which should be no laughing matter.
Control of our Internet experience now resides in
servers,
run by entities which make no secret of having their own interests.
Less overt, but increasingly obvious, is the single-mindedness with which they pursue
those interests.
&lt;/p&gt;
&lt;p&gt;
And the stakes have risen.
In the early days,
we used the Internet as a supplement in our intellectual lives.
Today, we depend on it in our financial and social lives.
Today, the server-sphere can be a hostile  place.
Going forward it may well become a theater of war.
    &lt;/p&gt;
    We could try to solve this problem by running our own servers.
    But this is a lot of work, and only leaves us in touch with those
    willing and able to do that.  In practice, this seems to be nobody.
    &lt;/p&gt;
    &lt;p&gt;
Urbit seeks to solve these problems with
hassle-free personal servers, called urbits.
Urbits are journaling databases, so they are incorruptable.
To make sure they can be run anywhere in the cloud&lt;a id=&quot;footnote-1-ref&quot; href=&quot;#footnote-1&quot;&gt;[1]&lt;/a&gt;,
they are based on a tiny virtual machine, called Nock.
To keep urbits compact and secure,
Urbit takes on code bloat directly --
Urbit is an original design from a clean slate,
with a new protocol stack.
    &lt;/p&gt;
    &lt;h2&gt;About Hoon&lt;/h2&gt;
    &lt;p&gt;
    Nock's &quot;machine language&quot; takes the form of trees of arbitrary precision integers.
    The integers can be interpreted as strings, floats, etc.,
    as desired.
    And the trees can be interpreted as lists,
    giving Nock a resemblance to a LISP VM.
    Nock does its own memory management
    and takes care of its own garbage collection.&lt;a id=&quot;footnote-2-ref&quot; href=&quot;#footnote-2&quot;&gt;[2]&lt;/a&gt;
    &lt;/p&gt;
    &lt;p&gt;
    Traditionally, there are two ways to enter machine language,
    &lt;ul&gt;
    &lt;li&gt;Physically, for example,
    by toggling it into a machine's front panel.
    Originally, entering it physically was the only way.
    &lt;/li&gt;
    &lt;li&gt;Indirectly, using
    assembler or some higher-level language, like C.
    Once these indirect methods existed, they
    rapidly took over as the most common way to create machine language.
    &lt;/li&gt;
    &lt;/ul&gt;
    Like traditional
    machine language, Nock cannot be written directly.
    Hoon is Urbit's equivalent of C -- it is Urbit's
    &quot;close to the metal&quot; higher level language.
    &lt;/p&gt;
    &lt;p&gt;
    Not that Hoon looks much like C,
    or anything else you've ever seen.
    This is a Hoon program that takes an integer argument,
    call it &lt;tt&gt;n&lt;/tt&gt;,
    and returns the first &lt;tt&gt;n&lt;/tt&gt; counting numbers:
    &lt;pre&gt;&lt;tt&gt;
    |=  end=@                                               ::  1
    =/  count=@  1                                          ::  2
    |-                                                      ::  3
    ^-  (list @)                                            ::  4
    ?:  =(end count)                                        ::  5
      ~                                                     ::  6
    :-  count                                               ::  7
    $(count (add 1 count))                                  ::  8
    &lt;/tt&gt;&lt;/pre&gt;
    &lt;/p&gt;
    &lt;p&gt;
    Hoon comments begin with a &quot;&lt;tt&gt;::&lt;/tt&gt;&quot; and run until the next
    newline.
    The above Hoon sample uses comments to show line numbers.
    &lt;/p&gt;
    &lt;p&gt;
    The example for this post will be
    a &lt;tt&gt;hoonlint&lt;/tt&gt; subset: a multi-line comment linter.
    Multi-line comments are the only Hoon syntax we will talk about.
    (For those who want to know more about Hoon,
    &lt;a href=&quot;https://urbit.org/docs/learn/hoon/&quot;&gt;there is a tutorial&lt;/a&gt;.)
    &lt;/p&gt;
    &lt;p&gt;
    &lt;/p&gt;
    &lt;h2&gt;About Hoon comments&lt;/h2&gt;
    &lt;p&gt;
    In basic Hoon syntax, multi-line comments are free-form.
    In practice, Hoon authors tend to follow a set of conventions.
    &lt;/p&gt;
    &lt;h3&gt;Pre-comments&lt;/h3&gt;
    &lt;p&gt;
    In the simplest case, a comment must precede the code it
    describes, and be at the same indent.
    These simple cases are called &quot;pre-comments&quot;.&lt;a id=&quot;footnote-3-ref&quot; href=&quot;#footnote-3&quot;&gt;[3]&lt;/a&gt;
    For example, this code contains a pre-comment:
    &lt;pre&gt;&lt;tt&gt;
	  :: pre-comment 1
	  [20 (mug bod)]
    &lt;/tt&gt;&lt;/pre&gt;
    &lt;p&gt;
    &lt;h3&gt;Inter-comments&lt;/h3&gt;
    Hoon multi-line comments may also
    contain &quot;inter-comments&quot;.
    The inter-comments are aligned depending on the syntax.
    In the display below, the inter-comments are aligned with the &quot;rune&quot; of the enclosing sequence.
    A &quot;rune&quot; is Hoon's rough equivalent of a &quot;keyword&quot;.
    Runes are always digraphs of special ASCII characters.
    The rune in the following code is
    &lt;tt&gt;:~&lt;/tt&gt;,
    and the sequence it introduces
    includes pre-comments, inter-comments and meta-comments.
    &lt;/p&gt;
    &lt;pre&gt;&lt;tt&gt;
      :~  [3 7]
      ::
	  :: pre-comment 1
	  [20 (mug bod)]
      ::
	  :: pre-comment 2
	  [2 yax]
      ::
	  :: pre-comment 3
	  [2 qax]
    ::::
    ::    :: pre-comment 4
    ::    [4 qax]
      ::
	  :: pre-comment 5
	  [5 tay]
      ==
    &lt;/tt&gt;&lt;/pre&gt;
    &lt;p&gt;
    When inter-comments are empty, as they are in the above,
    they are called &quot;breathing comments&quot;, because they serve to separate,
    or allow some &quot;air&quot; between, elements of a sequence.
    For clarity,
    the pre-comments in the above are further indicated:
    all and only pre-comments contain the text &quot;&lt;tt&gt;pre-comment&lt;/tt&gt;&quot;.
    &lt;/p&gt;
    &lt;h3&gt;Meta-comments&lt;/h3&gt;
    &lt;p&gt;
    The above code also contains a third kind of comment -- meta-comments.
    Meta-comments must occur at the far left margin -- at column 1.
    These are called meta-comments, because they are allowed
    to be outside the syntax structure.
    One common use for meta-comments is &quot;commenting out&quot; other syntax.
    In the above display, the meta-comments &quot;comment out&quot;
    the comment labeled &quot;&lt;tt&gt;pre-comment 4&lt;/tt&gt;&quot;
    and its associated code.
    &lt;/p&gt;
    &lt;h3&gt;Staircase comments&lt;/h3&gt;
    &lt;p&gt;Finally, there are &quot;staircase comments&quot;, which are used
    to indicate the larger structure of Hoon sequences and other
    code.
    For example,
    &lt;pre&gt;&lt;tt&gt;
    ::                                                      ::
    ::::  3e: AES encryption  (XX removed)                  ::
      ::                                                    ::
      ::
    ::                                                      ::
    ::::  3f: scrambling                                    ::
      ::                                                    ::
      ::    ob                                              ::
      ::
    &lt;/tt&gt; &lt;/pre&gt;
    &lt;p&gt;
    Each staircase consists of three parts.
    In lexical order, these parts are
    an upper riser,
    a tread, and a lower riser.
    The upper riser is a sequence of comments at the same
    alignment as an inter-comment.
    The tread is also at the inter-comment alignment,
    but must be 4 colons (&quot;&lt;tt&gt;::::&lt;/tt&gt;&quot;) followed
    by whitespace.
    The lower riser is a sequence of comments
    indented two spaces more than the tread.
    &lt;/p&gt;
    &lt;h2&gt;Hoon comment conventions&lt;/h2&gt;
    &lt;p&gt;Hoon's basic syntax allows comments to be free-form.
    In practice, there are strict conventions for these comments,
    conventions we would like to enforce with &lt;tt&gt;hoonlint&lt;/tt&gt;.
    &lt;ol&gt;
    &lt;li&gt;A multi-line comment may contain
    an &quot;inter-part&quot;, a &quot;pre-part&quot;,
    or both.
    &lt;/li&gt;
    &lt;li&gt;If both an inter-part and a pre-part are present,
    the inter-part must precede the pre-part.
    &lt;/li&gt;
    &lt;li&gt;The inter-part is a non-empty sequence of inter-comments
    and staircases.
    &lt;/li&gt;
    &lt;li&gt;A pre-part is a non-empty sequence of pre-comments.
    &lt;/li&gt;
    &lt;li&gt;Meta-comments may be inserted anywhere in either the pre-part
    or the inter-part.
    &lt;/li&gt;
    &lt;li&gt;Comments which do not obey the above rules are
    &lt;b&gt;bad comments&lt;/b&gt;.
    A &lt;b&gt;good comment&lt;/b&gt; is any comment which is not a bad comment.
    &lt;/li&gt;
    &lt;li&gt;A comment is not regarded as a meta-comment
    if it can be parsed as structural comment.
    An &lt;b&gt;structural comment&lt;/b&gt; is any good comment which is
    not a meta-comment.
    &lt;/li&gt;
    &lt;/ol&gt;
    &lt;h2&gt;Grammar&lt;/h2&gt;
    &lt;p&gt;We will implement these conventions using the BNF
    of this section.
    The sections to follow outline the strategy behind the BNF.
    &lt;pre&gt;&lt;tt&gt;
    :start ::= gapComments
    gapComments ::= OptExceptions Body
    gapComments ::= OptExceptions
    Body ::= InterPart PrePart
    Body ::= InterPart
    Body ::= PrePart
    InterPart ::= InterComponent
    InterPart ::= InterruptedInterComponents
    InterPart ::= InterruptedInterComponents InterComponent

    InterruptedInterComponents ::= InterruptedInterComponent+
    InterruptedInterComponent ::= InterComponent Exceptions
    InterComponent ::= Staircases
    InterComponent ::= Staircases InterComments
    InterComponent ::= InterComments

    InterComments ::= InterComment+

    Staircases ::= Staircase+
    Staircase ::= UpperRisers Tread LowerRisers
    UpperRisers ::= UpperRiser+
    LowerRisers ::= LowerRiser+

    PrePart ::= ProperPreComponent OptPreComponents
    ProperPreComponent ::= PreComment
    OptPreComponents ::= PreComponent*
    PreComponent ::= ProperPreComponent
    PreComponent ::= Exception

    OptExceptions ::= Exception*
    Exceptions ::= Exception+
    Exception ::= MetaComment
    Exception ::= BadComment
    Exception ::= BlankLine
    &lt;/tt&gt;&lt;/pre&gt;
    &lt;h2&gt;Technique: Combinator&lt;/h2&gt;
    Our comment linter is implemented as a combinator.
    The main &lt;tt&gt;hoonlint&lt;/tt&gt; parser invokes this combinator when it encounters
    a multi-line comment.
    Because of the main parser,
    we do not have to worry about confusing comments with
    Hoon's various string and in-line text syntaxes.
    &lt;/p&gt;
    &lt;p&gt;Note that while combinator parsing is useful,
    it is a technique that can be oversold.
    Combinators have been much talked about in the functional programming
    literature&lt;a id=&quot;footnote-4-ref&quot; href=&quot;#footnote-4&quot;&gt;[4]&lt;/a&gt;,
    but the current flagship functional programming language compiler,
    the Glasgow Haskell Compiler,
    does not use combinators to parse its version of the Haskell --
    instead it uses a parser in the yacc lineage.&lt;a id=&quot;footnote-5-ref&quot; href=&quot;#footnote-5&quot;&gt;[5]&lt;/a&gt;
    As a parsing technique on its own,
    the use of combinators is simply another way of packaging recursive
    descent with backtracking,
    and the two techniques share the same power,
    the same performance,
    and the same downsides.
    &lt;/p&gt;
    &lt;p&gt;Marpa is much more powerful than either LALR (yacc-lineage) parsers or combinators,
    so we can save combinator parsing for those cases where
    combinator parsing really is helpful.
    One such case is lexer mismatch.
    &lt;/p&gt;
    &lt;h3&gt;Lexer mismatch&lt;/h3&gt;
    &lt;p&gt;The first programming languages, like BASIC and FORTRAN,
    were line-structured -- designed to be parsed line-by-line.&lt;a id=&quot;footnote-6-ref&quot; href=&quot;#footnote-6&quot;&gt;[6]&lt;/a&gt;
    After ALGOL, new languages were usually block-structured.
    Blocks can start or end in the middle of a line,
    and can span multiple lines.
    And blocks are often nested.
    &lt;/p&gt;
    &lt;p&gt;A line-structured language requires its lexer to think in
    terms of lines,
    but this approach is completely useless for a block-structured
    language.
    Combining both line-structured and block-structured logic in the same lexer
    usually turns the lexer's code into a rat's nest.
    &lt;/p&gt;
    &lt;p&gt;Calling a combinator every time
    a line-structured block is encountered eliminates the problem.
    The main lexer can assume that the code is block-structured,
    and all the line-by-line logic can go into combinators.
    &lt;/p&gt;
    &lt;h2&gt;Technique: Non-determinism&lt;/h2&gt;
    &lt;p&gt;
    Our grammar is non-deterministic,
    but unambiguous.
    It is unambiguous because,
    for every input,
    it will produce no more than one parse.
    &lt;/p&gt;
    &lt;p&gt;
    It is non-deterministic because there is a case
    where it tracks two possible parses at once.
    The comment linter cannot immediately distinguish between
    a prefix of the upper riser of a staircase,
    and a prefix of a sequence of inter-comments.
    When a tread and lower riser is encountered,
    the parser knows it has found a staircase,
    but not until then.
    And if the parse is of an inter-comment sequence,
    the comment linter will
    not be sure of this until the end of the sequence.
    &lt;/p&gt;
    &lt;h2&gt;Technique: Infinite lookahead&lt;/h2&gt;
    &lt;p&gt;
    As just pointed out,
    the comment linter does not know whether it is parsing a staircase or
    an inter-comment sequence until either
    &lt;ul&gt;
    &lt;li&gt;it finds a tread and lower riser, in which case
    it knows the correct parse will be a staircase; or
    &lt;/li&gt;
    &lt;li&gt;it successfully reaches the end of the inter-comment sequence,
    in which case it knows the correct parse is an inter-comment sequence.
    &lt;/ul&gt;
    To determine which of these two choices is the correct parse,
    the linter needs to read
    an arbitrarily long sequence of tokens --
    in other words, the linter needs to perform infinite lookahead.
    &lt;/p&gt;
    &lt;p&gt;Humans deal with infinite lookaheads all the time --
    natural languages are full of situations that require them.&lt;a id=&quot;footnote-7-ref&quot; href=&quot;#footnote-7&quot;&gt;[7]&lt;/a&gt;
    Modern language designers labor to avoid the need
    for infinite lookahead,
    but even so
    cases where it is desirable pop up.&lt;a id=&quot;footnote-8-ref&quot; href=&quot;#footnote-8&quot;&gt;[8]&lt;/a&gt;
    &lt;/p&gt;
    &lt;p&gt;
    Fortunately, in 1991, Joop Leo published a method that
    allows computers to emulate infinite lookahead efficiently.
    Marpa uses Joop's technique.
    Joop's algorithm is complex,
    but the basic idea is to do what humans do in the same circumstance --
    keep all the possibilities in mind until the evidence comes in.
    &lt;/p&gt;
    &lt;p&gt;
    &lt;/p&gt;
    &lt;h2&gt;Technique: the Ruby Slippers&lt;/h2&gt;
    &lt;p&gt;Recall that, according to our conventions,
    our parser does not recognize a meta-comment unless
    no structural comment can be recognized.
    We could implement this in BNF,
    but it is much more elegant to use the Ruby Slippers.&lt;a id=&quot;footnote-9-ref&quot; href=&quot;#footnote-9&quot;&gt;[9]&lt;/a&gt;
    &lt;/p&gt;
    &lt;p&gt;As those already familiar with Marpa may recall,
    the Ruby Slippers are invoked when a Marpa parser finds itself
    unable to proceed with its current set of input tokens.
    At this point, the lexer can ask the Marpa parser what token it &lt;b&gt;does&lt;/b&gt; want.
    Once the lexer is told what the &quot;wished-for&quot; token is,
    it can concoct one, out of nowhere if necessary, and pass it to the Marpa parser,
    which then proceeds happily.
    In effect, the lexer acts like Glenda the Good Witch of Oz,
    while the Marpa parser plays the role of Dorothy.
    &lt;/p&gt;
    &lt;p&gt;In our implementation, the Marpa parser, by default,
    looks only for structural comments.
    If the Marpa parser of our
    comment linter finds that the current input line is not
    a structural comment,
    the Marpa parser halts
    and tells the lexer that there is a problem.
    The lexer then asks the Marpa parser what it is looking for.
    In this case, the answer will always be the same:
    the Marpa parser will be looking for a meta-comment.
    The lexer checks to see if the current line is a comment
    starting at column 1.
    If there is a comment starting at column 1,
    the lexer tells the Marpa parser that its wish has come true --
    there is a meta-comment.
    &lt;/p&gt;
    &lt;p&gt;Another way to view the Ruby Slippers is as a kind of exception
    mechanism for grammars.
    In this application, we treat inability to read an structural
    comment as an exception.
    When the exception occurs,
    if possible, we read a meta-comment.
    &lt;/p&gt;
    &lt;h2&gt;Technique: Error Tokens&lt;/h2&gt;
    &lt;p&gt;&lt;b&gt;Error tokens&lt;/b&gt; are a specialized use of the Ruby Slippers.
    The application for this parser is &quot;linting&quot; --
    checking that the comments follow conventions.
    As such, the main product of the parser is not the parse --
    it is the list of errors gathered along the way.
    So stopping the parser at the first error does not make sense.
    &lt;/p&gt;
    &lt;p&gt;
    What is desirable is to treat all inputs as valid,
    so that the parsing always runs to the end of input,
    in the process producing a list of the errors.
    To do this, we want to set up the parser so that it reads
    special &quot;error tokens&quot; whenever it encounters a reportable error.
    &lt;/p&gt;
    &lt;p&gt;This is perfect for the Ruby Slippers.
    If an &quot;exception&quot; occurs,
    as above described for meta-comments,
    but no meta-comment is available,
    we treat it as a second level exception.
    &lt;/p&gt;
    &lt;p&gt;When would no meta-comment be available?
    There are two cases:
    &lt;ul&gt;&lt;li&gt;The line read is a comment,
    but it does not start at column 1.
    &lt;/li&gt;
    &lt;li&gt;The line read is a blank line (all whitespace).
    &lt;/li&gt;
    &lt;/ul&gt;
    &lt;p&gt;On the second exception level, the current line
    will be read as either a &lt;tt&gt;&amp;lt;BlankLine&amp;gt;&lt;/tt&gt;,
    or a &lt;tt&gt;&amp;lt;BadComment&amp;gt;&lt;/tt&gt;.
    We know that every line must lex as either a
    &lt;tt&gt;&amp;lt;BlankLine&amp;gt;&lt;/tt&gt;
    or a &lt;tt&gt;&amp;lt;BadComment&amp;gt;&lt;/tt&gt; because our comment linter
    is called as a combinator,
    and the parent Marpa parser guarantees this.
    &lt;/p&gt;
    &lt;h2&gt;Technique: Ambiguity&lt;/h2&gt;
    &lt;p&gt;Marpa allows ambiguity,
    which could have been exploited as a technique.
    For example, in a simpler BNF than that we used above,
    it might be ambiguous whether a meta-comment belongs to an &lt;tt&gt;&amp;lt;InterPart&amp;gt;&lt;/tt&gt;
    which immediately precedes it;
    or to a &lt;tt&gt;&amp;lt;PrePart&amp;gt;&lt;/tt&gt; which immediately follows it.
    We could solve the dilemma by noting that it does not matter:
    All we care about is spotting bad comments and blank lines,
    so that picking one of two ambiguous parses at random will work fine.
    &lt;/p&gt;
    &lt;p&gt;
    But efficiency issues are sometimes a problem with ambiguity
    and unambiguity can be a good way of avoiding them.&lt;a id=&quot;footnote-10-ref&quot; href=&quot;#footnote-10&quot;&gt;[10]&lt;/a&gt;
    Also, requiring the grammar to be unambiguous allows
    an additional check that is useful in the development phase.
    In our code we test each parse for ambiguity.
    If we find one, we know that &lt;tt&gt;hoonlint&lt;/tt&gt; has a coding error.
    &lt;/p&gt;
    &lt;p&gt;
    Keeping the parser unambiguous makes the BNF
    less elegant than it could be.
    To avoid ambiguity,
    we introduced extra symbols;
    introduced extra rules;
    and restricted the use of ambiguous tokens.
    &lt;/p&gt;
    &lt;p&gt;Recall that I am using the term &quot;ambiguous&quot; in the strict technical
    sense that it has in parsing theory, so that a parser is only ambiguous
    if it can produce two valid parses for one string.
    An unambiguous parser
    can allow non-deterministism and
    can have ambiguous tokens.
    In fact, our example grammar does both of these things,
    but is nonetheless unambiguous.
    &lt;/p&gt;
    &lt;h3&gt;Extra symbols&lt;/h3&gt;
    One example of an extra symbol introduced to make this parser
    unambiguous is &lt;tt&gt;&amp;lt;ProperPreComment&amp;gt;&lt;/tt&gt;.
    &lt;tt&gt;&amp;lt;ProperPreComment&amp;gt;&lt;/tt&gt;
    is used to ensure that a
    &lt;tt&gt;&amp;lt;PrePart&amp;gt;&lt;/tt&gt;
    never begins with a meta-comment.&lt;a id=&quot;footnote-11-ref&quot; href=&quot;#footnote-11&quot;&gt;[11]&lt;/a&gt;
    &lt;/p&gt;
    &lt;p&gt;The BNF requires that the first line of a
    &lt;tt&gt;&amp;lt;PrePart&amp;gt;&lt;/tt&gt;
    must be a
    &lt;tt&gt;&amp;lt;ProperPreComment&amp;gt;&lt;/tt&gt;.
    This means that, if a
    &lt;tt&gt;&amp;lt;MetaComment&amp;gt;&lt;/tt&gt; is found
    at the boundary between an
    &lt;tt&gt;&amp;lt;InterPart&amp;gt;&lt;/tt&gt;
    and a
    &lt;tt&gt;&amp;lt;PrePart&amp;gt;&lt;/tt&gt;,
    it cannot be the first line of the
    &lt;tt&gt;&amp;lt;PrePart&amp;gt;&lt;/tt&gt;
    and so must be the last line of the
    &lt;tt&gt;&amp;lt;InterPart&amp;gt;&lt;/tt&gt;.
    &lt;/p&gt;
    &lt;/p&gt;
    &lt;h3&gt;Extra rules&lt;/h3&gt;
    &lt;p&gt;In our informal explanation of the comment conventions,
    we stated that an inter-part is a sequence, each element of
    which is an inter-comment or a staircase.
    While BNF that directly implemented this rule would be correct,
    it would also be highly ambiguous:
    If an inter-comment occurs before a tread or an upper riser line,
    it could also be parsed as part of the upper riser.
    &lt;/p&gt;
    &lt;p&gt;To eliminate the ambiguity,
    we stipulate that if comment &lt;b&gt;can&lt;/b&gt; be parsed as part of a staircase,
    then it &lt;b&gt;must&lt;/b&gt; be parsed as part of a staircase.
    This stipulation still leaves the grammar non-deterministic --
    we may not know if our comment could be part of a staircase until
    many lines later.
    &lt;/p&gt;
    &lt;p&gt;With our stipulation we know that, if an
    &lt;tt&gt;&amp;lt;InterComponent&amp;gt;&lt;/tt&gt;
    contains
    a staircase, then that staircase must come before any of the inter-comments.
    In an &lt;tt&gt;&amp;lt;InterComponent&amp;gt;&lt;/tt&gt;
    both staircases and inter-comments are optional, so the
    unambiguous representation of
    &lt;tt&gt;&amp;lt;InterComponent&amp;gt;&lt;/tt&gt;
    is
    &lt;pre&gt;&lt;tt&gt;
    InterComponent ::= Staircases
    InterComponent ::= Staircases InterComments
    InterComponent ::= InterComments
    &lt;/tt&gt;&lt;/pre&gt;
    Notice that, although
    both staircases and inter-comments are optional,
    we do not include the case where both are omitted.
    This is because we insist that an
    &lt;tt&gt;&amp;lt;InterComponent&amp;gt;&lt;/tt&gt;
    contain at least one line.
    &lt;/p&gt;
    &lt;h3&gt;Ambiguous tokens&lt;/h3&gt;
    &lt;p&gt;Our parser is not ambiguous, but
    it &lt;b&gt;does&lt;/b&gt; allow ambiguous tokens.
    For example, a comment with inter-comment alignment
    could be either an
    &lt;tt&gt;&amp;lt;InterComment&amp;gt;&lt;/tt&gt;
    or an
    &lt;tt&gt;&amp;lt;UpperRiser&amp;gt;&lt;/tt&gt;;
    and our lexer returns both.
    The parser remains unambiguous, however, because
    only one of these two tokens will wind up in the
    final parse.
    &lt;/p&gt;
    &lt;p&gt;Call the set of tokens returned
    by our parser for a single line,
    a &quot;token set&quot;.
    If the token set contains more than one token,
    the tokenization is ambiguous for that line.
    If the token set contains only one token,
    the token set is called a &quot;singleton&quot;,
    and tokenization is unambiguous for that line.
    &lt;/p&gt;
    &lt;p&gt;
    To keep
    this parser unambiguous, we restrict the
    ambiguity at the lexer level.
    For example,
    our lexer is set up so
    that a meta-comment is never one of the alternatives
    in a lexical ambiguity.
    If a token set contains a
    &lt;tt&gt;&amp;lt;MetaComment&amp;gt;&lt;/tt&gt;,
    that token set must be a singleton.
    The Ruby Slippers are used to enforce this.&lt;a id=&quot;footnote-12-ref&quot; href=&quot;#footnote-12&quot;&gt;[12]&lt;/a&gt;
    Similarly, the Ruby Slippers are used to guarantee that
    any set of tokens containing either
    a &lt;tt&gt;&amp;lt;BadComment&amp;gt;&lt;/tt&gt;
    or a
    &lt;tt&gt;&amp;lt;BlankLine&amp;gt;&lt;/tt&gt; is a singleton.
    &lt;/p&gt;
    &lt;h2&gt;Code&lt;/h2&gt;
    &lt;p&gt;This post did not walk the reader through the code.
    Instead, we talked in terms of strategy.
    &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/tree/gh-pages/code/vgap&quot;&gt;
    The code is available on Github&lt;/a&gt;
    in unit test form.
    For those who want to see the comment-linter combinator in a context,
    a version of the code embedded in &lt;tt&gt;hoonlint&lt;/tt&gt;
    in also on Github.&lt;a id=&quot;footnote-13-ref&quot; href=&quot;#footnote-13&quot;&gt;[13]&lt;/a&gt;
    &lt;h2&gt;Comments on this blog post, etc.&lt;/h2&gt;
    &lt;p&gt;
      To learn about Marpa,
      my Earley/Leo-based parser,
      there is the
      &lt;a href=&quot;http://savage.net.au/Marpa.html&quot;&gt;semi-official web site, maintained by Ron Savage&lt;/a&gt;.
      The official, but more limited, Marpa website
      &lt;a href=&quot;http://jeffreykegler.github.io/Marpa-web-site/&quot;&gt;is my personal one&lt;/a&gt;.
      Comments on this post can be made in
      &lt;a href=&quot;http://groups.google.com/group/marpa-parser&quot;&gt;
        Marpa's Google group&lt;/a&gt;,
      or on our IRC channel: #marpa at freenode.net.
    &lt;/p&gt;
    &lt;h2&gt;Footnotes&lt;/h2&gt;
&lt;p id=&quot;footnote-1&quot;&gt;&lt;b&gt;1.&lt;/b&gt;
In their present form, urbits run on top of Unix and UDP.
 &lt;a href=&quot;#footnote-1-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-2&quot;&gt;&lt;b&gt;2.&lt;/b&gt;
    Garbage collection and arbitrary precision may seem too high-level
    for something considered a &quot;machine language&quot;,
    but our concepts evolve.
    The earliest machine languages required programmers to
    write their own memory caching logic
    and to create their own floating
    point representations,
    both things we now regard as much too low-level
    to deal with even at the lowest software level.
 &lt;a href=&quot;#footnote-2-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-3&quot;&gt;&lt;b&gt;3.&lt;/b&gt;
    This post attempts to follow standard Hoon terminology, but
    for some details of Hoon's whitespace conventions,
    there is no settled terminology,
    and I have invented terms as necessary.
    The term &quot;pre-comment&quot; is one of those inventions.
 &lt;a href=&quot;#footnote-3-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-4&quot;&gt;&lt;b&gt;4.&lt;/b&gt;
    For a brief survey of this literature,
    see the entries from 1990 to 1996
    in my &lt;a href=&quot;https://jeffreykegler.github.io/personal/timeline_v3&quot;&gt;
    &quot;timeline&quot; of parsing history&lt;/a&gt;.
 &lt;a href=&quot;#footnote-4-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-5&quot;&gt;&lt;b&gt;5.&lt;/b&gt;
&lt;a
    href=&quot;https://github.com/ghc/ghc/blob/master/compiler/parser/Parser.y&quot;&gt;This
    is the LALR grammar for GHC&lt;/a&gt;, from GHC's Github mirror.
 &lt;a href=&quot;#footnote-5-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-6&quot;&gt;&lt;b&gt;6.&lt;/b&gt;
    This is simplified.
    There were provisions for line continuation, etc.
    But, nonetheless, the lexers for these languages worked in
    terms of lines, and had no true concept of a &quot;block&quot;.
 &lt;a href=&quot;#footnote-6-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-7&quot;&gt;&lt;b&gt;7.&lt;/b&gt;
    An example of a requirement for infinite lookahead
    is the sentence &quot;The horse raced past the barn fell&quot;.
    Yes, this sentence is not, in fact, infinitely long,
    but the subclause &quot;raced past the barn&quot; could be anything,
    and therefore could be arbitrarily long.
    In isolation, this example sentence may seem unnatural,
    a contrived &quot;garden path&quot;.
    But if you imagine the sentence as an answer to the question, &quot;Which horse fell?&quot;,
    expectations are set so that the sentence is quite reasonable.
 &lt;a href=&quot;#footnote-7-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-8&quot;&gt;&lt;b&gt;8.&lt;/b&gt;
    See my blog post &lt;a href=&quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/08/rntz.html&quot;&gt;
    &quot;A Haskell challenge&quot;&lt;/a&gt;.
 &lt;a href=&quot;#footnote-8-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-9&quot;&gt;&lt;b&gt;9.&lt;/b&gt;
        To find out more about Ruby Slippers parsing see the Marpa FAQ,
        &lt;a href=&quot;http://savage.net.au/Perl-modules/html/marpa.faq/faq.html#q122&quot;&gt;
          questions 122&lt;/a&gt;
        and
        &lt;a href=&quot;http://savage.net.au/Perl-modules/html/marpa.faq/faq.html#q123&quot;&gt;
          123&lt;/a&gt;;
        my
        &lt;a href=&quot;file:///mnt2/new/projects/Ocean-of-Awareness-blog/metapages/annotated.html#PARSE_HTML&quot;&gt;
          blog series on parsing HTML&lt;/a&gt;;
	  my recent blog post
	  &lt;a
	  href=&quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/05/combinator2.html&quot;&gt;
	  &quot;Marpa and combinator parsing 2&quot;&lt;/a&gt;;
	  and my much older blog post
        &lt;a href=&quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2011/11/marpa-and-the-ruby-slippers.html&quot;&gt;
          &quot;Marpa and the Ruby Slippers&quot;&lt;/a&gt;.
 &lt;a href=&quot;#footnote-9-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-10&quot;&gt;&lt;b&gt;10.&lt;/b&gt;
    This, by the way,
    is where I believe parsing theory went wrong,
    beginning in the 1960's.
    In an understandable search for efficiency,
    mainstream parsing theory totally excluded not just ambiguity,
    but non-determinism as well.
    These draconian restrictions limited the
    search for practical parsers to a subset of techniques
    so weak that they cannot
    even duplicate human parsing capabilities.
    This had the bizarre effect of committing
    parsing theory to a form of
    &quot;human exceptionalism&quot; --
    a belief that human beings have a special ability to
    parse that computers cannot emulate.
    For more on this story,
    see my &lt;a href=&quot;https://jeffreykegler.github.io/personal/timeline_v3&quot;&gt;
    &quot;timeline&quot; of parsing history&lt;/a&gt;.
 &lt;a href=&quot;#footnote-10-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-11&quot;&gt;&lt;b&gt;11.&lt;/b&gt;
    This example illustrates the efficiency considerations
    involved in the decision to tolerate,
    or to exclude,
    efficiency.
    If &lt;tt&gt;n&lt;/tt&gt; meta-comments occur between a
    &lt;tt&gt;&amp;lt;InterPart&amp;gt;&lt;/tt&gt;
    and a &lt;tt&gt;&amp;lt;PrePart&amp;gt;&lt;/tt&gt;,
    the dividing line is arbitrary,
    so that there are &lt;tt&gt;n+1&lt;/tt&gt; parses.
    This will, in theory, make the processing time quadratic.
    And, in fact, long sequences of meta-comments might occur
    between the inter- and pre-comments,
    so the inefficiency might be real.
 &lt;a href=&quot;#footnote-11-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-12&quot;&gt;&lt;b&gt;12.&lt;/b&gt;
    Inter-comments and
    comments that are part of upper risers may start at column 1,
    so that, without special precautions in the lexer,
    an ambiguity between a structural comment
    and a meta-comment is entirely
    possible.
 &lt;a href=&quot;#footnote-12-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-13&quot;&gt;&lt;b&gt;13.&lt;/b&gt;
    For the &lt;tt&gt;hoonlint&lt;/tt&gt;-embedded form,
    the Marpa grammar is
    &lt;a href=&quot;https://github.com/jeffreykegler/yahc/blob/714157124b46492e13968c786e400276017a3b85/Lint/Policy/Test/Whitespace.pm#L19&quot;&gt;
    here&lt;/a&gt;
    and the code is
    &lt;a href=&quot;https://github.com/jeffreykegler/yahc/blob/714157124b46492e13968c786e400276017a3b85/Lint/Policy/Test/Whitespace.pm#L341&quot;&gt;
    here&lt;/a&gt;.
    These are snapshots -- permalinks.
    The application is under development,
    and probably will change considerably.
    Documentation is absent
    and testing is minimal,
    so that this pre-alpha embedded form of the code will mainly be useful
    for those who want to take a quick glance at the
    comment linter in context.
 &lt;a href=&quot;#footnote-13-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
  &lt;/body&gt;
&lt;/html&gt;</description>
  </item>
  <item>
    <title>Sherlock Holmes and the Case of the Missing Parsing Solution</title>
    <link>http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2019/03/methodology.html</link>
    <description>&lt;html&gt;
  &lt;head&gt;
  &lt;/head&gt;
  &lt;body&gt;
    &lt;!--
      marpa_r2_html_fmt --no-added-tag-comment --no-ws-ok-after-start-tag
      --&gt;
    &lt;blockquote&gt;Always approach a case with an absolutely blank mind.
      It is always an advantage.
      Form no theories, just simply observe and draw inferences from your observations.
      &amp;mdash;
        Sherlock Holmes, quoted in &quot;The Adventure of the Cardboard Box&quot;.
        &lt;/blockquote&gt;
    &lt;blockquote&gt;It is a capital mistake to theorize before one has data.
      &amp;mdash;
        Holmes, in &quot;A Scandal in Bohemia&quot;.
    &lt;/blockquote&gt;
    &lt;blockquote&gt;I make a point of never having any prejudices, and of following docilely wherever fact may lead me.
      &amp;mdash;
      Holmes, in &quot;The Reigate Puzzle&quot;.
    &lt;/blockquote&gt;
    &lt;blockquote&gt;When you have eliminated the impossible, whatever remains, no matter how improbable, must be the truth.
      &amp;mdash;
      Holmes, in &quot;The Sign of Four&quot;.
    &lt;/blockquote&gt;
    &lt;blockquote&gt;
      In imagination there
      exists the perfect
      mystery story.
      Such a story presents
      the essential clues, and compels us to form our own
      theory of the case.
      If we
      follow the plot carefully, we arrive at the complete
      solution for ourselves just before the author's disclosure
      at the end of the book. The solution itself, contrary to
      those of inferior mysteries, does not disappoint us;
      moreover, it appears at the very moment we expect it.
      Can we liken the reader of such a book to the scientists,
      who throughout successive generations continue to seek
      solutions of the mysteries in the book of nature? The
      comparison is false and will have to be abandoned later,
      but it has a modicum of justification which may be
      extended and modified to make it more appropriate to
      the endeavour of science to solve the mystery of the
      universe.
      &amp;mdash;
      Albert Einstein
        and Leopold Infeld. &lt;a id=&quot;footnote-1-ref&quot; href=&quot;#footnote-1&quot;&gt;[1]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;h2&gt;The Sherlock Holmes approach&lt;/h2&gt;
    &lt;p&gt;My
    &lt;a href=&quot;https://jeffreykegler.github.io/personal/timeline_v3&quot;&gt;
    timeline history of parsing theory&lt;/a&gt;
      is my most popular writing, but
      it is not without its critics.
      Many of them accuse the timeline of lack of objectivity or of bias.
    &lt;/p&gt;
    &lt;p&gt;
      Einstein assumed his reader's idea of methods of proper investigation,
      in science as elsewhere,
      would be similar to those Conan Doyle's Sherlock Holmes.
      I will follow Einstein's lead in starting there.
    &lt;/p&gt;
    &lt;p&gt;
      The deductions recorded in the Holmes' canon
      often involve
      &lt;b&gt;a lot&lt;/b&gt;
      of theorizing.
      To make it a matter of significance what the dogs in &quot;Silver Blaze&quot; did in the night,
      Holmes needs a theory of canine behavior,
      and Holmes' theory sometimes outpaces its pack of facts by a considerable distance.
      Is it really true that only dangerous people own
      dangerous dogs?&lt;a id=&quot;footnote-2-ref&quot; href=&quot;#footnote-2&quot;&gt;[2]&lt;/a&gt;
    &lt;/p&gt;
    &lt;p&gt;
      Holmes's methods, at least as stated in the Conan Doyle stories,
      are incapable of solving anything
      but the fictional problems he encounters.
      In real life, a &quot;blank mind&quot; can observe nothing.
      There is no &quot;data&quot; without theory, just white noise.
      Every &quot;fact&quot; gathered relies on many prejudgements about what is
      relevant and what is not.
      And you certainly cannot characterize anything as &quot;impossible&quot;,
      unless you have, in advance, a theory about what is possible.
    &lt;/p&gt;
    &lt;h2&gt;The Einstein approach&lt;/h2&gt;
    &lt;p&gt;Einstein, in his popular account
    of the evolution of physics,
      finds the Doyle stories &quot;admirable&quot;&lt;a id=&quot;footnote-3-ref&quot; href=&quot;#footnote-3&quot;&gt;[3]&lt;/a&gt;.
      But to solve real-life mysteries, more is needed.
      Einstein begins his description of his methods at the start
      of his Chapter II:
    &lt;/p&gt;&lt;blockquote&gt;
      The following pages contain a dull report of
      some very simple experiments.
      The account will be boring
      not only because the description of experiments is uninteresting
      in comparison with their actual performance,
      but also because the meaning of the experiments does
      not become apparent until theory makes it so. Our
      purpose is to furnish a striking example of the role of
      theory in physics.
      &lt;a id=&quot;footnote-4-ref&quot; href=&quot;#footnote-4&quot;&gt;[4]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;p&gt;Einstein follows with a series of the kind of experiments
      that are performed in high school physics classes.
      One might imagine these experiments allowing an observer
      to deduce the basics of electromagnetism
      using materials and techniques available for centuries.
    &lt;/p&gt;
    &lt;p&gt;But, and this is Einstein's point,
      this is not how it happened.
      The theory came
      &lt;b&gt;first&lt;/b&gt;,
      and the experiments were devised afterwards.
    &lt;/p&gt;
    &lt;blockquote&gt;
      In the first pages
      of our book we compared the role
      of an investigator
      to that of a detective who, after
      gathering the requisite facts, finds the right solution
      by pure thinking. In one essential this comparison must
      be regarded as highly superficial. Both in life and in
      detective novels the crime is given. The detective must
      look for letters, fingerprints, bullets, guns, but at least
      he knows that a murder has been committed. This is
      not so for a scientist. It should not be difficult to
      imagine someone who knows absolutely nothing about
      electricity, since all the ancients lived happily enough
      without any knowledge of it. Let this man be given
      metal, gold foil, bottles, hard-rubber rod, flannel, in
      short, all the material required for performing our
      three experiments. He may be a very
      cultured person,
      but he will probably put wine into the bottles, use the
      flannel for cleaning, and never once entertain the idea
      of doing the things we have described.
      For the detective
      the crime is given, the problem formulated: who
      killed Cock Robin?
      The scientist must, at least in part,
      commit his own crime, as well as carry out the investigation.
      Moreover, his task is not to explain just one
      case, but all phenomena which have happened
      or may
      still happen. &amp;mdash; Einstein and Infeld &lt;a id=&quot;footnote-5-ref&quot; href=&quot;#footnote-5&quot;&gt;[5]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;h2&gt;Commiting our own crime&lt;/h2&gt;
    &lt;p&gt;If then,
      we must commit the crime of theorizing before the facts,
      where does out theory come from?
    &lt;/p&gt;
    &lt;blockquote&gt;
    Science is not just a collection of laws,
    a catalogue of unrelated facts.
    It is a creation of the human mind,
    with its freely invented ideas and concepts.
    Physical theories try to form a picture of reality
    and to establish its connection
    with the wide world of sense impressions.
    Thus the only justification for our mental structures
    is whether and in what way our theories form such
    a link. &amp;mdash; Einstein and Infeld &lt;a id=&quot;footnote-6-ref&quot; href=&quot;#footnote-6&quot;&gt;[6]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;blockquote&gt;
      In the case of planets moving around the sun
      it is found that the system of mechanics works
      splendidly.
      Nevertheless we can well imagine that another system,
      based on different assumptions,
      might work just as well.
      &lt;br&gt;
      Physical concepts are free creations
      of the human mind, and are not,
      however it may seem,
      uniquely determined by the external world.
      In our endeavor to understand reality
      we are somewhat like a man trying
      to understand the mechanism of a closed watch.
      He sees the face and the moving hands,
      even hears its ticking,
      but he has no way of opening the case.
      If he is ingenious
      he may form some picture of a mechanism
      which could be responsible
      for all the things he observes,
      but he may never be quite sure
      his picture is the only one
      which could explain his observations.
      He will never be able
      to compare his picture with the real mechanism
      and he cannot even imagine the possibility
      or the meaning of such a comparison.
      But he certainly believes that,
      as his knowledge increases,
      his picture of reality will become
      simpler and simpler
      and will explain a wider and wider range
      of his sensuous impressions.
      He may also be believe in the existence
      of the ideal limit of knowledge
      and that it is approached
      by the human mind.
      He may call this ideal limit
      the objective truth. -- Einstein and Infeld &lt;a id=&quot;footnote-7-ref&quot; href=&quot;#footnote-7&quot;&gt;[7]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;p&gt;It may sound as if Einstein believed that the soundness of
    our theories is a matter of faith.
    In fact, Einstein was quite comfortable with putting it
    exactly that way:
    &lt;blockquote&gt;However, it must be admitted
    that our knowledge of these laws is only imperfect
    and fragmentary, so that,
    actually the belief
    in the existence of basic all-embracing laws
    in Nature also rests on a sort of faith.
    All the same this faith has been largely
    justified so far by the success of
    scientific research. &amp;mdash; Einstein &lt;a id=&quot;footnote-8-ref&quot; href=&quot;#footnote-8&quot;&gt;[8]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;blockquote&gt;
    I believe that every true theorist
    is a kind of tamed metaphysicist,
    no matter how pure a &quot;positivist&quot; he may
    fancy himself.
    The metaphysicist believes that the logically
    simple is also the real.
    The tamed metaphysicist believes
    that not all that is logically simple
    is embodied in experienced reality,
    but that the totality of all sensory experience
    can be &quot;comprehended&quot; on the basis of a
    conceptual system built on premises of great
    simplicity.
    The skeptic will say this is a &quot;miracle creed.&quot;
    Admittedly so, but it is a miracle creed
    which has been borne out to an amazing extent by
    the development of science. &amp;mdash; Einstein &lt;a id=&quot;footnote-9-ref&quot; href=&quot;#footnote-9&quot;&gt;[9]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;blockquote&gt;
    The liberty of choice, however,
    is of a special kind;
    it is not in any way similar to the liberty of a
    writer of fiction.
    Rather, it is similar to that of a man engaged
    in solving a well-designed puzzle.
    He may, it is true, propose
    any word as the solution;
    but, there is only &lt;i&gt;one&lt;/i&gt;
    word which really solves the puzzle in all its
    parts.
    It is a matter of faith that nature
    &amp;mdash;
    as she is perceptible to our five senses
    &amp;mdash;
    takes the character of such a
    well-formulated puzzle.
    The successes reaped up to now
    by science do,
    it is true,
    give a certain encouragement for this faith. --
    Einstein &lt;a id=&quot;footnote-10-ref&quot; href=&quot;#footnote-10&quot;&gt;[10]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;p&gt;The puzzle metaphor of the last quote is revealing.
    Einstein believes there is a single truth,
    but that we will never know what it is &amp;mdash;
    even its existence can only be taken as a matter of faith.
    Existence is a crossword puzzle whose answer we will never
    know.
    Even the existence of an answer must be taken as
    a matter of faith.
    &lt;/p&gt;
    &lt;blockquote&gt;
    The very fact that the totality of our sense experience
    is such that by means of thinking
    (operations with concepts,
    and the creation and use of definite functional relations
    between them,
    and the coordination of sense experiences to these concepts)
    it can be put in order,
    this fact is one which leaves us in awe,
    but which we shall never understand.
    One may say that
    &quot;the eternal mystery of the world
    is its comprehensibility&quot;. &amp;mdash; Einstein &lt;a id=&quot;footnote-11-ref&quot; href=&quot;#footnote-11&quot;&gt;[11]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;blockquote&gt;
    In my opinion,
    nothing can be said &lt;i&gt;a priori&lt;/i&gt;
    concerning the manner in which the concepts
    are to be formed and connected,
    and how we are to coordinate them to sense experiences.
    In guiding us in the creation of such an order
    of sense experiences,
    success alone is the determining factor.
    All that is necessary is to fix a set of rules,
    since without such rules the acquisition
    of knowledge in the desired sense would be impossible.
    One may compare these rules with the rules of a game
    in which,
    while the rules themselves are arbitrary,
    it is their rigidity alone which
    makes the game possible.
    However, the fixation will never be final.
    It will have validity only for a special field
    of application. &amp;mdash; Einstein &lt;a id=&quot;footnote-12-ref&quot; href=&quot;#footnote-12&quot;&gt;[12]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;blockquote&gt;
    There are no eternal theories in science.
    It always happens that some of the facts
    predicted by a theory
    are disproved by experiment.
    Every theory has its period of
    gradual development and triumph,
    after which it may experience a
    rapid decline. &amp;mdash; Einstein and Infeld
    &lt;a id=&quot;footnote-13-ref&quot; href=&quot;#footnote-13&quot;&gt;[13]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;/p&gt;
    &lt;blockquote&gt;
    In our great mystery story there are no problems
    wholly solved and settled for all time. &amp;mdash; Einstein and Infeld
    &lt;a id=&quot;footnote-14-ref&quot; href=&quot;#footnote-14&quot;&gt;[14]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;blockquote&gt;
      This great mystery story
      is still
      unsolved.
      We
      cannot
      even be sure that it has a final solution. &amp;mdash;
      Einstein and Infeld &lt;a id=&quot;footnote-15-ref&quot; href=&quot;#footnote-15&quot;&gt;[15]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;h2&gt;Choosing a &quot;highway&quot;&lt;/h2&gt;
    In most of the above,
    Einstein is focusing on his work in a &quot;hard&quot; science: physics.
    Are his methods relevant to &quot;softer&quot; fields of study?
    Einstein thinks so:
    &lt;blockquote&gt;
      The whole of science is nothing
      more than a refinement of everyday thinking.
      It is for this reason that the critical thinking
      of the physicist cannot possibly be restricted to
      the examination of the concepts of his own
      specific field.
      He cannot proceed without considering critically
      a much more difficult problem,
      the problem of analyzing the nature of everyday
      thinking. &amp;mdash; Einstein
      &lt;a id=&quot;footnote-16-ref&quot; href=&quot;#footnote-16&quot;&gt;[16]&lt;/a&gt;
    &lt;/blockquote&gt;
    Einstein's collaboration with Infeld was, like the &quot;Timeline&quot;,
    a description of the evolution of ideas,
    and in the Einstein&amp;ndash;Infeld book they describe their approach:
    &lt;blockquote&gt;
      Through the maze of
      facts and concepts we had to choose some highway
      which seemed to us most characteristic and significant.
      Facts and theories not reached by this road had to be
      omitted. We were forced, by our general aim, to make
      a definite choice of facts and ideas. The importance of a
      problem should not be judged by the number of pages
      devoted to it. Some essential lines of thought have been
      left out, not because they seemed to us unimportant,
      but because they do not lie along the road we have
      chosen. &amp;mdash; Einstein and Infeld &lt;a id=&quot;footnote-17-ref&quot; href=&quot;#footnote-17&quot;&gt;[17]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;h2&gt;Truth and success&lt;/h2&gt;
    &lt;p&gt;Einstein says that objective truth, while
    it exists, is not to be attained in the hard sciences,
    so it is not likely he thought that a historical
    account could outdo physics in this respect.
    For Einstein, as quoted above,
    &quot;success alone is the determining factor&quot;.
    &lt;/p&gt;
    &lt;p&gt;Success, of course, varies with what the audience
    for a theory wants.
    In a very real sense,
    I consider a theory that can predict the
    stock market more successful than
    one which can predict perturbations of planetary orbits
    invisible to the naked eye.
    But this is not a reasonable expectation when applied
    to the theory of general relativity.
    &lt;/p&gt;
    Among the expectations reasonable for a timeline of parsing
    might be these:
    &lt;ul&gt;
    &lt;li&gt;It helps choose the right parsing algoithm for practical
    applications.
    &lt;li&gt;It helps a reader to understand articles in the
    literature of parsing.
    &lt;li&gt;It helps guide future research.
    &lt;li&gt;It predicts the outcome of future research.
    &lt;/ul&gt;
    &lt;/p&gt;When I wrote the first version of &lt;cite&gt;Timeline&lt;/cite&gt;,
    its goal was none of these.
    Instead I intended it to explain the sources behind my own
    research in the Earley/Leo lineage.
    &lt;/p&gt;
    &lt;p&gt;
    With such a criteria of &quot;success&quot;,
    I wondered if &lt;cite&gt;Timeline&lt;/cite&gt; would have an audience
    much larger than one,
    and was quite surprised when it started getting thousands of
    web hits a day.
      The large audience &lt;cite&gt;Timeline 1.0&lt;/cite&gt; drew
      was a sign that there is an large appetite
      out there for
      accounts of parsing theory,
      an appetite so strong that anything resembling
      a coherent account
      was quickly devoured.
    &lt;p&gt;In response to the unexpectedly large audience,
    later versions of the &lt;cite&gt;Timeline&lt;/cite&gt; widened
    their focus.
      &lt;cite&gt;Timeline 3.1&lt;/cite&gt;
      was broadened to give good coverage
      of mainstream parsing practice
      including a lot of new material and original analysis.
      This brought in lot of material on topics
      which had little or no influence on my Earley/Leo work.
      The parsing of arithmetic expressions,
      for example,
      is trivial in the Earley/Leo context,
      and before my research for &lt;cite&gt;Timeline 3.0&lt;/cite&gt;
      I had devoted little attention to
      approaches that I felt amounted to
      needlessly doing things the hard way.
      But arithmetic expressions are at the borderline of power
      for traditional approaches
      and parsing arithmetic expressions was a central motivation
      for the authors of the algorithms that have so far
      been most influential on mainstream parsing.
      So in
      &lt;cite&gt;Timeline 3.1&lt;/cite&gt;
      arithmetic expresssions became a recurring theme,
      being brought back for detailed examination time and time again.
    &lt;/p&gt;
    &lt;h2&gt;Is the &quot;Timeline&quot; false?&lt;/h2&gt;
    &lt;p&gt;
      Is the &quot;Timeline&quot; false?
      The answer is yes, in three increasingly practical senses.
    &lt;/p&gt;
    &lt;p&gt;As Einstein makes clear,
    every theory that is about reality,
    will eventually proved be false.
    The best a theory can hope for is the fate of
    Newton's physics &amp;mdash;
    to be shown to be a subcase of a larger theory.
    &lt;/p&gt;
    &lt;p&gt;In a more specific sense,
    the truth of any theory of parsing history depends
    on its degree of success in explaining the facts.
    This means that the truth of the &quot;Timeline&quot; depends on which facts
    you require it to explain.
    If arbitrary choices of facts to be explained are allowed,
    the &quot;Timeline&quot; will certainly be seen to be false.
    &lt;/p&gt;But can the &quot;Timeline&quot; be shown to be false
    for criteria of success which are non-arbitrary?
    In the next section, I will describe four non-arbitrary
    criteria of success,
    all of which are of practical interest,
    and for all of which the &quot;Timeline&quot; is false.
    &lt;/p&gt;
    &lt;h2&gt;The Forever Five&lt;/h2&gt;
    &lt;p&gt;&quot;Success&quot; depends a lot on judgement,
    but my studies have led me to conclude that all but five algorithms
    are &quot;unsuccessful&quot; in the sense that,
    for everything that they do,
    at least one other algorithm does it better in practice.
    But this means there are five algorithms which &lt;b&gt;do&lt;/b&gt; solve
    some practical problems
    better than any other algorithm,
    including each of the other four.
    I call these the &quot;forever five&quot; because,
    if I am correct,
    these algorithms will be of permanent interest.
    &lt;/p&gt;
    &lt;p&gt;
      My &quot;Forever Five&quot; are regular expressions, recursive descent, PEG, Earley/Leo and Sakai's
      algorithm.&lt;a id=&quot;footnote-18-ref&quot; href=&quot;#footnote-18&quot;&gt;[18]&lt;/a&gt;
      Earley/Leo is the focus of my
      &lt;cite&gt;Timeline&lt;/cite&gt;, so that an effective
      critique of my &quot;Timeline&quot;
      could be a parsing historiography centering on any other of the other four.
    &lt;/p&gt;
    &lt;p&gt;For example, of the five, regular expressions are the most limited in parsing power.
      On the other hand, most of the parsing problems you encounter in practice
      are handled quite nicely by regular expressions.&lt;a id=&quot;footnote-19-ref&quot; href=&quot;#footnote-19&quot;&gt;[19]&lt;/a&gt;
      Good implementations of regular expressions are widely available.
      And, for speed, they are literally unbeatable -- if a parsing problem is a
      regular expression, no other algorithm will beat a dedicated regular expression
      engine for parsing it.
    &lt;/p&gt;
    &lt;p&gt;Could a
      &lt;cite&gt;Timeline&lt;/cite&gt;
      competitor be written which
      centered on regular expressions?
      Certainly.
      And if immediate usefulness to the average programmer is the criterion
      (and it is a very good criterion),
      then the
      &lt;cite&gt;Regular Expressions Timeline&lt;/cite&gt;
      would certainly give
      my timeline a run for the money.
    &lt;/p&gt;
    &lt;h2&gt;What about a PEG Timeline?&lt;/h2&gt;
    &lt;p&gt;
      The immediate impetus for this article was
      &lt;a href=&quot;https://groups.google.com/d/msg/marpa-parser/8EEq92TjR4E/dIzCnsITBQAJ&quot;&gt;a very collegial inquiry&lt;/a&gt;
      from Nicolas Laurent, a researcher whose main interest is PEG.
      Could a
      &lt;cite&gt;PEG Timeline&lt;/cite&gt;
      challenge mine?
      Again, very certainly.
    &lt;/p&gt;
    &lt;p&gt;Because there are at least some
      problems for which PEG is superior to everything else,
      my own Earley/Leo approach included.
      As one example, PEG
      could be an more powerful alternative to regular expressions.
    &lt;/p&gt;
    &lt;p&gt;That does not mean that I might not come back with
    a counter-critique.
    Among the questions that I might ask:
    &lt;ul&gt;
    &lt;li&gt;
      Is the PEG algorithm being proposed a future,
      or does it have an implementation?
    &lt;/li&gt;
    &lt;li&gt;What claims of speed and time complexity are made?
      Is there a way of determining in advance of runtime how fast
      your algorithm will run?
      Or is the expectation of practical speed
      on an &quot;implement and pray&quot; basis?
    &lt;/li&gt;
    &lt;li&gt;Does the proposed PEG algorithm match human parsing
      capabilities?
      If not, it is a claim for human exceptionalism,
      of a kind not usually accepted in modern computer science.
      How is exceptionalism justified in this case?
    &lt;/li&gt;
    &lt;/ul&gt;
    &lt;blockquote&gt;
    The search for truth is more precious
    than its possession. -- Einstein, quoting Lessing&lt;a id=&quot;footnote-20-ref&quot; href=&quot;#footnote-20&quot;&gt;[20]&lt;/a&gt;
    &lt;/blockquote&gt;
    &lt;h2&gt;Comments, etc.&lt;/h2&gt;
    &lt;p&gt;
      The background material for this post is in my
      &lt;a href=&quot;https://jeffreykegler.github.io/personal/timeline_v3&quot;&gt;
        Parsing: a timeline 3.0&lt;/a&gt;,
      and this post may be considered a supplement to &quot;Timelime&quot;.
      To learn about Marpa,
      my Earley/Leo-based parsing project,
      there is the
      &lt;a href=&quot;http://savage.net.au/Marpa.html&quot;&gt;semi-official web site, maintained by Ron Savage&lt;/a&gt;.
      The official, but more limited, Marpa website
      &lt;a href=&quot;http://jeffreykegler.github.io/Marpa-web-site/&quot;&gt;is my personal one&lt;/a&gt;.
      Comments on this post can be made in
      &lt;a href=&quot;http://groups.google.com/group/marpa-parser&quot;&gt;
        Marpa's Google group&lt;/a&gt;,
      or on our IRC channel: #marpa at freenode.net.
    &lt;/p&gt;
    &lt;h2&gt;Footnotes&lt;/h2&gt;
&lt;p id=&quot;footnote-1&quot;&gt;&lt;b&gt;1.&lt;/b&gt;
      Einstein, Albert and Infeld, Leopold,
        &lt;cite&gt;The Evolution of Physics&lt;/cite&gt;,
        Simon and Schuster, 2007, p. 3
 &lt;a href=&quot;#footnote-1-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-2&quot;&gt;&lt;b&gt;2.&lt;/b&gt;
        &quot;A dog reflects the family life.
        Whoever saw a frisky dog in a gloomy family, or a sad dog in a happy one?
        Snarling people have snarling dogs, dangerous people have dangerous ones.&quot;
        From &quot;The Adventure of the Creeping Man&quot;.
 &lt;a href=&quot;#footnote-2-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-3&quot;&gt;&lt;b&gt;3.&lt;/b&gt;
      Einstein and Infeld, p. 4.
 &lt;a href=&quot;#footnote-3-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-4&quot;&gt;&lt;b&gt;4.&lt;/b&gt;
      Einstein and Infeld, p. 71.
 &lt;a href=&quot;#footnote-4-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-5&quot;&gt;&lt;b&gt;5.&lt;/b&gt;
        Einstein and Infeld, p 78.
 &lt;a href=&quot;#footnote-5-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-6&quot;&gt;&lt;b&gt;6.&lt;/b&gt;
    Einstein and Infeld, p. 294.
 &lt;a href=&quot;#footnote-6-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-7&quot;&gt;&lt;b&gt;7.&lt;/b&gt;
      Einstein and Infeld, p. 31.
        See also Einstein,
	&quot;On the Method of Theoretical Physics&quot;,
        &lt;cite&gt;Ideas and Opinions&lt;/cite&gt;,
	Wings Books, New York,
	no publication date, p. 272.
 &lt;a href=&quot;#footnote-7-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-8&quot;&gt;&lt;b&gt;8.&lt;/b&gt;
    Dukas and Hoffman,
    &lt;cite&gt;Albert Einstein: The Human Side&lt;/cite&gt;,
    Princeton University Press, 2013,
    pp 32-33.
 &lt;a href=&quot;#footnote-8-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-9&quot;&gt;&lt;b&gt;9.&lt;/b&gt;
    &quot;On the Generalized Theory of Gravitation&quot;, in
    &lt;cite&gt;Ideas and Opinions&lt;/cite&gt;, p 342.
 &lt;a href=&quot;#footnote-9-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-10&quot;&gt;&lt;b&gt;10.&lt;/b&gt;
    &quot;Physics and Reality&quot;, in
    &lt;cite&gt;Ideas and Opinions&lt;/cite&gt;, pp. 294-295.
 &lt;a href=&quot;#footnote-10-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-11&quot;&gt;&lt;b&gt;11.&lt;/b&gt;
    &quot;Physics and Reality&quot;, in
    &lt;cite&gt;Ideas and Opinions&lt;/cite&gt;,
    p. 292.
 &lt;a href=&quot;#footnote-11-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-12&quot;&gt;&lt;b&gt;12.&lt;/b&gt;
    &quot;Physics and Reality&quot;, in
    &lt;cite&gt;Ideas and Opinions&lt;/cite&gt;,
    p. 292.
 &lt;a href=&quot;#footnote-12-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-13&quot;&gt;&lt;b&gt;13.&lt;/b&gt;
    Einstein and Infeld, p. 75.
 &lt;a href=&quot;#footnote-13-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-14&quot;&gt;&lt;b&gt;14.&lt;/b&gt;
    Einstein and Infeld, p. 35.
 &lt;a href=&quot;#footnote-14-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-15&quot;&gt;&lt;b&gt;15.&lt;/b&gt;
      Einstein and Infeld, pp. 7-8
 &lt;a href=&quot;#footnote-15-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-16&quot;&gt;&lt;b&gt;16.&lt;/b&gt;
	&quot;Physics and Reality&quot;,
        &lt;cite&gt;Ideas and Opinions&lt;/cite&gt;, p 290.
 &lt;a href=&quot;#footnote-16-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-17&quot;&gt;&lt;b&gt;17.&lt;/b&gt;
        Einstein and Infeld, p. 78.
 &lt;a href=&quot;#footnote-17-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-18&quot;&gt;&lt;b&gt;18.&lt;/b&gt;
        Three quibbles:
        Regular expressions do not find structure,
        so pedantically they are recognizers,
        not parsers.
        Recursive descent is technique for creating a family of algorithms,
        not an algorithm.
        And the algorithm first described by Sakai is more commonly
        called CYK, from the initials of three other researchers who re-discovered
        it over the years.
 &lt;a href=&quot;#footnote-18-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-19&quot;&gt;&lt;b&gt;19.&lt;/b&gt;
      A lot of this is because programmers learn to formulate problems in
      ways which avoid complex parsing so that,
      in practice,
      the alternatives are
      using regular expressions or rationalizing away the
      need for parsing.
 &lt;a href=&quot;#footnote-19-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-20&quot;&gt;&lt;b&gt;20.&lt;/b&gt;
    &quot;The Fundaments of Theoretical Physics&quot;, in
    &lt;cite&gt;Ideas and Opinions&lt;/cite&gt;, p. 335.
 &lt;a href=&quot;#footnote-20-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
  &lt;/body&gt;
&lt;/html&gt;</description>
  </item>
  <item>
    <title>Parsing Timeline 3.1</title>
    <link>http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/10/timeline_3_1.html</link>
    <description>&lt;html&gt;
  &lt;head&gt;
  &lt;/head&gt;
  &lt;body style=&quot;max-width:850px&quot;&gt;
    &lt;!--
      marpa_r2_html_fmt --no-added-tag-comment --no-ws-ok-after-start-tag
      --&gt;
    &lt;p&gt;
    &lt;h2&gt;Announcing Timeline 3.1&lt;/h2&gt;
    &lt;p&gt;I have just released
    &lt;a href=
    &quot;https://jeffreykegler.github.io/personal/timeline_v3&quot;&gt;
    version 3.1 of my Parsing Timeline&lt;/a&gt;.
    It is a painless introduction to
    a fascinating and important story
    which is scattered among
    one of the most
    forbidding literatures in computer science.
    Previous versions of this timeline have been,
    by far,
    the most popular of my writings.
    &lt;/p&gt;
    &lt;p&gt;A third of Timeline 3.1 is new,
    added since the 3.0 version.
    Much of the new material is adapted from previous
    blog posts, both old and recent.
    Other material is completely new.
    The sections that are not new with 3.1
    has been carefully reviewed and
    heavily revised.
    &lt;/p&gt;
    &lt;h2&gt;Comments, etc.&lt;/h2&gt;
    &lt;p&gt;My interest in parsing stems from my 
    own approach to it -- a parser in the Earley/Leo
    lineage named Marpa.
    To learn more about Marpa,
      a good first stop is the
      &lt;a href=&quot;http://savage.net.au/Marpa.html&quot;&gt;semi-official web site, maintained by Ron Savage&lt;/a&gt;.
      The official, but more limited, Marpa website
      &lt;a href=&quot;http://jeffreykegler.github.io/Marpa-web-site/&quot;&gt;is my personal one&lt;/a&gt;.
      Comments on this post can be made in
      &lt;a href=&quot;http://groups.google.com/group/marpa-parser&quot;&gt;
        Marpa's Google group&lt;/a&gt;,
      or on our IRC channel: #marpa at freenode.net.
    &lt;/p&gt;
    &lt;!--
    No footnotes in this one !!!
    &lt;h2&gt;Footnotes&lt;/h2&gt;
    --&gt;
  &lt;/body&gt;
&lt;/html&gt;</description>
  </item>
  <item>
    <title>Measuring language popularity</title>
    <link>http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/10/popularity.html</link>
    <description>&lt;html&gt;
  &lt;head&gt;
  &lt;/head&gt;
  &lt;body style=&quot;max-width:850px&quot;&gt;
    &lt;!--
      marpa_r2_html_fmt --no-added-tag-comment --no-ws-ok-after-start-tag
      --&gt;
    &lt;h2&gt;Language popularity&lt;/h2&gt;
    &lt;p&gt;
      &lt;a href=&quot;https://github.com/github/linguist&quot;&gt;Github's
        linguist&lt;/a&gt;
      is seen as the most trustworthy tool
      for estimating language popularity&lt;a id=&quot;footnote-1-ref&quot; href=&quot;#footnote-1&quot;&gt;[1]&lt;/a&gt;,
      in large part because it reports its result as
      the proportion of code in a very large dataset,
      instead of web hits or searches.&lt;a id=&quot;footnote-2-ref&quot; href=&quot;#footnote-2&quot;&gt;[2]&lt;/a&gt;
      It is ironic, in this context,
      that
      &lt;tt&gt;linguist&lt;/tt&gt;
      avoids looking at the code,
      preferring to use
      metadata -- file name and the vim and shebang lines.
      Scanning the actual code is &lt;tt&gt;linguist&lt;/tt&gt;'s last resort.&lt;a id=&quot;footnote-3-ref&quot; href=&quot;#footnote-3&quot;&gt;[3]&lt;/a&gt;
    &lt;/p&gt;
    &lt;p&gt;How accurate is this?
      For files that are mostly in a single programming language,
      currently the majority of them,
      &lt;tt&gt;linguist&lt;/tt&gt;'s method are probably very accurate.
    &lt;/p&gt;
    &lt;p&gt;But literate programming often requires mixing languages.
      It is perhaps an extreme example,
      but much of the code used in this blog post
      comes from a Markdown file, which contains both C and Lua.
      This code is &quot;untangled&quot; from the Lua by ad-hoc scripts&lt;a id=&quot;footnote-4-ref&quot; href=&quot;#footnote-4&quot;&gt;[4]&lt;/a&gt;.
      In my codebase,
      &lt;tt&gt;linguist&lt;/tt&gt;
      indentifies this code simply
      as Markdown.&lt;a id=&quot;footnote-5-ref&quot; href=&quot;#footnote-5&quot;&gt;[5]&lt;/a&gt;
      &lt;tt&gt;linguist&lt;/tt&gt;
      then ignores it,
      as it does all documentation files.&lt;a id=&quot;footnote-6-ref&quot; href=&quot;#footnote-6&quot;&gt;[6]&lt;/a&gt;.
    &lt;/p&gt;
    &lt;p&gt;Currently, this kind of homegrown
      literate programming may be so rare
      that it is not worth taking into account.
      But if literate programming becomes more popular,
      that trend might well slip under
      &lt;tt&gt;linguist&lt;/tt&gt;'s radar.
      And even those with a lot of faith in
      &lt;tt&gt;linguist&lt;/tt&gt;'s numbers should be happy to
      know they could be confirmed by more careful methods.
    &lt;/p&gt;
    &lt;h2&gt;Token-by-token versus line-by-line&lt;/h2&gt;
    &lt;p&gt;&lt;tt&gt;linguist&lt;/tt&gt; avoids reporting results based on looking at the code,
    because careful line counting for multiple languages
      cannot be done with traditional parsing methods.&lt;a id=&quot;footnote-7-ref&quot; href=&quot;#footnote-7&quot;&gt;[7]&lt;/a&gt;
      To do careful line counting,
      a parser must be able to handle ambiguity in several forms --
      ambiguous parses, ambiguous tokens, and overlapping variable-length tokens.
    &lt;/p&gt;
    &lt;p&gt;
      The ability to deal with
      &quot;overlapping variable-length tokens&quot; may sound like a bizarre requirement,
      but it is not.
      Line-by-line languages (BASIC, FORTRAN, JSON, .ini files, Markdown)
      and token-by-token languages (C, Java, Javascript, HTML)
      are both common,
      and even today commonly occur in the same file (POD and Perl,
      Haskell's Bird notation, Knuth's CWeb).
    &lt;/p&gt;
    &lt;p&gt;
      Deterministic parsing can switch back and forth,
      though at the cost of some very hack-ish code.
      But for careful line counting,
      you need to parse line-by-line and token-by-token
      simultaneously.
      Consider this example:
    &lt;/p&gt;
    &lt;pre&gt;&lt;tt&gt;
    int fn () { /* for later
\begin{code}
   */ int fn2(); int a = fn2();
   int b = 42;
   return  a + b; /* for later
\end{code}
*/ }
    &lt;/tt&gt;&lt;/pre&gt;
    &lt;p&gt;A reader can imagine that this code is part of a test case using code
      pulled from a LaTeX file.
      The programmer wanted to indicate the copied portion of code,
      and did so by commenting out its original LaTeX delimiters.
      GCC compiles this code without warnings.
    &lt;/p&gt;
    &lt;p&gt;It is not really the case that LaTeX is a line-by-line language.
      But in literate programming systems&lt;a id=&quot;footnote-8-ref&quot; href=&quot;#footnote-8&quot;&gt;[8]&lt;/a&gt;,
      it is usually required that the
      &lt;tt&gt;\begin{code}&lt;/tt&gt;
      and
      &lt;tt&gt;\end{code}&lt;/tt&gt;
      delimiters begin at column 0,
      and that the code block between them be a set of whole lines so,
      for our purposes in this post,
      we can treat LaTeX as line-by-line.
      For LaTeX, our parser finds
    &lt;/p&gt;&lt;pre&gt;&lt;tt&gt;
  L1c1-L1c29 LaTeX line: &quot;    int fn () { /* for later&quot;
  L2c1-L2c13 \begin{code}
  L3c1-L5c31 [A CODE BLOCK]
  L6c1-L6c10 \end{code}
  L7c1-L7c5 LaTeX line: &quot;*/ }&quot;&lt;a id=&quot;footnote-9-ref&quot; href=&quot;#footnote-9&quot;&gt;[9]&lt;/a&gt;
&lt;/tt&gt;&lt;/pre&gt;&lt;p&gt;
      Note that in the LaTeX parse, line alignment is respected perfectly:
      The first and last are ordinary LaTeX lines,
      the 2nd and 6th are commands bounding the code,
      and lines 3 through 5 are a code block.
    &lt;/p&gt;
    &lt;p&gt;
      The C tokenization, on the other hand,
      shows no respect for lines.
      Most tokens are a small part of their line,
      and the two comments start in the middle of
      a line and end in the middle of one.
      For example, the first comment starts at column 17
      of line 1 and ends at column 5 of line 3.&lt;a id=&quot;footnote-10-ref&quot; href=&quot;#footnote-10&quot;&gt;[10]&lt;/a&gt;
    &lt;/p&gt;
    &lt;p&gt;What language is our example in?
    Our example is long enough to justify classification,
    and it compiles as C code.
    So it seems best to classify this example as C code&lt;a id=&quot;footnote-11-ref&quot; href=&quot;#footnote-11&quot;&gt;[11]&lt;/a&gt;.
    Our parses give us enough data for a heuristic
    to make a decision capturing this intuition.&lt;a id=&quot;footnote-12-ref&quot; href=&quot;#footnote-12&quot;&gt;[12]&lt;/a&gt;
    &lt;/p&gt;
    &lt;h2&gt;Earley/Leo parsing and combinators&lt;/h2&gt;
    &lt;p&gt;In a series of previous posts&lt;a id=&quot;footnote-13-ref&quot; href=&quot;#footnote-13&quot;&gt;[13]&lt;/a&gt;,
      I have been developing a parsing method that
      integrates
      Earley/Leo parsing and combinator parsing.
      Everything in my previous posts is available
      in &lt;a href=
      &quot;https://metacpan.org/pod/distribution/Marpa-R2/pod/Marpa_R2.pod&quot;
      &gt;Marpa::R2&lt;/a&gt;,
      which was Debian stable as of jessie.
    &lt;/p&gt;
    &lt;p&gt;
      The final piece, added in this post, is the
      ability to use variable length subparsing&lt;a id=&quot;footnote-14-ref&quot; href=&quot;#footnote-14&quot;&gt;[14]&lt;/a&gt;,
      which I have just added to Marpa::R3,
      Marpa::R2's successor.
      Releases of &lt;a href=
      &quot;https://metacpan.org/pod/release/JKEGL/Marpa-R3-4.001_053/pod/Marpa_R3.pod&quot;
      &gt;Marpa::R3&lt;/a&gt;
      pass a full test suite,
      and the documentation is kept up to date,
      but R3 is alpha, and the usual cautions&lt;a id=&quot;footnote-15-ref&quot; href=&quot;#footnote-15&quot;&gt;[15]&lt;/a&gt;
      apply.
    &lt;/p&gt;
    &lt;p&gt;Earley/Leo parsing is linear for a superset
    of the LR-regular grammars,
    which includes all other grammar classes in practical use,
    and Earley/Leo allows the equivalent of infinite lookahead.&lt;a id=&quot;footnote-16-ref&quot; href=&quot;#footnote-16&quot;&gt;[16]&lt;/a&gt;
    When the power of Earley/Leo gives out,
    Marpa allows combinators (subparsers)
    to be invoked.
    The subparsers can be anything, including
    other Earley/Leo parsers,
    and they can be called recursively&lt;a id=&quot;footnote-17-ref&quot; href=&quot;#footnote-17&quot;&gt;[17]&lt;/a&gt;.
    Rare will be the grammar of practical interest that
    cannot be parsed with this combination of methods.
    &lt;/p&gt;
    &lt;h2&gt;The example&lt;/h2&gt;
    &lt;p&gt;The code that ran this example is &lt;a href=
    &quot;https://github.com/jeffreykegler/Marpa--R3/tree/08fa873687130fcfbe199a5f573375ad11322f3a/pub/varlex&quot;
    &gt;available on Github&lt;/a&gt;.
      In previous posts,
      we gave larger examples&lt;a id=&quot;footnote-18-ref&quot; href=&quot;#footnote-18&quot;&gt;[18]&lt;/a&gt;,
      and our tools and techniques have scaled.
      We expect that the variable-length subparsing
      feature will also scale -- while it was not available in
      Marpa::R2, it is not in itself new.
      Variable-length tokens have been available in other Marpa interfaces for
      years and they were described in Marpa's theory paper.&lt;a id=&quot;footnote-19-ref&quot; href=&quot;#footnote-19&quot;&gt;[19]&lt;/a&gt;.
    &lt;/p&gt;
    &lt;p&gt;
      The grammars used in the example of this post are minimal.
      Only enough LaTex is implemented
      to recognize code blocks; and
      only enough C syntax is implemented to recognize comments.
    &lt;/p&gt;
    &lt;h2&gt;The code, comments, etc.&lt;/h2&gt;
    &lt;p&gt;To learn more about Marpa,
      a good first stop is the
      &lt;a href=&quot;http://savage.net.au/Marpa.html&quot;&gt;semi-official web site, maintained by Ron Savage&lt;/a&gt;.
      The official, but more limited, Marpa website
      &lt;a href=&quot;http://jeffreykegler.github.io/Marpa-web-site/&quot;&gt;is my personal one&lt;/a&gt;.
      Comments on this post can be made in
      &lt;a href=&quot;http://groups.google.com/group/marpa-parser&quot;&gt;
        Marpa's Google group&lt;/a&gt;,
      or on our IRC channel: #marpa at freenode.net.
    &lt;/p&gt;
    &lt;h2&gt;Footnotes&lt;/h2&gt;
&lt;p id=&quot;footnote-1&quot;&gt;&lt;b&gt;1.&lt;/b&gt;
	This github repo for &lt;tt&gt;linguist&lt;/tt&gt; is &lt;a href=
	&quot;https://github.com/github/linguist/&quot;
	&gt;https://github.com/github/linguist/&lt;/a&gt;.
 &lt;a href=&quot;#footnote-1-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-2&quot;&gt;&lt;b&gt;2.&lt;/b&gt;
	Their methodology is often left vague,
	but it seems safe to say the careful line-by-line counting
	discussed in this post
	goes well beyond the techniques used in
	the widely-publicized lists of &quot;most popular programming
	languages&quot;. 
	&lt;br&gt;&lt;br&gt;
	In fact, it seems likely these measures do not use line
	counts at all,
	but instead report the sum of blob sizes.
	Github's &lt;tt&gt;linguist&lt;/tt&gt; does give a line count but
	Github does not vouch for its accuracy:
&quot;if you really need to know the lines of code of an entire repo, there are much better tools for this than Linguist.&quot;
        (Quoted from
        &lt;a href=
	&quot;https://github.com/github/linguist/issues/3131&quot;
	&gt;the resolution of
	Github linguist issue #1331&lt;/a&gt;.)
	The Github API's &lt;tt&gt;list-languages&lt;/tt&gt; command reports language sizes
	in bytes.
	The &lt;a href=
	  &quot;https://developer.github.com/v3/repos/#list-languages&quot;
	&gt;API documentation&lt;/a&gt;
	is vague, but it seems the counts are the
	sum of blob sizes,
	with each blob classed as one and only one language.
	&lt;br&gt;&lt;br&gt;
	Some tallies seem even more coarsely grained than this --
	they are not even blob-by-blob,
	but assign entire repos to the &quot;primary language&quot;.
	For more, see
        &lt;a href=&quot;https://techcrunch.com/2018/09/30/what-the-heck-is-going-on-with-measures-of-programming-language-popularity/&quot;&gt;
          Jon Evan's
          &lt;cite&gt;Techcrunch&lt;/cite&gt;
          article&lt;/a&gt;;
	  and &lt;a href=
	  &quot;https://www.benfrederickson.com/ranking-programming-languages-by-github-users/&quot;
	  &gt;Ben Frederickson's project&lt;/a&gt;.
 &lt;a href=&quot;#footnote-2-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-3&quot;&gt;&lt;b&gt;3.&lt;/b&gt;
        &lt;tt&gt;linguist&lt;/tt&gt;'s methodology is described in its README.md
	(&lt;a href=
	&quot;https://github.com/github/linguist/blob/8cd9d744caa7bd3920c0cb8f9ca494ce7d8dc206/README.md&quot;
	&gt;permalink as of 30 September 2018&lt;/a&gt;).
 &lt;a href=&quot;#footnote-3-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-4&quot;&gt;&lt;b&gt;4.&lt;/b&gt;
        This custom literate programming system is not documented or packaged,
	but those who cannot resist taking a look can find the Markdown
	file it processes &lt;a href=
	&quot;https://github.com/jeffreykegler/Marpa--R3/blob/f16ef5798986da69fa8b437edc3930ce2cebd498/cpan/kollos/kollos.md&quot;
	&gt;here&lt;/a&gt;,
	and its own code &lt;a href=
	&quot;https://github.com/jeffreykegler/Marpa--R3/blob/f16ef5798986da69fa8b437edc3930ce2cebd498/cpan/kollos/miranda&quot;&gt;
	here&lt;/a&gt;
	(permalinks accessed 2 October 2018).
 &lt;a href=&quot;#footnote-4-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-5&quot;&gt;&lt;b&gt;5.&lt;/b&gt;
        For those who care about getting
        &lt;tt&gt;linguist&lt;/tt&gt;
        as
        accurate as possible.
        there is a workaround:
        the
        &lt;tt&gt;linguist-language&lt;/tt&gt;
        git attribute.
        This still requires that each blob be 
	reported as containing lines of only one language.
 &lt;a href=&quot;#footnote-5-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-6&quot;&gt;&lt;b&gt;6.&lt;/b&gt;
        For the treatment of Markdown, see
        &lt;tt&gt;linguist&lt;/tt&gt;
        &lt;a href=&quot;https://github.com/github/linguist/blob/8cd9d744caa7bd3920c0cb8f9ca494ce7d8dc206/README.md#my-repository-isnt-showing-my-language&quot;&gt;README.md&lt;/a&gt;
        (permalink accessed as of 30 September 2018).
 &lt;a href=&quot;#footnote-6-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-7&quot;&gt;&lt;b&gt;7.&lt;/b&gt;
        Another possibility is a multi-scan approach -- one
        pass per language.
        But that is likely to be expensive.
        At last count there were 381 langauges in
        &lt;tt&gt;linguist&lt;/tt&gt;'s
        database.
        Worse, it won't solve the problem:
        &quot;liberal&quot; recognition even of a single language
        requires more power than available from
        traditional parsers.
 &lt;a href=&quot;#footnote-7-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-8&quot;&gt;&lt;b&gt;8.&lt;/b&gt;
      For example, these line-alignment requirements match 
      those in
      &lt;a href=
      &quot;https://www.haskell.org/onlinereport/haskell2010/haskellch10.html&quot;
      &gt;Section 10.4&lt;/a&gt; of the 2010 Haskell Language Report.
 &lt;a href=&quot;#footnote-8-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-9&quot;&gt;&lt;b&gt;9.&lt;/b&gt;
  Adapted from
  &lt;a href=
  &quot;https://github.com/jeffreykegler/Marpa--R3/blob/08fa873687130fcfbe199a5f573375ad11322f3a/pub/varlex/idlit_ex2.t#L83&quot;
  &gt;test code in Github repo&lt;/a&gt;, permalink accessed 2 October 2018.
 &lt;a href=&quot;#footnote-9-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-10&quot;&gt;&lt;b&gt;10.&lt;/b&gt;
      See the &lt;a href=
      &quot;https://github.com/jeffreykegler/Marpa--R3/blob/08fa873687130fcfbe199a5f573375ad11322f3a/pub/varlex/idlit_ex2.t#L44&quot;
      &gt;test file&lt;/a&gt;
      on Gihub.
 &lt;a href=&quot;#footnote-10-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-11&quot;&gt;&lt;b&gt;11.&lt;/b&gt;
    Some might think the two LaTex lines should be counted as LaTex and,
    using subparsing of comments, that heuristic can be implemented.
 &lt;a href=&quot;#footnote-11-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-12&quot;&gt;&lt;b&gt;12.&lt;/b&gt;
    To be sure, a useful tool would want to include considerably more of
    C's syntax.
    It is perhaps not necessary to be sure that a file compiles
    before concluding it is C.
    And we might want to class a file as C in spite of a
    fleeting failure to compile.
    But we do want to lower the probably of a false positive.
 &lt;a href=&quot;#footnote-12-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-13&quot;&gt;&lt;b&gt;13.&lt;/b&gt;
    &lt;a href=
    &quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/05/csg.html&quot;
    &gt;Marpa and procedural parsing&lt;/a&gt;;
    &lt;a href=
    &quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/05/combinator.html&quot;
    &gt;Marpa and combinator parsing&lt;/a&gt;;
    and &lt;a href=
    &quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/05/combinator2.html&quot;
    &gt;Marpa and combinator parsing 2&lt;/a&gt;
 &lt;a href=&quot;#footnote-13-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-14&quot;&gt;&lt;b&gt;14.&lt;/b&gt;
      There is &lt;a href=
      &quot;https://metacpan.org/pod/distribution/Marpa-R2/pod/Marpa_R2.pod&quot;
      &gt;documentation of the interface&lt;/a&gt;,
      but it is not a good starting point
      for a reader who has just started to look at the Marpa::R3 project.
      Once a user is familiar with Marpa::R3 standard DSL-based
      interface,
      they can start to learn about its alternatives &lt;a href=
      &quot;https://metacpan.org/pod/release/JKEGL/Marpa-R3-4.001_053/pod/External/Basic.pod&quot;
      &gt;here&lt;/a&gt;.
 &lt;a href=&quot;#footnote-14-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-15&quot;&gt;&lt;b&gt;15.&lt;/b&gt;
        Specifically,
	since Marpa::R3 is alpha,
	its features are subject
        to change without notice, even between micro releases,
        and changes are made without concern for backward compatibility.
        This makes R3 unsuitable for a production application.
        Add to this that,
	while R3 is tested, it has seen much less
        usage and testing than R2, which has been very stable for
        some time.
 &lt;a href=&quot;#footnote-15-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-16&quot;&gt;&lt;b&gt;16.&lt;/b&gt;
    Technically, a grammar is LR-regular if it can be parsed
    deterministically using a regular set as its lookahead.
    A &quot;regular set&quot; is a set of regular expressions.
    The regular set itself must be finite,
    but the regular expressions it contains
    can match lookaheads of arbitrary length.
 &lt;a href=&quot;#footnote-16-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-17&quot;&gt;&lt;b&gt;17.&lt;/b&gt;
    See &lt;a href=
    &quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/05/combinator2.html&quot;
    &gt;Marpa and combinator parsing 2&lt;/a&gt;
 &lt;a href=&quot;#footnote-17-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-18&quot;&gt;&lt;b&gt;18.&lt;/b&gt;
    The largest example is in &lt;a href=
    &quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/05/combinator2.html&quot;
    &gt;Marpa and combinator parsing 2&lt;/a&gt;
 &lt;a href=&quot;#footnote-18-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-19&quot;&gt;&lt;b&gt;19.&lt;/b&gt;
 Kegler, Jeffrey. &lt;cite&gt;Marpa, A Practical General Parser: The Recognizer&lt;/cite&gt;.
 &lt;a href=
 &quot;http://dinhe.net/~aredridel/.notmine/PDFs/Parsing/KEGLER,%20Jeffrey%20-%20Marpa,%20a%20practical%20general%20parser:%20the%20recognizer.pdf&quot;
&gt;Online version accessed of 24 April 2018&lt;/a&gt;.
The link is to the 19 June 2013 revision of the 2012 original.
 &lt;a href=&quot;#footnote-19-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
  &lt;/body&gt;
&lt;/html&gt;</description>
  </item>
  <item>
    <title>A Haskell challenge</title>
    <link>http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/08/rntz.html</link>
    <description>&lt;html&gt;
  &lt;head&gt;
  &lt;/head&gt;
  &lt;body style=&quot;max-width:850px&quot;&gt;
    &lt;!--
      marpa_r2_html_fmt --no-added-tag-comment --no-ws-ok-after-start-tag
      --&gt;
    &lt;h2&gt;The challenge&lt;/h2&gt;
    &lt;p&gt;
    A &lt;a href=&quot;http://www.rntz.net/post/2018-07-10-parsing-list-comprehensions.html&quot;&gt;recent
    blog post by Michael Arntzenius&lt;/a&gt; ended with a friendly challenge to Marpa.
    Haskell list comprehensions are something that
    Haskell's own parser handles only with difficulty.
    A point of Michael's critique of Haskell's parsing was
    that Haskell's list comprehension could be even more powerful if not
    for these syntactic limits.
    &lt;/p&gt;
    Michael wondered aloud if Marpa could do better.
    It can.
    &lt;/p&gt;
    &lt;p&gt;The problem syntax occurs with the &quot;guards&quot;,
    a very powerful facility of
    Haskell's list comprehension.
    Haskell allows several kinds of &quot;guards&quot;.
    Two of these &quot;guards&quot; can have the same prefix,
    and these ambiguous prefixes can
    be of arbitrary length.
    In other words,
    parsing Haskell's list comprehension requires
    either lookahead of arbitrary length,
    or its equivalent.
    &lt;p&gt;
    &lt;p&gt;To answer Michael's challenge,
    I extended my Haskell subset parser to deal with
    list comprehension.
    That parser, with its test examples, is online.&lt;a id=&quot;footnote-1-ref&quot; href=&quot;#footnote-1&quot;&gt;[1]&lt;/a&gt;
    I have run it for examples thousands of tokens long and,
    more to the point,
    have checked the Earley sets to ensure that Marpa
    will stay linear,
    no matter how long the ambiguous prefix gets.&lt;a id=&quot;footnote-2-ref&quot; href=&quot;#footnote-2&quot;&gt;[2]&lt;/a&gt;
    &lt;/p&gt;
    Earley parsing, which Marpa uses,
    accomplishes the seemingly impossible here.
    It does the equivalent of infinite lookahead efficiently,
    without actually doing any lookahead or
    backtracking.
    That Earley's algorithm can do this has been a settled
    fact in the literature for some time.
    But today Earley's algorithm is little known even
    among those well acquainted with parsing,
    and to many claiming the equivalent of infinite lookahead,
    without actually doing any lookahead at all,
    sounds like a boast of magical powers.
    &lt;/p&gt;
    &lt;p&gt;
    In the rest of this blog post,
    I hope to indicate how Earley parsing follows more than
    one potential parse at a time.
    I will not describe Earley's algorithm in full.&lt;a id=&quot;footnote-3-ref&quot; href=&quot;#footnote-3&quot;&gt;[3]&lt;/a&gt;
    But I will show that no magic is involved,
    and that in fact the basic ideas behind Earley's method
    are intuitive and reasonable.
    &lt;/p&gt;
    &lt;h2&gt;A quick cheat sheet on list comprehension&lt;/h2&gt;
    &lt;p&gt;
    List comprehension in Haskell is impressive.
    Haskell allows
    you to build a list using a series of &quot;guards&quot;,
    which can be of several kinds.
    The parsing issue arises because two of the guard types --
    generators and boolean expressions --
    must be treated quite differently,
    but can look the same over an arbitrarily long prefix.
    &lt;/p&gt;
    &lt;h3&gt;Generators&lt;/h3&gt;
    &lt;p&gt;Here is one example of a Haskell generator,
    from the test case for this blog post:
    &lt;/p&gt;
    &lt;pre&gt;&lt;tt&gt;
          list = [ x | [x, 1729,
		      -- insert more here
		      99
		   ] &lt;- xss ] &lt;/tt&gt;&lt;a id=&quot;footnote-4-ref&quot; href=&quot;#footnote-4&quot;&gt;[4]&lt;/a&gt;&lt;/pre&gt;
    &lt;p&gt;
    This says to build a lists of &lt;tt&gt;x&lt;/tt&gt;'s
    such that the guard
    &lt;tt&gt;[x, 1729, 99 ] &amp;lt;- xss&lt;/tt&gt;
    holds.
    The clue that this guard is a generator is the
    &lt;tt&gt;&amp;lt;-&lt;/tt&gt; operator.
    The &lt;tt&gt;&amp;lt;-&lt;/tt&gt; operator
    will appear in every generator,
    and means &quot;draw from&quot;.
    &lt;/p&gt;
    &lt;p&gt;
    The LHS of the &lt;tt&gt;&amp;lt;-&lt;/tt&gt; operator is a pattern
    and the RHS is an expression.
    This generator draws all the elements from &lt;tt&gt;xss&lt;/tt&gt;
    which match the pattern &lt;tt&gt;[x, 1729, 99 ]&lt;/tt&gt;.
    In other words, it draws out
    all the elements of &lt;tt&gt;xss&lt;/tt&gt;,
    and tests that they
    are lists of length 3
    whose last two subelements are 1729 and 99.
    &lt;/p&gt;
    &lt;p&gt;The variable &lt;tt&gt;x&lt;/tt&gt; is set to the 1st subelement.
    &lt;tt&gt;list&lt;/tt&gt; will be a list of all those &lt;tt&gt;x&lt;/tt&gt;'s.
    In the test suite, we have
    &lt;pre&gt;&lt;tt&gt;
    xss = [ [ 42, 1729, 99 ] ] &lt;/tt&gt;&lt;a id=&quot;footnote-5-ref&quot; href=&quot;#footnote-5&quot;&gt;[5]&lt;/a&gt;&lt;/pre&gt;
    &lt;/p&gt;
    so that list becomes &lt;tt&gt;[42]&lt;/tt&gt; -- a list
    of one element whose value is 42.
    &lt;/p&gt;
    &lt;h3&gt;Boolean guards&lt;/h3&gt;
    &lt;p&gt;Generators can share very long prefixes with Boolean guards.
    &lt;pre&gt;&lt;tt&gt;
	list2 = [ x | [x, 1729, 99] &amp;lt;- xss,
               [x, 1729,
                  -- insert more here
                  99
               ] == ys,
             [ 42, 1729, 99 ] &amp;lt;- xss
             ] &lt;/tt&gt;&lt;a id=&quot;footnote-6-ref&quot; href=&quot;#footnote-6&quot;&gt;[6]&lt;/a&gt;&lt;/pre&gt;
    &lt;/p&gt;
    &lt;p&gt;The expression defining &lt;tt&gt;list2&lt;/tt&gt;
    has 3 comma-separated guards:
    The first guard is a generator,
    the same one as in the previous example.
    The last guard is also a generator.
    &lt;/p&gt;
    &lt;p&gt;
    The middle guard is of a new type: it is a Boolean:
    &lt;tt&gt;[x, 1729, 99 ] == ys&lt;/tt&gt;.
    This guard insists that &lt;tt&gt;x&lt;/tt&gt; be such that the triple
    &lt;tt&gt;[x, 1729, 99 ]&lt;/tt&gt; is equal to &lt;tt&gt;ys&lt;/tt&gt;.
    &lt;/p&gt;
    &lt;p&gt;
    In the test suite, we have
    &lt;pre&gt;&lt;tt&gt;
    ys = [ 42, 1729, 99 ] &lt;/tt&gt;&lt;a id=&quot;footnote-7-ref&quot; href=&quot;#footnote-7&quot;&gt;[7]&lt;/a&gt;&lt;/pre&gt;
    so that &lt;tt&gt;list2&lt;/tt&gt; is also
    &lt;tt&gt;[42]&lt;/tt&gt;.
    &lt;/p&gt;
    &lt;h2&gt;Boolean guards versus generators&lt;/h2&gt;
    &lt;p&gt;From the parser's point of view, Boolean guards
    and generators start out looking the same --
    in the examples above, three of our guards start out
    the same -- with the string &lt;tt&gt;[x, 1729, 99 ]&lt;/tt&gt;,
    but
    &lt;ul&gt;
    &lt;li&gt;in one case (the Boolean guard),
    &lt;tt&gt;[x, 1729, 99 ]&lt;/tt&gt; is the beginning of an expression; and &lt;/li&gt;
    &lt;li&gt;in the other two cases (the generators),
    &lt;tt&gt;[x, 1729, 99 ]&lt;/tt&gt; is a pattern.&lt;/li&gt;
    &lt;/ul&gt;
    Clearly patterns and expressions can look identical.
    And they can look identical for an arbitrarily long time --
    I tested the &lt;a href=&quot;https://www.haskell.org/ghc/&quot;&gt;Glasgow Haskell Compiler&lt;/a&gt;
    (GHC)
    with identical expression/pattern prefixes
    thousands of tokens in length.
    My virtual memory eventually gives out,
    but GHC itself never complains.&lt;a id=&quot;footnote-8-ref&quot; href=&quot;#footnote-8&quot;&gt;[8]&lt;/a&gt;
    (The comments &quot;&lt;tt&gt;insert more here&lt;/tt&gt;&quot; show the points at which the
    comma-separated lists of integers can be extended.)
    &lt;/p&gt;
    &lt;h2&gt;The problem for parsers&lt;/h2&gt;
    &lt;p&gt;So Haskell list comprehension presents a problem for parsers.
    A parser must determine whether it is parsing an expression or
    a pattern, but it cannot know this for an arbitrarily long time.
    A parser must keep track of two possibilities at once --
    something traditional parsing has refused to do.
    As I have pointed out&lt;a id=&quot;footnote-9-ref&quot; href=&quot;#footnote-9&quot;&gt;[9]&lt;/a&gt;,
    belief that traditional parsing &quot;solves&quot; the parsing problem is
    belief in human exceptionalism --
    that human have calculating abilities that Turing machines do not.
    Keeping two possibilites in mind for a long time is trivial for
    human beings -- in one form we call it worrying,
    and try to prevent ourselves from doing it obsessively.
    But it has been the orthodoxy that practical parsing algorithms
    cannot do this.
    &lt;/footnote&gt;
    &lt;/p&gt;
    &lt;p&gt;Arntzenius has a nice summary of the attempts to parse this
    construct while only allowing one possibility at a time --
    that is, determistically.
    Lookahead clearly cannot work -- it would have to be arbitrarily
    long.
    Backtracking can work, but can be very costly
    and is a major obstacle to quality error reporting.
    &lt;/p&gt;
    &lt;p&gt;
    GHC avoids the problems with backtracking by using post-processing.
    At parsing time, GHC treats an ambiguous guard as a
    Boolean.
    Then, if it turns out that is a generator,
    it rewrites it in post-processing.
    This inelegance incurs some real technical debt --
    either a pattern must &lt;b&gt;always&lt;/b&gt; be a valid expression,
    or even more trickery must be resorted to.&lt;a id=&quot;footnote-10-ref&quot; href=&quot;#footnote-10&quot;&gt;[10]&lt;/a&gt;
    &lt;h2&gt;The Earley solution&lt;/h2&gt;
    &lt;/p&gt;
    &lt;p&gt;Earley parsing deals with this issue by doing what 
    a human would do --
    keeping both possibilities in mind at once.
    Jay Earley's innovation was to discover a way for a computer
    to track multiple possible parses
    that is compact,
    efficient to create,
    and efficient to read.
    &lt;/p&gt;
    &lt;p&gt;
    Earley's algorithm maintains an &quot;Earley table&quot;
    which contains &quot;Earley sets&quot;,
    one for each token.
    Each Earley set contains &quot;Earley items&quot;.
    Here are some Earley items from Earley set 25 
    in one of our test cases:&lt;br&gt;
    &lt;pre&gt;&lt;tt&gt;
	origin = 22; &amp;lt;atomic expression&amp;gt; ::=   '[' &amp;ltexpression&amp;gt; '|' . &amp;ltguards&amp;gt; ']'
	origin = 25; &amp;lt;guards&amp;gt; ::= . &amp;lt;guard&lt;&amp;gt;
	origin = 25; &amp;lt;guards&amp;gt; ::= . &amp;lt;guards&amp;gt; ',' &amp;lt;guard&lt;&amp;gt;
	origin = 25; &amp;lt;guard&lt;&amp;gt;  ::= . &amp;lt;pattern&amp;gt; '&amp;lt; &amp;lt;expression&amp;gt;
	origin = 25; &amp;lt;guard&lt;&amp;gt;  ::= . &amp;lt;expression&amp;gt; &lt;/tt&gt;&lt;a id=&quot;footnote-11-ref&quot; href=&quot;#footnote-11&quot;&gt;[11]&lt;/a&gt;&lt;/pre&gt;
     &lt;p&gt;
     In the code, these represent the state of the parse just after
     the pipe symbol (&quot;&lt;tt&gt;|&lt;/tt&gt;&quot;) on line 4 of our test code.
    &lt;/p&gt;
    Each Earley item describes progress in one rule of the grammar.
    There is a dot (&quot;&lt;tt&gt;.&lt;/tt&gt;&quot;) in each rule,
    which indicates how far the parse
    has progressed inside the rule.
    One of the rules has the dot just after the pipe symbol,
    as you would expect, since we have just seen a pipe symbol.
    &lt;/p&gt;
    &lt;p&gt;
    The other four rules have the dot at the beginning of the RHS.
    These four rules are &quot;predictions&quot; -- none of their symbols
    have been parsed yet, but we know that these rules might occur,
    starting at the location of this Earley set.
    &lt;/p&gt;
    &lt;p&gt;
    Each item also records an &quot;origin&quot;: the location in the input where
    the rule described in the item began.
    For predictions the origin is always the same as the Earley set.
    For the first Earley item, the origin is 3 tokens earlier,
    in Earley set 22.
    &lt;/p&gt;
    &lt;p&gt;
    &lt;h2&gt;The &quot;secret&quot; of non-determinism&lt;/h2&gt;
    &lt;p&gt;
    And now we have come to the secret of efficient non-deterministic parsing --
    a &quot;secret&quot;
    which I hope to convince the reader is not magic,
    or even much of a mystery.
    Here, again, are two of the items from Earley set 25:&lt;/p&gt;
    &lt;pre&gt;&lt;tt&gt;
	origin = 25; &amp;lt;guard&lt;&amp;gt;  ::= . &amp;lt;pattern&amp;gt; '&amp;lt; &amp;lt;expression&amp;gt;
	origin = 25; &amp;lt;guard&lt;&amp;gt;  ::= . &amp;lt;expression&amp;gt; &lt;/tt&gt; &lt;a id=&quot;footnote-12-ref&quot; href=&quot;#footnote-12&quot;&gt;[12]&lt;/a&gt;&lt;/pre&gt;
    &lt;/p&gt;
    &lt;p&gt;At this point there are two possibilities going forward --
    a generator guard or a Boolean expression guard.
    And there is an Earley item for each of these possibilities in the Earley set.
    &lt;/p&gt;
    &lt;p&gt;
    That is the basic idea -- that is all there is to it.
    Going forward in the parse, for as long as both possibilities stay
    live, Earley items for both will appear in the Earley sets.
    &lt;/p&gt;
    &lt;p&gt;From this point of view,
    it should now be clear why the Earley algorithm can keep track
    of several possibilities without lookahead or backtracking.
    No lookahead is needed because all possibilities are in the
    Earley set, and selection among them will take place as the
    rest of the input is read.
    And no backtracking is needed because every possibility
    was already recorded -- there is nothing new to be found
    by backtracking.
    &lt;/p&gt;
    &lt;p&gt;It may also be clearer why I claim that Marpa is left-eidetic,
    and how the Ruby Slippers work.&lt;a id=&quot;footnote-13-ref&quot; href=&quot;#footnote-13&quot;&gt;[13]&lt;/a&gt;
    Marpa has perfect knowledge of everything in the parse so far,
    because it is all in the Earley tables.
    And, given left-eidetic knowledge, Marpa also knows what
    terminals are expected at the current location,
    and can &quot;wish&quot; them into existence as necessary.
    &lt;/p&gt;
    &lt;h2&gt;The code, comments, etc.&lt;/h2&gt;
    &lt;p&gt;A permalink to the
    full code and a test suite for this prototype,
    as described in this blog post,
    is
    &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/tree/0df0aef7d6cb8590d3a33f857619e75f84786dd7/code/haskell&quot;&gt;
    on Github&lt;/a&gt;.
    In particular,
    the permalink of the
    the test suite file for list comprehension is
    &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/blob/0df0aef7d6cb8590d3a33f857619e75f84786dd7/code/haskell/listcomp.t&quot;&gt;
    here&lt;/a&gt;.
    I expect to update this code,
    and the latest commit can be found
    &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/tree/gh-pages/code/haskell&quot;&gt;
    here&lt;/a&gt;.
    &lt;/p&gt;
    &lt;p&gt;
      To learn more about Marpa,
      a good first stop is the
      &lt;a href=&quot;http://savage.net.au/Marpa.html&quot;&gt;semi-official web site, maintained by Ron Savage&lt;/a&gt;.
      The official, but more limited, Marpa website
      &lt;a href=&quot;http://jeffreykegler.github.io/Marpa-web-site/&quot;&gt;is my personal one&lt;/a&gt;.
      Comments on this post can be made in
      &lt;a href=&quot;http://groups.google.com/group/marpa-parser&quot;&gt;
        Marpa's Google group&lt;/a&gt;,
      or on our IRC channel: #marpa at freenode.net.
    &lt;/p&gt;
    &lt;h2&gt;Footnotes&lt;/h2&gt;
&lt;p id=&quot;footnote-1&quot;&gt;&lt;b&gt;1.&lt;/b&gt;
    If you are interested in my Marpa-driven Haskell subset parser,
    &lt;a href=&quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/05/combinator2.html&quot;&gt;
    this blog post&lt;/a&gt;
    may be the best introduction.
    The code is
    &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/tree/gh-pages/code/haskell&quot;&gt;
    on Github&lt;/a&gt;.
 &lt;a href=&quot;#footnote-1-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-2&quot;&gt;&lt;b&gt;2.&lt;/b&gt;
    The Earley sets for the ambigious prefix immediately reach a size
    of 46 items, and then stay at that level.
    This is experimental evidence that the Earley set
    sizes stay constant.
    &lt;br&gt;&lt;br&gt;
    And, if the Earley items are examined,
    and their derivations traced,
    it can be seen that
    they must repeat the same Earley item count
    for as long as the ambiguous prefix continues.
    The traces I examined are
    &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/blob/0df0aef7d6cb8590d3a33f857619e75f84786dd7/code/haskell/listcomp_trace.out&quot;&gt;here&lt;/a&gt;,
    and the code which generated them is
    &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/blob/0df0aef7d6cb8590d3a33f857619e75f84786dd7/code/haskell/listcomp_ex.pl&quot;&gt;here&lt;/a&gt;,
    for the
    reader who wants to convince himself.
    &lt;br&gt;&lt;br&gt;
    The guard prefixes of Haskell are ambiguous,
    but (modulo mistakes in the standards)
    the overall Haskell grammar is not.
    In the literature on Earley's,
    it has been shown that for an unambiguous grammar,
    each Earley item has an constant amortized cost in time.
    Therefore,
    if a parse produces
    a Earley sets that are all of less than a constant size,
    it must have linear time complexity.
 &lt;a href=&quot;#footnote-2-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-3&quot;&gt;&lt;b&gt;3.&lt;/b&gt;
    There are many descriptions of Earley's algorithm out there.
    &lt;a href=&quot;https://en.wikipedia.org/wiki/Earley_parser&quot;&gt;The
    Wikipedia page on Earley's algorithm&lt;/a&gt;
    (accessed 27 August 2018)
    is one good place to start.
    I did
    another very simple introduction to Earley's in
    &lt;a href=&quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2010/06/jay-earleys-idea.html&quot;&gt;an
    earlier blog post&lt;/a&gt;,
    which may be worth looking at.
    Note that Marpa contains
    &lt;a href=&quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2011/11/what-is-the-marpa-algorithm.html&quot;&gt;
    improvements to Earley's algorithm&lt;/a&gt;.
    Particularly, to fulfill Marpa's claim of linear time for all
    LR-regular grammars, Marpa uses Joop Leo's speed-up.
    But Joop's improvement is &lt;b&gt;not&lt;/b&gt; necessary or useful
    for parsing
    Haskell list comprehension,
    is not used in this example,
    and will not be described in this post.
 &lt;a href=&quot;#footnote-3-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-4&quot;&gt;&lt;b&gt;4.&lt;/b&gt;
    &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/blob/0df0aef7d6cb8590d3a33f857619e75f84786dd7/code/haskell/listcomp.t#L30&quot;&gt;
    Permalink to this code&lt;/a&gt;,
    accessed 27 August 2018.
 &lt;a href=&quot;#footnote-4-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-5&quot;&gt;&lt;b&gt;5.&lt;/b&gt;
    &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/blob/0df0aef7d6cb8590d3a33f857619e75f84786dd7/code/haskell/listcomp.t#L25&quot;&gt;
    Permalink to this code&lt;/a&gt;,
    accessed 27 August 2018.
 &lt;a href=&quot;#footnote-5-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-6&quot;&gt;&lt;b&gt;6.&lt;/b&gt;
    &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/blob/0df0aef7d6cb8590d3a33f857619e75f84786dd7/code/haskell/listcomp.t#L35&quot;&gt;
    Permalink to this code&lt;/a&gt;,
    accessed 27 August 2018.
 &lt;a href=&quot;#footnote-6-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-7&quot;&gt;&lt;b&gt;7.&lt;/b&gt;
    &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/blob/0df0aef7d6cb8590d3a33f857619e75f84786dd7/code/haskell/listcomp.t#L28&quot;&gt;
    Permalink to this code&lt;/a&gt;,
    accessed 27 August 2018.
 &lt;a href=&quot;#footnote-7-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-8&quot;&gt;&lt;b&gt;8.&lt;/b&gt;
    Note that if the list is extended, the patterns matches and Boolean
    tests fail, so that 42 is no longer the answer.
    From the parsing point of view, this is immaterial.
 &lt;a href=&quot;#footnote-8-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-9&quot;&gt;&lt;b&gt;9.&lt;/b&gt;
    In several places, including
    &lt;a href=&quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/07/knuth_1965_2.html&quot;&gt;
    this blog post&lt;/a&gt;.
 &lt;a href=&quot;#footnote-9-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-10&quot;&gt;&lt;b&gt;10.&lt;/b&gt;
    This account of the state of the art summarizes
    &lt;a href=&quot;http://www.rntz.net/post/2018-07-10-parsing-list-comprehensions.html&quot;&gt;
    Arntzenius's recent post&lt;/a&gt;,
    which should be consulted for the details.
 &lt;a href=&quot;#footnote-10-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-11&quot;&gt;&lt;b&gt;11.&lt;/b&gt;
     Adapted from
     &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/blob/0df0aef7d6cb8590d3a33f857619e75f84786dd7/code/haskell/listcomp_trace.out#L811&quot;&gt;
     this trace output&lt;/a&gt;,
     accessed 27 August 2018.
 &lt;a href=&quot;#footnote-11-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-12&quot;&gt;&lt;b&gt;12.&lt;/b&gt;
     Adapted from
     &lt;a href=&quot;https://github.com/jeffreykegler/Ocean-of-Awareness-blog/blob/0df0aef7d6cb8590d3a33f857619e75f84786dd7/code/haskell/listcomp_trace.out#L811&quot;&gt;
     this trace output&lt;/a&gt;,
     accessed 27 August 2018.
 &lt;a href=&quot;#footnote-12-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
&lt;p id=&quot;footnote-13&quot;&gt;&lt;b&gt;13.&lt;/b&gt;
    For more on the Ruby Slippers see
    my &lt;a href=&quot;http://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2018/05/combinator2.html&quot;&gt;
    just previous blog post&lt;/a&gt;,
 &lt;a href=&quot;#footnote-13-ref&quot;&gt;&amp;#8617;&lt;/a&gt;&lt;/p&gt;
  &lt;/body&gt;
&lt;/html&gt;</description>
  </item>
  </channel>
</rss>