thesis.tex

\documentclass[a4paper,10pt,twoside]{report}
%\documentclass[b5paper,8pt,twoside]{article}
%\documentclass[a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
%\usepackage{fontspec}
\usepackage{fixltx2e}
\usepackage{graphicx}
\usepackage{longtable}
\usepackage{float}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{textcomp}
\usepackage{marvosym}
\usepackage{wasysym}
\usepackage{amssymb}
\usepackage[hidelinks]{hyperref}
\usepackage{cite}
\tolerance=1000
%\usepackage{minted}
\usepackage{todonotes}
\usepackage{comment}
\usepackage{listings}
\usepackage{lipsum}
\usepackage{tikz}
\usepackage{syntax}

\setcounter{secnumdepth}{4}

\usetikzlibrary{trees}
%\tikzstyle op=[draw, circle]
%\tikzstyle arg=[]
\tikzstyle op=[inner sep=0,draw,circle,minimum size=0.8cm,execute at begin node=$,execute at end node=$]
\tikzstyle arg=[minimum size=.6cm,inner sep=0,draw,circle,fill=gray!50!white]

\author{Nathanael Schweers}
\date{\today}
\title{Proof of Concept: An implementation of a reader for extensible syntaxes for Emacs Lisp}
% \hypersetup{
%   pdfkeywords={},
%   pdfsubject={},
%   pdfcreator={Emacs 25.0.91.1 (Org mode 8.2.10)}}

\newcommand{\el}{Emacs Lisp}
\newcommand{\cl}{Common Lisp}
\newcommand{\elr}{el-reader}
\newcommand{\sym}[1]{\texttt{#1}}
\newcommand{\fun}[1]{\texttt{#1}}
\newcommand{\emacs}{GNU~Emacs}
\newcommand{\emacsv}[1]{GNU~Emacs~#1}
\newcommand{\unix}{\textsc{unix}}
\newcommand{\windows}{MS Windows\texttrademark}
\newcommand{\Read}{\fun{read}}
\newcommand{\nil}{\sym{nil}}
\newcommand{\tee}{\sym{t}}

% \newenvironment{myEnv} [2]
% {\begin{frame} \frametitle{#1} \framesubtitle{#2} \begin{center}}
% {\end{center} \end{frame}}

% \newenvironment{sourcecode} [2]
% {\begin{listing}[h] \label{#1} \caption{#2} \begin{verbatim}}
% {\end{verbatim} \end{listing}}

%\newminted{cl}{}

\lstset{         %
  language=Lisp, %
  frame=single,  %
  %float,         %
  captionpos=b}

\lstdefinestyle{lispcode}{
  language=Lisp,
  basicstyle=\scriptsize,
  frame=single,
  %float,
  captionpos=b,
  %stringstyle=\color{red},
  %keywordstyle=\bfseries\color{purple}
  showstringspaces=false
}

\lstdefinestyle{lispinline}{
  language=Lisp, 
  showstringspaces=false}

\lstdefinestyle{pythoncode}{
  language=python,
  basicstyle=\small,
  showstringspaces=false
}

\lstdefinestyle{rubycode}{
  language=ruby,
  basicstyle=\small,
  float,
  captionpos=b,
  showstringspaces=false
}

\lstdefinestyle{rubyinline}{
  language=ruby,
  basicstyle=\small,
  showstringspaces=false
}

\begin{document}

\maketitle

\begin{abstract}
  This thesis discusses the design and implementation of a proof of concept for
  an \el{} compatible reader for extensible syntaxes, in \el{}, named
  ``\elr{}''.  In Lisp parlance, the reader is the part of the
  interpreter/compiler, which converts the characters in the source code to data
  structures, which are then further processed.  In an extensible reader, new
  syntactic constructs may be defined from within the language.  These new
  constructs take effect when specific characters are encountered in the source
  code, and may interact with the existing reader to create arbitrary data
  structures which are later either interpreted or compiled by the Lisp
  environment.

  Some other languages feature such capabilities, of which \cl{} is most widely
  known.  Some Scheme implementations also provide these or similar
  possibilities, yet this is not part of the standard.  SRFI-10 provides a
  specific character sequence with which all read macros must
  start. \cite{srfi-10}

  In \cl{} the possibility to interact with the reader is called a read macro or
  reader macro, and these terms will also be used throughout this thesis.

  \el{} does not provide reader macros.  As of the upcoming version 25.1 the
  reader is implemented in C and does not provide any hooks into it, apart from
  the usual advice system which all \el{} functions provide.

  The code underlying this thesis provides a means to replace the reader built
  into Emacs.  The main point of this thesis is to show that the flexible nature
  of a Lisp environment may be used to replace even such a crucial part of the
  system as the reader at runtime without affecting unrelated parts or making
  the original part unavailable.

  While a short recap of Lisps basics---especially where relevant to reading
  code---is given, a working knowledge of Lisp and modern language constructs
  like closures is assumed.  Pointers to additional Lisp resources are provided
  in the text.
\end{abstract}

\tableofcontents

\chapter{Introduction}
\label{chapter:introduction}

In this thesis a proof of concept for a new reader for \emacs{} is discussed.
The reader (called \elr{}) provides \cl{}-style reader macros, which allow the
user of the language to extend the syntax of the language without leaving the
language or even restarting the program.

The advantage of such a system is that new syntactic constructs may be
implemented without the restriction of merely re-using existing ones, as would
be the case using operator overloading.  Also, several completely
different---even incompatible---syntaxes may coexist within one program, as long
as their uses do not overlap.  To a certain extent these syntaxes can be
composable, yet in such a case caution is advised.

The outline of this thesis is as follows:

Chapter~\ref{chapter:introduction} provides a short history of how \emacs{} came
into being and how the language extended here (\el{}) is part of \emacs{}.
Other Emacs variants (also known as \emph{emacsen}) are not discussed here.
Also a short history of Lisp is given, as well as a few basics of the language
and what makes it special.  Also some motivation is given as to why the reader
should be replaced.

% Chapter ~\ref{chapter:lisp-hist} gives a brief history of the conception and
% further development of Lisp and its dialects.

% Chapter ~\ref{chapter:lisp-special} shows how Lisps in general differ from other
% programming languages.  This chapter is especially important for people
% unfamiliar with Lisps in general, yet will hopefully also be of interest to the
% more casual lisper.

% Chapter ~\ref{chapter:lisp-basics} gives a short introduction to programming in
% Lisp, as Lisp knowledge will make the rest of this thesis much easier to
% understand.

% Armed with the knowledge of previous chapters, chapter ~\ref{chapter:motivation}
% finally gives some motivation of why one would want to allow users to extend the
% syntax of a language.  As with all features, taste is advised in doing so.

% Chapter ~\ref{chapter:repl-reader} discusses how the reader may be replaced and
% what the pros and cons of each approach are.

Chapter~\ref{chapter:reader-printer} discusses what a reader actually is and how
it differs from the more traditional parsers found in other programming
languages.  This chapter also discusses the inner workings of the algorithm used
to implement \elr{}.

Chapter~\ref{chapter:api-overview} gives an overview on how to use the
reader presented in this thesis, including some explanations often lacking in
tutorials found both on the web and in books.

Finally chapter~\ref{chapter:future-work} outlines some possible work in the
future to improve the situation.

\section{Lexical conventions}
\label{sec:lexical-conv}

Throughout this thesis, the names of functions, classes, variables and plain
symbols are written in a \texttt{monospace} font.  Code is mostly placed in
boxes and listings, but the occasional short expression may be put between
``double quotes''.  Single characters embedded into the text are surrounded by
single quotes.  Note that only the quote signs as used in source code of
virtually all programming languages (i.e. the characters ‚'’ and ‚"’) are to be
taken as part of such an expression or character.

When functions are presented, their signature is shown too.
Figure~\ref{sig:example} shows an example.

\begin{lstlisting}[style=lispcode,label={sig:example},caption={Example function
signature},numbers=left]
(el-reader/read (&optional INPUT-STREAM
                           (EOF-ERROR-P T)
                           EOF-VALUE
                           RECURSIVE-P
                           KEYS))
\end{lstlisting}

The parentheses denote a list.  The first element is the name of the function in
question.  All other elements are either arguments or qualifiers for further
arguments.  One such qualifier is ``\&optional''.  It means that all further
arguments are optional.  The following arguments may either be an argument name
or a list.  If a name is given, the value will be \nil{} if the caller does not
supply an argument.  In case of a list, the first element of said list must be a
name, while the second may be an arbitrary expression which will be evaluated as
the function is called.  The result of said expression will be bound to the
argument.\footnote{Note that this is a consequence of using the \texttt{CL}
  package distributed with \emacs{}, which provides some \cl{} emulation.  The
  list may also provide another value which may be used by the called function
  to determine whether an argument was supplied or not, yet this is not used in
  this thesis.  The appropriate info page discusses these features in more
  depth.}  Another argument is ``\&rest''.  This must always be followed by
exactly one argument name.  It means that all further arguments given---which
may be zero or more---are collected into a list and bound to this argument.

As \el{} does not provide any namespace support worthy of the name, it is sadly
necessary to prefix all names one uses to avoid clashes.  As the package
developed for this thesis has been named ``el-reader'', this is also used as a
prefix.  Many \el{} packages simply delimit prefixes with a dash for public
variables or functions and a double dash for private ones, yet in this package a
single and double forward slash are used for this purpose as it is more easily
distinguished from the other dashes used to separate names.

Special (a.k.a. dynamic, a.k.a. global) variables are---by
convention---surrounded by asterisks, yet only the part of the name which does
not include a prefix.  For example the readtable is placed in
\sym{el-reader/*readtable*}, not \sym{*el-reader/readtable*}.

Hence, every symbol which begins with \sym{el-reader/} and is not followed by a
second immediate slash, denotes a public variable or function.

To make the names used in this thesis shorter, the full name is used once, and
from then on the part after the slash(es) is used.

\section{A brief history of (GNU) Emacs}
\label{sec:emacs-intro}

Emacs was at first a text editor at MIT, written in machine language and
extensible with an interpreted command language.  The interpreter was taken from
TECO, another text editor used at MIT at the time.  According to Richard
Stallman \cite{emacs-hist} there was an implementation on a Lisp Machine and
later on Multics.  Both of these were written in Lisp, hence were extensible in
Lisp.  When Stallman started the GNU project, he needed an implementation for
\unix{}.  As Lisp programs on regular machines could not perform runtime type
checks at the same speed as Lisp Machines could one had to choose performance or
safety, but could not have both.  Stallman decided to write a Lisp interpreter
and editor primitives like display handling in C while writing as many higher
level functions as possible in Lisp.\footnote{Why Stallman felt that using C
  instead of an optimizing Lisp compiler is beyond me.  Both suffer from the
  lack of type checking, while using Lisp would have been far more productive
  than programming in C.  Possibly the lack of decent free Lisp compilers played
  a part in this decision.}  The dialect he designed and implemented is called
\el{}, or elisp for short\footnote{Note though, that there once was an unrelated
  dialect of the name elisp.}.  In 1984 Stallman wrote what is now known as
\emacs{}\cite[See ``History'']{emacs-first-release}. Since then, \el{} has grown
and changed a lot.

While \emacs{} is primarily a text editor, it achieves its extensibilty by
being/providing a virtual Lisp Machine.  So far only some experimental features
exist to incorporate shared objects into a running instance of \emacs{}, so the
only ways to add features is to modify the C code base, which requires
recompilation, or by loading \el{} code.  All C functions which users may need
are exported to Lisp and are largely indistinguishable from Lisp functions,
macros or special forms.  This extensibility has made \emacs{} one of the most
powerful editors available.  For the rest of this thesis, the editing
capabilities of \emacs{} are rather unimportant.  The thesis will concentrate
more on the Lisp Machine part of \emacs{}.

The capabilities of \el{} have drastically improved over the last years.  Until
\emacsv{24.1} \el{} only featured dynamic scope\cite[See “* Lisp changes in
Emacs 24.1”]{Emacs-Lexical}.\footnote{See \cite{CLTL2}
  (\url{http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node43.html})for an
  explanation of why ``dynamic scope'' is a misnomer, yet still useful as a
  term.  While the differences in scope and extent are interesting, they are not
  relevant to this thesis.}  In what will probably be called \emacsv{25.1},
support for watered down \cl{} style multimethods will be included.  As these
methods were used for the code in this thesis, said code requires at least a
pretest version of \emacs{}\cite{emacs-pretest}.

\section{A brief history of Lisp}
\label{sec:lisp-hist}

Pinpointing Lisps exact moment of birth is rather difficult, as it is often the
case with ideas which take some time to develop.  John McCarthy published a
paper in 1960 on the language\cite{rec-fun-sym-expr} in which some ideas,
notation among others, from Alonzo Church’s $\lambda$-calculus are used.

The gist of the paper is that McCarthy devised a theoretic system to represent
functions in programs.  This was originally meant as a theoretic foundation,
yet, as McCarthy writes in his retrospective notes on Lisp history
\cite{lisp-hist}: “S.R. Russell\footnote{One of his students at the time.}
noticed that eval could serve as an interpreter for LISP, promptly hand coded
it, and we now had a programming language with an interpreter.”

While it may have been an accident that Lisp got an implementation, thus
becoming an actual programming language at the time that it did, the foundation
was laid down at that time, so it could have happened at any later point.

As Lisp introduced several ideas which were new to programming, resulting in
high expressive power, it quickly became popular in some circles, especially the
artificial intelligence community, of which McCarthy was a part.  This was
probably simply because Lisp was more powerful than any other language of the
time, even though it was very slow.  It was slow compared to other languages of
the time, namely machine language/assembly and FORTRAN.  It was also slow simply
because compiler technology was not nearly as advanced as it is today.

What makes Lisp so expressive is discussed in greater detail in
section~\ref{sec:lisp-special}.

Lisp was originally supposed to adopt a different syntax, especially for
function calls and definitions.  McCarthy even used this syntax, called
M-expressions, in his original paper \cite{rec-fun-sym-expr}.  They were to be
translated into S-expressions (short for Symbolic expressions).  While in the
original paper the syntactic representation of S-expressions and lists was
slightly different\footnote{McCarthy originally allowed spaces in atoms,
  i.e. symbol names, and separated list elements with a comma.  Today spaces are
  used to delimit elements.}, they essentially had their current form, namely
consisting of symbols and lists, using a prefix scheme, i.e. having the operator
as the first element of a list.

McCarthy himself used the term S-expression for symbols while today most Lisp
programmers use the term S-expression, or sexp for short, to mean any string of
characters which may be \emph{read} by a Lisp implementation.  The latter
meaning will be used in this thesis from now on.

This proposed translation was not merely a syntactic, let alone cosmetic change.
Compilers and interpreters in general need some form of source representation
which is above the character level of the source code which they take as input.
This is typically called an abstract syntax tree (AST)\footnote{The exact
  differences and relationships between ASTs, concrete syntax trees and the like
  are beyond the scope of this thesis.}.  Lisp, being a dynamically typed
language, allows the syntax tree to be particularly simple.  While a tree in a
statically typed language may need structures to differentiate between a
function call (which has a name and zero or more arguments), and a number
literal, in Lisp one can simply use dynamic typing.  This way, numbers, strings,
symbols etc. can be put directly in place, while nested lists provide the tree
structure needed.

As Lisp programs can be represented with data structures which Lisp provides,
higher-order programs became possible quite early in the history of computers.
Lisps syntax and its relationship to metaprogramming is discussed in further
detail in section~\ref{sec:lisp-special}.

As Lisp is rather a set of ideas on how a language should be, rather than a
single language, different Lisp dialects were developed.  Among the most well
known are \cl{} and Scheme.  Another rather recent addition to the Lisp family
of languages is Clojure, which is rather important, as it is very different from
other Lisps, especially \cl{}.  Another Lisp dialect which is somewhat
widespread is \el{}, which is rather similar to \cl{} is some respects.  This
language is used---as the name suggests---as an extension language for \emacs{}
and some other Emacs variants.

\section{What makes Lisp special}
\label{sec:lisp-special}

As outlined at the end of section~\ref{sec:lisp-hist}, Lisp code can be
represented in its own data structures.  This sets it apart from most (if not
all) mainstream languages in use today. As this is vital to the rest of this
thesis, it is discussed in detail here.

As an example, we will transform a simple algebraic expression.  The simple
expression \(a + b \cdot c\) can be put into a tree form
(figure~\ref{fig:simple-tree}).  Note that in tree form parentheses are no
longer needed to group terms.  Also, all precedence becomes explicit.

\begin{figure}[h]
  \centering
  \begin{tikzpicture} %[every node/.style = {shape=circle, draw}]
    \node [op] {+} child { node [arg] {a} } child { node [op] {\times}
      child { node [arg] {b} } child { node [arg] {c} } };
  \end{tikzpicture}
  
  \caption{Tree form of an algebraic expression}
  \label{fig:simple-tree}
\end{figure}

Alternatively, given lists of arbitrary length (represented by surrounding
parenthesis), symbols and dynamic typing, the same tree may also be represented
like this:

% Given a list notation as used in Lisp, i.e. parentheses and spaces which delimit
% elements, together with prefix notation, it may as well look like this:

\begin{lstlisting}[style=lispinline]
(+ a (* b c))
\end{lstlisting}

Given that Lisp syntax is given as nested lists, symbols and prefix notation, it
becomes apparent that Lisp only has (linear) syntax for ASTs.  This means that
programmers do not enter syntactic forms, but directly enter the AST for their
programs.

This may put off some programmers as unaesthetic\footnote{Many non-lispers claim
  that Lisps syntax---or lack thereof---is difficult to read, yet I and many
  other Lisp programmers disagree.  Once a programmer has become accustomed to
  this kind of source code, it can become quite appealing, and the parenthesis
  are not as intrusive to the mind as many initially suspect.}.  While it is
difficult to meaningfully argue over such subjective topics, there are technical
advantages to this approach, which come in at least four categories, each of
which are discussed in more detail in the following sections: macros, sexp
communication, editing tools, customizable syntax.

\subsection{Macros}
\label{subsec:macros}

As programs can be represented by data structures which are accessible to the
user of the language, an implementation may provide hooks which allow the user
to add transformation routines.  Such a routine may take a source tree, which
has been read, but not evaluated, and return such a source tree, which is
evaluated instead of the invocation of the transformer itself.  In lisp, such
transformers are called macros.  A macro is a procedure, written in Lisp, which
operates on Lisp source code.  The result of calling a macro procedure is called
the expansion of said macro.  Listing~\ref{code:increment} is taken from the
``GNU Emacs Lisp Reference Manual''.\footnote{Note that this definition has some
  (fixable) problems, which make it unsuitable for production work.}

\pagebreak

\begin{lstlisting}[style=lispcode,caption={Increment as a
      macro.},label={code:increment}]
(defmacro inc (var)
  (list 'setq var (list '1+ var)))
\end{lstlisting}

This macro can be invoked via \texttt{(inc x)} where \sym{x} is an already
existing variable.  Inside the macro procedure \sym{val} is \emph{not} bound to
the value which \sym{x} will hold at runtime, but with the \emph{symbol}
\sym{x}.  This makes it possible to perform macro expansion at runtime.  The
procedure returns a list which is evaluated at runtime.  As the first element of
this list is the symbol \sym{setq}, it sets a variable to a value.  The code
which produces the expansion as well the expansion itself may be arbitrarily
complex.  There is a lot more to write about how to use macros in practice.  The
``GNU Emacs Lisp Reference Manual'' has a section devoted to macros in \el, and
Paul Grahams book ``On Lisp'' \cite{on-lisp} features an excellent chapter on
macros in \cl{}.

A more sophisticated macro can be found in the source code to this thesis (see
listing~\ref{code:make-accessors}, which generates getters and setters for a
class).  The arguments to the macro are a symbol which denotes the class for
which accessors shall be created, and an optional second symbol which denotes a
prefix for all accessors.

Note that the mechanism to create classes is not part of the core of \el{} but
is implemented on top of \el{}.  This is done with a package named
EIEIO\footnote{EIEIO stands for “Enhanced Implementation of Emacs Interpreted
  Objects” and is shipped with \emacs{}}.\cite{eieio}

\begin{lstlisting}[style=lispcode,caption={Create accessors for all data members
  of a given class.},label={code:make-accessors}]
(defmacro el-reader//make-public-accessors (class &optional prefix)
    (cons 'progn
          (seq-map
           (lambda (e)
             `(progn
                (defun ,(intern (s-concat (if prefix (symbol-name prefix)
                                            "")
                                          (symbol-name e)))
                    (obj)
                  ,(s-concat "Extract the element " (symbol-name e)
                             " from OBJ. ")
                  (slot-value obj ',e))
                (cl-defmethod (setf ,(intern (s-concat
                                              (if prefix
                                                  (symbol-name prefix)
                                                "")
                                              (symbol-name e))))
                    (new-elt (obj el-reader/readtable))
                  (setf (slot-value obj ',e) new-elt))))
           (seq-map #'eieio-slot-descriptor-name
                    (eieio-class-slots class)))))
\end{lstlisting}

For every data member \sym{memb}, prefix \sym{prefix/} in the class \sym{klass},
the code in listing~\ref{code:data-member} has to be generated.

\begin{lstlisting}[style=lispcode,caption={Getter and Setter for a member called
  \sym{memb}.},label={code:data-member}]
(defun prefix/memb (obj)
  "Extract the element invalid-chars from OBJ. "
  (slot-value obj (quote invalid-chars)))
(cl-defmethod (setf prefix/memb)
    (new-elt (obj klass))
  (setf (slot-value obj (quote memb)) new-elt))
\end{lstlisting}

This code inspects the slots (i.e. member variables) of the class given in the
first argument with the function \fun{eieio-class-slots} which returns a list of
slot descriptors.  The function \fun{eieio-slot-descriptor-name} in turn returns
the name of the given slot, which is done for each slot (with \fun{seq-map}).
For each slot name a getter (\fun{defun}) and setter (\fun{cl-defmethod}) is
created.  Note that the macro itself does not create these functions.  The macro
returns code which---when run---creates said functions.  The alternative to this
macro would have been to write getters and setters for every data member in a
class manually.\footnote{For this program, the macro was used only once.  The
  expansion is almost 80 lines long, if properly formatted.}  This is not only a
menial task not worthy of a self respecting programmer, but also error prone and
highly unmaintainable.  If for instance the desired prefix changes, every getter
and every setter has to be inspected and changed.

\subsection{Communication via sexps}
\label{subsec:sexp-communication}

When a given program needs to save or read data from a file, a stream or some
other medium, some form of structure is needed.  The simplest case is a binary
format, yet these schemes often suffer from lack of extensibility and
debugability.  If a text based scheme is called for, some sort of syntax is
needed.  In the \unix{} culture plaintext files with assignments using the
\texttt{=} is and was often used.  Programs on \windows{} often used INI files.
Both cultures seem to tend towards XML.  They do so with good reason: there are
parsers for XML which are readily available, and XML offers a tree structure.

XML and sexps are quite similar.  Both describe a tree shaped structure without
specifying which labels are permitted.  XML seems to focus on cases where there
is a lot of text and little markup (hence making it quite suitable for the web),
while sexps are well suited for cases with lots of markup and little text.  One
could say that sexps focus more on symbols, while XML focuses on strings.

While it is possible to describe both a document and a program with XML or with
sexps, it quickly becomes ugly to describe a document with sexps because text
must be escaped.  Likewise, describing programs in XML is very verbose and ugly.

Both XML and sexps serve the purpose of providing a linear syntax for data
objects.  A syntax which may not only be read by a program, but also which may
be written, as data objects may be printed in XML or sexps notation.

The obvious benefit of using sexps to share data between processes for a Lisp
programmer is rather obvious: the syntax is the same as the language itself and
both readers and printers are available by definition.  At least in \cl{} both
are customizable, which allows for new syntax for new data types.  Often new
syntax is not needed, as printing can produce an expression which can be
transformed at runtime.  A hash table might be printed like this:

\pagebreak

\begin{lstlisting}[style=lispinline]
(hash-table size 65 test eql rehash-size 1.5
            rehash-threshold 0.8
            data (:foo "FOO!" :bar "BAR!"))
\end{lstlisting}

In such an example the mere reading of an expression will not yield a hash
table.  Yet interpreting such a structure gives enough details on how to
construct the table in memory, similar as would be done using XML.

\subsection{Editing Tools}
\label{subsec:edit-tools}

While editing any kind of language one may encounter intermediate states which
are invalid.  While typing ``<a href=http://example.com>link</a>'' in an XML
document, the person typing this will break the structure of the file in several
ways:  ``<'' alone is not valid in XML.  ``<a'' too, is invalid.  The same
problem exists for sexps: ``(a ((href . "http://example.com")) "link")'' is an
sexps which has roughly the same structure as the previous XML example.  Here,
unbalanced parentheses cause the text to become invalid syntax, i.e. no tree can
be constructed from it.

For \emacs{} there are at least two tools which address this problem:
paredit\cite{paredit} and lispy\cite{lispy}.  Both tools provide alternative
editing commands which ensure a valid tree:  if an open paren is inserted, a
closing paren is also inserted.  A list may be expanded to include the next
element, or eject the last element from the list (thus always keeping the tree
intact).  These tools also offer commands which raise a whole expression (of
arbitrary size) up one level in a tree, deleting one level.  An example may be
used to illustrate:

\begin{lstlisting}[style=lispinline]
(if (some-predicate arg1 arg2 and some more really
                    complex code!)
    simple-value)
\end{lstlisting}

Given this code, a programmer might discover that the whole if expression is not
needed, and that \sym{simple-value} should always be used.  Given very few
commands, both paredit and lispy can \emph{raise} the token \sym{simple-value}
to produce a new tree:

\begin{lstlisting}[style=lispinline]
simple-value
\end{lstlisting}

Note that this is done (somewhat) atomically: at no point is there an opening or
closing paren which distorts the syntax tree.

While such tools may not be impossible for other languages, sexps (and XML
probably too, yet the I am not aware of such tools for XML) make this much
easier than in other languages.\footnote{To show how ``simple'' this can be, one
  can simply take a look at the length of the source code of such tools: the
  lispy version I use at the time of writing consists of 12 files for several
  languages, and less than 10,000 lines of \el{} code (including comments etc).
  While this is still quite a lot, it probably would be a lot more for languages
  with more ``traditional'' syntax, let alone for languages with undecidable
  grammars like C++.}

\pagebreak

\subsection{Customizable Syntax}
\label{subsec:custom-syntax}

As Lisp syntax is merely syntax for an AST, i.e. data, this syntax can be
customized to include new data types, or to redefine existing syntactic
constructs.  It is important to note that this is not true of Lisps in general,
only that it is possible for a Lisp to exhibit this kind of behavior.  \cl{}
features a customizable reader \el{} does not, which is why this thesis
addresses the issue for \emacs{}.

Not many users of \cl{} seem to use this set of features, as free and good
introductory material on the topic is quite lacking on the web, although
Graham’s On Lisp does feature a short chapter on read macros.\cite[Chapter
17---``Read Macros'']{on-lisp}

\section{Lisp Basics}
\label{section:lisp-basics}

This chapter is both a short recap of some Lisp basics, and on the stages code
goes through, which is more explicit in Lisp than in other languages.  At
first, the symbol and list type will be introduced, which play a very important
role in Lisp, and without neither reader nor compiler would make much sense.
Later on the stages through which code may flow will be discussed.  This
includes the range from reading to evaluation.

Unless stated otherwise the examples given refer to \el{}, not to every Lisp in
general.

\subsection{Datatypes}
\label{subsec:datatypes}

\subsubsection{Symbols}
\label{subsubsec:symbols}

The symbol is a type which represents a name, often together with places for
values and/or functions.  The latter depend on the Lisp dialect.  An important
property of symbols is that fast lookup is supported, and that symbols are
interned.  This means that each symbol which is read, which has the same name,
returns the same Lisp object.\footnote{This is, in effect, a singleton.}

In \el{}, which is the Lisp under consideration for this thesis, a symbol has
the following attributes:

\begin{description}
\item[{Name}] A name for the symbol.  This may be retrieved as a string at any
  time by calling \fun{symbol-name}.  Also, a string may be used to retrieve the
  symbol instance with the function \fun{intern}.
\item[{Value}] This is the value of the symbol.  It may be retrieved by a call
  to \fun{symbol-value} or by simply appearing in a program (macros or special
  forms may treat symbols differently).
\item[{Function}] This cell is looked up if the symbol is the first element of a
  list which is evaluated (more on this later).  Alternatively, a call to
  \fun{symbol-function} returns the function to which it points (if any).
\item[{Property List}] A symbol may also have a Property List, or plist for
  short, which is a mapping from a key (mostly a symbol) to a value.  This is
  intended as a mechanism to store arbitrary metadata in a symbol.
\end{description}

In \el{} a fifth attribute exists, which is not visible from Lisp code.  Symbols
are placed into so called ``obarrays'', which are similar to hashtables, but are
specialized to symbols (i.e. no other objects can be stored in them).  To
support fast lookup and low memory usage, each symbol contains a pointer to the
next symbol in the same bucket of the obarray.  For this reason a symbol may
only be interned in one obarray.\cite[Section 8.3 ``Creating and Interning
Symbols'']{elisp-reference} Because of this limitation it is not possible (at
the time) to implement \cl{} compatible packages for \emacs{} (not to be
confused with the packages provided in \fun{package.el}, which are installable
units).

\subsubsection{List}
\label{subsubsec:list}

In Lisp parlance, when a list is mentioned, it is almost always a singly linked
list.  While by far not the only compound data structure\footnote{Strictly
  speaking, lists by themselves do not exist: there are only cons-cells, yet
  this nuance is not very important for the current discussion.}, it is a very
important one.

For the moment, the most important aspect of lists is that they may be
heterogeneous, i.e. contain objects of different types, and that they may have
varying length.

\subsection{Stages}
\label{subsec:stages}

When a Lisp expression is to be evaluated, it goes through up to three stages,
all of which are discussed here at least briefly, including some implications.

Note that all of these steps can be invoked---independently of each other---by a
running Lisp program.  It is also possible for each stage to invoke the others.
Depending on the Lisp environment a user chooses to
employ, these steps (apart from evaluation) may be invoked in sequence to
produce an executable.  This is not exactly the case for \emacs{}; here, the
last step in a batch compile is bytecode.

\subsubsection{Reading}
\label{subsubsec:reading}

The first stage is reading.  Before this stage, the source code is present in
the form of a linear string of characters, for instance from a file.  The result
of this is a Lisp object.  This object may be very simple, but it may be a huge
object composed of many sub-objects.  The object returned mirrors the structure
of the source.  Variable and function names are symbols, parenthesis denote
lists, number literals are converted into numeric objects etc.

This stage already exists in stock \emacs{}, yet \elr{} replaces this stage in
order to enable this stage to a) recognize additional (user supplied) patterns
and b) to reach out into the other stages.  In a way the reader included in
stock \emacs{} will call functions which are also exposed to \el{}, such as
\fun{intern}, yet this is not in any way user controllable.  In stock \emacs{},
the reader is the only stage which can never reach out to other stages.

With \elr{} it is possible to run and/or compile code at readtime.  This may be
used to provide nonstandard semantics for an embedded domain specific language
(EDSL), or may be used to pull calculations from runtime to readtime in order to
improve performance.

Besides providing syntax for data types (like symbols and lists), readers may
also provide shorthand notation for common scenarios.  The first shorthand
everyone who learns to program in lisp encounters is quote (\sym{'}).  This
reads another expression and wraps it in a list where the first element is the
symbol \sym{quote}.  Hence ``'foo'' becomes \texttt{(quote foo)}.  This special
form is relevant when code is evaluated, as it prevents evaluation of its
argument.  If its argument is a symbol, the symbol is returned, if it is a list,
it is not treated as a function call or macro expansion or special form
invocation.

\paragraph{Example}
\label{par:read-example}

Assuming a file with the contents shown in figure~\ref{code:read-example},
reading the contents yields a list as its result: the first element is the
symbol \fun{defun}, followed by the symbol \sym{factorial}, followed by a list
containing the symbol \sym{n} as its sole element, and so on.

Note that when the code is evaluated, it defines a function (hence the symbol
\sym{defun}, a contraction of \textbf{def}ine \textbf{fun}ction, yet reading it
merely results in a list.  This means that Lisp does not have keywords in the
sense that more mainstream languages have.

\begin{lstlisting}[style=lispcode,label={code:read-example},caption={A function
which calculates the factorial of its argument.},numbers=left]
(defun factorial (n)
  (let ((result 1))
    (while (not (zerop n))
      (setf result (* result n))
      (decf n))
    result))
\end{lstlisting}

While it has nothing to do with reading, it is very important for the next
sections that the first element of a list (should it ever be evaluated) should
be either a special form, a macro or a function (i.e. something ``invokable'').
The exception to this rule is quoting which will be discussed in the section on
evaluation~(\ref{subsubsec:evaluation}).

\subsubsection{Macro expansion and compilation}
\label{subsubsec:compilation}

The next step is to optionally compile the code.  During this step (if it is
performed) macros are expanded.  In the end, every single step involved must be
a primitive operation, such as addition, calling and defining functions or
setting variables.  In the example given, three macros are used: \fun{defun},
\fun{setf} and \fun{decf}.  The invocation of \fun{(decf n)} expands to the form
\texttt{(setq n (1- n))}, where \fun{setq} is a primitive which performs
assignments\footnote{Either a single assignment, or many assignments may be
  performed with \fun{setq}.}.

While the transformation which \fun{decf} performs may be trivial, it is already
very useful, as it shows the intent of the code a little more clearly than
spelling it out.  Also, macros may become arbitrarily complex, effectively
becoming languages in their own right.  The multimethod facility discussed in
section~\ref{subsubsec:multimethods} is implemented entirely as macros.  Also,
\fun{cl-loop}, a port of the \cl{} looping construct is in effect a very large
macro.

This works by associating a function with a name.  When this name is the first
element of a list which is evaluated, the function is called, yet not in the
normal way.  The function is called with the rest of the list elements, which
have been read but not yet evaluated.  This means that a macro invocation which
has the form \texttt{(+ 1 2)} as its argument is given a list of the symbol
\sym{+}, the number one and the number two.  This means that the code has not
yet been evaluated (i.e. is not the number three)!  The result of this function
is used as a replacement for the macro invocation.  This is applied repeatedly
until no further unexpanded macro invocations remain.  This mechanism allows
users to write code which transforms other code (takes code as input and
produces code as output).

This is one of the reasons Lisp code is so much more compressible than that of
many other languages.\footnote{It may be that full support for lazy evaluation
  is similarly powerful.}

For a more in depth discussion of (\cl{}) macros and how to employ them, see On
Lisp by Paul Graham.\cite{on-lisp}

\subsubsection{Evaluation}
\label{subsubsec:evaluation}

If no compilation has been performed (i.e. the code is interpreted) macro
expansion as described in the previous section is performed as needed.

The exact evaluation semantics strongly depend on the Lisp dialect at hand.
Here are merely a few rules of thumb.  When a list is evaluated, the first
element denotes what operation to perform while the other elements of the list
are arguments to this operation.  The operation may be a macro expansion (see
the previous section), a function call or a special form.  The latter is an
operation which must be built into the language runtime.  In \el{} \fun{setq} is
such an operation.  Special forms---like macros---may choose whether and how
often to evaluate their arguments.\footnote{Macros themselves often do not
  evaluate their arguments at all, yet the code emitted by the macro may
  evaluate its arguments once, several times or not at all!}  The special form
\fun{setq} for instance does not evaluate its first argument, but checks whether
it is a symbol or not.  If it is, the next argument is evaluated and its value
is assigned to the value slot of the symbol at hand.

If the operation is a symbol which is not a macro, all arguments are evaluated
and passed to the function which is to be found in the function slot of the
symbol.

Evaluating a symbol is done by referencing the value slot of the symbol.

% \chapter{Lisp Basics}
% \label{chapter:lisp-basics}

% This section is only intended as a short recap of what Lisp syntax is, and how
% it relates to the reader.  For more details, a Lisp textbook is more
% appropriate.  A good reference for \el{} is found at \cite{elisp-reference}.

% Lisp syntax never directly describes control flow, function definitions, or
% other actions, but merely data.  If the described data consists of lists and
% symbols, it will later on be treated as code, which can be compiled and
% executed.

% Such data structures can represent code by having a list with an ``action'',
% which may be a special form, a macro or a function, as its first element.  Any
% further elements are arguments to said operator.

% As an example, here is some code which performs a variable binding with
% \fun{let} (which is a special form), an assignment with \fun{setf} (a macro)
% and a function call to \fun{message}.

% \begin{lstlisting}[style=lispinline]
% (let ((var 5)) (setf var "Hello") (message "%s" var))
% \end{lstlisting}

% When Lisp source code is processed, it goes through three distinct stages:
% reading, macro expansion (optionally this can also be a compile phase), and
% evaluation (which can be the execution of compiled code).

% The first stage is somewhat unique to Lisp---most languages only have this as an
% implementation detail in the compiler or interpreter.

% \section{Important Lisp Data Types}
% \label{subsec:important-types}

% A few Lisp data types are of particular importance, both because they are
% relatively rare in other languages, but also because they are very widespread in
% Lisp code, i.e. the reader often creates them.

% Every Lisp dialect the author is aware of provides at least the following:

% \subsection{Symbols}
% \label{subsubsec:symbols}

% The symbol is a type which represents a name, often together with places for
% values and/or functions.  The latter depend on the Lisp dialect.  An important
% property of symbols is that fast lookup is supported, and that symbols are
% interned.  This means that each symbol which is read, which has the same name,
% returns the same Lisp object.\footnote{This is, in effect, a singleton.}

% In \el{}, which is the Lisp under consideration for this thesis, a symbol has
% the following attributes:

% \begin{description}
% \item[{Name}] A (typically hashed) name for the symbol.  This may be retrieved
%   as a string at any time.
% \item[{Value}] This is the value of the symbol.  It may be retrieved by a call
%   to \fun{symbol-value} or by simply appearing in a program (macros or special
%   forms may treat symbols differently).
% \item[{Function}] This cell is looked up if the symbol is the first element of a
%   list which is evaluated (more on this later).  Alternatively, a call to
%   \fun{symbol-function} returns the function to which it points (if any).
% \item[{Property List}] A symbol may also have a Property List, or plist for
%   short, which is a mapping from a key (mostly a symbol) to a value.  This is
%   intended as a mechanism to store arbitrary metadata in a symbol.
% \end{description}

% A fifth attribute exists, which is not visible from Lisp code.  Symbols are
% placed into so called ``obarrays'', which are similar to hashtables, but are
% specialized to symbols (i.e. no other objects can be stored in them).  To
% support fast lookup and low memory usage, each symbol contains a pointer to the
% next symbol in the same bucket of the obarray.  For this reason a symbol may
% only be interned in one obarray.  Because of this limitation it is not possible
% (at the time) to implement \cl{} compatible packages for \emacs{} (not to be
% confused with the packages provided in \fun{package.el}, which are installable
% units).

% \subsubsection{Read Syntax}
% \label{subsubsec:symbol-syntax}

% The read syntax for a symbol is any token of text which is not also a valid
% number, does not contain any unescaped terminating macro characters and does not
% start with an unescaped macro character.

% Thus \sym{foo}, \sym{1+}, \sym{contains?} and \sym{set!} are valid symbols.
% \sym{+1} is not, as it is the positive number one.  \sym{\textbackslash{}+1} on
% the other hand is a valid symbol, as the \sym{+} sign is escaped.  Its name is
% ``+1'' (i.e. the escape character is only part of the read syntax, not of the
% outcome). 

% \subsection{List}
% \label{subsubsec:list}

% In Lisp parlance, when a list is mentioned, it is almost always a singly linked
% list.  While by far not the only compound data structure\footnote{Strictly
%   speaking, lists by themselves to not exist: there are only cons-cells, yet
%   this nuance is not very important for the current discussion.}, it is a very
% important one.

% Not only is there syntax for lists; together with the symbol, the list is used
% as the prime representation for code.  Lists are made up of cons cells, which
% are ordered pairs of Lisp objects.  They can be constructed at runtime with a
% call to \fun{cons}, passing the two objects as arguments.  Two additional
% operations fetch the contents and serve as setf-able places: \fun{car} and
% \fun{cdr}.  The former fetches the first element of the cons, the latter the
% second.  Cons cells can be linked.  By convention, the cdr of a cons points to
% the next cons, or \sym{nil}, if at the end of a list.  This type of structure is
% called a proper list.  A chain of conses which do not end in a cons with a cdr
% of \sym{nil} is called an improper list.

% \subsubsection{Read Syntax}
% \label{subsubsec:list-syntax}

% Cons cells, proper and improper lists have read syntax to create them at
% read-time:

% \begin{description}
% \item[cons] A single cons may be notated as a ``dotted-pair'': \texttt{(car
%     . cdr)}

%   Here a new cons is created which has a car of \sym{car} and a cdr of
%   \sym{cdr}.
% \item[proper list] A proper list is notated with whitespace separated elements
%   surrounded by parentheses: \texttt{(foo bar)}

%   This is equivalent to \texttt{(foo . (bar . nil))}
% \item[improper list] An improper list looks like a proper list, yet uses
%   embedded dotted pair syntax at its end: \texttt{(foo bar . spam)} is
%   equivalent to \texttt{(foo . (bar . spam))}.
% \end{description}

% \subsection{Other Types}
% \label{subsec:other-types}

% Most other types are rather unspectacular and depend on the dialect under
% consideration.  Practically all Lisps provide strings, \el{} even provides an
% alternate notation which provides text properties: \texttt{\#("foo" 0 1 (escapedp
%   t))}.  This represents a string ``foo'', with the property \sym{'escaped} and
% the corresponding value \sym{t} on the first letter.

% Numbers are also often a little different than in mainstream languages.  Both
% \cl{} and scheme provide exact fractions.  \cl{} and \el{} provide read syntax
% for integers in several bases: \texttt{\#b1111} \texttt{\#o17} \texttt{\#xF} and
% \texttt{15} all represent the number 15 in base 10.  This notation is the same
% in \cl{} and \el{}.  This notation has a certain air of elegance to it as soon
% as one realizes that \texttt{\#} is a dispatching macro character (see section
% ~\ref{subsec:dispatching-macro-chars} for details).  This means that
% \texttt{\#b}, \texttt{\#o} and \texttt{\#x} may be implemented as read macros.

\section{Motivation}
\label{section:motivation}
Users of other Lisp dialects, especially of \cl{}, have enjoyed the possibility
to modify the syntax they use to write programs for a long time.  This
possibility provides a huge benefit in the course of evolving a language, as
well as implementing readers for custom languages.  Macros---which are expanded
at compile-time---provide a very powerful mechanism for code transformation
(i.e. ASTs to ASTs).  These allow to control the order of evaluation of forms
within the macro, as well as the opportunity to transform them, yet sometimes
even this kind of power is not quite enough, as the macro call must still
consist of valid s-expressions.

A very simple case is the addition of syntax for data structures, since this is
what the reader is there for anyway: transforming characters into data
structures.  Most modern programming languages provide some syntax to enter hash
tables.  Take for instance python, where the keys and values may be specified
between curly braces:

% \begin{minted}[]{python}
% {"foo": "bar", "five": 5}
% \end{minted}

\begin{lstlisting}[style=pythoncode]
{"foo": "bar", "five": 5}
\end{lstlisting}

When entered into a python interpreter, this expression returns a hash table (or
Dictionary in Python terminology).

Clojure, a rather recent addition to the Lisp family of languages also features
a similar syntax:

% \begin{minted}[]{clojure}
% {"foo" "bar" "five" 5}
% \end{minted}

\begin{lstlisting}[style=lispinline]
{"foo" "bar" "five" 5}
\end{lstlisting}

This also returns a mapping from keys to values.

Unfortunately, neither \cl{} nor \el{} feature such (concise)
syntax\footnote{\el{} has read (and even print) syntax for hash tables, yet is
  is rather ugly.  The same table looks like this: \#s(hash-table size 65 test
  eql rehash-size 1.5 rehash-threshold 0.8 data ("foo" "bar" "five" 5))}.  \cl{}
users may create such a syntax themselves, yet \el{} users never had this
possibility.

In the absence of read macros, one possible shortcut is to simply define a
function to do so; consider the function \fun{ht} in listing~\ref{code:cl-ht}
from the source code of this thesis, ported to \cl{}.

% \begin{listing}[h]
% \caption{Creating a hash table.}
% \label{mint:cl-ht}
% \begin{clcode}
% (defun ht (&rest args)
%   "Create and return a hashtable.

% Keys and values are given alternating in args."
%   (let ((h (make-hash-table)))
%     (loop for (key value) on args by #'cddr
%        do (if (and key value)
%               (setf (gethash key h) value)
%               (error "Odd number of arguments passed")))
%     h))
% \end{clcode}
% \end{listing}

\begin{lstlisting}[style=lispcode,caption={Creating a hash
  table.},label={code:cl-ht}]
(defun ht (&rest args)
  "Create and return a hashtable.

Keys and values are given alternating in args."
  (let ((h (make-hash-table)))
    (loop for (key value) on args by #'cddr
       do (if (and key value)
              (setf (gethash key h) value)
              (error "Odd number of arguments passed")))
    h))
\end{lstlisting}
This may be used as

% \begin{clcode}
% (ht "foo" "bar" "five" 5)
% \end{clcode}

\begin{lstlisting}[style=lispinline]
(ht "foo" "bar" "five" 5)
\end{lstlisting}


instead of the more cumbersome and repetitive code shown in
listing~\ref{code:cl-canon-ht}.

% \begin{listing}[h]
% \begin{clcode}
% (let ((h (make-hash-table)))
%   (prog1 h
%     (setf (gethash "foo" h) "bar")
%     (setf (gethash "five" h) 5)))
% \end{clcode}
% \caption{Canonical way to create a hash table.}
% \label{mint:cl-canon-ht}
% \end{listing}

\begin{lstlisting}[style=lispcode,caption={Canonical way to create a hash
  table.},label={code:cl-canon-ht}]
(let ((h (make-hash-table)))
  (prog1 h
    (setf (gethash "foo" h) "bar")
    (setf (gethash "five" h) 5)))
\end{lstlisting}

To further shorten and clarify the creation of hash tables, \cl{} users may use
a read macro. Listing~\ref{code:cl-hash-reader} shows how this may be done
(assuming \fun{ht} has been defined as in listing~\ref{code:cl-ht}).
\footnote{The details of this code are not of particular importance, merely that
  it is possible, and quite short.}

\pagebreak

\begin{lstlisting}[style=lispcode,caption={A \cl{} read macro to read hash
  tables}.,label={code:cl-hash-reader}]
(set-macro-character #\} (get-macro-character #\)))

(set-macro-character
 #\{
 (lambda (stream char)
   (declare (ignore char))
   (let ((k-v (read-delimited-list #\} stream t)))
     (if (oddp (length k-v))
         (error "Invalid syntax: {}")
         `(ht ,@k-v)))))
\end{lstlisting}

Now the \cl{} reader reads ``\{\ldots{}\}'' constructs in a similar fashion as
the Clojure reader does.

Given \elr{}, i.e. the code from this thesis, the code in
listing~\ref{code:el-hash-reader} may be used in \el{} to produce the same
effect.

\begin{lstlisting}[style=lispcode,caption={An equivalent read macro using
  \elr{}.}, label={code:el-hash-reader}]
(defun ht (&rest args)
  "Create and return a hashtable.

Keys and values are given alternating in args."
  (let ((h (make-hash-table)))
    (cl-loop for (key value) on args by #'cddr
             do (if (and key value) (puthash key value h)
                  (error "Odd number of arguments passed")))
    h))

(el-reader/set-macro-character
 ?\} (car (el-reader/get-macro-character ?\))))

(el-reader/set-macro-character
 ?\{
 (lambda (stream _char)
   (cl-values
    (let ((k-v (el-reader/read-delimited-list ?\} stream t)))
      (if (= (mod (length k-v) 2) 1)
          (error "Invalid syntax: {}")
        `(el-reader//ht ,@k-v))))))
\end{lstlisting}

Note the similarity of the API.  The prefixes have been put in place, as \el{}
provides no packaging facilities.\footnote{Should Emacs change the internal
  representation of symbols, packaging may be possible, given this reader, yet
  is beyond the scope of this thesis.}

A more sophisticated use case might be to provide read syntax for regular
expressions---regexps for short.  In \el{} regexps are notoriously ugly to enter
and read, because they are opaque strings, which are passed to a function, which
parses and compiles them at runtime.  This means that many constructs must be
escaped multiple times, which makes the strings difficult to read and write.  A
particular notorious offender is the regular expression which matches a single
backslash; in Clojure one may write \texttt{\#"\textbackslash\textbackslash"},
\el{} users are forced to write the much more cumbersome
\texttt{"\textbackslash\textbackslash\textbackslash\textbackslash"}.  Here again
Clojure shows a nice way to handle the issue: by providing syntax for them.
Clojure also has the opportunity to compile the regexp at read time, while this
cannot be done in \el{}.

While Clojure provides more syntax than \el{} does, and more than \cl{} does out
of the box, it provides no way for users to add their own constructs.

Finally, it is also possible to define non-standard evaluation semantics, as the
code for read macros is run at read time.  This means that calculations may be
performed at read time, which---for most code---is effectively compile-time.

An example for this is something similar to rubys \texttt{\%w} construct.  It
reads a list of tokens, converts them to strings and makes an array out of
them.  

\begin{lstlisting}[style=rubyinline]
%w(foo bar spam and eggs) 
=> ["foo", "bar", "spam", "and", "eggs"]
\end{lstlisting}

If one is willing to ignore that ruby is very flexible about the delimiters, one
might define something similar to \texttt{\#w(...)} in Lisp as in
listing~\ref{code:elr-ruby-words}.

\begin{lstlisting}[style=lispcode,caption={A read macro for rubys word
  syntax.}, label={code:elr-ruby-words}]
(defvar *my-read-stuff/word-seq-types*
  {'cons #'list 
   'vector #'vector})

(defun my-read-stuff/read-words (stream char _num)
   (let ((words (el-reader/read stream t nil t)))
     (if (not (seqp words))
         (error "Invalid syntax: #w")
       (cl-values
        `(quote
          ,(apply
            (gethash (type-of words)
                     *my-read-stuff/word-seq-types*)
            (seq-map (lambda (s)
                       (if (not (symbolp s))
                           (error "Invalid syntax: #w")
                         (symbol-name s)))
                     words)))))))

(el-reader/set-dispatch-macro-character
 ?# ?w
 #'my-read-stuff/read-words)
\end{lstlisting}

While this version uses different delimiters for different types of return value
(parentheses form a list, square brackets form a vector), it can only use
certain delimiters, namely those which have read syntax for a sequential data
structure.  On the other hand it is extensible in the types it provides, and
most importantly forms the structure at read-time.  This may be of some worth
for large constants.

Here is how it is used:

\begin{lstlisting}[style=rubyinline]
#w(foo bar spam eggs) => ("foo" "bar" "spam" "eggs")
#w[foo bar spam eggs] => ["foo" "bar" "spam" "eggs"]
\end{lstlisting}

% \chapter{Replacing Emacs’ built-in reader}
% \label{chapter:repl-reader}

% When contemplating the replacement of such a critical and internal part of the
% implementation of a programming language, one must find a way to do so.  The
% obvious way is to replace the C code within \emacs{} itself.  The obvious
% advantage to this approach is speed and good integration.  The drawback is that
% programming in C is rather slow going, and that C and \el{} have very different
% data and execution models.  Another problem is that this makes distribution an
% all or nothing proposal.  The changes needed for this approach really are
% modifications of existing code, not mere additions.  Applying these changes
% would require the user to patch and recompile Emacs.  On the other hand, if
% these changes got included into \emacs{} by its maintainers, distribution would
% be automatic.

% Another possibility is to implement the reader in \el.  This not only has the
% advantage of easing development, by virtue of \el{} being a more expressive
% language than C, and not having to switch between different data and execution
% models.  It also makes it possible to distribute the resulting reader in an
% optional package, so every user may independently choose whether or not to use
% it.

% A compromise would be to write a module in C (or any other language which
% compiles to C callable native code).  This would make distribution easier than a
% patch, but more difficult than distributing an \el{} package, as it requires the
% user to compile code. This is already difficult on \unix{} based systems, as not
% every user has \emacs{}’ header files installed, and not all users have the
% appropriate compiler installed.  Also, at the time of writing, \emacsv{25.x} has
% not yet officially released, so not only would users have to build \emacs{} from
% source themselves, they’d have to manually add (experimental) module support.
% On MS Windows systems, this would be even more problematic, as hardly any user
% has any compiler in the first place, let alone an environment in which \emacs{}
% can be built.  The experimental status may also mean changes in the API to which
% modules must adhere.

% The second approach has one problem: the original reader must be replaced at
% runtime.  Fortunately, this is possible due to the dynamic nature of \el.  A
% function may not only be replaced wholesale, as is possible in many languages
% (such as python), but can also be ``advised''.  This is a feature which has been
% present in many Lisp dialects in the past, and has recently seen some popularity
% under the name of ``Aspect oriented programming''

% \emacs{} offers many different ways to add code to a function, all documented in
% the manual \cite[Section 12.11 ``Advising Emacs Lisp
% Functions'']{elisp-reference}.  In order to replace the reader (which consists
% of only one function: \fun{read} and a few variables) it is sufficient to place
% \sym{:around} advice onto the function in question.  This has the effect that
% all calls to \fun{read} will be redirected to a different, user supplied
% function with all arguments, plus the original function as an extra argument.

% This makes it possible to check whether \elr{} shall be used or not, and
% redirect to the original function instead.  Also, it makes it possible for
% \elr{} to reuse the original reader, where appropriate\footnote{Note that this
%   was done only for performance reasons.  Also, nested structures can not in
%   general be reused, as the original reader in \emacs{} is not aware of macro
%   chars and the like.  If the code for reading lists had been reused, ``(foo
%   bar)'' would still read correctly, yet ``(foo \{:bar :spam\})'' would fail, as
%   the built-in reader does not know what to do about curly braces.}.

% Given this possibility, replacing the built-in reader is a matter of a few lines
% of \el{} code, as can be seen in listing ~\ref{code:replace-reader}.

% \begin{lstlisting}[style=lispcode,caption={Replacing the built-in
%     reader},label={code:replace-reader}]
% (define-advice read (:around (oldfun &optional stream)
%                              el-reader//replace-read)
%   (if (and use-el-reader (not el-reader-bytecode))
%       (el-reader/read stream)
%     (funcall oldfun stream)))
% \end{lstlisting}

% %\section{Compatibility}
% %\label{subsec:compat}

% When replacing such a critical part of a language, it is important to maintain
% compatibility.  For this, several criteria must be met:

% \begin{enumerate}
% \item Previously accepted code must be accepted by the new system.
% \item Previously accepted code must have the same meaning in the new system.
% \end{enumerate}

% Both criteria may seem obvious, yet they apply both to syntax and to a function
% call, namely to \Read{}.

% To achieve this, it is necessary to take a look at the signatures of \Read{}
% both in \el{} and in \cl{}, as \elr{} takes a lot of inspiration from \cl{}.  In
% \el{} read takes 0 or 1 arguments, where stream is either:
% \begin{description}
% \item[nil:] the value of \sym{standard-input} (defaults to \sym{t}) is used.
% \item[a buffer:] begin at point and advance it.
% \item[a marker:] read from where it points and advance it.
% \item[a function:] called without arguments to retrieve a character, called with
%   one argument to unread a char.
% \item[a string:] starts at the beginning, does not change its argument
% \item[\sym{t}:] read a line of text either from the minibuffer (very Emacs
%   specific), or from standard input, if used in batch mode
% \end{description}

% \begin{lstlisting}[style=lispinline]
% (read &optional STREAM)
% \end{lstlisting}

% In \cl{} \Read{} has the following signature:

% \begin{lstlisting}[style=lispinline]
% (read &optional (stream *standard-input*) (eof-error-p t)
%       (eof-value nil) (recursive-p nil))
% \end{lstlisting}

% As in \el{} stream may be called without arguments, yet if exactly one is given,
% it refers to a stream.  All extra arguments are also optional, and their default
% behavior is---as far as it applies to \emacs{}, identical.

% The signature chosen for \elr{} is a little more complex, yet is fully
% compatible with \emacs{}’ \Read{} and reasonably compatible with \cl{}:

% \begin{lstlisting}[style=lispinline]
% (el-reader/read &optional input-stream (eof-error-p t)
%                 eof-value recursive-p keys)
% \end{lstlisting}

% The last argument needs some explaining.  Unfortunately, \el{} lacks several
% features which are present in \cl{}, hence the authors of \cl{}’s \Read{} took
% for granted.  Among these features is the possibility to have a function return
% a number of values different than one (i.e. more than one, or even none at all)
% without forcing the caller to know or care about the fact.  In \cl{} the first
% value returned by a function is called the primary value.  This value can be
% used as if the called function merely returned one value.  Should any other
% values be of interest, they must be explicitly bound by
% \sym{multiple-value-bind} or similar operators.  The multiple values facility of
% \cl{} also allows for no values to be present.  In such a case, the primary
% value (if bound) is \sym{nil}, yet the caller \emph{may} check for the number of
% returned arguments.  As \el{} does not support this mechanism, \elr{} passes an
% additional argument named \sym{keys} to switch between returning exactly one
% value, and returning a (possibly empty) list of values.  This is done, as it is
% possible for a read macro to return no value.  This is done for comments.  For
% this reason, read macros must always wrap their return value in \fun{list}, or
% \fun{cl-values} (which---at the time of writing---is an alias for \fun{list}).

% Note that the keys argument never has to be given by the user.  It is given only
% in one two places, which is in \fun{el-reader/read} itself, i.e. in a
% self-recursive call, and in the function \fun{el-reader/read-delimited-list}.
% Nevertheless, the user must always return a list of values in read macros, which
% is a shame, as it departs from the way it is done in \cl{}.

% There are other features which are present in \cl{} but not in \el{}, yet none
% of them are so critical (and difficult to retrofit) as multiple values.  Some
% other features can be emulated at various degrees of awkwardness, such as having
% non-nil default values for optional arguments.  These can be used with
% \fun{cl-defun}, and have been used in the code for \elr{}.

% Other features have only started to appear in \emacsv{25}, such as multiple
% dispatch, which is the main reason \elr{} doesn’t work on \emacsv{24.5}.

% While \emacs{} has a pre-defined set of types which may form a valid stream
% (with the function type being the most general), \elr{} provides a
% \fun{defclass} and \fun{cl-defgeneric} based extension mechanism.  This is
% discussed in detail in section ~\ref{subsec:streams}.

% \begin{enumerate}
% \item Possibility of higher order programs, i.e. macros.
% \item Possibility of communicating via sexps, similar to XML.
% \item Possibility of very powerful editing tools.
% \item Possibility of a customizable syntax.
% \end{enumerate}

% The first category enables programs which transform and write programs.  This
% can even happen at runtime, if needed, and---depending on the particular Lisp
% implementation---may even be compiled at runtime to provide high speed.

% The second category simply means that data may be printed and read back in the
% languages own syntax, without having the need to devise an own syntax along with
% a parser.  This also frees the programmer from having to learn another language.

% The third category is---for the most part---beyond the scope of this thesis,
% although the customizability of the languages syntax does pose a problem for
% such tools.  The idea behind these tools is that they can view the source code
% in a similar fashion as the reader does: as a tree of expressions.  Such tools
% attempt to disallow and thus prevent the user from making any changes to the
% source which would result in an incorrect tree.  This does not mean that the
% source will always be free of errors: function calls may still have too many or
% too few arguments, functions or variables may be undefined.  Yet it is not
% possible to have too many opening or closing parens.

% These tools offer tremendous editing power, as they allow the programmer to
% perform operations on a tree.  Two examples of such tools I am aware of are
% paredit\cite{paredit} and lispy\cite{lispy}.  Both these tools are for Emacs and
% both are primarily directed at \el{}.  Yet because of the simplicity and
% (syntactic) similarity of Lisps, both tools can also be used for other Lisps
% like \cl{}, albeit with some restrictions and annoyances.\todo{add a reference
%   to the appropriate section, should I opt to add one.}

% The fourth category is the subject of this thesis.  Before discussing how an
% extensible reader works, it is necessary to first understand what a reader does,
% whether it is extensible or not.  This is discussed in section
% ~\ref{chapter:reader-printer}.

\chapter{Reader and Printer}
\label{chapter:reader-printer}

\section{Readers vs. Parsers}
\label{subsec:readers-vs-parsers}

Most texts discussing compilers, interpreters and other language processors
often use the terms \emph{lexer} and \emph{parser} to describe how text is at
first processed.  In the Lisp world, there is another term, often used instead:
the reader.

Lisps split the compiler or interpreter apart in a different way than most other
languages by allowing the user to read/parse an expression at runtime and to
provide hooks to the compiler in the form of macros.  Even Lisps which allow for
read macros like \cl{} lump the lexer and parser together and name it the reader
and make it a first-class service to the user.  This is because the Lisp reader
and its macros operate on single characters, and return anything from a single
token (a symbol) to a whole tree.  The details of this are discussed in
section~\ref{sec:read-algo}.  The book ``Practical Common Lisp'' also has a good
take on this issue.\cite[Chapter 4, Section ``Breaking Open the Black
Box'']{pcl}

\section{Overview of the reader (in \emacs{})}
\label{subsec:reader}

The Lisp reader is responsible for transforming source code into Lisp objects.
In \el{} the rules for doing so are fixed.  The reader is implemented in C,
mostly by large switch statements, which look at the current character.
Normally parsers---and therefore readers---can perform their work by reading one
char, and being able to put one character back.  The built-in reader for \el{}
normally does this too, yet deviates from this procedure at least
once\footnote{This is done at least in the implementation of a rather obscure
  macro character: \#@.  This char is used to skip over bytes in the stream.
  The byte compiler places such characters into its output in order to skip over
  docstrings.  The strange behavior is probably for historic reasons, as wrong
  input is purposefully accepted in certain situations as \emacs{} used to
  produce wrong output due to a bug.}.

%\subsection{A sketch of rules for \emacs’ reader}
%\label{subsubsec:rules-sketch}

The reader API only provides for the possibility to read \emph{one} expression
from a stream, at the current position: the function \Read{}.  This
expression can be very complex, because of composite structures like lists.
When the reader encounters a special char (a macro character) as its first char,
it follows the rules for such a char.  Should such a char be encountered while
reading a symbol or number (both are called ``atoms''), it stops in order to
return the atom read so far\footnote{An example of such behavior is when
  reading the expression ``(foo bar)''.  When the reader attempts to read the
  second symbol it encounters a closing paren after the letter `r'.  At this
  point, the reader puts the read closing paren back onto the stream.}.  Only
when the reader is called again, it sees the macro char again.

When encountering a macro char as the first char, the according procedure is
called.  Obviously the opening parenthesis must be such a char: it triggers the
creation of a list.  It recursively calls \Read{} until it sees a closing
parenthesis.  This means that recursive calls to \Read{} may also see an open
paren as their respective first char, thus allowing nested calls to macro char
functions.

Note that \emacs{}’ reader does not actually have a strict notion of macro
characters and the like.  This is because \emacs{}’ reader does not have an
extensible reader; the reader itself knows all rules and how to handle macro
characters.

Most syntactic constructs simply start with a triggering char and have somewhat
straightforward rules regarding the construction of an object from characters.
The aforementioned list is one example, the vector, written as ``[a b c]'' is
another: here the tokens (\sym{a}, \sym{b} and \sym{c}) are not
evaluated\footnote{This means that reading ``[(list 1 2)]'' does not result in a
  vector with a single element which in turn is a list containing the numbers 1
  and 2, but a vector with the list consisting of the symbol \sym{list} and the
  numbers 1 and 2 as its sole element.}.  Quoting is also very simple: the
single-quote character triggers a macro, which in turn reads exactly one
expression and returns a two element list: the symbol \sym{quote} as the first
element, and the read expression as the second element.

Other forms are more intricate and require non-trivial solutions, most notably
dotted pair syntax.  In Lisp, a cons cell (which is an ordered pair) may be
constructed with the function \fun{cons}, with list syntax or with dotted pair
notation.  The function \fun{cons} takes two arguments and returns a cell
consisting of these two objects.  List notation with parentheses constructs a
series of cells which point to their successor in the second field.  This means
that the second element must be another cons cell, or \nil{}, which designated the
empty list.  To construct a cons which does \emph{not} point to another cons or
\nil{} at read time, one may employ dotted pair syntax.  This constuct is like a
list, yet between the last and second to last elements a sole dot is inserted,
like so:

\begin{lstlisting}[style=lispinline]
(foo bar spam . eggs)
(spam . eggs)
\end{lstlisting}

The first example constructs a so called \emph{non-proper} list; such a list
does not end with a pointer to \nil{}.  It thus produces 3 cons cells.  Without
the dot, 4 cells would have been produced, in order to have a reference to
\nil{} at the end.  The second example produces a single cons cell, equivalent
to calling \texttt{(cons 'spam 'eggs)} at read time.

Note that this read construct is a deviation from a previously introduced one:
the list.  Also, symbols consisting only of a dot, as used here, are otherwise
not permitted, neither in \el{} nor \cl{}.  Both \emacs{} and the code used in
this thesis use a similar strategy to resolve this issue, as is discussed in
section~\ref{subsec:irregular-dotted-pair}.

% The reader is a part of a Lisp system.  As outlined before, a string of
% characters (source code) is transformed into an AST in any language.  In Lisp,
% the syntactic layer is very thin, which makes the structure of the resulting AST
% apparent in the source code, as no special constructs are needed for loops,
% function definitions, types or the like.

\section{Overview of print syntax}
\label{subsec:print-syntax}

The printer is a part of a Lisp system, which works in tandem with the reader.
While a reader converts a textual representation into an object, the printer
converts an object into a textual representation.  It can be seen as the inverse
to the reader, yet while being a nice metaphor, it does not hold for all inputs.

As Lisp systems tend to be highly interactive, most lisps provide a means to
print values to a stream.  What Lisp does is slightly different to what many
other languages offer, as Lisp can often print back values in a way which are
valid input syntax.  How a data type is printed depends on the Lisp dialect.
Also not every dialect can print every data type.  Common Lisp by default cannot
print hash tables, and hardly any lisp has a meaningful printed representation
for functions---\el{} being an exception here.\footnote{\el{} can print
  interpreted functions as lists of symbols, and byte-compiled functions as
  bytevectors.  These may not be readable by a human, and barely compatible with
  other \emacs{} versions, yet is still more than most other dialects and
  implementations have to offer.}

Lists are among the types which have not only meaningful print syntax, but also
one which is compatible with its read syntax.  This means that a list, as shown
above, looks the same when printed, as it did in the source code---with minor
exceptions, which have to do with evaluation.  Also, formatting and comments are
lost.

As a fully fledged and extensible printer is a topic in its own right, further
details are beyond the scope of this thesis.  An extensible printer for \emacs{}
would not only bring \el{} a little closer to \cl{}, it would also make \elr{} a
little more valuable, especially in aiding communication between programs.

% \begin{lstlisting}[style=lispinline]
%   (+ a (* b c))
% \end{lstlisting}

% results in a Lisp object.  When this object is printed back, it looks (almost)
% exactly the same\footnote{A few things, such as comments and indentation are
%   lost, but otherwise it looks the same.}.  In Emacs, this can easily be tried
% out: In the *scratch* buffer, put the following expression: \textbf{(read "(+ a
%   (* b c))")}, place the cursor on the end of the expression and press C-c C-e.
% The very same text should appear on the status line.  The same can be done in
% almost any lisp environment in some form.  

% This allows users to print data to a file or network and read them back in
% again.  This is what Paul Graham meant when he claimed that programs
% communicating via sexps was an idea recently reinvented by XML.

\begin{comment}
  The reason why lisp code has so many parentheses, is that parentheses denote a
  list in Lisp.  Expressions in general, save the most trivial ones, are
  represented of lists containing other lists and symbols.\footnote{Plus of
    course other entities such as numbers, yet these are not relevant to this
    discussion.}
\end{comment}
% \chapter{Introduction}
% \label{sec:introduction}

% This thesis discusses the design, implementation, and idiosyncrasies of \elr{},
% an extensible reader for \el{}.  An overview over the history of Lisp, Emacs and
% \el{} are given, as well as an introduction to 

% \chapter{Introduction}
% \label{sec:introduction}
% In this thesis, a \cl{} like extensible reader for \el{} is discussed.  Before
% discussing the reader itself, it is important to discuss what a reader does,
% what it can do, and how the built-in reader in \emacs{} differs from the
% extensible one discussed in this thesis, or the \cl{} reader, from which this
% reader draws its inspiration.

% For this purpose, Lisps syntax, and how it is different from almost all other
% languages, is discussed.

% It is also shown what can be done with such an extensible reader, and why one
% might want to use such extensions.

% The application of such an extensible reader allows the creation of new
% syntactic constructs, which may be embedded into the language.  Thus, it enables
% the user to not only create a DSL, but also to embed it into a greater program,
% without making the new constructs opaque by putting them into strings, which
% must be parsed at runtime.


% In this thesis, the author wishes to first convey how Lisps syntax is different
% from other languages and why this is relevant to the discussion of extending the
% syntax.  Secondly what is meant by an extensible syntax or read macro, and why
% they might be relevant.  Also, depending on the background of the reader of this
% text, one might confuse such syntactic extensions with Domain Specific
% Languages—or DSLs for short.

% While there is some overlap between the two, DSLs are not necessarily embedded,
% but instead may come with a separate compiler and/or interpreter.  An extensible
% syntax, as it is presented here, allows to change the syntax on-the-fly,
% possibly without altering any present syntactic constructs, thus allowing new
% constructs in an existing programming language.

\section{Replacing Emacs’ built-in reader}
\label{sec:repl-reader}

When contemplating the replacement of such a critical and internal part of the
implementation of a programming language, one must find a way to do so.  The
obvious way is to replace the C code within \emacs{} itself.  The obvious
advantage to this approach is speed and good integration.  The drawback is that
programming in C is rather slow going, and that C and \el{} have very different
data and execution models.  Another problem is that this makes distribution an
all or nothing proposal.  The changes needed for this approach really are
modifications of existing code, not mere additions.  Applying these changes
would require the user to patch and recompile Emacs.  On the other hand, if
these changes got included into \emacs{} by its maintainers, distribution would
be automatic.

Another possibility is to implement the reader in \el.  This not only has the
advantage of easing development, by virtue of \el{} being a more expressive
language than C, and not having to switch between different data and execution
models.  It also makes it possible to distribute the resulting reader in an
optional package, so every user may independently choose whether or not to use
it.

A compromise would be to write a module in C (or any other language which
compiles to C callable native code).  This would make distribution easier than a
patch, but more difficult than distributing an \el{} package, as it requires the
user to compile code. This is already difficult on \unix{} based systems, as not
every user has \emacs{}’ header files installed, and not all users have the
appropriate compiler installed.  Also, at the time of writing, \emacsv{25.x} has
not yet officially released, so not only would users have to build \emacs{} from
source themselves, they’d have to manually add (experimental) module support.
On MS Windows systems, this would be even more problematic, as hardly any user
has any compiler in the first place, let alone an environment in which \emacs{}
can be built.  The experimental status may also mean changes in the API to which
modules must adhere.

The second approach has one problem: the original reader must be replaced at
runtime.  Fortunately, this is possible due to the dynamic nature of \el.  A
function may not only be replaced wholesale, as is possible in many languages
(such as python), but can also be ``advised''.  This is a feature which has been
present in many Lisp dialects in the past, and has recently seen some popularity
under the name of ``Aspect oriented programming''

\emacs{} offers many different ways to add code to a function, all documented in
the manual \cite[Section 12.11 ``Advising Emacs Lisp
Functions'']{elisp-reference}.  In order to replace the reader (which consists
of only one function: \fun{read} and a few variables) it is sufficient to place
\sym{:around} advice onto the function in question.  This has the effect that
all calls to \fun{read} will be redirected to a different, user supplied
function with all arguments, plus the original function as an extra argument.

This makes it possible to check whether \elr{} shall be used or not, and
redirect to the original function instead.  Also, it makes it possible for
\elr{} to reuse the original reader, where appropriate\footnote{Note that this
  was done only for performance reasons.  Also, nested structures can not in
  general be reused, as the original reader in \emacs{} is not aware of macro
  chars and the like.  If the code for reading lists had been reused, ``(foo
  bar)'' would still read correctly, yet ``(foo \{:bar :spam\})'' would fail, as
  the built-in reader does not know what to do about curly braces.}.

Given this possibility, replacing the built-in reader is a matter of a few lines
of \el{} code, as can be seen in listing~\ref{code:replace-reader}.

\begin{lstlisting}[style=lispcode,caption={Replacing the built-in
    reader},label={code:replace-reader}]
(define-advice read (:around (oldfun &optional stream)
                             el-reader//replace-read)
  (if use-el-reader
      (el-reader/read stream)
    (funcall oldfun stream)))
\end{lstlisting}

The use of \sym{use-el-reader} is explained in section~\ref{sec:activation}.

%\section{Compatibility}
%\label{subsec:compat}

When replacing such a critical part of a language, it is important to maintain
compatibility.  For this, several criteria must be met:

\begin{enumerate}
\item Previously accepted code must be accepted by the new system.
\item Previously accepted code must have the same meaning in the new system.
\end{enumerate}

Both criteria may seem obvious, yet they apply both to syntax and to a function
call, namely to \Read{}.

To achieve this, it is necessary to take a look at the signatures of \Read{}
both in \el{} and in \cl{}, as \elr{} takes a lot of inspiration from \cl{}.  In
\el{} read takes 0 or 1 arguments, where stream is one of the following:
\begin{description}
\item[nil:] the value of \sym{standard-input} (defaults to \sym{t}) is used.
\item[a buffer:] begin at point and advance it.
\item[a marker:] read from where it points and advance it.
\item[a function:] called without arguments to retrieve a character, called with
  one argument to unread a char.
\item[a string:] starts at the beginning, does not change its argument
\item[\sym{t}:] read a line of text either from the minibuffer (very Emacs
  specific), or from standard input, if used in batch mode
\end{description}

\begin{lstlisting}[style=lispinline]
(read &optional STREAM)
\end{lstlisting}

In \cl{} \Read{} has the following signature:

\begin{lstlisting}[style=lispinline]
(read &optional (stream *standard-input*) (eof-error-p t)
      (eof-value nil) (recursive-p nil))
\end{lstlisting}

As in \el{} stream may be called without arguments, yet if exactly one is given,
it refers to a stream.  All extra arguments are also optional, and their default
behavior is---as far as it applies to \emacs{}, identical.

The signature chosen for \elr{} is a little more complex, yet is fully
compatible with \emacs{}’ \Read{} and reasonably compatible with \cl{}:

\begin{lstlisting}[style=lispinline]
(el-reader/read &optional input-stream (eof-error-p t)
                eof-value recursive-p keys)
\end{lstlisting}

The last argument needs some explaining.  Unfortunately, \el{} lacks several
features which are present in \cl{}, hence the authors of \cl{}’s \Read{} took
for granted.  Among these features is the possibility to have a function return
a number of values different than one (i.e. more than one, or even none at all)
without forcing the caller to know or care about the fact.  In \cl{} the first
value returned by a function is called the primary value.  This value can be
used as if the called function merely returned one value.  Should any other
values be of interest, they must be explicitly bound by
\sym{multiple-value-bind} or similar operators.\cite[Section 5.3 ``The Data and
Control Flow Dictionary'']{hyperspec} The multiple values facility of \cl{} also
allows for no values to be present.  In such a case, the primary value (if
bound) is \sym{nil}, yet the caller \emph{may} check for the number of returned
arguments.  As \el{} does not support this mechanism, \elr{} passes an
additional argument named \sym{keys} to switch between returning exactly one
value, and returning a (possibly empty) list of values.  This is done, as it is
possible for a read macro to return no value.  This is done for comments.  For
this reason, read macros must always wrap their return value in \fun{list}, or
\fun{cl-values} (which---at the time of writing---is an alias for \fun{list}).

Note that the keys argument never has to be given by the user.  It is given only
in one two places, which is in \fun{el-reader/read} itself, i.e. in a
self-recursive call, and in the function \fun{el-reader/read-delimited-list}.
Nevertheless, the user must always return a list of values in read macros, which
is a shame, as it departs from the way it is done in \cl{}.

There are other features which are present in \cl{} but not in \el{}, yet none
of them are so critical (and difficult to retrofit) as multiple values.  Some
other features can be emulated at various degrees of awkwardness, such as having
non-nil default values for optional arguments.  These can be used with
\fun{cl-defun}, and have been used in the code for \elr{}.

Other features have only started to appear in \emacsv{25}, such as multiple
dispatch, which is the main reason \elr{} doesn’t work on \emacsv{24.5}.

While \emacs{} has a pre-defined set of types which may form a valid stream
(with the function type being the most general), \elr{} provides a
\fun{defclass} and \fun{cl-defgeneric} based extension mechanism.  This is
discussed in detail in section~\ref{subsec:streams}.

\section{Terminology}
\label{subsec:terminology}

Before describing the algorithm used to transform source code to lisp objects,
it is important to define some terms which will be used to describe the
process.  These terms are all defined in the Hyperspec, which is often regarded
as a free alternative to the \cl{} standard.

\subsection{General Terms}
\label{subsubsec:general-terms}

\begin{description}
\item[Terminating macro character] A character which has been marked as a
  terminating macro character must have a function attached to it, even if this
  function only signals an error when called.  Such characters also terminate
  any token which occurs before them.  If, for instance, `(' is such a
  character, the character sequence ``bar(...'' will stop reading between the
  `r' and `(', resulting in the symbol \sym{bar}.

  If such a character is encountered as the first char to \Read{}, its
  associated function is called.
\item[Non-terminating macro character] These characters work in a similar
  fashion to Terminating macro characters, yet do not terminate a token
  preceding them.  As with a terminating macro char, the associated function is
  called if it is the first character encountered.
\item [Read macro] A read macro is a pair of a (non-)terminating macro character
  and a function, which is associated with it.  The two together create the
  possibility to define new syntactic constructs.
\item[Syntax Type] At any given time every character has exactly one syntax
  type.  Note that different occurrences of the same character can have
  different syntax types.  It is thus not sufficient to provide a relation
  between the character value and a syntax type.  See
  paragraph~\ref{subsubsec:syntax-type} for more details.
\item[Token] A token is an atomic unit of text which has been read by the
  reader.  As it is atomic, no unescaped terminating macro characters can be
  part of it, as they trigger a separate call to \Read{}.  When a token has been
  accumulated, the reader tries to interpret said token as a number (or in
  \cl{}’s case also as a so-called ``potential number'').  Should such an
  interpretation be successful, the given number is returned.  Otherwise the
  token is interpreted as a symbol.  As this step is rather isolated, this step
  is performed by the built-in reader in \elr{}.  This yields significant
  performance gains, as this step is typically performed very often.
\end{description}

\subsection{Character Syntax Types}
\label{subsubsec:syntax-type}

The six following Syntax Types for characters exist.  Note that while each
character always has exactly one type, this type may change over time, and does
this per character instance.  I.e not every `A' has the same syntax type.

\begin{description}
\item[constituent] Constituent characters are characters which may be part of a
  symbol or (potential) number.
\item[macro character] Macro characters come in two flavours: terminating and
  non-terminating (see also section~\ref{sec:read-algo}).
\item[single-escape] A single escape character escapes the following character
  in the stream.  This means that the syntax type of the next char is changed to
  constituent, no matter what its original type.  It also hat \emph{alphabetic}
  as its only trait (see section~\ref{subsubsec:traits} for details).
\item[multiple escape] A multiple escape character escapes all following
  characters until another multiple escape char occurs.  Nested single escape
  characters are ignored, yet do not reverse escaping.
\item[whitespace] Characters of this type end the accumulation of a token
  (i.e. number or symbol).  They are thus used to separate expressions (for
  instance the elements of a list).
\item[invalid] Characters which may not be part of a token.  Note that this set
  may be empty and is in \elr{}.
\end{description}

\subsection{Character Traits}
\label{subsubsec:traits}

A character which has the syntax type constituent has one or more traits.
The following traits are available:

\begin{itemize}
\item alphabetic
\item digit
\item plus sign
\item minus sign
\item dot
\item decimal point
\item ratio marker
\item exponent marker
\item invalid
\end{itemize}

Additionally \cl{} has a trait package marker, which is not particularly
relevant in \elr{} as \el{} cannot sustain package support.  The trait was
nevertheless included, although it has no effect.

The listed traits should be rather self explanatory, with the possible exception
of \emph{alphabetic}: characters with the trait alphabetic can be considered as
``not special''; i.e. can be part of a token, but not of a number.  By virtue of
having the syntax type constituent, no character having any traits can be a
macro character or escape further characters.  When considering that ``3e5'' is
a number, yet ``35e'' is a symbol, it becomes clear that the two occurrences of
the letter `e' can have different traits.  The same holds true for syntax types.

Note that most of these traits are used to construct (potential) numbers.

% \section{Extensible Readers}
% \label{sec:ext-reader}

% \todo{Write something or delete this section!}

\section{Reader algorithm}
\label{sec:read-algo}

As it is the goal of this thesis to provide an extensible reader in spirit and
style of \cl{} it is imporant to highlight the rules of \cl{}’s reader and how
\elr{} deviates from these rules.

The algorithm which \cl{} adheres to is detailed in the HyperSpec in section
2.2\cite{hyperspec}.

It is worth noting that in \cl{} the reader revolves around a central opaque
data structure: the readtable.  Every customization which is made, is made to a
readtable.  Some functions take a readtable as an optional argument while
defaulting to the current readtable (stored in the special variable
\sym{*readtable*}), or simply always using the default table.  Note that this
table can be dynamically rebound, as it is a so called special variable,
i.e. its binding follows a stack discipline.

The reader consists of 10 numbered steps:

\begin{enumerate}
\item Read a character from the input stream into \texttt{X}.  If at
  end-of-file, act according to \texttt{eof-error-p} and \texttt{eof-value}.
  If \texttt{eof-error-p} is non-\texttt{nil} (the default) signal an error.
  Otherwise return \texttt{eof-value} (\texttt{nil} by default).

  According to the syntax type of \texttt{X}, control is dispatched to one of
  the steps 2 to 7, some of which transfer control to steps 8 to 10.
\item If \texttt{X} is an invalid character, signal an error of type
  \texttt{'reader-error}.  In the current implementation of \elr{} this will not
  happen, as no character is marked as explicitly invalid.
\item If \texttt{X} is a whitespace character, discard it and jump to step 1.
  Normally this would be done with a tail call, yet had to be implemented with a
  sort of trampoline.  This branch returns a marker object, which determines
  whether to start again or return the value returned by the conditional
  expression as a whole.
\item If \texttt{X} is a macro character, either terminating or non-terminating,
  its associated function is executed.  The current input stream and the
  character (\texttt{X}) itself are passed as arguments.  Note that this step
  only occurs when looking at the first character since the invocation of the
  current \Read{} frame!  The open parenthesis in ``foo(bar'' would \emph{not}
  trigger this behavior, ``(foo'' on the other hand will.

  Before calling the associated function, a special variable which allows for
  dotted pair notation is bound to \texttt{nil} for the duration of said inner
  call.  The return value of the associated function is the return value of the
  conditional being discussed at the moment, unless it is not a list of
  arbitrary length.  Further dispatching on the length of the list (especially
  if the list is empty) is done outside of this conditional and is discussed
  later.  This will become critical to comments.

  It is critical that the reader macros perform no side-effects other than
  advancing the stream, as code may be read zero or more times.  This is
  especially true when \Read{} is called by editing tools, as they may read
  inner or outer parts of an expression which overlap, and do not care for any
  effects designed for the program which is being edited.
\item If \texttt{X} is a single escape character, another character from
  the stream is read into \texttt{Y}.

  The syntax type of \texttt{Y} is set to constituent with only one trait:
  alphabetic.  An additional property is added to \texttt{Y}, namely
  \texttt{escapedp}.  The resulting character starts a new token, which is
  further processed in step 8.  Note that syntax type, traits and other
  properties are unique to every character.  An unescaped 'A' is a different
  object from an escaped 'A'.  More details on data structures used are given in
  section~\ref{sec:datastructures}.

  Escaping characters can be used to allow characters in a symbol for which
  there is normally no read syntax.  For instance, in and of itself, a symbol
  may be named ``(foo)'' (note the parens, which would normally make up a list).
  One way of obtaining such a symbol is by calling \texttt{intern}---which takes
  a string---at runtime:

\begin{lstlisting}[style=lispinline]
(intern "(foo)")
\end{lstlisting}

  The alternative is using escaping to allow this name at read time:
  
\begin{lstlisting}[style=lispinline]
\(foo\)
\end{lstlisting}
\item If \texttt{X} is a multiple escape character, a new empty token is begun
  and control is transferred to step 9.  Note that the escape character itself
  is consumed from the stream.
\item If \texttt{X} is a constituent character, it begins a new token and enters
  step 8.

  The new token initially consists of one character, \texttt{X}, which has
  copies of its syntax type and traits attached to it.
\item At this point an even number of multiple escape chars have occurred
  (i.e. none are currently in effect), and a token is accumulated.

  A new character \texttt{Y} is read from the input stream.  If end-of-file is
  reached, the token so far is processed.  The resulting token is the return
  value for this branch.

  Another dispatch on the syntax type of \texttt{Y} is performed:
  \begin{enumerate}
  \item If \texttt{Y} is invalid, an error of type \texttt{'reader-error} is
    signaled.  In the current implementation of \elr{} this will not happen, as
    no character is marked as explicitly invalid.
  \item If \texttt{Y} is a single escape char, a new char \texttt{Z} is fetched
    from the input stream and appended to the current token as an alphabetic
    constituent with the \texttt{'escapedp} property.
  \item If \texttt{Y} is a multiple escape char, the character is consumed as
    usual, the token is left as is (possibly left empty).  Finally, step 9 is
    entered.
  \item If \texttt{Y} is a terminating macro character, \texttt{Y} is put back
    into the stream and the token so far is processed.  The processed token (a
    symbol or a number) is returned, thus \Read{} is done.
  \item If \texttt{Y} is a whitespace character, \texttt{Y} is put back into the
    stream if \texttt{el-reader/*preserve-whitespace*} is true.  In both cases
    the token is processed and returned, thus \Read{} is done.
  \item If \texttt{Y} is a constituent or \emph{non}-terminating macro
    character, it is appended onto the token and step 8 repeats.
  \end{enumerate}

  In summary: a single escape char is dealt with in place, a multiple escape
  char hands the token over to code expecting to escape everything (step 9),
  macro characters and whitespace terminate reading and constituent and
  non-terminating macro characters are appended to the token.

\item At this point an odd number of multiple escape characters have been
  encountered (i.e. escaping is in effect). 
  \begin{enumerate}
  \item If \texttt{Y} is either constituent, macro character (terminating and
    non-terminating alike) or whitespace, \texttt{Y} is appended onto the stream
    as an alphabetic constituent with the property \texttt{'escapedp} before
    repeating step 9.
  \item If \texttt{Y} is a single escape character, it is appended to the token
    as an alphabetic constituent with the \texttt{'escapedp} property and step 9
    is repeated.
  \item If \texttt{Y} is a multiple escape character, it consumed without adding
    it to the token before switching back to step 8.
  \end{enumerate}
\item An entire token has been accumulated.  The token is first checked for
  whether or not it constitutes a valid number.  If so, that number is the
  result of this step and hence of \Read{}.  Otherwise a symbol is constructed.
  While \elr{} contains code to do this in \el{}, it is only used if escaped
  characters were encountered.  Otherwise \elr{} removes its own advice from
  \Read{}, calls \Read{} with the token and reinstates the advice.  This is done
  for performance reasons, as the steps taken are exactly the same in \elr{} and
  the built-in reader.  It seems as if is were faster to replace the escaped
  characters in the token with backslashes again and let the built-in reader do
  the job in all cases, yet such a string transformation is so costly that it is
  almost exactly as fast.  Such code exists in \elr{}, yet is not used.
\end{enumerate}

Figure~\ref{code:read-algo} shows the corresponding \el{} code.  Note that many
steps have very straightforward translations, while the lack of multiple-values
and tail-call optimization make rather ugly looping a necessity.  Also, steps 8
and 9 have been pulled out as extra functions and put in front of
\fun{el-reader/read}. 

\begin{lstlisting}[style=lispcode,label={code:read-algo},caption={Code for the
    reader algorithm.},numbers=left]
(cl-defun el-reader//step-8 (input-stream token _x)
  ;; Loop until we do an explicit return.  This would have been so much
  ;; nicer with tco.  *sigh*
  (let (y)
    (while t
      (setf y (condition-case nil
                  (el-reader/getch input-stream)
                (end-of-file (cl-return-from el-reader//step-8
                               (el-reader//process-token token)))))
      (cond ((el-reader//rt/invalid-syntax-type-p
              el-reader/*readtable* y)
             (signal 'reader-error (list "Invalid character" y)))
            ((el-reader//rt/single-escape-char-p
              el-reader/*readtable* y)
             (setf token (s-concat token
                                   (el-reader//put-escaped-prop
                                    (el-reader//force-alphabetic
                                     (el-reader/getch input-stream))))))
            ((el-reader//rt/multiple-escape-char-p
              el-reader/*readtable* y)
             (cl-return-from el-reader//step-8
               (el-reader//step-9 input-stream
                                  token
                                  y)))
            ((el-reader//rt/terminating-macro-char-p
              el-reader/*readtable* y)
             (el-reader/getch input-stream y)
             (cl-return-from el-reader//step-8
               (el-reader//process-token token)))
            ((el-reader//rt/whitespacep el-reader/*readtable* y)
             (when el-reader/*preserve-whitespace*
               (el-reader/getch input-stream y))
             (cl-return-from el-reader//step-8
               (el-reader//process-token token)))
            ((or (el-reader//rt/constituentp el-reader/*readtable* y)
                 (el-reader//rt/non-terminating-macro-char-p
                  el-reader/*readtable* y))
             (setf token (s-concat token
                                   (el-reader//defaults-to-str-props
                                    y))))))))

(cl-defun el-reader//step-9 (input-stream token x)
  ;; At this point a token is being accumulated, and an odd
  ;; number of multiple escape characters have been encountered.
  (let (y)
    (while t
      (setf y (el-reader/getch input-stream))
      (cond
       ((funcall (apply
                  #'-orfn
                  (seq-map
                   (lambda (fn)
                     (-partial fn el-reader/*readtable*))
                   (list
                    #'el-reader//rt/constituentp
                    #'el-reader//rt/terminating-macro-char-p
                    #'el-reader//rt/non-terminating-macro-char-p
                    #'el-reader//rt/whitespacep)))
                 y)
        (setf token (s-concat token (el-reader//put-escaped-prop
                                     (el-reader//force-alphabetic y)))))
       ((el-reader//rt/single-escape-char-p el-reader/*readtable* y)
        (setf token (s-concat
                     token
                     (el-reader//put-escaped-prop
                      (el-reader//force-alphabetic
                       (el-reader/getch input-stream))))))
       ((el-reader//rt/multiple-escape-char-p el-reader/*readtable* y)
        (cl-return-from el-reader//step-9
          (el-reader//step-8 input-stream token x)))
       ((el-reader//rt/invalidp el-reader/*readtable* y)
        (signal 'reader-error "Invalid char"))))))

(cl-defun el-reader/read (&optional input-stream (eof-error-p t)
                                    eof-value recursive-p keys)
  (let ((res el-reader/*repeat-read*)
        (input-stream (el-reader//get-getch-state input-stream)))
    (while (eq res el-reader/*repeat-read*)
      (setf
       res
       (let ((x (condition-case c
                    (el-reader/getch input-stream)
                  (end-of-file
                   (if eof-error-p
                       (signal (car c) (cdr c))
                     (cl-return-from el-reader/read eof-value))))))
         (when (not recursive-p)
           (setf el-reader//*read-objects* nil
                 *el-reader//circular-read-functions* nil))
         (cond ((el-reader//rt/invalid-syntax-type-p
                                    el-reader/*readtable* x)
                (signal 'reader-error (list "Invalid char" x)))
               ((el-reader//rt/whitespacep el-reader/*readtable* x)
                el-reader/*repeat-read*)
               ((el-reader//rt/terminating-macro-char-p
                 el-reader/*readtable* x)
                (let ((el-reader//*allow-single-dot-symbol* nil))
                  (let ((res (funcall
                              (gethash x (el-reader//rt/term-mac-fns
                                el-reader/*readtable*))
                              input-stream x)))
                    (when (not (listp res))
                      (error (s-join
                              " "
                              '("Read-macro functions must always"
                                "return a (possibly empty) list."))))
                    res)))
               ((el-reader//rt/non-terminating-macro-char-p
                 el-reader/*readtable* x)
                (let ((el-reader//*allow-single-dot-symbol* nil))
                  (let ((res (funcall
                              (gethash
                               x
                               (el-reader//rt/non-term-mac-fns
                                el-reader/*readtable*))
                              input-stream x)))
                    (when (not (listp res))
                      (error (s-join
                              " "
                              '("Read-macro functions must always"
                                "return a (possibly empty) list."))))
                    res)))
               ((el-reader//rt/single-escape-char-p
                                    el-reader/*readtable* x)
                (cl-values (el-reader//step-8
                            input-stream
                            (el-reader//put-escaped-prop
                             (el-reader//force-alphabetic
                              (el-reader/getch input-stream)))
                            x)))
               ((el-reader//rt/multiple-escape-char-p
                                    el-reader/*readtable* x)
                (cl-values (el-reader//step-9 input-stream "" x)))
               ((el-reader//rt/constituentp el-reader/*readtable* x)
                (cl-values (el-reader//step-8
                            input-stream
                            (el-reader//defaults-to-str-props x)
                            x)))))))
    (if (plist-get keys :return-list)
        res
      (if res
          (car res)
        (let ((res))
          (while (not res)
            (setf res
                  (el-reader/read input-stream eof-error-p eof-value
                                  recursive-p '(:return-list t))))
          (car res))))))
\end{lstlisting}


\section{Additional algorithms}
\label{sec:add-algos}

While section~\ref{sec:read-algo} details the reader algorithm itself, it alone
is not sufficient to read lisp code.  For instance, the reader algorithm does
not specify a single macro character.  While simple ones like \fun{quote} are
very simple, matching pairs of characters is a little more tricky.  When an open
parenthesis is encountered, it starts reading elements with recursive calls to
\Read{}.  Yet the reader algorithm itself does not specify how to reliably
detect when to stop reading, unless end-of-file is encountered.  One cannot
simply check whether the next character in the stream is a close paren, as
comments and whitespace may lie in between the current position and the closing
paren:

\begin{lstlisting}[style=lispinline]
(foo bar ;; comment
 )
\end{lstlisting}

This is why the function \fun{read-delimited-list} exists.

\subsection{read-delimited-list}
\label{subsec:read-delimited-list}

The purpose of \fun{read-delimited-list} is to read successive elements from the
input stream until a terminating character is found, returning the read elements
as a list.  For almost all purposes the terminating character must be a
terminating macro character, as it may otherwise be consumed by inner tokens.
For instance, in ``(foo bar)'' the second element would read as ``bar)'' if
``)'' was not a terminating macro character.  The terminating character is
consumed when encountered, thus the associated code is only executed on
incorrect input.  For this reason, most characters used as closing delimiters
reuse the definition for the closing parenthesis, which unconditionally signals
an error.

The function \fun{read-delimited-list} must know details about the inner
workings of the reader implementation, as it must check a characters syntax
type, which is not normally part of the public API.

Its signature is as follows:

\begin{lstlisting}[style=lispinline]
(el-reader/read-delimited-list char &optional stream
                               recursive-p)
\end{lstlisting}

As before, the discussion of how the function works will be followed by the code
itself.

\fun{read-delimited-list} looks one char ahead, i.e. reads a character without
consuming it.  None of the code discussed so far does this directly.  While not
relevant to any stream type built into \elr{}, it may be possible for a stream
to support unreading of only one character, thus \fun{el-reader/peek-char} is
used.

The following steps constitute the algorithm for \fun{read-delimited-list}:

\begin{enumerate}
\item The character \sym{C} is obtained by \fun{peek-char}.
\item \label{loop-step}
  If \sym{C} matches \sym{char} (i.e. the character to terminate the list)
  the character is consumed from the list.  Finally, the reversed list of read
  objects is returned.
\item Depending on the syntax type of \sym{C}, one of the following actions is
  performed:
  \begin{enumerate}
  \item If \sym{C} is invalid, a \sym{'reader-error} is signaled.
  \item If \sym{C} is whitespace, \sym{C} is consumed from the input stream and
    a new character is read with \fun{peek-char} and stored in
    \sym{C}.
  \item If \sym{C} is a constituent, macro character (terminating or
    non-terminating), or an escape character (single or multiple), \Read{} is
    called to potentially produce an object.  If \Read{} returns no object (in
    the default configuration this only happens on comments), the whole process
    is repeated, now with an advanced input stream.  Otherwise, the returned
    object is added to the front of a list of objects (initially empty).
  \end{enumerate}
\item Control is transferred back to step~\ref{loop-step}.
\end{enumerate}


\begin{lstlisting}[style=lispcode,label={code:read-delim-list},caption={Code for
  \fun{read-delimited-list}},numbers=left]
(defun el-reader/read-delimited-list (char &optional stream
                                           _recursive-p)
  (let ((end-res el-reader/*repeat-read*))
    (while (eq end-res el-reader/*repeat-read*)
      (setf
       end-res
       (let ((res-list nil)
             (c (el-reader/peek-char stream)))
         (while (/= c char)
           (cond
            ((el-reader//rt/invalid-syntax-type-p
              el-reader/*readtable* c)
             (signal 'reader-error (list "Invalid char" c)))
            ((el-reader//rt/whitespacep el-reader/*readtable* c)
             (el-reader/getch stream)
             (setf c (el-reader/peek-char stream)))
            ((or (el-reader//rt/constituentp el-reader/*readtable* c)
                 (el-reader//rt/terminating-macro-char-p
                  el-reader/*readtable* c)
                 (el-reader//rt/non-terminating-macro-char-p
                  el-reader/*readtable* c)
                 (el-reader//rt/single-escape-char-p
                  el-reader/*readtable* c)
                 (el-reader//rt/multiple-escape-char-p
                  el-reader/*readtable* c))
             (let ((res (el-reader/read stream t nil t
                                        '(:return-list t))))
               (if res
                   (if (listp res)
                       (push (car res) res-list)
                     (error (s-join
                             " "
                             '("Read-macro functions must always "
                               "return a (possibly empty) list."))))
                 el-reader/*repeat-read*))
             (setf c (el-reader/peek-char stream)))))
         (el-reader/getch stream)
         (reverse res-list))))
    end-res))
\end{lstlisting}

\subsection{Reading lists and dotted pair notation}
\label{subsec:list-dotted-pair}

Dotted pair notation has been mentioned in section~\ref{sec:repl-reader}.  In
this section the relationship between regular list notation and dotted pair
syntax, how dotted pair syntax deviates from the regularity of lisp syntax, and
how both are implemented.

\subsection{The regularity of macro character based syntax}
\label{subsec:reg-by-macro}

Most of lisps syntax is highly regular, as an expression either consists only of
a single token like a number or a symbol, or because there is a single character
which is a macro character.  Note that even such seemingly complex rules as
\texttt{\#36rZ} follow this rule: \texttt{\#} is a so called \emph{dispatching
  macro character}.  It thus reads an optional decimal integer, and another
single character which itself is treated as a macro character.  Thus
\texttt{\#r} is a macro which takes a number as a parameter, then reads another
token which is treated as an integer.  The decimal integer which forms the
argument specifies the base which shall be used.  Hence \texttt{\#36rZ} means:
treat \texttt{Z} as an integer in base 36.

Hence, constructs which at first glance seem to deviate from the very regular
form, do not necessarily do so.

\subsection{Regular Lists}
\label{subsec:regular-lists}

Regular lists follow a very simple rule which has been sketched before: begin
with an opening parenthesis, read expressions which form the elements of the
list until a closing parenthesis is encountered.

Thus, from the perspective of the Lisp reader, we again follow prefix notation
in a very direct sense: a macro character triggers a new procedure which
consumes expressions from the input stream until a terminating condition is met.

\subsection{The irregularity of dotted pairs}
\label{subsec:irregular-dotted-pair}

Dotted pair notation at first seems trivial enough: an opening parenthesis, an
expression, a dot, another expression, and a closing parenthesis.  Yet, this
ruleset is not how dotted pairs work.  Otherwise they would indeed be trivial to
implement, despite their somewhat unlispy notation.

A dot may be the second to last element in any list, causing the structure of
the list to be created to change.  The dot means: the \sym{cdr} of the second to
last read element shall be set to the last read element, instead of placing the
last element into a \sym{cons} of its own.

As regular lists, and lists which include a dot share the same triggering macro
character, both constructs must be read by the same code.

A further hindrance is that normally the reader immediately signals an error if
a symbol consisting of only a single dot is read.  The second to last element of
a list is the only place such a symbol may be read!

Thus, the algorithm to read such a list is as follows:

\begin{enumerate}
\item Set the global variable \sym{*el-reader//allow-single-dot-symbol} to
  \sym{t}, which prevents subsequent calls to \Read{} to signal an error when
  a dot is read.
\item Read a list using \fun{read-delimited-list} which ends with a
  close paren (see section~\ref{subsec:read-delimited-list} for details).
\item \label{lbl:read-lisp-list-step-3}
  Check for these three conditions:
  \begin{enumerate}
  \item Is the length of the reads list at least 3?
  \item Is the symbol \sym{.} the second to last element of the list?
  \item Is there exactly one such dot in the list?
  \end{enumerate}
\item If all conditions in step~\ref{lbl:read-lisp-list-step-3} are true, return
  a list which has the correct form, i.e. append the last element to all other
  elements.  Otherwise proceed to step~\ref{lbl:read-lisp-list-step-4}.
\item \label{lbl:read-lisp-list-step-4}
  If the number of dot symbols is not zero, signal an error, otherwise proceed
  to step~\ref{lbl:read-lisp-list-step-5}.
\item \label{lbl:read-lisp-list-step-5}
  Return the originally read list unchanged.  In this case, no dotted pair
  syntax was used.
\end{enumerate}

This is the corresponding lisp code:
\begin{lstlisting}[style=lispcode,label={code:read-lisp-list},caption={Code for
    \fun{el-reader//read-lisp-list}},numbers=left]
(defun el-reader//read-lisp-list (stream _char)
  (cl-values
   (let ((el-reader//*allow-single-dot-symbol* t))
     (let ((l (el-reader/read-delimited-list ?\) stream t))
           (dot (intern ".")))
       (cond ((and (>= (seq-length l) 3)
                   (eq (seq-elt l (- (seq-length l) 2)) dot)
                   (= 1 (cl-loop for c = 0 for s in l if (eq s dot)
                                 count c)))
              (append (seq-subseq l 0 (- (seq-length l) 2))
                      (seq-elt l (1- (seq-length l)))))
             ((not (zerop (cl-loop for c = 0 for s in l if (eq s dot)
                                   count c)))
              (unintern "." obarray)
              (signal 'reader-error "invalid-read-syntax: \".\""))
             (t l))))))
\end{lstlisting}

\subsection{Reading Circular Datastructures}
\label{subsec:circular-datastructures}

Another irregular syntactic construct is the use of \texttt{\#n=} (where
\texttt{n} is any decimal integer) and \texttt{\#n\#} (again, \texttt{n} is any
decimal integer) to read circular data structures.  The two constructs by
themselves are regular enough: they are dispatching macro characters which
happen to take an argument.\footnote{\texttt{\#\#} is a little irregular, as is
  serves two purposes: \texttt{\#n\#} is used to read circular datastructures,
  while \texttt{\#\#} returns the empty symbol.}

Circular datastructures by themselves are not something which is too uncommon:
it is not unheard of for a program to contain a linked list which at some point
references its own head, this forming a list with no apparent end.  What seems
to be rather rare is syntactic support for such a feature.

In \el{} \texttt{\#1=} may be placed before an expression.  This expression is
then read and saved.  The use of \texttt{\#1\#} returns this saved expression.
Implementing this has two major difficulties:  1. A reference may be used before
the saving construct is finished reading, so references have to be replaced
later by walking the read object.  2. Each \emph{toplevel} call to read has its
own scope.  If both of the following expressions were \texttt{toplevel}
expressions in a file, or were part of different toplevel reads, it would not
work as one might expect.

\begin{lstlisting}[style=lispcode,label={code:circular-toplevel-broken},caption={This
  does not work as expected},numbers=left]
#1=(d e f)
(a b c #1#)
\end{lstlisting}

One might expect the result of the second expression to be \texttt{(a b c (d e
  f))}, yet it signals an error, as both operate in a different scope.  This on
the other hand works as expected, and returns a circular list of repeating
\texttt{a} \texttt{b} and \texttt{c} symbols.

\begin{lstlisting}[style=lispinline]
'#1=(a b c . #1#)
\end{lstlisting}

In order to implement this, it is necessary to know whether a call to \Read{} is
toplevel or not.  \el{}s \Read{} has an argument for this purpose: the fourth
argument (\sym{recursive-p}) shall be non-nil if called from within read.

Given this piece of information, the saved references can be cleared and objects
can be walked every time a toplevel call exits.

\subsubsection{Implementation details}
\label{subsubsec:impl-details}

When \texttt{\#n=} is encountered, \fun{el-reader//read-=} is called.  A marker
object is created.  This objects sole purpose is to be distinguishable from all
other objects.  This can be achieved by creating a new cons cell.  As the
contents don’t matter, a call like this is fully sufficient: \texttt{(cons nil
  nil)}.  This allocated a new cons cell, i.e. its memory location is unique.
As \el{} provides an operator to check whether two Lisp objects refer to the
same memory location, namely \fun{eq}, this object is distinguishable from any
other object which may be created.

This newly created marker object is placed into a structure which also records
the number under which the object shall be referenceable, as well as the object
itself (which must be nil for the moment, as the object has not been read yet).
This structure (a vector of three elements) is placed onto a global list which
is cleared every time a toplevel read exits.

Then an object is read from the stream via a call to \Read{}.  The read object
is then recorded in the previously mentioned vector.

When \texttt{\#n\#} is encountered, it is first checked whether or not a number
was passed.  The code implementing this macro character can be found in the
function \fun{el-reader//read-hash-num-hash}.  If not, the empty symbol is
returned.  Otherwise the global list of object references is searched for the
number passed.  If no such number is found an error is signaled, as it is
invalid to reference something before it was defined.

In case such a reference was found, the marker object (the empty cons cell)
which was put into the three element vector mentioned previously, is returned.
Note that no further object is read, and that no immediately useful object is
returned, as the marker object by itself is of little use.

Every time \Read{} exits, it first checks whether it is toplevel.  If it is, the
value which is to be returned is walked in search for marker objects with a call
to \fun{el-reader//replace-placeholders}.

This is done by distinguishing between cons cells (which contain further
elements, and are a potential marker object), arrays (which contain further
elements), hash-tables (which contain both keys and values), and atoms (which
are returned as is).

Each of the compound structures (cons, array and hash-table) are recursively
walked for each of their elements.  Cons cells are treated a little differently,
as they may represent a marker object.  In such a case a lookup on the global
list is performed, and the appropriate object is inserted.  In all cases the
objects are changed destructively, as changing their memory location might cause
other references to fail!

\subsection{Interpreting Numbers}
\label{subsec:interpreting-numbers}

When \Read{} begins to read an expression which begins with a
\texttt{constituent} character it is read as a token.  This means that the
result is either a number or a symbol, and that no macro chars play any role.
The only thing macro characters can do is to delimit a token.  For instance in
the expression \texttt{bar(...)} the opening paren delimits the token
\texttt{bar}.  Note that a token may be read in a recursive call, so in the
expression \texttt{(foo bar)} \texttt{foo} and \texttt{bar} are both tokens
which are read because lists call \Read{} recursively.

When a token has been read completely, \Read{} tries to interpret it as a
number.  Should this succeed, the number is returned.  Otherwise a symbol is
constructed.  There are certain properties which a token may have, which prevent
it from being interpreted as a number.  Any escape characters for instance
prevent this.

The grammar which \cl{} uses is seen in figure~\ref{fig:cl-grammar}.

\begin{figure}
  \begin{grammar}
    <numeric-token> ::= <integer> \alt <ratio> \alt <float>
  
    <integer> ::= {[}<sign>{]} \\
    <decimal-digit>+ \\
    <decimal-point>
    \alt {[}<sign>{]} \\
    <digit>+

    <ratio> ::= {[}<sign>{]} \\
    <digit>+ \\
    <slash> \\
    <digit>+
  
    <float> ::= {[}<sign>{]} \\
    <decimal-digit>* \\
    <decimal-point> \\
    <decimal-digit>+ \\
    {[}<exponent>{]}
    \alt {[}<sign>{]} \\
    <decimal-digit>+ \\
    {[}<decimal-point> \\
    <decimal-digit>*{]} \\
    <exponent>
  
    <exponent> ::= <exponent-marker> \\
    {[}<sign>{]} \\
    <digit>+
                                       
    <sign> ::= "a sign."
  
    <slash> ::= "a slash."
  
    <decimal-point> ::= "a dot."
  
    <exponent-marker> ::= "an exponent marker."
  
    <decimal-digit> ::= "a digit in radix 10."
  
    <digit> ::= "a digit in the current input radix."
  \end{grammar}
  
  \caption{\cl{}s grammar for numbers}
  \label{fig:cl-grammar}
\end{figure}
As \el{} has no support for fractions, but has syntax for positive and negative
infinity and NaN (not a number), the grammar had to be altered.  The grammar
used can be seen in figure~\ref{fig:el-grammar}.

\begin{figure}
  \begin{grammar}
    <numeric-token> ::= <integer> \alt <float>
  
    <integer> ::= {[}<sign>{]} \\
    <decimal-digit>+ \\
    <decimal-point>
    \alt {[}<sign>{]} \\
    <digit>+
  
    <float> ::= {[}<sign>{]} \\
    <decimal-digit>* \\
    <decimal-point> \\
    <decimal-digit>+ \\
    {[}<extension>{]}
    \alt {[}<sign>{]} \\
    <decimal-digit>+ \\
    {[}<decimal-point> \\
    <decimal-digit>*{]} \\
    <extension>

    <exponent> ::= <exponent-marker> \\
    {[}<sign>{]} \\
    <digit>+

    <inf-marker> ::= <exponent-marker> `+INF'
  
    <nan-marker> ::= <exponent-marker> `+NaN'

    <extension> ::= <exponent> \alt <exponent-marker> (<inf-marker> |
    <nan-marker>)
                                       
    <sign> ::= "a sign."
  
    <slash> ::= "a slash."
  
    <decimal-point> ::= "a dot."
  
    <exponent-marker> ::= "an exponent marker."
  
    <decimal-digit> ::= "a digit in radix 10."
  
    <digit> ::= "a digit in the current input radix."
  \end{grammar}
  
  \caption[\elr{}s grammar]{The grammar for numbers used in \elr{}.}
  \label{fig:el-grammar}
\end{figure}

As both grammars are not left-recursive, a simple recursive descent parser can
be used to parse these grammars.  Implementing functions which parse terminals
like a sign or a decimal digit and functions which combine several such
functions or call them repeatedly, specifying a grammar can be quite
declarative.  While the code to extract the information from the resulting tree
is a little more involved, the code to create the parse tree for a float can be
seen in listing~\ref{code:parse-float-raw}.  Note that, apart from terminals and
combinators, there are also convenience functions which use both and can be used
as if for a terminal.  These functions (like \fun{el-reader//parse-extension})
collapse the tree into a more convenient and more shallow form.  This is also
the reason why not only are floats parsed separately from integers, but also why
the production for float has been split up into two functions.  The code which
parses integers has been omitted from this document, as it is much simpler than
the code for floats.

The datastructures used for this code are discussed in
section~\ref{sec:datastructures}.

Note that parsing numbers is the only place where a classic parser is used.
While the rest of Lisps syntax is tree structured, this is achieved by the use
of read macros, not by hardwiring the rules for the syntax into a parser.

\pagebreak

\begin{lstlisting}[style=lispcode,label={code:parse-float-raw},caption={Create a
  function which parses a float into a tree.},numbers=left]
(defvar *el-reader//parse-left-float*
  (el-reader//ensure-complete-token
   (el-reader//parse-seq
    (el-reader//parse-optional #'el-reader//parse-sign)
    (el-reader//parse-kleene-star #'el-reader//parse-decimal-digit)
    #'el-reader//parse-decimal-point
    (el-reader//parse-plus #'el-reader//parse-digit)
    (el-reader//parse-optional #'el-reader//parse-extension))))

(defvar *el-reader//parse-right-float*
  (el-reader//ensure-complete-token
   (el-reader//parse-seq
    (el-reader//parse-optional #'el-reader//parse-sign)
    (el-reader//parse-plus #'el-reader//parse-decimal-digit)
    (el-reader//parse-optional
     (el-reader//parse-seq
      #'el-reader//parse-decimal-point
      (el-reader//parse-kleene-star #'el-reader//parse-decimal-digit)))
    #'el-reader//parse-extension)))
\end{lstlisting}

\subsubsection{Explanation}
\label{subsubsec:parser-explanation}

Functions like \fun{el-reader//parse-seq} take one or more functions which
themselves take a token and a position and return a \sym{result}, and return a
function of the same type, i.e. take a token and position and return a result.
To better illustrate this, it may be useful to employ a more formal and succinct
syntax for function signatures.

Haskell\cite{haskell.org} has such a syntax which I shall borrow: ``typeA ->
typeB'' represents a function which takes an argument of type ``typeA'' and
returns an argument of type ``typeB''.  ``A -> B -> C'' represents a function
which takes an ``A'' and a ``B'' and returns a ``C''.  ``(A -> B) -> C'' on the
other hand represents a function which takes a function from an ``A'' to a ``B''
and returns a ``C''.

So the type of \fun{parse-seq} is:

\begin{lstlisting}[style=lispinline]
(token -> pos -> result) -> (token -> pos -> result)
... -> (token -> pos -> result)
\end{lstlisting}

On the other hand, functions which directly parse an entity have the following
signature:

\begin{lstlisting}[style=lispinline]
(token -> pos -> result)
\end{lstlisting}

Thus, functions which parse an entity may be passed to functions which combine
such functions.

Last but not least there are also functions which return a function which parse
an entity based on some input.  One such function which does this by parsing a
single character is \fun{el-reader//parse-single-char}, which has the following
signature:

\begin{lstlisting}[style=lispinline]
char -> (token -> pos)
\end{lstlisting}

This means that it takes a single character as an argument and returns a
function which parses this exact character given both a token and a position.
If one wanted to parse the literal character sequence ``+INF'', as is needed to
correctly parse floating point numbers representing infinity (either positive or
negative), one can construct such a function using \fun{parse-seq} and
\fun{parse-single-char}, as can be seen in listing~\ref{code:parse-inf-marker}.

\begin{lstlisting}[style=lispcode,label={code:parse-inf-marker},caption={Parse
    ``+INF''},numbers=left]
(defvar *el-reader//pf/parse-inf-marker*
  (el-reader//parse-seq
   (el-reader//parse-single-char ?+)
   (el-reader//parse-single-char ?I)
   (el-reader//parse-single-char ?N)
   (el-reader//parse-single-char ?F)))
\end{lstlisting}

Using this form of function combination it is possible to write parts of the
parser in a fairly declarative form without even having to resort to macros.

\subsection{Dispatching Macro Characters}
\label{subsec:dispatching-macro-chars}

There is another algorithm with directly user visible consequences, which does
not directly belong to the reader: dispatching macro characters.

These are macro characters which themselves provide a sort of ``sub macro
characters''.  This means that a dispatching macro character does not by itself
do much.  It merely checks the next characters to check if a previously defined
sub character is encountered.  If this is the case, the appropriate function is
called.  In addition an optional decimal number between the dispatching char and
the sub char is read and passed (if present, \nil{} otherwise) to the function.
Both in \cl{} and \el{}, only one such character exists in standard
syntax.\footnote{While it is not exactly correct to call such a character a
  dispatching macro character in \el{}, as \el{} does not even have macro
  characters in the actual sense of the word, it is a dispatching macro
  character in \elr{}.}  The character \texttt{\#} is such a dispatching macro
character.  Note that while this character is also a \emph{non-terminating}
macro character in \cl{}, it is not in \elr{}, as the same character terminates
a token in standard \el{}.

While the mechanism of the implementation is different from regular macro
characters, to a user aiming to employ read macros, they appear to work much the
same.  The purpose of dispatching macro characters is of course to provide a way
to allow more possible macro characters, without having to worry about clashes.
For instance, in both \cl{} and \el{}, curly braces have no special meaning.
This means that they may be part of a symbols name, or even provide the whole
name.  In both of these languages, the expression \texttt{\{\}} yield a symbol.
The advantage of using a dispatching character is that the dispatch
character may not be used as a part of a symbol.\footnote{Note that in \cl{} it
  may be part of a token for a symbol; it is merely illegal to start a symbol
  with a \texttt{\#}.  This is not because \texttt{\#} is a dispatching
  character, but because it is non-terminating.}

The reason this algorithm is not directly part of the reader, is because it
itself is implemented as a regular read macro.  The function which registers a
given character to be a dispatching macro character by associating a predefined
function which does the dispatching based on the current readtable is
\fun{make-dispatch-macro-character}.

% The function
% \fun{make-dispatch-macro-character} registers the given character to be a macro
% character and associates a predefined function to it.

\section{Datastructures}
\label{sec:datastructures}

All but the most trivial programs need some form of custom datastructures, and
this program is of course no exception.  In this section some of the more
interesting datastructures are discussed, especially where their use relates to
language features not necessarily present in more mainstream languages.

\subsection{The Readtable}
\label{subsec:readtable-class}

The most important datastructure is the readtable, as it is user visible, and
user customizable.

It contains almost everything which is customizable by the user.  This not only
includes which characters are macro characters and what their associated
functions are, but also which characters have which default syntax type and
traits.  This is different from \cl{}, which does not allow the user to change
the syntax types of characters quite as freely.  \cl{} has a function
\fun{set-syntax-from-char} which allows the user to copy the syntax type of one
character to another, i.e. the user must first find a character which already
hat the desired syntax type.  This may be difficult if the user wishes to make a
character invalid.  Another difference is that \cl{} does not allow the user to
set traits.\footnote{See section 2.1.4.2 of the Hyperspec.}

The readtable in \elr{} also contains a relation from characters to numbers in
form of a hash-table.  This can be used in the rare cases in which number input
shall be different than expected.  For example, when using base 16, the
character `A' represents the number 10.  Using this table, this meaning can be
changed.  This is a feature which will probably hardly ever be used.

Most interactions with the readtable will probably take place with the following
functions:
\begin{itemize}
\item \fun{el-reader/set-macro-character} 
\item \fun{el-reader/get-macro-character}
\item \fun{el-reader/make-dispatch-macro-character}
\item \fun{el-reader/set-dispatch-macro-character}
\item \fun{el-reader/get-dispatch-macro-character}
\end{itemize}
All of these functions take a readtable and either give a macro character a
function, or retrieve said function.

These functions also exist (without the prefix) in \cl{} in almost the same
form.\cite[Section 23.2 ``The Reader Dictionary'']{hyperspec}

Most other functions which interact with the readtable (e.g. \Read{}) do not
take a readtable as an argument, as this would be very cumbersome.  These
functions use the global default, which is the variable
\sym{el-reader/*readtable*}.  It is important to note that `global' means that
the variable is dynamically bound, i.e. can be rebound on a stack frame with
\fun{let}.  This makes global variables much more powerful and useful than in
more static languages like C, as they are globally visible, but can be rebound
for anything below the current stack frame.

The readtable holds two slots which determine the association of macro
characters to their appropriate function: \sym{term-mac-fns} and
\sym{non-term-mac-fns} which are both hash-tables from a character (i.e. an
integer) to a function.  The former is for terminating, the latter for
non-terminating characters.  Otherwise, both work the same.  These slots also
serve the purpose of determining whether a character is a (non-)terminating
macro character or not, as one can check whether a key is present in a table or
not.

The readtable also has a slot which holds a mapping for dispatching macro
characters.  By default this table only has one key: `\#'.  The corresponding
value is in turn also a hash-table which in turn maps characters to functions.

The function \fun{make-dispatch-macro-character} associates a function
with the dispatching character.  This function first tries to read a decimal
number from the input stream.  This does \texttt{not} use \Read{}, but uses
\fun{el-reader//getch} instead.  As soon as \fun{getch} does not return a
character between `0' and `9', the offending character is pushed back onto the
stream and the read number (if any) is converted.  As the next step, the offending
character is read from the stream (again), and used as a key in a lookup table
which records how to dispatch further.  This way the characters used in
dispatching macros and regular macro characters may overlap without causing
problems.  For instance the construct ``\#4r0123'' uses the character `r' as a
sort of macro character without causing the rest of the reader to mistake `r'
for a macro character.  Also, the number---if any---is passed to the function
associated with the dispatch character.

\subsection{Streams}
\label{subsec:streams}

Both in \cl{}, \el{}, and in \el{} with \elr{}, the first argument to \Read{} is
a stream.  As Emacs does not provide a good abstraction over streams, one had to
be created.  Before showing how these are implemented, a short explanation of
multimethods is in order.  For a more thorough explanation, Peter Seibels book
``Practical Common Lisp'' has two very good chapters on how Object
Orientation---and hence multimethods---is done in \cl{}.\cite{pcl} Emacs 25.1
tries to mimic this behavior.

\subsubsection{Multimethods}
\label{subsubsec:multimethods}

In contrast to more mainstream Object Oriented languages, in \cl{} methods do
not belong to classes.  Rather, in \cl{} the programmer may define a generic
function, which consists only of name, a lambda-list (i.e. an argument list)
and---in \el{}---optionally a default implementation.  Additionally, methods may
be defined which provide an implementation for particular classes of arguments.
As long as methods only specialize on the first argument, they work very similar
to classic single-dispatch (i.e. virtual methods).  This means that a concrete
method is chosen according to the type of the first argument.

The power and name of multimethods come from the possibility to dispatch on more
than one argument (also called ``multiple dispatch'').  Hence the argument on
which dispatching is performed does not necessarily have to be the first.  This
means that the concrete method is chosen according to the classes of
(potentially) \emph{all} arguments of the argument list.

Another advantage of multimethods is that methods may specialize on types not
defined by the author of the method.  This is of critical importance to the code
of this thesis.

As multimethods are not part of the language core, but are a set of macros which
were added later, there are some types, for which dispatching is not exactly
straightforward.  This is the case in \el{}, not \cl{}!

It is not possible to directly dispatch on functions, as there is no universal
function type.  A function may be byte-compiled, or it may be a list, where the
first element just happens to be the symbol \sym{lambda}.  A function may also
be a list, where the first symbol just happens to be \sym{closure},
followed---among other things---by an environment which represents the lexical
environment which was in place when the closure was created.

\subsubsection{Generic functions and methods for streams}
\label{subsubsec:multimethods-streams}

In order to handle streams in an extensible way, at least two operations have to
be provided for all objects which may be a stream: fetching a character and
putting a character back onto the stream.  This means that state must be kept.
In the case of a buffer or marker this is already done.  A function may also
function as a stream by returning a character when called without any argument,
and putting a character back when called with an argument (putting the given
character back onto the stream).  For strings however, this does not work
directly, so another mechanism is needed first: one must be able to retrieve a
stateful object from a stateless one.  For this, the generic function
\fun{el-reader//get-getch-state} was defined.  It takes an argument and returns
a stateful version of the given argument, or the unchanged argument if no
conversion is needed (as for buffers and markers).

% The default implementation checks whether the given argument is a function or
% not.  If not, it is returned as is, assuming that a specialization on
% \fun{getch} already exists.  If a specialization for the given
% argument existed, it would have been chosen over the default method.

If the given argument \texttt{is} a function, it is wrapped in a
\sym{function-read-state} object, which serves as a workaround for
the fact that \el{} cannot dispatch on functions.  This object merely holds the
function given.  Also, a method for \fun{getch} exists which merely
calls the function given, i.e. doesn't do much.

The previous paragraph does not hold if the given argument is the function
\fun{get-file-char}.  This is because this particular function is passed in by
\emacs{}, yet signals an error if given an argument to push back onto the
stream.  As a workaround for this quirk the function is wrapped in a closure
which holds its own list of pushed back characters before it is wrapped by a
\sym{function-read-state}.

Additional methods were provided for strings and functions.  Strings need
special handling, as they do not hold a current position as part of their
state.  Functions need a wrapper which does not hold any additional data but
provides a type on which a method may dispatch.

The other generic function needed provides the actual mechanism of fetching and
restoring a character, named \fun{getch}.  This function accepts---besides an
object returned from \fun{get-getch-state}---an optional character.  If no
character is given, a character is read from the stream and the position of the
stream is advanced by one.  If a char is given it is returned to the stream and
the position is decreased.  The code in the actual reader assumes that only one
character may be put back, and further assumes that only characters which have
been read may be returned.  This is because the reader does not need more
capabilities and also works on input sources which shall not change or are even
read only.  Calling \Read{} on a buffer should not mutate said buffer, apart
from the position of point in the buffer.  Note that while \elr{} does not push
back more than one character explicitly, this may still happen when
\fun{peek-char} is used, as \fun{peek-char} calls \fun{getch} twice,
i.e. \emph{unreads} a character.  This may lead to unreading two characters.
This situation is not a problem (at least not yet), as all streamtypes and their
wrappers support multilevel unreading.  Also the generic function
\fun{peek-char} may be specialized with a method on a stream type to remedy this
situation.  It may even improve performance.

\subsection{Structures for the recursive descent parser}
\label{subsec:parser-structs}

The recursive descent parser used in order to parse numbers uses several classes
which serve both as structures and as dispatch targets.

\subsubsection{The \sym{result} type}
\label{subsubsec:result}

Every operation in the parser returns an object of type \fun{el-reader//result},
regardless of success.  The object itself contains these fields:

\begin{description}
\item[success] A symbol which is either \nil{} (false) or non-\nil{}, mostly
  \tee{}.
\item[token] The token which is processed.  This is either a complete number, a
  symbol, or a syntax error.  This is of type string.
\item[pos] An offset at which the token begins.  An integer.
\item[newpos] The position at which the token (so far) ends.  This is used in
  later calls to know where to continue reading.  An integer.
\item[result] This may be an object of any type, even \nil{}.  It represents the
  result of a parsing function.  This is only of interest if \sym{success} is
  non-\nil{}.  This value is set by virtually all parsing functions.  Some
  functions add to it, possibly first converting a previous value to a list
  (\fun{parse-seq} does this).  Others condense it into a different object.  An
  example for this is \fun{parse-float} and several other functions it calls.
  These functions successively condense the result until a single \sym{float}
  remains.
\item[metadata] This may be any value, but mostly is used---if at all---as a
  plist, representing mappings from keywords to arbitrary Lisp objects.  This is
  set by \fun{parse-alt} to a value which hints which of the given alternatives
  was successful, hence makes up the value.  This was not really used after
  all, so it may be removed in future versions.
\end{description}

While the slots \sym{success} \sym{token} \sym{pos} \sym{newpos} and
\sym{metadata} are only needed for the parsing, i.e. \emph{recognizing} a given
structure, \sym{result} is at the heart of actually producing a value.  This is
why every parsing function directly or indirectly returns such an outer
\sym{result}.  Some functions take one or more \sym{result} objects
and combine their \sym{result} \emph{slots}.  The function \fun{parse-seq} does
this by putting objects in a list, but several functions rather create objects
of the type \sym{el-reader//syntax-element} or a derivative.

\subsubsection{The \sym{syntax-element} type}
\label{subsubsec:syntax-element}

Many functions put an object of type \sym{syntax-element} or one of
its descendants into the \sym{result} slot of the returned
\sym{result} object.  In the most simple case such an object merely
contains of a \sym{value} slot, but descendants may add additional slots.

These are all syntax elements used are the following (as far as not mentioned
otherwise, all types descend from \sym{syntax-element}.

\begin{description}
\item[syntax-element] Only contains a \sym{value} which may be an object of any
  type. 
\item[digit] Besides a \sym{value} which represents the numeric value of the
  string of text which was parsed (i.e. recognized) as a digit, it also contains
  a \sym{base} slot, which contains the base in which it was read as an integer.
\item[decimal-digit] This type is the same as a \sym{digit}, yet always has the
  value 10 as its base.  The main difference is that it has a different type
  tag.  This type descends from \sym{digit}.
\item[exponent-marker] Apart from the \sym{value} slot which contains a
  symbol which represents the type of marker used
  (e.g. \sym{'single-exponent-marker}).
\item[exponent] Directly descends from \sym{syntax-element}, contains an integer
  as its \sym{value} slot.
\item[char] Contains a single character.
\item[sign] Contains a symbol which encodes which sign (if any) was parsed.
\item[plus-sign] Descends from \sym{sign}, always contains a plus sign.  Serves
  a similar purpose as \sym{decimal-digit}.
\item[decimal-point] Represents a decimal digit.
\item[inf-marker] Represents an infinity marker.
\item[nan-marker] Represents a \textbf{N}ot \textbf{a} \textbf{N}umber marker.
\item[float] Represents a float.  Its \sym{value} slot is the float it
  represents.
\end{description}

\subsubsection{Strings with text properties for syntax types and traits}
\label{subsubsec:stings-syntax-types}

As a given character may have one syntax type at one moment, yet a different
type at another (see section~\ref{sec:read-algo}, one must encode this.  A
simple character will not do, as an open parenthesis may have the syntax type
terminating macro character, or (if it was escaped) have the syntax type
constituent and the trait alphabetic.  Fortunately \el{} makes this particular
case extremely easy: it provides strings with text properties.\cite[Section
2.3.8.4 ``Text Properties in Strings'']{elisp-reference}

This allows the user to specify any value for any property (usually a symbol)
for a given stretch of a string.  Note that any string will do, it is not
necessary to create special strings for this behavior.  When strings are
concatenated, their text properties are copied too.  Their positions are also
adjusted.  For example a string ``('' with the syntax type set to terminating
macro character looks like this:

\begin{lstlisting}[style=lispinline]
#("(" 0 1 (syntax-type terminating-macro-character))
\end{lstlisting}

A string ``.'' with the syntax type constituent and the traits alphabetic, dot
and decimal point looks like this:

\begin{lstlisting}[style=lispinline]
#("." 0 1 (syntax-type constituent traits 
                       (alphabetic dot decimal-point)))
\end{lstlisting}

When both are concatenated the result looks like this:

\begin{lstlisting}[style=lispinline]
#("(."
  0 1 (syntax-type
       terminating-macro-character)
  1 2 (syntax-type
       constituent
       traits
       (alphabetic dot decimal-point)))
\end{lstlisting}

Note that the positions of the second string have been adjusted.  This way
single characters are represented by strings of length one, while sequences of
characters are strings of the appropriate length.  In either case, text
properties are used to encode syntax type and traits of the characters.  These
are also used in the recursive descent parser for numbers to determine whether a
character fulfills the necessary criteria.

\chapter{API overview}
\label{chapter:api-overview}

The API of \elr{} takes lots of inspiration from the corresponding API in \cl{},
as the two languages are sufficiently similar, and \cl{} has a very mature
library.\footnote{As the two libraries are so similar, often eliding the
  \sym{el-reader/} prefix yields a name present in \cl{}.}  Also, it may be
expected that some \el{} developers are familiar with \cl{} read macros.  Those
who are not may benefit from the similarity, as learning and reference material
on \cl{} read macros seems to be more widely available than for any other Lisp,
probably due to its stable standard.  \footnote{Paul Grahams book ``On
  Lisp''\cite[p.~224]{on-lisp} contains a chapter explaining both regular macros
  and read macros.}

The intention of this chapter is to serve as a short user manual/tutorial on how
to use read macros.  It should not be necessary to have read the rest of this
thesis, as long as one already knows a fair amount of Lisp, preferably \cl{}
and/or \el{}.

At first, the general outline of how to setup a read macro will be discussed.
Afterwards an example will be implemented, which will explain why the steps are
done in the way they are.  The last section of this chapter will provide a short
reference.  For the most part, any \cl{} reference will do, as the APIs are
deliberately similar.  Finally the most important differences with \cl{} will be
summarized.

\section{Activation}
\label{sec:activation}

To prevent \elr{} from interfering with every code, it has to be activated for
each file or buffer.  This can be done by exploiting \emacs{}’ buffer local
variables\cite[11.10 ``Buffer-Local Variables'']{elisp-reference}.

Two buffer local variables were introduced by \elr{}, one of which is
experimental.  The other is \sym{use-el-reader}.  This informs the advice around
\Read{} on whether to use \elr{} or the stock reader built into \emacs{}.  The
best way to set it is to ensure that \elr{} is loaded (for instance by using
\sym{require} or \sym{load})\cite[15 ``Loading'']{elisp-reference} and by
putting the line shown in figure~\ref{code:activate-el-reader} before any use of
\elr{} specific functionality.  Figure~\ref{code:activate-el-reader} shows a
complete file which activates \elr{}, adds and uses syntax for hash-tables.

\begin{lstlisting}[style=lispcode,label={code:activate-el-reader},
caption={Activating \elr{}},numbers=left]
(eval-and-compile
  (setf use-el-reader t))
\end{lstlisting}

\begin{lstlisting}[style=lispcode,label={code:using-and-adding-elr},
caption={Using \elr{} and adding syntax},numbers=left]
;; -*- lexical-binding: t -*-

(eval-and-compile
  (setf use-el-reader t))

(defun el-reader//read-hash-table (stream _char)
   (cl-values
    (let ((k-v (el-reader/read-delimited-list ?\} stream t)))
      (if (= (mod (length k-v) 2) 1)
          (error "Invalid syntax: {}")
        `(el-reader//ht ,@k-v)))))

(el-reader/set-macro-character ?\{ #'el-reader//read-hash-table)

(cl-multiple-value-bind (fun _term)
    (el-reader/get-macro-character ?\))
  (el-reader/set-macro-character ?\} fun))

(message "use-el-reader: \%s" use-el-reader)
(message "hash-table: \%s" {:foo "FOO!"})
\end{lstlisting}

\section{Outline}
\label{sec:outline}

The reader has a few user visible datastructures, but one is the most important:
the readtable.  This structure holds all information from the syntax types and
traits of characters to the mapping of characters to functions.

There is always exactly one readtable in effect: the structure stored in the
special (i.e. dynamic) variable \sym{*readtable*}.  This table may either be
modified in place, thus affecting all code under the same binding (possibly all
code in the Emacs process, if the table is never rebound), or may be rebound,
possibly with a copy of the existing table.  Such a copy may be acquired with
the function \fun{el-reader/copy-readtable}.

For those rare cases in which a completely new table is needed, it is also
possible to create a somewhat empty table with \fun{make-instance}.\cite[Section
4, ``Making New Objects'']{eieio} The function
\fun{el-reader/make-default-elisp-readtable} may be used to obtain the readtable
with was in effect before any changes were made.  The only values present in a
readtable created via \fun{make-instance} are whitespace characters and a
mapping from characters to numbers used in number conversion (i.e. `0' maps to
0, `A' to 10, `Z' to 35).

After having chosen which table to use and modify, it is time to actually modify
the table.  This is done---as mentioned in section~\ref{subsec:readtable-class}
(\nameref{subsec:readtable-class})---with the following functions, documented in
section~\ref{sec:api-reference}:

\begin{itemize}
\item \fun{set-macro-character} 
\item \fun{get-macro-character}
\item \fun{make-dispatch-macro-character}
\item \fun{set-dispatch-macro-character}
\item \fun{get-dispatch-macro-character}
\end{itemize}

\section{An Example}
\label{sec:example}

In this section an example of a read macro is presented and discussed in detail.
It is a fairly simple, yet useful macro.  As mentioned earlier in this thesis
(see section~\ref{section:motivation}), Clojure---like many other more modern
languages---features a concise notation for hash-tables, namely surrounding a
list of keys and values with curly braces.  Such notation is implemented and
shown in this example.  Listing~\ref{code:hash-table} shows the code.

\begin{lstlisting}[style=lispcode,label={code:hash-table},caption={Create syntax
    for hash-tables.},numbers=left]
(defun el-reader//ht (&rest args)
  "Create and return a hashtable.

Keys and values are given alternating in args."
  (let ((h (make-hash-table)))
    (cl-loop for (key value) on args by #'cddr
             do (if (and key value) (puthash key value h)
                  (error "Odd number of arguments passed")))
    h))

(defun el-reader//read-hash-table (stream _char)
   (cl-values
    (let ((k-v (el-reader/read-delimited-list ?\} stream t)))
      (if (= (mod (length k-v) 2) 1)
          (error "Invalid syntax: {}")
        `(el-reader//ht ,@k-v)))))

(el-reader/set-macro-character ?\{ #'el-reader//read-hash-table)

(cl-multiple-value-bind (fun _term)
    (el-reader/get-macro-character ?\))
  (el-reader/set-macro-character ?\} fun))
\end{lstlisting}

As with regular macros, it is important to remember which code is run at which
time.  The code shown in the function \fun{read-hash-table} is run at read time,
namely as soon as \Read{} encounters an opening curly brace.  This is the case
because of the line of code directly following the definition of the function.
It registers the open curly brace to be a terminating (because non-terminating
would have required an extra flag, which was not given) macro character, and
associates it with the function \fun{read-hash-table}.  The last line registers
another character as a macro character, namely the closing brace.  As the
closing brace does not do much, except serve as a closing marker, it may not be
obvious what its associated function should do.  The last expression in the
shown code fetches the function used for the closing parenthesis and reuses it.
The only thing this function does is raise an error.  This is because a closing
paren or brace may never be read directly.

As the proposed syntax consists of a list of alternating keys and values
directly surrounded by curly braces, the obvious tool to reach for is
\fun{read-delimited-list} (for the \cl{} version of this function see
\cite[Section 23.2 ``The Reader Dictionary'']{hyperspec}).  This function is
passed a character to look out for (which must also be a macro character) as
well as a stream and whether the call is performed from within a recursive call
to read.  While \elr{} (currently) completely ignores this argument, it is a
good idea to always set this to \tee{}, if only for the sake of consistency with
\cl{}.  The call to \fun{read-delimited-list} not only reads a list, it also
removes the closing character from the stream.  This is done without invoking
the function associated with the character.  This means that if \Read{} directly
encounters a closing paren or brace, it must be a syntax error.  It is for this
reason that the only thing the function associated with these characters does is
to signal the appropriate error.

The result of the call to \fun{read-delimited-list} is---not surprisingly---a
list of all expressions between the curly braces.  Remember that the expressions
given by the user have \texttt{not} been evaluated.  They have merely been read!
If the user places an unquoted symbol as a key, the symbol itself is returned.
Only in the next step something possibly unexpected would happen.  If the number
of expressions is indeed odd, an error is signaled, which occurs at read-time.
This means that informing the user of the error in the program is not delayed
until runtime, as it is so often the case in dynamic languages like Lisp.
Instead the user is informed immediately.

The last step (which is only run in case an even number of expressions was read)
is seemingly innocuous, yet may hold some surprises for users not particularly
familiar with (either kind of) macros.  The backquote at the beginning of the
expression---itself a read macro---represents a sort of template.  It does
\texttt{not} execute the form (i.e. does not call the function
\fun{el-reader//ht}), but returns a list.  The first element of the list is the
symbol \sym{ht} (i.e. the symbol was implicitly quoted).  As the next
expression begins with a comma (,)---more specifically a comma followed by an @
sign---the expression is inserted (spliced) into the template.  If for example
the user entered the form \texttt{\{:foo "bar"\}}, \fun{read-delimited-list} would
return the list \texttt{(:foo "bar")}.  Given the fact that said list is stored
in the variable \sym{k-v}, and that the number of elements is even, consider the
following evaluation:

\begin{lstlisting}[style=lispinline]
`(el-reader//ht ,@k-v)
=> (el-reader//ht :foo "bar")
\end{lstlisting}

Note that the list form shown as the result above is \emph{not evaluated}!  It
is returned as a Lisp object which just happens to be a list.  It is the result
of our macro function, and thus also the result of a call to \Read{}.  The
runtime effect of the given code is that a the function \fun{ht} will
be called with the arguments \texttt{:foo} and \texttt{"bar"}, thus constructing
a hash table at runtime.

Note that it is perfectly legal to perform code evaluation at readtime, just as
regular macros may perform evaluations at compile-time.  This may have all sorts
of reasons, performance being one of them.  This was not done here, as the user
shall have the possibility to use variables which are only available at runtime.
Had the code not said \texttt{`(el-reader//ht ,@k-v)} but \texttt{(apply
  \#'el-reader//ht k-v)}, the syntactic construct would have been more in line
of an \el{} vector, i.e. variable references would not have been looked up, but
would have been quoted instead.  In such a case \texttt{\{:foo bar}\} would
always result in a table where the key \sym{:foo} was mapped to the symbol
\sym{bar}, regardless of whether \sym{bar} was bound---and if so to what.

\pagebreak

\section{Reference}
\label{sec:api-reference}

\subsection{Functions}
\label{subsec:functions}

\subsubsection{set-macro-character}
\label{subsubsec:set-macro-character}

\begin{lstlisting}[style=lispinline]
(cl-defun el-reader/set-macro-character
    (char new-function &optional
          non-terminating-p
          (readtable el-reader/*readtable*)))
\end{lstlisting}

This function is used to set a given character to be a macro character.  As can
be seen in the last argument, a readtable may be given, but if it is omitted,
the current table is used.  The macro character is associated with the given
function (\sym{new-function}).  This means that \Read{} calls said function with
the current input stream and said macro character, in that order.  Another
important consequence of making a character a (terminating) macro character is
that it now delimits tokens.  If for instance one were to make `a' a macro
character, the expression \texttt{fooabar} would yield the token \texttt{foo}.
Another read would trigger the macro character.

If, on the other hand, the character is made into a non-terminating macro
character (by passing non-\nil{} as the third argument), the expression
\texttt{afoobar} would trigger the function, yet \texttt{fooabar} would not.
Instead, the latter expression would return \texttt{fooabar} as a single token.

As can be seen in the next section, it can be desirable to make a character into
a macro character without having it’s function ever be called in a correct
program.  This is the case when using delimited lists, as was seen in the
example in section~\ref{sec:example}.

\subsubsection{get-macro-character}
\label{subsubsec:get-macro-character}

\begin{lstlisting}[style=lispinline]
(cl-defun el-reader/get-macro-character
    (char &optional
          (readtable el-reader/*readtable*)))
\end{lstlisting}

This function retrieves the function set in the given (or current) readtable for
the given character.  It is pretty much the opposite of
\fun{set-macro-character}.  It returns a list of two elements, the function in
question and a flag denoting whether the character in question is a terminating
(\nil{}) or non-terminating \tee{}.\footnote{In \cl{} this function returns
  multiple (two) values; the primary value being the corresponding function (if
  any), and the second value being a flag denoting whether the character is a
  terminating or non-terminating macro character.}

\subsubsection{make-dispatch-macro-character}
\label{subsubsec:make-dispatch-macro-character}

\begin{lstlisting}[style=lispinline]
(cl-defun el-reader/make-dispatch-macro-character
    (char &optional non-terminating-p
          (readtable el-reader/*readtable*)))
\end{lstlisting}

This function creates a new dispatching macro character.  Macro characters
involving dispatching consist of a character which begins a sequence, optionally
a decimal number, and then another (non-decimal) character which is mapped to a
function supplied by the user.  While the only dispatching macro character in
\cl{}, the hash-sign (\texttt{\#}), is also the only non-terminating macro
character, the two concepts are completely unrelated.  In fact, in \elr{},
\texttt{\#} is a terminating character, as it is in standard \el{}.

\subsubsection{set-dispatch-macro-character}
\label{subsubsec:set-dispatch-macro-character}

\begin{lstlisting}[style=lispinline]
(cl-defun el-reader/set-dispatch-macro-character
    (disp-char sub-char new-function &optional
               (readtable el-reader/*readtable*)))
\end{lstlisting}

As with \fun{set-macro-character}, this function takes a character (in this case
two, namely a dispatching and a sub character), a function, and optionally a
readtable.  Associates the given character sequence with the given function.
Always returns \tee{}.

\subsubsection{get-dispatch-macro-character}
\label{subsubsec:get-dispatch-macro-character}

\begin{lstlisting}[style=lispinline]
(cl-defun el-reader/get-dispatch-macro-character
    (disp-char sub-char &optional
               (readtable el-reader/*readtable*)))
\end{lstlisting}

As with \fun{get-macro-character}, this function returns the function (if any)
associated with the given macro character.  Unlike \fun{get-macro-character}
however, it only returns one value, as dispatched macro characters cannot be
non-terminating.  However, the \texttt{dispatching} (i.e. first) character can
(and in \cl{} also is).

\subsubsection{copy-readtable}
\label{subsubsec:copy-readtable}

\begin{lstlisting}[style=lispinline]
(defun el-reader/copy-readtable
    (&optional from-readtable to-readtable))
\end{lstlisting}

This function copies the contents of one table to another.  If no source is
given (or is \nil{}), the current readtable is used.  If no destination is given
(or is \nil{}), a new table is allocated.  In either case, the table where data
was copied into is returned.

\subsubsection{getch}
\label{subsubsec:getch}

\begin{lstlisting}[style=lispinline]
(cl-defgeneric el-reader/getch (stream &optional char))
\end{lstlisting}

When methods of this function are called with only a stream, a character is
returned from that stream.  If the end of the stream is reached, it signals an
error of type \sym{end-of-file}.

If a method is called with an additional argument, it shall be a character.
This character is put back onto the stream.  Methods may or may not allow
putting back more than one character, or putting back characters which were not
read from the stream, yet a read macro may never assume this is valid!

\subsubsection{peek-char}
\label{subsubsec:peek-char}

\begin{lstlisting}[style=lispinline]
(cl-defgeneric el-reader/peek-char (stream))
\end{lstlisting}

Works as \fun{getch}, yet leaves the character on the stream.  This is a generic
function, i.e. it may be specialized for other data types.  The default behavior
calls \fun{getch} twice---once without an argument, then again with the return
value of the first call as an argument.  Hence, a character is read and put back
again.  For most types this is not a problem, yet for streams which---for
whatever reason---cannot unread more than one character, it may be necessary to
define a method specializing on the stream type at hand.

\subsubsection{read}
\label{subsubsec:read}

\begin{lstlisting}[style=lispinline]
(cl-defun el-reader/read (&optional input-stream
                                    (eof-error-p t)
                                    eof-value
                                    recursive-p
                                    keys))
\end{lstlisting}

This function is the main entry point to the reader.  As it’s name implies, it
reads code.  Namely, it reads one expression from a given (or implied) input
source.  User may call this function directly, yet for many programs this is not
done, as the reader is invoked by the Lisp environment by loading code.  Other
use cases were discussed in section~\ref{subsec:sexp-communication}.

Users who wish to implement their own read macros on the other hand, have to use
\Read{} directly, as well as previously discussed functions to assign a function
to a macro character.

A call to \Read{} does not require any arguments, yet those which may be given
are discussed here.

\begin{description}
\item[input-stream] This is an object on which a method for the generic function
  \fun{get-getch-state} is defined, and which returns an object for which a
  method for the generic function \fun{getch} is defined.
\item[eof-error-p] This flag indicates whether encountering the end of the input
  stream should raise an error or return a specific value.  The default is to
  raise an error. 
\item[eof-value] If \sym{eof-error-p} is non-\nil{}, this value will be
  returned.  It defaults to \nil{}.
\item[recursive-p] This should be true if \Read{} is called inside another read
  call, i.e. when it is a recursive call.  Why it is important for \Read{} to
  know this is discussed in section~\ref{subsec:circular-datastructures}.  As a
  rule of thumb one can say that this argument shall always be non-\nil{} if
  called from within a read macro and \nil{} if done with no reference to
  macros.

  Note that a call may be recursive, have \sym{recursive-p} set to \nil{} and
  still be valid: if the user needs to read from a stream (possibly a different
  one than is currently being read from) while processing a read macro.  In such
  a case a call to read is recursive in the sense that there are at the time of
  calling \Read{} already other calls to \Read{} on the call stack.  On the
  other hand, it has nothing to do with those parent calls, and should work
  independently of those.
\item[keys] This is only used internally---users should never supply a
  non-\nil{} value!  The only reason this exists, is because \el{}---in contrast
  to \cl{}---does not support returning multiple values.
\end{description}

\subsubsection{read-preserving-whitespace}
\label{subsubsec:read-preserving-whitespace}

\begin{lstlisting}[style=lispinline]
(cl-defun el-reader/read-preserving-whitespace
    (&optional input-stream (eof-error-p t) eof-value
               recursive-p))
\end{lstlisting}

This function works in the same way as \Read{}, with one difference.  When done
reading an expression which ends with whitespace, \Read{} consumes a whitespace
character, yet does not put it back.  This function does this.

% It is implemented by creating a new binding of the variable
% \sym{*preserve-whitespace*} (see section ~\ref{subsec:preserve-whitespace}) to
% \tee{} before calling \Read{}.
Creating a new binding of the variable \sym{*preserve-whitespace*} (see
section~\ref{subsubsec:preserve-whitespace}) to \tee{} before calling \Read{} is
sufficient to implement this function.  It only does so if not called as a
recursive call.  This brings us to the next section: variables.

\subsection{Variables}
\label{subsec:variables}

Besides functions, there are also variables which influence the behavior of
several functions.  It is important to note that all of these variables may be
rebound at the stack level, hence a caller may influence not only the direct
callee, but all further callees.

\subsubsection{*readtable*}
\label{subsubsec:readtable-var}

This variable holds the \texttt{current} readtable.  Some functions allow the
caller to pass in a readtable as an argument, some don’t.  In either case, the
default is this variable.

\subsubsection{*read-base*}
\label{subsubsec:read-base}

When no extra construct is used, integers are read in base 10.  This is because
the default value for \sym{*read-base*} is 10, and the reader always uses this
variable to determine which base to use.  Thus reading hex numbers may be done
in three different ways:

\begin{lstlisting}[style=lispinline]
\#xF
\#16rF
(let ((el-reader/*read-base* 16))
  (el-reader/read "F"))
\end{lstlisting}

All of the above expressions return the number \(15_{10}\).

\subsubsection{preserve-whitespace}
\label{subsubsec:preserve-whitespace}

This variable controls whether \Read{} leaves whitespace at the end of an
expression intact or not.  The default is \nil{}.

\section{Differences to \cl{}}
\label{sec:cl-differences}

While \elr{} takes inspiration from the reader used in \cl{}, it is not exactly
the same.  Some of this is by design, to slightly improve the reader, but some
is due to deficiencies in \el{}.  Some are due to idiosyncrasies of either \el{}
or \cl{}.

\subsection{Improvements}
\label{subsec:improvements}

The API of \elr{} is a little more flexible than the reader standardized for
\cl{}.\footnote{While it should be allowed for a \cl{} implementation to allow a
more flexible reader, such extensions do not at all seem to be very common.}
There are two differences, which were both already discussed in previous
sections.

\subsubsection{Setting syntax and traits}
\label{subsubsec:set-syntax-traits}

As the readtable in \elr{} is not opaque it is possible for a user to set the
syntax types of characters more flexibly than in \cl{}.  In \cl{} the user must
first find a character which has the desired syntax type and may copy the type
from said character to a desired other one.  This may be difficult if the user
wishes to make characters invalid (\elr{} for instance does not define any
characters to be invalid).  A user may either modify an existing table directly
(although this should go through a function in future versions of this reader),
or pass the character(s) in question to the appropriate fields of
\fun{make-instance}.

While \elr{} doesn’t go further with traits, it does go much further than \cl{}
does: in \elr{}, traits can be set the same way as syntax types, yet \cl{} does
not allow traits to be set at all.

\subsubsection{Mapping characters to numbers}
\label{subsubsec:chars-to-nums}

Both in \el{} and \cl{} it is possible to read numbers in different bases,
namely from 2 to 36.  The maximum is 36 because the code for the character ‘Z’
is 35 (i.e. 0-9, A-Z).  In \cl{} the mapping of characters to numbers is fixed,
which is not generally a problem.  In \elr{} this mapping is not fixed.
Instead, the readtable holds a hash table which provides a mapping from
characters to numbers.

A user can achieve a different mapping by adjusting which characters are
alphadigits (a trait) in the first place, and by changing the mapping from
characters to numbers.  So, if the user so desires, a ‘\(\lambda{}\)’ could be a
number.

\subsection{Differences due to deficiencies in \el{}}
\label{subsec:deficiency-diffs}

Sadly, \el{} lacks some features of \cl{}.  While the support for multimethods
is not nearly as thorough as in \cl{}, it is sufficient for the purposes of this
project.  Other features lacking on the other hand, do cause problems.

\subsubsection{Return values of read macro procedures}
\label{subsubsec:ret-vals}

The function associated with a read macro reads from the stream and constructs a
Lisp object from the read data.  This object is returned from the function.
While this sounds very straightforward, the question arises what a function
shall do if no object is created.  This does not necessarily be an error!  One
particular case is the syntax for comments.  The ‘;’ character is a macro
character which reads and ignores characters from the stream until the end of
the line is reached.  This means that no value is produced, hence no value can
be returned.  In \cl{} a function may return multiple values.  This may be a bit
of a misnomer, because it not only means that a function may return more than
one object, it may return more than \emph{zero} objects!  This feature is used
in \cl{} to cope with the fact that some read macros do not produce lisp
objects.  For \elr{} the obvious solution was to simply return a list, which may
be a singleton list, or an empty list.  The disadvantage is that users must
remember to always wrap their code either in a \fun{list} or \fun{cl-values}
form.\footnote{The function \fun{cl-values} comes from the \sym{CL} package in
  \emacs{} which tries to provide emulation for some \cl{} features missing in
  \el{}.}

The advantage of \cl{}’s multiple values support is that returning a single
value does not require special syntax on behalf of the user.  Also, a user may
simply ignore any excess values a function returns.  In such a case only the
\emph{primary value} is used.

This is the reason why read macros written in \cl{} simply return a value, while
for \elr{} the return value must be wrapped in a list.

\subsubsection{Package support}
\label{subsubsec:package-support}

In \cl{} the colon has a special meaning in symbol read syntax.  If the first
character of a token is a colon, it is interned into the keywords package, and
has itself as its value cell.  In \el{} only the latter is the case.

If a symbol contains a colon somewhere in the middle, the first part is used as
a package qualifier, while the second denotes a symbol in said package.  Also a
double colon may be used for non-exported symbols.

The exact details of \cl{}’s package system are beyond the scope of this thesis,
let alone this section.  Suffice to say that \el{} cannot support this behavior,
at least not in its present state.  This is because symbols form a linked list
inside of an obarray bucket.  This is discussed in
section~\ref{subsubsec:symbols} in more detail.  As symbols may not exist in
more than one obarray, package support can not be implemented by the reader.
Hence another difference with \cl{}.

\subsubsection{Support for fractions}
\label{subsubsec:fractions}

\cl{} not only has a datatype for fractions, it also provides read and print
syntax for them.  In \elr{} no read support is offered, simply because \el{} has
no support for such numbers.

\subsection{Idiosyncrasies}
\label{subsec:idiosyncrasies}

Some differences are neither obviously good or bad, but simply a little
different.

\subsubsection{Non-terminating dispatching macro character (‘\#’)}
\label{subsubsec:non-term-dispatch}

In \cl{}, the ’\#’ denotes a non-terminating dispatching macro character, while
in \elr{} it is---by default---a terminating dispatching macro character.  This
only because this is the default behavior in \el{}, and may be changed by users.

\subsubsection{Case conversion}
\label{subsubsec:case-conversion}

When the \cl{} reader reads a symbol, the default behavior is to convert the
characters which make up the name of the symbol to upper case.  In \cl{} this
may be customized on a per-readtable basis, yet this was not incorporated into
\elr{} as this was never done in \el{}.

\subsubsection{Name and signature differences}
\label{subsubsec:name-diffs}

The functions \fun{getch} and \fun{peek-char} have different names and
signatures in \cl{}.

\chapter{Future work}
\label{chapter:future-work}

The code presented here is merely a prototype, a proof of concept.  It is
\emph{not} ready for distribution to the masses.  There are several things
missing as is discussed in the next section.

Section~\ref{sec:improve-situation} discusses some possibilities to improve
\elr{}, especially as its performance is appallingly bad.

\section{What is missing?}
\label{sec:missing}

As with all language implementations and parts of those, one is never finished.
There will always be more to do.

Currently, not all constructs can be read with \elr{}.  This is not because of
some fundamental flaw, yet simply because not every syntactic construct of
standard \el{} has been implemented yet.  This can be done, yet requires some
more time, and has to be done before releasing code for use by the public.

Because of this, it is currently not possible to run code which was compiled
using \elr{}.  This is because \emacs{}---quite elegantly---reads bytecode with
the reader.  Standard emacs provides a type named byte-vector, which has read
syntax. \footnote{See section 16.7 “Byte-Code Function Objects” of the \el{}
  reference info file.\cite{elisp-reference}} Also, \emacs{} has at least one
idiosyncrasy which prevented a straightforward use, which is discussed in
section~\ref{subsubsec:multimethods-streams}.

\section{What can be done to improve the situation?}
\label{sec:improve-situation}

The most grievous problem for usability (apart from the aforementioned lack of
complete coverage of standard \el{}) is bad performance.  On a rather old laptop
containing a Core i3 with two cores clocked at 1.33GHz and Hyperthreading,
reading the source code of the reader (less than 2000 lines) with the byte
compiled reader takes around 20 seconds, while doing the same with the built in
reader is done practically instantly.

There are at least three ways to address this problem, one of which has nothing
to do with \elr{}:

\subsection{Improve \emacs{}’ performance}
\label{subsec:impr-emacs-perf}

One obvious, yet nontrivial way around the performance problem is to improve
\emacs{}’ performance.  This would not only benefit \elr{}, but also some other
packages, like the Gnus mail and newsreader.\cite{gnus}  The problem is of
course that this is something virtually every user would like, yet still doesn’t
happen as fast and as much as desired, as it is not exactly an easy task.

There are at least two ways this could be addressed: using a faster bytecode VM,
or using a JIT (Just In Time compiler) to compile to native code.

The Guile Emacs project\cite{guile-emacs}\footnote{Which does not seem to have a
  proper website, only a git repository which does not seem to be very active.}
is an attempt to use the Guile\cite{guile} VM instead of \emacs{}’ VM, yet
progress seems to be very slow.  Another advantage to this approach would be to
allow writing extensions in scheme, in addition to \el{}.

There seems to be another rather undocumented effort to replace \emacs{}’ VM
with an LLVM\cite{llvm} based JIT which can be found at \cite{emacs-llvm}

\subsection{Rewrite \elr{} in C as part of \emacs{}}
\label{subsec:rewrite-in-C}

Having finished a prototype, it is a possibility to port \elr{} to C, completely
replacing the built in reader.  This would obviously need the approval of the
\emacs{} maintainers in order to be properly distributable.

The disadvantage is that one would have to rewrite the whole reader in one go,
there is no usable state between having just started, and having completed it.
Which brings us to the next section.

\subsection{Write a C module}
\label{subsec:C-module}

Writing a C module may be a good way to enhance the performance of \elr{}
without leaving it in an unusable state.  It should be possible to rewrite
\elr{} one function at a time.  Each function used in \elr{} could be replaced
by a lisp callable procedure written in C, thus allowing one to concentrate on
the performance hungry portions first.  The problem with this approach is that
the module system is not (yet) very well documented and that the module system
is considered experimental.

% \section{The readtable}
% \label{subsec:readtable}

% The only user visible data structure is the readtable.  In \cl{} this type has
% non-enumerable types, while in \elr{} it is defined in terms of
% \fun{defclass}, hence is enumerable and even has public attributes.  Most users
% will need none of these.  They are mostly needed when creating a completely new
% readtable from scratch.

% \section{Functions}
% \label{subsec:functions}

% Here be dragons.

\vfill
\pagebreak
\bibliographystyle{abbrvdin}
\bibliography{references}

\end{document}