[doc] Describe the doc toolchain, with examples.

Fix lint error too.

doc/doc-plugins.md

@@ -0,0 +1,81 @@
+---
+in_progress: yes
+default_highlighter: oil-sh
+---
+
+Examples of HTML Plugins
+========================
+
+This file is a sort of unit test for [doctools/]($oil-src).
+
+<div id="toc">
+</div>
+
+## Link Shortcuts with `$`
+
+- `$xref`: [bash]($xref)
+- `$blog-tag`: [oil-release]($blog-tag)
+- `$oil-src`: [INSTALL.txt]($oil-src), [INSTALL.txt]($oil-src:INSTALL.txt)
+- `$blog-code-src`: [interactive-shell/README.md]($blog-code-src)
+- `$issue`: [issue 11]($issue:11)
+- `$oil-commit`: [this commit]($oil-commit:a1dad10d53b1fb94a164888d9ec277249ae98b58)
+
+
+## Syntax Highlighting With Markdown Code Blocks
+
+### sh-prompt
+
+```sh-prompt
+$ echo hi   # comment
+hi
+```
+
+### Pygments
+
+```python
+x = 42
+print(x, file=sys.stderr)
+```
+
+### `default_highlighter` in Front matter
+
+`oil-sh` is a generic formatter that works for both shell and Oil code.  This
+is what we use in [idioms.html](idioms.html).
+
+No:
+
+    pat='*.py'         # pattern stored in a string
+    echo $pat          # implicit glob in shell
+
+Yes:
+
+    var pat = '*.py'   # Oil assignment
+    echo @glob(pat)    # explicit
+
+### Plugins We Should Have
+
+- Side-by-side sh and Oil
+- Side-by-side PCRE and Eggex
+- sh-session - How to replace the data?
+
+A shell session could look like this:
+
+<div shell="sh">
+
+```
+$ echo one
+$ echo two
+```
+
+</div>
+
+Embeddings:
+
+- Embed Image Preview of Web Page?
+- Embed Github Commit?
+- Graphviz
+  - LaTeX (although I don't really use it)
+
+## Related Docs
+
+- [doc-toolchain](doc-toolchain.html)

doc/doc-toolchain.md

@@ -2,128 +2,126 @@
 in_progress: yes
 ---
 
-Oil Documentation Toolchain
-===========================
+How We Build Oil's Documentation
+================================
 
-The toolchain is based on hand-written Markdown and HTML filters.  See
-[lazylex/README.md]().
+1. Write Markdown by hand, with optional "front matter".
+2. Render Markdown to HTML, and run the result through our own HTML filters.
+3. Publish static HTML to <https://www.oilshell.org/>.
+
+The code is in the [doctools/]($oil-src) directory, which uses the
+[lazylex/]($oil-src)  library.
 
 <div id="toc">
 </div>
 
-## Plugins
-
-### Link Shortcuts
+## Quick Start
 
-This markdown:
+To build and preview this doc, run:
 
-```
-The [GNU bash shell]($xref:bash)
-```
+    build/doc.sh split-and-render doc/doc-toolchain.md
 
-is translated to this HTML:
+Open the path in prints in your browser
+(`_release/VERSION/doc/doc-toolchain.html`).
 
-```
-The <a href="$xref:bash">GNU bash shell</a>
-```
+## Front Matter and Title
 
-which is expanded by our plugin to:
+Most docs start with something like this:
 
-```
-The <a href="/cross-ref.html#bash">GNU bash shell</a>
-```
+    ---
+    in_progress: yes
+    default_highlighter: oil-sh
+    ---
 
-If the argument is omitted, then the anchor text is used.  For example,
+    My Title
+    ========
 
-```
-<a href="$xref">bash</a>
-```
+    Hello
 
-becomes:
+The "front matter" between `---` lines is metadata for rendering the doc.
+Github's web UI understands and renders it.
 
-```
-The <a href="/cross-ref.html#bash">bash</a>
-```
+## Plugins That Transform HTML
 
-List of plugins:
+We have some HTML plugins that make writing **markdown** easier.
+Note that [CommonMark][] tightens up the rules for embedding HTML in Markdown,
+and that is very useful.
 
-- `$xref:bash` expands to `/cross-ref.html#bash` (shown above)
-- `$blog-tag:oil-release` expands to `/blog/tags.html#oil-release`
+[CommonMark]: https://www.oilshell.org/blog/2018/02/14.html 
 
+### Table of Contents
 
-[bash]($xref)
+Insert this into the doc
 
-[oil-release]($blog-tag)
+    <div id="toc">
+    </div>
 
-[INSTALL.txt]($oil-src)
+and it will be expanded into a table of contents derived from `h2` and `h3`
+tags.
 
-[INSTALL.txt]($oil-src:INSTALL.txt)
+### Link Shortcuts, e.g. `$xref`
 
-[interactive-shell/README.md]($blog-code-src)
+Here's an example of how it works.  This Markdown:
 
-[issue 11]($issue:11)
+    The [GNU bash shell]($xref:bash)
 
-[this commit]($oil-commit:a1dad10d53b1fb94a164888d9ec277249ae98b58)
+is translated to HTML by [CommonMark][]:
 
-### Syntax Highlighting
+    The <a href="$xref:bash">GNU bash shell</a>
 
+Our `$xref:` plugin expands it to:
 
-Plugins:
+    The <a href="/cross-ref.html#bash">GNU bash shell</a>
 
-`sh-prompt`:
+If the argument is omitted, then the **anchor text** is used.  So you can just write:
 
-``` sh-prompt
-oil$ var x = 'hello world'
-oil$ echo $x
-hello world
-```
+    [bash][]
 
-`python`:
+and it will become:
 
-``` python
-x = 42  # comment
+    The <a href="/cross-ref.html#bash">bash</a>
 
-def add(y):
-  return x + y
+List of plugins:
 
-print(add(x * 7))
-```
+- `$xref:bash` expands to `/cross-ref.html#bash` (shown above)
+- `$blog-tag:oil-release` expands to `/blog/tags.html#oil-release`
+- `$oil-src`
 
+See the raw and rendered versions of this doc for more:
 
-### TODO:
+- [doc-plugins.md][]
+- [doc-plugins.html](doc-plugins.html)
 
-- Side-by-side sh and Oil
-- Side-by-side PCRE and Eggex
-- sh-session - How to replace the data?
+[doc-plugins.md]: $oil-src:doc/doc-plugins.md
 
+### Syntax Highlighting of Code Blocks
 
-A shell session could look like this:
+Use Markdown's fenced code blocks like this:
 
-<div shell="sh">
+    ``` sh-prompt
+    oil$ var x = 'hello world'
+    oil$ echo $x
+    hello world
+    ```
 
-```
-$ echo one
-$ echo two
-```
+Or you can set `default_highlighter` for blocks indented by 4 sapces.
 
-</div>
+Again see [doc-plugins.md][] for examples.
 
-Embeddings:
+## The Help Toolchain Renders to HTML and ANSI
 
-- Embed Image Preview of Web Page?
-- Embed Github Commit?
-- Graphviz
-  - LaTeX (although I don't really use it)
+TODO: Describe how this works.  We have a plugin for the table of contents.
 
+It's published to the web as well as underlying the `help` builtin.  TODO:
+Render to ANSI.
 
-## Help Toolchain
+We also split `{osh,oil}-help.md` into "cards".
 
-- It splits `help-index.md` and `help.md` into "cards"
-- TODO: Render to ANSI
+Table of contents: `{osh,oil}-help-topics.md`.
 
 ## Code Location
 
-Right now there are several scripts in `devtools/*.py`.  And `build/doc.sh`.
+- [build/doc.sh]($oil-src) drives the tools in [doctools/]($oil-src).
+- Markdown files are in [doc/]($oil-src).
 
-TODO: Move into `doc/`?
 

doc/framing.md

@@ -15,7 +15,7 @@ contexts.
 <div id="toc">
 </div>
 
-## Delimiter Based
+## Delimiter Based Solutions
 
 ### Newline as Delimiter
 
@@ -30,24 +30,25 @@ contain newlines, but not `NUL` (`\0`).
 So naturally we also support the the format that `find -print0` emits, and
 `xargs -0` consumes.
 
-## Quoting / Escaping Special characters: [QSN][]
+## Solutions That Can Express Arbitrary Bytes
 
-Quoted String Notation
+### Quoting / Escaping Special characters: [QSN][]
 
-TODO: Describe this.
+QSN uses backslash escapes like `\n`, `\x00`, and `\u{3bc}` to express
+arbitrary bytes (and characters).
 
-## Length-Prefixed: Netstrings
+### Length-Prefixed: Netstrings
 
-TODO: Implement this
+TODO: Implement this.
 
 Like [QSN][], this format is "8-bit clean", but:
 
 - A netstring encoder is easier to write than a QSN encoder.  This may be
   useful if you don't have a library handy.
 - It's more efficient to decode, in theory.
 
-However the encoded payload may contain binary data, so it can't be viewed in
-a terminal, in general.
+However, the encoded output may contain binary data, which is hard to view in a
+terminal.
 
 [QSN]: qsn.html
 

doc/index.md

@@ -27,7 +27,7 @@ language.  It's **our upgrade path from bash**.
 - [Known Differences](known-differences.html) is trivia for advanced users.
   It lists differences between OSH and other shells.
 - [Quirks](quirks.html) for compatibility.
-- [errexit](errexit.html) (in progress)
+- [Error Handling with `errexit`](errexit.html)
 - [OSH Help Topics](osh-help-topics.html) (incomplete).  This document
   underlies the `help` builtin.
 
@@ -79,6 +79,7 @@ More:
 - [JSON](json.html): Currently supported only in the Python prototype of Oil.
 - [QSN](qsn.html): Quoted String Notation.  A human- and machine-readable
   format for byte strings.
+  - [Framing](framing.html)
 - [QTSV](qtsv.html): An extension of TSV, built on top of QSN.
 - [Unicode](unicode.html).  Oil supports and prefers UTF-8.
 

doctools/README.md

@@ -0,0 +1,14 @@
+Doctools
+========
+
+Tools we use to generate [Oil's documentation](../doc/).  Some of this code is
+used to build the [the blog](//www.oilshell.org/blog/) as well.
+
+See [doc/doc-toolchain.md](../doc/doc-toolchain.md) for details.
+
+- `cmark.py`: Our wrapper around CommonMark.
+- `html_head.py`: Common HTML fragments.
+- `oil_doc.py`: HTML filters.
+- `split_doc.py`: Split "front matter" from Markdown.
+- `make_help.py`: For `doc/{osh,oil}-help-topics.html`.
+

oil_lang/expr_eval.py

@@ -17,7 +17,6 @@
 from oil_lang import objects
 from osh import braces
 from osh import word_compile
-from osh import word_eval
 
 import libc