Privatize the Lexer API

[djc]

Becomes an implementation detail of the Parser.

[bluejekyll]

There are a bunch of changes to the state machine for the parsing, do you think you could describe some of the background there?

[djc]

There are a bunch of changes to the state machine for the parsing, do you think you could describe some of the background there?

Sure. I usually write commits with the intention that they can be reviewed commit-by-commit. The changes to the parser state machine are from "handle includes in the parser". Previously there were some TODOs in the lexer:

    /// TODO: it looks hacky as far we effectively duplicate parser's functionality
    /// (at least partially) and performing lexing twice.
    /// Better solution requires us to change lexer to deal
    /// with Lines-like iterator instead of String buf (or capability to combine a few
    /// lexer instances into a single lexer).
    ///
    /// TODO: $INCLUDE could specify domain name -- to support on-flight swap for Origin
    /// value we definitely need to rethink and rework loader/parser/lexer

This basically tries to address some of those issues by having the Parser hold a stack of Lexers. When an $INCLUDE is encountered, we push the stack; when a lexer runs to completion, we pop the stack. Because the parser takes its data by &str but for the case of includes we own the string inside the parser/lexer stack, the Lexer now also has to abstract over String vs &str input, which is where the CowChars type comes in.

Let me know if you have more questions.

(I wrote this code about 2 months ago so I no longer remember all the details.)

[bluejekyll]

Everything looks good. Thanks for cleaning all of this up.