Factor lexer logic out into reusable state machine, improve ergonomics

Factor lexer logic out into reusable state machine, improve ergonomics

The biggest change here is factoring out the lexical analysis logic out of BufLexer and into lex::mach::Mach, a reusable lexical analysis state machine that should be able to provide all the core logic for the other expected Lexer implementations: synchronous reading from an io::Read and asynchronous polling.

There are also made a bunch of ergonomics changes that should hopefully make the user experience easier, including:

• Lexer::next() now returns a simple lex::Token
• lex::Token enumerates Eof and Err as special cases
• Lexer::value() now returns a Result that is not wrapped in an Option.

Now BufLexer::next() should continue returning an infinite stream of Eof at end of file and an infinite stream of Err if it hits a lexing error.