Add ES2023 tokenization

[arhadthedev]

For starters, it can be Tokenizer struct right inside the crate.

The struct should recognize lexical grammar described in https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar.

The struct should provide methods to switch between goal symbols on the fly. The spec section linked above gives the following reasoning for such a feature:

There are several situations where the identification of lexical input elements is sensitive to the syntactic grammar context that is consuming the input elements. This requires multiple goal symbols for the lexical grammar. The InputElementRegExpOrTemplateTail goal is used in syntactic grammar contexts where a RegularExpressionLiteral, a TemplateMiddle, or a TemplateTail is permitted.

Syntactic grammar contexts is defined by a parser state so it's the parser that needs to switch a current lexical grammar goal symbol to adjust tokenization.

The user entry point is Tokenizer object with its get_next_symbol method:

/// From <https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar>:
///
/// > There are several situations where the identification of lexical input
/// > elements is sensitive to the syntactic grammar context that is consuming
/// > the input elements. This requires multiple goal symbols for the lexical
/// > grammar.
pub enum GoalSymbols {
    /// From <https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar>:
    ///
    /// > Used at the start of a Script or Module
    InputElementHashbangOrRegExp,

    /// From <https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar>:
    ///
    /// > Used in syntactic grammar contexts where a `RegularExpressionLiteral`,
    /// > a `TemplateMiddle`, or a `TemplateTail` is permitted.
    InputElementRegExpOrTemplateTail,

    /// From <https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar>:
    ///
    /// > Used in all syntactic grammar contexts where a
    /// > `RegularExpressionLiteral` is permitted but neither a
    /// > `TemplateMiddle`, nor a TemplateTail is permitted.
    InputElementRegExp,

    /// From <https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar>:
    ///
    /// > Used in all syntactic grammar contexts where a `TemplateMiddle` or a
    /// > `TemplateTail` is permitted but a `RegularExpressionLiteral` is
    /// > not permitted
    InputElementTemplateTail,

    /// From <https://262.ecma-international.org/14.0/#sec-ecmascript-language-lexical-grammar>:
    ///
    /// > In all other contexts, [...] used as the lexical goal symbol.
    InputElementDiv
}

#[derive(Debug)]
pub struct Tokenizer {
    /// A current set of lexical grammar
    current_goal: GoalSymbols,
}

Grammar parameter implementation

For notation like Production[Param1, Param2] add the following comment into the code:

Notes on how specification features are mapped into YACC features:

• grammatical parameters (like Nonterminal[Param1, Param2]) are implemented as a special FecerBrowser_...KeywordUsage static semantics returning a boolean, one for each pair of a left side parameter and its right side usage
• lookahead restriction
• [No Line Terminator Here]
• automatic semicolon insertion

Cover grammar

Also add a note on what cover grammar is.

Todo list:

• Add ES2023 whitespace and line terminator tokenization #12
• Add ES2023 hashbang and common comment tokenization #13
• Add ES2023 common token tokenization #14
• Add ES2023 names tokenization #15
• Add ES2023 punctuators tokenization #16
• Add ES2023 keyword literal tokenization #17
• Add ES2023 numeric literal tokenization #18
• Add ES2023 string literal tokenization #19
• Add ES2023 regex literal tokenization #20
• Add ES2023 template literal tokenization #21