Support for custom implementations of Token patterns.

[bd82]

Currently a Token's pattern is defined using regular expressions.

let IntegerToken = createToken({name: "IntegerToken", pattern: /\d+/})

However in some circumstances the capability to provide a custom pattern matching implementation may be required. Perhaps a special Token which cannot be easily defined using regular expressions, or perhaps
to enable solving performance problems in a specific RegularExpression engine, for example:

• WebKit/Safari multiple orders of magnitude performance degradation for specific regExp patterns 😞

[bd82]

re-opening to remind myself to update to performance page (when running on safari)

[bd82]

Docs can be found here:
https://github.com/SAP/chevrotain/blob/master/docs/custom_token_patterns.md