Add a way to override the lexer in a Parser instance #317
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, it's possible to do,
but would require a lot of code duplication and pinning to an exact
version of tolerant-php-parser, which discourages overriding the lexer.
I can think of the following use cases for this.
My main reason is to reuse token_get_all() elsewhere,
but being able to parse
T_FNin php < 7.4 (for short arrow functions) is also convenient.Multiple applications needing to use the result of token_get_all
for the same file. If none of them modify the array,
it's much faster to reuse the same array than to create this. (benchmarking, the time savings aren't as large as I thought, but there might be some memory savings for reusing the strings in the array)
For example, Phan's language server mode will potentially use
tolerant-php-parser. In addition to that, it also uses token_get_all
in InlineHTMLPlugin (to check for misuse of inline HTML) and sometimes
in BuiltinSuppressionPlugin
(to list T_COMMENT/T_DOC_COMMENT containing
@phan-suppress-*)Aside: https://wiki.php.net/rfc/token_as_object would be faster and
more memory efficient than token_get_all() in php 8.0, if it gets approved
Needing to call tolerant-php-parser on the same token stream,
multiple times.
(e.g. an application that modifies the Microsoft\PhpParser\Node
instances (but not tokens), or which creates data structures from the
original Node but usually discards them to save memory)