Lexical Feedback

[marcioAlmada]

This is related to the "Context Sensitive Lexer" RFC: https://wiki.php.net/rfc/context_sensitive_lexer. This new W.I.P patch is an alternative implementation of #1054.

About this new implementation:

• Has no regression | forward compatibility risks and is highly predictable
• Has an very small footprint compared to the previous attempt involving a pure lexical approach
• Is highly configurable, to make a word semi-reserved you only have to edit this inclusive parser rule.
• Doesn't require compile time check

Tasks

• Pass all tests
• Keep %expect equals to 2
• Port ext tokenizer
  • 30%
  • 60%
  • 90%
  • 100%
• Review

[ghost]

Can one of the admins verify this patch?

[nikic]

Why not handle property names with the same system?

[marcioAlmada]

Because property names have sigils and therefore no naming limitations, if you handle it the same way as the other member names they will be restricted by the inclusive SEMI_RESERVED rule https://github.com/php/php-src/pull/1158/files#diff-7eff82c2c5b45db512a9dc49fb990bb8R273

I'm sure you don't want that.

[nikic]

Good point, I forgot that method names have a more limited set of allowed keywords :)

[marcioAlmada]

BTW, I found a way to uncomment https://github.com/php/php-src/pull/1158/files#diff-7eff82c2c5b45db512a9dc49fb990bb8R280 and have these words semi reserved too, but that would require that people aliasing methods that clash with method modifiers names to use the verbose syntax. Ex:

trait TraitA {
   private function private(){}
}

Class A {
    use TraitA {
        private as protected protected;
    }
}

I'm leaving this out of the patch though, as it has to be compatible with the older one.

[dstogov]

It should be possible to get the string representation of keyword without "rewind" and "rematch".
If something like the following may work, you won't need to change lexer at all.

identifier:
T_STRING { $$ = $1; }
| SEMI_RESERVED { zval zv; ZVAL_STRINGL(&zv, LANG_SCNG(yy_text), LANG_SCNG(yy_leng)); $$ = zend_ast_create_zval(&zv); }
;

[marcioAlmada]

Looks like a great idea because we don't waste time interacting with the lexer twice anymore. On the other side I was doing the ext tokenizer port based on this lexical feedback and now the lexer is completely unaware of context again.

I'll use this anyway and will try to find another solution on the tokenizer extension side again. Thanks ^^

[marcioAlmada]

Just a heads up: this PR is still active and soon we'll have updates.

[marcioAlmada]

FYI, I'll be waiting for the anonymous classes patch to be merged before continue.

[jrnickell]

@marcioAlmada will this support methods named empty()? It looks like you have empty included in the tokenizer, but I don't see it listed in the RFC or your tests here.

[marcioAlmada]

@jrnickell absolutely. Looks like I forgot empty on the tests. The following works:

php -r "class X { static function empty(){ return true; } } var_dump(X::empty());"
# bool(true)
php -r "class X { function empty(){ return true; } } var_dump((new X)->empty());"
# bool(true)
php -r "class X { const EMPTY = true; } var_dump(X::EMPTY);"
# bool(true)

Good catch, the tests will be updated soon.

[marcioAlmada]

@jrnickell 66c17c0 ✅

[jrnickell]

@marcioAlmada awesome! Thanks

[marcioAlmada]

I'm also working on this other experiment #1221 as we still have some time.

[marcioAlmada]

Closed in favor of #1221.