Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use function/const not tokenized as T_STRING when preceded by comment #2431

Closed
jrfnl opened this issue Feb 23, 2019 · 1 comment
Closed

Use function/const not tokenized as T_STRING when preceded by comment #2431

jrfnl opened this issue Feb 23, 2019 · 1 comment

Comments

@jrfnl
Copy link
Contributor

jrfnl commented Feb 23, 2019

The function and const keywords used in use statements are normally tokenized as T_STRING.

However, when there is a comment between the use keyword and the function/const keyword, this fails and they are tokenized as T_FUNCTION/T_CONST.

Example code where the tokenizer correction fails:

use /* comment */ function bar\math\{
    Msin,
    level\Mcos as BarCos,
    Mcosh,
};

use // phpcs:ignore Standard.Category
    const
	    bar\math\{ BGAMMA as BAR_GAMMA, BGOLDEN_RATIO };

Relevant code in the Tokenizer:

// This is a special case for PHP 5.6 use function and use const
// where "function" and "const" should be T_STRING instead of T_FUNCTION
// and T_CONST.
if (($newToken['code'] === T_FUNCTION
|| $newToken['code'] === T_CONST)
&& ($finalTokens[$lastNotEmptyToken]['code'] === T_USE || $insideUseGroup === true)
) {
$newToken['code'] = T_STRING;
$newToken['type'] = 'T_STRING';
}

As that code appears to look at the $lastNonEmptyToken, I'm not sure how to fix this....

@gsherwood gsherwood added this to Backlog in PHPCS v3 Development via automation Feb 27, 2019
@gsherwood gsherwood added this to the 3.4.1 milestone Feb 27, 2019
@gsherwood
Copy link
Member

$lastNonEmptyToken was only used to store the last non-whitespace token. Changing this store the last non-empty token as per the token util class gets this working and doesn't look to break anything.

@gsherwood gsherwood changed the title PHP Tokenizer bug: use function/const not always tokenized as T_STRING Use function/const not tokenized as T_STRING when preceded by comment Feb 27, 2019
gsherwood added a commit that referenced this issue Feb 27, 2019
PHPCS v3 Development automation moved this from Backlog to Ready for Release Feb 27, 2019
jrfnl added a commit to PHPCompatibility/PHPCompatibility that referenced this issue Aug 9, 2020
As noted inline:
> In PHPCS < 3.4.1, the class keyword after a double colon + comment may be tokenized as
> `T_CLASS` instead of as `T_STRING`, so registering both.

Adjusted an existing unit test to safeguard this change.

Upstream issue about the tokenizer problem:
* squizlabs/PHP_CodeSniffer#2431
jrfnl added a commit to PHPCompatibility/PHPCompatibility that referenced this issue Aug 10, 2020
As noted inline:
> In PHPCS < 3.4.1, the class keyword after a double colon + comment may be tokenized as
> `T_CLASS` instead of as `T_STRING`, so registering both.

Adjusted an existing unit test to safeguard this change.

Upstream issue about the tokenizer problem:
* squizlabs/PHP_CodeSniffer#2431
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
PHPCS v3 Development
Ready for Release
Development

No branches or pull requests

2 participants