Skip to content

Commit

Permalink
Tokenizer/PHP: efficiency improvement for DNF type handling
Browse files Browse the repository at this point in the history
The PHP::processAdditional()` method walks _back_ from the end of the file to the beginning.

With that in mind, and knowing that a type can never end on an open parenthesis, and an open parenthesis can never be seen in a type before the close parenthesis has been seen (at least for valid/non-parse error types), it makes no sense to trigger the type handling logic for open parentheses.

This should make the tokenizer slightly more efficient as (open) parentheses are used a lot in code ;-)

It also prevents the type handling layer from acting on these type of invalid/parse error types, while it previously would.

Includes tests safeguarding the behaviour of the type handling layer for this type of invalid/parse error types.

Of these tests, the type for the OO constant and for the property were previously not handled consistently/correctly.
The parameter type + the return type were fine.
  • Loading branch information
jrfnl committed May 21, 2024
1 parent 027c0cb commit cac9038
Show file tree
Hide file tree
Showing 3 changed files with 86 additions and 1 deletion.
1 change: 0 additions & 1 deletion src/Tokenizers/PHP.php
Original file line number Diff line number Diff line change
Expand Up @@ -3036,7 +3036,6 @@ protected function processAdditional()
continue;
} else if ($this->tokens[$i]['code'] === T_BITWISE_OR
|| $this->tokens[$i]['code'] === T_BITWISE_AND
|| $this->tokens[$i]['code'] === T_OPEN_PARENTHESIS
|| $this->tokens[$i]['code'] === T_CLOSE_PARENTHESIS
) {
/*
Expand Down
17 changes: 17 additions & 0 deletions tests/Core/Tokenizer/PHP/DNFTypesParseError1Test.inc
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<?php

// Parentheses in broken DNF type declarations will remain tokenized as normal parentheses.
// This test is in a separate file as the 'nested_parenthesis' indexes will be off after this code.
class ParseErrors {
/* testBrokenConstDNFTypeEndOnOpenParenthesis */
const A|(B PARSE_ERROR = null;

/* testBrokenPropertyDNFTypeEndOnOpenParenthesis */
public A|(B $parseError;

function unmatchedParens {
/* testBrokenParamDNFTypeEndOnOpenParenthesis */
A|(B $parseError,
/* testBrokenReturnDNFTypeEndOnOpenParenthesis */
) : A|(B {}
}
69 changes: 69 additions & 0 deletions tests/Core/Tokenizer/PHP/DNFTypesParseError1Test.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
<?php
/**
* Tests that parentheses tokens are not converted to type parentheses tokens in broken DNF types.
*
* @author Juliette Reinders Folmer <phpcs_nospam@adviesenzo.nl>
* @copyright 2024 PHPCSStandards and contributors
* @license https://github.com/PHPCSStandards/PHP_CodeSniffer/blob/master/licence.txt BSD Licence
*/

namespace PHP_CodeSniffer\Tests\Core\Tokenizer\PHP;

use PHP_CodeSniffer\Tests\Core\Tokenizer\AbstractTokenizerTestCase;

final class DNFTypesParseError1Test extends AbstractTokenizerTestCase
{


/**
* Document handling for a DNF type / parse error where the last significant type specific token is an open parenthesis.
*
* @param string $testMarker The comment prefacing the target token.
*
* @dataProvider dataBrokenDNFTypeCantEndOnOpenParenthesis
* @covers PHP_CodeSniffer\Tokenizers\PHP::processAdditional
*
* @return void
*/
public function testBrokenDNFTypeCantEndOnOpenParenthesis($testMarker)
{
$tokens = $this->phpcsFile->getTokens();

$openPtr = $this->getTargetToken($testMarker, [T_OPEN_PARENTHESIS, T_TYPE_OPEN_PARENTHESIS], '(');
$token = $tokens[$openPtr];

// Verify that the open parenthesis is tokenized as a normal parenthesis.
$this->assertSame(T_OPEN_PARENTHESIS, $token['code'], 'Token tokenized as '.$token['type'].', not T_OPEN_PARENTHESIS (code)');
$this->assertSame('T_OPEN_PARENTHESIS', $token['type'], 'Token tokenized as '.$token['type'].', not T_OPEN_PARENTHESIS (type)');

// Verify that the type union is still tokenized as T_BITWISE_OR as the type declaration
// is not recognized as a valid type declaration.
$unionPtr = $this->getTargetToken($testMarker, [T_BITWISE_OR, T_TYPE_UNION], '|');
$token = $tokens[$unionPtr];

$this->assertSame(T_BITWISE_OR, $token['code'], 'Token tokenized as '.$token['type'].', not T_BITWISE_OR (code)');
$this->assertSame('T_BITWISE_OR', $token['type'], 'Token tokenized as '.$token['type'].', not T_BITWISE_OR (type)');

}//end testBrokenDNFTypeCantEndOnOpenParenthesis()


/**
* Data provider.
*
* @see testBrokenDNFTypeCantEndOnOpenParenthesis()
*
* @return array<string, array<string, string>>
*/
public static function dataBrokenDNFTypeCantEndOnOpenParenthesis()
{
return [
'OO const type' => ['/* testBrokenConstDNFTypeEndOnOpenParenthesis */'],
'OO property type' => ['/* testBrokenPropertyDNFTypeEndOnOpenParenthesis */'],
'Parameter type' => ['/* testBrokenParamDNFTypeEndOnOpenParenthesis */'],
'Return type' => ['/* testBrokenReturnDNFTypeEndOnOpenParenthesis */'],
];

}//end dataBrokenDNFTypeCantEndOnOpenParenthesis()


}//end class

0 comments on commit cac9038

Please sign in to comment.