Skip to content

Commit

Permalink
Tokenizers/Comment: bug fix - empty docblock
Browse files Browse the repository at this point in the history
This commit fixes an edge case tokenizer bug, where a - completely empty, not even whitespace - _DocBlock_, would not be tokenized correctly.

Without this commit, the `/***/` code snippet was tokenized as:
```
 13 | L10 | C  1 | CC 0 | ( 0) | T_DOC_COMMENT_OPEN_TAG     | [  5]: /***/
 14 | L10 | C  6 | CC 0 | ( 0) | T_DOC_COMMENT_CLOSE_TAG    | [  0]:
```

With the fix applied, it will be tokenized as:
```
 13 | L10 | C  1 | CC 0 | ( 0) | T_DOC_COMMENT_OPEN_TAG     | [  3]: /**
 14 | L10 | C  4 | CC 0 | ( 0) | T_DOC_COMMENT_CLOSE_TAG    | [  2]: */
```
  • Loading branch information
jrfnl committed May 18, 2024
1 parent c54cb10 commit 252675c
Show file tree
Hide file tree
Showing 3 changed files with 38 additions and 4 deletions.
13 changes: 10 additions & 3 deletions src/Tokenizers/Comment.php
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,16 @@ public function tokenizeString($string, $eolChar, $stackPtr)
extra star when they are used for function and class comments.
*/

$char = ($numChars - strlen(ltrim($string, '/*')));
$openTag = substr($string, 0, $char);
$string = ltrim($string, '/*');
$char = ($numChars - strlen(ltrim($string, '/*')));
$lastChars = substr($string, -2);
if ($char === $numChars && $lastChars === '*/') {
// Edge case: docblock without whitespace or contents.
$openTag = substr($string, 0, -2);
$string = $lastChars;
} else {
$openTag = substr($string, 0, $char);
$string = ltrim($string, '/*');
}

$tokens[$stackPtr] = [
'content' => $openTag,
Expand Down
3 changes: 3 additions & 0 deletions tests/Core/Tokenizer/Comment/SingleLineDocBlockTest.inc
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
/* testEmptyBlockCommentNoWhiteSpace */
/**/

/* testEmptyDocblockNoWhiteSpace */
/***/

/* testEmptyDocblockWithWhiteSpace */
/** */

Expand Down
26 changes: 25 additions & 1 deletion tests/Core/Tokenizer/Comment/SingleLineDocBlockTest.php
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ final class SingleLineDocBlockTest extends CommentTestCase
public static function dataDocblockOpenerCloser()
{
return [
'Single line docblock: empty, no whitespace' => [
'marker' => '/* testEmptyDocblockNoWhiteSpace */',
'closerOffset' => 1,
'expectedTags' => [],
],
'Single line docblock: only whitespace' => [
'marker' => '/* testEmptyDocblockWithWhiteSpace */',
'closerOffset' => 2,
Expand Down Expand Up @@ -79,12 +84,31 @@ public function testEmptyBlockCommentNoWhiteSpace()


/**
* Verify tokenization of an empty, single line DocBlock.
* Verify tokenization of an empty, single line DocBlock without whitespace between the opener and closer.
*
* @phpcs:disable Squiz.Arrays.ArrayDeclaration.SpaceBeforeDoubleArrow -- Readability is better with alignment.
*
* @return void
*/
public function testEmptyDocblockNoWhiteSpace()
{
$expectedSequence = [
[T_DOC_COMMENT_OPEN_TAG => '/**'],
[T_DOC_COMMENT_CLOSE_TAG => '*/'],
];

$target = $this->getTargetToken('/* '.__FUNCTION__.' */', T_DOC_COMMENT_OPEN_TAG);

$this->checkTokenSequence($target, $expectedSequence);

}//end testEmptyDocblockNoWhiteSpace()


/**
* Verify tokenization of an empty, single line DocBlock.
*
* @return void
*/
public function testEmptyDocblockWithWhiteSpace()
{
$expectedSequence = [
Expand Down

0 comments on commit 252675c

Please sign in to comment.