Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shorthand binary cast causes tokenizer errors #371

Closed
bskendig opened this issue Dec 15, 2014 · 8 comments
Closed

Shorthand binary cast causes tokenizer errors #371

bskendig opened this issue Dec 15, 2014 · 8 comments
Labels
Type: bug

Comments

@bskendig
Copy link

My codebase passed phpcs 1.5.6 cleanly. When I run phpcs.phar 2.0.0 on it, I get "Undefined index: parenthesis_closer" and "Undefined index:" errors on a large number of files. Here's one iteration of the two errors; this block of errors repeats hundreds of times.

$ phpcs -v src
Registering sniffs in the PEAR standard... DONE (27 sniffs registered)
Creating file list... DONE (4 files in queue)
Changing into directory src
Processing run-tests.php PHP Notice:  Undefined index: parenthesis_closer in phar:///usr/local/bin/phpcs/CodeSniffer/File.php on line 1856
PHP Stack trace:
PHP   1. {main}() /usr/local/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /usr/local/bin/phpcs:6
PHP   3. PHP_CodeSniffer_CLI->process() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:95
PHP   4. PHP_CodeSniffer->processFiles() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:854
PHP   5. PHP_CodeSniffer->processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:615
PHP   6. PHP_CodeSniffer->_processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:1622
PHP   7. PHP_CodeSniffer_File->start() phar:///usr/local/bin/phpcs/CodeSniffer.php:1744
PHP   8. PHP_CodeSniffer_File->_parse() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:462
PHP   9. PHP_CodeSniffer_File::tokenizeString() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:706
PHP  10. PHP_CodeSniffer_File::_createScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1432
PHP  11. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1774
PHP  12. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:2026
PHP  13. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:2026

Notice: Undefined index: parenthesis_closer in phar:///usr/local/bin/phpcs/CodeSniffer/File.php on line 1856

Call Stack:
    0.0002     226784   1. {main}() /usr/local/bin/phpcs:0
    0.0103    1648160   2. PHP_CodeSniffer_CLI->runphpcs() /usr/local/bin/phpcs:6
    0.0108    1704640   3. PHP_CodeSniffer_CLI->process() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:95
    0.0262    3185328   4. PHP_CodeSniffer->processFiles() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:854
    0.0278    3188112   5. PHP_CodeSniffer->processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:615
    0.0279    3188984   6. PHP_CodeSniffer->_processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:1622
    0.0284    3223432   7. PHP_CodeSniffer_File->start() phar:///usr/local/bin/phpcs/CodeSniffer.php:1744
    0.0285    3225472   8. PHP_CodeSniffer_File->_parse() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:462
    0.0285    3297312   9. PHP_CodeSniffer_File::tokenizeString() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:706
    0.2047   20254528  10. PHP_CodeSniffer_File::_createScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1432
    0.2339   20463880  11. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1774
    0.2563   20570624  12. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:2026
    0.2573   20580664  13. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:2026

PHP Notice:  Undefined index:  in phar:///usr/local/bin/phpcs/CodeSniffer/File.php on line 1860
PHP Stack trace:
PHP   1. {main}() /usr/local/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /usr/local/bin/phpcs:6
PHP   3. PHP_CodeSniffer_CLI->process() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:95
PHP   4. PHP_CodeSniffer->processFiles() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:854
PHP   5. PHP_CodeSniffer->processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:615
PHP   6. PHP_CodeSniffer->_processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:1622
PHP   7. PHP_CodeSniffer_File->start() phar:///usr/local/bin/phpcs/CodeSniffer.php:1744
PHP   8. PHP_CodeSniffer_File->_parse() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:462
PHP   9. PHP_CodeSniffer_File::tokenizeString() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:706
PHP  10. PHP_CodeSniffer_File::_createScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1432
PHP  11. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1774
PHP  12. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:2026
PHP  13. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:2026

Notice: Undefined index:  in phar:///usr/local/bin/phpcs/CodeSniffer/File.php on line 1860

Call Stack:
    0.0002     226784   1. {main}() /usr/local/bin/phpcs:0
    0.0103    1648160   2. PHP_CodeSniffer_CLI->runphpcs() /usr/local/bin/phpcs:6
    0.0108    1704640   3. PHP_CodeSniffer_CLI->process() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:95
    0.0262    3185328   4. PHP_CodeSniffer->processFiles() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:854
    0.0278    3188112   5. PHP_CodeSniffer->processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:615
    0.0279    3188984   6. PHP_CodeSniffer->_processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:1622
    0.0284    3223432   7. PHP_CodeSniffer_File->start() phar:///usr/local/bin/phpcs/CodeSniffer.php:1744
    0.0285    3225472   8. PHP_CodeSniffer_File->_parse() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:462
    0.0285    3297312   9. PHP_CodeSniffer_File::tokenizeString() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:706
    0.2047   20254528  10. PHP_CodeSniffer_File::_createScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1432
    0.2339   20463880  11. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1774
    0.2563   20570624  12. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:2026
    0.2573   20580664  13. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:2026
...
@oliverklee
Copy link
Contributor

Can you please post the PHP code that - when scanned - triggers the notices, e.g., in a GitHub Gist?

@bskendig
Copy link
Author 

$ cat bar.php
<?php
foo($a, b"
$c
");

$ phpcs -v bar.php 
Registering sniffs in the PEAR standard... DONE (27 sniffs registered)
Creating file list... DONE (1 files in queue)
Changing into directory src
Processing bar.php [PHP => 11 tokens in 4 lines]...

This then hangs without generating further output.

@bskendig
Copy link
Author 

$ cat bar.php
<?php
if (foo($a, b"
$c
")) {}

$ phpcs -v bar.php 
Registering sniffs in the PEAR standard... DONE (27 sniffs registered)
Creating file list... DONE (1 files in queue)
Changing into directory src
Processing bar.php PHP Notice:  Undefined index: parenthesis_closer in phar:///usr/local/bin/phpcs/CodeSniffer/File.php on line 1856
PHP Stack trace:
PHP   1. {main}() /usr/local/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /usr/local/bin/phpcs:6
PHP   3. PHP_CodeSniffer_CLI->process() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:95
PHP   4. PHP_CodeSniffer->processFiles() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:854
PHP   5. PHP_CodeSniffer->processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:615
PHP   6. PHP_CodeSniffer->_processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:1622
PHP   7. PHP_CodeSniffer_File->start() phar:///usr/local/bin/phpcs/CodeSniffer.php:1744
PHP   8. PHP_CodeSniffer_File->_parse() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:462
PHP   9. PHP_CodeSniffer_File::tokenizeString() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:706
PHP  10. PHP_CodeSniffer_File::_createScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1432
PHP  11. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1774

Notice: Undefined index: parenthesis_closer in phar:///usr/local/bin/phpcs/CodeSniffer/File.php on line 1856

Call Stack:
    0.0002     226720   1. {main}() /usr/local/bin/phpcs:0
    0.0149    1648064   2. PHP_CodeSniffer_CLI->runphpcs() /usr/local/bin/phpcs:6
    0.0154    1704568   3. PHP_CodeSniffer_CLI->process() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:95
    0.0315    3185312   4. PHP_CodeSniffer->processFiles() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:854
    0.0316    3186840   5. PHP_CodeSniffer->processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:615
    0.0316    3187672   6. PHP_CodeSniffer->_processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:1622
    0.0321    3222120   7. PHP_CodeSniffer_File->start() phar:///usr/local/bin/phpcs/CodeSniffer.php:1744
    0.0321    3224160   8. PHP_CodeSniffer_File->_parse() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:462
    0.0322    3225864   9. PHP_CodeSniffer_File::tokenizeString() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:706
    0.0327    3282944  10. PHP_CodeSniffer_File::_createScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1432
    0.0327    3283152  11. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1774

PHP Notice:  Undefined index:  in phar:///usr/local/bin/phpcs/CodeSniffer/File.php on line 1860
PHP Stack trace:
PHP   1. {main}() /usr/local/bin/phpcs:0
PHP   2. PHP_CodeSniffer_CLI->runphpcs() /usr/local/bin/phpcs:6
PHP   3. PHP_CodeSniffer_CLI->process() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:95
PHP   4. PHP_CodeSniffer->processFiles() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:854
PHP   5. PHP_CodeSniffer->processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:615
PHP   6. PHP_CodeSniffer->_processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:1622
PHP   7. PHP_CodeSniffer_File->start() phar:///usr/local/bin/phpcs/CodeSniffer.php:1744
PHP   8. PHP_CodeSniffer_File->_parse() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:462
PHP   9. PHP_CodeSniffer_File::tokenizeString() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:706
PHP  10. PHP_CodeSniffer_File::_createScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1432
PHP  11. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1774

Notice: Undefined index:  in phar:///usr/local/bin/phpcs/CodeSniffer/File.php on line 1860

Call Stack:
    0.0002     226720   1. {main}() /usr/local/bin/phpcs:0
    0.0149    1648064   2. PHP_CodeSniffer_CLI->runphpcs() /usr/local/bin/phpcs:6
    0.0154    1704568   3. PHP_CodeSniffer_CLI->process() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:95
    0.0315    3185312   4. PHP_CodeSniffer->processFiles() phar:///usr/local/bin/phpcs/CodeSniffer/CLI.php:854
    0.0316    3186840   5. PHP_CodeSniffer->processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:615
    0.0316    3187672   6. PHP_CodeSniffer->_processFile() phar:///usr/local/bin/phpcs/CodeSniffer.php:1622
    0.0321    3222120   7. PHP_CodeSniffer_File->start() phar:///usr/local/bin/phpcs/CodeSniffer.php:1744
    0.0321    3224160   8. PHP_CodeSniffer_File->_parse() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:462
    0.0322    3225864   9. PHP_CodeSniffer_File::tokenizeString() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:706
    0.0327    3282944  10. PHP_CodeSniffer_File::_createScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1432
    0.0327    3283152  11. PHP_CodeSniffer_File::_recurseScopeMap() phar:///usr/local/bin/phpcs/CodeSniffer/File.php:1774

[PHP => 14 tokens in 4 lines]...

This also then hangs.

@bskendig
Copy link
Author

The b"..." is binary casting (http://php.net/manual/en/language.types.type-juggling.php), which I hadn't seen before I ran into this.

@bskendig
Copy link
Author

Even more minimal test cases:

b"$c"; - hangs phpcs

if (b"$c") {} - throws the errors then hangs phpcs

@gsherwood
Copy link
Member

I hadn't seen binary casting before either. Looks like PHP doesn't assign a specific token to it, so I might need to detect it in some way. Thanks for posting this code.

@gsherwood gsherwood changed the title "Undefined index: parenthesis_closer" Shorthand binary cast causes tokenizer errors Dec 15, 2014
gsherwood added a commit that referenced this issue Dec 15, 2014
@gsherwood
Fixed bug #371 : Shorthand binary cast causes tokenizer errors
66230c8
@gsherwood
Copy link
Member

The fix that went in now tokenizes this code:

<?php
if (foo($a, b"
$c
")) {}

b"$c";

if (b"$c") {}

like this:

Process token 0 on line 1 [col:1;len:5;lvl:0;]: T_OPEN_TAG => <?php\n
Process token 1 on line 2 [col:1;len:2;lvl:0;]: T_IF => if
Process token 2 on line 2 [col:3;len:1;lvl:0;]: T_WHITESPACE => ·
Process token 3 on line 2 [col:4;len:1;lvl:0;]: T_OPEN_PARENTHESIS => (
Process token 4 on line 2 [col:5;len:3;lvl:0;]: T_STRING => foo
Process token 5 on line 2 [col:8;len:1;lvl:0;]: T_OPEN_PARENTHESIS => (
Process token 6 on line 2 [col:9;len:2;lvl:0;]: T_VARIABLE => $a
Process token 7 on line 2 [col:11;len:1;lvl:0;]: T_COMMA => ,
Process token 8 on line 2 [col:12;len:1;lvl:0;]: T_WHITESPACE => ·
Process token 9 on line 2 [col:13;len:1;lvl:0;]: T_BINARY_CAST => b
Process token 10 on line 2 [col:14;len:1;lvl:0;]: T_DOUBLE_QUOTED_STRING => "\n
Process token 11 on line 3 [col:1;len:2;lvl:0;]: T_DOUBLE_QUOTED_STRING => $c\n
Process token 12 on line 4 [col:1;len:1;lvl:0;]: T_DOUBLE_QUOTED_STRING => "
Process token 13 on line 4 [col:2;len:1;lvl:0;]: T_CLOSE_PARENTHESIS => )
Process token 14 on line 4 [col:3;len:1;lvl:0;]: T_CLOSE_PARENTHESIS => )
Process token 15 on line 4 [col:4;len:1;lvl:0;]: T_WHITESPACE => ·
Process token 16 on line 4 [col:5;len:1;lvl:0;]: T_OPEN_CURLY_BRACKET => {
Process token 17 on line 4 [col:6;len:1;lvl:1;conds;T_IF;]: T_CLOSE_CURLY_BRACKET => }
Process token 18 on line 4 [col:7;len:0;lvl:0;]: T_WHITESPACE => \n
Process token 19 on line 5 [col:1;len:0;lvl:0;]: T_WHITESPACE => \n
Process token 20 on line 6 [col:1;len:1;lvl:0;]: T_BINARY_CAST => b
Process token 21 on line 6 [col:2;len:4;lvl:0;]: T_DOUBLE_QUOTED_STRING => "$c"
Process token 22 on line 6 [col:6;len:1;lvl:0;]: T_SEMICOLON => ;
Process token 23 on line 6 [col:7;len:0;lvl:0;]: T_WHITESPACE => \n
Process token 24 on line 7 [col:1;len:0;lvl:0;]: T_WHITESPACE => \n
Process token 25 on line 8 [col:1;len:2;lvl:0;]: T_IF => if
Process token 26 on line 8 [col:3;len:1;lvl:0;]: T_WHITESPACE => ·
Process token 27 on line 8 [col:4;len:1;lvl:0;]: T_OPEN_PARENTHESIS => (
Process token 28 on line 8 [col:5;len:1;lvl:0;]: T_BINARY_CAST => b
Process token 29 on line 8 [col:6;len:4;lvl:0;]: T_DOUBLE_QUOTED_STRING => "$c"
Process token 30 on line 8 [col:10;len:1;lvl:0;]: T_CLOSE_PARENTHESIS => )
Process token 31 on line 8 [col:11;len:1;lvl:0;]: T_WHITESPACE => ·
Process token 32 on line 8 [col:12;len:1;lvl:0;]: T_OPEN_CURLY_BRACKET => {
Process token 33 on line 8 [col:13;len:1;lvl:1;conds;T_IF;]: T_CLOSE_CURLY_BRACKET => }

Note the new T_BINARY_CAST tokens in there. Thanks again for submitting that code.

@bskendig
Copy link
Author

The fix you checked in solves the problem for me. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@gsherwood @oliverklee @bskendig