PHPCheckstyle is timing out on itself

[jbrooksuk]

If you run PHPCheckstyle against the actual PHPCheckstyle.php file itself, it crashes. I don't know if this is related to a memory bug or something, but it's unable to process itself.

[jbrooksuk]

I've narrowed it down to the Tokenizer.

[jbrooksuk]

The PHPCheckstyle.php file has 20151 tokens to parse (that's without the pull requests that are waiting to come in). It'd be easy to remove some of these from the file, but this won't be possible in other files, so we need to figure this out.

[jbrooksuk]

Tokenizer.php seems to be the cause of the issue. It may be worth switching to an AST such as PHP-Parser, although I have no idea how it works.

[jbrooksuk]

Looking at PhpParser, I think we're going to need to rewrite a lot of the code, this is a pretty big project, but hopefully we should see major benefit in it and also have a true AST to traverse which already includes support for PHP 5.5

[jbrooksuk]

Turns out we don't need to use an AST. This was fixed by using a local class variable to store tokens within.

Closed by 6b755ae