Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PHP 8.2 | Tokenizer/PHP: add support for DNF types #461

Merged
merged 1 commit into from
Apr 28, 2024

Conversation

jrfnl
Copy link
Member

@jrfnl jrfnl commented Apr 22, 2024

Description

This commit adds tokenizer support for DNF types as per the proposal outlined in #387.

This means that:

  • Two new tokens are introduced T_TYPE_OPEN_PARENTHESIS and T_TYPE_CLOSE_PARENTHESIS for the parentheses used in DNF types. This allows for sniffs to specifically target those tokens and prevents sniffs which are looking for the "normal" open/close parenthesis tokens from acting on DNF parentheses.
  • These new tokens, like other parentheses, will get the parenthesis_opener and parenthesis_closer token array indexes and the tokens between them will have the nested_parenthesis index.

Based on the currently added tests, the commit safeguards that:

  • The | in types is still tokenized as T_TYPE_UNION, even in DNF types.
  • The & in types is still tokenized as T_TYPE_INTERSECTION, even in DNF types.
  • The static keyword for properties is still tokenized as T_STATIC, even when right before a DNF type (which could be confused for a function call).
  • The arrow function retokenization to T_FN with a T_FN_ARROW scope opener is handled correctly, even when DNF types are involved and including when the arrow function is declared to return by reference.
  • The keyword tokens, like self, parent, static, true or false, when used in DNF types are still tokenized to their own token and not tokenized as T_STRING.
  • The array keyword when used in DNF types is still tokenized as T_STRING and not as T_ARRAY.
  • A ? intended as an (illegal) nullability operator in combination with a DNF type is still tokenized as T_NULLABLE and not as T_INLINE_THEN.
  • A function declaration open parenthesis before a typed parameter isn't accidentally retokenized to T_TYPE_OPEN_PARENTHESIS.

Includes ample unit tests.

Even so, strenuous testing of this PR is recommended as there are so many moving parts involved, it is very easy for something to have been overlooked.

Suggested changelog entry

Added: tokenizer support for Disjunctive Normal Form types.

Related issues/external references

Related to #105

Closes #387
Closes squizlabs/PHP_CodeSniffer#3731

Types of changes

  • New feature (non-breaking change which adds functionality)

This commit adds tokenizer support for DNF types as per the proposal outlined in 387.

This means that:
* Two new tokens are introduced `T_TYPE_OPEN_PARENTHESIS` and `T_TYPE_CLOSE_PARENTHESIS` for the parentheses used in DNF types.
    This allows for sniffs to specifically target those tokens and prevents sniffs which are looking for the "normal" open/close parenthesis tokens from acting on DNF parentheses.
* These new tokens, like other parentheses, will get the `parenthesis_opener` and `parenthesis_closer` token array indexes and the tokens between them will have the `nested_parenthesis` index.

Based on the currently added tests, the commit safeguards that:
* The `|` in types is still tokenized as `T_TYPE_UNION`, even in DNF types.
* The `&` in types is still tokenized as `T_TYPE_INTERSECTION`, even in DNF types.
* The `static` keyword for properties is still tokenized as `T_STATIC`, even when right before a DNF type (which could be confused for a function call).
* The arrow function retokenization to `T_FN` with a `T_FN_ARROW` scope opener is handled correctly, even when DNF types are involved and including when the arrow function is declared to return by reference.
* The keyword tokens, like `self`, `parent`, `static`, `true` or `false`, when used in DNF types are still tokenized to their own token and not tokenized as `T_STRING`.
* The `array` keyword when used in DNF types is still tokenized as `T_STRING` and not as `T_ARRAY`.
* A `?` intended as an (illegal) nullability operator in combination with a DNF type is still tokenized as `T_NULLABLE` and not as `T_INLINE_THEN`.
* A function declaration open parenthesis before a typed parameter isn't accidentally retokenized to `T_TYPE_OPEN_PARENTHESIS`.

Includes ample unit tests.

Even so, strenuous testing of this PR is recommended as there are so many moving parts involved, it is very easy for something to have been overlooked.

Related to 105

Closes 387
Closes squizlabs/PHP_CodeSniffer 3731
@jrfnl jrfnl force-pushed the php-8.2/387-tokenizer-add-support-for-dnf-types branch from ceab210 to 5ce28e9 Compare April 28, 2024 11:16
@jrfnl
Copy link
Member Author

jrfnl commented Apr 28, 2024

Rebased without changes.

Even though there has been no feedback so far, I am going to merge this PR to improve the chances of people using the master branch testing this change in hopes of any undesired side-effects being caught before the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Proposal for supporting PHP 8.2+ Disjunctive Normal Form Types in the Tokenizer PHP 8.2 - DNF Support
1 participant