Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes for C and C++ functions and namespaces #1722

Merged
merged 11 commits into from
Feb 27, 2021

Conversation

amitkummer
Copy link
Contributor

@amitkummer amitkummer commented Feb 23, 2021

This fixes #1561 and also fixes #1719.

Fix lexing of function names (#1561) by adding a keywords state for matching keywords, inside and outside functions.
Before this, when a keyword would appear the lexer would go to the statements state, in which functions were not matched, so functions prefixed with keywords were not matched.

  • Apart from the c and cpp lexers, I only updated the swig lexer and made sure it's keywords are in the new keywords state, as it was the only lexer that was breaking from this change.

  • The new keywords state is (almost) entirely optional on purpose for deriving lexers. They don't benefit from using it themselves. Because of that I think it's a waste of time to incorporate this state in deriving lexers so I decided against it.

Fix matching functions with a name or return argument that includes a '::' (also #1561) by introducing a second identifiers regex that matches all the previous identifiers, and also '::', and use it to match for function names and return arguments.

  • I took the decision to create a second identifiers regex with '::' inside, simply because using the old identifiers regex would hurt
    performance massively on every solution I tried to craft. I think this solution is still pretty elegant though, and the issue it fixes is big.

Fix lexing of namespaces (#1719) wherever possible, by adding a namespace state that is entered each time the namespace keyword is matched, and lexes all names matched as namespaces.

Cases this approach doesn't cover:

  • Namespaces in using declarations.
  • Namespaces that prefix names in random code.

Unfortunately, in both of these cases the names before and after '::' are not always namespaces so these cases can't be covered.

This fixes pygments#1561.

Add a keywords state that matches inside and outside functions for
keywords.

Before this, when a keyword would appear the lexer would go to the
statements state, in which functions were not matched.
Add a second identifiers regex that matces all the previous identifiers
and also '::'.

I took the decision to create a second identifiers regex with '::'
inside, simply because using the old identifiers regex would hurt
performance massively on every solution I tried to craft.
Add a namespace state that is entered each time the namespace keyword
is matched and lexes all name matches as namespaces.

Cases this approach doesn't cover:
* Namespaces in using declarations.
* Namespaces that prefix names in random code.
Unfortunately, in both of these cases the names before and after '::'
are not always namespaces.
Remove group nesting that became unneeded after fc56ab8 (the last big
refactor).
@amitkummer amitkummer changed the title Fixes for C++ functions and namespaces Fixes for C and C++ functions and namespaces Feb 23, 2021
@Anteru Anteru merged commit c1a0d82 into pygments:master Feb 27, 2021
@Anteru Anteru added the changelog-update Items which need to get mentioned in the changelog label Feb 27, 2021
@Anteru Anteru added this to the 2.9 milestone Feb 27, 2021
@Anteru
Copy link
Collaborator

Anteru commented Feb 27, 2021

Merged, thanks a lot for your contribution! Nicely done!

@Anteru Anteru removed the changelog-update Items which need to get mentioned in the changelog label Mar 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants