Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

c++: has_value() function marked as generic Name rather than Function.Name #1561

Closed
tabatkins opened this issue Oct 2, 2020 · 2 comments · Fixed by #1722
Closed

c++: has_value() function marked as generic Name rather than Function.Name #1561

tabatkins opened this issue Oct 2, 2020 · 2 comments · Fixed by #1722
Milestone

Comments

@tabatkins
Copy link

Another change I noticed when bumping from 2.6.1 to 2.7.1, probably as part of fc56ab8.

In the following code:

constexpr bool has_value() const noexcept;

has_value was previously marked as Name.Function, but now it's marked as just Name. I suspect it should still be Name.Function.

@tabatkins
Copy link
Author

I'm seeing changes in both directions, actually:

In:

 E& error() &;

The error token was Name in 2.6.1, and is Name.Function is 2.7.1 (looks like a correct change).

While in:

 virtual const char* what() const noexcept override;

The what was Name.Function in 2.6.1, and is just Name in 2.7.1 (looks like an error).

All of these examples were taken from http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p0323r8.html, btw.

@Anteru Anteru added the help wanted Community help appreciated! label Feb 6, 2021
@amitkummer
Copy link
Contributor

amitkummer commented Feb 18, 2021

After some investigation I am seeing more problematic cases. Function names with :: are tokenized as Names:

int Type::has_value() {}

has_value is tokenized as a Name and not Name.Function. I think this is because the lexer has a list of identifiers it uses to match function names and ':' is not on this list.

Another case I see is that the lexer matches the if here as Name.Function:

else if () {}

And T here is matched as Name.Function:

new T();

@birkenfeld birkenfeld changed the title has_value() function marked as generic Name rather than Function.Name c++: has_value() function marked as generic Name rather than Function.Name Feb 18, 2021
amitkummer added a commit to amitkummer/Pygments that referenced this issue Feb 23, 2021
This fixes pygments#1561.

Add a keywords state that matches inside and outside function for
keywords.

Before this, when a keyword would appear the lexer would go to the
statements state, in which functions were not matched.
amitkummer added a commit to amitkummer/Pygments that referenced this issue Feb 23, 2021
This fixes pygments#1561.

Add a keywords state that matches inside and outside functions for
keywords.

Before this, when a keyword would appear the lexer would go to the
statements state, in which functions were not matched.
Anteru pushed a commit that referenced this issue Feb 27, 2021
* Fix lexing of function names

This fixes #1561.

Add a keywords state that matches inside and outside functions for
keywords.

Before this, when a keyword would appear the lexer would go to the
statements state, in which functions were not matched.

* Add tests for lexing of function names

* Unbreak previous tests

* Allow namespaced names in function statements

Add a second identifiers regex that matces all the previous identifiers
and also '::'.

I took the decision to create a second identifiers regex with '::'
inside, simply because using the old identifiers regex would hurt
performance massively on every solution I tried to craft.

* Add tests for namespaced names in functions

* Unbreak previous tests

* Add support for namespaces in namespace declarations

Add a namespace state that is entered each time the namespace keyword
is matched and lexes all name matches as namespaces.

Cases this approach doesn't cover:
* Namespaces in using declarations.
* Namespaces that prefix names in random code.
Unfortunately, in both of these cases the names before and after '::'
are not always namespaces.

* Add tests for namespace declartions

* Unbreak previous tests

* Tidy functions regex

Remove group nesting that became unneeded after fc56ab8 (the last big
refactor).

* Remove f string usage I introduced by mistake
@Anteru Anteru added this to the 2.9 milestone Feb 27, 2021
@Anteru Anteru removed the help wanted Community help appreciated! label Mar 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants