C++ Lexer ErrorToken: \ # '

[martinburchell]

I've attached some files that are failing with the C++ Lexer. Most have been broken since c1a0d82. One has been broken since fc56ab8

pygments_cpp_tests.tar.gz

[jeanas]

@amitkummer @hgruniaux

[amitkummer]

Looking into this.

[jeanas]

It seems to work if you wrap the snippets in a function. For example, this is lexed okay:

int main() {
  const QString EMAIL_RE_STR(
      // Regex for an e-mail address.
      // From colander.__init__.py, in turn from
      // https://html.spec.whatwg.org/multipage/input.html#e-mail-state-(type=email)
      // Note that C++ raw strings start R"( and end )"

      R"(^[a-zA-Z0-9.!#$%&'*+\/=?^_`{|}~-]+@[a-zA-Z0-9])"
      R"((?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9])"
      R"((?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$)"
  );
}

AFAICS, the problem is that without the enclosing function, we try to parse the statement as a function definition/declaration. Of course, the code given is not valid as a standalone program, but in documentation, it makes sense to give examples without context (e.g. without a main() function). Then, when we see

type identifier(stuff ...);

it is tricky to know if this is a function declaration or a declaration of a variable that is called by initializing a constructor. Cf. “most vexing parse”.

I'm not sure yet what's the best way to fix this. Maybe just this?

diff --git a/pygments/lexers/c_cpp.py b/pygments/lexers/c_cpp.py
index 5d3b9c7d..198d288d 100644
--- a/pygments/lexers/c_cpp.py
+++ b/pygments/lexers/c_cpp.py
@@ -132,9 +132,9 @@ class CFamilyLexer(RegexLexer):
              r'(' + _possible_comments + r')'    # possible comments
              r'(' + _namespaced_ident + r')'             # method name
              r'(' + _possible_comments + r')'    # possible comments
-             r'(\([^;]*?\))'                          # signature
+             r'(\([^;"]*?\))'                          # signature
              r'(' + _possible_comments + r')'    # possible comments
-             r'([^;{/]*)(\{)',
+             r'([^;{/"]*)(\{)',
              bygroups(using(this), using(this, state='whitespace'), Name.Function, using(this, state='whitespace'),
                       using(this), using(this, state='whitespace'), using(this), Punctuation),
              'function'),
@@ -143,9 +143,9 @@ class CFamilyLexer(RegexLexer):
              r'(' + _possible_comments + r')'    # possible comments
              r'(' + _namespaced_ident + r')'             # method name
              r'(' + _possible_comments + r')'    # possible comments
-             r'(\([^;]*?\))'                          # signature
+             r'(\([^;"]*?\))'                          # signature
              r'(' + _possible_comments + r')'    # possible comments
-             r'([^;/]*)(;)',
+             r'([^;/"]*)(;)',
              bygroups(using(this), using(this, state='whitespace'), Name.Function, using(this, state='whitespace'),
                       using(this), using(this, state='whitespace'), using(this), Punctuation)),
             include('types'),

It's heuristic, and given that we're not going to reimplement a full-fledged C++ parser which is complex like hell, it will always remain heuristic ...

[jeanas]

which is complex like hell

(actually not possible for us since parsing C++ requires knowing the contents of the include files)

[martinburchell]

@jean-abou-samra @amitkummer Thanks for looking into this. In case it is of any help, the original code is at https://github.com/ucam-department-of-psychiatry/camcops/tree/master/tablet_qt