Use SIMD to accelerate comment lexing

[dlangBugzillaToGithub]

Walter Bright (@WalterBright) reported this on 2015-06-01T19:37:55Z

Transferred from https://issues.dlang.org/show_bug.cgi?id=14641

CC List

Description

We encourage use of Ddoc to document functions. But this can result in voluminous comments, which slow down the lexer. Lexing comments can be accelerated by using SIMD vector instructions.

A little inline assembler in the lexer.c dmd source code would implement this.

[dlangBugzillaToGithub]

briancschott commented on 2015-06-01T20:17:09Z

The best way to do this that I've found is to skip everything other than a set of bytes that varies based on the comment being lexed:

For /* */ comments:
0x0c (\n)
0x0d (\r)
0x2a (*)
0x2f (/)
x0e2 (Beginning of multi-byte UTF-8 newline)

For /+ +/ comments:
0x0c (\n)
0x0d (\r)
0x2b (+)
0x2f (/)
x0e2 (Beginning of multi-byte UTF-8 newline)

For // comments:
0x0c (\n)
0x0d (\r)
x0e2 (Beginning of multi-byte UTF-8 newline)

The instruction used in libdparse to do this is "pcmpestri", which requires SSE4.2 (First released in 2008 according to wikipedia). My advice is to leave most of the logic intact and implement the assembly code such that it may advance the lexer 0 or more bytes, so that the rest of the algorithm is not disrupted on machines that don't support SSE4.2.