Replace the tokenizer with a flex-based scanner

[kivikakk]

Preliminary benchmarks put this in at a 12x speedup.

[1] ~/github/linguist  $ bundle exec bin/bench
Rehearsal -----------------------------------------
c:      0.460000   0.000000   0.460000 (  0.464052)
rb:     5.120000   0.020000   5.140000 (  5.142903)
-------------------------------- total: 5.600000sec
            user     system      total        real
c:      0.420000   0.000000   0.420000 (  0.426902)
rb:     5.170000   0.020000   5.190000 (  5.188762)
Tokens extracted by each method:
 * extract_tokens_c: 3041830
 * extract_tokens_rb: 2836080

It doesn't produce identical results, but very near enough to. (Enough that all the tests should pass.)

/cc @vmg because he luuuuurves C

[vmg]

Looking good. I assume we're now properly handling all the weird Flex crashes you were seeing?

I also think it'd be neat if the Rakefile had a way to rebuild the lexer (and also to check that we're using the right version of Flex).

[lildude]

I've not tested this yet, but I wonder how well this new tokenizer will fair with non-ASCII. From the look of things, it should be 👌 , but thought I'd ask to be sure.

For context, an attempt to improve Linguist's support of non-ASCII in the ruby implementation has been started in #3748.

[kivikakk]

I assume we're now properly handling all the weird Flex crashes you were seeing?

Yeah; I constrained our use of features which turn out to be dangerous (LOOKING AT YOU, TRAILING CONTEXT), and everything works as expected now. ✨

I also think it'd be neat if the Rakefile had a way to rebuild the lexer (and also to check that we're using the right version of Flex).

+1, will add.

I've not tested this yet, but I wonder how well this new tokenizer will fair with non-ASCII.

It'll do as well as it currently does, which is to say Not Hugely Well; non-ASCII stuff will get skipped. It wouldn't be too hard to make it grok things we're likely to see in UTF-8 text, though it'd be a lot harder to do this and only match word-characters (since we'd have to add actual Unicode understanding to our lexer at that stage).

[Alhadis]

Does the \R extension not work here?

I also take it Ruby's regex engine doesn't have the equivalent of Perl's /a modifier?

[kivikakk]

I'm changing as little code as I can; this is just a refactor from:

https://github.com/github/linguist/blob/0b9c05f989a66e3e92ca9a1e0b236781ef54229f/lib/linguist/blob_helper.rb#L278-L279

\R also catches [\v\f] which we definitely don't want.

I also take it Ruby's regex engine doesn't have the equivalent of Perl's /a modifier?

~$ ruby -e '//a'
-e:1: unknown regexp option - a

It doesn't, and more to the point, it wouldn't help for our use here, which isn't about Unicode-aware matching so much as avoiding terrible encoding exceptions rising from the deep. /a modifies the meaning of several sequences in the regular expressions itself, rather than changing how a regular expression is applied to a given byte-sequence-tagged-with-an-encoding (i.e. a String), whatever the meaning of its contents.

[Alhadis]

Ah, I see. ;) Just thought to ask, since it's used very little in Perl (for good reasons). Thanks!

[kivikakk]

I'd like to merge this! Anyone feel like doing a final review?

[lildude]

Caveat pre-emptor: I have a copy of Dennis Richie's book but I'm far from being a C expert.

From what I do know this looks good to me, and the perf improvement is fantastic!!

[kivikakk]

@lildude Thank you! The responsibility is mine if this somehow goes belly-up.