[test] Write exhaustive tests over the bjorn dfa and crockford utf8 decoders

[PossiblyAShrub]

I've found a way to exhaustively test all the interesting cases for our UTF-8 decoder which is only ~9 million byte sequences. My reasoning for that is covered in the header of data_lang/utf8_test.cc

These tests required some meta-programming so I could avoid lots of manual bit-math. They are generated from bit patterns like 1111 xxxx which denotes the byte range 0xF0 to 0xFF. This is done using the data_lang/utf8_decoder_tests_gen.py script which is invoked at build time.

I've been running the tests as so:

ninja _bin/cxx-asan/data_lang/utf8_test
./_bin/cxx-asan/data_lang/utf8_test -t utf8_decoder
# or just ./_bin/cxx-asan/data_lang/utf8_test for all tests

They should be run in CI as well.

[andychu]

OK I poked at this slightly, I ran the opt version of the utf8_test and it took 2 seconds, which is fast enough

I merged master into the branch

[andychu]

Hm one thing is I wonder if we can generate a C++ data file that different TEST() blocks read, rather than the .inc file with TEST() blocks

I think this is cleaner for tests

---

I will have to understand the strategy a little more ... I thought there were SOME cases that could be done "brute force", and then maybe 1 case that needs to be clever about not being too slow