Support tokenize option for FTS

[simonw]

FTS5 supports things like porter stemming using a tokenize= option:

https://www.sqlite.org/fts5.html#tokenizers

Something like this in code:

            CREATE VIRTUAL TABLE [{table}_fts] USING {fts_version} (
                {columns},
                tokenize='porter',
                content=[{table}]
            );

I tried this out just now and it worked exactly as expected.

So... db[table].enable_fts(...) should accept a 'tokenize= argument, and sqlite-utils enable-fts ... should support a --tokenize option.

[simonw]

API documentation here: 617e6f0?short_path=e156262#diff-e1562629b8def6da772d9b0903faf703

[simonw]

Turns out it works for FTS4 as well: https://www.sqlite.org/fts3.html#tokenizer

[simonw]

CLI documentation: 57e4eb8?short_path=7240b7c#diff-7240b7c71b1a8194da0c001c64fc8d40