CNDB-9422 vsearch: Add CQL function to get the tokens produced by a Lucene analyzer

[adelapena]

The function replaces the virtual table system_views.analyzer, as described in CNDB-9422.

[adelapena]

I wonder if the function should be named simply "analyze", or we should mention Lucene in the name, since there can be other analyzers. For example, I think SASI analyzer doesn't use Lucene.

[jkni]

I wonder if the function should be named simply "analyze", or we should mention Lucene in the name, since there can be other analyzers. For example, I think SASI analyzer doesn't use Lucene.

This is a good point. I think I'd prefer calling it sai_analyze if we want to distinguish that this is the SAI analysis path. While we use Lucene for analysis at the moment, I think that's more of an implementation detail, and I'd rather not couple function naming to that.

[jkni]

LGTM from an implementation perspective. Should we add in-tree doc coverage for this?

[adelapena]

Thanks for the review. I have renamed the function to sai_analyze and added some doc. Documentation for SAI is something that we should definitively add at some point; I haven't found anything about it in the in-tree doc.

[sonarcloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
97.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[jkni]

LGTM! Test failure is benign and related to test quality, not this PR. Thanks for the PR. Agreed on SAI coverage in-tree.