STAR-123 custom lucene analyzer and some language unit tests

[jasonrutherglen]

added more language tests

added brazilian

cql test passes

added support for setting a lucene analyzer

cql json test passes

fixed up some things

[mike-tr-adamson]

I've done an initial review of the changes here.

My main concern about this is how we are going to document it. I know we've discussed this but it seems to me that usage of these is a bit of a black art. The lucene documentation, I could find, points to the Javadoc for the analyzer/tokenizer classes. I can't see new users getting to grips with that very easily.

Seems to me that we need to, at least, document common usages. Then we can add documentation as we go on.

This is just observation, it doesn't stop these changes getting merged.

[mike-tr-adamson]

Definitely in favour of WITH JSON OPTIONS here. It can be done on a follow on ticket.

[mike-tr-adamson]

This has been left over from another ticket but we do need to figure out what should be done at this point rather than just throwing.

[jasonrutherglen]

@mike-tr-adamson I think most users can simply use the analyzer classes of which we can list them. They're available at https://lucene.apache.org/core/7_7_3/analyzers-common/index.html

[mike-tr-adamson]

LGTM

[pkolaczk]

I looked at the code-smells reported by Sonar and it complains a lot about too generic exceptions plus a few minor issues.
Can you fix them, please?

[pkolaczk]

What is sasi code? I'm commenting on the code you've added / modified. org.apache.cassandra.index.sai.analyzer.LuceneAnalyzer looks like our class, same for AbstractAnalyzer.

This class implements an Iterator but breaks the Iterator contract in a few ways.
One way is fortunately documented on the next() call (next() doesn't work correctly before calling hasNext()).
But another one is having to call reset(...) before calling hasNext().

The top level documentation for the class should say something like that:

This class wraps a lucene `Analyzer` in order to <describe the purpose of this class; what value it adds to the lucene's Analyzer functionality>.

Caution!
Although this class implements `java.util.Iterator`, it does not adhere at all to the iterator contract:
1. Before the first use after construction, you have to call `reset(ByteBuffer)` in order to provide the input data to be analyzed. You are allowed to call `reset` multiple times to change the analyzed input.
2. To obtain the next analyzed element, you have to call `hasNext()` before calling `next()`. The call to `next()` does not advance the iterator, so calling it multiple times will result in the same element (or throwing NoSuchElementException if `hasNext` has never been called before`
3. A call to `hasNext` advances the iterator.  

This is the bare minimum I'd expect. A better documentation would also tell me why we're really making iterators that do not work like proper iterators. I haven't written the code, so I really don't know why - you're the author and you should explain.

[jasonrutherglen]

@pkolaczk

The reply in thread doesn't work from:

#192 (comment)

The iterator style is from sasi. https://github.com/datastax/cassandra/blob/ds-trunk/src/java/org/apache/cassandra/index/sasi/analyzer/AbstractAnalyzer.java

Again... comments directed at why it's this way may be addressed to the sasi devs on a mailing list.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

84.5% Coverage
0.0% Duplication