KeywordSearch: enabling "case sensitive" also breaks punctuation handling — `perfect` stops matching `"...perfect."`

[kz930]

What happened?

KeywordSearchOpExec picks one of two Lucene analyzers based on the
isCaseSensitive flag:

common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/keywordSearch/KeywordSearchOpExec.scala:38

@transient private lazy val analyzer: Analyzer = {
  if (desc.isCaseSensitive) new CaseSensitiveAnalyzer() else new StandardAnalyzer()
}

• Case-insensitive (default) → StandardAnalyzer: StandardTokenizer splits
  on Unicode word boundaries, so punctuation is stripped and "perfect."
  tokenizes to perfect. Query perfect matches.
• Case-sensitive → CaseSensitiveAnalyzer, which is a bare
  WhitespaceTokenizer:

common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/keywordSearch/CaseSensitiveAnalyzer.scala:29

// Achieves case sensitivity by skipping the lowercasing and normalization
// pipeline used in StandardAnalyzer.
class CaseSensitiveAnalyzer extends Analyzer {
  override protected def createComponents(fieldName: String): TokenStreamComponents = {
    val tokenizer = new WhitespaceTokenizer()
    val stream: TokenStream = new StopFilter(tokenizer, CharArraySet.EMPTY_SET)
    new TokenStreamComponents(tokenizer, stream)
  }
}

WhitespaceTokenizer splits only on whitespace, so punctuation stays glued
to the token: "...absolutely perfect." tokenizes to [absolutely, perfect.].
The query term perfect (tokenized the same way → perfect) does not equal the
indexed token perfect., so the row is not matched.

The intent of CaseSensitiveAnalyzer was only to skip lowercasing. But swapping
StandardTokenizer for WhitespaceTokenizer also dropped Unicode word-boundary
tokenization. So toggling "case sensitive" silently changes punctuation
handling too — a side effect the option name does not imply. A user who turns on
case sensitivity to match Trump exactly will find that USA. no longer matches
USA, day! no longer matches day, etc.

Impact: any keyword search over real text (which contains commas, periods,
!, ?) behaves inconsistently between the two modes. Words adjacent to
punctuation become unmatchable in case-sensitive mode.

Proposed fix (untested — direction only, not yet compiled/verified): make
CaseSensitiveAnalyzer mirror StandardAnalyzer minus the lowercasing step —
i.e. use StandardTokenizer (Unicode word-boundary tokenization, strips
punctuation) and simply omit the LowerCaseFilter, instead of falling back to
WhitespaceTokenizer:

override protected def createComponents(fieldName: String): TokenStreamComponents = {
  val tokenizer = new StandardTokenizer()
  // StandardAnalyzer pipeline without LowerCaseFilter: keep case, still strip
  // punctuation on Unicode word boundaries.
  val stream: TokenStream = new StopFilter(tokenizer, CharArraySet.EMPTY_SET)
  new TokenStreamComponents(tokenizer, stream)
}

Then "case sensitive" changes only case behavior, matching user expectation and
staying consistent with the default mode's tokenization.

Notes:

• Present on main; not introduced by the standalone-translation work.
• Related: the standalone translation (KeywordSearchOpDesc.generateStandaloneCode)
  does not implement case sensitivity at all (always case-insensitive \b-regex),
  so it already diverges from this mode. Fixing the analyzer would make the
  case-insensitive behavior of both paths line up on punctuated text; full
  case-sensitive parity in the standalone is a separate gap.

How to reproduce?

Option A — user-facing (workflow):

1. Add a Keyword Search operator over a text column whose values contain
   trailing punctuation, e.g. a row "everything was absolutely perfect.".
2. Set keyword = perfect, Case Sensitive = off → the row matches (expected).
3. Flip Case Sensitive = on, rerun → the same row no longer matches, even
   though perfect obviously appears. Only the case handling was supposed to
   change.

Option B — unit test (mirrors KeywordSearchOpExecSpec):

val opDesc = new KeywordSearchOpDesc()
opDesc.attribute = "text"
opDesc.keyword = "perfect"

val schema = Schema().add(new Attribute("text", AttributeType.STRING))
def row(t: String) = Tuple.builder(schema).add(schema.getAttribute("text"), t).build()
val data = List(row("everything was absolutely perfect."))

// case-insensitive: matches
opDesc.isCaseSensitive = false
val a = new KeywordSearchOpExec(objectMapper.writeValueAsString(opDesc)); a.open()
assert(data.exists(t => a.processTuple(t, 0).nonEmpty))   // passes

// case-sensitive: SHOULD still match "perfect", but does not
opDesc.isCaseSensitive = true
val b = new KeywordSearchOpExec(objectMapper.writeValueAsString(opDesc)); b.open()
assert(data.exists(t => b.processTuple(t, 0).nonEmpty))   // FAILS on current main

Version/Branch

1.3.0-incubating-SNAPSHOT (main)

[kz930]

/take