Introduce new hasPhrase function

[ahmadov]

The first stage for the #101473. The new function is independent of how the positional data would be stored.

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Adds a hasPhrase (alias matchPhrase) function for phrase search (continuous sequences of tokens). Search is brute-force, i.e. not supported by the text index yet.

[clickhouse-gh]

Workflow [PR], commit [3b03b2d]

Summary: ❌

job_name	test_name	status	info	comment
AST fuzzer (amd_debug, targeted)		failure
	Assertion `px != 0' failed (STID: 0250-3d88)	FAIL	cidb
AST fuzzer (amd_debug, targeted, old_compatibility)		failure
	Assertion `px != 0' failed (STID: 0250-3d88)	FAIL	cidb

---

AI Review

Summary

This PR adds a new string-search function hasPhrase (with alias matchPhrase) that matches consecutive token sequences using tokenizer-aware tokenization and a KMP-style matcher, plus comprehensive stateless tests. I reviewed all changed files and did not find correctness, safety, concurrency, or performance blockers in the current patch.

Missing context

• ⚠️ Full CI signal is not yet available (PR status is still pending), so this review is code-focused.

ClickHouse Rules

Item	Status	Notes
Deletion logging	➖
Serialization versioning	➖
Core-area scrutiny	✅
No test removal	✅
Experimental gate	➖
No magic constants	✅
Backward compatibility	✅
SettingsChangesHistory.cpp	➖
PR metadata quality	✅
Safe rollout	✅
Compilation time	✅
No large/binary files	✅

Final Verdict

• Status: ✅ Approve

[Ergus]

Overall LGTM, but the added a few comment it worth address before merging.

My question with this is how is this different from preexisting functions like multiSearch or substring family of functions?

[Ergus]

else 
   UNREACHABLE()

or

else
   throw Exception...

It is safe to always add a last resource guard error.

[ahmadov]

actually, cannot happen because of the validator. I removed the similar checks #101997 (comment)

[Ergus]

But you can change the validator and forget this, or the other way around ;)

[Ergus]

Some time ago I had an issue with code like this:

onMatchCallback is captured to the current scope in a move. But the lambda captures it by reference. So it conceptually goes OOS when this operator() function returns... in principle this is a potential dangling pointer.

C++ compilers hide this with a trick: when a reference-type variable is captured by reference, C++ captures the referent (not the reference variable itself). So the closure holds a reference directly to the original temporary callback, which stays alive for the full expression.

Not a bug in this code. But a reason why I don't like complex lambdas ;)

Do we really need to return a lambda?

[ahmadov]

forEachToken accepts a callback (lambda) to process each tokenized token. We can directly do that implementation there as

for (size_t i = 0; i < input_rows_count; ++i)
{
    std::string_view input = col_input.getDataAt(i);
    col_result[i] = 0;

    forEachToken(*tokenizer, input.data(), input.size(), [...](...) {
       ...
    });
}

but I prefer to extract the matching logic to its own place and keep the implementation simpler. So, in any case we would need a lambda.

[ahmadov]

My question with this is how is this different from preexisting functions like multiSearch or substring family of functions?

@Ergus, both multiSearch or substring search functions do not remove tokenizer separators in between. see these tests:

ClickHouse/tests/queries/0_stateless/02346_function_matchPhrase.sql

Lines 49 to 53 in 31f5790

	SELECT '-- tokenizer separators in phrase are removed before matching';
	SELECT matchPhrase('error: connection refused', 'error---connection');
	SELECT matchPhrase('error: connection refused', 'error:connection');
	SELECT matchPhrase('one two three', 'one...two...three');
	SELECT matchPhrase('one two three', 'one!@#two$%^three');

.

It is similar to hasAllTokens(input, 'token_1 token_2 ... token_n'), but it makes sure the tokens are consecutive.

[clickhouse-gh]

LLVM Coverage Report

Metric	Baseline	Current	Δ
Lines	84.00%	84.00%	+0.00%
Functions	90.90%	90.90%	+0.00%
Branches	76.50%	76.60%	+0.10%

Changed lines: 98.81% (166/168) · Uncovered code

Full report · Diff report