feat: TEXT field search with exact phrase, fuzzy, proximity, OR/union, scoring

[nkanu17]

Summary

Comprehensive TEXT field search overhaul: fixes exact phrase matching (= operator) and adds the full RediSearch text search surface area, including fuzzy LD levels, suffix/infix matching, OR/union, proximity search, BM25 scoring, and special character escaping.

Feature Reference

Feature	SQL Syntax	RediSearch Output	Notes
Exact phrase	WHERE name = 'bank of red'	@name:"bank red"	Stopwords stripped (not indexed by Redis)
Negated phrase	WHERE name != 'test'	-@name:"test"
Tokenized search	WHERE fulltext(title, 'gaming laptop')	@title:(gaming laptop)	AND semantics
OR / union	WHERE fulltext(title, 'laptop OR tablet')	@title:(laptop|tablet)	Terms escaped
Prefix	WHERE title LIKE 'lap%'	@title:lap*	Unchanged
Suffix	WHERE title LIKE '%phone'	@title:*phone	New
Contains / infix	WHERE title LIKE '%phone%'	@title:*phone*	New
Fuzzy LD=1	WHERE fuzzy(title, 'laptap')	@title:%laptap%	Default level
Fuzzy LD=2	WHERE fuzzy(title, 'laptap', 2)	@title:%%laptap%%
Fuzzy LD=3	WHERE fuzzy(title, 'laptap', 3)	@title:%%%laptap%%%	Max LD
Proximity (slop)	WHERE fulltext(title, 'a b', 2)	@title:(a b) => { $slop: 2; }
Proximity + inorder	WHERE fulltext(title, 'a b', 2, true)	@title:(a b) => { $slop: 2; $inorder: true; }
BM25 scoring	SELECT score() AS s FROM ...	WITHSCORES SCORER BM25	Default scorer
Custom scorer	SELECT score('TFIDF') AS s FROM ...	WITHSCORES SCORER TFIDF
Score-only SELECT	SELECT score() AS s FROM ...	RETURN 0 + WITHSCORES	No payload leak
Verbatim	ParsedQuery.verbatim = True	VERBATIM	API-only flag
Nostopwords	ParsedQuery.nostopwords = True	NOSTOPWORDS	API-only flag

Problem

WHERE name = 'bank of red' previously stripped stopwords and produced @name:(bank red) as a tokenized query, matching any document containing "bank" and "red" anywhere rather than the exact phrase. The = operator now produces a proper double-quoted exact phrase query (@name:"bank red"), with stopwords stripped to match how RediSearch indexes text (stopwords are not stored in the inverted index and their positions are not preserved). This was the entry point; the PR grew to cover all missing RediSearch text search features.

Key Implementation Details

Stopword handling

Both = (exact phrase) and fulltext() (tokenized search) strip default Redis stopwords before sending queries to RediSearch. This is necessary because RediSearch does not index stopwords, so including them in queries causes syntax errors or failed matches. A UserWarning is emitted when stopwords are removed. Users who need stopwords indexed can create their index with STOPWORDS 0.

Escaping

All text search operators escape RediSearch-reserved characters (|, -, (, ), @, ~, ", \) via _escape_fulltext_term(). OR operands are individually escaped before joining with |.

Score parsing

WITHSCORES + RETURN 0 changes the Redis response stride from 3 to 2 (no field arrays). Both sync and async executors handle this via _ScoreParseMixin._has_return_0(). Score alias collision with document fields is detected across all rows and auto-prefixed with __score_. Bytes field keys are normalized to strings for consistent collision detection regardless of decode_responses setting.

Validation

• Fuzzy level must be 1, 2, or 3
• Slop must be a non-negative integer
• score() + GROUP BY (FT.AGGREGATE) raises ValueError
• fulltext(), fuzzy(), and score() arguments must be literals (placeholders allowed for parameter substitution)

Double-negation normalization

NOT (field != 'x') resolves to @field:"x" instead of double-negating.

Files Changed

File	Change
sql_redis/query_builder.py	Exact phrase with stopword stripping for =/!=, fuzzy levels, suffix/infix, OR, proximity, escaping
sql_redis/parser.py	fulltext(f, v, slop, inorder), fuzzy(f, v, level), score() in SELECT, argument validation
sql_redis/analyzer.py	Routes FULLTEXT/FUZZY conditions, scoring spec
sql_redis/translator.py	Wires fuzzy_level/slop/inorder, emits WITHSCORES/SCORER, double-negation normalization
sql_redis/executor.py	_ScoreParseMixin for WITHSCORES+RETURN0 response parsing, score alias collision guard with bytes normalization
tests/test_query_builder.py	All new operators, escaping, fuzzy levels, proximity, OR, stopword stripping
tests/test_translator.py	End-to-end: fuzzy LD 2/3, suffix/infix, OR, proximity, scoring
tests/test_sql_parser.py	Parser tests for fulltext/fuzzy args, score(), slop validation

Tests

• 456 tests pass (all unit + integration)
• make lint clean (black, isort)

Breaking Changes

= on TEXT fields now produces @field:"value" (double-quoted exact phrase) instead of @field:value (tokenized). Stopwords are stripped from the phrase to match how RediSearch indexes text. Callers relying on the old tokenized behavior should switch to fulltext().

[nkanu17]

@codex review

[Copilot]

Pull request overview

Updates TEXT equality semantics in the SQL→RediSearch translation layer so = on TEXT fields uses exact phrase matching (quoted) to preserve stopwords, addressing cases like "bank of red" where tokenization/stopword stripping changes meaning.

Changes:

• Change TEXT = (and !=) query generation to wrap values in quotes for exact phrase semantics.
• Keep MATCH as tokenized search, using parenthesized multi-term syntax and stopword warnings.
• Update/expand tests to reflect the new query syntax and error-message variability across Redis versions.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
sql_redis/query_builder.py	Adjusts TEXT condition building to quote =/!= values and changes multi-word handling for tokenized search.
tests/test_translator.py	Updates translator expectations for TEXT = to produce quoted phrase queries.
tests/test_query_builder.py	Updates/adds QueryBuilder TEXT tests for exact phrase behavior and MATCH multi-word behavior.
tests/test_parameter_substitution.py	Relaxes empty-string TEXT error assertion to accommodate Redis version differences.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 👍

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[nkanu17]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🎉

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

sql_redis/query_builder.py:90

• The multi-field branch returns early using the raw value, which bypasses the operator-specific formatting/escaping and also drops negation handling. This reintroduces incorrect behavior for =/!= (missing quotes / stopword preservation), and multi-word values won’t use the new parenthesized MATCH semantics. Consider computing search_value first (including escaping/stopword filtering) and then applying it to both single- and multi-field cases, and include prefix for negated / != queries (e.g., -(@f1|f2:...)).

        # Derive negation from both the flag and the operator itself,
        # consistent with how build_tag_condition handles != via operator.
        prefix = "-" if negated or operator == "!=" else ""

        # Handle multi-field search
        if isinstance(field, list):
            field_str = "|".join(field)
            return f"(@{field_str}:{value})"

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[nkanu17]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

sql_redis/query_builder.py:90

• The multi-field early return bypasses all operator handling (including the new exact-phrase quoting/escaping for =/!=) and also ignores negated/!= when building the prefix. If multi-field search is part of the public API, it should apply the same operator semantics as the single-field path (or explicitly disallow operators other than FULLTEXT/MATCH and raise).

        # Derive negation from both the flag and the operator itself,
        # consistent with how build_tag_condition handles != via operator.
        prefix = "-" if negated or operator == "!=" else ""

        # Handle multi-field search
        if isinstance(field, list):
            field_str = "|".join(field)
            return f"(@{field_str}:{value})"

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[nkanu17]

@copilot review

[nkanu17]

@copilot review

[nkanu17]

@copilot review

[Copilot]

@copilot review

Addressed all three open review threads in commit 506307b:

1. MATCH/FULLTEXT consistency — removed all "MATCH" operator references from tests and the docstring. build_text_condition() now documents only =, !=, FULLTEXT, LIKE, FUZZY.

2. RediSearch special-char escaping — added TEXT_QUERY_SPECIAL_CHARS = frozenset({"\\", "-", "@", "|", "(", ")"}) and _escape_fulltext_term() classmethod. Applied to all FULLTEXT single-term, multi-word, and OR paths to prevent characters like - (e.g., "anti-virus") or @ in user data from altering query semantics. Intentional RediSearch features (~, *, %, ^) are excluded from escaping. New test test_text_fulltext_special_chars_escaped verifies "anti-virus" → @description:anti\-virus.

3. Black formatting — ran black on tests/test_sql_queries.py to fix pre-existing f-string formatting issues in a file I modified in a prior commit.

@copilot review

Same changes apply — see reply above (commit 506307b).

[nkanu17]

@copilot review