Skip to content

fix!: reject invalid filter input and make the parser correct & concurrency-safe#9

Merged
christiangda merged 1 commit into
mainfrom
fix/filter-parser-validation-correctness
Jul 11, 2026
Merged

fix!: reject invalid filter input and make the parser correct & concurrency-safe#9
christiangda merged 1 commit into
mainfrom
fix/filter-parser-validation-correctness

Conversation

@christiangda

Copy link
Copy Markdown
Contributor

Summary

The filter parser was accepting several classes of invalid input and producing a misleading AST, which undermines its role as a query validator. This PR fixes the core correctness bugs, removes a data race, and modernizes a few idioms. All findings were reproduced with tests before fixing.

Bugs fixed

  1. Trailing / incomplete input silently accepted. Parse stopped at the first complete expression and ignored the rest. Parse now requires the entire input to be consumed.

    Input Before After
    first_name = 'John' garbage ✅ accepted ❌ error
    age > 30 40 ✅ accepted ❌ error
    first_name = 'John') ✅ accepted ❌ error
    1 = 1 ✅ accepted ❌ error
  2. FilterParser was not safe for concurrent use — confirmed data race under go test -race. Per-parse state (lexer, currentToken, errors) moved off the shared struct into a per-call filterParseState. SortParser/FieldsParser were already stateless; now all three can be created once and shared.

  3. Dead IsNot fields → misleading AST. NOT IN / NOT BETWEEN / NOT SIMILAR TO / NOT DISTINCT FROM were wrapped in a UnaryOperatorNode{NOT} around a node whose own IsNot was always false. They now set IsNot = true on the operator node itself (consistent with how RegexMatchNode already worked); NOT LIKE uses TokenOperatorNotLike. Type() and String() now honor negation.

  4. DISTINCT FROM <value> discarded the value. Added DistinctNode.Value, which now holds the compared value.

  5. Stray + lexed to an empty-type token that slipped past the illegal-token check. It now lexes to ILLEGAL.

Idiom / modernization

  • map[string]struct{} sets instead of map[string]any (the previous "avoid allocating" comment was backwards) across all three parsers.
  • FilterParser.Parse returns errors.Join(...) of *QFVFilterError so callers can use errors.As/errors.Is.
  • go directive bumped to 1.25.0 (already pending in the working tree); range-over-int in new tests.

Tests

  • New filter_parser_regression_test.go: trailing tokens, DISTINCT value retention, concurrent reuse (run under -race), and + lexing.
  • Existing AST tests updated to assert the new negation representation.
  • go test -race ./... green; go vet clean; gofmt clean; coverage 80.1%.

⚠️ Breaking changes

This is a pre-1.0 breaking change (suggested tag: v0.1.0). Full migration guide is in the README under "Breaking Changes & Migration". In short:

  • Negated-operator AST shape changed (IsNot on the node; NOT LIKETokenOperatorNotLike).
  • DistinctNode gained a Value field.
  • Parse returns a joined error instead of one formatted string.
  • Inputs previously accepted by mistake are now rejected.

Code that only calls Parse and checks err != nil needs no changes (beyond fixing inputs that were never actually valid).

🤖 Generated with Claude Code

…rrency-safe

The filter parser accepted several classes of invalid input and produced a
misleading AST, undermining its role as a validator. This fixes the core
correctness bugs and modernizes a few idioms.

Fixes:
- Reject trailing/incomplete input. Parse now requires the whole input to be
  consumed; previously it stopped at the first complete expression and silently
  ignored the rest (e.g. "name = 'John' garbage", "age > 30 40", "1 = 1" all
  validated).
- Make FilterParser safe for concurrent use. Per-parse state moved out of the
  shared struct into a per-call filterParseState, removing a data race
  (confirmed under -race). SortParser/FieldsParser were already stateless.
- Negation is now recorded on the operator node. NOT IN / NOT BETWEEN /
  NOT SIMILAR TO / NOT DISTINCT FROM set IsNot=true instead of being wrapped in
  a UnaryOperatorNode(NOT) around a node whose IsNot was always false; NOT LIKE
  uses TokenOperatorNotLike. Type() and String() now honor negation.
- Preserve the DISTINCT FROM value in a new DistinctNode.Value field; it was
  parsed and discarded before.
- Lex a stray '+' as ILLEGAL instead of an empty-type token that slipped past
  the illegal-token check.

Idioms:
- Use map[string]struct{} sets (was map[string]any) across all three parsers.
- FilterParser.Parse returns errors.Join of *QFVFilterError values so callers
  can use errors.As/errors.Is.
- Bump go directive to 1.25.0; use range-over-int in new tests.

Adds regression tests for trailing tokens, DISTINCT value retention, concurrent
reuse (-race), and '+' lexing. Existing AST tests updated for the new negation
representation. A migration guide is documented in the README.

BREAKING CHANGE: The filter AST for negated operators changed (IsNot on the
operator node instead of a wrapping UnaryOperatorNode; NOT LIKE via
TokenOperatorNotLike), DistinctNode gained a Value field, and FilterParser.Parse
returns a joined error. Inputs that were previously accepted by mistake are now
rejected. See the "Breaking Changes & Migration" section in the README.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@christiangda christiangda self-assigned this Jul 11, 2026
@christiangda christiangda merged commit ba8ab7c into main Jul 11, 2026
1 check passed
@christiangda christiangda deleted the fix/filter-parser-validation-correctness branch July 11, 2026 11:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant