Skip to content

feat: add token source position tracking to lexer#22

Open
label-hook[bot] wants to merge 1 commit intomasterfrom
phoenix/issue-17
Open

feat: add token source position tracking to lexer#22
label-hook[bot] wants to merge 1 commit intomasterfrom
phoenix/issue-17

Conversation

@label-hook
Copy link
Copy Markdown

@label-hook label-hook bot commented Apr 7, 2026

Summary

Adds optional token source position tracking to the sqlparse lexer to support use cases like syntax highlighting. This enhancement allows users to track the character position of each token in the original SQL string while maintaining full backwards compatibility.

Changes

  • Enhanced lexer functionality: Added position tracking capability to the tokenization process
  • Backwards compatible API: Existing tokenize() method unchanged, new functionality available through optional parameters
  • Position-aware token processing: Updated filters and processing pipeline to handle tokens with optional position information
  • Flexible token representation: Tokens can now optionally include a third element containing character position data

Key Implementation Details

  • Modified sqlparse/lexer.py to track character positions during tokenization
  • Updated token filters in sqlparse/filters/tokens.py to gracefully handle both 2-tuple and 3-tuple tokens
  • Enhanced sqlparse/engine/filter_stack.py to support position-aware token processing
  • Extended sqlparse/sql.py Token class to optionally store and provide access to position information

Testing

Comprehensive test coverage added to verify:

  • Position tracking accuracy across various SQL constructs
  • Backwards compatibility with existing tokenization behavior
  • Proper handling of edge cases (empty strings, whitespace, multi-line SQL)
  • Filter pipeline compatibility with both token formats
  • Position information preservation through the parsing process

All existing tests continue to pass, ensuring no breaking changes to the current API.

Closes #17


Closes #17

@label-hook label-hook bot added the ai:review Phoenix AI: PR ready for review label Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:review Phoenix AI: PR ready for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tracking token source positions

1 participant