fix(tokenizer): allow $ as PG identifier continuation char

[renecannao]

Summary

• Tokenizer now allows $ as an identifier continuation char in PostgreSQL mode (per PG lexical-syntax docs).
• First-character constraint preserved: $<letter> at token start still emits TK_ERROR.
• MySQL behaviour unchanged.

Why

ProxySQL's set_parser_algorithm=3 path was breaking on inputs like SET search_path = schema$1 — the walker only saw schema and the trailing $1 fell through as a placeholder, producing pgsql-set_parameter_validation_test-t failures. The PG spec explicitly allows $ in unquoted identifiers after the first character.

Test plan

• 4 new tests in tests/test_set.cpp covering: PG mid-ident $, multi-$ idents, PG $<word> still erroring, MySQL behaviour unchanged
• make test — 1258 tests pass, 0 fail
• End-to-end validated in ProxySQL: spliced libsqlparser.a into deps and ran setparser_parsersql_test-t — strict byte-exact tests for SET search_path = schema$1 and SET search_path = my$schema$2_name both pass

Summary by CodeRabbit

Release Notes

• New Features

  • PostgreSQL identifiers can now contain $ characters (after the first character), e.g., schema$1 is recognized as a single identifier.

• Tests

  • Added regression tests for PostgreSQL identifier parsing and MySQL compatibility verification.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3ae83bdd-9bb7-4450-b3e3-992df528adaa

📥 Commits

Reviewing files that changed from the base of the PR and between 9226f87 and 78c5908.

📒 Files selected for processing (2)

• include/sql_parser/tokenizer.h
• tests/test_set.cpp

---

📝 Walkthrough

Walkthrough

The tokenizer's identifier scanning loop was refactored to permit $ as a continuation character in PostgreSQL identifiers only after the first character, enabling tokens like schema$1 to be recognized as single identifiers. Regression tests for PostgreSQL SET parsing and a MySQL guardrail ensure the feature works correctly and does not affect other SQL dialects.

Changes

PostgreSQL $ identifier continuation

Layer / File(s)	Summary
Tokenizer $ continuation logic include/sql_parser/tokenizer.h	The identifier-character loop in scan_identifier_or_keyword computes an is_cont flag and extends PostgreSQL rules to accept $ as a continuation character only when it occurs after the first character, allowing multi-part identifiers like schema$1.
Regression and guardrail tests tests/test_set.cpp	Adds PostgreSQL SET parsing regression tests for $ continuation (e.g., schema$1, my$schema$2_name) and error cases (e.g., reserved $word tokens), plus MySQL guardrail tests to verify $ continuation does not leak into MySQL parsing.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

A tokenizer hops with PostgreSQL cheer,
Allowing dollars mid-identifiers here:
schema$one now scans as a whole,
While MySQL stands guard at the goal. 🐰💰

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/pg-dollar-in-ident

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}