Skip to content

Snowflake: decode octal/hex/unicode string-literal escapes - #16

Closed
wojpadlo wants to merge 1 commit into
lav-1362-masking-policyfrom
lav-1329-string-literal-escape-parity
Closed

Snowflake: decode octal/hex/unicode string-literal escapes#16
wojpadlo wants to merge 1 commit into
lav-1362-masking-policyfrom
lav-1329-string-literal-escape-parity

Conversation

@wojpadlo

Copy link
Copy Markdown

Decodes Snowflake single-quoted string-literal escapes in the tokenizer:

  • \ooo (octal), \xhh (hex) and \uhhhh (unicode) → the corresponding code point
  • \0 stays NUL
  • unrecognized letters (\a, \Z, …) collapse to the bare letter
  • malformed \x / \u raise a tokenizer error

Gated behind a new supports_snowflake_string_literal_escapes dialect capability so other backslash-escaping dialects keep their MySQL/BigQuery semantics.

Files: src/tokenizer.rs (+192/-5), src/dialect/mod.rs, src/dialect/snowflake.rs, tests/sqlparser_common.rs.

Why

This is the parser-fork side of LAV-1329 ("string-literal escape parity"). It is a prerequisite for LAV-1330 (multi-character FIELD_DELIMITER / RECORD_DELIMITER on the CSV load path): that task requires a hex-escaped delimiter value such as FIELD_DELIMITER = '\x7C\x7C' to decode to ||. The currently pinned fork rev (b5e11f27) maps unknown escapes to the bare char (\x7Cx7C), so the emulator never sees ||. Real Snowflake decodes it.

Stacking

Stacked on top of #14 (lav-1362-masking-policy), matching the existing PR chain (#15#14 → …). Diff against this base is the single escape-parity commit. Once this and its ancestors land in main, the downstream snowflake-rs Cargo.toml sqlparser pin must be bumped to consume it before LAV-1330 can proceed.

🤖 Generated with Claude Code

Decode \ooo, \xhh and \uhhhh as Unicode code points, keep \0 as NUL,
collapse unrecognized letters (\a, \Z) to the bare letter, and raise a
tokenizer error on malformed \x / \u — gated behind a new
supports_snowflake_string_literal_escapes dialect capability so other
backslash-escaping dialects keep MySQL/BigQuery semantics.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@wojpadlo

Copy link
Copy Markdown
Author

Superseded by #17, which rebases the same escape-parity commit directly onto main. This PR was stacked on lav-1362-masking-policy, whose ancestors already have red CI (fmt in ast/mod.rs, clippy in parser/mod.rs, and CreateTable missing-field errors in the duckdb/mssql tests) — none of it from the escape change. #17 decouples it so it can go green and merge independently.

@wojpadlo wojpadlo closed this Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant