Skip to content

Fix: normalize empty strings consistently across MySQL, PostgreSQL, and Snowflake - #4

Open
vmatt wants to merge 2 commits into
masterfrom
empty-string
Open

Fix: normalize empty strings consistently across MySQL, PostgreSQL, and Snowflake#4
vmatt wants to merge 2 commits into
masterfrom
empty-string

Conversation

@vmatt

@vmatt vmatt commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Problem

Cross-database diffs produced spurious differences on string columns where one side stored '' (empty string) and the other stored NULL.

MySQL and PostgreSQL treat empty string and NULL as distinct values. Snowflake can store genuine '' but also sometimes stores NULL for the same logical "no value" — depending on how data was loaded. When reladiff hashed rows, '' and NULL produced different hashes, causing false diffs.

Example false diff on a text/varchar column:

+ (None, '393665', ..., 'Developer Support', 'O', '0', '',   ...)
- (None, '393665', ..., 'Developer Support', 'O', '0', None, ...)

Root cause

normalize_value_by_type dispatches StringType columns to normalize_text(), which defaulted to a plain to_string() cast in all dialects. No normalization was applied to empty strings before hashing.

Fix

Added a normalize_text hook to AbstractMixin_NormalizeValue (in sqeleton/abcs/mixins.py) with a passthrough default. MySQL, PostgreSQL, and Snowflake all override it to emit NULLIF(<value>, ''), so empty strings are treated as NULL before hashing.

  • MySQL (sqeleton/databases/mysql.py): NULLIF(cast({value} as char), '')
  • PostgreSQL (sqeleton/databases/postgresql.py): NULLIF({value}::varchar, '')
  • Snowflake (sqeleton/databases/snowflake.py): NULLIF(cast({value} as string), '')

Oracle is intentionally excluded — it auto-converts '' to NULL at storage time, so Oracle already behaves consistently and needs no override.

This makes all four cases match correctly across any pair of these databases:

Side A Side B After normalization Result
'' NULL NULL / NULL match
NULL '' NULL / NULL match
'' '' NULL / NULL match
NULL NULL NULL / NULL match

Files changed

  • sqeleton/abcs/mixins.py — added normalize_text() hook + StringType dispatch in normalize_value_by_type()
  • sqeleton/databases/mysql.pynormalize_text override: NULLIF(cast(... as char), '')
  • sqeleton/databases/postgresql.pynormalize_text override: NULLIF(... ::varchar, '')
  • sqeleton/databases/snowflake.pynormalize_text override: NULLIF(cast(... as string), '')

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant