Conversation
…tches When pgx receives a map[string]any for JSONB columns, it re-serializes using encoding/json, which can produce different output than Sonic (used to parse wal2json). This mismatch causes 'invalid input syntax for type json' errors on the target database for complex JSONB with Unicode, emojis, or special escapes. Fix: Pre-serialize JSONB/JSON columns to []byte using Sonic in filterRowColumns() before passing to pgx. This ensures consistent encoding throughout the pipeline.
- Only serialize map[string]any and []any types with Sonic - Pass string values through unchanged (fixes schemalog snapshot) - Add comprehensive tests for all JSONB value types - Compact code for production readiness
eminano
force-pushed
the
jsonb-fix-for-upstream
branch
from
d2a1978 to
b9a4985
Compare
Merging this branch will increase overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. Changed unit test files
|
There was a problem hiding this comment.
Pull request overview
This PR fixes a JSON encoding mismatch issue where JSONB columns were causing 'invalid input syntax for type json' errors in the target database. The problem occurred because pgx was re-serializing JSONB values using encoding/json, which could produce different output than Sonic (used to parse wal2json), especially for complex JSONB with Unicode, emojis, or special escapes.
Changes:
- Pre-serialize JSONB/JSON map and slice values using Sonic before passing to pgx
- Add comprehensive tests for JSONB serialization with Unicode, emojis, and special characters
- Ensure string values pass through unchanged to avoid double-encoding
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| pkg/wal/processor/postgres/postgres_wal_dml_adapter.go | Added serializeJSONBValue function to pre-serialize JSONB/JSON columns with Sonic, integrated into filterRowColumns and buildWhereQuery |
| pkg/wal/processor/postgres/jsonb_serialization_test.go | Added comprehensive tests for JSONB handling including maps, arrays, strings, and WHERE clauses with Unicode and special characters |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Appreciate you guys being able to get this in! Had very little time on my end to clean it up so much appreciated :) |
2 participants
Same as #685 with linting fixed.
Thanks @VikramChennai 🙏