sql: add ClickHouse Map type support for sql_insert#4178
sql: add ClickHouse Map type support for sql_insert#4178josephwoodward merged 3 commits intoredpanda-data:mainfrom
Conversation
The ClickHouse driver requires map arguments to have concrete Go types matching the column schema (e.g. map[string]int64), but Bloblang args_mapping produces generic map[string]any values. This causes inserts to fail for any column using ClickHouse Map types. Introduce an argsConverter for ClickHouse that introspects column scan types at connect time via a `SELECT ... LIMIT 0` query, then normalizes map and slice arguments through a JSON round-trip into the expected Go types before passing them to the driver. Applied to both sql_insert output and processor components. Includes integration tests covering common map types.
|
|
|
In reference to #3819 |
|
Full disclosure, I used Claude to assist with this patch as I don't normally write in Go. Feel free to make any changes to conform to the project's standards. |
|
Hi @rickysaltzer, thank you for your pull request. There's a number of linter violations that would be good to clean up and I've added a few comments. If you are using Claude then you can also use the Claude skill here to review the pull request which can be pretty helpful. |
- Removes unused regex based approach, as this was originally intended for `sql_raw`, but felt was better left to a future PR. - Add logging to normalization failures
|
Hey @josephwoodward - Thanks for such a quick review, I've sent in some changes to hopefully address your comments. |
|
Commits
Review LGTM once the commits are squashed. |
|
Thanks for addressing the comments @rickysaltzer. I see the linter is complaining about something, if you wouldn't mind addressing that when you have a moment that'd be great. Also, if you rebase your changes on the main branch the tests should go green. |
|
Hey @josephwoodward - linting should be taken care of if you want to approve the workflows to check. |
|
Thank you for your contribution @rickysaltzer, your change is now available in v4.86.0 |

The ClickHouse driver requires map arguments to have concrete Go types matching the column schema (e.g. map[string]int64), but Bloblang args_mapping produces generic map[string]any values. This causes inserts to fail for any column using ClickHouse Map types.
Introduce an argsConverter for ClickHouse that introspects column scan types at connect time via a
SELECT ... LIMIT 0query, then normalizes map and slice arguments through a JSON round-trip into the expected Go types before passing them to the driver.Applied to both sql_insert output and processor components. Includes integration tests covering common map types.