fixing json conversion and adding tests #168
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem Statement
When exporting query results containing resolved array fields to CSV using DuckDB's COPY command, array elements were not properly quoted, resulting in invalid JSON format
Before:
This format is problematic because:
Array elements lack quotes, making it invalid JSON
Cannot be parsed by standard JSON parsers
Breaks downstream CSV consumers expecting proper JSON arrays
Inconsistent with JSON standards for string arrays
Root Cause:
DuckDB's ARRAY_AGG() function, when exported to CSV via COPY, serializes arrays as plain text without proper JSON escaping. While in-memory query results return native JavaScript arrays, the CSV serialization layer converts them to string representation without quotes around individual elements.
Solution
Wrapped ARRAY_AGG() calls with to_json() in the aggregation step to ensure proper JSON formatting:
Result:
The csv readers will read this and serialize it as "["Alice Smith"",""Bob Jones"]"
https://app.devrev.ai/devrev/works/ISS-219002