Skip to content

Conversation

@senthilb-devrev
Copy link
Contributor

@senthilb-devrev senthilb-devrev commented Nov 12, 2025

Problem Statement
When exporting query results containing resolved array fields to CSV using DuckDB's COPY command, array elements were not properly quoted, resulting in invalid JSON format

Before:

"Owners - Display Name"
"[Alice Smith, Bob Jones]"

This format is problematic because:
Array elements lack quotes, making it invalid JSON
Cannot be parsed by standard JSON parsers
Breaks downstream CSV consumers expecting proper JSON arrays
Inconsistent with JSON standards for string arrays

Root Cause:
DuckDB's ARRAY_AGG() function, when exported to CSV via COPY, serializes arrays as plain text without proper JSON escaping. While in-memory query results return native JavaScript arrays, the CSV serialization layer converts them to string representation without quotes around individual elements.

Solution
Wrapped ARRAY_AGG() calls with to_json() in the aggregation step to ensure proper JSON formatting:

// Before
const aggregationFn = isArrayColumn
  ? `COALESCE(ARRAY_AGG(DISTINCT ${columnRef}) FILTER (WHERE ${columnRef} IS NOT NULL), [])`
  : `MAX(${columnRef})`;

// After
const aggregationFn = isArrayColumn
  ? `to_json(COALESCE(ARRAY_AGG(DISTINCT ${columnRef}) FILTER (WHERE ${columnRef} IS NOT NULL), []))`
  : `MAX(${columnRef})`;

Result:

"Owners - Display Name"
"[Alice Smith, Bob Jones]"
"Owners - Display Name"
"[""Alice Smith"",""Bob Jones""]"

The csv readers will read this and serialize it as "["Alice Smith"",""Bob Jones"]"

https://app.devrev.ai/devrev/works/ISS-219002

@senthilb-devrev senthilb-devrev merged commit bd83abc into main Nov 12, 2025
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants