Skip to content

v0.3.6 - Snowflake → local Parquet streaming + capability cleanup

Choose a tag to compare

View all tags
@hfmsio hfmsio released this 04 May 16:00
· 104 commits to main since this release
v0.3.6
b2243b5

Snowflake → local Parquet streaming

  • Cloud-streaming export strategy: Snowflake/BigQuery → re-query the source, pipe each yielded chunk through DuckDB-WASM's Parquet writer. Constant-memory profile per chunk; the result never materializes in JS heap.
  • True chunked Parquet write: each chunk → its own chunk_N.parquet (no temp table buffering). Final merge via COPY (read_parquet([...])) TO 'final.parquet'.
  • INSERT path uses read_json on a registered virtual JSON file (~20-50x faster than the prior SQL VALUES escape-and-parse path).

Connector capability cleanup

  • New ParquetExportCapable interface + isParquetExportCapable typeguard in @ide/connectors. Replaces 4 structural casts across the app.
  • New CloudProxyUnavailableError typed throw on 501 cloud_proxy_unavailable responses.
  • Strategy registry under apps/web-client/src/components/table/exporters/ picks via capability check, not connector-name match.
  • setActiveConnector swap-and-restore retired from operations - documented as user-preference state only.

Performance + UX

  • Snowflake JSON-partition parsing moved to a Web Worker for partitions ≥ 2000 rows. Main thread stays responsive during 500k+ row queries and exports.
  • Drag-drop file upload uses zero-copy registerFileHandle for parquet/csv/json. 600k-row Parquet drops/uploads in milliseconds instead of seconds.
  • Executing-query overlay stays alive for the full streaming-mode fetch.
  • Streaming-mode useTableData no longer auto-reruns on connector switch (DuckDB ↔ Snowflake toggle no longer kicks off credit-burning warehouse queries).

Bug fixes

  • DuckDB Promise executor missed reject binding; parser/runtime errors silently disappeared. Added unit tests covering the error path.
  • Console flood of "Unknown Snowflake type: Utf8" warnings: per-render log demoted to debug + Arrow-type fallthrough in normalizers.
  • /api/* paths in npx dbxlite-ui no longer return index.html; now return 501 + JSON error.

Removed

  • Dead useResultExport hook + ResultPane.tsx (1500+ lines). Two parallel exporters guaranteed drift; only PaginatedTable + useTableExport were wired to production.

Tests: 995 passing, 10 documented skips.

Assets 2
Loading