Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Bug: Metabase truncates CSV/xlsx downloads with large data volume. #6960
Metabase truncates CSV/xlsx downloads with large data volume.
Minimal required SQL
#standardsql select * from `bigquery-public-data.samples.github_timeline` limit 20000
referenced this issue
Feb 4, 2019
We're seeing the same issue when trying to download large-ish .csv files (~77k rows, well under the 1 million row limit for CSVs).
Digging into the BigQuery driver (bigquery.clj) it looks like the
(log/info (u/format-color 'cyan (str "BigQuery QueryResponse details, Total rows: " (.getTotalRows response) " Fetched Rows: " (.size (.getRows response)) " Page Token: " (.getPageToken response))))
and I'm seeing fetched rows less than the total rows available when downloading full results via .csv:
03-05 13:44:03 INFO driver.bigquery :: BigQuery QueryResponse details, Total rows: 77699 Fetched Rows: 31818 Page Token: BFSFOYSPNEAQAAASA4EAAEEAQCAAKGQIBDFPQAIQUCGQMIFQVYKQ====
This issue is a blocker for BigQuery - even when results are limited by the UI's imposed 10k limit. Depending on the content of your results, it is entirely feasible for a result set with less than 10k rows getting truncated.