allows nested types in BigQuery via autodetect_schema #1591
Conversation
✅ Deploy Preview for dlt-hub-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
rudolfix
force-pushed
the
feat/allows-autodetect-schema-bigquery
branch
from
e744b4b
to
b5df237
Compare
| table = conn.execute(f"SELECT * FROM read_json_auto('{json_file_path}')").fetch_arrow_table() | ||
| ``` | ||
|
|
||
| Note that **duckdb** and **pyarrow** methods will generate [nested types](#loading-nested-types) for nested data, which are only partially by `dlt`. |
There was a problem hiding this comment.
there is a word missing here, probably "supported"
|
@rudolfix the PR looks good, but I have a question. Are you sure that this schema auto detection really creates the schema the exact same way in bigquery as our explicit code would (with the exception of the complex fields of course)? At least this is what I understand it says in the docs. I think I would at least add a test that checks all data types or something like that. |
|
Or maybe it would be possible to only have the complex fields be autodetected and run the other migrations the way we always do, but I am actually not sure this makes sense. |
|
@sh-rp the schema is identical in case of parquet (because parquet schema is source of truth both for bigquery and dlt) - with the exception of JSON documents are inferred differently and our schema does not match 100% regarding the types and column order. What is unfortunate - I can't create some columns with dlt and some (complex) with BigQuery because it complains that the schema is different. I do not think the problem is too great. The biggest difference so far is anyway this is a trick for power user until we do #1592 is done |
|
@rudolfix can we do this with Redshift SUPER type? Loading from jsonl. |
Description
This PR allows users to offload schema inference and evolution (type inference from data, create and alter tables) to BigQuery.
As a consequence nested data types may be created (if present in arrow tables/parquet or inferred from json).
Besides being useful in itself it is a PoC of how nested types support may look in
dlt.max_table_nestingstill works - user has a choice how fardltshould decompose json into tables and when to switch to nested types