Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HUDI-6857] Update BigQuerySyncTool docs #9710

Merged
merged 2 commits into from
Sep 14, 2023
Merged

[HUDI-6857] Update BigQuerySyncTool docs #9710

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fd0a290
Update documentation around big query sync tool
the-other-tim-brown Sep 13, 2023
5a3330c
update based on feedback
the-other-tim-brown Sep 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
37 changes: 29 additions & 8 deletions website/docs/gcp_bigquery.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,27 @@ summary: Introduce BigQuery integration in Hudi.
---

Hudi tables can be queried from [Google Cloud BigQuery](https://cloud.google.com/bigquery) as external tables. As of
now, the Hudi-BigQuery integration only works for hive-style partitioned Copy-On-Write tables.
now, the Hudi-BigQuery integration only works for hive-style partitioned Copy-On-Write and Read-Optimized Merge-On-Read tables.

## Sync Modes
### Manifest File
As of version 0.14.0, the `BigQuerySyncTool` supports syncing table to BigQuery using [manifests](https://cloud.google.com/blog/products/data-analytics/bigquery-manifest-file-support-for-open-table-format-queries). On the first run, the tool will create a manifest file representing the current base files in the table and a table in BigQuery based on the provided configurations. The tool produces a new manifest file on each subsequent run and will update the schema of the table in BigQuery if the schema changes in your Hudi table.
nsivabalan marked this conversation as resolved.
Show resolved Hide resolved
#### Benefits of using the new manifest approach:
<ol>
<li>Only the files in the manifest can be scanned leading to less cost and better performance for your queries</li>
<li>The schema is now synced from the Hudi commit metadata allowing for proper schema evolution</li>
<li>Lists no longer have unnecessary nesting when querying in BigQuery as list inference is enabled by default</li>
<li>Partition column no longer needs to be dropped from the files due to new schema handling improvements</li>
</ol>

To enable this feature, set `hoodie.gcp.bigquery.sync.use_bq_manifest_file` to true.

### View Over Files (Legacy)
This is the current default behavior to preserve compatibility as users upgrade to 0.14.0 and beyond.
After run, the sync tool will create 2 tables and 1 view in the target dataset in BigQuery. The tables and the view
nsivabalan marked this conversation as resolved.
Show resolved Hide resolved
share the same name prefix, which is taken from the Hudi table name. Query the view for the same results as querying the
Copy-on-Write Hudi table.
**NOTE:** The view can scan all of the parquet files under your table's base path so it is recommended to upgrade to the manifest based approach for improved cost and performance.

## Configurations

Expand All @@ -17,16 +37,21 @@ setting sync tool class. A few BigQuery-specific configurations are required.
| `hoodie.gcp.bigquery.sync.project_id` | The target Google Cloud project |
| `hoodie.gcp.bigquery.sync.dataset_name` | BigQuery dataset name; create before running the sync tool |
| `hoodie.gcp.bigquery.sync.dataset_location` | Region info of the dataset; same as the GCS bucket that stores the Hudi table |
| `hoodie.gcp.bigquery.sync.source_uri` | A wildcard path pattern pointing to the first level partition; make sure to include the partition key. |
| `hoodie.gcp.bigquery.sync.source_uri` | A wildcard path pattern pointing to the first level partition; partition key can be specified or auto-inferred. Only required for partitioned tables |
| `hoodie.gcp.bigquery.sync.source_uri_prefix` | The common prefix of the `source_uri`, usually it's the path to the Hudi table, trailing slash does not matter. |
| `hoodie.gcp.bigquery.sync.base_path` | The usual basepath config for Hudi table. |
| `hoodie.gcp.bigquery.sync.use_bq_manifest_file` | Set to true to enable the manifest based sync |

Refer to `org.apache.hudi.gcp.bigquery.BigQuerySyncConfig` for the complete configuration list.

In addition to the BigQuery-specific configs, set the following Hudi configs to write the Hudi table in the desired way.
### Partition Handling
In addition to the BigQuery-specific configs, you will need to use hive style partitioning for partition pruning in BigQuery. On top of that, the value in partition path will be the value returned for that field in your query. For example if you partition on a time-millis field, `time`, with an output format of `time=yyyy-MM-dd`, the query will return `time` values with day level granularity instead of the original milliseconds so keep this in mind while setting up your tables.

```
hoodie.datasource.write.hive_style_partitioning = 'true'
```

For the view based sync you must also specify the following configurations:
```
hoodie.datasource.write.drop.partition.columns = 'true'
hoodie.partition.metafile.use.base.format = 'true'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see this option in the hudi docs so I removed it. Also we don't use this internally when running the sync tool. Anyone know what this may have been for?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah. each hudi partition has a metadata file called .hoodie_partition_metadata. this has to be in parquet format just for BQ sync purpose. for other query engines or metastores, we did not have this requirement.
So, this meta file will be just a text file in general. but if user prefers to sync to BQ, the recommendation is to enable this

```
Expand Down Expand Up @@ -74,7 +99,3 @@ spark-submit --master yarn \
--hoodie-conf hoodie.partition.metafile.use.base.format=true \
--hoodie-conf hoodie.metadata.enable=true \
```

After run, the sync tool will create 2 tables and 1 view in the target dataset in BigQuery. The tables and the view
share the same name prefix, which is taken from the Hudi table name. Query the view for the same results as querying the
Copy-on-Write Hudi table.