Support of direct query to BQ

[khorshuheng]

Databrick's forked version of the connector supports direct query to BQ, for the purpose of reducing data transfer:
https://databricks.com/blog/2020/07/31/announcing-support-for-google-bigquery-in-databricks-runtime-7-1.html

Would this feature be eventually supported by the connector? Or is there no plan for such feature at the moment?

[davidrabinowitz]

We are actively looking at it and will have an update soon.

[chimerasaurus]

To echo the comment above it's something that is on our roadmap and we're planning on doing it.

Having said that, it's also unfortunate that a private fork chose to effectively fork and close-source an improvement, which isn't very open-source friendly in my opinion.

[jtvmatos]

Apparently, databricks implementation is an extension of the "BigQuery views" support.
When a query is add to parameters, a materialization of the result generate a BQ table with same "sbc******" nomenclature and expiration date in the viewMaterializationProject.

Is there an expected date for this feature?

[davidrabinowitz]

Fixed in version 0.19.0, see https://github.com/GoogleCloudDataproc/spark-bigquery-connector#reading-data-from-a-bigquery-query.