Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error with BigQuery data connector and RuntimeDataConnector #2959

Closed
fosk06 opened this issue Jun 25, 2021 · 1 comment · Fixed by #3219
Closed

Error with BigQuery data connector and RuntimeDataConnector #2959

fosk06 opened this issue Jun 25, 2021 · 1 comment · Fixed by #3219

Comments

@fosk06
Copy link

fosk06 commented Jun 25, 2021

Environment:
Python 3.7.8
great expectation version : 0.13.20
MacOS

I'm trying to do some expectations on a batch of data from a BigQuery dataset/table.
Some of the expectations that computes metrics are failing.

Error message:

provider.py", line 57, in inner_func
return metric_fn(*args, **kwargs)
File "/Users/tTrividic/code/python/data_quality_exploration/venv/lib/python3.7/site-packages/great_expectations/expectations/metrics/map_metric.py", line 397, in inner_func
_metrics=metrics,
File "/Users/tTrividic/code/python/data_quality_exploration/venv/lib/python3.7/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_unique.py", line 65, in _sqlalchemy_window
if sql_engine and dialect and dialect.dialect.name == "mysql":
AttributeError: module 'pybigquery.sqlalchemy_bigquery' has no attribute 'dialect'

the error come from the file : great_expectations/expectations/metrics/column_map_metrics/column_values_unique.py
Line 65 the if condition is failing, commenting the whole if block solve the error.

Here is the code i use:

Datasource = biggquery
datasource_config = { "name": "bigquery_datasource", "class_name": "Datasource", "execution_engine": { "class_name": "SqlAlchemyExecutionEngine", "connection_string": "{0}".format(connection_string), }, "data_connectors": { "runtime_data_connector": { "class_name": "RuntimeDataConnector", "batch_identifiers": ["default_identifier_name"], } }, } context.add_datasource(**datasource_config) batch_request = RuntimeBatchRequest( datasource_name="addons_bigquery_datasource", data_connector_name="runtime_data_connector", data_asset_name="addons_customer", runtime_parameters={"query": "SELECT * FROM myproject.bronze_addons.ps_customer limit 50"}, batch_identifiers={"default_identifier_name": "something_something"}, batch_spec_passthrough={ "bigquery_temp_table": "ge_temp" } ) expectation_suite_name = "bronze_addons.ps_customer" context.create_expectation_suite( expectation_suite_name=expectation_suite_name, overwrite_existing=True ) validator = context.get_validator( batch_request=batch_request, expectation_suite_name=expectation_suite_name ) validator.expect_column_values_to_be_unique(column='id_customer') # this line trigger the bug

@talagluck
Copy link
Contributor

Thanks for submitting, @fosk06 and apologies for delays! We will review internally and be in touch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants