Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bigquery regex unsupported dialect #3131

Closed
jdimatteo opened this issue Jul 28, 2021 · 3 comments · Fixed by #3219
Closed

bigquery regex unsupported dialect #3131

jdimatteo opened this issue Jul 28, 2021 · 3 comments · Fixed by #3219

Comments

@jdimatteo
Copy link
Contributor

Describe the bug
When running regex expectations with bigquery, exceptions like the following occur:

Traceback (most recent call last):
  File "/home/jdimatteo/dev/ge_tutorials_fork/expect_column_values_to_not_match_regex/bigquery_python_example.py", line 61, in <module>
    validator.expect_column_values_to_match_regex(column="station_id", regex="[at]+")
  File "/home/jdimatteo/dev/ge_tutorials_fork/expect_column_values_to_not_match_regex/venv/lib/python3.9/site-packages/great_expectations/validator/validator.py", line 280, in inst_expectation
    raise err
  File "/home/jdimatteo/dev/ge_tutorials_fork/expect_column_values_to_not_match_regex/venv/lib/python3.9/site-packages/great_expectations/validator/validator.py", line 235, in inst_expectation
    validation_result = expectation.validate(
  File "/home/jdimatteo/dev/ge_tutorials_fork/expect_column_values_to_not_match_regex/venv/lib/python3.9/site-packages/great_expectations/expectations/expectation.py", line 635, in validate
    evr = validator.graph_validate(
  File "/home/jdimatteo/dev/ge_tutorials_fork/expect_column_values_to_not_match_regex/venv/lib/python3.9/site-packages/great_expectations/validator/validator.py", line 474, in graph_validate
    metrics = self.resolve_validation_graph(graph, metrics, runtime_configuration)
  File "/home/jdimatteo/dev/ge_tutorials_fork/expect_column_values_to_not_match_regex/venv/lib/python3.9/site-packages/great_expectations/validator/validator.py", line 515, in resolve_validation_graph
    self._resolve_metrics(
  File "/home/jdimatteo/dev/ge_tutorials_fork/expect_column_values_to_not_match_regex/venv/lib/python3.9/site-packages/great_expectations/validator/validator.py", line 559, in _resolve_metrics
    return execution_engine.resolve_metrics(
  File "/home/jdimatteo/dev/ge_tutorials_fork/expect_column_values_to_not_match_regex/venv/lib/python3.9/site-packages/great_expectations/execution_engine/execution_engine.py", line 281, in resolve_metrics
    resolved_metrics[metric_to_resolve.id] = metric_fn(
  File "/home/jdimatteo/dev/ge_tutorials_fork/expect_column_values_to_not_match_regex/venv/lib/python3.9/site-packages/great_expectations/expectations/metrics/metric_provider.py", line 57, in inner_func
    return metric_fn(*args, **kwargs)
  File "/home/jdimatteo/dev/ge_tutorials_fork/expect_column_values_to_not_match_regex/venv/lib/python3.9/site-packages/great_expectations/expectations/metrics/map_metric.py", line 389, in inner_func
    expected_condition = metric_fn(
  File "/home/jdimatteo/dev/ge_tutorials_fork/expect_column_values_to_not_match_regex/venv/lib/python3.9/site-packages/great_expectations/expectations/metrics/column_map_metrics/column_values_match_regex.py", line 32, in _sqlalchemy
    "Regex is not supported for dialect %s" % str(_dialect.dialect.name)
AttributeError: module 'pybigquery.sqlalchemy_bigquery' has no attribute 'dialect'

To Reproduce

Use a regex expectation such as expect_column_values_to_match_regex with bigquery based on the example at https://knoxpod.netlify.app/docs/guides/connecting_to_your_data/database/bigquery.

More specifically, to get the above stack trace you can:

  1. git clone --branch jdimatteo/reproduce_expect_column_values_to_not_match_regex git@github.com:jdimatteo/ge_tutorials.git && cd ge_tutorials/expect_column_values_to_not_match_regex/
  2. python3.9 -m venv venv && source venv/bin/activate && pip install -r requirements.txt
  3. Copy the public dataset austin_bikeshare table bikeshare_stations to a google cloud project you can run queries against, e.g. bq mk <GCP_PROJECT_NAME>:austin_bikeshare && bq cp bigquery-public-data:austin_bikeshare.bikeshare_stations <GCP_PROJECT_NAME>:austin_bikeshare.bikeshare_stations
  4. Update gcp_project in the bigquery_python_example.py to be your <GCP_PROJECT_NAME>
  5. python bigquery_python_example.py

Expected behavior
No exception raised.

Environment (please complete the following information):

  • Operating System: Linux
  • Great Expectations Version: 0.13.24
@jdimatteo jdimatteo changed the title [BUG] bigquery regex expectations raise exception bigquery regex expectations raise exception Jul 28, 2021
@jdimatteo jdimatteo changed the title bigquery regex expectations raise exception bigquery regex unsupported dialect Jul 28, 2021
@jdimatteo
Copy link
Contributor Author

This might be related to #2978

@jdimatteo
Copy link
Contributor Author

Possibly a duplicate of #2959 as well

@jdimatteo
Copy link
Contributor Author

Before closing this issue, please ensure bigquery test coverage passing with:

  1. expect_column_values_to_match_regex
  2. expect_column_values_to_not_match_regex
  3. expect_column_values_to_match_regex_list
  4. expect_column_values_to_not_match_regex_list

E.g. see how these tests are commented out at

"expect_column_values_to_not_match_regex", # TODO: "column_name with space" -- follow up / triage with https://github.com/great-expectations/great_expectations/issues/3132.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants