[FEATURE] Support to include ID/PK in validation result for each row - SQL

[Shinnnyshinshin]

Changes proposed in this pull request:

SQL implementation of unexpected_index_columns value that allows users to specify a primary key (PK) column for identifying rows that failed an Expectation (usually returned as part of the unexpected_index_list). The PR also enables unexpected_index_query to be returned to the user, which

Changes were made to the map_metric_provider to take in the parameter from result_format and output unexpected_index_list as key-value pairs of the primary key column.

Note : This is only the SQL implementation, Pandas has been merged already, and Spark to follow.

• (Almost) closes #3195

What has changed?

• unexpected_index_list and unexpected_index_query was added as validation dependencies for SqlAlchemyExecutionEngine

• _sqlalchemy_map_condition_index was added as MapMetric.

  • This Metric returns the indices (truncated at 10 results for SQL) that contain unexpected values, according to the Primary Key values specified using unexpected_index_column_names. Note Unlike Pandas, the current SQL implementation does not have a default index that will be returned if unexpected_index_column_names is not specified (In other words, in order to see unexpected_index_list, the user must specify unexpected_index_column_names)

• _sqlalchemy_map_condition_query was added as MapMetric.

  • This Metric returns the query that can be used to retrieve the full unexpected results.
  • This method depends on two helper methods :
    • sql_post_compile_to_string() : Used by the _sqlalchemy_map_condition_query() method to compile SQL select statement with post-compile parameters into a string. Logic lifted directly from the sqlalchemy documentation documentation.
    • get_sqlalchemy_source_table_and_schema_selectable(): Used by _sqlalchemy_map_condition_query() metric function to return the table associated with the current Batch, rather than the temp_table that is created as part of running the query.

Can you give me an Example? (heavily adopted from the Pandas example)

Given the following table animal_names

sqlite_path = file_relative_path(
       __file__, "../../test_sets/metrics_test.db"
    )
sqlite_engine = sa.create_engine(f"sqlite:///{sqlite_path}")
df = pd.DataFrame(
       {
           "pk_1": [0, 1, 2, 3, 4, 5],
           "pk_2": ["zero", "one", "two", "three", "four", "five"],
            "animals": [
               "cat",
               "fish",
               "dog",
               "giraffe",
                "lion",
                "zebra",
            ],
        }
    )
df.to_sql(
     name="animal_names",
     con=sqlite_engine,
     index=False,
     if_exists="replace",
 )

We could run the expect_column_values_to_be_in_set Expectation on the animals column with ["cat", "fish", "dog"] as the value_set (ie domestic animals).

expectationConfiguration = ExpectationConfiguration(
    expectation_type="expect_column_values_to_be_in_set",
    kwargs={
        "column": "animals",
        "value_set": ["cat", "fish", "dog"],
        "result_format": {
            "result_format": "COMPLETE",
        },
    },
)

After running the ExpectationConfiguration, we would expect the unexpected_index_list to be ["giraffe", "lion", "zebra"] which correspond to the indices of 4, 5, and 6, the values that are not in the value_set.

This PR enables the following configuration, which sets unexpected_index_columns to be pk_1.

expectationConfiguration = ExpectationConfiguration(
    expectation_type="expect_column_values_to_be_in_set",
    kwargs={
        "column": "animals",
        "value_set": ["cat", "fish", "dog"],
        "result_format": {
            "result_format": "COMPLETE",
            "unexpected_index_column_names": ["pk_1"],
        },
    },
)

After running this new ExpectationConfiguration, we expect the unexpected_index_list to be [{"pk_1": 3}, {"pk_1": 4}, {"pk_1": 5}] which correspond to the values in the pk_1 column of the indices of ["giraffe", "lion", "zebra"].

What if I have a lot of rows in my table?

In order to retrieve the full list unexpected values from the table, the result also contains a unexpected_index_query, which can be copied into a db client to retrieve all the unexpected rows.

SELECT animals, pk_1 
FROM animal_names 
WHERE animals IS NOT NULL AND (animals NOT IN "
  "('cat', 'fish', 'dog'))

The query (along with unexpected_index_list) is returned as part of the result values returned by the Validator.

...
"unexpected_index_list": [{"pk_1": 3}, {"pk_1": 4}, {"pk_1": 5}],
"unexpected_index_query": "SELECT animals, pk_1 \n"
"FROM animal_names \n"
  "WHERE animals IS NOT NULL AND (animals NOT IN "
  "('cat', 'fish', 'dog'))",
...

Definition of Done

Please delete options that are not relevant.

• My code follows the Great Expectations style guide
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added unit tests where applicable and made sure that new and existing tests are passing.
• I have run any local integration tests and made sure that nothing is broken.

Thank you for submitting!

[ghost]

👇 Click on the image for a new way to code review

• Make big changes easier — review code in small groups of related files

• Know where to start — see the whole change at a glance

• Take a code tour — explore the change with an interactive tour

• Make comments and review — all fully sync’ed with github

  Try it now!

Legend

[netlify]

✅ Deploy Preview for niobium-lead-7998 ready!

Name	Link
🔨 Latest commit	2fe0627
🔍 Latest deploy log	https://app.netlify.com/sites/niobium-lead-7998/deploys/638e4b59a7a82f0007c37fa8
😎 Deploy Preview	https://deploy-preview-6448--niobium-lead-7998.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[Shinnnyshinshin]

Return SQL query according to backend

[Shinnnyshinshin]

TODO: adjust test to compare the query from bigquery, rather than adjust the query from bigquery to match the test

[anthonyburdi]

LGTM! Thank you for the synchronous review and for adding the tests for param substitution for each backend 🙇