[FEATURE] Add sqlite datasource #6657

billdirks · 2022-12-28T23:15:57Z

There are 2 substantive changes in this PR"

We make the Datasource class generic over the data asset. This lets us use properties of the particular data asset in the datasource without casting/ignoring errors.
Adds the new sqlite datasource.

My code follows the Great Expectations style guide
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added unit tests where applicable and made sure that new and existing tests are passing.
I have run any local integration tests and made sure that nothing is broken.

netlify · 2022-12-28T23:16:01Z

✅ Deploy Preview for niobium-lead-7998 ready!

Name	Link
🔨 Latest commit	`62be266`
🔍 Latest deploy log	https://app.netlify.com/sites/niobium-lead-7998/deploys/63ae1cb75801a200091ef950
😎 Deploy Preview	https://deploy-preview-6657--niobium-lead-7998.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

ghost · 2022-12-28T23:17:49Z

👇 Click on the image for a new way to code review

Make big changes easier — review code in small groups of related files
Know where to start — see the whole change at a glance
Take a code tour — explore the change with an interactive tour
Make comments and review — all fully sync’ed with github

Try it now!

Legend

billdirks · 2022-12-28T23:18:20Z

great_expectations/experimental/datasources/sql_datasource.py

-        assert isinstance(data_asset, TableAsset), "data_asset must be a TableAsset"
-        assert isinstance(
-            data_asset.datasource.execution_engine, SqlAlchemyExecutionEngine
+def _query_for_year_and_month(


The strategy here is to take the code that use to be in param_defaults and put it here. We parameterize the query_datetime_range Callable which let's us break out the sqlite specific code.

billdirks · 2022-12-28T23:20:31Z

great_expectations/experimental/datasources/sqlite_datasource.py

+    column_splitter: Optional[SqliteYearMonthSplitter] = None  # type: ignore[assignment]
+
+    # This asset type will support a variety of splitters
+    def add_year_and_month_splitter(


I made the new SqliteTableAsset so I could have this helper function add correct splitter.

billdirks · 2022-12-28T23:22:50Z

great_expectations/experimental/datasources/sqlite_datasource.py

+        default_factory=lambda: ["year", "month"]
+    )
+
+    def param_defaults(self, table_asset: TableAsset) -> Dict[str, List]:


I tried making the table_asset input parameter more specific (ie SqliteTableAsset) using Generics but ran into some problems using pydantic and Generics together. I decided not to resolve the issue right now.

billdirks · 2022-12-28T23:26:36Z

great_expectations/experimental/datasources/sqlite_datasource.py

+            order_by=order_by or [],  # type: ignore[arg-type]  # coerce list[str]
+            # see TableAsset._parse_order_by_sorter()
+        )
+        # TODO (kilo59): custom init for `DataAsset` to accept datasource in constructor?


This was basically lifted from the sql datasource. I'm not sure if I should keep or delete this comment.

We may want to use the setter for the datasource instead of accessing _datasource directly. I didn't do that now because I then wanted to typecheck on assignment which led to defining more methods on each class. I can do this in a followup PR.

If the comments are still on the parent method then I think we could delete them here.

Kilo59 · 2022-12-29T14:25:43Z

great_expectations/experimental/datasources/sql_datasource.py

+    # We should make an assertion about the execution_engine earlier. Right now it is assigned to
+    # after construction. We may be able to use a hook in a property setter.
+    from great_expectations.execution_engine import SqlAlchemyExecutionEngine
+
+    assert isinstance(
+        table_asset.datasource.execution_engine, SqlAlchemyExecutionEngine


Can this be a custom pydantic validator on the SQLDatasource?

Because this happens in assignment I think the way to use pydantic validators would be to turn on validate_assignment on a model config which would turn it on for all fields. I'm not sure we want to do that. I do think it would be better to do this check on the property setter and when we use it, like here. Right now we set it using the private variable _datasource so some cleanup is needed. I'll do this in a followup PR.

Kilo59 · 2022-12-29T14:35:23Z

great_expectations/experimental/datasources/sqlite_datasource.py

+    # right side of the operator determines the type name
+    # left side enforces the names on instance creation
+    type: Literal["sqlite"] = "sqlite"  # type: ignore[assignment]
+    connection_string: str


I think we should consider adding some constraints to this connection_string field.

It could as simple a custom validator that checks that the connection_string .startswith('sqlite')

Or is could be with a pydantic constrained string type (with a regex).
A benefit of using the constrained type is the regex will show up in the JSON schema for the model.
https://docs.pydantic.dev/usage/types/#constrained-types
https://docs.pydantic.dev/usage/types/#arguments-to-constr

https://docs.sqlalchemy.org/en/14/dialects/sqlite.html#

Added constr and some tests.

Kilo59 · 2022-12-29T14:42:31Z

great_expectations/experimental/datasources/interfaces.py

+class Datasource(
+    ExperimentalBaseModel, Generic[DataAssetType], metaclass=MetaDatasource
+):


So now the type-system knows that a PostgresDatasource contains TableAssets and not a general/non-specific DataAsset type?
🙌

…eat-expectations/great_expectations into f/dx-106/pandas-named-indices * 'f/dx-106/pandas-named-indices' of https://github.com/great-expectations/great_expectations: [FEATURE] Add sqlite datasource (#6657) [DOCS] doc-356 importable prerequisite box with correct theme applied and default entries (#6666) [DOCS] DOC-400 remove an outdated video link (#6667) [FEATURE] Added new expectation: ExpectYesterdayCountComparedToAvgEquivalentDaysOfWeek… (#6622) [FEATURE] ZEP - load from shared config (#6655) [MAINTENANCE] Add redshift/snowflake casting for sample data on expect_column_values_to_be_of_type (#6659) [MAINTENANCE] Add separate tests for exact stdev for redshift and snowflake (#6658) [MAINTENANCE] Make only column y data_alt different in col_pair_equal tests (#6661)

…lag-for-returning-query * develop: [MAINTENANCE] Only spark v2 for special test in not_be_in_set (#6669) [BUGFIX] Avoid key collisions between Rule-Based Profiler terminal literals and MetricConfiguration value keys (#6675) [MAINTENANCE] Make only column y data_alt different in col_pair_in_set tests (#6670) [MAINTENANCE] Add redshift/snowflake casting for sample data on expect_column_values_to_be_in_type_list (#6668) [FEATURE] Add sqlite datasource (#6657) [DOCS] doc-356 importable prerequisite box with correct theme applied and default entries (#6666) [DOCS] DOC-400 remove an outdated video link (#6667) [FEATURE] Added new expectation: ExpectYesterdayCountComparedToAvgEquivalentDaysOfWeek… (#6622) [FEATURE] ZEP - load from shared config (#6655) [MAINTENANCE] Add redshift/snowflake casting for sample data on expect_column_values_to_be_of_type (#6659) [MAINTENANCE] Add separate tests for exact stdev for redshift and snowflake (#6658) [MAINTENANCE] Make only column y data_alt different in col_pair_equal tests (#6661)

billdirks added 2 commits December 27, 2022 12:08

Add sqlite datasource

cf39c75

Specialize some types in sqlite_datasource.py

f24b485

github-actions bot added core-team platform labels Dec 28, 2022

billdirks commented Dec 28, 2022

View reviewed changes

Kilo59 reviewed Dec 29, 2022

View reviewed changes

Kilo59 approved these changes Dec 29, 2022

View reviewed changes

billdirks added 3 commits December 29, 2022 14:14

Add pydantic constraint for sqlite connection string.

52d1cd0

Fix file formatting

3713e3f

Merge branch 'develop' into add-sqlite-datasource

1ad431a

billdirks enabled auto-merge (squash) December 29, 2022 22:22

billdirks changed the title ~~Add sqlite datasource~~ [FEATURE] Add sqlite datasource Dec 29, 2022

billdirks disabled auto-merge December 29, 2022 22:27

Add workaround for constr mypy issue.

f46c0da

billdirks enabled auto-merge (squash) December 29, 2022 22:41

billdirks added 2 commits December 29, 2022 14:57

Remove unused import.

01cf851

Fix spacing in imports.

62be266

billdirks merged commit 9ff03a9 into develop Dec 29, 2022

billdirks deleted the add-sqlite-datasource branch December 29, 2022 23:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Add sqlite datasource #6657

[FEATURE] Add sqlite datasource #6657

billdirks commented Dec 28, 2022 •

edited

Loading

netlify bot commented Dec 28, 2022 •

edited

Loading

ghost commented Dec 28, 2022 •

edited by ghost

Loading

billdirks Dec 28, 2022

billdirks Dec 28, 2022

billdirks Dec 28, 2022

billdirks Dec 28, 2022

Kilo59 Dec 29, 2022 •

edited

Loading

billdirks Dec 29, 2022

Kilo59 Dec 29, 2022

billdirks Dec 29, 2022

Kilo59 Dec 29, 2022 •

edited

Loading

billdirks Dec 29, 2022

Kilo59 Dec 29, 2022

[FEATURE] Add sqlite datasource #6657

[FEATURE] Add sqlite datasource #6657

Conversation

billdirks commented Dec 28, 2022 • edited Loading

netlify bot commented Dec 28, 2022 • edited Loading

✅ Deploy Preview for niobium-lead-7998 ready!

ghost commented Dec 28, 2022 • edited by ghost Loading

Legend

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kilo59 Dec 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kilo59 Dec 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

billdirks commented Dec 28, 2022 •

edited

Loading

netlify bot commented Dec 28, 2022 •

edited

Loading

ghost commented Dec 28, 2022 •

edited by ghost

Loading

Kilo59 Dec 29, 2022 •

edited

Loading

Kilo59 Dec 29, 2022 •

edited

Loading