-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] Add sqlite datasource #6657
Conversation
✅ Deploy Preview for niobium-lead-7998 ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
👇 Click on the image for a new way to code review
Legend |
assert isinstance(data_asset, TableAsset), "data_asset must be a TableAsset" | ||
assert isinstance( | ||
data_asset.datasource.execution_engine, SqlAlchemyExecutionEngine | ||
def _query_for_year_and_month( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The strategy here is to take the code that use to be in param_defaults
and put it here. We parameterize the query_datetime_range
Callable
which let's us break out the sqlite specific code.
column_splitter: Optional[SqliteYearMonthSplitter] = None # type: ignore[assignment] | ||
|
||
# This asset type will support a variety of splitters | ||
def add_year_and_month_splitter( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I made the new SqliteTableAsset
so I could have this helper function add correct splitter.
default_factory=lambda: ["year", "month"] | ||
) | ||
|
||
def param_defaults(self, table_asset: TableAsset) -> Dict[str, List]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried making the table_asset
input parameter more specific (ie SqliteTableAsset
) using Generics
but ran into some problems using pydantic and Generics
together. I decided not to resolve the issue right now.
order_by=order_by or [], # type: ignore[arg-type] # coerce list[str] | ||
# see TableAsset._parse_order_by_sorter() | ||
) | ||
# TODO (kilo59): custom init for `DataAsset` to accept datasource in constructor? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was basically lifted from the sql datasource. I'm not sure if I should keep or delete this comment.
We may want to use the setter for the datasource
instead of accessing _datasource
directly. I didn't do that now because I then wanted to typecheck on assignment which led to defining more methods on each class. I can do this in a followup PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the comments are still on the parent method then I think we could delete them here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
# We should make an assertion about the execution_engine earlier. Right now it is assigned to | ||
# after construction. We may be able to use a hook in a property setter. | ||
from great_expectations.execution_engine import SqlAlchemyExecutionEngine | ||
|
||
assert isinstance( | ||
table_asset.datasource.execution_engine, SqlAlchemyExecutionEngine |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can this be a custom pydantic
validator on the SQLDatasource
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because this happens in assignment I think the way to use pydantic validators would be to turn on validate_assignment
on a model config which would turn it on for all fields. I'm not sure we want to do that. I do think it would be better to do this check on the property setter and when we use it, like here. Right now we set it using the private variable _datasource
so some cleanup is needed. I'll do this in a followup PR.
# right side of the operator determines the type name | ||
# left side enforces the names on instance creation | ||
type: Literal["sqlite"] = "sqlite" # type: ignore[assignment] | ||
connection_string: str |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should consider adding some constraints to this connection_string
field.
It could as simple a custom validator that checks that the connection_string
.startswith('sqlite')
Or is could be with a pydantic
constrained string type (with a regex).
A benefit of using the constrained type is the regex will show up in the JSON schema for the model.
https://docs.pydantic.dev/usage/types/#constrained-types
https://docs.pydantic.dev/usage/types/#arguments-to-constr
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added constr
and some tests.
class Datasource( | ||
ExperimentalBaseModel, Generic[DataAssetType], metaclass=MetaDatasource | ||
): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So now the type-system knows that a PostgresDatasource
contains TableAsset
s and not a general/non-specific DataAsset
type?
🙌
…eat-expectations/great_expectations into f/dx-106/pandas-named-indices * 'f/dx-106/pandas-named-indices' of https://github.com/great-expectations/great_expectations: [FEATURE] Add sqlite datasource (#6657) [DOCS] doc-356 importable prerequisite box with correct theme applied and default entries (#6666) [DOCS] DOC-400 remove an outdated video link (#6667) [FEATURE] Added new expectation: ExpectYesterdayCountComparedToAvgEquivalentDaysOfWeek… (#6622) [FEATURE] ZEP - load from shared config (#6655) [MAINTENANCE] Add redshift/snowflake casting for sample data on expect_column_values_to_be_of_type (#6659) [MAINTENANCE] Add separate tests for exact stdev for redshift and snowflake (#6658) [MAINTENANCE] Make only column y data_alt different in col_pair_equal tests (#6661)
…lag-for-returning-query * develop: [MAINTENANCE] Only spark v2 for special test in not_be_in_set (#6669) [BUGFIX] Avoid key collisions between Rule-Based Profiler terminal literals and MetricConfiguration value keys (#6675) [MAINTENANCE] Make only column y data_alt different in col_pair_in_set tests (#6670) [MAINTENANCE] Add redshift/snowflake casting for sample data on expect_column_values_to_be_in_type_list (#6668) [FEATURE] Add sqlite datasource (#6657) [DOCS] doc-356 importable prerequisite box with correct theme applied and default entries (#6666) [DOCS] DOC-400 remove an outdated video link (#6667) [FEATURE] Added new expectation: ExpectYesterdayCountComparedToAvgEquivalentDaysOfWeek… (#6622) [FEATURE] ZEP - load from shared config (#6655) [MAINTENANCE] Add redshift/snowflake casting for sample data on expect_column_values_to_be_of_type (#6659) [MAINTENANCE] Add separate tests for exact stdev for redshift and snowflake (#6658) [MAINTENANCE] Make only column y data_alt different in col_pair_equal tests (#6661)
There are 2 substantive changes in this PR"
Datasource
class generic over the data asset. This lets us use properties of the particular data asset in the datasource without casting/ignoring errors.