Skip to content

Commit

Permalink
Merge branch 'develop' into f/DX-103/update-id-pk-metric
Browse files Browse the repository at this point in the history
* develop:
  [FEATURE] Ensure `result_format` accessed is through Checkpoint, and warns users if `Expectation` or `Validator`-level (#6562)
  [BUGFIX] Remove rendered header from Cloud-rendering tests (#6597)
  [MAINTENANCE] Refactor `BaseDataContext` and `DataContext` into factory functions (#6531)
  [MAINTENANCE] Utilize a `StrEnum` for `ConfigPeer` modes (#6596)
  [BUGFIX] Use v3.3.6 or higher of google-cloud-bigquery (with shapely bugfix) (#6590)
  [MAINTENANCE] Add docs snippet checker to `dev` CI (#6594)
  [MAINTENANCE] Leverage `RendererConfiguration` in existing prescriptive templates (3 of 3) (#6530)
  [BUGFIX] Support non-string `datetime` evaluation parameters (#6571)
  [RELEASE] 0.15.41 (#6593)
  [FEATURE] ZEP - PG `BatchSorter` loading + dumping (#6580)
  [MAINTENANCE] Leverage `RendererConfiguration` in existing prescriptive templates (2 of 3) (#6488)
  [MAINTENANCE] Remove `ExplorerDataContext` (#6592)
  [MAINTENANCE] Small refactor of ExecutionEngine.resolve_metrics() for better code readability (and miscellaneous additional clean up) (#6587)
  [MAINTENANCE] `mypy` config update (#6589)
  [BUGFIX] convert_to_json_serializable does not accept numpy datetime (#6553)
  [BUGFIX] Return unique list of batch_definitions (#6579)
  [MAINTENANCE] typo in method name (#6585)
  • Loading branch information
Will Shin committed Dec 16, 2022
2 parents 425d70b + acae9ef commit 12421eb
Show file tree
Hide file tree
Showing 166 changed files with 3,062 additions and 3,136 deletions.
14 changes: 8 additions & 6 deletions azure-pipelines-dev.yml
Expand Up @@ -60,12 +60,6 @@ stages:
inputs:
verbose: true
rules: |
[ContribChanged]
contrib/**
[ExperimentalChanged]
contrib/experimental/**
[DocsChanged]
docs/**
tests/integration/docusaurus/**
Expand Down Expand Up @@ -137,6 +131,14 @@ stages:
flake8 --select F401 great_expectations tests
name: UnusedImportChecker
- job: docs_snippet_checker
condition: eq(stageDependencies.scope_check.changes.outputs['CheckChanges.DocsChanged'], true)
steps:
- script: |
yarn install
python scripts/validate_docs_snippets.py
name: DocsSnippetChecker
- stage: import_ge
dependsOn: scope_check
pool:
Expand Down
15 changes: 14 additions & 1 deletion constraints-dev.txt
Expand Up @@ -23,4 +23,17 @@ botocore==1.20.106 # From aiobotocore v1.4.0 dependencies https://pypi.org/proj
# 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
#
# More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
protobuf==3.20
# google-cloud-bigquery has a fix for a refactor to shapely in their 3.3.6 release
# https://github.com/googleapis/python-bigquery/pull/1376
# With the new requirement of google-cloud-bigquery>=3.3.6, there is a hard conflict with protobuf==3.20
# ERROR: Cannot install -r reqs/requirements-dev-bigquery.txt (line 2) because these package versions have conflicting dependencies.
#
# The conflict is caused by:
# google-cloud-bigquery 3.4.1 depends on protobuf!=3.20.0, !=3.20.1, !=4.21.0, !=4.21.1, !=4.21.2, !=4.21.3, !=4.21.4, !=4.21.5, <5.0.0dev and >=3.19.5
# google-cloud-bigquery 3.4.0 depends on protobuf!=3.20.0, !=3.20.1, !=4.21.0, !=4.21.1, !=4.21.2, !=4.21.3, !=4.21.4, !=4.21.5, <5.0.0dev and >=3.19.5
# google-cloud-bigquery 3.3.6 depends on protobuf!=3.20.0, !=3.20.1, !=4.21.0, !=4.21.1, !=4.21.2, !=4.21.3, !=4.21.4, !=4.21.5, <5.0.0dev and >=3.19.5
# The user requested (constraint) protobuf==3.20
# protobuf 3.20.0 and 3.20.1 were released in April 2022
# protobuf 3.19.5, 3.19.6, and 3.20.2 were released in September 2022
# https://pypi.org/project/protobuf/#history
protobuf!=3.20.0,!=3.20.1,<3.21
Expand Up @@ -273,7 +273,7 @@ class ExpectColumnDiscreteEntropyToBeBetween(ColumnExpectation):
}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -290,7 +290,7 @@ class ExpectColumnDistributionToMatchBenfordsLaw(ColumnExpectation):
# "catch_exceptions": False,
# }

# def validate_configuration(self, configuration: Optional[ExpectationConfiguration]):
# def validate_configuration(self, configuration: Optional[ExpectationConfiguration] = None):
# """
# Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
# necessary configuration arguments have been provided for the validation of the expectation.
Expand Down
Expand Up @@ -303,7 +303,7 @@ class ExpectColumnKurtosisToBeBetween(ColumnExpectation):
"catch_exceptions": False,
}

# def validate_configuration(self, configuration: Optional[ExpectationConfiguration]):
# def validate_configuration(self, configuration: Optional[ExpectationConfiguration] = None):
# """
# Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
# necessary configuration arguments have been provided for the validation of the expectation.
Expand Down
Expand Up @@ -103,7 +103,7 @@ class ExpectColumnPairValuesToHaveDifferenceOfCustomPercentage(
}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -394,7 +394,7 @@ class ExpectColumnSkewToBeBetween(ColumnExpectation):
"catch_exceptions": False,
}

# def validate_configuration(self, configuration: Optional[ExpectationConfiguration]):
# def validate_configuration(self, configuration: Optional[ExpectationConfiguration] = None):
# """
# Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
# necessary configuration arguments have been provided for the validation of the expectation.
Expand Down
Expand Up @@ -63,7 +63,7 @@ class ExpectColumnSumToBe(ColumnExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -383,7 +383,7 @@ class ExpectColumnValuesToBeEdtfParseable(ColumnMapExpectation):
}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
super().validate_configuration(configuration)

Expand Down
Expand Up @@ -175,7 +175,7 @@ class ExpectColumnValuesToBeHexadecimal(ColumnMapExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -98,7 +98,7 @@ class ExpectColumnValuesToBeInSetSparkOptimized(ColumnExpectation):
default_kwarg_values = {"mostly": 1, "strict": True, "value_set": []}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -231,7 +231,7 @@ class ExpectColumnValuesToBeNormallyDistributed(ColumnExpectation):
}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -94,7 +94,7 @@ class ExpectColumnValuesToBeValidCrc32(ColumnMapExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -95,7 +95,7 @@ class ExpectColumnValuesToBeValidGithubUsername(ColumnMapExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -97,7 +97,7 @@ class ExpectColumnValuesToBeValidIataCode(ColumnMapExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -100,7 +100,7 @@ class ExpectColumnValuesToBeValidJson(ColumnMapExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -203,7 +203,7 @@ class ExpectColumnValuesToMatchPowersOfBase(ColumnMapExpectation):
default_kwarg_values = {"mostly": 1}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -147,7 +147,7 @@ class ExpectColumnValuesToMatchXmlSchema(ColumnMapExpectation):
}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
super().validate_configuration(configuration)

Expand Down
Expand Up @@ -246,7 +246,7 @@ class ExpectColumnWassersteinDistanceToBeLessThan(ColumnExpectation):
]

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -49,7 +49,7 @@ class ExpectQueriedColumnPairValuesToHaveDiff(QueryExpectation):
}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
super().validate_configuration(configuration)
diff = configuration["kwargs"].get("diff")
Expand Down
Expand Up @@ -50,7 +50,7 @@ class ExpectQueriedColumnValueFrequencyToMeetThreshold(QueryExpectation):
}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
super().validate_configuration(configuration)
value = configuration["kwargs"].get("value")
Expand Down
Expand Up @@ -44,7 +44,7 @@ class ExpectQueriedTableRowCountToBe(QueryExpectation):
}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
super().validate_configuration(configuration)
value = configuration["kwargs"].get("value")
Expand Down
Expand Up @@ -459,7 +459,9 @@ class ExpectTableChecksumToEqualOtherTable(TableExpectation):
"meta": None,
}

def validate_configuration(self, configuration: Optional[ExpectationConfiguration]):
def validate_configuration(
self, configuration: Optional[ExpectationConfiguration] = None
):
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
necessary configuration arguments have been provided for the validation of the expectation.
Expand Down
Expand Up @@ -98,7 +98,7 @@ class ExpectColumnValuesToBeValidFloridaZip(ColumnMapExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -122,7 +122,7 @@ class ExpectColumnValuesToBeValidGeneralZipcode(ColumnMapExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -98,7 +98,7 @@ class ExpectColumnValuesToBeValidIdahoZip(ColumnMapExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -98,7 +98,7 @@ class ExpectColumnValuesToBeValidKansasZip(ColumnMapExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -98,7 +98,7 @@ class ExpectColumnValuesToBeValidNewMexicoZip(ColumnMapExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -100,7 +100,7 @@ class ExpectColumnValuesToBeValidNorthCarolinaZip(ColumnMapExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
Expand Up @@ -98,7 +98,7 @@ class ExpectColumnValuesToBeValidUtahZip(ColumnMapExpectation):
default_kwarg_values = {}

def validate_configuration(
self, configuration: Optional[ExpectationConfiguration]
self, configuration: Optional[ExpectationConfiguration] = None
) -> None:
"""
Validates that a configuration has been set, and sets a configuration if it has yet to be set. Ensures that
Expand Down
15 changes: 15 additions & 0 deletions docs/changelog.md
Expand Up @@ -2,6 +2,21 @@
title: Changelog
---

### 0.15.41
* [FEATURE] enable mostly for expect_compound_columns_to_be_unique (#6533) (thanks @kimfrie)
* [BUGFIX] Return unique list of batch_definitions (#6579) (thanks @tanelk)
* [BUGFIX] convert_to_json_serializable does not accept numpy datetime (#6553) (thanks @tmilitino)
* [DOCS] Clean up misc snippet violations (#6582)
* [MAINTENANCE] Update json schema validation on usage stats to filter based on format. (#6502)
* [MAINTENANCE] Renaming Metric Name Suffixes using Enum Values for Better Code Readability (#6575)
* [MAINTENANCE] M/great 1433/cloud tests to async (#6543)
* [MAINTENANCE] Add static type checking to `rule.py` and `rule_based_profiler_result.py` (#6573)
* [MAINTENANCE] Update most contrib Expectation docstrings to be consistent and decently formatted for gallery (#6577)
* [MAINTENANCE] Update changelogs to reflect PyPI yanks (0.15.37-0.15.39) (#6581)
* [MAINTENANCE] Refactor ExecutionEngine.resolve_metrics() for better code readability (#6578)
* [MAINTENANCE] adding googletag manager to docusaurus (#6584)
* [MAINTENANCE] typo in method name (#6585)

### 0.15.40
* [FEATURE] F/great 1397/zep checkpoints (#6525)
* [FEATURE] Add integration test for zep sqlalchemy datasource with renderering. (#6564)
Expand Down
@@ -1,3 +1,3 @@
```
great_expectations, version 0.15.40
great_expectations, version 0.15.41
```
16 changes: 16 additions & 0 deletions docs_rtd/changelog.rst
Expand Up @@ -4,6 +4,22 @@
Changelog
#########

0.15.41
-----------------
* [FEATURE] enable mostly for expect_compound_columns_to_be_unique (#6533) (thanks @kimfrie)
* [BUGFIX] Return unique list of batch_definitions (#6579) (thanks @tanelk)
* [BUGFIX] convert_to_json_serializable does not accept numpy datetime (#6553) (thanks @tmilitino)
* [DOCS] Clean up misc snippet violations (#6582)
* [MAINTENANCE] Update json schema validation on usage stats to filter based on format. (#6502)
* [MAINTENANCE] Renaming Metric Name Suffixes using Enum Values for Better Code Readability (#6575)
* [MAINTENANCE] M/great 1433/cloud tests to async (#6543)
* [MAINTENANCE] Add static type checking to `rule.py` and `rule_based_profiler_result.py` (#6573)
* [MAINTENANCE] Update most contrib Expectation docstrings to be consistent and decently formatted for gallery (#6577)
* [MAINTENANCE] Update changelogs to reflect PyPI yanks (0.15.37-0.15.39) (#6581)
* [MAINTENANCE] Refactor ExecutionEngine.resolve_metrics() for better code readability (#6578)
* [MAINTENANCE] adding googletag manager to docusaurus (#6584)
* [MAINTENANCE] typo in method name (#6585)

0.15.40
-----------------
* [FEATURE] F/great 1397/zep checkpoints (#6525)
Expand Down

0 comments on commit 12421eb

Please sign in to comment.