[MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ #2256

anthonyburdi · 2021-01-14T20:58:44Z

This PR proposes a change to our requirements files to ensure compatibility with the new pip resolver in pip v20.3+

Specifically, the change is to:

add a constraints-dev.txt file that will be considered via pip install -r requirements-dev.txt -c constraints-dev.txt to pin sub-dependencies of installed packages. The constraints.txt file pattern is used to constrain versions of packages rather than adding those packages to our pip requirements files, which would force install. By using the constraints.txt file, if those packages are no longer sub-dependencies of our requirements then they would not be installed as dependencies of Great Expectations. See the pip user guide for more info.
upgrade snowflake-connector-python to the latest version, which includes several sub-dependency upgrades (see release notes: https://pypi.org/project/snowflake-connector-python/)
Update our contributor "Setting up your dev environment" guide to add the constraints file to the pip install command.

# Conflicts: # docs/changelog.rst

* Typing cleanup. * [MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ (#2256) * upgrade snowflake-connector-python * pin boto3 and botocore for aiobotocore * Change dev environment setup guide to add constraints file * Update changelog Co-authored-by: anthonyburdi <anthony@superconductive.com>

…nt Classes and Maintained Using Checkpoint Configuration Store) (#2311) * Add checkpoint dir and module * Bring in checkpoint-related changes from Abe’s branch - https://github.com/superconductive/great_expectations/tree/refactor/create_checkpoint_class * Update tests * Update module name to reflect different dir structure * Fix typos * Update test * Add filesystem_csv_data_context_v3 fixture * Update tests * Update tests * Change Checkpoint to LegacyCheckpoint and NewStyleCheckpoint to Checkpoint * Checkpoint class rob (#2157) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Alexsherstinsky/wip/checkpoints sync with develop 12 15 2020 1 (#2166) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Alexsherstinsky/wip/checkpoint store 12 15 2020 2 (#2170) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * [DOCS] Add some links to GreatExpectationsOperator repo, links to in-code docs (#2167) * Add some links to GreatExpectationsOperator repo, links to in-code docs * fix line spacing and numbering * fix link * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Checkpoint store tests. Co-authored-by: anthonyburdi <anthony@superconductive.com> * Alexsherstinsky/wip/adding checkpoint store to data context 12 15 2020 3 (#2173) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * [DOCS] Add some links to GreatExpectationsOperator repo, links to in-code docs (#2167) * Add some links to GreatExpectationsOperator repo, links to in-code docs * fix line spacing and numbering * fix link * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Checkpoint store tests. * Checkpoint Store into DataContext * cleanup Co-authored-by: anthonyburdi <anthony@superconductive.com> * Checkpoint class rob (#2175) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> * Alexsherstinsky/wip/checkpoint api 12 16 2020 4 (#2182) * cleanup * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Alexsherstinsky/wip/checkpoint update tests 12 17 2020 5 (#2186) * Fix a configuration related test for checkpoints. * [ENHANCEMENT] Docstrings for DataConnector and associated classes (#2172) * Update changelog for develop * some clean up on docstrings * tests * Fixing tests. * More tests. Co-authored-by: William Shin <will@superconductive.com> * Alexsherstinsky/wip/checkpoint update tests 12 17 2020 6 (#2187) * Fix a configuration related test for checkpoints. * [ENHANCEMENT] Docstrings for DataConnector and associated classes (#2172) * Update changelog for develop * some clean up on docstrings * tests * Fixing tests. * More tests. * self_check for checkpoint store * linting Co-authored-by: William Shin <will@superconductive.com> * Checkpoint class rob (#2188) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * cleanup * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Update schema * Allow user to pass in all checkpoint attributes * Update Checkpoint __init__ * Allow user to provide all checkpoint attributes * Update run methods * Update run_checkpoint * Fix typo * Update formatting * Change template to template_name; add setters and update methods * Add update method * Change template to template_name * Add limit and batch_spec_passthrough to json dict output * Update run methods; add method to get substituted config * Isort and linting * Isort * Switch order * Fix typo * Nix old code * Update sig of update_action_list, add delete behavior * Fix config version for legacy checkpoint * Fix config version for legacy checkpoint * Add unit test for checkpoint config template/runtime substitution * Isort and linting * Update test * Add comment * Update test * Nix inheritance from object * Nix cast Co-authored-by: Alex Sherstinsky <alex@superconductivehealth.com> Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> * Alexsherstinsky/wip/checkpoint using test yaml config 12 17 2020 7 (#2192) * test_yaml_config for CheckpointStore * Lint * self_check and test_yaml_config * merge * Merge. * linting * linting * Fix tests. * Alexsherstinsky/wip/checkpoint fix tests in azure 12 18 2020 8 (#2195) * tests in Azure * debub * [MAINTENANCE] Anthony/bugfix/show cta footer issue 2120 (#2190) * remove show_cta_footer from base.py defaults * Add note that show_cta_footer was deprecated * remove show_cta_footer from in-code config tests * reformat with new black version * troubleshooting azure errors * cleanup Co-authored-by: anthonyburdi <anthony@superconductive.com> * Checkpoint.run (#2202) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * cleanup * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Update schema * Allow user to pass in all checkpoint attributes * Update Checkpoint __init__ * Allow user to provide all checkpoint attributes * Update run methods * Update run_checkpoint * Fix typo * Update formatting * Change template to template_name; add setters and update methods * Add update method * Change template to template_name * Add limit and batch_spec_passthrough to json dict output * Update run methods; add method to get substituted config * Isort and linting * Isort * Switch order * Fix typo * Nix old code * Update sig of update_action_list, add delete behavior * Fix config version for legacy checkpoint * Fix config version for legacy checkpoint * Add unit test for checkpoint config template/runtime substitution * Isort and linting * Update test * Add comment * Update test * Nix inheritance from object * Nix cast * tests in Azure * debub * troubleshooting azure errors * cleanup * linting * Set runtime_configuration and evaluation_parameters to {} if None * Instantiate BatchRequest if input batch_request is a dict * Change update_action_list to class_method get_updated_action_list; update batch_request update handling * Add util methods - get_runtime_batch_request - validate_validation_dict - get_substituted_validation_dict - get_run_name_from_template * Fill out run method * Add test for newstyle checkpoint .run * Add test for data_context.run_checkpoint for newstyle checkpoint * Update action_list update from runtime kwargs * Isort and linting * Add type hints; add exception case * isort and linting * Address pr review comments - add type hints - make methods private * Add type hints and fix typo * Linting Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> Co-authored-by: Alex Sherstinsky <alex@superconductivehealth.com> * [ENHANCEMENT][BUGFIX] Alexsherstinsky/wip/checkpoint test yaml config doc 12 21 2020 9 (#2206) * test_yaml_config -- more tests in preparation for documentation * Various checkpoint configuration options. * Cleanup. * Update explore_expectations_in_a_notebook.rst (#2174) * Update to support not using BatchRequest in configuration. * [BUGFIX] Made some improvements for the treatment of decimals for database backends (#2207) * Updated lossy conversion message in ensure_json_serializable and updated sqlalchemy_dataset to convert numeric metrics to json_serializable up front. * Update test cases for improved lossy conversion logic Signed-off-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: James Campbell <james.p.campbell@gmail.com> * pass manually_initialize_store_backend_id to database store backends, add test (#2205) * Update validations and tests. * fix messaging and tests Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: anthonyburdi <anthony@superconductive.com> * [DOCS][ENHANCEMENT] Alexsherstinsky/wip/checkpoint test yaml config doc 12 22 2020 10 (#2212) * Checkpoints howto using test_yaml_config * Rename * [BUGFIX][ENHANCEMENT] Alexsherstinsky/wip/checkpoint test yaml config doc examples 01 04 2021 11 (#2226) * Add an assertion for checkpoint test. * clean up * New packages * Debug * Debug * Debug * Fix tests for new Pandas version. * [ENHANCEMENT][BUGFIX] Alexsherstinsky/wip/checkpoint test yaml config doc examples 01 04 2021 12 (#2228) * Add an assertion for checkpoint test. * clean up * Adding run_checkpoint to all Checkpoint configuration examples. * Adding run_checkpoint to all Checkpoint configuration examples. * cleanup * Minor * [MAINTENANCE] Alexsherstinsky/maintenance/checkpoint test yaml config notebook 01 06 2021 15 (#2236) * rename method to add checkpoint to fit with the spec * typo * fix documentation * More flexibility to checkpoint_run test. * [BUGFIX] [ENHANCEMENT] Alexsherstinsky/maintenance/checkpoint test yaml config notebook 01 07 2021 16 (#2242) * DataFrame in batch_spec is not needed after it is processing by ExecutionEngine. * Make warning about non-existent class_name disappear. * Asset config. * Minor * Warnings * Printing * Formatting. * Schema adherence check. * [ENHANCEMENT] Alexsherstinsky/maintenance/checkpoint versioning test yaml config notebook 01 08 2021 18 (#2245) * Fixtures refactoring. * Error messages. * tests * Fix a test. * Refactor Checkpoints to allow easy addition of the SimpleCheckpoint class. (#2248) * Refactor `Checkpoint` class instantiation in such a way that the addition of `LegacyCheckpoint` and `SimpleCheckpoint` subclasses of the `Checkpoint` class becomes very easy. * [ENHANCEMENT] CheckpointResult type (#2238) Changes proposed in this pull request: - addition of new CheckpointResult type, which is output from the Checkpoint.run method - the new class is nearly equivalent to ValidationOperatorResult, with some exceptions: - no top-level evaluation_parameters key - the "actions_results" key is now expected (not enforced atm though) in each validation dict within run_results - batch_definition replaces batch_kwargs Previous Design Review notes: - https://github.com/superconductive/design/pull/24#discussion_r552861330 * * DataContext.get_checkpoint now returns a Checkpoint (#2249) Changes proposed in this pull request: This PR removes the return_config parameter to make DataContext.get_checkpoint() do what it says by returning a Checkpoint. CheckpointConfigs are easy to access from the Checkpoint.config property. * empty * [ENHANCEMENT] [BUGFIX] Configuration Version Increase from 2 to 3; Fix for DataAsset Configuration Schema Validation (#2253) * Version control against CheckpointStore * Warning for using validation_operators with version 3. * Warning for using validation_operators with version 3. * checkpoint_store error messaging * Configuration Version update from 2 to 3. * Adding batch_spec_passthrough to the AssetConfig and AssetConfigSchema for properly passing it to ExecutionEngine classes. * Cleanup * Cleanup * Cleanup * Use monkeypatch for environment variables. * Fixed a typo. * Cleanup. * Update tests to reflect new packages impacting notebook rendering. * CLI warnings update. * [DOCS][ENHANCEMENT] move validation actions into checkpoints; docs updates (#2255) * Move validation actions to checkpoint module * Move action util functions from validation_operator module to checkpoint module * Add .actions * Update doc with new CheckpointResult info * Add warning to doc * NIx empty util.py * Isort and linting * Update docs/guides/how_to_guides/validation/how_to_create_a_new_checkpoint_using_test_yaml_config.rst Co-authored-by: Taylor Miller <Aylr@users.noreply.github.com> * NIx util.py from verify_dynamic_loading_support call * Linting Co-authored-by: Taylor Miller <Aylr@users.noreply.github.com> * [BUGFIX] GE DataContext with multiple DataSources should not scan all configurations (#2250) * push before tests * Update data_context.py * fixing bug, added comments * update base.py to always include datasource as dict * added test and fixtures * changed comment * Update tests/conftest.py * cleaned up unnecessary refactor * Update data_context.py * linting * Update test_data_context.py * Update test_data_context.py * [ENHANCEMENT] Upgrade Capability for config_version 2 to config_version 3 with Checkpoints Configuration Migration (#2279) * Checkpoint upgrade * Typo on merge. * Update tests to account for Checkpoint store data migration. * Upgrade tests. * Manual steps tests. * Rename to a more general upgrade test module. * Add a test for v2_to_v3_upgrade (Checkpoints migration). * Robust error handling in upgrade tests. * Small typo in tests. * Update documentation by including full checkpoints capabilities. * Updates per review comments. * A more elegant upgrade logic. * A more elegant upgrade logic. * [WIP][ENHANCEMENT] SimpleCheckpoint (#2235) * * some more type annotations * action_list property on Checkpoint * SimpleCheckpoint stub * removed an unnecessary re-raise * added a few tests for Checkpoint WIP * decent default actions * decent site names functionality * a few more tests for Checkpoint * slack webhooks now validated using existing util (which was moved to prevent a circular dep) * new properties on SimpleCheckpiont More progress on SimpleCheckpoint * slack action built using notifyon/with * cleaner code * fixed default notify_with behavior * implemented guiding error messages in slack parameters * cleaned up code for readability slightly better docstring * remove name from actions * BaseDataContext now has a .get_site_names() method * better site validation on SimpleCheckpoint * removed redundant exception catch * put names back into action dicts * improved error messages on load_class util with tests * 3 mostly working SimpleCheckpoint.run() tests * convert SimpleCheckpoint class to a convenience class * cleanup and refactor * fix test after remvoing cherry pick * fix after merging CheckpointResult * * refactor fixtures * * docstring with examples * fixed a few tests * Checkpoint now validates datacontext * remove todo * * fixed a few load_class tests * removed some dead imports * added spark fixture to a few spark tests to prevent them from running when users select --no-spark * * configurator name change and design review notes * Add configurator to CheckpointConfig and schema * Isort; linting; test updates * Move SimpleCheckpointConfigurator into own module * Update tests * Add configurator arg to Checkpoint * Add **kwargs * Isort and linting * Update tests * Add typehints * Add code to use configurator if present at __init__ * Nix configurator from CheckpointConfig and schema * Fix typo * Fix import issues * Update tests * Add args specific to configurator * Add **kwargs and pass to configurator; assert no extra kwargs with no configurator * Nix missing kwarg to support LegacyCheckcpoint * Add configurator-specific fields to checkpoint config and schema * Isort and linting * Add new attributes to checkpoint config; fix typos * Add SimpleCheckpoint fixtures; update tests to use checkpoint from store * Isort and linting * Add post_dump processing of CheckpointConfig Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> * Checkpoint documentation for running and expanding configuration. (#2288) * [ENHANCEMENT] Remove the Re-Use of BatchRequest as Internal DataConnector Protocol Object (#2297) * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * [ENHANCEMENT] Simple checkpoint adjustment (#2301) * Add args specific to SimpleCheckpoint * Move configurator logic out of Checkpoint and into SimpleCheckpoint * Clarify comment * Add SimpleCheckpoint to test_yaml_config * Clarify comments * Add SimpleCheckpoint-specific attributes to update * Linting * Add comment clarification * Nix SimpleCheckpoint attributes from update; nix setters * [MAINTENANCE] Clean Up Type Hints (#2305) * Typing cleanup. * [MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ (#2256) * upgrade snowflake-connector-python * pin boto3 and botocore for aiobotocore * Change dev environment setup guide to add constraints file * Update changelog Co-authored-by: anthonyburdi <anthony@superconductive.com> * [DOCS] WIP: docs updates for Checkpoints (#2307) * [DOCS] WIP: docs updates for checkpoints * Some more tweaks to the formatting * Changed presumed release version * Added content for SimpleCheckpoint * Refactoring. * Checkpoint release tweaks (#2322) * Nix comment * Make TODO comment more precise * Update docs/guides/how_to_guides/configuring_datasources/how_to_configure_a_pandas_filesystem_datasource.rst Co-authored-by: James Campbell <james.p.campbell@gmail.com> * Checkpoints upgrade help page. * Checkpoints upgrade help page. * Checkpoints upgrade help page. * Checkpoints upgrade help page. * Fix the docs. * Undo incorrect fix for the explanation of relative directory. * Update Checkpoints feature maturity (#2323) * Update Checkpoints feature maturity * Update great_expectations/cli/checkpoint.py Co-authored-by: William Shin <will@superconductive.com> * Nix icon key Co-authored-by: William Shin <will@superconductive.com> * [BUGFIX] Updating the CLI for Checking Configuration -- Consistent Operation with Legacy Checkpoint Run (#2324) * Checking configuration and version upgrade. * Cleanup. * [BUGFIX] Minor Fix to Y/N Prompt Behavior During Upgrade (#2326) * Upgrade fix. * Fix to upgrade prompt logic. * [BUGFIX] Upgrade should NOT be invoked by CLI Checkpoint Run command (#2328) * Upgrade must run seamlesly when checkpoint_run CLI command is run. * Default checkpoint store behavior. * Correct default checkpoint store behavior. * Do not upgrade when running a checkpoint manually. Upgrade should be explicitly requested. * wording * wording * A more robust check for existing checkpoints. * Typo * Cleaup. * error messages * Error messages. * More robust error checking. * Style of comments. * Style of comments. * [BUGFIX] ConfigurationStore TupleStoreBackend was incorrectly turning "." into "/" -- Fixed via File Template (#2335) * Fix path for ConfigurationStore objects. * Update tests. * Typo. * Making tests work better with Azure. Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> Co-authored-by: anthonyburdi <anthony@superconductive.com> Co-authored-by: William Shin <will@superconductive.com> Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: Aylr <Aylr@users.noreply.github.com> Co-authored-by: Sam Bail <samanthapbail@gmail.com>

* Add checkpoint dir and module * Bring in checkpoint-related changes from Abe’s branch - https://github.com/superconductive/great_expectations/tree/refactor/create_checkpoint_class * Update tests * Update module name to reflect different dir structure * Fix typos * Update test * Add filesystem_csv_data_context_v3 fixture * Update tests * Update tests * Change Checkpoint to LegacyCheckpoint and NewStyleCheckpoint to Checkpoint * Checkpoint class rob (#2157) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Alexsherstinsky/wip/checkpoints sync with develop 12 15 2020 1 (#2166) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Alexsherstinsky/wip/checkpoint store 12 15 2020 2 (#2170) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * [DOCS] Add some links to GreatExpectationsOperator repo, links to in-code docs (#2167) * Add some links to GreatExpectationsOperator repo, links to in-code docs * fix line spacing and numbering * fix link * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Checkpoint store tests. Co-authored-by: anthonyburdi <anthony@superconductive.com> * Alexsherstinsky/wip/adding checkpoint store to data context 12 15 2020 3 (#2173) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * [DOCS] Add some links to GreatExpectationsOperator repo, links to in-code docs (#2167) * Add some links to GreatExpectationsOperator repo, links to in-code docs * fix line spacing and numbering * fix link * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Checkpoint store tests. * Checkpoint Store into DataContext * cleanup Co-authored-by: anthonyburdi <anthony@superconductive.com> * Checkpoint class rob (#2175) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> * Alexsherstinsky/wip/checkpoint api 12 16 2020 4 (#2182) * cleanup * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Alexsherstinsky/wip/checkpoint update tests 12 17 2020 5 (#2186) * Fix a configuration related test for checkpoints. * [ENHANCEMENT] Docstrings for DataConnector and associated classes (#2172) * Update changelog for develop * some clean up on docstrings * tests * Fixing tests. * More tests. Co-authored-by: William Shin <will@superconductive.com> * Alexsherstinsky/wip/checkpoint update tests 12 17 2020 6 (#2187) * Fix a configuration related test for checkpoints. * [ENHANCEMENT] Docstrings for DataConnector and associated classes (#2172) * Update changelog for develop * some clean up on docstrings * tests * Fixing tests. * More tests. * self_check for checkpoint store * linting Co-authored-by: William Shin <will@superconductive.com> * Checkpoint class rob (#2188) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * cleanup * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Update schema * Allow user to pass in all checkpoint attributes * Update Checkpoint __init__ * Allow user to provide all checkpoint attributes * Update run methods * Update run_checkpoint * Fix typo * Update formatting * Change template to template_name; add setters and update methods * Add update method * Change template to template_name * Add limit and batch_spec_passthrough to json dict output * Update run methods; add method to get substituted config * Isort and linting * Isort * Switch order * Fix typo * Nix old code * Update sig of update_action_list, add delete behavior * Fix config version for legacy checkpoint * Fix config version for legacy checkpoint * Add unit test for checkpoint config template/runtime substitution * Isort and linting * Update test * Add comment * Update test * Nix inheritance from object * Nix cast Co-authored-by: Alex Sherstinsky <alex@superconductivehealth.com> Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> * Alexsherstinsky/wip/checkpoint using test yaml config 12 17 2020 7 (#2192) * test_yaml_config for CheckpointStore * Lint * self_check and test_yaml_config * merge * Merge. * linting * linting * Fix tests. * Alexsherstinsky/wip/checkpoint fix tests in azure 12 18 2020 8 (#2195) * tests in Azure * debub * [MAINTENANCE] Anthony/bugfix/show cta footer issue 2120 (#2190) * remove show_cta_footer from base.py defaults * Add note that show_cta_footer was deprecated * remove show_cta_footer from in-code config tests * reformat with new black version * troubleshooting azure errors * cleanup Co-authored-by: anthonyburdi <anthony@superconductive.com> * Checkpoint.run (#2202) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * cleanup * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Update schema * Allow user to pass in all checkpoint attributes * Update Checkpoint __init__ * Allow user to provide all checkpoint attributes * Update run methods * Update run_checkpoint * Fix typo * Update formatting * Change template to template_name; add setters and update methods * Add update method * Change template to template_name * Add limit and batch_spec_passthrough to json dict output * Update run methods; add method to get substituted config * Isort and linting * Isort * Switch order * Fix typo * Nix old code * Update sig of update_action_list, add delete behavior * Fix config version for legacy checkpoint * Fix config version for legacy checkpoint * Add unit test for checkpoint config template/runtime substitution * Isort and linting * Update test * Add comment * Update test * Nix inheritance from object * Nix cast * tests in Azure * debub * troubleshooting azure errors * cleanup * linting * Set runtime_configuration and evaluation_parameters to {} if None * Instantiate BatchRequest if input batch_request is a dict * Change update_action_list to class_method get_updated_action_list; update batch_request update handling * Add util methods - get_runtime_batch_request - validate_validation_dict - get_substituted_validation_dict - get_run_name_from_template * Fill out run method * Add test for newstyle checkpoint .run * Add test for data_context.run_checkpoint for newstyle checkpoint * Update action_list update from runtime kwargs * Isort and linting * Add type hints; add exception case * isort and linting * Address pr review comments - add type hints - make methods private * Add type hints and fix typo * Linting Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> Co-authored-by: Alex Sherstinsky <alex@superconductivehealth.com> * [ENHANCEMENT][BUGFIX] Alexsherstinsky/wip/checkpoint test yaml config doc 12 21 2020 9 (#2206) * test_yaml_config -- more tests in preparation for documentation * Various checkpoint configuration options. * Cleanup. * Update explore_expectations_in_a_notebook.rst (#2174) * Update to support not using BatchRequest in configuration. * [BUGFIX] Made some improvements for the treatment of decimals for database backends (#2207) * Updated lossy conversion message in ensure_json_serializable and updated sqlalchemy_dataset to convert numeric metrics to json_serializable up front. * Update test cases for improved lossy conversion logic Signed-off-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: James Campbell <james.p.campbell@gmail.com> * pass manually_initialize_store_backend_id to database store backends, add test (#2205) * Update validations and tests. * fix messaging and tests Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: anthonyburdi <anthony@superconductive.com> * [DOCS][ENHANCEMENT] Alexsherstinsky/wip/checkpoint test yaml config doc 12 22 2020 10 (#2212) * Checkpoints howto using test_yaml_config * Rename * [BUGFIX][ENHANCEMENT] Alexsherstinsky/wip/checkpoint test yaml config doc examples 01 04 2021 11 (#2226) * Add an assertion for checkpoint test. * clean up * New packages * Debug * Debug * Debug * Fix tests for new Pandas version. * [ENHANCEMENT][BUGFIX] Alexsherstinsky/wip/checkpoint test yaml config doc examples 01 04 2021 12 (#2228) * Add an assertion for checkpoint test. * clean up * Adding run_checkpoint to all Checkpoint configuration examples. * Adding run_checkpoint to all Checkpoint configuration examples. * cleanup * Minor * [MAINTENANCE] Alexsherstinsky/maintenance/checkpoint test yaml config notebook 01 06 2021 15 (#2236) * rename method to add checkpoint to fit with the spec * typo * fix documentation * More flexibility to checkpoint_run test. * [BUGFIX] [ENHANCEMENT] Alexsherstinsky/maintenance/checkpoint test yaml config notebook 01 07 2021 16 (#2242) * DataFrame in batch_spec is not needed after it is processing by ExecutionEngine. * Make warning about non-existent class_name disappear. * Asset config. * Minor * Warnings * Printing * Formatting. * Schema adherence check. * [ENHANCEMENT] Alexsherstinsky/maintenance/checkpoint versioning test yaml config notebook 01 08 2021 18 (#2245) * Fixtures refactoring. * Error messages. * tests * Fix a test. * Refactor Checkpoints to allow easy addition of the SimpleCheckpoint class. (#2248) * Refactor `Checkpoint` class instantiation in such a way that the addition of `LegacyCheckpoint` and `SimpleCheckpoint` subclasses of the `Checkpoint` class becomes very easy. * [ENHANCEMENT] CheckpointResult type (#2238) Changes proposed in this pull request: - addition of new CheckpointResult type, which is output from the Checkpoint.run method - the new class is nearly equivalent to ValidationOperatorResult, with some exceptions: - no top-level evaluation_parameters key - the "actions_results" key is now expected (not enforced atm though) in each validation dict within run_results - batch_definition replaces batch_kwargs Previous Design Review notes: - https://github.com/superconductive/design/pull/24#discussion_r552861330 * * DataContext.get_checkpoint now returns a Checkpoint (#2249) Changes proposed in this pull request: This PR removes the return_config parameter to make DataContext.get_checkpoint() do what it says by returning a Checkpoint. CheckpointConfigs are easy to access from the Checkpoint.config property. * empty * [ENHANCEMENT] [BUGFIX] Configuration Version Increase from 2 to 3; Fix for DataAsset Configuration Schema Validation (#2253) * Version control against CheckpointStore * Warning for using validation_operators with version 3. * Warning for using validation_operators with version 3. * checkpoint_store error messaging * Configuration Version update from 2 to 3. * Adding batch_spec_passthrough to the AssetConfig and AssetConfigSchema for properly passing it to ExecutionEngine classes. * Cleanup * Cleanup * Cleanup * Use monkeypatch for environment variables. * Fixed a typo. * Cleanup. * Update tests to reflect new packages impacting notebook rendering. * CLI warnings update. * [DOCS][ENHANCEMENT] move validation actions into checkpoints; docs updates (#2255) * Move validation actions to checkpoint module * Move action util functions from validation_operator module to checkpoint module * Add .actions * Update doc with new CheckpointResult info * Add warning to doc * NIx empty util.py * Isort and linting * Update docs/guides/how_to_guides/validation/how_to_create_a_new_checkpoint_using_test_yaml_config.rst Co-authored-by: Taylor Miller <Aylr@users.noreply.github.com> * NIx util.py from verify_dynamic_loading_support call * Linting Co-authored-by: Taylor Miller <Aylr@users.noreply.github.com> * [BUGFIX] GE DataContext with multiple DataSources should not scan all configurations (#2250) * push before tests * Update data_context.py * fixing bug, added comments * update base.py to always include datasource as dict * added test and fixtures * changed comment * Update tests/conftest.py * cleaned up unnecessary refactor * Update data_context.py * linting * Update test_data_context.py * Update test_data_context.py * [ENHANCEMENT] Upgrade Capability for config_version 2 to config_version 3 with Checkpoints Configuration Migration (#2279) * Checkpoint upgrade * Typo on merge. * Update tests to account for Checkpoint store data migration. * Upgrade tests. * Manual steps tests. * Rename to a more general upgrade test module. * Add a test for v2_to_v3_upgrade (Checkpoints migration). * Robust error handling in upgrade tests. * Small typo in tests. * Update documentation by including full checkpoints capabilities. * Updates per review comments. * A more elegant upgrade logic. * A more elegant upgrade logic. * [WIP][ENHANCEMENT] SimpleCheckpoint (#2235) * * some more type annotations * action_list property on Checkpoint * SimpleCheckpoint stub * removed an unnecessary re-raise * added a few tests for Checkpoint WIP * decent default actions * decent site names functionality * a few more tests for Checkpoint * slack webhooks now validated using existing util (which was moved to prevent a circular dep) * new properties on SimpleCheckpiont More progress on SimpleCheckpoint * slack action built using notifyon/with * cleaner code * fixed default notify_with behavior * implemented guiding error messages in slack parameters * cleaned up code for readability slightly better docstring * remove name from actions * BaseDataContext now has a .get_site_names() method * better site validation on SimpleCheckpoint * removed redundant exception catch * put names back into action dicts * improved error messages on load_class util with tests * 3 mostly working SimpleCheckpoint.run() tests * convert SimpleCheckpoint class to a convenience class * cleanup and refactor * fix test after remvoing cherry pick * fix after merging CheckpointResult * * refactor fixtures * * docstring with examples * fixed a few tests * Checkpoint now validates datacontext * remove todo * * fixed a few load_class tests * removed some dead imports * added spark fixture to a few spark tests to prevent them from running when users select --no-spark * * configurator name change and design review notes * Add configurator to CheckpointConfig and schema * Isort; linting; test updates * Move SimpleCheckpointConfigurator into own module * Update tests * Add configurator arg to Checkpoint * Add **kwargs * Isort and linting * Update tests * Add typehints * Add code to use configurator if present at __init__ * Nix configurator from CheckpointConfig and schema * Fix typo * Fix import issues * Update tests * Add args specific to configurator * Add **kwargs and pass to configurator; assert no extra kwargs with no configurator * Nix missing kwarg to support LegacyCheckcpoint * Add configurator-specific fields to checkpoint config and schema * Isort and linting * Add new attributes to checkpoint config; fix typos * Add SimpleCheckpoint fixtures; update tests to use checkpoint from store * Isort and linting * Add post_dump processing of CheckpointConfig Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> * Checkpoint documentation for running and expanding configuration. (#2288) * [ENHANCEMENT] Remove the Re-Use of BatchRequest as Internal DataConnector Protocol Object (#2297) * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * [ENHANCEMENT] Simple checkpoint adjustment (#2301) * Add args specific to SimpleCheckpoint * Move configurator logic out of Checkpoint and into SimpleCheckpoint * Clarify comment * Add SimpleCheckpoint to test_yaml_config * Clarify comments * Add SimpleCheckpoint-specific attributes to update * Linting * Add comment clarification * Nix SimpleCheckpoint attributes from update; nix setters * [MAINTENANCE] Clean Up Type Hints (#2305) * Typing cleanup. * [MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ (#2256) * upgrade snowflake-connector-python * pin boto3 and botocore for aiobotocore * Change dev environment setup guide to add constraints file * Update changelog Co-authored-by: anthonyburdi <anthony@superconductive.com> * [DOCS] WIP: docs updates for Checkpoints (#2307) * [DOCS] WIP: docs updates for checkpoints * Some more tweaks to the formatting * Changed presumed release version * Added content for SimpleCheckpoint * Checkpoint release tweaks (#2322) * Nix comment * Make TODO comment more precise * Update Checkpoints feature maturity (#2323) * Update Checkpoints feature maturity * Update great_expectations/cli/checkpoint.py Co-authored-by: William Shin <will@superconductive.com> * Nix icon key Co-authored-by: William Shin <will@superconductive.com> * [BUGFIX] Updating the CLI for Checking Configuration -- Consistent Operation with Legacy Checkpoint Run (#2324) * Checking configuration and version upgrade. * Cleanup. * [BUGFIX] Minor Fix to Y/N Prompt Behavior During Upgrade (#2326) * Upgrade fix. * Fix to upgrade prompt logic. * [BUGFIX] Upgrade should NOT be invoked by CLI Checkpoint Run command (#2328) * Upgrade must run seamlesly when checkpoint_run CLI command is run. * Default checkpoint store behavior. * Correct default checkpoint store behavior. * Do not upgrade when running a checkpoint manually. Upgrade should be explicitly requested. * wording * wording * A more robust check for existing checkpoints. * Typo * Cleaup. * error messages * Error messages. * More robust error checking. * [BUGFIX] ConfigurationStore TupleStoreBackend was incorrectly turning "." into "/" -- Fixed via File Template (#2335) * Fix path for ConfigurationStore objects. * Update tests. * Typo. * Making tests work better with Azure. * [DOCS] checkpoint core concepts (#2261) * very rough WIP first draft on a Checkpoints core concepts page * Isort and linting * Add specifying_batches ref * Fill out core concept doc * Minor tweaks * Add section on CheckpointResult and SimpleCheckpoint; expand examples section * Fix typo * Just some minor format tweaks Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> Co-authored-by: Sam Bail <sam@superconductive.com> Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> Co-authored-by: anthonyburdi <anthony@superconductive.com> Co-authored-by: William Shin <will@superconductive.com> Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: Aylr <Aylr@users.noreply.github.com> Co-authored-by: Sam Bail <samanthapbail@gmail.com> Co-authored-by: Sam Bail <sam@superconductive.com>

…t-expectations#2256) * upgrade snowflake-connector-python * pin boto3 and botocore for aiobotocore * Change dev environment setup guide to add constraints file * Update changelog

…nt Classes and Maintained Using Checkpoint Configuration Store) (great-expectations#2311) * Add checkpoint dir and module * Bring in checkpoint-related changes from Abe’s branch - https://github.com/superconductive/great_expectations/tree/refactor/create_checkpoint_class * Update tests * Update module name to reflect different dir structure * Fix typos * Update test * Add filesystem_csv_data_context_v3 fixture * Update tests * Update tests * Change Checkpoint to LegacyCheckpoint and NewStyleCheckpoint to Checkpoint * Checkpoint class rob (great-expectations#2157) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Alexsherstinsky/wip/checkpoints sync with develop 12 15 2020 1 (great-expectations#2166) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Alexsherstinsky/wip/checkpoint store 12 15 2020 2 (great-expectations#2170) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * [DOCS] Add some links to GreatExpectationsOperator repo, links to in-code docs (great-expectations#2167) * Add some links to GreatExpectationsOperator repo, links to in-code docs * fix line spacing and numbering * fix link * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Checkpoint store tests. Co-authored-by: anthonyburdi <anthony@superconductive.com> * Alexsherstinsky/wip/adding checkpoint store to data context 12 15 2020 3 (great-expectations#2173) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * [DOCS] Add some links to GreatExpectationsOperator repo, links to in-code docs (great-expectations#2167) * Add some links to GreatExpectationsOperator repo, links to in-code docs * fix line spacing and numbering * fix link * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Checkpoint store tests. * Checkpoint Store into DataContext * cleanup Co-authored-by: anthonyburdi <anthony@superconductive.com> * Checkpoint class rob (great-expectations#2175) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> * Alexsherstinsky/wip/checkpoint api 12 16 2020 4 (great-expectations#2182) * cleanup * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Alexsherstinsky/wip/checkpoint update tests 12 17 2020 5 (great-expectations#2186) * Fix a configuration related test for checkpoints. * [ENHANCEMENT] Docstrings for DataConnector and associated classes (great-expectations#2172) * Update changelog for develop * some clean up on docstrings * tests * Fixing tests. * More tests. Co-authored-by: William Shin <will@superconductive.com> * Alexsherstinsky/wip/checkpoint update tests 12 17 2020 6 (great-expectations#2187) * Fix a configuration related test for checkpoints. * [ENHANCEMENT] Docstrings for DataConnector and associated classes (great-expectations#2172) * Update changelog for develop * some clean up on docstrings * tests * Fixing tests. * More tests. * self_check for checkpoint store * linting Co-authored-by: William Shin <will@superconductive.com> * Checkpoint class rob (great-expectations#2188) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * cleanup * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Update schema * Allow user to pass in all checkpoint attributes * Update Checkpoint __init__ * Allow user to provide all checkpoint attributes * Update run methods * Update run_checkpoint * Fix typo * Update formatting * Change template to template_name; add setters and update methods * Add update method * Change template to template_name * Add limit and batch_spec_passthrough to json dict output * Update run methods; add method to get substituted config * Isort and linting * Isort * Switch order * Fix typo * Nix old code * Update sig of update_action_list, add delete behavior * Fix config version for legacy checkpoint * Fix config version for legacy checkpoint * Add unit test for checkpoint config template/runtime substitution * Isort and linting * Update test * Add comment * Update test * Nix inheritance from object * Nix cast Co-authored-by: Alex Sherstinsky <alex@superconductivehealth.com> Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> * Alexsherstinsky/wip/checkpoint using test yaml config 12 17 2020 7 (great-expectations#2192) * test_yaml_config for CheckpointStore * Lint * self_check and test_yaml_config * merge * Merge. * linting * linting * Fix tests. * Alexsherstinsky/wip/checkpoint fix tests in azure 12 18 2020 8 (great-expectations#2195) * tests in Azure * debub * [MAINTENANCE] Anthony/bugfix/show cta footer issue 2120 (great-expectations#2190) * remove show_cta_footer from base.py defaults * Add note that show_cta_footer was deprecated * remove show_cta_footer from in-code config tests * reformat with new black version * troubleshooting azure errors * cleanup Co-authored-by: anthonyburdi <anthony@superconductive.com> * Checkpoint.run (great-expectations#2202) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * cleanup * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Update schema * Allow user to pass in all checkpoint attributes * Update Checkpoint __init__ * Allow user to provide all checkpoint attributes * Update run methods * Update run_checkpoint * Fix typo * Update formatting * Change template to template_name; add setters and update methods * Add update method * Change template to template_name * Add limit and batch_spec_passthrough to json dict output * Update run methods; add method to get substituted config * Isort and linting * Isort * Switch order * Fix typo * Nix old code * Update sig of update_action_list, add delete behavior * Fix config version for legacy checkpoint * Fix config version for legacy checkpoint * Add unit test for checkpoint config template/runtime substitution * Isort and linting * Update test * Add comment * Update test * Nix inheritance from object * Nix cast * tests in Azure * debub * troubleshooting azure errors * cleanup * linting * Set runtime_configuration and evaluation_parameters to {} if None * Instantiate BatchRequest if input batch_request is a dict * Change update_action_list to class_method get_updated_action_list; update batch_request update handling * Add util methods - get_runtime_batch_request - validate_validation_dict - get_substituted_validation_dict - get_run_name_from_template * Fill out run method * Add test for newstyle checkpoint .run * Add test for data_context.run_checkpoint for newstyle checkpoint * Update action_list update from runtime kwargs * Isort and linting * Add type hints; add exception case * isort and linting * Address pr review comments - add type hints - make methods private * Add type hints and fix typo * Linting Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> Co-authored-by: Alex Sherstinsky <alex@superconductivehealth.com> * [ENHANCEMENT][BUGFIX] Alexsherstinsky/wip/checkpoint test yaml config doc 12 21 2020 9 (great-expectations#2206) * test_yaml_config -- more tests in preparation for documentation * Various checkpoint configuration options. * Cleanup. * Update explore_expectations_in_a_notebook.rst (great-expectations#2174) * Update to support not using BatchRequest in configuration. * [BUGFIX] Made some improvements for the treatment of decimals for database backends (great-expectations#2207) * Updated lossy conversion message in ensure_json_serializable and updated sqlalchemy_dataset to convert numeric metrics to json_serializable up front. * Update test cases for improved lossy conversion logic Signed-off-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: James Campbell <james.p.campbell@gmail.com> * pass manually_initialize_store_backend_id to database store backends, add test (great-expectations#2205) * Update validations and tests. * fix messaging and tests Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: anthonyburdi <anthony@superconductive.com> * [DOCS][ENHANCEMENT] Alexsherstinsky/wip/checkpoint test yaml config doc 12 22 2020 10 (great-expectations#2212) * Checkpoints howto using test_yaml_config * Rename * [BUGFIX][ENHANCEMENT] Alexsherstinsky/wip/checkpoint test yaml config doc examples 01 04 2021 11 (great-expectations#2226) * Add an assertion for checkpoint test. * clean up * New packages * Debug * Debug * Debug * Fix tests for new Pandas version. * [ENHANCEMENT][BUGFIX] Alexsherstinsky/wip/checkpoint test yaml config doc examples 01 04 2021 12 (great-expectations#2228) * Add an assertion for checkpoint test. * clean up * Adding run_checkpoint to all Checkpoint configuration examples. * Adding run_checkpoint to all Checkpoint configuration examples. * cleanup * Minor * [MAINTENANCE] Alexsherstinsky/maintenance/checkpoint test yaml config notebook 01 06 2021 15 (great-expectations#2236) * rename method to add checkpoint to fit with the spec * typo * fix documentation * More flexibility to checkpoint_run test. * [BUGFIX] [ENHANCEMENT] Alexsherstinsky/maintenance/checkpoint test yaml config notebook 01 07 2021 16 (great-expectations#2242) * DataFrame in batch_spec is not needed after it is processing by ExecutionEngine. * Make warning about non-existent class_name disappear. * Asset config. * Minor * Warnings * Printing * Formatting. * Schema adherence check. * [ENHANCEMENT] Alexsherstinsky/maintenance/checkpoint versioning test yaml config notebook 01 08 2021 18 (great-expectations#2245) * Fixtures refactoring. * Error messages. * tests * Fix a test. * Refactor Checkpoints to allow easy addition of the SimpleCheckpoint class. (great-expectations#2248) * Refactor `Checkpoint` class instantiation in such a way that the addition of `LegacyCheckpoint` and `SimpleCheckpoint` subclasses of the `Checkpoint` class becomes very easy. * [ENHANCEMENT] CheckpointResult type (great-expectations#2238) Changes proposed in this pull request: - addition of new CheckpointResult type, which is output from the Checkpoint.run method - the new class is nearly equivalent to ValidationOperatorResult, with some exceptions: - no top-level evaluation_parameters key - the "actions_results" key is now expected (not enforced atm though) in each validation dict within run_results - batch_definition replaces batch_kwargs Previous Design Review notes: - https://github.com/superconductive/design/pull/24#discussion_r552861330 * * DataContext.get_checkpoint now returns a Checkpoint (great-expectations#2249) Changes proposed in this pull request: This PR removes the return_config parameter to make DataContext.get_checkpoint() do what it says by returning a Checkpoint. CheckpointConfigs are easy to access from the Checkpoint.config property. * empty * [ENHANCEMENT] [BUGFIX] Configuration Version Increase from 2 to 3; Fix for DataAsset Configuration Schema Validation (great-expectations#2253) * Version control against CheckpointStore * Warning for using validation_operators with version 3. * Warning for using validation_operators with version 3. * checkpoint_store error messaging * Configuration Version update from 2 to 3. * Adding batch_spec_passthrough to the AssetConfig and AssetConfigSchema for properly passing it to ExecutionEngine classes. * Cleanup * Cleanup * Cleanup * Use monkeypatch for environment variables. * Fixed a typo. * Cleanup. * Update tests to reflect new packages impacting notebook rendering. * CLI warnings update. * [DOCS][ENHANCEMENT] move validation actions into checkpoints; docs updates (great-expectations#2255) * Move validation actions to checkpoint module * Move action util functions from validation_operator module to checkpoint module * Add .actions * Update doc with new CheckpointResult info * Add warning to doc * NIx empty util.py * Isort and linting * Update docs/guides/how_to_guides/validation/how_to_create_a_new_checkpoint_using_test_yaml_config.rst Co-authored-by: Taylor Miller <Aylr@users.noreply.github.com> * NIx util.py from verify_dynamic_loading_support call * Linting Co-authored-by: Taylor Miller <Aylr@users.noreply.github.com> * [BUGFIX] GE DataContext with multiple DataSources should not scan all configurations (great-expectations#2250) * push before tests * Update data_context.py * fixing bug, added comments * update base.py to always include datasource as dict * added test and fixtures * changed comment * Update tests/conftest.py * cleaned up unnecessary refactor * Update data_context.py * linting * Update test_data_context.py * Update test_data_context.py * [ENHANCEMENT] Upgrade Capability for config_version 2 to config_version 3 with Checkpoints Configuration Migration (great-expectations#2279) * Checkpoint upgrade * Typo on merge. * Update tests to account for Checkpoint store data migration. * Upgrade tests. * Manual steps tests. * Rename to a more general upgrade test module. * Add a test for v2_to_v3_upgrade (Checkpoints migration). * Robust error handling in upgrade tests. * Small typo in tests. * Update documentation by including full checkpoints capabilities. * Updates per review comments. * A more elegant upgrade logic. * A more elegant upgrade logic. * [WIP][ENHANCEMENT] SimpleCheckpoint (great-expectations#2235) * * some more type annotations * action_list property on Checkpoint * SimpleCheckpoint stub * removed an unnecessary re-raise * added a few tests for Checkpoint WIP * decent default actions * decent site names functionality * a few more tests for Checkpoint * slack webhooks now validated using existing util (which was moved to prevent a circular dep) * new properties on SimpleCheckpiont More progress on SimpleCheckpoint * slack action built using notifyon/with * cleaner code * fixed default notify_with behavior * implemented guiding error messages in slack parameters * cleaned up code for readability slightly better docstring * remove name from actions * BaseDataContext now has a .get_site_names() method * better site validation on SimpleCheckpoint * removed redundant exception catch * put names back into action dicts * improved error messages on load_class util with tests * 3 mostly working SimpleCheckpoint.run() tests * convert SimpleCheckpoint class to a convenience class * cleanup and refactor * fix test after remvoing cherry pick * fix after merging CheckpointResult * * refactor fixtures * * docstring with examples * fixed a few tests * Checkpoint now validates datacontext * remove todo * * fixed a few load_class tests * removed some dead imports * added spark fixture to a few spark tests to prevent them from running when users select --no-spark * * configurator name change and design review notes * Add configurator to CheckpointConfig and schema * Isort; linting; test updates * Move SimpleCheckpointConfigurator into own module * Update tests * Add configurator arg to Checkpoint * Add **kwargs * Isort and linting * Update tests * Add typehints * Add code to use configurator if present at __init__ * Nix configurator from CheckpointConfig and schema * Fix typo * Fix import issues * Update tests * Add args specific to configurator * Add **kwargs and pass to configurator; assert no extra kwargs with no configurator * Nix missing kwarg to support LegacyCheckcpoint * Add configurator-specific fields to checkpoint config and schema * Isort and linting * Add new attributes to checkpoint config; fix typos * Add SimpleCheckpoint fixtures; update tests to use checkpoint from store * Isort and linting * Add post_dump processing of CheckpointConfig Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> * Checkpoint documentation for running and expanding configuration. (great-expectations#2288) * [ENHANCEMENT] Remove the Re-Use of BatchRequest as Internal DataConnector Protocol Object (great-expectations#2297) * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * [ENHANCEMENT] Simple checkpoint adjustment (great-expectations#2301) * Add args specific to SimpleCheckpoint * Move configurator logic out of Checkpoint and into SimpleCheckpoint * Clarify comment * Add SimpleCheckpoint to test_yaml_config * Clarify comments * Add SimpleCheckpoint-specific attributes to update * Linting * Add comment clarification * Nix SimpleCheckpoint attributes from update; nix setters * [MAINTENANCE] Clean Up Type Hints (great-expectations#2305) * Typing cleanup. * [MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ (great-expectations#2256) * upgrade snowflake-connector-python * pin boto3 and botocore for aiobotocore * Change dev environment setup guide to add constraints file * Update changelog Co-authored-by: anthonyburdi <anthony@superconductive.com> * [DOCS] WIP: docs updates for Checkpoints (great-expectations#2307) * [DOCS] WIP: docs updates for checkpoints * Some more tweaks to the formatting * Changed presumed release version * Added content for SimpleCheckpoint * Refactoring. * Checkpoint release tweaks (great-expectations#2322) * Nix comment * Make TODO comment more precise * Update docs/guides/how_to_guides/configuring_datasources/how_to_configure_a_pandas_filesystem_datasource.rst Co-authored-by: James Campbell <james.p.campbell@gmail.com> * Checkpoints upgrade help page. * Checkpoints upgrade help page. * Checkpoints upgrade help page. * Checkpoints upgrade help page. * Fix the docs. * Undo incorrect fix for the explanation of relative directory. * Update Checkpoints feature maturity (great-expectations#2323) * Update Checkpoints feature maturity * Update great_expectations/cli/checkpoint.py Co-authored-by: William Shin <will@superconductive.com> * Nix icon key Co-authored-by: William Shin <will@superconductive.com> * [BUGFIX] Updating the CLI for Checking Configuration -- Consistent Operation with Legacy Checkpoint Run (great-expectations#2324) * Checking configuration and version upgrade. * Cleanup. * [BUGFIX] Minor Fix to Y/N Prompt Behavior During Upgrade (great-expectations#2326) * Upgrade fix. * Fix to upgrade prompt logic. * [BUGFIX] Upgrade should NOT be invoked by CLI Checkpoint Run command (great-expectations#2328) * Upgrade must run seamlesly when checkpoint_run CLI command is run. * Default checkpoint store behavior. * Correct default checkpoint store behavior. * Do not upgrade when running a checkpoint manually. Upgrade should be explicitly requested. * wording * wording * A more robust check for existing checkpoints. * Typo * Cleaup. * error messages * Error messages. * More robust error checking. * Style of comments. * Style of comments. * [BUGFIX] ConfigurationStore TupleStoreBackend was incorrectly turning "." into "/" -- Fixed via File Template (great-expectations#2335) * Fix path for ConfigurationStore objects. * Update tests. * Typo. * Making tests work better with Azure. Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> Co-authored-by: anthonyburdi <anthony@superconductive.com> Co-authored-by: William Shin <will@superconductive.com> Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: Aylr <Aylr@users.noreply.github.com> Co-authored-by: Sam Bail <samanthapbail@gmail.com>

…t-expectations#2338) * Add checkpoint dir and module * Bring in checkpoint-related changes from Abe’s branch - https://github.com/superconductive/great_expectations/tree/refactor/create_checkpoint_class * Update tests * Update module name to reflect different dir structure * Fix typos * Update test * Add filesystem_csv_data_context_v3 fixture * Update tests * Update tests * Change Checkpoint to LegacyCheckpoint and NewStyleCheckpoint to Checkpoint * Checkpoint class rob (great-expectations#2157) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Alexsherstinsky/wip/checkpoints sync with develop 12 15 2020 1 (great-expectations#2166) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Alexsherstinsky/wip/checkpoint store 12 15 2020 2 (great-expectations#2170) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * [DOCS] Add some links to GreatExpectationsOperator repo, links to in-code docs (great-expectations#2167) * Add some links to GreatExpectationsOperator repo, links to in-code docs * fix line spacing and numbering * fix link * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Checkpoint store tests. Co-authored-by: anthonyburdi <anthony@superconductive.com> * Alexsherstinsky/wip/adding checkpoint store to data context 12 15 2020 3 (great-expectations#2173) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * [DOCS] Add some links to GreatExpectationsOperator repo, links to in-code docs (great-expectations#2167) * Add some links to GreatExpectationsOperator repo, links to in-code docs * fix line spacing and numbering * fix link * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Checkpoint store tests. * Checkpoint Store into DataContext * cleanup Co-authored-by: anthonyburdi <anthony@superconductive.com> * Checkpoint class rob (great-expectations#2175) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> * Alexsherstinsky/wip/checkpoint api 12 16 2020 4 (great-expectations#2182) * cleanup * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Alexsherstinsky/wip/checkpoint update tests 12 17 2020 5 (great-expectations#2186) * Fix a configuration related test for checkpoints. * [ENHANCEMENT] Docstrings for DataConnector and associated classes (great-expectations#2172) * Update changelog for develop * some clean up on docstrings * tests * Fixing tests. * More tests. Co-authored-by: William Shin <will@superconductive.com> * Alexsherstinsky/wip/checkpoint update tests 12 17 2020 6 (great-expectations#2187) * Fix a configuration related test for checkpoints. * [ENHANCEMENT] Docstrings for DataConnector and associated classes (great-expectations#2172) * Update changelog for develop * some clean up on docstrings * tests * Fixing tests. * More tests. * self_check for checkpoint store * linting Co-authored-by: William Shin <will@superconductive.com> * Checkpoint class rob (great-expectations#2188) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * cleanup * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Update schema * Allow user to pass in all checkpoint attributes * Update Checkpoint __init__ * Allow user to provide all checkpoint attributes * Update run methods * Update run_checkpoint * Fix typo * Update formatting * Change template to template_name; add setters and update methods * Add update method * Change template to template_name * Add limit and batch_spec_passthrough to json dict output * Update run methods; add method to get substituted config * Isort and linting * Isort * Switch order * Fix typo * Nix old code * Update sig of update_action_list, add delete behavior * Fix config version for legacy checkpoint * Fix config version for legacy checkpoint * Add unit test for checkpoint config template/runtime substitution * Isort and linting * Update test * Add comment * Update test * Nix inheritance from object * Nix cast Co-authored-by: Alex Sherstinsky <alex@superconductivehealth.com> Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> * Alexsherstinsky/wip/checkpoint using test yaml config 12 17 2020 7 (great-expectations#2192) * test_yaml_config for CheckpointStore * Lint * self_check and test_yaml_config * merge * Merge. * linting * linting * Fix tests. * Alexsherstinsky/wip/checkpoint fix tests in azure 12 18 2020 8 (great-expectations#2195) * tests in Azure * debub * [MAINTENANCE] Anthony/bugfix/show cta footer issue 2120 (great-expectations#2190) * remove show_cta_footer from base.py defaults * Add note that show_cta_footer was deprecated * remove show_cta_footer from in-code config tests * reformat with new black version * troubleshooting azure errors * cleanup Co-authored-by: anthonyburdi <anthony@superconductive.com> * Checkpoint.run (great-expectations#2202) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * cleanup * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Update schema * Allow user to pass in all checkpoint attributes * Update Checkpoint __init__ * Allow user to provide all checkpoint attributes * Update run methods * Update run_checkpoint * Fix typo * Update formatting * Change template to template_name; add setters and update methods * Add update method * Change template to template_name * Add limit and batch_spec_passthrough to json dict output * Update run methods; add method to get substituted config * Isort and linting * Isort * Switch order * Fix typo * Nix old code * Update sig of update_action_list, add delete behavior * Fix config version for legacy checkpoint * Fix config version for legacy checkpoint * Add unit test for checkpoint config template/runtime substitution * Isort and linting * Update test * Add comment * Update test * Nix inheritance from object * Nix cast * tests in Azure * debub * troubleshooting azure errors * cleanup * linting * Set runtime_configuration and evaluation_parameters to {} if None * Instantiate BatchRequest if input batch_request is a dict * Change update_action_list to class_method get_updated_action_list; update batch_request update handling * Add util methods - get_runtime_batch_request - validate_validation_dict - get_substituted_validation_dict - get_run_name_from_template * Fill out run method * Add test for newstyle checkpoint .run * Add test for data_context.run_checkpoint for newstyle checkpoint * Update action_list update from runtime kwargs * Isort and linting * Add type hints; add exception case * isort and linting * Address pr review comments - add type hints - make methods private * Add type hints and fix typo * Linting Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> Co-authored-by: Alex Sherstinsky <alex@superconductivehealth.com> * [ENHANCEMENT][BUGFIX] Alexsherstinsky/wip/checkpoint test yaml config doc 12 21 2020 9 (great-expectations#2206) * test_yaml_config -- more tests in preparation for documentation * Various checkpoint configuration options. * Cleanup. * Update explore_expectations_in_a_notebook.rst (great-expectations#2174) * Update to support not using BatchRequest in configuration. * [BUGFIX] Made some improvements for the treatment of decimals for database backends (great-expectations#2207) * Updated lossy conversion message in ensure_json_serializable and updated sqlalchemy_dataset to convert numeric metrics to json_serializable up front. * Update test cases for improved lossy conversion logic Signed-off-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: James Campbell <james.p.campbell@gmail.com> * pass manually_initialize_store_backend_id to database store backends, add test (great-expectations#2205) * Update validations and tests. * fix messaging and tests Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: anthonyburdi <anthony@superconductive.com> * [DOCS][ENHANCEMENT] Alexsherstinsky/wip/checkpoint test yaml config doc 12 22 2020 10 (great-expectations#2212) * Checkpoints howto using test_yaml_config * Rename * [BUGFIX][ENHANCEMENT] Alexsherstinsky/wip/checkpoint test yaml config doc examples 01 04 2021 11 (great-expectations#2226) * Add an assertion for checkpoint test. * clean up * New packages * Debug * Debug * Debug * Fix tests for new Pandas version. * [ENHANCEMENT][BUGFIX] Alexsherstinsky/wip/checkpoint test yaml config doc examples 01 04 2021 12 (great-expectations#2228) * Add an assertion for checkpoint test. * clean up * Adding run_checkpoint to all Checkpoint configuration examples. * Adding run_checkpoint to all Checkpoint configuration examples. * cleanup * Minor * [MAINTENANCE] Alexsherstinsky/maintenance/checkpoint test yaml config notebook 01 06 2021 15 (great-expectations#2236) * rename method to add checkpoint to fit with the spec * typo * fix documentation * More flexibility to checkpoint_run test. * [BUGFIX] [ENHANCEMENT] Alexsherstinsky/maintenance/checkpoint test yaml config notebook 01 07 2021 16 (great-expectations#2242) * DataFrame in batch_spec is not needed after it is processing by ExecutionEngine. * Make warning about non-existent class_name disappear. * Asset config. * Minor * Warnings * Printing * Formatting. * Schema adherence check. * [ENHANCEMENT] Alexsherstinsky/maintenance/checkpoint versioning test yaml config notebook 01 08 2021 18 (great-expectations#2245) * Fixtures refactoring. * Error messages. * tests * Fix a test. * Refactor Checkpoints to allow easy addition of the SimpleCheckpoint class. (great-expectations#2248) * Refactor `Checkpoint` class instantiation in such a way that the addition of `LegacyCheckpoint` and `SimpleCheckpoint` subclasses of the `Checkpoint` class becomes very easy. * [ENHANCEMENT] CheckpointResult type (great-expectations#2238) Changes proposed in this pull request: - addition of new CheckpointResult type, which is output from the Checkpoint.run method - the new class is nearly equivalent to ValidationOperatorResult, with some exceptions: - no top-level evaluation_parameters key - the "actions_results" key is now expected (not enforced atm though) in each validation dict within run_results - batch_definition replaces batch_kwargs Previous Design Review notes: - https://github.com/superconductive/design/pull/24#discussion_r552861330 * * DataContext.get_checkpoint now returns a Checkpoint (great-expectations#2249) Changes proposed in this pull request: This PR removes the return_config parameter to make DataContext.get_checkpoint() do what it says by returning a Checkpoint. CheckpointConfigs are easy to access from the Checkpoint.config property. * empty * [ENHANCEMENT] [BUGFIX] Configuration Version Increase from 2 to 3; Fix for DataAsset Configuration Schema Validation (great-expectations#2253) * Version control against CheckpointStore * Warning for using validation_operators with version 3. * Warning for using validation_operators with version 3. * checkpoint_store error messaging * Configuration Version update from 2 to 3. * Adding batch_spec_passthrough to the AssetConfig and AssetConfigSchema for properly passing it to ExecutionEngine classes. * Cleanup * Cleanup * Cleanup * Use monkeypatch for environment variables. * Fixed a typo. * Cleanup. * Update tests to reflect new packages impacting notebook rendering. * CLI warnings update. * [DOCS][ENHANCEMENT] move validation actions into checkpoints; docs updates (great-expectations#2255) * Move validation actions to checkpoint module * Move action util functions from validation_operator module to checkpoint module * Add .actions * Update doc with new CheckpointResult info * Add warning to doc * NIx empty util.py * Isort and linting * Update docs/guides/how_to_guides/validation/how_to_create_a_new_checkpoint_using_test_yaml_config.rst Co-authored-by: Taylor Miller <Aylr@users.noreply.github.com> * NIx util.py from verify_dynamic_loading_support call * Linting Co-authored-by: Taylor Miller <Aylr@users.noreply.github.com> * [BUGFIX] GE DataContext with multiple DataSources should not scan all configurations (great-expectations#2250) * push before tests * Update data_context.py * fixing bug, added comments * update base.py to always include datasource as dict * added test and fixtures * changed comment * Update tests/conftest.py * cleaned up unnecessary refactor * Update data_context.py * linting * Update test_data_context.py * Update test_data_context.py * [ENHANCEMENT] Upgrade Capability for config_version 2 to config_version 3 with Checkpoints Configuration Migration (great-expectations#2279) * Checkpoint upgrade * Typo on merge. * Update tests to account for Checkpoint store data migration. * Upgrade tests. * Manual steps tests. * Rename to a more general upgrade test module. * Add a test for v2_to_v3_upgrade (Checkpoints migration). * Robust error handling in upgrade tests. * Small typo in tests. * Update documentation by including full checkpoints capabilities. * Updates per review comments. * A more elegant upgrade logic. * A more elegant upgrade logic. * [WIP][ENHANCEMENT] SimpleCheckpoint (great-expectations#2235) * * some more type annotations * action_list property on Checkpoint * SimpleCheckpoint stub * removed an unnecessary re-raise * added a few tests for Checkpoint WIP * decent default actions * decent site names functionality * a few more tests for Checkpoint * slack webhooks now validated using existing util (which was moved to prevent a circular dep) * new properties on SimpleCheckpiont More progress on SimpleCheckpoint * slack action built using notifyon/with * cleaner code * fixed default notify_with behavior * implemented guiding error messages in slack parameters * cleaned up code for readability slightly better docstring * remove name from actions * BaseDataContext now has a .get_site_names() method * better site validation on SimpleCheckpoint * removed redundant exception catch * put names back into action dicts * improved error messages on load_class util with tests * 3 mostly working SimpleCheckpoint.run() tests * convert SimpleCheckpoint class to a convenience class * cleanup and refactor * fix test after remvoing cherry pick * fix after merging CheckpointResult * * refactor fixtures * * docstring with examples * fixed a few tests * Checkpoint now validates datacontext * remove todo * * fixed a few load_class tests * removed some dead imports * added spark fixture to a few spark tests to prevent them from running when users select --no-spark * * configurator name change and design review notes * Add configurator to CheckpointConfig and schema * Isort; linting; test updates * Move SimpleCheckpointConfigurator into own module * Update tests * Add configurator arg to Checkpoint * Add **kwargs * Isort and linting * Update tests * Add typehints * Add code to use configurator if present at __init__ * Nix configurator from CheckpointConfig and schema * Fix typo * Fix import issues * Update tests * Add args specific to configurator * Add **kwargs and pass to configurator; assert no extra kwargs with no configurator * Nix missing kwarg to support LegacyCheckcpoint * Add configurator-specific fields to checkpoint config and schema * Isort and linting * Add new attributes to checkpoint config; fix typos * Add SimpleCheckpoint fixtures; update tests to use checkpoint from store * Isort and linting * Add post_dump processing of CheckpointConfig Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> * Checkpoint documentation for running and expanding configuration. (great-expectations#2288) * [ENHANCEMENT] Remove the Re-Use of BatchRequest as Internal DataConnector Protocol Object (great-expectations#2297) * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * [ENHANCEMENT] Simple checkpoint adjustment (great-expectations#2301) * Add args specific to SimpleCheckpoint * Move configurator logic out of Checkpoint and into SimpleCheckpoint * Clarify comment * Add SimpleCheckpoint to test_yaml_config * Clarify comments * Add SimpleCheckpoint-specific attributes to update * Linting * Add comment clarification * Nix SimpleCheckpoint attributes from update; nix setters * [MAINTENANCE] Clean Up Type Hints (great-expectations#2305) * Typing cleanup. * [MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ (great-expectations#2256) * upgrade snowflake-connector-python * pin boto3 and botocore for aiobotocore * Change dev environment setup guide to add constraints file * Update changelog Co-authored-by: anthonyburdi <anthony@superconductive.com> * [DOCS] WIP: docs updates for Checkpoints (great-expectations#2307) * [DOCS] WIP: docs updates for checkpoints * Some more tweaks to the formatting * Changed presumed release version * Added content for SimpleCheckpoint * Checkpoint release tweaks (great-expectations#2322) * Nix comment * Make TODO comment more precise * Update Checkpoints feature maturity (great-expectations#2323) * Update Checkpoints feature maturity * Update great_expectations/cli/checkpoint.py Co-authored-by: William Shin <will@superconductive.com> * Nix icon key Co-authored-by: William Shin <will@superconductive.com> * [BUGFIX] Updating the CLI for Checking Configuration -- Consistent Operation with Legacy Checkpoint Run (great-expectations#2324) * Checking configuration and version upgrade. * Cleanup. * [BUGFIX] Minor Fix to Y/N Prompt Behavior During Upgrade (great-expectations#2326) * Upgrade fix. * Fix to upgrade prompt logic. * [BUGFIX] Upgrade should NOT be invoked by CLI Checkpoint Run command (great-expectations#2328) * Upgrade must run seamlesly when checkpoint_run CLI command is run. * Default checkpoint store behavior. * Correct default checkpoint store behavior. * Do not upgrade when running a checkpoint manually. Upgrade should be explicitly requested. * wording * wording * A more robust check for existing checkpoints. * Typo * Cleaup. * error messages * Error messages. * More robust error checking. * [BUGFIX] ConfigurationStore TupleStoreBackend was incorrectly turning "." into "/" -- Fixed via File Template (great-expectations#2335) * Fix path for ConfigurationStore objects. * Update tests. * Typo. * Making tests work better with Azure. * [DOCS] checkpoint core concepts (great-expectations#2261) * very rough WIP first draft on a Checkpoints core concepts page * Isort and linting * Add specifying_batches ref * Fill out core concept doc * Minor tweaks * Add section on CheckpointResult and SimpleCheckpoint; expand examples section * Fix typo * Just some minor format tweaks Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> Co-authored-by: Sam Bail <sam@superconductive.com> Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> Co-authored-by: anthonyburdi <anthony@superconductive.com> Co-authored-by: William Shin <will@superconductive.com> Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: Aylr <Aylr@users.noreply.github.com> Co-authored-by: Sam Bail <samanthapbail@gmail.com> Co-authored-by: Sam Bail <sam@superconductive.com>

…t-expectations#2256) * upgrade snowflake-connector-python * pin boto3 and botocore for aiobotocore * Change dev environment setup guide to add constraints file * Update changelog

…nt Classes and Maintained Using Checkpoint Configuration Store) (great-expectations#2311) * Add checkpoint dir and module * Bring in checkpoint-related changes from Abe’s branch - https://github.com/superconductive/great_expectations/tree/refactor/create_checkpoint_class * Update tests * Update module name to reflect different dir structure * Fix typos * Update test * Add filesystem_csv_data_context_v3 fixture * Update tests * Update tests * Change Checkpoint to LegacyCheckpoint and NewStyleCheckpoint to Checkpoint * Checkpoint class rob (great-expectations#2157) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Alexsherstinsky/wip/checkpoints sync with develop 12 15 2020 1 (great-expectations#2166) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Alexsherstinsky/wip/checkpoint store 12 15 2020 2 (great-expectations#2170) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * [DOCS] Add some links to GreatExpectationsOperator repo, links to in-code docs (great-expectations#2167) * Add some links to GreatExpectationsOperator repo, links to in-code docs * fix line spacing and numbering * fix link * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Checkpoint store tests. Co-authored-by: anthonyburdi <anthony@superconductive.com> * Alexsherstinsky/wip/adding checkpoint store to data context 12 15 2020 3 (great-expectations#2173) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * [DOCS] Add some links to GreatExpectationsOperator repo, links to in-code docs (great-expectations#2167) * Add some links to GreatExpectationsOperator repo, links to in-code docs * fix line spacing and numbering * fix link * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Checkpoint store tests. * Checkpoint Store into DataContext * cleanup Co-authored-by: anthonyburdi <anthony@superconductive.com> * Checkpoint class rob (great-expectations#2175) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> * Alexsherstinsky/wip/checkpoint api 12 16 2020 4 (great-expectations#2182) * cleanup * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Alexsherstinsky/wip/checkpoint update tests 12 17 2020 5 (great-expectations#2186) * Fix a configuration related test for checkpoints. * [ENHANCEMENT] Docstrings for DataConnector and associated classes (great-expectations#2172) * Update changelog for develop * some clean up on docstrings * tests * Fixing tests. * More tests. Co-authored-by: William Shin <will@superconductive.com> * Alexsherstinsky/wip/checkpoint update tests 12 17 2020 6 (great-expectations#2187) * Fix a configuration related test for checkpoints. * [ENHANCEMENT] Docstrings for DataConnector and associated classes (great-expectations#2172) * Update changelog for develop * some clean up on docstrings * tests * Fixing tests. * More tests. * self_check for checkpoint store * linting Co-authored-by: William Shin <will@superconductive.com> * Checkpoint class rob (great-expectations#2188) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * cleanup * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Update schema * Allow user to pass in all checkpoint attributes * Update Checkpoint __init__ * Allow user to provide all checkpoint attributes * Update run methods * Update run_checkpoint * Fix typo * Update formatting * Change template to template_name; add setters and update methods * Add update method * Change template to template_name * Add limit and batch_spec_passthrough to json dict output * Update run methods; add method to get substituted config * Isort and linting * Isort * Switch order * Fix typo * Nix old code * Update sig of update_action_list, add delete behavior * Fix config version for legacy checkpoint * Fix config version for legacy checkpoint * Add unit test for checkpoint config template/runtime substitution * Isort and linting * Update test * Add comment * Update test * Nix inheritance from object * Nix cast Co-authored-by: Alex Sherstinsky <alex@superconductivehealth.com> Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> * Alexsherstinsky/wip/checkpoint using test yaml config 12 17 2020 7 (great-expectations#2192) * test_yaml_config for CheckpointStore * Lint * self_check and test_yaml_config * merge * Merge. * linting * linting * Fix tests. * Alexsherstinsky/wip/checkpoint fix tests in azure 12 18 2020 8 (great-expectations#2195) * tests in Azure * debub * [MAINTENANCE] Anthony/bugfix/show cta footer issue 2120 (great-expectations#2190) * remove show_cta_footer from base.py defaults * Add note that show_cta_footer was deprecated * remove show_cta_footer from in-code config tests * reformat with new black version * troubleshooting azure errors * cleanup Co-authored-by: anthonyburdi <anthony@superconductive.com> * Checkpoint.run (great-expectations#2202) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * cleanup * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Update schema * Allow user to pass in all checkpoint attributes * Update Checkpoint __init__ * Allow user to provide all checkpoint attributes * Update run methods * Update run_checkpoint * Fix typo * Update formatting * Change template to template_name; add setters and update methods * Add update method * Change template to template_name * Add limit and batch_spec_passthrough to json dict output * Update run methods; add method to get substituted config * Isort and linting * Isort * Switch order * Fix typo * Nix old code * Update sig of update_action_list, add delete behavior * Fix config version for legacy checkpoint * Fix config version for legacy checkpoint * Add unit test for checkpoint config template/runtime substitution * Isort and linting * Update test * Add comment * Update test * Nix inheritance from object * Nix cast * tests in Azure * debub * troubleshooting azure errors * cleanup * linting * Set runtime_configuration and evaluation_parameters to {} if None * Instantiate BatchRequest if input batch_request is a dict * Change update_action_list to class_method get_updated_action_list; update batch_request update handling * Add util methods - get_runtime_batch_request - validate_validation_dict - get_substituted_validation_dict - get_run_name_from_template * Fill out run method * Add test for newstyle checkpoint .run * Add test for data_context.run_checkpoint for newstyle checkpoint * Update action_list update from runtime kwargs * Isort and linting * Add type hints; add exception case * isort and linting * Address pr review comments - add type hints - make methods private * Add type hints and fix typo * Linting Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> Co-authored-by: Alex Sherstinsky <alex@superconductivehealth.com> * [ENHANCEMENT][BUGFIX] Alexsherstinsky/wip/checkpoint test yaml config doc 12 21 2020 9 (great-expectations#2206) * test_yaml_config -- more tests in preparation for documentation * Various checkpoint configuration options. * Cleanup. * Update explore_expectations_in_a_notebook.rst (great-expectations#2174) * Update to support not using BatchRequest in configuration. * [BUGFIX] Made some improvements for the treatment of decimals for database backends (great-expectations#2207) * Updated lossy conversion message in ensure_json_serializable and updated sqlalchemy_dataset to convert numeric metrics to json_serializable up front. * Update test cases for improved lossy conversion logic Signed-off-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: James Campbell <james.p.campbell@gmail.com> * pass manually_initialize_store_backend_id to database store backends, add test (great-expectations#2205) * Update validations and tests. * fix messaging and tests Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: anthonyburdi <anthony@superconductive.com> * [DOCS][ENHANCEMENT] Alexsherstinsky/wip/checkpoint test yaml config doc 12 22 2020 10 (great-expectations#2212) * Checkpoints howto using test_yaml_config * Rename * [BUGFIX][ENHANCEMENT] Alexsherstinsky/wip/checkpoint test yaml config doc examples 01 04 2021 11 (great-expectations#2226) * Add an assertion for checkpoint test. * clean up * New packages * Debug * Debug * Debug * Fix tests for new Pandas version. * [ENHANCEMENT][BUGFIX] Alexsherstinsky/wip/checkpoint test yaml config doc examples 01 04 2021 12 (great-expectations#2228) * Add an assertion for checkpoint test. * clean up * Adding run_checkpoint to all Checkpoint configuration examples. * Adding run_checkpoint to all Checkpoint configuration examples. * cleanup * Minor * [MAINTENANCE] Alexsherstinsky/maintenance/checkpoint test yaml config notebook 01 06 2021 15 (great-expectations#2236) * rename method to add checkpoint to fit with the spec * typo * fix documentation * More flexibility to checkpoint_run test. * [BUGFIX] [ENHANCEMENT] Alexsherstinsky/maintenance/checkpoint test yaml config notebook 01 07 2021 16 (great-expectations#2242) * DataFrame in batch_spec is not needed after it is processing by ExecutionEngine. * Make warning about non-existent class_name disappear. * Asset config. * Minor * Warnings * Printing * Formatting. * Schema adherence check. * [ENHANCEMENT] Alexsherstinsky/maintenance/checkpoint versioning test yaml config notebook 01 08 2021 18 (great-expectations#2245) * Fixtures refactoring. * Error messages. * tests * Fix a test. * Refactor Checkpoints to allow easy addition of the SimpleCheckpoint class. (great-expectations#2248) * Refactor `Checkpoint` class instantiation in such a way that the addition of `LegacyCheckpoint` and `SimpleCheckpoint` subclasses of the `Checkpoint` class becomes very easy. * [ENHANCEMENT] CheckpointResult type (great-expectations#2238) Changes proposed in this pull request: - addition of new CheckpointResult type, which is output from the Checkpoint.run method - the new class is nearly equivalent to ValidationOperatorResult, with some exceptions: - no top-level evaluation_parameters key - the "actions_results" key is now expected (not enforced atm though) in each validation dict within run_results - batch_definition replaces batch_kwargs Previous Design Review notes: - https://github.com/superconductive/design/pull/24#discussion_r552861330 * * DataContext.get_checkpoint now returns a Checkpoint (great-expectations#2249) Changes proposed in this pull request: This PR removes the return_config parameter to make DataContext.get_checkpoint() do what it says by returning a Checkpoint. CheckpointConfigs are easy to access from the Checkpoint.config property. * empty * [ENHANCEMENT] [BUGFIX] Configuration Version Increase from 2 to 3; Fix for DataAsset Configuration Schema Validation (great-expectations#2253) * Version control against CheckpointStore * Warning for using validation_operators with version 3. * Warning for using validation_operators with version 3. * checkpoint_store error messaging * Configuration Version update from 2 to 3. * Adding batch_spec_passthrough to the AssetConfig and AssetConfigSchema for properly passing it to ExecutionEngine classes. * Cleanup * Cleanup * Cleanup * Use monkeypatch for environment variables. * Fixed a typo. * Cleanup. * Update tests to reflect new packages impacting notebook rendering. * CLI warnings update. * [DOCS][ENHANCEMENT] move validation actions into checkpoints; docs updates (great-expectations#2255) * Move validation actions to checkpoint module * Move action util functions from validation_operator module to checkpoint module * Add .actions * Update doc with new CheckpointResult info * Add warning to doc * NIx empty util.py * Isort and linting * Update docs/guides/how_to_guides/validation/how_to_create_a_new_checkpoint_using_test_yaml_config.rst Co-authored-by: Taylor Miller <Aylr@users.noreply.github.com> * NIx util.py from verify_dynamic_loading_support call * Linting Co-authored-by: Taylor Miller <Aylr@users.noreply.github.com> * [BUGFIX] GE DataContext with multiple DataSources should not scan all configurations (great-expectations#2250) * push before tests * Update data_context.py * fixing bug, added comments * update base.py to always include datasource as dict * added test and fixtures * changed comment * Update tests/conftest.py * cleaned up unnecessary refactor * Update data_context.py * linting * Update test_data_context.py * Update test_data_context.py * [ENHANCEMENT] Upgrade Capability for config_version 2 to config_version 3 with Checkpoints Configuration Migration (great-expectations#2279) * Checkpoint upgrade * Typo on merge. * Update tests to account for Checkpoint store data migration. * Upgrade tests. * Manual steps tests. * Rename to a more general upgrade test module. * Add a test for v2_to_v3_upgrade (Checkpoints migration). * Robust error handling in upgrade tests. * Small typo in tests. * Update documentation by including full checkpoints capabilities. * Updates per review comments. * A more elegant upgrade logic. * A more elegant upgrade logic. * [WIP][ENHANCEMENT] SimpleCheckpoint (great-expectations#2235) * * some more type annotations * action_list property on Checkpoint * SimpleCheckpoint stub * removed an unnecessary re-raise * added a few tests for Checkpoint WIP * decent default actions * decent site names functionality * a few more tests for Checkpoint * slack webhooks now validated using existing util (which was moved to prevent a circular dep) * new properties on SimpleCheckpiont More progress on SimpleCheckpoint * slack action built using notifyon/with * cleaner code * fixed default notify_with behavior * implemented guiding error messages in slack parameters * cleaned up code for readability slightly better docstring * remove name from actions * BaseDataContext now has a .get_site_names() method * better site validation on SimpleCheckpoint * removed redundant exception catch * put names back into action dicts * improved error messages on load_class util with tests * 3 mostly working SimpleCheckpoint.run() tests * convert SimpleCheckpoint class to a convenience class * cleanup and refactor * fix test after remvoing cherry pick * fix after merging CheckpointResult * * refactor fixtures * * docstring with examples * fixed a few tests * Checkpoint now validates datacontext * remove todo * * fixed a few load_class tests * removed some dead imports * added spark fixture to a few spark tests to prevent them from running when users select --no-spark * * configurator name change and design review notes * Add configurator to CheckpointConfig and schema * Isort; linting; test updates * Move SimpleCheckpointConfigurator into own module * Update tests * Add configurator arg to Checkpoint * Add **kwargs * Isort and linting * Update tests * Add typehints * Add code to use configurator if present at __init__ * Nix configurator from CheckpointConfig and schema * Fix typo * Fix import issues * Update tests * Add args specific to configurator * Add **kwargs and pass to configurator; assert no extra kwargs with no configurator * Nix missing kwarg to support LegacyCheckcpoint * Add configurator-specific fields to checkpoint config and schema * Isort and linting * Add new attributes to checkpoint config; fix typos * Add SimpleCheckpoint fixtures; update tests to use checkpoint from store * Isort and linting * Add post_dump processing of CheckpointConfig Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> * Checkpoint documentation for running and expanding configuration. (great-expectations#2288) * [ENHANCEMENT] Remove the Re-Use of BatchRequest as Internal DataConnector Protocol Object (great-expectations#2297) * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * [ENHANCEMENT] Simple checkpoint adjustment (great-expectations#2301) * Add args specific to SimpleCheckpoint * Move configurator logic out of Checkpoint and into SimpleCheckpoint * Clarify comment * Add SimpleCheckpoint to test_yaml_config * Clarify comments * Add SimpleCheckpoint-specific attributes to update * Linting * Add comment clarification * Nix SimpleCheckpoint attributes from update; nix setters * [MAINTENANCE] Clean Up Type Hints (great-expectations#2305) * Typing cleanup. * [MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ (great-expectations#2256) * upgrade snowflake-connector-python * pin boto3 and botocore for aiobotocore * Change dev environment setup guide to add constraints file * Update changelog Co-authored-by: anthonyburdi <anthony@superconductive.com> * [DOCS] WIP: docs updates for Checkpoints (great-expectations#2307) * [DOCS] WIP: docs updates for checkpoints * Some more tweaks to the formatting * Changed presumed release version * Added content for SimpleCheckpoint * Refactoring. * Checkpoint release tweaks (great-expectations#2322) * Nix comment * Make TODO comment more precise * Update docs/guides/how_to_guides/configuring_datasources/how_to_configure_a_pandas_filesystem_datasource.rst Co-authored-by: James Campbell <james.p.campbell@gmail.com> * Checkpoints upgrade help page. * Checkpoints upgrade help page. * Checkpoints upgrade help page. * Checkpoints upgrade help page. * Fix the docs. * Undo incorrect fix for the explanation of relative directory. * Update Checkpoints feature maturity (great-expectations#2323) * Update Checkpoints feature maturity * Update great_expectations/cli/checkpoint.py Co-authored-by: William Shin <will@superconductive.com> * Nix icon key Co-authored-by: William Shin <will@superconductive.com> * [BUGFIX] Updating the CLI for Checking Configuration -- Consistent Operation with Legacy Checkpoint Run (great-expectations#2324) * Checking configuration and version upgrade. * Cleanup. * [BUGFIX] Minor Fix to Y/N Prompt Behavior During Upgrade (great-expectations#2326) * Upgrade fix. * Fix to upgrade prompt logic. * [BUGFIX] Upgrade should NOT be invoked by CLI Checkpoint Run command (great-expectations#2328) * Upgrade must run seamlesly when checkpoint_run CLI command is run. * Default checkpoint store behavior. * Correct default checkpoint store behavior. * Do not upgrade when running a checkpoint manually. Upgrade should be explicitly requested. * wording * wording * A more robust check for existing checkpoints. * Typo * Cleaup. * error messages * Error messages. * More robust error checking. * Style of comments. * Style of comments. * [BUGFIX] ConfigurationStore TupleStoreBackend was incorrectly turning "." into "/" -- Fixed via File Template (great-expectations#2335) * Fix path for ConfigurationStore objects. * Update tests. * Typo. * Making tests work better with Azure. Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> Co-authored-by: anthonyburdi <anthony@superconductive.com> Co-authored-by: William Shin <will@superconductive.com> Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: Aylr <Aylr@users.noreply.github.com> Co-authored-by: Sam Bail <samanthapbail@gmail.com>

…t-expectations#2338) * Add checkpoint dir and module * Bring in checkpoint-related changes from Abe’s branch - https://github.com/superconductive/great_expectations/tree/refactor/create_checkpoint_class * Update tests * Update module name to reflect different dir structure * Fix typos * Update test * Add filesystem_csv_data_context_v3 fixture * Update tests * Update tests * Change Checkpoint to LegacyCheckpoint and NewStyleCheckpoint to Checkpoint * Checkpoint class rob (great-expectations#2157) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Alexsherstinsky/wip/checkpoints sync with develop 12 15 2020 1 (great-expectations#2166) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Alexsherstinsky/wip/checkpoint store 12 15 2020 2 (great-expectations#2170) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * [DOCS] Add some links to GreatExpectationsOperator repo, links to in-code docs (great-expectations#2167) * Add some links to GreatExpectationsOperator repo, links to in-code docs * fix line spacing and numbering * fix link * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Checkpoint store tests. Co-authored-by: anthonyburdi <anthony@superconductive.com> * Alexsherstinsky/wip/adding checkpoint store to data context 12 15 2020 3 (great-expectations#2173) * Enabling in-memory DataContextV3 instantiation. * Refactoring load_batch(). * Linting. * Remove unused imports. * Cleanup. * Refactoring and more tests. * Refactoring and docstrings. * Cleanup. * Cleanup. * Added a test of in-memory data_context configuration and instantiation from dictionary config. * Removing V3 WIP * Removing V3 DC WIP. * Delete DataContextV3 -- everything is unified in DataContext. * Linting. * Update the docs to stop using DataContextV3 and just use DataContext (it supports both V2 and V3 Datasource APIs). * Merge. * Merge. * Typo. * WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store WIP * Checkpoints configuration store tests. * minor * minor * tests * tests * Config testing. * Cleanup. * Linting. * Linting. * Linting. * Cleanup * Cleanup * Cleanup * cleanup * Cleanup. * Cleanup. * Config store improvements. * Configuration Store with tests. * WIP * WIP * Store Config merge. * Configuration Store * Configuration Store with tests. * Configuration Store with tests. * Config * Class hierarchy * Renamings * Linting. * Refactoring. * [DOCS] Add some links to GreatExpectationsOperator repo, links to in-code docs (great-expectations#2167) * Add some links to GreatExpectationsOperator repo, links to in-code docs * fix line spacing and numbering * fix link * Refactor * More tests for ConfigurationStore. * Cleanup. * Test errors. * Checkpoint store tests. * Checkpoint Store into DataContext * cleanup Co-authored-by: anthonyburdi <anthony@superconductive.com> * Checkpoint class rob (great-expectations#2175) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> * Alexsherstinsky/wip/checkpoint api 12 16 2020 4 (great-expectations#2182) * cleanup * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Alexsherstinsky/wip/checkpoint update tests 12 17 2020 5 (great-expectations#2186) * Fix a configuration related test for checkpoints. * [ENHANCEMENT] Docstrings for DataConnector and associated classes (great-expectations#2172) * Update changelog for develop * some clean up on docstrings * tests * Fixing tests. * More tests. Co-authored-by: William Shin <will@superconductive.com> * Alexsherstinsky/wip/checkpoint update tests 12 17 2020 6 (great-expectations#2187) * Fix a configuration related test for checkpoints. * [ENHANCEMENT] Docstrings for DataConnector and associated classes (great-expectations#2172) * Update changelog for develop * some clean up on docstrings * tests * Fixing tests. * More tests. * self_check for checkpoint store * linting Co-authored-by: William Shin <will@superconductive.com> * Checkpoint class rob (great-expectations#2188) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * cleanup * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Update schema * Allow user to pass in all checkpoint attributes * Update Checkpoint __init__ * Allow user to provide all checkpoint attributes * Update run methods * Update run_checkpoint * Fix typo * Update formatting * Change template to template_name; add setters and update methods * Add update method * Change template to template_name * Add limit and batch_spec_passthrough to json dict output * Update run methods; add method to get substituted config * Isort and linting * Isort * Switch order * Fix typo * Nix old code * Update sig of update_action_list, add delete behavior * Fix config version for legacy checkpoint * Fix config version for legacy checkpoint * Add unit test for checkpoint config template/runtime substitution * Isort and linting * Update test * Add comment * Update test * Nix inheritance from object * Nix cast Co-authored-by: Alex Sherstinsky <alex@superconductivehealth.com> Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> * Alexsherstinsky/wip/checkpoint using test yaml config 12 17 2020 7 (great-expectations#2192) * test_yaml_config for CheckpointStore * Lint * self_check and test_yaml_config * merge * Merge. * linting * linting * Fix tests. * Alexsherstinsky/wip/checkpoint fix tests in azure 12 18 2020 8 (great-expectations#2195) * tests in Azure * debub * [MAINTENANCE] Anthony/bugfix/show cta footer issue 2120 (great-expectations#2190) * remove show_cta_footer from base.py defaults * Add note that show_cta_footer was deprecated * remove show_cta_footer from in-code config tests * reformat with new black version * troubleshooting azure errors * cleanup Co-authored-by: anthonyburdi <anthony@superconductive.com> * Checkpoint.run (great-expectations#2202) * Add new types and schema for CheckpointConfig and CheckpointValidationConfig * Update constant name * Add additional attributes * Isort and linting * Fix typo * Fix indent * Restore test_yaml_config * Update names * Linting * Update name * Change template name * Add checkpoint_template.yml * Update var name * Add placeholder for SimpleCheckpointConfig * Update test * Add LegacyCheckpointConfig and schema * Add support for LegacyConfig * Isort and linting * cleanup * Add checkpoint_store to fixture * Nix @post_lost * Fix typos * Add yaml indent * Nix deprecated code and implement list/create/get checkpoint * Update script template * Make CheckpointConfiguraiton store implementation more elegant by using the class variable that is the class of the actual store. * Checkpoint store operations. * Linting. * Minor * Tests. * Update schema * Allow user to pass in all checkpoint attributes * Update Checkpoint __init__ * Allow user to provide all checkpoint attributes * Update run methods * Update run_checkpoint * Fix typo * Update formatting * Change template to template_name; add setters and update methods * Add update method * Change template to template_name * Add limit and batch_spec_passthrough to json dict output * Update run methods; add method to get substituted config * Isort and linting * Isort * Switch order * Fix typo * Nix old code * Update sig of update_action_list, add delete behavior * Fix config version for legacy checkpoint * Fix config version for legacy checkpoint * Add unit test for checkpoint config template/runtime substitution * Isort and linting * Update test * Add comment * Update test * Nix inheritance from object * Nix cast * tests in Azure * debub * troubleshooting azure errors * cleanup * linting * Set runtime_configuration and evaluation_parameters to {} if None * Instantiate BatchRequest if input batch_request is a dict * Change update_action_list to class_method get_updated_action_list; update batch_request update handling * Add util methods - get_runtime_batch_request - validate_validation_dict - get_substituted_validation_dict - get_run_name_from_template * Fill out run method * Add test for newstyle checkpoint .run * Add test for data_context.run_checkpoint for newstyle checkpoint * Update action_list update from runtime kwargs * Isort and linting * Add type hints; add exception case * isort and linting * Address pr review comments - add type hints - make methods private * Add type hints and fix typo * Linting Co-authored-by: Alex Sherstinsky <alexsherstinsky@users.noreply.github.com> Co-authored-by: Alex Sherstinsky <alex@superconductivehealth.com> * [ENHANCEMENT][BUGFIX] Alexsherstinsky/wip/checkpoint test yaml config doc 12 21 2020 9 (great-expectations#2206) * test_yaml_config -- more tests in preparation for documentation * Various checkpoint configuration options. * Cleanup. * Update explore_expectations_in_a_notebook.rst (great-expectations#2174) * Update to support not using BatchRequest in configuration. * [BUGFIX] Made some improvements for the treatment of decimals for database backends (great-expectations#2207) * Updated lossy conversion message in ensure_json_serializable and updated sqlalchemy_dataset to convert numeric metrics to json_serializable up front. * Update test cases for improved lossy conversion logic Signed-off-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: James Campbell <james.p.campbell@gmail.com> * pass manually_initialize_store_backend_id to database store backends, add test (great-expectations#2205) * Update validations and tests. * fix messaging and tests Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: anthonyburdi <anthony@superconductive.com> * [DOCS][ENHANCEMENT] Alexsherstinsky/wip/checkpoint test yaml config doc 12 22 2020 10 (great-expectations#2212) * Checkpoints howto using test_yaml_config * Rename * [BUGFIX][ENHANCEMENT] Alexsherstinsky/wip/checkpoint test yaml config doc examples 01 04 2021 11 (great-expectations#2226) * Add an assertion for checkpoint test. * clean up * New packages * Debug * Debug * Debug * Fix tests for new Pandas version. * [ENHANCEMENT][BUGFIX] Alexsherstinsky/wip/checkpoint test yaml config doc examples 01 04 2021 12 (great-expectations#2228) * Add an assertion for checkpoint test. * clean up * Adding run_checkpoint to all Checkpoint configuration examples. * Adding run_checkpoint to all Checkpoint configuration examples. * cleanup * Minor * [MAINTENANCE] Alexsherstinsky/maintenance/checkpoint test yaml config notebook 01 06 2021 15 (great-expectations#2236) * rename method to add checkpoint to fit with the spec * typo * fix documentation * More flexibility to checkpoint_run test. * [BUGFIX] [ENHANCEMENT] Alexsherstinsky/maintenance/checkpoint test yaml config notebook 01 07 2021 16 (great-expectations#2242) * DataFrame in batch_spec is not needed after it is processing by ExecutionEngine. * Make warning about non-existent class_name disappear. * Asset config. * Minor * Warnings * Printing * Formatting. * Schema adherence check. * [ENHANCEMENT] Alexsherstinsky/maintenance/checkpoint versioning test yaml config notebook 01 08 2021 18 (great-expectations#2245) * Fixtures refactoring. * Error messages. * tests * Fix a test. * Refactor Checkpoints to allow easy addition of the SimpleCheckpoint class. (great-expectations#2248) * Refactor `Checkpoint` class instantiation in such a way that the addition of `LegacyCheckpoint` and `SimpleCheckpoint` subclasses of the `Checkpoint` class becomes very easy. * [ENHANCEMENT] CheckpointResult type (great-expectations#2238) Changes proposed in this pull request: - addition of new CheckpointResult type, which is output from the Checkpoint.run method - the new class is nearly equivalent to ValidationOperatorResult, with some exceptions: - no top-level evaluation_parameters key - the "actions_results" key is now expected (not enforced atm though) in each validation dict within run_results - batch_definition replaces batch_kwargs Previous Design Review notes: - https://github.com/superconductive/design/pull/24#discussion_r552861330 * * DataContext.get_checkpoint now returns a Checkpoint (great-expectations#2249) Changes proposed in this pull request: This PR removes the return_config parameter to make DataContext.get_checkpoint() do what it says by returning a Checkpoint. CheckpointConfigs are easy to access from the Checkpoint.config property. * empty * [ENHANCEMENT] [BUGFIX] Configuration Version Increase from 2 to 3; Fix for DataAsset Configuration Schema Validation (great-expectations#2253) * Version control against CheckpointStore * Warning for using validation_operators with version 3. * Warning for using validation_operators with version 3. * checkpoint_store error messaging * Configuration Version update from 2 to 3. * Adding batch_spec_passthrough to the AssetConfig and AssetConfigSchema for properly passing it to ExecutionEngine classes. * Cleanup * Cleanup * Cleanup * Use monkeypatch for environment variables. * Fixed a typo. * Cleanup. * Update tests to reflect new packages impacting notebook rendering. * CLI warnings update. * [DOCS][ENHANCEMENT] move validation actions into checkpoints; docs updates (great-expectations#2255) * Move validation actions to checkpoint module * Move action util functions from validation_operator module to checkpoint module * Add .actions * Update doc with new CheckpointResult info * Add warning to doc * NIx empty util.py * Isort and linting * Update docs/guides/how_to_guides/validation/how_to_create_a_new_checkpoint_using_test_yaml_config.rst Co-authored-by: Taylor Miller <Aylr@users.noreply.github.com> * NIx util.py from verify_dynamic_loading_support call * Linting Co-authored-by: Taylor Miller <Aylr@users.noreply.github.com> * [BUGFIX] GE DataContext with multiple DataSources should not scan all configurations (great-expectations#2250) * push before tests * Update data_context.py * fixing bug, added comments * update base.py to always include datasource as dict * added test and fixtures * changed comment * Update tests/conftest.py * cleaned up unnecessary refactor * Update data_context.py * linting * Update test_data_context.py * Update test_data_context.py * [ENHANCEMENT] Upgrade Capability for config_version 2 to config_version 3 with Checkpoints Configuration Migration (great-expectations#2279) * Checkpoint upgrade * Typo on merge. * Update tests to account for Checkpoint store data migration. * Upgrade tests. * Manual steps tests. * Rename to a more general upgrade test module. * Add a test for v2_to_v3_upgrade (Checkpoints migration). * Robust error handling in upgrade tests. * Small typo in tests. * Update documentation by including full checkpoints capabilities. * Updates per review comments. * A more elegant upgrade logic. * A more elegant upgrade logic. * [WIP][ENHANCEMENT] SimpleCheckpoint (great-expectations#2235) * * some more type annotations * action_list property on Checkpoint * SimpleCheckpoint stub * removed an unnecessary re-raise * added a few tests for Checkpoint WIP * decent default actions * decent site names functionality * a few more tests for Checkpoint * slack webhooks now validated using existing util (which was moved to prevent a circular dep) * new properties on SimpleCheckpiont More progress on SimpleCheckpoint * slack action built using notifyon/with * cleaner code * fixed default notify_with behavior * implemented guiding error messages in slack parameters * cleaned up code for readability slightly better docstring * remove name from actions * BaseDataContext now has a .get_site_names() method * better site validation on SimpleCheckpoint * removed redundant exception catch * put names back into action dicts * improved error messages on load_class util with tests * 3 mostly working SimpleCheckpoint.run() tests * convert SimpleCheckpoint class to a convenience class * cleanup and refactor * fix test after remvoing cherry pick * fix after merging CheckpointResult * * refactor fixtures * * docstring with examples * fixed a few tests * Checkpoint now validates datacontext * remove todo * * fixed a few load_class tests * removed some dead imports * added spark fixture to a few spark tests to prevent them from running when users select --no-spark * * configurator name change and design review notes * Add configurator to CheckpointConfig and schema * Isort; linting; test updates * Move SimpleCheckpointConfigurator into own module * Update tests * Add configurator arg to Checkpoint * Add **kwargs * Isort and linting * Update tests * Add typehints * Add code to use configurator if present at __init__ * Nix configurator from CheckpointConfig and schema * Fix typo * Fix import issues * Update tests * Add args specific to configurator * Add **kwargs and pass to configurator; assert no extra kwargs with no configurator * Nix missing kwarg to support LegacyCheckcpoint * Add configurator-specific fields to checkpoint config and schema * Isort and linting * Add new attributes to checkpoint config; fix typos * Add SimpleCheckpoint fixtures; update tests to use checkpoint from store * Isort and linting * Add post_dump processing of CheckpointConfig Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> * Checkpoint documentation for running and expanding configuration. (great-expectations#2288) * [ENHANCEMENT] Remove the Re-Use of BatchRequest as Internal DataConnector Protocol Object (great-expectations#2297) * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * Removing the usage of BatchRequest as internal protocol in the DataConnector class hierarchy. * [ENHANCEMENT] Simple checkpoint adjustment (great-expectations#2301) * Add args specific to SimpleCheckpoint * Move configurator logic out of Checkpoint and into SimpleCheckpoint * Clarify comment * Add SimpleCheckpoint to test_yaml_config * Clarify comments * Add SimpleCheckpoint-specific attributes to update * Linting * Add comment clarification * Nix SimpleCheckpoint attributes from update; nix setters * [MAINTENANCE] Clean Up Type Hints (great-expectations#2305) * Typing cleanup. * [MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ (great-expectations#2256) * upgrade snowflake-connector-python * pin boto3 and botocore for aiobotocore * Change dev environment setup guide to add constraints file * Update changelog Co-authored-by: anthonyburdi <anthony@superconductive.com> * [DOCS] WIP: docs updates for Checkpoints (great-expectations#2307) * [DOCS] WIP: docs updates for checkpoints * Some more tweaks to the formatting * Changed presumed release version * Added content for SimpleCheckpoint * Checkpoint release tweaks (great-expectations#2322) * Nix comment * Make TODO comment more precise * Update Checkpoints feature maturity (great-expectations#2323) * Update Checkpoints feature maturity * Update great_expectations/cli/checkpoint.py Co-authored-by: William Shin <will@superconductive.com> * Nix icon key Co-authored-by: William Shin <will@superconductive.com> * [BUGFIX] Updating the CLI for Checking Configuration -- Consistent Operation with Legacy Checkpoint Run (great-expectations#2324) * Checking configuration and version upgrade. * Cleanup. * [BUGFIX] Minor Fix to Y/N Prompt Behavior During Upgrade (great-expectations#2326) * Upgrade fix. * Fix to upgrade prompt logic. * [BUGFIX] Upgrade should NOT be invoked by CLI Checkpoint Run command (great-expectations#2328) * Upgrade must run seamlesly when checkpoint_run CLI command is run. * Default checkpoint store behavior. * Correct default checkpoint store behavior. * Do not upgrade when running a checkpoint manually. Upgrade should be explicitly requested. * wording * wording * A more robust check for existing checkpoints. * Typo * Cleaup. * error messages * Error messages. * More robust error checking. * [BUGFIX] ConfigurationStore TupleStoreBackend was incorrectly turning "." into "/" -- Fixed via File Template (great-expectations#2335) * Fix path for ConfigurationStore objects. * Update tests. * Typo. * Making tests work better with Azure. * [DOCS] checkpoint core concepts (great-expectations#2261) * very rough WIP first draft on a Checkpoints core concepts page * Isort and linting * Add specifying_batches ref * Fill out core concept doc * Minor tweaks * Add section on CheckpointResult and SimpleCheckpoint; expand examples section * Fix typo * Just some minor format tweaks Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> Co-authored-by: Sam Bail <sam@superconductive.com> Co-authored-by: Robert Moses Lim <robert.m.lim@gmail.com> Co-authored-by: anthonyburdi <anthony@superconductive.com> Co-authored-by: William Shin <will@superconductive.com> Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: Aylr <Aylr@users.noreply.github.com> Co-authored-by: Sam Bail <samanthapbail@gmail.com> Co-authored-by: Sam Bail <sam@superconductive.com>

anthonyburdi added 2 commits January 14, 2021 15:49

upgrade snowflake-connector-python

5ffeba7

pin boto3 and botocore for aiobotocore

ee46f2b

anthonyburdi self-assigned this Jan 14, 2021

anthonyburdi added 2 commits January 19, 2021 16:13

Change dev environment setup guide to add constraints file

ec94d31

Merge branch 'develop' into anthony/bugfix/new-pip-resolver

3ca58ea

anthonyburdi changed the title ~~Anthony/bugfix/new pip resolver~~ [MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ Jan 19, 2021

anthonyburdi marked this pull request as ready for review January 19, 2021 21:42

anthonyburdi requested a review from jcampbell January 19, 2021 21:43

anthonyburdi added 4 commits January 19, 2021 16:50

Update changelog

ea5b7ba

Merge branch 'develop' into anthony/bugfix/new-pip-resolver

83432a1

Merge branch 'develop' into anthony/bugfix/new-pip-resolver

da957ec

# Conflicts: # docs/changelog.rst

Merge branch 'develop' into anthony/bugfix/new-pip-resolver

cdc5d84

jcampbell approved these changes Jan 22, 2021

View reviewed changes

anthonyburdi merged commit f76fe54 into develop Jan 22, 2021

anthonyburdi deleted the anthony/bugfix/new-pip-resolver branch January 22, 2021 21:13

spbail mentioned this pull request Feb 8, 2021

Installing development dependencies is extremely slow when using new pip 20.3's new resolver #2191

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ #2256

[MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ #2256

anthonyburdi commented Jan 14, 2021 •

edited

Loading

[MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ #2256

[MAINTENANCE] Ensure compatibility with new pip resolver v20.3+ #2256

Conversation

anthonyburdi commented Jan 14, 2021 • edited Loading

anthonyburdi commented Jan 14, 2021 •

edited

Loading