[MAINTENANCE] Minor Validator code clean up -- for better code clarity #4147
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
✔️ Deploy Preview for niobium-lead-7998 ready! 🔨 Explore the source changes: 382dbbf 🔍 Inspect the deploy log: https://app.netlify.com/sites/niobium-lead-7998/deploys/62022ac92f0b1f0007201aa5 😎 Browse the preview: https://deploy-preview-4147--niobium-lead-7998.netlify.app |
HOWDY! This is your friendly 🤖 CHANGELOG bot 🤖Please don't forget to add a clear and succinct description of your change under the Develop header in ✨ Thank you! ✨ |
alexsherstinsky
requested review from
anthonyburdi,
cdkini,
NathanFarmer,
Shinnnyshinshin and
donaldheppner
and removed request for
anthonyburdi
anthonyburdi
approved these changes
Feb 8, 2022
alexsherstinsky
deleted the
feature/GREAT-564/GREAT-568/alexsherstinsky/rule_based_profiler_instantiate_rule_based_profiler_using_config_object-2022_02_07-13
branch
Rachel-Reverie
pushed a commit
that referenced
this pull request
Feb 25, 2022
Rachel-Reverie
added a commit
that referenced
this pull request
Mar 7, 2022
* Added universal map resources: imagery, mdx files for quickly loading that imagery, and css for creating borderless and/or centered tables. * Doc 49 overview update (#3919) * Added universal map resources: imagery, mdx files for quickly loading that imagery, and css for creating borderless and/or centered tables. * Added setup_overview.md * Added the "Step 1: Setup" section to the ToC, along with related How To guides and the overview page for setup. * Added a space insisde of a comment so we could have a commit to trigger a netlify build. * Moved setup_overview.md into `guides/setup` Updated sidebars.js Removed broken links from setup_overview.md * Wording change in setup_overview.md intro and wrapping up paragraphs. Added index pages for Setup how to guides. Linked to index pages for Setup how to guides in setup_overview.md and sidebars.js * Update docs/guides/setup/setup_overview.md Co-authored-by: Abe Gong <abegong@users.noreply.github.com> * Update docs/guides/setup/setup_overview.md Co-authored-by: Abe Gong <abegong@users.noreply.github.com> * Update docs/guides/setup/setup_overview.md Co-authored-by: Abe Gong <abegong@users.noreply.github.com> * Updated wording in the prerequisites section * Changed references to "Metadata Stores" in the Setup documentation to "Stores" * corrected typo, better phrasing for last line of Wrapping up * Minor change in wording in the wrapping up section. * Update setup_overview.md Corrected minimum version of Python required. * Moved how to guides' category indexes for Step 1: Setup to the end of each subsection in the ToC. * Moved how to guides' category indexes for Step 1: Setup (Data Docs and Stores sections) to the end of each subsection in the ToC. * Correction to header levels for sub-headings and sub-sub-headings of The Setup Process. Co-authored-by: Abe Gong <abegong@users.noreply.github.com> * correction to format of relative links in index page. * - Reorganized ToC for "Step 3: Create Expectations" contents. (#3999) - Added index page for "Step 3: Create Expectations" how to guides. - Added placeholder page for "Step 3: Overview" so that corresponding links could be added. (Guidance on content is pending) * Doc 52 connect to data overview (#3970) * Partial rough draft of overview.md for the Connect to Data step. * More fleshed out rough draft of overview.md * complete rough draft for overview.md * Minor edits to overview.md Added overview.md and corresponding how to documents into the ToC * Partial rough draft of overview.md for the Connect to Data step. * More fleshed out rough draft of overview.md * complete rough draft for overview.md * rebased on parent branch * Nested Connect to Data guides under How to guides in the ToC * Starting updates on index pages for the Connect to Data how to documents. * Update docs/guides/connecting_to_your_data/overview.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/guides/connecting_to_your_data/overview.md Co-authored-by: talagluck <tal@superconductive.com> * Extended section on `test_yaml_config()` * Removed sections of the ToC that were moved into "1. Setup" and "2. Connecting to Data". * Added `batch_identifiers` as a potential key to replace <additional_keys_based_on_source_data_systems> in the examples when the source data system is a filesystem. * Updated `InferredAssetDataConnector` section to include that `InferredAssetSqlDataConnector` provides similar functionality for SQL based source data systems. * Added index for Step 2: Connect to Data to sidebar.js; updated contents of index to correspond to folder structure in sidebar.js * Updated relative links for index.md * Update docs/guides/connecting_to_your_data/overview.md Co-authored-by: Abe Gong <abegong@users.noreply.github.com> * Update docs/guides/connecting_to_your_data/overview.md Co-authored-by: Abe Gong <abegong@users.noreply.github.com> * - Changed heading of section 1. Boilerplate to 1. Prepare scaffolding - Refactored name of overview file to match naming convention of other step's overview files. - Updated internal links and references to the refactored overview file's name. * - Corrected spelling of `test_yaml_config()` Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: Abe Gong <abegong@users.noreply.github.com> * Docs/doc 72/doc 88/validate data toc reorg (#4003) * - Reorganized ToC for "Step 4: Validate Data" contents. - Added index page for "Step 4L Validate Data" how to guides. - Added placeholder page for "Step 4: Overview" so that corresponding links could be added. (Guidance on content is pending) * Added universal map graphical outline to Validate Data: Overview * - Made corrections to sidebars.js for improperly merged contents of Step 2: Connect to Data how to guides. - Standardized Index headers * First draft content for the Create Expectations: Overview page. (#4017) * First draft content for the Create Expectations: Overview page. * Update docs/guides/expectations/create_expectations_overview.md Co-authored-by: William Shin <will@superconductive.com> * Update docs/guides/expectations/create_expectations_overview.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/guides/expectations/create_expectations_overview.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/guides/expectations/create_expectations_overview.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/guides/expectations/create_expectations_overview.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/guides/expectations/create_expectations_overview.md I'm fine with calling it the profiler workflow. I'll make that change as well. Co-authored-by: talagluck <tal@superconductive.com> * Update docs/guides/expectations/create_expectations_overview.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/guides/expectations/create_expectations_overview.md I think this needs to be "When following one of" but I'll make that edit after committing these changes. Co-authored-by: talagluck <tal@superconductive.com> * Update docs/guides/expectations/create_expectations_overview.md Co-authored-by: talagluck <tal@superconductive.com> * Changed "automatic" workflow to "Profiler" workflow. Minor edits for grammar. * changed "in depth" to "in-depth" where appropriate. Co-authored-by: William Shin <will@superconductive.com> Co-authored-by: talagluck <tal@superconductive.com> * Draft for "Validate Data: Overview" page (#4032) * Rough draft for "Validate Data: Overview" page * Updated some phrasing for clarity. * Update docs/guides/validation/validate_data_overview.md Co-authored-by: talagluck <tal@superconductive.com> Co-authored-by: talagluck <tal@superconductive.com> * Updated relative paths in index.md for Setup folder. * Added illustrations to the Connect to data: Overview page. Minor edits to text where the illustrations were inserted. (#4048) * Added illustration to the "Validate Data: Overview" page. Minor edits to text where the illustration was inserted. (#4053) * Added illustration to the "Create Expectations: Overview" page. Minor edits to text where the illustration was inserted. (#4052) * Updates to the links in site_builder.py _get_call_to_action_buttons() to all be V3 links instead of V2. (#4054) * Doc 50 glossary update (#4065) * First Pass of creating glossary terms files. * Added additional Technical Terms to the glossary, and added some notes for the eventual detailed definitions to some Technical Terms. * Added docusaurus-terminology plugin and compiled the glossary page. * reformatted and added the glossary. removed links to details pages from glossary.md Corrected typo in validation_result_store.md * removed remaining links in glossary.md * rough drafts/notes on datasource.md and store.md * created _tag.mdx, which uses tag.json to easily provide Technical Terms with a hover text definition and url link. Added custom styling necessary for hovertext Corrected typo in batch_request.md Tested _tag.mdx and terms.json imports in glossary.md (This will need to be redone, but for now it provides an example of how to use the _tag.mdx import in other files and should provide the basis for having glossary.md automatically update based on the contents of terms.json) * Added all technical terms and definitions to terms.json * Fixed typo in "Batch" * Updates to sidebars.js to include technical terms folder. Updates to technical term files to include tagged terms. update _tag.mdx to accept relative path directions to the docs folder. * Update terms.json * Batch Definition Candidate Edits (#4020) * Terms language tweaks * First draft Batch extended definition * datasource language tweak * Added build_glossary_page.py as a utility script - it will generate an alphabetically sorted glossary page based on the contents of terms.json. Updated the glossary.md using build_glossary_page.py * Added build_glossary_page.py as a utility script - it will generate an alphabetically sorted glossary page based on the contents of terms.json. Updated the glossary.md using build_glossary_page.py * Minor edit to glossary of terms page for consistency in formatting (separating content from metadata by one line in source files.) * Cleaned up custom CSS after conflict in import from parent branch. Corrected links in batch.md Co-authored-by: James Campbell <james.p.campbell@gmail.com> * Docs/doc 72/doc 91/toc and misc universal map updates (#4066) * Consolidated all contributing documentation in the Contributing folder of the ToC * Moved all contributing guides into the contributing folder. Moved miscellaneous how to guides into appropriate folders based on when they are applicable. Combined changelog and migrationg guide into "Updates and Migration" section. Combined API Reference and Core Concepts into Reference section. Moved how to use the GE CLI into Reference section. Added Advanced section to Setup How to guides to contain moved miscellaneous guides. * Updated definitions in terms.json and glossary to reduce emphasis on terms as objects. * Added notice for GE Cloud into Setup: Overview document under "The alternative to manual Setup" section. * Corrected "in depth guide" to "in-depth guide" (in-depth is the adjective version.) * Added index page to "Deployment Guides" section; updated indexes in all "how to guide" sections. * Docs/doc 95/doc 100/technical term data context (#4074) * - Updated content for the Data Context technical term page. - Minor edits to the Setup: Overview how-to guide index. * Added tags for technical terms in data_context.md * - Corrected some headings that were improperly title cased. * - Updated admonition boxes to have proper syntax highlighting and titles. * plugin technical term (#4077) * - Added Plugin technical term page and content. - Added Plugin entry to terms.json * - Updated Glossary to include Plugin entry. * - minor edits for grammar/spelling * Update docs/glossary.md Co-authored-by: talagluck <tal@superconductive.com> * Update plugin.md Co-authored-by: talagluck <tal@superconductive.com> * Added technical term documentation for Execution Engines. * checkpoint technical term page (#4075) * - Add Checkpoint technical term page content. - Add Checkpoint technical term page to ToC. - Add information from Checkpoint core concepts to other technical term page's notes sections (for when those documents are created in full.) * - Corrected a technicla term that was not title cased. * - Added technical term tags and updated technical terms to proper title case format. * Docs/doc 95/doc 109/technical term validation (#4118) * - Linked the Validation technical term to the "Validate Data: Overview" page. * - Added reference to Plugins in setup_overview - Minor correction to spelling in plugin doc. * Docs/doc 95/doc 105/technical term expectation suite (#4105) * - Added base content to expectation_suite.md - Added expectation_suite.md to the ToC * - Added tags to Expectation Suite technical term page. - Minor edits to Expectation Suite technical term page. * Update docs/terms/expectation_suite.md Co-authored-by: James Campbell <james.p.campbell@gmail.com> * Update docs/terms/expectation_suite.md Co-authored-by: James Campbell <james.p.campbell@gmail.com> * Update docs/terms/expectation_suite.md Co-authored-by: James Campbell <james.p.campbell@gmail.com> * - updated Expectation Suite page Reusability section to account for individual Expectation Suites being paired with individual Batch Requests, rather than all Expectation Suites running on all Batch Requests in the Checkpoint. Co-authored-by: James Campbell <james.p.campbell@gmail.com> * Docs/doc 95/doc 102/technical term action (#4089) * Updated Validation Action to Action in the files and terms.json Added Action content and page * Added tags to technical terms. * Corrected keys in some technical tags. * - updated information on `data_asset` parameter in Actions to indicate it will be a Validator instance. * Update docs/terms/action.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/action.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/action.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/action.md * - Updated Action definition to be more concise - Added note regarding contributions and Slack consultation in relation to developing custom Actions. - Corrected casing for Python - Added clarification regarding order Actions are run in. Co-authored-by: talagluck <tal@superconductive.com> * technical terms: expectation and execution engine (#4094) * - started Expectation documentation - Added missing documentation to Execution Engine technical term page. * - Expanded content in Expectation technical term page. * - finished adding content to expectation.md * - Added tags to expectation.md - Converted Validation Action to Action in linked documents. * -updates to sidebar layout; added Additional Documentation to the reference section, for docs that are supplemental to technical term documentation but aren't API documentation. * slight tweak to Reference category in ToC * - Added base content to expectation_suite.md - Added expectation_suite.md to the ToC * - Added tags to Expectation Suite technical term page. - Minor edits to Expectation Suite technical term page. * Update docs/terms/expectation_suite.md Co-authored-by: James Campbell <james.p.campbell@gmail.com> * Update docs/terms/expectation_suite.md Co-authored-by: James Campbell <james.p.campbell@gmail.com> * Update docs/terms/expectation_suite.md Co-authored-by: James Campbell <james.p.campbell@gmail.com> * Update docs/terms/execution_engine.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/expectation.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/expectation.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/expectation.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/expectation.md Co-authored-by: talagluck <tal@superconductive.com> * - Updates to the Expectation page based on conversations on GitHub. * - minor edits to execution engine page. * Update docs/terms/execution_engine.md Co-authored-by: talagluck <tal@superconductive.com> * - Added note pertaining to developing new Execution Engines. - Removed section on Design motivations. Co-authored-by: James Campbell <james.p.campbell@gmail.com> Co-authored-by: talagluck <tal@superconductive.com> * - Added content to supporting resource technical term page. (#4126) - Added supporting resource technical term page to ToC * Docs/doc 95/doc 110/technical term validation result (#4122) * - Updated definition of "Validation Result" to include `ExpectationValidationResult` and `ExpectationSuiteValidationResult` objects. - Added content to "Validation Result" technical term page. - Added "Validation Result" page to ToC. * - Minor updates for clarity. * - Additional edits for clarity between `ExpectationValidationResult` and `ExpectationSuiteValidationResult` instances. - Added technical term tags. * Update docs/terms/validation_result.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/validation_result.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/validation_result.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/validation_result.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/validation_result.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/validation_result.md Co-authored-by: talagluck <tal@superconductive.com> * - added clarification of the differences between `ExpectationSuiteValidationResult` and `ExpectationValidationResult`. Co-authored-by: talagluck <tal@superconductive.com> * "Profiler" technical term page. (#4120) * - Added content to "Profiler" technical term page. - Added "Profiler" technical term page to the ToC * - updated content to reference RuleBasedProfiler class instead of Profiler class in accordance with the updated how-to guides. * Update docs/terms/profiler.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/profiler.md Co-authored-by: talagluck <tal@superconductive.com> * - updates to the documentation based on GitHub discussions. * - fixed missing comma in sidebars.js Co-authored-by: talagluck <tal@superconductive.com> * Docs/doc 95/doc 106/technical term metric (#4116) * - Added content for Metric technical term page. - Added Metric technical term page to ToC * - Added technical term tags to Metric page - Minor editing to comply with documentation standards - Updated "Validation Action" references to "Action" * - Added technical term tags to Metric page - Minor editing to comply with documentation standards * Update docs/terms/metric.md Co-authored-by: talagluck <tal@superconductive.com> * Update docs/terms/metric.md Co-authored-by: talagluck <tal@superconductive.com> * - updates to the documentation based on GitHub discussions. * Removed nonsensical statement. Will replace with clarified version if anyone can figure out what it was meant to convey. * - Removed nonsensical statement. Will replace with clarified version if anyone can figure out what it was meant to convey. - Removed extraneous path information alongside github url. * -Updated technical term tag for Action after merging in updates. Co-authored-by: talagluck <tal@superconductive.com> * Docs/doc 95/doc 116/technical term data docs (#4199) * - Added base content to technical term page for Data Docs. - Added Data Docs technical term page to ToC. * - Added technical term tags to the Data Docs technical term page. * - Updated according to GitHub suggestions. * - Added technical term page for Custom Expectations - Added Custom Expectations page to ToC - Added technical term tags to Custom Expectations page * Docs/doc 95/doc 126/technical term validator (#4254) * - Added technical term page for Validator - Added Custom Validator page to ToC * - Added technical term tags to Validator page * typo in validator.md Co-authored-by: Joshua Stauffer <66793731+joshua-stauffer@users.noreply.github.com> * - Added technical term page for Renderers (#4250) - Added Custom Renderers page to ToC - Added technical term tags to Renderers page * Docs/doc 95/doc 107/technical term evaluation parameter (#4108) * - Added initial content to technical terms page for Evaluation Parameters. - Added technical term page for Evaluation Parameters to the ToC * - Added tags to technical terms in Evaluation Prameters technical terms page. - Updated Evaluation Parameter's technical terms page to reflect interaction of Evaluation Parameters, Checkpoints and Actions. * - Content updates for clarity and completeness per resolved discussions on GitHub. * - Corrected issue where definition of Evaluation Parameter had been assigned as the definition of Evaluation Parameter Store. * - Added mention of writing Evaluation Parameters based on domain knowledge while interactive evaluation is disabled. * - Added example for configuring Evaluation Parameter to pull value from environment varibale. - Added example for passing Evaluation Parameter to Checkpoint as a dictionary. * - Updated all "my_df." examples from V2 to "validator." examples compliant with V3 - Added the examples regarding the `evaluation_parameters` parameter in Checkpoints (but they still need tested). * - corrected missing closed parenthesis in one of the code examples. * -Updated technical term tag for Action after merging in updates. * Datasource technical term page (#4205) * - Updated documentation for Datasource technical term page * - Made corrections and additions corresponding to GitHub conversation. * - Made correction corresponding to GitHub conversation. * Update datasource.md (#4208) * added missing word * removed extra punctuation Co-authored-by: Joshua Stauffer <66793731+joshua-stauffer@users.noreply.github.com> * Docs/doc 95/doc 119/technical terms batch and request (#4227) * - clarification that Batch Requests and Expectation Suites are configured or given to Checkpoints in pairs. - added Batch and Batch Request technical term pages to the ToC - updated Batch Request technical term page with the blank template for technical term pages. * - Updated Batch technical term page to reflect technical term page template. - Added content to Batch Request technical term page. - Added content to _batches_and_batch_requests.mdx which will be displayed in the "More details" section of both the Batch and Batch Request technical term pages. - Corected link in how_to_add_validations_data_or_suites_to_a_checkpoint.md that was pointing to the wrong file. * Minor edits to content. * Added technical term tags to Batch and Batch Request pages * incorporate new 'life of a batch request' diagram (#4233) * - Made further correction corresponding to GitHub conversation. * capitalize checkpoint * update to will's formatted diagram Co-authored-by: Joshua Stauffer <66793731+joshua-stauffer@users.noreply.github.com> * technical term data connector (#4207) * - Added basic content to Data Connector technical term page. - Added Data Connector technical term page to the ToC. * - Added technical term tags to Data Connector page. * - Update to Features section. * Update data_connector.md (#4212) * Update data_connector.md * Update data_connector.md * Update data_connector.md * - Made corrections and additions corresponding to GitHub conversation. * Update data_connector.md (#4210) * Update data_connector.md (#4213) no space in SQLAlchemy Co-authored-by: Joshua Stauffer <66793731+joshua-stauffer@users.noreply.github.com> * technical term CLI (#4228) * - Added technical term page for CLI - added CLI page to ToC * - Added technical term tags to CLI page. * - Made additions and corrections corresponding to GitHub conversation. * [MAINTENANCE] Additional distinction made between V2 and V3 upgrade script (#4046) * push of changes and note * added note to the V11 upgrader * Delete test_v2_to_v3_project_upgrade_expected_v012_stdout.fixture2 * Update test_project_upgrade_with_manual_steps_expected_v012_stdout.fixture * [MAINTENANCE] Correcting Checkpoint Configuration and Execution Implementation (#4015) * [FEATURE] add bigqquery json credentials kwargs for sqlalchemy connect (#4039) * add kwargs for sqlalchemy connect * added note about kwarg * define new execution_engine passthru kwarg * add credentials_info to datasource and execution engine schemas * updated Simple SQL Datasource to configure bigquery creds * revert changes for v2 * added test for credentials_info kwarg * reformatted, changed assert * add valid but useless pkey * dummy creds conform to format, and added changelog * match input and output * omit conn string * update version (#4055) * [MAINTENANCE] Refactor RBP constructor to work with **kwargs instantiation pattern through config objects (#4043) * refactor: start refactor * refactor: continue modifying __init__ * refactor: clean up constructor * refactor: additional refactoring * docs passing sqlalchemy.create_engine kwargs through connection_string Added to docs in docs/guides/connecting_to_your_data/database/*.md; explains how to pass custom params via use of `connection_string` passing kwargs. * discuss and check with sql, stub submitted; passed reader_options and reader_method thus far * refactor: post-Alex comments * refactor: continue refactor * fix: ensure profiler store works with changes * docs: update docs under test * chore: restore misc docs changes * test: update docs under test * chore: add linted file * chore: remove DataContext import to remove cyclic issue * refactor: continue impl * docs: revert docs changes * feat: misc changes after convo with Alex & Will * test: fix all tests * docs: misc comments to clarify `pop()` * refactor: clean up init * chore: misc changes from Alex review * chore: type annotate batch requests * chore: remove accidental whitespace * chore: additional refactors per Alex comments * chore: add back str to batch request type annotation * refactor: ensure rename from parameter_name to name * fix: fix schema validation in RBP constructor Co-authored-by: h4nn3rs <spacehanners@gmail.com> Co-authored-by: Hannah (✿◕‿◕) <hannah@superconductive.com> * [MAINTENANCE] Remove unnecessary metric dependency evaluations and add common table column types metric. (#4063) * [BUGFIX] Checkpoint Configurator fix to allow notebook logging suppression (#4057) * swap print for str * Update configurator.py * [MAINTENANCE] Clean up new RBP types, method signatures, and method names for the long term. (#4064) * Created a page containing our glossary of terms and definitions. (#4056) * Created a page containing our glossary of terms and definitions. Added the glossary to the bottom of the Core Concepts section of the ToC * Update docs/glossary.md * Update docs/glossary.md * Update docs/glossary.md * [FEATURE] Get profilers from DataContext (#4033) * feat: start impl * chore: reference new store in init and anonymizer * chore: also reference class in test_yaml_config * feat: update class interface * chore: start marking parts of codebase that need ref to ProfilerStore * chore: fix defaults in schema * feat: implement serialization_self_check * test: start writing unit tests * test: continue test impl * test: continue test impl * feat: update backend defaults schemas to include profiler * test: fix all DataContext UI tests to work with profiler store * feat: implement properties * chore: saving work from Alex pairing * feat: continue refactored impl * feat: finish refactor impl * chore: remove print statement * chore: fix typo * test: fix tests * test: rename test * feat: add other schema vars * feat: update properties per Alex's review * refactor: move utility to test utils * chore: minor change to error message in properties * feat: ensure that filesystem and config is updated * chore: reuse profilers_path var * chore: misc cleanup * feat: add toolkit * feat: start impl * test: add final tests * feat: start impl * test: write test stubs * fix: correct cyclic import issue with type hints * test: continue test impl * chore: remove invalid path from Checkpoint toolkit * feat: continue impl * refactor: start refactor * refactor: continue modifying __init__ * refactor: clean up constructor * refactor: additional refactoring * docs passing sqlalchemy.create_engine kwargs through connection_string Added to docs in docs/guides/connecting_to_your_data/database/*.md; explains how to pass custom params via use of `connection_string` passing kwargs. * discuss and check with sql, stub submitted; passed reader_options and reader_method thus far * refactor: post-Alex comments * refactor: continue refactor * fix: ensure profiler store works with changes * docs: update docs under test * chore: restore misc docs changes * test: update docs under test * chore: add linted file * chore: remove DataContext import to remove cyclic issue * refactor: continue impl * docs: revert docs changes * feat: misc changes after convo with Alex & Will * test: fix all tests * docs: misc comments to clarify `pop()` * refactor: clean up init * chore: misc changes from Alex review * chore: type annotate batch requests * chore: remove accidental whitespace * chore: additional refactors per Alex comments * chore: add back str to batch request type annotation * refactor: ensure rename from parameter_name to name * fix: fix schema validation in RBP constructor * chore: misc updates * chore: another update * chore: revert checkpoint change * chore: misc changes * chore: continue debugging * fix: update post_dump to include unknown fields * docs: write docstring for post_dump * refactor: bump up fixtures to global conftest Co-authored-by: h4nn3rs <spacehanners@gmail.com> Co-authored-by: Hannah (✿◕‿◕) <hannah@superconductive.com> * fixed broken function call (#4068) * [FEATURE] Add RuleBasedProfiler to `test_yaml_config` utility (#4038) * feat: start impl _test_instantiation_of_profiler_from_yaml_config * feat: create anonymizer * feat: utilize anonymizer in helper method * refactor: remove DataContext imports in RBP due to cyclic issues * test: add test stub * chore: undo some `sd` usage * test: start test impl * chore: misc updates after develop merge * feat: complete initial impl * refactor: enable checkpoint-like pattern in test_yaml_config * feat: flesh out self_check * test: write tests * chore: add refactor helper method per Alex * test: add test to ensure `self_check` in test_yaml_config * chore: misc changes per review * release candidate for 0.14.3 (#4078) * [BUGFIX] Fix typing_extensions requirement to allow for proper build (#4083) * fix: update requirements * feat: add try/except * [MAINTENANCE] Use `logger.warning` instead of `logger.warn` due to deprecation warning (#4085) * data docs action rewrite (#4087) * metric store how to rewrite (#4086) * metric store how to rewrite * revision per review * release candidate for 0.14.4 (#4091) * [MAINTENANCE] fix typos in changelogs (#4093) * [MAINTENANCE] Migration of GCP tests to new project (#4072) * pushing current changes * Update test_script_runner.py * update the pipelines * pushing new pipeline * Update azure-pipelines-docs-integration-testing.yml * Update how_to_host_and_share_data_docs_on_gcs.py * get rid of this * changed gsutil script * Update azure-pipelines-docs-integration-testing.yml * Update azure-pipelines-docs-integration-testing.yml * migration to normal docs-integration script * bring test_script_runner back to speed * Update test_bigquery_benchmarks.py * Update azure-pipelines-cloud-db-integration.yml * dumb mistake * Update expect_column_values_to_be_unique.json * Update expect_column_values_to_be_unique.json * Update expect_column_values_to_be_unique.json * Update expect_column_values_to_be_unique.json * fixed column_values_to_be_unique * Update test_expectations.py * reversion * reversion of stale changes * fixed variable * [BUGFIX] Fix for `expect_column_values_to_be_unique` that was not compatible with BigQuery (#4073) * fix for test that is not compatible with bigquery * Update expect_column_values_to_be_unique.json * added actual condition * made clean * bugfix * clean up and additional error messages * push before cleaning up docs * push before final clean up * Delete yellow_tripdata_sample_2019-03.csv * clean up * more clean up * push with additional clean up * adding the additional information * more clean up * pushing additional docs changes * push * final clean up * after review * oops * Add default BIGQUERY_TYPES (#4096) * [MAINTENANCE] Refactor Validator methods (#4095) * [FEATURE] Delete profilers from DataContext (#4067) * feat: start impl * chore: reference new store in init and anonymizer * chore: also reference class in test_yaml_config * feat: update class interface * chore: start marking parts of codebase that need ref to ProfilerStore * chore: fix defaults in schema * feat: implement serialization_self_check * test: start writing unit tests * test: continue test impl * test: continue test impl * feat: update backend defaults schemas to include profiler * test: fix all DataContext UI tests to work with profiler store * feat: implement properties * chore: saving work from Alex pairing * feat: continue refactored impl * feat: finish refactor impl * chore: remove print statement * chore: fix typo * test: fix tests * test: rename test * feat: add other schema vars * feat: update properties per Alex's review * refactor: move utility to test utils * chore: minor change to error message in properties * feat: ensure that filesystem and config is updated * chore: reuse profilers_path var * chore: misc cleanup * feat: add toolkit * feat: start impl * test: add final tests * feat: start impl * test: write test stubs * fix: correct cyclic import issue with type hints * test: continue test impl * chore: remove invalid path from Checkpoint toolkit * feat: continue impl * refactor: start refactor * refactor: continue modifying __init__ * refactor: clean up constructor * refactor: additional refactoring * docs passing sqlalchemy.create_engine kwargs through connection_string Added to docs in docs/guides/connecting_to_your_data/database/*.md; explains how to pass custom params via use of `connection_string` passing kwargs. * discuss and check with sql, stub submitted; passed reader_options and reader_method thus far * refactor: post-Alex comments * refactor: continue refactor * fix: ensure profiler store works with changes * docs: update docs under test * chore: restore misc docs changes * test: update docs under test * chore: add linted file * chore: remove DataContext import to remove cyclic issue * refactor: continue impl * docs: revert docs changes * feat: misc changes after convo with Alex & Will * test: fix all tests * docs: misc comments to clarify `pop()` * refactor: clean up init * chore: misc changes from Alex review * chore: type annotate batch requests * chore: remove accidental whitespace * chore: additional refactors per Alex comments * chore: add back str to batch request type annotation * refactor: ensure rename from parameter_name to name * fix: fix schema validation in RBP constructor * chore: misc updates * chore: another update * chore: revert checkpoint change * chore: misc changes * chore: continue debugging * fix: update post_dump to include unknown fields * docs: write docstring for post_dump * refactor: bump up fixtures to global conftest * feat: implement method and tests * test: add additional assert to test Co-authored-by: h4nn3rs <spacehanners@gmail.com> Co-authored-by: Hannah (✿◕‿◕) <hannah@superconductive.com> * [BUGFIX] Pin `pip --upgrade` to a specific version to bypass bug in pip's underlying algorithm (#4100) * fix: pin pip upgrade * chore: test change to trigger pipeline * fix: also pin dependency graph pip upgrades * chore: revert changes * [BUGFIX] Use `pip==20.2.4` for usage statistics stage of CI/CD (#4102) * fix: pin pip upgrade * chore: test change to trigger pipeline * fix: also pin dependency graph pip upgrades * chore: revert changes * chore: revert usage stats pip versions * [MAINTENANCE] Fix Configuration Schema and Refactor Rule-Based Profiler; Initial Implementation of Reconciliation Logic Between Configuration and Runtime Arguments (#4088) * [BUGFIX] Fix shared state issue in renderer test (#4000) * chore: add assert to test problem * test: use abspath to find JSON files * test: add UNSUPPORTED_EXPECTATIONS * chore: add comment to explain test * refactor: remove defaultdict * chore: update comment * fix: bring back defaultdict * chore: fix typo * chore: update comments * fix: use context manager pattern for cd * chore: move certain asserts out of context manager scope * refactor: use context manager cd for tests * chore: remove unnecessary imports * test: fix test identation * refactor: split up os join usage * [BUGFIX] Missing docstrings on validator expect_ methods (#4062) (#4081) * Fix issue where docstrings were not added to expectations on Validator (#4062) * linting Co-authored-by: talagluck <tal@superconductive.com> * minor cleanup (#4110) * [FEATURE][BUGFIX] Support nullable int column types (#4044) * Bugfix for Pandas dtypes that need instantiation to work properly. Add Pandas nullable integer types to default int type list. * Applied formatting * Fixed tests for new int types * [WIP] Attempt introduce test skips for invalid versions of Pandas * [WIP] Test to try to fix bug with Pandas ~0.25 in expect_column_values_to_be_in_type_list * [WIP] Another attempt at a bugfix. The issue appears to be with Numpy <1.21 rather than with Pandas. * Since Numpy was the issue rather than Pandas 0.25, and that was dealt with in the type checking function, we can expand the nullable int test to include Pandas 0.25 Co-authored-by: David Maxson <david.maxson@rearc.io> * [FEATURE] Rule-Based Profiler Configuration and Runtime Arguments Reconciliation Logic (#4111) * [BUGFIX] Fix s3 path suffix bug on windows (#4042) * Bugfix for S3 path prefixes being assigned invalid suffix when executed on Windows host * Updated S3 sanitizer to behave more like previous sanitizer when a file or empty prefix is given * Applied black formatting * Added docstring * Renamed to avoid naming conflict. Added same functionality to ConfiguredAssetS3DataConnector. * Returned line endings to their proper values... * [WIP] Resolved circular dependency * Bugfix for empty prefixes * Add tests to ensure that S3 prefix sanitization behaves the same as the original sanitizer (besides OS-specific differences) * Expanded sanitizer test slightly Co-authored-by: David Maxson <david.maxson@rearc.io> * release candidate for 0.14.5 (#4121) * [FEATURE] Add scheduled runs for primary Azure pipeline * [BUGFIX] minor updates to test definition json files (#4123) * Rename the two JSON files that have '_test_set' at the end of filenames * Update 2 test_definition json files with proper values * put base.py from RBP into its own config directory -- new pattern of organization (#4125) * [MAINTENANCE] enable filter properties dict to handle both inclusion and exclusion lists (#4127) * [FEATURE] Promote dependency graph test strategy to production (#4124) * feat: add cron schedule for main pipeline * feat: write update YAML for pipelines * chore: test change * fix: add postgres to new stage * chore: remove new stage * chore: add comments to YAMLs * chore: revert test change * chore: test change * chore: update pip * chore: revert test changes * [MAINTENANCE] Remove unused Great Expectations imports (#4135) * chore: start cleanup * chore: continue impl * chore: complete cleanup * chore: revert renderer content block imports * [MAINTENANCE] Update trigger for scheduled Azure runs (#4134) * feat: add cron schedule for main pipeline * feat: write update YAML for pipelines * chore: test change * fix: add postgres to new stage * chore: remove new stage * chore: add comments to YAMLs * chore: revert test change * chore: test change * chore: update pip * chore: revert test changes * chore: make always true * Maintenance/upgrade black (#4136) Upgraded black; required minimal reformatting on some files * [FEATURE] Create profiler from DataContext (#4070) * docs passing sqlalchemy.create_engine kwargs through connection_string Added to docs in docs/guides/connecting_to_your_data/database/*.md; explains how to pass custom params via use of `connection_string` passing kwargs. * discuss and check with sql, stub submitted; passed reader_options and reader_method thus far * feat: start impl * feat: continue add_profiler method * feat: continue impl * feat: package config before sending to toolkit * docs: add comments to DataContext method * feat: use Schema.validate() * chore: revert docs files * feat: finish initial impl * refactor: make check a boolean function * chore: misc changes from reviews * chore: remove unused import * test: fix tests ref to fixtures * chore: misc changes from review * chore: misc changes made per Don's review * refactor: add profiler store to signature * test: write tests at DataContext level * chore: move comment to separate line * chore: update test error assert * chore: add comments around class_name refactoring * fix: ensure proper import for schema Co-authored-by: h4nn3rs <spacehanners@gmail.com> Co-authored-by: Hannah (✿◕‿◕) <hannah@superconductive.com> * [MAINTENANCE] Alter `great_expectations` pipeline trigger to be more consistent (#4138) * feat: add cron schedule for main pipeline * feat: write update YAML for pipelines * chore: test change * fix: add postgres to new stage * chore: remove new stage * chore: add comments to YAMLs * chore: revert test change * chore: test change * chore: update pip * chore: revert test changes * chore: make always true * refactor: use Azure checks instead of scope_check stage * [MAINTENANCE] Remove remaining unused imports (#4137) * chore: start cleanup * chore: continue impl * chore: complete cleanup * chore: revert renderer content block imports * feat: remove remaining unused imports * chore: remove additional import * Update great_expectations/render/renderer/content_block/profiling_column_properties_table_content_block.py * [MAINTENANCE] Remove `class_name` as mandatory field from `RuleBasedProfiler` (#4139) * refactor: make class_name an optional field * refactor: remove usages of class_name * fix: ensure tests fixtures remove class and module name * [MAINTENANCE] Ensure `AWSAthena` does not create temporary table as part of processing Batch by default, which is currently not supported (#4103) * push with fix and warning * push with warning and bugfix * After review * pushing initial test * Update azure-pipelines-cloud-db-integration.yml * Update test_script_runner.py * Update azure-pipelines-cloud-db-integration.yml * Update test_script_runner.py * ensure the integration tests work * moving tests so dependencies are clear * commit before final cleanup * clean up * clean up * pushing final changes * pushing final changes * pushing changes after review * feat: remove unused exceptions (#4143) * [MAINTENANCE] Standardize DictDot Method Behaviors Formally for Consistent Usage Patterns in Subclasses (#4131) * [MAINTENANCE] Remove unused f-strings (#4142) * [MAINTENANCE] Minor Validator code clean up -- for better code clarity (#4147) * [BUGFIX] Fix typo for metric name in expect_column_values_to_be_edtf_parseable (#4140) * Fix typo for metric name in expect_column_values_to_be_edtf_parseable and comment out _spark * Add a comment about why _spark method is commented out * [DOCS] Reenable docs-under-test for RuleBasedProfiler (#4149) * test: re-enable test * docs: update doc contents * docs: update warning * [FEATURE] Add read_sas function (#3972) * Add Pandas read_sas function * Add .sas7bdat test file to test_set * Add unit test for read_sas function * Fix linting * Modify import order * Fix test function * Add V3 API test * Return raised exception to original depth Co-authored-by: AFineDayFor <AFineDayFor@users.noreply.github.com> * [MAINTENANCE] Refactoring of `test_script_runner.py`. Integration and Docs tests (#4145) * [MAINTENANCE] Refactoring of `test_script_runner.py`. Integration and Docs tests (#4145) * [FEATURE] Run profiler from DataContext (#4141) * feat: init commit * feat: add basic signature and toolkit method * feat: add basic argument validation * chore: clean up exception message * test: write test stubs * feat: update to name and signature after discussion with Alex * test: write tests * [MAINTENANCE]: Remove `compatability` stage from dependency-graph (#4161) * [BUGFIX] Ensure that CheckpointResult object can be pickled (#4157) * Enable CheckpointResult objects to be pickled. * [BUGFIX] custom notebook templates (#2619) * bugfixes for custom notebook templates - fix custom_templates_module allowed to be None. This is needed for the docs example "Overwriting the class" to work - improved documentation noting that `notebook_assets` needs to be a python package - added tests for these cases * add line to changelog * [MAINTENANCE] CLOUD-618: GE Cloud "account" to "organization" rename (#4146) * Change 'account' to 'organization' * Update with organization_id, but allow backward compatibility with account_id * Add account_id property * Change 'accounts' to 'organizations' * Change account_id to organization_id but maintain backwards compatibility * Handle case where ge_cloud_credentials has account_id * Add todo * Lint * Lint * Lint * Lint * Revert "Lint" This reverts commit 914453bc5a57d891f5e9c04d88b29411c0f5b591. * Revert "Lint" This reverts commit a4f8b2b3ab5443696701aad2e40fcbf6265eff7c. * Revert "Lint" This reverts commit c8b5892f37c6cd6884af3b3d1d975ff7cb9c68a6. * Revert "Lint" This reverts commit 7be882398fd123f6d20bcbb1343b2d0a99feb6ab. * Lint * [FEATURE] Instantiate Rule-Based Profiler Using Typed Configuration Object (#4150) * [BUGFIX] Include public fields in property_names (#4159) * Include public fields in property_names * Use set union operator instead of .union method * [FEATURE] Provide ability to instantiate Checkpoint using CheckpointConfig typed object (#4166) * [FEATURE] Misc cleanup around CLI `suite` command and related utilities (#4158) * chore: type hints * chore: add notes * refactor: misc cleanup * chore: revert CLI signature * refactor: more cleanup * chore: revert change to test util * Provided details for using GE_HOME in commandline. (#4164) * release candidate for 0.14.6 (#4172) * [FEATURE] Allow Data Docs to be rendered in night mode (#4130) * add dark mode themeing to data docs pages * darkmode logo * remove extra css file and move logo to CDN Co-authored-by: talagluck <tal@superconductive.com> * [FEATURE] SimpleDateFormatParameterBuilder (#4156) * Implement Class SimpleDateFormatStringParameterBuilder * Update changelog.rst (#4163) Modify the 0.10.8 entry to reference the proper environment variable for custom jupyter command (GE_JUPYTER_CMD vs GE_JUPYTER_COMMAND) * [FEATURE] Introduce Top-Level Abstract "ConfigPeer" class for handling configuration output for Marshmallo Schema validated Config objects (#4183) * [MAINTENANCE] Make `cli_integration` stage in primary `great_expectations` pipeline run in parallel (#4187) * feat: add cron schedule for main pipeline * feat: write update YAML for pipelines * chore: test change * fix: add postgres to new stage * chore: remove new stage * chore: add comments to YAMLs * chore: revert test change * chore: test change * chore: update pip * chore: revert test changes * chore: make always true * refactor: use Azure checks instead of scope_check stage * feat: remove test_cli prereqs * chore: revert test changes * chore: test change * chore: try making usage_stats last * fix: remove cycle from dependsOn * refactor: move usage_stats to end of YAML * fix: update cli stage ref * feat: parallelize cli_integration * [FEATURE] Update `suite new --profile` to work with Rule Based Profiler (#4171) * chore: type hints * chore: add notes * refactor: misc cleanup * chore: revert CLI signature * refactor: more cleanup * chore: revert change to test util * feat: add profiler_name as valid arg to --profile * chore: bump requirements.txt * feat: configure optional option arg * feat: start modifying notebooks * test: fix broken tests * chore: revert requirements * chore: type hint and add comment * feat: update renderer logic * test: add CLI tests * test: write tests for renderer * feat: add CLI messaging about profiler type * [MAINTENANCE] Refactor RuleBasedProfiler toolkit pattern (#4191) * docs passing sqlalchemy.create_engine kwargs through connection_string Added to docs in docs/guides/connecting_to_your_data/database/*.md; explains how to pass custom params via use of `connection_string` passing kwargs. * refactor: remove toolkit in place of static methods * chore: make context helper method private * chore: revert docs * fixes two references to the Getting Started tutorial (#4189) * [MAINTENANCE] Revert changes to `dependency_graph` pipeline * [MAINTENANCE] Refactor relative imports (#4195) * refactor: start cleanup * chore: revert to dev * refactor: fix all relative imports * [MAINTENANCE]: Remove temp file accidentally committed (#4201) * [FEATURE] Support Multi-Dimensional Metric Computations Generically for Multi-Batch Parameter Builders (#4206) * [MAINTENANCE] Update default candidate strings SimpleDateFormatString parameter builder (#4193) * [MAINTENANCE] minor type hints clean up (#4214) * [MAINTENANCE] RBP testing framework changes (#4184) * push of proposed refactor * Merge branch 'develop' into feature/GREAT-494/refactor-rbp-tests * develop: [FEATURE] SimpleDateFormatParameterBuilder (#4156) * tests pass * Update test_user_workflow_fixtures.py * Update conftest.py * bugfix * pushing refactor after review * final update * pushing recent changes * [BUGFIX] Update validate_configuration for core Expectations that don't return… (#4216) * Update validate_configuration for core Expectations that don't return a value * Annotate return type of validate_configuration method on Expectations that have it * add conditional check for 'expect_column_values_to_be_in_type_list' (#4200) * add conditional check for column * fix parallel bug * lint * [BUGFIX] Datadocs filter on date is broken (#4217) * update date filter function * update lib version and fix formatter * update linting and fix pull request template to use new docs * update the changelog * Bugfix/devrel 189/datadocs filter on date is broken (#4218) * update date filter function * update lib version and fix formatter * update linting and fix pull request template to use new docs * update the changelog * update changelog in the right place * [ENHANCEMENT] Allow users to pass in any set of polygon points in expectation for point to be within region (#2520) * Create expectation file expect record point to be within geo region * Initial expectation setup * Implement pandas expectation * Add in negative test and read in data as class variable, so doesn't load on every expectation invocation * Rename package to be consistent with Expectation class * black formatter * isort * Add parameter polygon_points in _pandas function * Update input variable definitions to expect polygon_points, and update tests for passing in input latitude-longitude points to expectation * Update documentation given addition of support for passing in polygon points * Add new test * New polygon points for the USA test * linting Co-authored-by: DXcarlos <clopezh@uni.pe> Co-authored-by: anthonyburdi <anthony@superconductive.com> Co-authored-by: Austin Robinson <austin@superconductive.com> * [ENHANCEMENT] Better support Hive, better support BigQuery. (#2624) * Add hive to the dataset model * Add support for project id in bigquery datasources If the credentials do not have the same project as the table to be queried, the project id needs to be specified. * linting Co-authored-by: Austin Robinson <austin@superconductive.com> * [DOCS] Deepnote Deployment Pattern Guide (#4169) * first half of deepnote deployment guide * first half of deployment guide * first pass at DN deployment pattern * update sidebar.js * fixed img refs * typo fixes & codeblock linting * codeblock fix * replaced images * move process_evaluation_parameters into conditional (#4109) * [MAINTENANCE] Type hint usage statistics (#4226) * [FEATURE] Add support for sqlalchemy-bigquery while falling back on pybigquery (#4182) * Add support for sqlalchemy-bigquery while falling back on pybigquery * Black reformatting * Update package to pip install in docs * Update doc url in credentials_snippet * Feature/add support for sqlalchemy bigquery + geo columns (#4222) * updated bigquery import path * add warning to expectation if geometry package isnt installed * updated dev reqs * lint * update reqs-dev * update bigquery import path * updated import paths * lint * Use warnings.warn for DeprecationWarning about using pybigquery Co-authored-by: Ken Wade <ken@superconductive.com> * Add comment in requirements-dev-sqlalchemy.txt for sqlalchemy-bigquery Co-authored-by: Joshua Stauffer <66793731+joshua-stauffer@users.noreply.github.com> * [DOCS] Deepnote Deployment Pattern Image Fixes (#4229) * first half of deepnote deployment guide * first half of deployment guide * first pass at DN deployment pattern * update sidebar.js * fixed img refs * typo fixes & codeblock linting * codeblock fix * replaced images * image linking * linking fix * release candidate for 0.14.7 (#4230) * [MAINTENANCE] Rule-Based Profiler: Fix Circular Imports; Configuration Schema Fixes; Enhanced Unit Tests; Pre-Requisites/Refactoring for Self-Estimating Expectations (#4234) * [FEATURE] Add `run_profiler_on_data` method to DataContext (#4190) * feat: add method stub * feat: start impl * chore: write docstr * test: write test stubs * feat: copy batch requests each time * refactor: make reconciliation method private * feat: finish initial version * refactor: convert batch request to dict to ensure proper serializability * feat: lay out plan after talking with Alex * feat: convert batch request to dict * feat: finish impl * feat: leverage enum check instead of isinstance * feat: add docstring and use safe_deep_copy * chore: use deepcopy instead of safe_deecopy * feat: remove copy * [FEATURE] `RegexPatternStringParameterBuilder` for `RuleBasedProfiler` (#4167) * [FEATURE] `RegexPatternStringParameterBuilder` for `RuleBasedProfiler` (#4167) Co-authored-by: Chetan Kini <chetan@superconductive.com> * fix: fix issue (#4240) * remove duplicate test (#4241) * experimental column map expectation checking for vectors (#3102) * Reformat contrib expectation with black (#4244) * Improve pandas version checking in test_expectations[_cfe].py files (#4248) * [MAINTENANCE] Resolve cyclic import issue with usage stats (#4251) * chore: move imports to be in-line * chore: add comment * fix: fix cfg setting * [FEATURE] Pre-requisites in Rule-Based Profiler component relevant for Self-Estimating Expectations: customizable/flexible configuration reconciliation at Variables, DomainBuilder, ParameterBuilder, and ExpectationConfigurationBuilder levels; additional method accessors. (#4242) * [BUGFIX] Ensure `test_script_runner.py` actually raises AssertionErrors correctly (#4239) * Update test_script_runner.py * Update gcp_deployment_patterns_file_gcs_yaml_configs.py * bugfix * update this with actual values * Update multi_batch_rule_based_profiler_example.py * Update multi_batch_rule_based_profiler_example.py * clean up * [MAINTENANCE] Additional refactor to clean up cyclic imports in usage stats (#4256) * chore: move imports to be in-line * chore: add comment * fix: fix cfg setting * refactor: move away from init and use full paths * refactor: remove imports in init * [MAINTENANCE] Rule-Based Profiler prerequisite: fix quantiles profiler configuration and add comments (#4255) * [FEATURE] Add optional parameter `condition` to DefaultExpectationConfigurationBuilder (#4246) - Conditional logic parser using pyparsing added to DefaultExpectationConfigurationBuilder - Testing for DefaultExpectationConfigurationBuilder * [MAINTENANCE] Introspect Batch Request Dictionary for its kind and instantiate accordingly (#4259) * [MAINTENANCE] Minor clean up in style of an RBP test fixture; making variables access more robust (#4261) * Check for pandas>=024 not pandas>=24 (#4263) * [BUGFIX] Add support for SqlAlchemyQueryStore connection_string credentials (#4224) * Add support for SqlAlchemyQueryStore connection_string credentials * Add test for SqlAlchemyQueryStore connection_string credentials * Fix formatting * Fix formatting * [DOCS/FEATURE] Hackathon Contribution Docs (#3897) * Create how_to_enable_additional_execution_engines_for_custom_expectations.md add documentation for implementing metrics in SQLAlchemy and Spark * initial commit of custom expectations docs restructure * clean up structure and titles * put creating expectation doc scripts under test * - Reworks custom expectation doc into a more explicit guide for custom column aggregate expectations - Fixes pointers in other docs to this doc - Brings supporting example into great_expectations from superconductive/ge_tutorials * cleanup on column_aggregate docs | skeleton for column_map doc * fixed references to map_metric * moving reference file under tests, fixing references * Rename custom_column_max_example.py * Restore colors to `contributing maturity.md` (#3910) * Draft overview for custom_expectations (#3911) * Draft Overview for custom_expectations * Add creating_custom_expectations/components/prerequisites.jsx * Feature/expectation completeness checklist (#3906) * Implement the first several steps of generate_diagnostic_checklist, with tests and unit-testable supporting methods. * Refactor tests so that test_run_diagnostics uses the same test cases as test_generate_diagnostic_checklist * Move expect_column_values_to_equal_three.py to a fixtures directory * expansion of contributing_maturity with specifics for expectations * Updates and edits to custom expectations overview and contributing maturity docs * progress on adapting column agg doc to fit pattern from column map doc | edits to template and example to support | linting * rework of agg doc re: map doc | updates to supporting code | linting and typo fixes * updates to agg doc | small edits to metrics & prereqs * Feature/refactor and type expectation diagnostic report (#3929) * Initial scaffolding. One type and one test * Scaffold most of the rest of the objects we'll need, with some tests. * Add tests to unpack the behavior of @dataclass combined with SerializableDotDict * More WIP, especially tests * A spate of renaming and tightening * Add SerializableDotDict.keys * Update expectation_test_data_cases to use SerializableDictDot * Fix gtypos * Update SerializableDictDot, with tests * The meat of the refactor, mostly done with some WIP * Fixed test__count_positive_and_negative_example_cases * Decruft * Fix test_include_in_gallery_flag * Make .to_dict recursive * Linting * Remove pydantic; Remove submessages in execution_engines; test_generate_diagnostic_checklist__first_iteration now passes * Refactor _count_unexpected_cases_and_get_sub_messages into _count_unexpected_cases * Fix docstring * Clean up types and methods around LibraryMetadata * Add support for enum serialization * Implement LegacyAugmentedLibraryMetadataAdapter; Tidy up a little * Minor bugfixes. Many docstrings * Add docstrings for test_data_cases * Improve _get_metric_diagnostics_list * Clean up _get_test_results * Work on docstrings and more bug fixes. Re-up pydantic. Need to get out of the weeds soon. * Deprecate LegacyAugmentedLibraryMetadataAdapter * Tidy up rendering methods * Add RendererTestDiagnostics object * Add unit tests for _get_augmented_library_metadata and _get_examples * Unit test for _get_description_diagnostics * Implement OtherExpectationTestDiagnostics * Flow typed ExecutedExpectationTestCases through * Remove dependence on global _registered_renderers. Get another test to pass. * Clean up tests and get business logic for Has at least one positive and negative example case, and all test cases pass working * Decruft. Skip one failing test * Feature/changes from novuum jan3 review (#3941) * Add ExpectationDiagnosticMaturityMessages class and refactor around it * Add optional AugmentedLibraryMetadata.package and ExpectationDiagnosticCheckMessage.doc_url * Decruft * Fix mismatched string in tests * Delete test_expectation__get_execution_engine_dict _get_execution_engine_dict was replaced by _get_execution_engine_diagnostics * Update tests/expectations/test_expectation_diagnostics.py Co-authored-by: Don Heppner <donald.heppner@gmail.com> * Implement ExpectationDiagnostics.to_json_dict method and use in build_gallery.py script * Update build_gallery.py script to collect expectation tracebacks and display before writing gallery json file * Allow ExpectationTestDataCases to receive 'schemas' * Add commented-out 'test_backends' attribute to ExpectationTestDataCases class * Add a mysql docker image that works with M1 Macs % docker-compose up Pulling mysql_db (mysql:8.0.20)... 8.0.20: Pulling from library/mysql ERROR: no matching manifest for linux/arm64/v8 in the manifest list entries * Ensure that the 'configuration' object passed to renderers are typed objects * Update build_gallery.py script to not die if there are expectation tracebacks * Use dot notation for args passed to ExpectationDiagnosticCheckMessage Co-authored-by: Ken Wade <ken@superconductive.com> Co-authored-by: Don Heppner <donald.heppner@gmail.com> * [FEATURE] Add maturity_checklist to ExpectationDiagnostics (#4061) * Add maturity_checklist to ExpectationDiagnostics and use in Expectation.run_diagnostics * Remove concept_only from ExpectationDiagnosticMaturityMessages * Re-format files with black * Update the 'broken links' identified by docs_link_checker.py script * [DOCS] WIP Docs/custom expectation test cases (#3969) * updated opening paragraph * further edits * completed ex_cases doc | re-orged | edited sidebar & example file to support * fixed codeblock line refs * Update docs/guides/expectations/features_custom_expectations/how_to_add_example_cases_for_an_expectation.md Co-authored-by: Rachel-Reverie <94694058+Rachel-Reverie@users.noreply.github.com> * Update docs/guides/expectations/features_custom_expectations/how_to_add_example_cases_for_an_expectation.md Co-authored-by: Rachel-Reverie <94694058+Rachel-Reverie@users.noreply.github.com> * Update docs/guides/expectations/features_custom_expectations/how_to_add_example_cases_for_an_expectation.md Co-authored-by: Rachel-Reverie <94694058+Rachel-Reverie@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Rachel-Reverie <94694058+Rachel-Reverie@users.noreply.github.com> Co-authored-by: Rachel-Reverie <94694058+Rachel-Reverie@users.noreply.github.c…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Scope
Please annotate your PR title to describe what the PR does, then give a brief bulleted description of your PR below. PR titles should begin with [BUGFIX], [FEATURE], [DOCS], or [MAINTENANCE]. If a new feature introduces breaking changes for the Great Expectations API or configuration files, please also add [BREAKING]. You can read about the tags in our contributor checklist.
Changes proposed in this pull request:
After submitting your PR, CI checks will run and @ge-cla-bot will check for your CLA signature.
For a PR with nontrivial changes, we review with both design-centric and code-centric lenses.
In a design review, we aim to ensure that the PR is consistent with our relationship to the open source community, with our software architecture and abstractions, and with our users' needs and expectations. That review often starts well before a PR, for example in github issues or slack, so please link to relevant conversations in notes below to help reviewers understand and approve your PR more quickly (e.g.
closes #123).Previous Design Review notes:
Definition of Done
Please delete options that are not relevant.
Thank you for submitting!