Merge branch 'm/_/credentials-doc-update' of https://github.com/great…

…-expectations/great_expectations into m/_/credentials-doc-update * 'm/_/credentials-doc-update' of https://github.com/great-expectations/great_expectations: [BUGFIX] Setting maximum on `typing-extension` package version (#7970) [BUGFIX] Change GXSqlDialect.AWSATHENA to awsathena (#7950) [FEATURE] Implementing Python code snippets under test for "https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/fluent/cloud/how_to_connect_to_data_on_gcs_using_spark" (#7964) [FEATURE] Implementing Python code snippets under test for "https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/fluent/cloud/how_to_connect_to_data_on_gcs_using_pandas" (#7959) [DOCS] Add small doc change (#7957) [MAINTENANCE] Fix docs deploy (#7958) [MAINTENANCE] ignore black formatting and ruff auto-fix revisions (#7953) [FEATURE] Implementing Python code snippets under test for "https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/fluent/database/how_to_connect_to_sqlite_data" (#7947) [BUGFIX] Cloud - fix `add_or_update_*()` when using `name` as keyword arg (#7952) [MAINTENANCE] black `23.3` (#7946) [BUGFIX] fix `add_or_update_expectation_suite` update path (#7911) [BUGFIX] Return qualified name when calling `TableAsset.as_selectable()` (#7942) [FEATURE] Implementing Python code snippets under test for "https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/fluent/cloud/how_to_connect_to_data_on_s3_using_spark" (#7943) [MAINTENANCE] Update ruff to `0.0.269` and target `python 3.8` (#7945) [MAINTENANCE] bump python minimum version to 3.8 (#7916)
great-expectations · May 23, 2023 · 4691abf · 4691abf
2 parents 2338c02 + e60b6d7
commit 4691abf
Show file tree

Hide file tree

Showing 477 changed files with 992 additions and 1,126 deletions.
diff --git a/.git-blame-ignore-revs b/.git-blame-ignore-revs
@@ -3,6 +3,7 @@
 # https://git-scm.com/docs/git-blame#Documentation/git-blame.txt---ignore-revltrevgt
 # A typical use of these would be if a large formatting change was applied across the whole codebase but otherwise
 # no runtime changes were made. Each entry should be preceded with a comment explaining the change.
+#
 # https://black.readthedocs.io/en/stable/guides/introducing_black_to_your_project.html#avoiding-ruining-git-blame
 # Apply noqa markers for all TCH001 violations
 f5e7df1846102d9a62cc9b9110387925ffae60cc
@@ -12,3 +13,9 @@ f5e7df1846102d9a62cc9b9110387925ffae60cc
 # Apply noqa markers for TID251 (sqlalchemy) violations
 # https://github.com/great-expectations/great_expectations/pull/7564
 e55b3484a86f654e8b819041dd6cc73730e01a8f
+# python 3.7 -> 3.8 Ruff auto-fixes
+# https://github.com/great-expectations/great_expectations/pull/7945
+85b81029c759dc2b6a8c8956a5babb19e3280494
+# black 2.2 -> 2.3 formatting changes
+# https://github.com/great-expectations/great_expectations/pull/7946
+582fe60179b9b8bf31ba0a394bfa669f495ce8c6
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -13,11 +13,11 @@ repos:
       - id: requirements-txt-fixer
         exclude: docs/.*
   - repo: https://github.com/psf/black
-    rev: 22.3.0
+    rev: 23.3.0
     hooks:
       - id: black-jupyter
   - repo: https://github.com/charliermarsh/ruff-pre-commit
-    rev: 'v0.0.265'
+    rev: 'v0.0.269'
     hooks:
       - id: ruff
         files: ^(great_expectations|contrib|scripts|tasks\.py) # TODO: add tests/  docs/ etc.

diff --git a/CONTRIBUTING_CODE.md b/CONTRIBUTING_CODE.md
@@ -20,7 +20,7 @@ To modify existing Great Expectations code, you complete the following tasks:
 
 - [Submit a pull request](#submit-a-pull-request)
 
-To discuss your code change before you implement it, join the [Great Expectations Slack community](https://greatexpectations.io/slack) and make your suggestion in the [#contributing](https://greatexpectationstalk.slack.com/archives/CV828B2UX) channel. 
+To discuss your code change before you implement it, join the [Great Expectations Slack community](https://greatexpectations.io/slack) and make your suggestion in the [#contributing](https://greatexpectationstalk.slack.com/archives/CV828B2UX) channel.
 
 To request a documentation change, or a change that doesn't require local testing, see the [README](https://github.com/great-expectations/great_expectations/tree/develop/docs) in the `docs` repository.
 
@@ -63,9 +63,9 @@ To submit a custom package to Great Expectations for consideration, see [CONTRIB
     ```
 ## Install Python dependencies
 
-Python dependencies are required to modify Great Expectations code, submit a new feature, or submit a Custom Expectation. 
+Python dependencies are required to modify Great Expectations code, submit a new feature, or submit a Custom Expectation.
 
-1. Run the following command to create a virtual environment in your local repository using Python versions 3.7 to 3.11, activate the environment, and then install the necessary dependencies:
+1. Run the following command to create a virtual environment in your local repository using Python versions 3.8 to 3.11, activate the environment, and then install the necessary dependencies:
 
     ```python
     python3 -m venv gx_dev
@@ -102,21 +102,21 @@ Python dependencies are required to modify Great Expectations code, submit a new
     If your Mac computer has an Apple M1 chip, you might need to specify additional compiler or linker options. For example:
 
     `export LDFLAGS="-L/opt/homebrew/Cellar/unixodbc/[your version]/lib"`
-    
+
     `export CPPFLAGS="-I/opt/homebrew/Cellar/unixodbc/[your version]/include"`
 
-5. Add `ulimit -n 4096` to the `~/.zshrc` or `~/.bashrc` files to prevent `OSError: [Errno 24] Too many open files` errors. 
+5. Add `ulimit -n 4096` to the `~/.zshrc` or `~/.bashrc` files to prevent `OSError: [Errno 24] Too many open files` errors.
 
 6. Run the following command to confirm pandas and SQLAlchemy with SQLite tests are passing:
 
    ```sh
     ulimit -n 4096
-    
+
     pytest -v
    ```
 ## Create a virtual environment (optional)
 
-A virtual environment allows you to create and test code without affecting the primary Great Expectations codebase. 
+A virtual environment allows you to create and test code without affecting the primary Great Expectations codebase.
 
 ### Python
 
@@ -125,7 +125,7 @@ A virtual environment allows you to create and test code without affecting the p
    ```python
     python3 -m venv <path_to_environments_folder\>/great_expectations_dev
    ```
-2. Run the following command to activate the virtual environment: 
+2. Run the following command to activate the virtual environment:
 
    ```python
     <source path_to_environments_folder\>/great_expectations_dev/bin/activate
@@ -137,7 +137,7 @@ A virtual environment allows you to create and test code without affecting the p
    ```python
     conda create --name great_expectations_dev
    ```
-2. Run the following command to activate the virtual environment: 
+2. Run the following command to activate the virtual environment:
 
    ```python
     conda activate great_expectations_dev
@@ -270,12 +270,12 @@ Use the following information to use Spark for code testing.
 
 - Java. See [Java downloads](https://www.java.com/en/download/).
 
-- The PATH or JAVA_HOME environment variables set correctly. See [Setting Java variables in Windows](https://www.ibm.com/docs/en/b2b-integrator/5.2?topic=installation-setting-java-variables-in-windows) or [Setting Java variables in Linux](https://www.ibm.com/docs/en/b2b-integrator/5.2?topic=installation-setting-java-variables-in-linux). 
+- The PATH or JAVA_HOME environment variables set correctly. See [Setting Java variables in Windows](https://www.ibm.com/docs/en/b2b-integrator/5.2?topic=installation-setting-java-variables-in-windows) or [Setting Java variables in Linux](https://www.ibm.com/docs/en/b2b-integrator/5.2?topic=installation-setting-java-variables-in-linux).
 
 On Mac, run the following commands to set the PATH and JAVA_HOME environment variables:
 ```
 export JAVA_HOME=`/usr/libexec/java_home`
-export PATH=$PATH:$JAVA_HOME/bin 
+export PATH=$PATH:$JAVA_HOME/bin
 ```
 
 
@@ -301,7 +301,7 @@ Great Expectations code is not tested against all SQL database types. Continuous
 
 To perform unit testing, run `pytest` in the `great_expectations` directory root. By default, tests are run against `pandas` and `sqlite`. You can use `pytest` flags to test additional backends like `postgresql`, `spark`, and `mssql`. For example, to run a test against PostgreSQL backend, you run `pytest --postgresql`.
 
-The following are the supported `pytest` flags for general testing:   
+The following are the supported `pytest` flags for general testing:
 
 - `--spark`: Execute tests against Spark backend.
 - `--postgresql`: Execute tests against PostgreSQL.
@@ -310,7 +310,7 @@ The following are the supported `pytest` flags for general testing:
 - `--bigquery`: Execute tests against Google BigQuery (requires additional set up).
 - `--aws`: Execute tests against AWS resources such as Amazon S3, Amazon Redshift, and Athena (requires additional setup).
 
-To skip all local backend tests (except pandas), run `pytest --no-sqlalchemy`. 
+To skip all local backend tests (except pandas), run `pytest --no-sqlalchemy`.
 
 Testing can generate warning messages. These warnings are often caused by dependencies such as pandas or SQLAlchemy. Run `pytest --no-sqlalchemy --disable-pytest-warnings` to suppress these warnings.
 
@@ -325,7 +325,7 @@ Testing can generate warning messages. These warnings are often caused by depend
 4. Run the following command to test your project with the `GE_TEST_BIGQUERY_PROJECT` and `GE_TEST_BIGQUERY_DATASET` environment variables:
 
     ```bash
-    GE_TEST_BIGQUERY_PROJECT=<YOUR_GOOGLE_CLOUD_PROJECT> 
+    GE_TEST_BIGQUERY_PROJECT=<YOUR_GOOGLE_CLOUD_PROJECT>
     GE_TEST_BIGQUERY_DATASET=test_ci
     pytest tests/test_definitions/test_expectations_cfe.py --bigquery
     ```
@@ -496,9 +496,9 @@ Test the performance of code changes to determine they perform as expected. BigQ
     - [FEATURE] for significant PRs that add a new feature likely to require being added to our feature maturity matrix.
 
     - [MAINTENANCE] for PRs that focus on updating repository settings or related changes.
-		
+
     - [CONTRIB] for the contribution of Custom Expectations and supporting work into the `contrib/` directory.
-      
+
     - [HACKATHON] for submissions to an active Great Expectations Hackathon.
 
     In the section for design review, include a description of any prior discussion or coordination on the features in the PR, such as mentioning the number of the issue where discussion has taken place. For example: Closes #123”, linking to a relevant discuss or slack article, citing a team meeting, or even noting that no discussion is relevant because the issue is small.
@@ -517,7 +517,6 @@ Additionally, Great Expectations adds the following tags to indicate issue statu
 
 - The`help wanted` tag identifies useful issues that require help from community contributors to accelerate development.
 
-- The `enhacement` and `expectation-request` tags identify new Great Expectations features that require additional investigation and discussion. 
+- The `enhacement` and `expectation-request` tags identify new Great Expectations features that require additional investigation and discussion.
 
 - The `good first issue` tag identifies issues that provide an introduction to the Great Expectations contribution process.
-
diff --git a/assets/partners/anthonydb/just_connect.py b/assets/partners/anthonydb/just_connect.py
@@ -4,7 +4,6 @@
 e = sa.create_engine(connection)
 results = e.execute(sa.text("SELECT TOP 10 * from dbo.taxi_data")).fetchall()
 for r in results:
-
     print(r)
 
 print("finish")
diff --git a/assets/partners/anthonydb/run.py b/assets/partners/anthonydb/run.py
@@ -7,7 +7,6 @@
 
 
 def test_ge():
-
     CONNECTION_STRING = os.environ.get("DB_URL")
 
     context = gx.get_context()

diff --git a/assets/partners/rubenshift/run.py b/assets/partners/rubenshift/run.py
@@ -7,7 +7,6 @@
 
 
 def test_ge():
-
     CONNECTION_STRING = os.environ.get("DB_URL")
 
     context = gx.get_context()

diff --git a/ci/azure-pipelines-dev.yml b/ci/azure-pipelines-dev.yml
@@ -100,8 +100,8 @@ stages:
         steps:
           - task: UsePythonVersion@0
             inputs:
-              versionSpec: 3.7
-            displayName: 'Use Python 3.7'
+              versionSpec: 3.8
+            displayName: 'Use Python 3.8'
 
           - script: |
               pip install $(grep -E '^(black|invoke|ruff)' reqs/requirements-dev-contrib.txt)
@@ -121,8 +121,8 @@ stages:
         steps:
           - task: UsePythonVersion@0
             inputs:
-              versionSpec: '3.7'
-            displayName: 'Use Python 3.7'
+              versionSpec: '3.8'
+            displayName: 'Use Python 3.8'
 
           - bash: python -m pip install --upgrade pip
             displayName: 'Update pip'
@@ -163,7 +163,6 @@ stages:
           # initial run doesn't check `.pyi` source files
           invoke type-check --ci --pretty --check-stub-sources --python-version $(python.version)
         name: TypeCheckStubSourceFiles
-
     - job: docstring_linter
       steps:
       - script: |
@@ -217,8 +216,8 @@ stages:
         steps:
          - task: UsePythonVersion@0
            inputs:
-             versionSpec: '3.7'
-           displayName: 'Use Python 3.7'
+             versionSpec: '3.8'
+           displayName: 'Use Python 3.8'
 
          - bash: python -m pip install --upgrade pip
            displayName: 'Update pip'
@@ -230,7 +229,6 @@ stages:
          - script: |
              python -c "import great_expectations as gx; print('Successfully imported GX Version:', gx.__version__)"
            displayName: 'Import Great Expectations'
-
   - stage: required
     dependsOn: [scope_check, lint, import_ge, custom_checks, unit_tests]
     pool:
@@ -255,8 +253,8 @@ stages:
         steps:
           - task: UsePythonVersion@0
             inputs:
-              versionSpec: '3.7'
-            displayName: 'Use Python 3.7'
+              versionSpec: '3.8'
+            displayName: 'Use Python 3.8'
 
           - bash: python -m pip install --upgrade pip
             displayName: 'Update pip'
@@ -318,8 +316,8 @@ stages:
         steps:
           - task: UsePythonVersion@0
             inputs:
-              versionSpec: '3.7'
-            displayName: 'Use Python 3.7'
+              versionSpec: '3.8'
+            displayName: 'Use Python 3.8'
 
           - bash: python -m pip install --upgrade pip
             displayName: 'Update pip'
@@ -581,8 +579,8 @@ stages:
         steps:
           - task: UsePythonVersion@0
             inputs:
-              versionSpec: '3.7'
-            displayName: 'Use Python 3.7'
+              versionSpec: '3.8'
+            displayName: 'Use Python 3.8'
 
           - script: ./ci/checks/check_min_airflow_dependency_compatibility.sh
             name: CheckMinAirflowDependencyCompatibility

diff --git a/ci/azure-pipelines-sqlalchemy-compatibility.yml b/ci/azure-pipelines-sqlalchemy-compatibility.yml
@@ -66,8 +66,8 @@ stages:
         steps:
           - task: UsePythonVersion@0
             inputs:
-              versionSpec: 3.7
-            displayName: 'Use Python 3.7'
+              versionSpec: 3.8
+            displayName: 'Use Python 3.8'
 
           - script: |
               pip install $(grep -E '^(black|invoke|ruff)' reqs/requirements-dev-contrib.txt)
@@ -87,8 +87,8 @@ stages:
         steps:
          - task: UsePythonVersion@0
            inputs:
-             versionSpec: '3.7'
-           displayName: 'Use Python 3.7'
+             versionSpec: '3.8'
+           displayName: 'Use Python 3.8'
 
          - script: |
              pip install .
@@ -124,8 +124,8 @@ stages:
         steps:
           - task: UsePythonVersion@0
             inputs:
-              versionSpec: '3.7'
-            displayName: 'Use Python 3.7'
+              versionSpec: '3.8'
+            displayName: 'Use Python 3.8'
 
           - script: |
               cp constraints-dev.txt constraints-dev-temp.txt