(1/x) Create pyproject.toml and move package name from setup.py

Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (2/x) Move package description from setup.py Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (3/x) Move package author from setup.py to pyproject.toml Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (4/x) Move classifiers from setup.py to pyproject.toml Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> squash with 56005dc Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (5/x) Move long description from setup.py to pyproject.toml Poetry will take the configured readme and set the long_description to its contents and the long_description_content_type to "text/markdown" automatically. Ref: python-poetry/poetry#1979 Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (6/x) Move python dependency spec from setup.py to pyproject.toml Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (7/x) Move package version from setup.py to pyproject.toml We now specify the package version in a single place: pyproject.toml so it's not necessary to dynamically figure it out in setup.py. Copped the new contents of __version__.py from the Poetry github issue tracker here: python-poetry/poetry#144 (comment) Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (8/x) Move namespace package inclusion from setup.py to pyproject.toml Note that per poetry docs [^1] > Poetry is clever enough to detect Python subpackages. > > Thus, you only have to specify the directory where your root package resides. [1]: https://python-poetry.org/docs/pyproject/ Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (9/x) Move homepage from setup.py to pyproject.toml Also add the repository metadata which appears in the Pypi sidebar Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (10/x) Entirely remove zip_safe config It's obsolete: https://setuptools.pypa.io/en/latest/deprecated/zip_safe.html and not required by poetry python-poetry/poetry#928 (comment) Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (11/x) Move project dependencies from setup.py and requirements.txt (where they were duplicated) into pyproject.toml This adds the locked dependencies in poetry.lock Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (12/x) Entirely remove setup.py Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (13/x) Move dev dependencies from dev-requirements.txt Re-lock the dependencies Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (14/x) Completely remove MANIFEST.in It's not needed by poetry Ref: https: //stackoverflow.com/questions/64654860/replacing-manifest-in-with-pyproject-toml Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> (15/x) Entirely remove /scripts This script is to be replaced with `poetry build` Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> I compared a build from `poetry build` to the output from `python setup.py sdist bdist_wheel` and then ran a diff against the generated files. Here's what I found. I added line numbers that I can reference below. ``` 1 Only in dbt-databricks-1.6.1-pypi: MANIFEST.in 2 Files dbt-databricks-1.6.1-pypi/PKG-INFO and dbt_databricks-1.6.1-poetry/PKG-INFO differ 3 Files dbt-databricks-1.6.1-pypi/dbt/adapters/databricks/__version__.py and dbt_databricks-1.6.1-poetry/dbt/adapters/databricks/__version__.py differ 4 Only in dbt-databricks-1.6.1-pypi: dbt_databricks.egg-info 5 Only in dbt_databricks-1.6.1-poetry: pyproject.toml 6 Only in dbt-databricks-1.6.1-pypi: setup.cfg 7 Only in dbt-databricks-1.6.1-pypi: setup.py ``` 1: OK see e55e261 2: OK differences are because poetry writes an extra classifier and includes the new homepage 3: OK difference is because we now automatically fetch the package version set by pyproject.toml 4: OK we're not uploading eggs anymore. They're about to be deprecated by pypi anyway https://blog.pypi.org/posts/2023-06-26-deprecate-egg-uploads/ 5: OK to add pyproject.toml to the output distribution. This is standard for poetry. 6: OK this is egg-related. See diff 4. 7: OK since we don't use setup.py anymore Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> moving linting to nox (and eventually everything from tox) unit tests to nox migrating integration tests typo use cluster http path Replace build_dist.sh to perform build checks Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com> Preliminary GitHub Action to publish to test-pypi with poetry Signed-off-by: Jesse Whitehouse <jesse.whitehouse@databricks.com>
databricks · Sep 14, 2023 · c124e90 · c124e90
1 parent a6ddca9
commit c124e90
Show file tree

Hide file tree

Showing 17 changed files with 2,960 additions and 216 deletions.
diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
@@ -5,34 +5,55 @@ on:
       - "**.MD"
       - "**.md"
 jobs:
-  run-tox-tests-uc-cluster:
+  run-integration-tests-uc-cluster:
     runs-on: ubuntu-latest
     environment: azure-prod
     env:
       DBT_DATABRICKS_HOST_NAME: ${{ secrets.DATABRICKS_HOST }}
       DBT_DATABRICKS_CLIENT_ID: ${{ secrets.TEST_PECO_SP_ID }}
       DBT_DATABRICKS_CLIENT_SECRET: ${{ secrets.TEST_PECO_SP_SECRET }}
       DBT_DATABRICKS_UC_INITIAL_CATALOG: peco
-      DBT_DATABRICKS_LOCATION_ROOT: ${{ secrets.TEST_PECO_EXTERNAL_LOCATION }}
       TEST_PECO_UC_CLUSTER_ID: ${{ secrets.TEST_PECO_UC_CLUSTER_ID }}
+
     steps:
       - name: Check out repository
         uses: actions/checkout@v3
+
       - name: Set up python
         id: setup-python
         uses: actions/setup-python@v4
         with:
           python-version: "3.10"
+
       - name: Get http path from environment
         run: python .github/workflows/build_cluster_http_path.py
         shell: sh
-      - name: Install tox
-        id: install-dependencies
-        run: pip install tox
-      - name: Run integration-uc-databricks-cluster
-        run: DBT_TEST_USER=notnecessaryformosttests@example.com DBT_DATABRICKS_LOCATION_ROOT=$DBT_DATABRICKS_LOCATION_ROOT DBT_DATABRICKS_HOST_NAME=$DBT_DATABRICKS_HOST_NAME DBT_DATABRICKS_UC_CLUSTER_HTTP_PATH=$DBT_DATABRICKS_UC_CLUSTER_HTTP_PATH DBT_DATABRICKS_CLIENT_ID=$DBT_DATABRICKS_CLIENT_ID DBT_DATABRICKS_CLIENT_SECRET=$DBT_DATABRICKS_CLIENT_SECRET tox -e integration-databricks-uc-cluster
 
-  run-tox-tests-uc-sql:
+      - name: Install Poetry
+        uses: snok/install-poetry@v1
+        with:
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+          installer-parallel: true
+
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v3
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
+
+      - name: Install dependencies
+        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+        run: poetry install --no-interaction --no-root
+
+      - name: Install project
+        run: poetry install --no-interaction
+
+      - name: Run UC cluster integration tests
+        run: poetry run nox -t uc_cluster
+
+  run-integration-tests-uc-sql:
     runs-on: ubuntu-latest
     environment: azure-prod
     env:
@@ -43,44 +64,88 @@ jobs:
       DBT_DATABRICKS_UC_INITIAL_CATALOG: peco
       DBT_DATABRICKS_LOCATION_ROOT: ${{ secrets.TEST_PECO_EXTERNAL_LOCATION }}
       TEST_PECO_UC_CLUSTER_ID: ${{ secrets.TEST_PECO_UC_CLUSTER_ID }}
+
     steps:
       - name: Check out repository
         uses: actions/checkout@v3
+
       - name: Set up python
         id: setup-python
         uses: actions/setup-python@v4
         with:
           python-version: "3.10"
+
       - name: Get http path from environment
         run: python .github/workflows/build_cluster_http_path.py
         shell: sh
-      - name: Install tox
-        id: install-dependencies
-        run: pip install tox
-      - name: Run integration-databricks-uc-sql-endpoint
-        run: DBT_TEST_USER=notnecessaryformosttests@example.com DBT_DATABRICKS_LOCATION_ROOT=$DBT_DATABRICKS_LOCATION_ROOT DBT_DATABRICKS_HOST_NAME=$DBT_DATABRICKS_HOST_NAME DBT_DATABRICKS_UC_CLUSTER_HTTP_PATH=$DBT_DATABRICKS_UC_CLUSTER_HTTP_PATH DBT_DATABRICKS_CLIENT_ID=$DBT_DATABRICKS_CLIENT_ID DBT_DATABRICKS_CLIENT_SECRET=$DBT_DATABRICKS_CLIENT_SECRET tox -e integration-databricks-uc-sql-endpoint
 
-  run-tox-tests-non-uc:
+      - name: Install Poetry
+        uses: snok/install-poetry@v1
+        with:
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+          installer-parallel: true
+
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v3
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
+
+      - name: Install dependencies
+        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+        run: poetry install --no-interaction --no-root
+
+      - name: Install project
+        run: poetry install --no-interaction
+
+      - name: Run UC SQL Warehouse integration tests
+        run: poetry run nox -t uc_sql_endpoint
+
+  run-integration-tests-non-uc:
     runs-on: ubuntu-latest
     environment: azure-prod
     env:
       DBT_DATABRICKS_HOST_NAME: ${{ secrets.DATABRICKS_HOST }}
-      DBT_DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}
+      DBT_DATABRICKS_CLIENT_ID: ${{ secrets.TEST_PECO_SP_ID }}
+      DBT_DATABRICKS_CLIENT_SECRET: ${{ secrets.TEST_PECO_SP_SECRET }}
       TEST_PECO_CLUSTER_ID: ${{ secrets.TEST_PECO_CLUSTER_ID }}
-      DBT_DATABRICKS_LOCATION_ROOT: ${{ secrets.TEST_PECO_EXTERNAL_LOCATION }}
+
     steps:
       - name: Check out repository
         uses: actions/checkout@v3
+
       - name: Set up python
         id: setup-python
         uses: actions/setup-python@v4
         with:
           python-version: "3.10"
+
       - name: Get http path from environment
         run: python .github/workflows/build_cluster_http_path.py
         shell: sh
-      - name: Install tox
-        id: install-dependencies
-        run: pip install tox
-      - name: Run integration-databricks-cluster
-        run: DBT_TEST_USER=notnecessaryformosttests@example.com DBT_DATABRICKS_LOCATION_ROOT=$DBT_DATABRICKS_LOCATION_ROOT DBT_DATABRICKS_HOST_NAME=$DBT_DATABRICKS_HOST_NAME DBT_DATABRICKS_HTTP_PATH=$DBT_DATABRICKS_CLUSTER_HTTP_PATH DBT_DATABRICKS_CLIENT_ID=$DBT_DATABRICKS_CLIENT_ID DBT_DATABRICKS_CLIENT_SECRET=$DBT_DATABRICKS_CLIENT_SECRET tox -e integration-databricks-cluster
+
+      - name: Install Poetry
+        uses: snok/install-poetry@v1
+        with:
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+          installer-parallel: true
+
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v3
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
+
+      - name: Install dependencies
+        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+        run: poetry install --no-interaction --no-root
+
+      - name: Install project
+        run: poetry install --no-interaction
+
+      - name: Run Non-UC cluster integration tests
+        run: poetry run nox -t cluster
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
@@ -42,19 +42,13 @@ defaults:
 
 jobs:
   code-quality:
-    name: ${{ matrix.toxenv }}
+    name: Linting
 
     runs-on: ubuntu-latest
     timeout-minutes: 10
 
     strategy:
       fail-fast: false
-      matrix:
-        toxenv: [linter]
-
-    env:
-      TOXENV: ${{ matrix.toxenv }}
-      PYTEST_ADDOPTS: "-v --color=yes"
 
     steps:
       - name: Check out the repository
@@ -63,19 +57,31 @@ jobs:
           persist-credentials: false
 
       - name: Set up Python
+        id: setup-python
         uses: actions/setup-python@v4.3.0
         with:
-          python-version: "3.8"
+          python-version: "3.10"
 
-      - name: Install python dependencies
-        run: |
-          python -m pip install --user --upgrade pip
-          python -m pip --version
-          python -m pip install tox
-          tox --version
+      - name: Install Poetry
+        uses: snok/install-poetry@v1
+        with:
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+          installer-parallel: true
+
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v3
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
 
-      - name: Run tox
-        run: tox
+      - name: Install dependencies
+        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+        run: poetry install --no-interaction --no-root
+
+      - name: Run linting
+        run: poetry run nox -t lint
 
   unit:
     name: unit test / python ${{ matrix.python-version }}
@@ -89,7 +95,6 @@ jobs:
         python-version: ["3.8", "3.9", "3.10", "3.11"]
 
     env:
-      TOXENV: "unit"
       PYTEST_ADDOPTS: "-v --color=yes --csv unit_results.csv"
 
     steps:
@@ -99,18 +104,31 @@ jobs:
           persist-credentials: false
 
       - name: Set up Python ${{ matrix.python-version }}
+        id: setup-python
         uses: actions/setup-python@v4.3.0
         with:
           python-version: ${{ matrix.python-version }}
 
-      - name: Install python dependencies
-        run: |
-          python -m pip install --user --upgrade pip
-          python -m pip --version
-          python -m pip install tox
-          tox --version
-      - name: Run tox
-        run: tox
+      - name: Install Poetry
+        uses: snok/install-poetry@v1
+        with:
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+          installer-parallel: true
+
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v3
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
+
+      - name: Install dependencies
+        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+        run: poetry install --no-interaction --no-root
+
+      - name: Run unit tests
+        run: poetry run nox -s unit-${{ matrix.python-version }}
 
       - name: Get current date
         if: always()
@@ -138,15 +156,33 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v4.3.0
         with:
-          python-version: "3.8"
+          python-version: "3.10"
 
       - name: Install python dependencies
         run: |
           python -m pip install --user --upgrade pip
           python -m pip install --upgrade setuptools wheel twine check-wheel-contents
           python -m pip --version
-      - name: Build distributions
-        run: ./scripts/build-dist.sh
+      - name: Install Poetry
+        uses: snok/install-poetry@v1
+        with:
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+          installer-parallel: true
+
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v3
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
+
+      - name: Install dependencies
+        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+        run: poetry install --no-interaction --no-root
+
+      - name: Build distribution
+        run: poetry build
 
       - name: Show distributions
         run: ls -lh dist/

diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml
@@ -0,0 +1,55 @@
+name: Publish to PyPI [Production]
+on:
+  release:
+    types: [published]
+jobs:
+  publish:
+    name: Publish
+    runs-on: ubuntu-latest
+    steps:
+      #----------------------------------------------
+      #       check-out repo and set-up python
+      #----------------------------------------------
+      - name: Check out repository
+        uses: actions/checkout@v2
+      - name: Set up python
+        id: setup-python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+      #----------------------------------------------
+      #  -----  install & configure poetry  -----
+      #----------------------------------------------
+      - name: Install Poetry
+        uses: snok/install-poetry@v1
+        with:
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+          installer-parallel: true
+      #----------------------------------------------
+      #       load cached venv if cache exists
+      #----------------------------------------------
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v2
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ github.event.repository.name }}-${{ hashFiles('**/poetry.lock') }}
+      #----------------------------------------------
+      # install dependencies if cache does not exist
+      #----------------------------------------------
+      - name: Install dependencies
+        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+        run: poetry install --no-interaction --no-root
+      #-----------------------------------------------------------------------------
+      # Make sure the version is set to the current tagged release
+      #-----------------------------------------------------------------------------
+      - name: Update pyproject.toml
+        run: poetry version ${{ github.ref_name }}
+      #----------------------------------------------
+      # Attempt push to test-pypi
+      #----------------------------------------------
+      - name: Build and publish to pypi
+        uses: JRubics/poetry-publish@v1.10
+        with:
+          pypi_token: ${{ secrets.PYPI_TEST_SECRET }}
diff --git a/MANIFEST.in b/MANIFEST.in
diff --git a/black.ini b/black.ini
diff --git a/dbt/adapters/databricks/__version__.py b/dbt/adapters/databricks/__version__.py
@@ -1 +1,4 @@
-version: str = "1.6.4"
+import importlib.metadata
+
+__version__ = importlib.metadata.version("dbt-databricks")
+version = __version__