Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setup airflow e2e execution in workflow #1466

Merged
merged 14 commits into from
Sep 13, 2023
52 changes: 52 additions & 0 deletions .github/workflows/integrations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -637,3 +637,55 @@ jobs:
with:
Raalsky marked this conversation as resolved.
Show resolved Hide resolved
slack-webhook: ${{ secrets.E2E_INTEGRATIONS_SLACK_WEBHOOK }}
job-status: ${{ needs.mlflow.result }}

airflow:
env:
NEPTUNE_PROJECT: e2e-tests/integrations
timeout-minutes: 75
strategy:
fail-fast: false
matrix:
python-version: [ "3.8", "3.10" ]
os: [ ubuntu-latest, macos-latest, windows-latest ]
runs-on: ${{ matrix.os }}
name: 'airflow (${{ matrix.os }} - py${{ matrix.python-version }})'
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
ref: ${{ github.event.client_payload.pull_request.head.ref }}

- name: Install package
uses: ./.github/actions/install-package
with:
python-version: ${{ matrix.python-version }}
os: ${{ matrix.os }}

- name: Install airflow
uses: neptune-ai/neptune-airflow/.github/actions/airflow-install@main
with:
python-version: ${{ matrix.python-version }}
working_directory: neptune-airflow

- name: Test integration
uses: neptune-ai/neptune-airflow/.github/actions/e2e@main
with:
working_directory: neptune-airflow

airflow-notify:
needs: [ airflow ]
runs-on: ubuntu-latest
if: (success() || failure()) && github.ref == 'refs/heads/master'
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
ref: ${{ github.event.client_payload.pull_request.head.ref }}

- name: Notify
uses: ./.github/actions/job-notify
with:
slack-webhook: ${{ secrets.E2E_INTEGRATIONS_SLACK_WEBHOOK }}
job-status: ${{ needs.airflow.result }}
6 changes: 4 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,8 @@ neptune-xgboost = { version = "*", optional = true }
transformers = { version = "*", optional = true }
zenml = { version = "*", optional = true, python = "<3.9" }
neptune-aws = { version = "*", optional = true }
mlflow = { version = ">1.30", optional = true }
neptune-mlflow = { version = "*", optional = true }
neptune-airflow = { version = "*", optional = true }

[tool.poetry.extras]
aws = ["neptune-aws"]
Expand All @@ -103,7 +104,8 @@ tensorboard = ["neptune-tensorboard"]
transformers = ["transformers"]
xgboost = ["neptune-xgboost"]
zenml = ["zenml"]
mlflow = ["mlflow"]
mlflow = ["neptune-mlflow"]
airflow = ["neptune-airflow"]
dev = [
"altair",
"bokeh",
Expand Down