-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RD-WATCH/Smartflow Integration #412
Merged
mvandenburgh
merged 22 commits into
ResonantGeoData:main
from
mvandenburgh:smartflow-integration
May 31, 2024
Merged
Changes from all commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
f02f82b
Add airflow dev container to docker-compose
mvandenburgh 820e869
Install `apache-airflow-client`
mvandenburgh 688ed5f
Add settings value for smartflow url
mvandenburgh dc8d779
Add smartflow healthcheck to server status endpoint
mvandenburgh 27286fb
Fix smartflow auth, move to its own app
mvandenburgh 5bb13cd
Add endpoint for starting a DAG run
mvandenburgh 1bb462e
Add get/list endpoints for DAG runs
mvandenburgh 9bf5665
wip smartflow UI
mvandenburgh d72aaef
Generate openapi client from new smartflow endpoints
mvandenburgh d741347
[wip] hook up smartflow UI to backend
mvandenburgh 813dc74
Add missing parameter to dag run endpoint
mvandenburgh cbb9c47
Add running a new dag to UI
mvandenburgh a62bfa8
Fix typescript errors
mvandenburgh ed60b73
Add smartflow init file to dockerfile
mvandenburgh 23c82a9
Add sample DAG to dev environment
mvandenburgh e5a1b0b
Require all smartflow env vars to be set to install app
mvandenburgh 4f56a85
Fix status endpoint
mvandenburgh b2de3b7
Fix smartflow url in docker .env file
mvandenburgh 0748e28
Put script section first in sfc
mvandenburgh a84ff45
Add default value for dag run conf
mvandenburgh e87305b
Fix healthcheck
mvandenburgh e51b451
Pin airflow image
mvandenburgh File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
FROM apache/airflow:slim-latest-python3.12 | ||
|
||
RUN airflow db migrate | ||
|
||
RUN airflow users create \ | ||
--username rdwatch \ | ||
--password rdwatch \ | ||
--firstname rdwatch \ | ||
--lastname admin \ | ||
--role Admin \ | ||
--email admin@localhost |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
from datetime import datetime | ||
|
||
from airflow import DAG | ||
from airflow.decorators import task | ||
from airflow.models import Param | ||
|
||
with DAG( | ||
dag_id="RD-WATCH-AIRFLOW-DEMO-DAG", | ||
description="Test DAG", | ||
params={ | ||
"region_id": Param( | ||
default="BR_R002", type="string", pattern=r"^[A-Z]{2}_[RCST]\d{3}$" | ||
), | ||
"model_run_title": Param(default="test_run", type="string"), | ||
}, | ||
start_date=datetime(2022, 3, 1), | ||
catchup=False, | ||
schedule_interval=None, | ||
max_active_runs=1, | ||
default_view="grid", | ||
tags=["demo", "watch", "ta2", "rgd", "random"], | ||
) as dag1: | ||
|
||
@task | ||
def print_region(**context): | ||
region_id = context["params"]["region_id"] | ||
print(f"region_id: {region_id}") | ||
|
||
@task | ||
def print_model_run_title(**context): | ||
model_run_title = context["params"]["model_run_title"] | ||
print(f"model_run_title: {model_run_title}") | ||
|
||
print_region() >> print_model_run_title() |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
from ninja import NinjaAPI | ||
|
||
from . import views | ||
|
||
api = NinjaAPI(urls_namespace='smartflow') | ||
|
||
api.add_router('/', views.router) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
from django.urls import path | ||
|
||
from .api import api | ||
|
||
urlpatterns = [ | ||
path('', api.urls), | ||
] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,93 @@ | ||
from functools import cached_property | ||
from typing import Any | ||
|
||
import airflow_client.client | ||
import requests | ||
from airflow_client.client.api import dag_api, dag_run_api, monitoring_api | ||
from airflow_client.client.models import ( | ||
DAGCollection, | ||
DAGRun, | ||
DAGRunCollection, | ||
HealthInfo, | ||
) | ||
from bs4 import BeautifulSoup | ||
|
||
from django.conf import settings | ||
|
||
|
||
class SmartFlowClient: | ||
@cached_property | ||
def _session_token(self) -> str: | ||
session = requests.Session() | ||
smartflow_url: str = settings.SMARTFLOW_URL | ||
username: str = settings.SMARTFLOW_USERNAME | ||
password: str = settings.SMARTFLOW_PASSWORD | ||
|
||
res = session.get(smartflow_url) | ||
soup = BeautifulSoup(res.text, 'html.parser') | ||
csrf_token = soup.find('input', {'name': 'csrf_token'})['value'] | ||
|
||
login_url = f'{smartflow_url}/login/' | ||
login_form_data = { | ||
'csrf_token': csrf_token, | ||
'username': username, | ||
'password': password, | ||
} | ||
login_headers = {'referer': login_url} | ||
res = session.post( | ||
url=login_url, | ||
data=login_form_data, | ||
headers=login_headers, | ||
) | ||
res.raise_for_status() | ||
return session.cookies.get('session') | ||
|
||
@property | ||
def _client(self): | ||
host: str = f'{settings.SMARTFLOW_URL}/api/v1' | ||
|
||
configuration = airflow_client.client.Configuration( | ||
host=host, | ||
# api_key={'session': self._session_token} | ||
) | ||
with airflow_client.client.ApiClient( | ||
configuration=configuration, | ||
header_name='cookie', | ||
header_value=f'session={self._session_token}', | ||
) as api_client: | ||
return api_client | ||
|
||
@property | ||
def _monitoring_api(self): | ||
return monitoring_api.MonitoringApi(self._client) | ||
|
||
@property | ||
def _dag_api(self): | ||
return dag_api.DAGApi(self._client) | ||
|
||
@property | ||
def _dag_run_api(self): | ||
return dag_run_api.DAGRunApi(self._client) | ||
|
||
def get_health(self) -> HealthInfo: | ||
return self._monitoring_api.get_health() | ||
|
||
def list_dags(self, **kwargs) -> DAGCollection: | ||
return self._dag_api.get_dags(**kwargs) | ||
|
||
def list_dag_runs(self, dag_id: str, **kwargs) -> DAGRunCollection: | ||
return self._dag_run_api.get_dag_runs(dag_id=dag_id, **kwargs) | ||
|
||
def create_dag_run( | ||
self, dag_id: str, dag_run_title: str, conf: dict[str, Any] | ||
) -> DAGRun: | ||
""" | ||
Parameters: | ||
dag_id: the id of the DAG to run | ||
dag_run_title: the title to give this new DAG run | ||
""" | ||
dag_run = DAGRun(conf=conf, dag_run_id=dag_run_title) | ||
return self._dag_run_api.post_dag_run(dag_id=dag_id, dag_run=dag_run) | ||
|
||
def get_dag_run(self, dag_id: str, dag_run_id: str) -> DAGRun: | ||
return self._dag_run_api.get_dag_run(dag_id=dag_id, dag_run_id=dag_run_id) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is hardcoded so in the .env file should we make a note of this? It isn't like other containers where the username/password is parameterized. Or maybe we should use the airflow environment variables?
https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#environment-variables-supported-by-docker-compose
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I saw this, but I couldn't get it to work so I just gave up and did that command in the docker file. Does it work for you?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
took a little searching but I think you can remove the local airflow DockerFile. These
_
based commands are only meant for local development, but I assume your infrastructure code handles all this.I wouldn't worry about implementation now, just logging this comment for the future.