Skip to content
Permalink
Branch: master
Commits on Aug 13, 2019
  1. Initial k8s operator and split out of shared functionality

    Eronarn authored and natekupp committed Jul 27, 2019
    Summary:
    1. Moving shit around: move our Python and Docker operator into two files
    2. Vendor Airflow k8s operator
    3. Add James' `DagsterKubernetesPodOperator`
    
    As a follow-up, will need to figure out how we add tests for (3).
    
    Test Plan:
    tested manually in local airflow + minikube with:
    
    ```
    '''
    The airflow DAG scaffold for dagster_examples.toys.optional_outputs.optional_outputs
    
    Note that this docstring must contain the strings "airflow" and "DAG" for
    Airflow to properly detect it as a DAG
    See: http://bit.ly/307VMum
    '''
    import datetime
    import yaml
    
    from dagster_airflow.factory import make_airflow_dag_kubernetized
    
    ENVIRONMENT = '''
    loggers:
      console:
        config:
          log_level: DEBUG
    
    solids:
      multiply_the_word:
        inputs:
          word:
            value: bar
        config:
          factor: 2
    
    storage:
      s3:
        config:
          s3_bucket: dagster-airflow-scratch
    
    '''
    
    DEFAULT_ARGS = {
        'owner': 'airflow',
        'depends_on_past': False,
        'start_date': datetime.datetime(2019, 5, 7),
        'email': ['airflow@example.com'],
        'email_on_failure': False,
        'email_on_retry': False,
    }
    
    dag, tasks = make_airflow_dag_kubernetized(
        module_name='dagster_airflow_tests.test_project.dagster_airflow_demo',
        pipeline_name='demo_pipeline',
        environment_dict=yaml.load(ENVIRONMENT),
        dag_kwargs={'default_args': DEFAULT_ARGS, 'max_active_runs': 1},
        image='dagster-airflow-demo',
        namespace='default',
    )
    ```
    
    Reviewers: #ft, schrockn
    
    Reviewed By: #ft, schrockn
    
    Subscribers: max, schrockn
    
    Differential Revision: https://dagster.phacility.com/D801
You can’t perform that action at this time.