Skip to content
Permalink
Branch: master
Commits on Aug 12, 2019
  1. dagster-aws test fixes

    natekupp committed Aug 7, 2019
    Test Plan: unit
    
    Reviewers: max
    
    Reviewed By: max
    
    Subscribers: max
    
    Differential Revision: https://dagster.phacility.com/D780
Commits on Aug 11, 2019
  1. Fix dagster-aws CloudWatch tests

    natekupp committed Aug 11, 2019
    Summary: Also had to manually attach `arn:aws:iam::aws:policy/CloudWatchLogsFullAccess` to the buildkite roles to make this work (one more reason we should get this all in pulumi...)
    
    Test Plan: unit
    
    Reviewers: #ft, schrockn
    
    Reviewed By: #ft, schrockn
    
    Subscribers: schrockn
    
    Differential Revision: https://dagster.phacility.com/D796
Commits on Aug 9, 2019
  1. Update S3 buckets to use new dagster-scratch bucket

    natekupp committed Aug 8, 2019
    Summary: Centralizing our "scratch" usage in one place, `dagster-scratch`, which is a new bucket with appropriate ACLs/lifecycle policies
    
    Test Plan: unit
    
    Reviewers: #ft, alangenfeld
    
    Reviewed By: #ft, alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D786
Commits on Aug 8, 2019
  1. Small publish script cleanup

    natekupp committed Aug 7, 2019
    Test Plan: na
    
    Reviewers: alangenfeld, max
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D776
  2. Add smoke tests for clean pipenv install

    natekupp committed Aug 7, 2019
    Summary: Fixes our `pyyaml` dep issue and adds smoke tests to ensure installing with `pipenv` works
    
    Test Plan: unit
    
    Reviewers: #ft, max
    
    Reviewed By: #ft, max
    
    Differential Revision: https://dagster.phacility.com/D778
Commits on Aug 7, 2019
  1. Add more tests for pyspark page rank example

    natekupp committed Aug 7, 2019
    Summary: this gets test coverage for this example to 97%
    
    Test Plan: unit
    
    Reviewers: #ft, alangenfeld
    
    Reviewed By: #ft, alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D773
  2. Rewrote airline demo readme

    natekupp committed Aug 1, 2019
    Test Plan: docs only
    
    Reviewers: #ft, max
    
    Reviewed By: #ft, max
    
    Subscribers: max, alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D755
Commits on Aug 6, 2019
  1. Add SFTP solid

    natekupp committed Aug 4, 2019
    Summary:
    Depends on D760.
    
    Implements a working SFTP solid (ported from Airflow)
    
    Test Plan: unit
    
    Reviewers: #ft, alangenfeld, max
    
    Reviewed By: #ft, max
    
    Subscribers: max, alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D761
  2. Remove unnecessary pins now that Airflow 1.10.4 is released

    natekupp committed Aug 6, 2019
    Test Plan: unit
    
    Reviewers: #ft, max
    
    Reviewed By: #ft, max
    
    Differential Revision: https://dagster.phacility.com/D770
  3. Update publish script

    natekupp committed Aug 6, 2019
    Test Plan: na
    
    Reviewers: alangenfeld
    
    Reviewed By: alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D769
Commits on Aug 5, 2019
  1. Add Dagster SSH resource

    natekupp committed Aug 3, 2019
    Summary: Adding this as a pre-req for a SFTP solid
    
    Test Plan: unit
    
    Reviewers: #ft, alangenfeld
    
    Reviewed By: #ft, alangenfeld
    
    Subscribers: alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D760
  2. Add Dagster bash solid

    natekupp committed Aug 2, 2019
    Test Plan: unit
    
    Reviewers: #ft, alangenfeld
    
    Reviewed By: #ft, alangenfeld
    
    Subscribers: alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D756
  3. Add Papertrail integration

    natekupp committed Aug 2, 2019
    Test Plan: unit
    
    Reviewers: #ft, alangenfeld
    
    Reviewed By: #ft, alangenfeld
    
    Subscribers: alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D757
Commits on Aug 2, 2019
  1. Remove use of SolidDefinition in airline demo

    natekupp committed Aug 1, 2019
    Test Plan: unit
    
    Reviewers: #ft, alangenfeld
    
    Reviewed By: #ft, alangenfeld
    
    Subscribers: alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D754
Commits on Jul 31, 2019
  1. More minor cleanup in seven (#1634)

    natekupp committed Jul 31, 2019
    Summary: Addresses #1634
    
    Test Plan: unit
    
    Reviewers: #ft, schrockn
    
    Reviewed By: #ft, schrockn
    
    Subscribers: schrockn
    
    Differential Revision: https://dagster.phacility.com/D747
Commits on Jul 30, 2019
  1. Fixes for dagster-airflow scaffold CLI

    natekupp committed Jul 30, 2019
    Summary: This adds support for presets and adopts the same CLI interface as dagster proper
    
    Test Plan: unit
    
    Reviewers: #ft, max
    
    Reviewed By: #ft, max
    
    Subscribers: max
    
    Differential Revision: https://dagster.phacility.com/D743
  2. Fix Airflow skip behavior

    natekupp committed Jul 30, 2019
    Summary:
    Turns out that marking `TaskInstance` objects as skipped doesn't work in Airflow proper, because the `DagRun` overwrites the task state after user code execution.
    
    Instead, we do the Python exceptions-as-control-flow thing and use `AirflowSkipException`, this appears to be the canonical way to implement skipping-yourself behavior: https://blog.godatadriven.com/zen-of-python-and-apache-airflow
    
    Test Plan: unit and hand-tested in Airflow
    
    Reviewers: #ft, alangenfeld
    
    Reviewed By: #ft, alangenfeld
    
    Subscribers: max, alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D739
Commits on Jul 29, 2019
  1. Snowflake library updates

    natekupp committed Jul 29, 2019
    Test Plan: unit
    
    Reviewers: alangenfeld
    
    Reviewed By: alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D735
Commits on Jul 27, 2019
  1. Surface step skipped in Airflow

    natekupp committed Jul 26, 2019
    Test Plan: unit
    
    Reviewers: #ft, schrockn
    
    Reviewed By: #ft, schrockn
    
    Subscribers: schrockn
    
    Differential Revision: https://dagster.phacility.com/D712
Commits on Jul 26, 2019
  1. Modernize Snowflake solids

    natekupp committed Jul 25, 2019
    Summary: this removes the old subclasses of `SolidDefinition` in favor of decorator syntax, and moves the query execution logic from a solid to the resource.
    
    Test Plan: unit
    
    Reviewers: #ft, alangenfeld
    
    Reviewed By: #ft, alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D709
Commits on Jul 25, 2019
  1. Simplify Airflow operators

    natekupp committed Jul 24, 2019
    Summary: This diff simplifies the two base airflow operators we use, in advance of implementing correct Airflow task skip behavior when a step skip event is returned by Dagster step execution
    
    Test Plan: unit
    
    Reviewers: #ft, max
    
    Reviewed By: #ft, max
    
    Subscribers: max
    
    Differential Revision: https://dagster.phacility.com/D685
  2. Add deployment docs for Airflow and Dask

    natekupp committed Jul 18, 2019
    Test Plan: docs only
    
    Reviewers: #ft, alangenfeld, max
    
    Reviewed By: #ft, alangenfeld, max
    
    Subscribers: max, alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D663
Commits on Jul 24, 2019
  1. Handle composites correctly in Airflow

    natekupp committed Jul 22, 2019
    Test Plan: unit
    
    Reviewers: #ft, max
    
    Reviewed By: #ft, max
    
    Subscribers: alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D672
  2. Consolidate Airflow and Dask GraphQL client logic

    natekupp committed Jul 23, 2019
    Test Plan: unit
    
    Reviewers: max
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D684
  3. Consolidate Airflow and Dask GraphQL client logic

    natekupp committed Jul 24, 2019
    Summary:
    This diff cleans up the divergent code paths that `dagster-airflow` and `dagster-dask` used to execute pipelines through `dagster-graphql`.
    
    Now, a single function is provided in `dagster-graphql`, `execute_start_pipeline_execution_query()` to execute the pipelines and return a list of DagsterEvents.
    
    This is in preparation for a following diff which will consume the dagster events to inform Airflow that a particular task was skipped using the Airflow `SkipMixin`.
    
    Test Plan: unit
    
    Reviewers: #ft, max
    
    Reviewed By: #ft, max
    
    Subscribers: max
    
    Differential Revision: https://dagster.phacility.com/D677
Commits on Jul 19, 2019
  1. Update contributing guide (#1583)

    natekupp committed Jul 19, 2019
    Test Plan: docs only
    
    Reviewers: #ft, schrockn
    
    Reviewed By: #ft, schrockn
    
    Subscribers: max, schrockn
    
    Differential Revision: https://dagster.phacility.com/D666
Commits on Jul 18, 2019
  1. Update docs to remove PipelineDefinition usage

    natekupp committed Jul 17, 2019
    Test Plan: docs only
    
    Reviewers: #ft, max
    
    Reviewed By: #ft, max
    
    Subscribers: max
    
    Differential Revision: https://dagster.phacility.com/D659
  2. Update Airflow demo to use DSL

    natekupp committed Jul 17, 2019
    Test Plan: unit
    
    Reviewers: max
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D658
Commits on Jul 16, 2019
  1. [Dask] Fix crash for dagster_pd example (#1568)

    natekupp committed Jul 16, 2019
    Test Plan: unit
    
    Reviewers: #ft
    
    Differential Revision: https://dagster.phacility.com/D639
  2. [Dask] Put Dask execution in IEngine interface

    natekupp committed Jul 16, 2019
    Summary:
    This diff breaks up `execute_on_dask` to split out a `DaskEngine` object; it also uses the executor config to construct the engine, so the core can invoke the `DaskEngine` without needing to know about Dask via the `executor_config` property on `RunConfig`
    
    I'll use this in a follow-up diff which will drop `execute_on_dask` entirely in favor of `execute_pipeline` w/ `RunConfig`.
    
    Test Plan: unit
    
    Reviewers: #ft, max
    
    Reviewed By: #ft, max
    
    Subscribers: max
    
    Differential Revision: https://dagster.phacility.com/D637
  3. Cleanup and bug fixes for Airflow guide code

    natekupp committed Jul 15, 2019
    Test Plan: n/a
    
    Reviewers: alangenfeld, schrockn
    
    Reviewed By: alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D621
Commits on Jul 15, 2019
  1. Add tests for automation

    natekupp committed Jul 13, 2019
    Summary: This gets automation test coverage from 0% to > 90%
    
    Test Plan: unit
    
    Reviewers: max, schrockn, alangenfeld
    
    Reviewed By: alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D612
Commits on Jul 12, 2019
  1. (#1555) Name tasks after pipelines + solids in Dask

    natekupp committed Jul 12, 2019
    Test Plan: tested on local Dask cluster - doesn't seem like there's an easy way to query the Dask scheduler to confirm this works / set up a unit test
    
    Reviewers: schrockn, max
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D607
  2. (#1548) Fix Dask + composites

    natekupp committed Jul 12, 2019
    Summary: We previously were mashing `foo.bar` into a single `SolidHandle`; this changes that logic to build up the stack of solid handles for `foo` and `bar` that `get_solid` expects
    
    Test Plan: unit
    
    Reviewers: schrockn, alangenfeld, max, sashank
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D606
  3. Fix bad merge

    natekupp committed Jul 12, 2019
    Summary: Had a bad merge in D582 that reverted Alex’s fix in D579. D582 added a bunch of tests for `handle.py`, and left only a single line untested -- which happened to be this one.
    
    Test Plan: unit, tested dagit manually
    
    Reviewers: schrockn, alangenfeld
    
    Reviewed By: alangenfeld
    
    Differential Revision: https://dagster.phacility.com/D602
Older
You can’t perform that action at this time.