Skip to content
Permalink
Branch: master
Commits on May 16, 2019
  1. fix preset sorting

    alangenfeld committed May 16, 2019
    Test Plan: buildkite
    
    Reviewers: natekupp
    
    Reviewed By: natekupp
    
    Differential Revision: https://dagster.phacility.com/D152
  2. remove need to yield None for Nothing output

    alangenfeld committed May 16, 2019
    Test Plan: updated unit tests, run many_events toy in dagit
    
    Reviewers: max, natekupp
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D148
  3. [nightly] dont prune in phab tag cleanup

    alangenfeld committed May 16, 2019
    Summary: -P still isnt supported in v2.11 which is what the default circle image uses. I don't think we need it since we are doing a fresh checkout - so just do the clean up part.
    
    Test Plan: eyes
    
    Reviewers: max
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D147
  4. move presets from repo yml to pipeline

    alangenfeld committed May 16, 2019
    Summary: Move preset definition from repository.yml to `PresetDefinition`s being passed to `PipelineDefinition`.
    
    Test Plan: unit, load error_monster in dagit and select the preset
    
    Reviewers: schrockn, natekupp, max
    
    Reviewed By: natekupp
    
    Differential Revision: https://dagster.phacility.com/D144
  5. [dagit] change Nothing type display

    alangenfeld committed May 16, 2019
    Summary: In the initial pass, I choose to hide details about `Nothing` type IO. This was probably not the right choice, so add it back everywhere except the type signature section.
    
    Test Plan: screenshots
    
    Reviewers: schrockn, natekupp, bengotow, max
    
    Reviewed By: natekupp
    
    Differential Revision: https://dagster.phacility.com/D145
  6. [nightly] add --autoclean to publish.py

    alangenfeld committed May 16, 2019
    Summary:
    hitting 1 pyc file in the nightly job, so lets add an autoclean flag for automation
    https://circleci.com/gh/dagster-io/dagster/166468
    
    Test Plan: not sure how to test this without doing a publish
    
    Reviewers: max
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D146
  7. enable arc lint

    alangenfeld committed May 3, 2019
    Summary:
    Turn on local linting to reduce round trips from lint failure.
    
    Note: this is blocked on a bug with arc: https://discourse.phabricator-community.org/t/arcanistpylintlinter-version-regex-issue/2688
    
    Test Plan: arc lint
    
    Reviewers: schrockn, natekupp, max
    
    Reviewed By: natekupp, max
    
    Differential Revision: https://dagster.phacility.com/D36
  8. fixing nightly

    alangenfeld committed May 16, 2019
Commits on May 15, 2019
  1. guard against npm install

    alangenfeld committed May 15, 2019
    Test Plan: {P7}
    
    Reviewers: max
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D139
  2. [nightly] do phab tags last

    alangenfeld committed May 15, 2019
    Summary: We check out the nightly branch (from N days ago since this has been broken) so we should run this step after we've updated to master and pushed the new nightly.
    
    Test Plan: ???
    
    Reviewers: max
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D140
  3. [dagster] Fan In Dependency

    alangenfeld committed May 10, 2019
    Summary:
    Add `FanInDependencyDefinition` which is constructed with a list of `DependencyDefinition`s. Currently only supports `Nothing` type inputs.
    
    Updates dagit and dagster-graphql to support an input depending on multiple outputs.
    
    Test Plan: added unit tests
    
    Reviewers: schrockn, max, natekupp
    
    Reviewed By: schrockn
    
    Differential Revision: https://dagster.phacility.com/D109
Commits on May 14, 2019
  1. Expose CompositeSolid in GraphQL

    alangenfeld committed May 10, 2019
    Summary: See `schema.graphql` changes
    
    Test Plan:
    Query
    {P5}
    
    Response
    {P4}
    
    Reviewers: bengotow, schrockn
    
    Reviewed By: schrockn
    
    Differential Revision: https://dagster.phacility.com/D79
  2. fix nightly

    alangenfeld committed May 14, 2019
    Summary: Circle uses an older version of git that doesnt have --prune-tags / -P so use -p -t to attempt to prune tags
    
    Test Plan: ???
    
    Reviewers: max
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D130
  3. hide *.generated.json in phab

    alangenfeld committed May 13, 2019
    Test Plan: shows up as generated in phabricator
    
    Reviewers: schrockn, max
    
    Reviewed By: schrockn
    
    Differential Revision: https://dagster.phacility.com/D116
Commits on May 13, 2019
  1. [buildkite] fix GraphQL change detection

    alangenfeld committed May 13, 2019
    Summary:
    * switch to bash and use +x to see commands being invoked in logs more clearly
    * fix GraphQL check by adding accidentally omitted download-schema invocation
    
    Test Plan: https://dagster.phacility.com/D111?id=542 failed with graphql change
    
    Reviewers: schrockn, max
    
    Reviewed By: schrockn
    
    Differential Revision: https://dagster.phacility.com/D111
Commits on May 10, 2019
  1. [dagster] CompositeSolid (4/N) - IO mapping

    alangenfeld committed May 8, 2019
    Summary:
    Introduce `InputMapping` and `OutputMapping` for mapping inputs and outputs in a composite to solids it contains.
    
    depends on D54
    
    Test Plan: new unit tests
    
    Reviewers: schrockn, max, natekupp
    
    Reviewed By: schrockn
    
    Differential Revision: https://dagster.phacility.com/D68
  2. [dagster] CompositeSolid (3/N) - config

    alangenfeld committed May 7, 2019
    Summary:
    Support configuration for CompositeSolids. The current approach is the most straightforward one, nesting the config in the same way the solids are nested.
    
    ```
    solids:
      composite_name:
        solids:
          solid_name:
            inputs:
            config:
    ```
    
    We may want to add a way to use solid handles in the top level to flatten out things from getting too crazy nested, but I think we'll probably always want to support this approach as well?
    
    Interested to hear what thoughts others have.
    
    depends on D49
    
    Test Plan: new unit tests
    
    Reviewers: max, natekupp, schrockn
    
    Reviewed By: schrockn
    
    Differential Revision: https://dagster.phacility.com/D54
  3. [dagster] CompositeSolid (2/N) - execution

    alangenfeld committed May 7, 2019
    Summary:
    Basic execution of composite solids.
    * Introduces recursion to the execution plan creation
    * fastmod `result_for_solid` to `result_for_top_level_solid`
    
    Subsequent diffs will handle config and in/out mappings.
    
    Test Plan: new test executes in a composite
    
    Reviewers: max, natekupp, schrockn
    
    Reviewed By: schrockn
    
    Differential Revision: https://dagster.phacility.com/D49
Commits on May 8, 2019
  1. fix nightly job

    alangenfeld committed May 8, 2019
    Summary:
    not sure whats up with tag cleaning, seems to be trying to delete the same ones every day.
    
    Also we deleted the script its trying to run.
    
    Test Plan: ???
    
    Reviewers: max, schrockn
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D58
  2. [buildkite] add timeout

    alangenfeld committed May 7, 2019
    Test Plan: buildkite
    
    Reviewers: schrockn, natekupp, max
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D46
Commits on May 7, 2019
  1. [dagster] CompositeSolid (1/N)

    alangenfeld committed May 3, 2019
    Summary:
    Introduces `CompositeSolidDefinition` and `ISolidDefinition`.
    
    Changes the public API of `ISolidDefinition` from using `input_defs` and `output_defs` to `input_dict`/`output_dict`. This is to facilite the remapping that is done by CompositeSolids to their internal solids.
    
    Test Plan: No new functionality at this point so just existing tests.
    
    Reviewers: max, schrockn, natekupp
    
    Reviewed By: schrockn
    
    Differential Revision: https://dagster.phacility.com/D45
  2. [dagster] SolidHandle

    alangenfeld committed May 3, 2019
    Summary:
    Introduce SolidHandle, a way to uniquely identify a Solid within a pipeline.
    
    This diff doesn't introduce any new behavior, just preps for subsequent changes to create CompositeSolid.
    
    From exploration D30
    
    Test Plan: buildkite
    
    Reviewers: schrockn, max, natekupp
    
    Reviewed By: schrockn, natekupp
    
    Differential Revision: https://dagster.phacility.com/D38
  3. run black on examples as well

    alangenfeld committed Apr 25, 2019
    Summary: Not sure if this was intentional, but we should autoformat examples.
    
    Test Plan: CI passes
    
    Reviewers: schrockn, max, natekupp
    
    Reviewed By: max
    
    Differential Revision: https://dagster.phacility.com/D12
Commits on May 3, 2019
  1. clean phabricator tags in nightly job

    alangenfeld committed May 3, 2019
    Summary:
    I've been doing this manually everyonce and a while. Lets automote it.
    
    This cleans up the tags generated by phab to handoff builds to buildkite.
    
    Test Plan: ???
    
    Reviewers: max, natekupp, schrockn
    
    Reviewed By: natekupp, schrockn
    
    Differential Revision: https://dagster.phacility.com/D37
Commits on Apr 30, 2019
  1. [buildkite] remove circle test jobs

    alangenfeld committed Apr 29, 2019
    Summary:
    With all the jobs moved to buildkite, we can stop using circle.
    
    Landing this will finalize our cut over, whenever we are ready to do so.
    
    depends on D22
    
    Test Plan: buildkite
    
    Reviewers: max, schrockn, natekupp
    
    Reviewed By: schrockn
    
    Differential Revision: https://dagster.phacility.com/D23
  2. [buildkite] add dagster-airflow tests

    alangenfeld committed Apr 29, 2019
    Test Plan: buildkite
    
    Reviewers: max, schrockn, natekupp
    
    Reviewed By: natekupp
    
    Differential Revision: https://dagster.phacility.com/D22
  3. [dagster] naive multiproc engine upgrade

    alangenfeld committed Apr 25, 2019
    Summary:
    A very simple approach to actually use multiple processes in multiproc mode.
    
    Run every step in a step_level from a topological sort in parallel. To do this we change the loop thats polling the child process to yield, and then thread that yielding up to a point where we are holding on to iterators for every step that we are executing at the same time. Keep spinning over these untl they are done.
    
    Test Plan: {P3}
    
    Reviewers: schrockn, natekupp, max
    
    Reviewed By: schrockn
    
    Differential Revision: https://dagster.phacility.com/D11
Commits on Apr 29, 2019
  1. [buildkite] events demo tests

    alangenfeld committed Apr 25, 2019
    Test Plan:
    buildkite
    
    depends on D3
    
    Reviewers: natekupp, max
    
    Reviewed By: natekupp
    
    Differential Revision: https://dagster.phacility.com/D8
  2. [buildkite] airline demo tests

    alangenfeld committed Apr 23, 2019
    Summary:
    * Create `buildkite-integration` images that contain the super set of things for our complicated CI jobs.
    * Port the airline-demo test to buildkite
      * The step is invoked a `buildkite-integration` docker container. Were going to run docker from within this container, so we have to map `dock.sock` so we can use the host docker daemon. We then build the airflow container from this container to contain up to date versions of dagster. We then invoke docker-compose to run postgres. To communicate between the now sibling container, we introduce an environment variable which we populate with the ip address of the db container.
    
    Test Plan: buildkite
    
    Reviewers: max, schrockn, natekupp
    
    Reviewed By: schrockn
    
    Differential Revision: https://dagster.phacility.com/D3
Commits on Apr 25, 2019
  1. fix expectations in dagster-graphql

    alangenfeld committed Apr 25, 2019
    Test Plan: updated unit tests
    
    Reviewers: schrockn
    
    Reviewed By: schrockn
    
    Differential Revision: https://dagster.phacility.com/D9
  2. [buildkite] pipeline improvements (#1306)

    alangenfeld committed Apr 25, 2019
Commits on Apr 23, 2019
  1. buildkite setup (#1286)

    alangenfeld committed Apr 23, 2019
    Test Plan: pipeline works in buildkite
  2. fix lint (#1288)

    alangenfeld committed Apr 23, 2019
Commits on Apr 18, 2019
  1. [dagster-airflow] upgraded airflow dep (#1271)

    alangenfeld committed Apr 18, 2019
Commits on Apr 16, 2019
  1. [dagit] Runs table (#1257)

    alangenfeld authored and schrockn committed Apr 16, 2019
Older
You can’t perform that action at this time.