Skip to content
This repository has been archived by the owner on Oct 9, 2023. It is now read-only.

Instrument ArrayNode #550

Merged
merged 68 commits into from
Jul 31, 2023
Merged

Instrument ArrayNode #550

merged 68 commits into from
Jul 31, 2023

Commits on Apr 3, 2023

  1. updated flyteidl to local to get ArrayNode 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    4bb3138 View commit details
    Browse the repository at this point in the history

  2. added boilerplate to support ArrayNode 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 3, 2023
    Configuration menu
    Copy the full SHA
    33deb42 View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2023

  1. pushing forward 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    bf0a1ff View commit details
    Browse the repository at this point in the history

  2. refactored node executor interfaces to fix dependency cycle 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    6dc7fb1 View commit details
    Browse the repository at this point in the history

  3. refactoring almost complete 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    188a675 View commit details
    Browse the repository at this point in the history

  4. refactor complete 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    6f379e7 View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2023

  1. supporting environment variables 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    caa9b07 View commit details
    Browse the repository at this point in the history

Commits on Apr 13, 2023

  1. minimum viable product 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    03dcc56 View commit details
    Browse the repository at this point in the history

  2. update print statements for debugging 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    658abc7 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2023

  1. massive refactor fixing NodeExecutionContext override for ArrayNode 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    e33af40 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2023

  1. refactoring TODOs 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    110e1ea View commit details
    Browse the repository at this point in the history

  2. subnode retries working 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    0a5c58a View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2023

  1. parallelism working 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    7fee1e6 View commit details
    Browse the repository at this point in the history

  2. cache and cache_serialize working - first new functionality in maptask 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 27, 2023
    Configuration menu
    Copy the full SHA
    dc0c164 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2023

  1. adding implementation notes 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    0f880cd View commit details
    Browse the repository at this point in the history

Commits on May 22, 2023

  1. Merge branch 'master' into feature/array-node 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed May 22, 2023
    Configuration menu
    Copy the full SHA
    0aa773e View commit details
    Browse the repository at this point in the history

Commits on May 24, 2023

  1. removed eventing from subtasks 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed May 24, 2023
    Configuration menu
    Copy the full SHA
    e740627 View commit details
    Browse the repository at this point in the history

Commits on May 25, 2023

  1. adding correct requirements 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed May 25, 2023
    Configuration menu
    Copy the full SHA
    b8271ef View commit details
    Browse the repository at this point in the history

Commits on May 26, 2023

  1. working end-2-end with flytekit 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed May 26, 2023
    Configuration menu
    Copy the full SHA
    4549591 View commit details
    Browse the repository at this point in the history

  2. reporting output directory on success 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed May 26, 2023
    Configuration menu
    Copy the full SHA
    b4c6f3e View commit details
    Browse the repository at this point in the history

Commits on May 30, 2023

  1. fixed output directory append 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed May 30, 2023
    Configuration menu
    Copy the full SHA
    134f215 View commit details
    Browse the repository at this point in the history

  2. mocking TaskTemplate interface to enable caching 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed May 30, 2023
    Configuration menu
    Copy the full SHA
    d36cd44 View commit details
    Browse the repository at this point in the history

  3. capture failure reasons 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed May 30, 2023
    Configuration menu
    Copy the full SHA
    fa472d4 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2023

  1. wrapped up abort and finalize functionality 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed May 31, 2023
    Configuration menu
    Copy the full SHA
    a6e20c7 View commit details
    Browse the repository at this point in the history

  2. mocking initialization events 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed May 31, 2023
    Configuration menu
    Copy the full SHA
    f5a46b9 View commit details
    Browse the repository at this point in the history

  3. sending all events 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed May 31, 2023
    Configuration menu
    Copy the full SHA
    6dcbe53 View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2023

  1. minor refactoring of debug prints and formatting 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    59df842 View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2023

  1. intratask checkpointing working 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 2, 2023
    Configuration menu
    Copy the full SHA
    9a9d0f0 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2023

  1. support for and 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    377497e View commit details
    Browse the repository at this point in the history

  2. setting node log ids correctly 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    5cf3259 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2023

  1. reporting cache status 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    437dc91 View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2023

  1. Merge branch 'master' into feature/array-node 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    a6d31d7 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2023

  1. correctly setting subnode abort phase 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    d2abccc View commit details
    Browse the repository at this point in the history

  2. removing dead code 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    1de3e16 View commit details
    Browse the repository at this point in the history

  3. cleaned up most random TODO items 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    a66bb60 View commit details
    Browse the repository at this point in the history

  4. refactored into new files 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    3609dd7 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2023

  1. refactoring for ArrayNode unit tests 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    fdf8d6a View commit details
    Browse the repository at this point in the history

  2. refactored for unit testing to allow creation of NodeExecutor in arra… 

    …y package

Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    b150f1a View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2023

  1. first unit test for handling ArrayNodePhaseNone 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    b871b23 View commit details
    Browse the repository at this point in the history

  2. most of executing unit tests completed 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    49f4d32 View commit details
    Browse the repository at this point in the history

  3. finished executing unit tests 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    7767668 View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2023

  1. finished succeeding unit tests 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    73a26f4 View commit details
    Browse the repository at this point in the history

  2. wrote failing phase unit tests 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    575deea View commit details
    Browse the repository at this point in the history

  3. moving towards complete unit_test success 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    6dc6b88 View commit details
    Browse the repository at this point in the history

  4. unit tests passing 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    a38ec17 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2023

  1. fixed lint issues 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    db1e361 View commit details
    Browse the repository at this point in the history

  2. updated flyteidl dep 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    e0f156d View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2023

  1. Merge branch 'master' into feature/array-node 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    509900b View commit details
    Browse the repository at this point in the history

  2. added unit tests for Abort 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    211ff29 View commit details
    Browse the repository at this point in the history

  3. adding unit test for Finalize 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    c6df3fd View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2023

  1. added utils unit tests 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 30, 2023
    Configuration menu
    Copy the full SHA
    7c1931d View commit details
    Browse the repository at this point in the history

  2. moved state structs to handler package 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jun 30, 2023
    Configuration menu
    Copy the full SHA
    0df9572 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2023

  1. added docs 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 5, 2023
    Configuration menu
    Copy the full SHA
    4496efb View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. cleaned up abort event reporting 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    b13ef67 View commit details
    Browse the repository at this point in the history

  2. fixed RecordNodeEvent unit tests 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    1c446b7 View commit details
    Browse the repository at this point in the history

  3. removed taskEventRecorder from nodes package 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    7782460 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2023

  1. adding interface checking for arraynode 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    837379d View commit details
    Browse the repository at this point in the history

  2. added transform unit test 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    23d5312 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into feature/array-node 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    5615a86 View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2023

  1. fixed input bindings issue 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    5dbc665 View commit details
    Browse the repository at this point in the history

  2. fixed unit tests 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    7914641 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2023

  1. fixed unit tests 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    d5eb484 View commit details
    Browse the repository at this point in the history

  2. go generate 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    b044277 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2023

  1. addressing random TODO 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    14fae22 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into feature/array-node 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    34937b3 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2023

  1. fixed unit tests 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    bdbef61 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2023

  1. Merge branch 'master' into feature/array-node 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    d0edb9a View commit details
    Browse the repository at this point in the history

  2. addressing pr comments 

    Signed-off-by: Daniel Rammer <daniel@union.ai>
    @hamersaw
    hamersaw committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    70eda6a View commit details
    Browse the repository at this point in the history