This repository has been archived by the owner on Oct 9, 2023. It is now read-only.
Instrument ArrayNode #550
Merged
Instrument ArrayNode #550
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
kumare3
reviewed
Jul 27, 2023
kumare3
reviewed
Jul 27, 2023
kumare3
reviewed
Jul 27, 2023
kumare3
reviewed
Jul 27, 2023
kumare3
reviewed
Jul 27, 2023
kumare3
reviewed
Jul 27, 2023
kumare3
reviewed
Jul 27, 2023
There was a problem hiding this comment.
To be honest this is a crazy big code change and it looks pretty good. Boy though, when you look at some of the repeated verbosity in golang haha. But, mostly looks great to me, couple small comments.
Since this is behind a feature gate, i think this is safe to go
Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Daniel Rammer <daniel@union.ai>
kumare3
approved these changes
Jul 28, 2023
eapolinario
pushed a commit
to eapolinario/flytepropeller
that referenced
this pull request
Aug 9, 2023
* updated flyteidl to local to get ArrayNode Signed-off-by: Daniel Rammer <daniel@union.ai> * added boilerplate to support ArrayNode Signed-off-by: Daniel Rammer <daniel@union.ai> * pushing forward Signed-off-by: Daniel Rammer <daniel@union.ai> * refactored node executor interfaces to fix dependency cycle Signed-off-by: Daniel Rammer <daniel@union.ai> * refactoring almost complete Signed-off-by: Daniel Rammer <daniel@union.ai> * refactor complete Signed-off-by: Daniel Rammer <daniel@union.ai> * supporting environment variables Signed-off-by: Daniel Rammer <daniel@union.ai> * minimum viable product Signed-off-by: Daniel Rammer <daniel@union.ai> * update print statements for debugging Signed-off-by: Daniel Rammer <daniel@union.ai> * massive refactor fixing NodeExecutionContext override for ArrayNode Signed-off-by: Daniel Rammer <daniel@union.ai> * refactoring TODOs Signed-off-by: Daniel Rammer <daniel@union.ai> * subnode retries working Signed-off-by: Daniel Rammer <daniel@union.ai> * parallelism working Signed-off-by: Daniel Rammer <daniel@union.ai> * cache and cache_serialize working - first new functionality in maptask Signed-off-by: Daniel Rammer <daniel@union.ai> * adding implementation notes Signed-off-by: Daniel Rammer <daniel@union.ai> * removed eventing from subtasks Signed-off-by: Daniel Rammer <daniel@union.ai> * adding correct requirements Signed-off-by: Daniel Rammer <daniel@union.ai> * working end-2-end with flytekit Signed-off-by: Daniel Rammer <daniel@union.ai> * reporting output directory on success Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed output directory append Signed-off-by: Daniel Rammer <daniel@union.ai> * mocking TaskTemplate interface to enable caching Signed-off-by: Daniel Rammer <daniel@union.ai> * capture failure reasons Signed-off-by: Daniel Rammer <daniel@union.ai> * wrapped up abort and finalize functionality Signed-off-by: Daniel Rammer <daniel@union.ai> * mocking initialization events Signed-off-by: Daniel Rammer <daniel@union.ai> * sending all events Signed-off-by: Daniel Rammer <daniel@union.ai> * minor refactoring of debug prints and formatting Signed-off-by: Daniel Rammer <daniel@union.ai> * intratask checkpointing working Signed-off-by: Daniel Rammer <daniel@union.ai> * support for and Signed-off-by: Daniel Rammer <daniel@union.ai> * setting node log ids correctly Signed-off-by: Daniel Rammer <daniel@union.ai> * reporting cache status Signed-off-by: Daniel Rammer <daniel@union.ai> * correctly setting subnode abort phase Signed-off-by: Daniel Rammer <daniel@union.ai> * removing dead code Signed-off-by: Daniel Rammer <daniel@union.ai> * cleaned up most random TODO items Signed-off-by: Daniel Rammer <daniel@union.ai> * refactored into new files Signed-off-by: Daniel Rammer <daniel@union.ai> * refactoring for ArrayNode unit tests Signed-off-by: Daniel Rammer <daniel@union.ai> * refactored for unit testing to allow creation of NodeExecutor in array package Signed-off-by: Daniel Rammer <daniel@union.ai> * first unit test for handling ArrayNodePhaseNone Signed-off-by: Daniel Rammer <daniel@union.ai> * most of executing unit tests completed Signed-off-by: Daniel Rammer <daniel@union.ai> * finished executing unit tests Signed-off-by: Daniel Rammer <daniel@union.ai> * finished succeeding unit tests Signed-off-by: Daniel Rammer <daniel@union.ai> * wrote failing phase unit tests Signed-off-by: Daniel Rammer <daniel@union.ai> * moving towards complete unit_test success Signed-off-by: Daniel Rammer <daniel@union.ai> * unit tests passing Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed lint issues Signed-off-by: Daniel Rammer <daniel@union.ai> * updated flyteidl dep Signed-off-by: Daniel Rammer <daniel@union.ai> * added unit tests for Abort Signed-off-by: Daniel Rammer <daniel@union.ai> * adding unit test for Finalize Signed-off-by: Daniel Rammer <daniel@union.ai> * added utils unit tests Signed-off-by: Daniel Rammer <daniel@union.ai> * moved state structs to handler package Signed-off-by: Daniel Rammer <daniel@union.ai> * added docs Signed-off-by: Daniel Rammer <daniel@union.ai> * cleaned up abort event reporting Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed RecordNodeEvent unit tests Signed-off-by: Daniel Rammer <daniel@union.ai> * removed taskEventRecorder from nodes package Signed-off-by: Daniel Rammer <daniel@union.ai> * adding interface checking for arraynode Signed-off-by: Daniel Rammer <daniel@union.ai> * added transform unit test Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed input bindings issue Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed unit tests Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed unit tests Signed-off-by: Daniel Rammer <daniel@union.ai> * go generate Signed-off-by: Daniel Rammer <daniel@union.ai> * addressing random TODO Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed unit tests Signed-off-by: Daniel Rammer <daniel@union.ai> * addressing pr comments Signed-off-by: Daniel Rammer <daniel@union.ai> --------- Signed-off-by: Daniel Rammer <daniel@union.ai>
8 tasks
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR
Instrumenting support for ArrayNodes to provide a functionally complete map task experience.
Type
Are all requirements met?
Complete description
^^^
Tracking Issue
flyteorg/flyte#1131
Follow-up issue
NA