Update Propeller architecture documentation #5117
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
March 26, 2024 14:51
Signed-off-by: davidmirror-ops <david espejo@union.ai>
Signed-off-by: davidmirror-ops <david espejo@union.ai>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #5117 +/- ##
==========================================
+ Coverage 58.99% 59.11% +0.11%
==========================================
Files 645 645
Lines 55542 55574 +32
==========================================
+ Hits 32769 32852 +83
+ Misses 20182 20129 -53
- Partials 2591 2593 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
added 3 commits
March 26, 2024 15:13
Signed-off-by: davidmirror-ops <david espejo@union.ai>
Signed-off-by: davidmirror-ops <david espejo@union.ai>
Signed-off-by: davidmirror-ops <david espejo@union.ai>
davidmirror-ops
changed the title
dosubot
bot
added
size:L
Signed-off-by: davidmirror-ops <david espejo@union.ai>
neverett
previously approved these changes
Mar 28, 2024
| @@ -15,13 +15,20 @@ Introduction | |||
| A Flyte :ref:`workflow <divedeep-workflows>` is represented as a Directed Acyclic Graph (DAG) of interconnected Nodes. Flyte supports a robust collection of Node types to ensure diverse functionality. | |||
|
|
|||
| - ``TaskNodes`` support a plugin system to externally add system integrations. | |||
| - Control flow can be altered during runtime using ``BranchNodes``, which prune downstream evaluation paths based on input. | |||
| - ``BranchNodes`` allow altering the Control flow during runtime; pruning downstream evaluation paths based on input. | |||
There was a problem hiding this comment.
Suggested change
| - ``BranchNodes`` allow altering the Control flow during runtime; pruning downstream evaluation paths based on input. | |
| - ``BranchNodes`` allow altering the control flow during runtime, pruning downstream evaluation paths based on input. |
| NAME AGE | ||
| f7616dc75400f43e6920 3h42m | ||
|
|
||
| 3. Describe the contents of the CR, for example the ``spec`` section: |
There was a problem hiding this comment.
Suggested change
| 3. Describe the contents of the CR, for example the ``spec`` section: | |
| 3. Describe the contents of the Custom Resource, for example the ``spec`` section: |
| @@ -30,12 +37,99 @@ This document attempts to break down the FlytePropeller architecture by tracking | |||
| Components | |||
| ========== | |||
|
|
|||
|
|
|||
| FlyteAdmin is the common entry point, where initialization of FlyteWorkflow CRs may be triggered by user workflow definition executions, automatic relaunches, or periodically scheduled workflow definition executions. | |||
There was a problem hiding this comment.
Suggested change
| FlyteAdmin is the common entry point, where initialization of FlyteWorkflow CRs may be triggered by user workflow definition executions, automatic relaunches, or periodically scheduled workflow definition executions. | |
| FlyteAdmin is the common entry point, where initialization of FlyteWorkflow Custom Resources may be triggered by user workflow definition executions, automatic relaunches, or periodically scheduled workflow definition executions. |
| @@ -45,12 +139,12 @@ FlytePropeller supports concurrent execution of multiple, unique workflows using | |||
| The WorkQueue is a FIFO queue storing workflow ID strings that require a lookup to retrieve the FlyteWorkflow CR to ensure up-to-date status. A workflow may be added to the queue in a variety of circumstances: | |||
|
|
|||
| #. A new FlyteWorkflow CR is created or an existing instance is updated | |||
| #. The K8s Informer resyncs the FlyteWorkflow periodically (necessary to detect workflow timeouts and ensure liveness) | |||
| #. The K8s Informer detects a workflow timeout or failed liveness check during its periodic resync operation on the FlyteWorkflow. | |||
| #. A FlytePropeller worker experiences an error during a processing loop | |||
| #. The WorkflowExecutor observes a completed downstream node | |||
| #. A NodeHandler observes state change and explicitly enqueues its owner (For example, K8s pod informer observes completion of a task) | |||
There was a problem hiding this comment.
Suggested change
| #. A NodeHandler observes state change and explicitly enqueues its owner (For example, K8s pod informer observes completion of a task) | |
| #. A NodeHandler observes state change and explicitly enqueues its owner. (For example, K8s pod informer observes completion of a task.) |
| @@ -60,7 +154,9 @@ The WorkflowExecutor is responsible for handling high-level workflow operations. | |||
| NodeExecutor | |||
| ------------ | |||
|
|
|||
| The NodeExecutor is executed on a single node, beginning with the workflow's start node. It traverses the workflow using a visitor pattern with a modified depth-first search (DFS), evaluating each node along the path. A few examples of node evaluation based on phase: successful nodes are skipped, unevaluated nodes are queued for processing, and failed nodes may be reattempted up to a configurable threshold. There are many configurable parameters to tune evaluation criteria including max parallelism which restricts the number of nodes which may be scheduled concurrently. Additionally, nodes may be retried to ensure recoverability on failure. | |||
| The NodeExecutor is executed on a single node, beginning with the workflow's start node. It traverses the workflow using a visitor pattern with a modified depth-first search (DFS), evaluating each node along the path. A few examples of node evaluation based on phase include: successful nodes are skipped, unevaluated nodes are queued for processing, and failed nodes may be reattempted up to a configurable threshold. There are many configurable parameters to tune evaluation criteria including max parallelism which restricts the number of nodes which may be scheduled concurrently. Additionally, nodes may be retried to ensure recoverability on failure. | |||
There was a problem hiding this comment.
Suggested change
| The NodeExecutor is executed on a single node, beginning with the workflow's start node. It traverses the workflow using a visitor pattern with a modified depth-first search (DFS), evaluating each node along the path. A few examples of node evaluation based on phase include: successful nodes are skipped, unevaluated nodes are queued for processing, and failed nodes may be reattempted up to a configurable threshold. There are many configurable parameters to tune evaluation criteria including max parallelism which restricts the number of nodes which may be scheduled concurrently. Additionally, nodes may be retried to ensure recoverability on failure. | |
| The NodeExecutor is executed on a single node, beginning with the workflow's start node. It traverses the workflow using a visitor pattern with a modified depth-first search (DFS), evaluating each node along the path. A few examples of node evaluation based on phase include: | |
| * Successful nodes are skipped. | |
| * Unevaluated nodes are queued for processing. | |
| * Failed nodes may be reattempted up to a configurable threshold. | |
| There are many configurable parameters to tune evaluation criteria including max parallelism which restricts the number of nodes which may be scheduled concurrently. Additionally, nodes may be retried to ensure recoverability on failure. |
Signed-off-by: davidmirror-ops <david espejo@union.ai>
neverett
approved these changes
Apr 1, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
What changes were proposed in this pull request?
This PR aims to expand details on how the different components of propeller play a role during a workflow execution, adding steps for users to verify on their own how the execution engine works. This work sets a baseline to further discuss performance optimization in other sections of the docs.
How was this patch tested?
Setup process
Screenshots
Check all the applicable boxes
Related PRs
Docs link