Cherry pick make local run work by machichima · Pull Request #6943 · flyteorg/flyte

machichima · 2026-02-26T05:34:38Z

Tracking issue

Why are the changes needed?

This PR cherry picks from #6904, to ensure we can:

start the local env with k3d
- Development guide is in: manager/DEVELOPMENT.md
submit task, create pod for actions
Connect from flyte-sdk

What changes were proposed in this pull request?

How was this patch tested?

Local test

Labels

Please add one or more of the following labels to categorize your PR:

added: For new features.
changed: For changes in existing functionality.
deprecated: For soon-to-be-removed features.
removed: For features being removed.
fixed: For any bug fixed.
security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

Docs link

main
- Flyte 2 #6583
  - Fix vet issues #6901
    - Implement state service on top of etcd. #6902
      - Cherry pick make local run work #6943 👈

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com>

…chema translation. The converter handles bidirectional conversion Add literal-JSON converter and TranslatorService for RSJF-compliant schema translation Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com>

…ue and watch pipelines. Pass inputUri and runOutputBase as parameters to CreateRun instead of computing them after DB insert. Fix WatchActions to subscribe before Add ShortName field to TaskActionSpec and populate it through the queue and watch pipelines Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com>

Signed-off-by: machichima <nary12321@gmail.com>

* Switch to pflags Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> * Let it rain Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> * mostly working Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> * Add literal-JSON converter and TranslatorService for RSJF-compliant schema translation. The converter handles bidirectional conversion Add literal-JSON converter and TranslatorService for RSJF-compliant schema translation Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> * Add ShortName field to TaskActionSpec and populate it through the queue and watch pipelines. Pass inputUri and runOutputBase as parameters to CreateRun instead of computing them after DB insert. Fix WatchActions to subscribe before Add ShortName field to TaskActionSpec and populate it through the queue and watch pipelines Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> * fix: Pass parent name to Subscribe() and add TODO Signed-off-by: machichima <nary12321@gmail.com> * fix: clean-up main and add StopWatching call Signed-off-by: machichima <nary12321@gmail.com> * build: update CR fields Signed-off-by: machichima <nary12321@gmail.com> * feat: build local env with k3d and run successfully Signed-off-by: machichima <nary12321@gmail.com> * docs: add TODO in CreateRun Signed-off-by: machichima <nary12321@gmail.com> * docs: add brief development guide Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> Co-authored-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Implement state service on top of etcd. Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Initial executor <> Plugins integration Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix build and create_run.sh Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix: subscribe before listing Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * feat: store state to DB from state service watch Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix: skip update if state json does not change Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * feat: scope subscribe with parent action name Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor: rename regularK8sClient to k8sClientWithoutWatch Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix: log when update channel full Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor: remove unused function isChildOf Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * remove state service db persistant logic Signed-off-by: Alex Wu <c.alexwu@gmail.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix: a parent name link to multiple subscribers Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Initial executor <> Plugins integration (#6903) * refactor: add error log when fail to add finalizer Signed-off-by: machichima <nary12321@gmail.com> * fix: handle err when status update failed Signed-off-by: machichima <nary12321@gmail.com> * fix: remove phase is terminal check rely fully on task action conditions Signed-off-by: machichima <nary12321@gmail.com> * refactor: add event when failed to build context in finalize Signed-off-by: machichima <nary12321@gmail.com> * fix: reconcile again after update CR Signed-off-by: machichima <nary12321@gmail.com> * fix: only update status if changed Signed-off-by: machichima <nary12321@gmail.com> * build: remove tracked Cargo.lock Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> Co-authored-by: machichima <nary12321@gmail.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Cherry pick make local run work (#6943) * Switch to pflags Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> * Let it rain Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> * mostly working Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> * Add literal-JSON converter and TranslatorService for RSJF-compliant schema translation. The converter handles bidirectional conversion Add literal-JSON converter and TranslatorService for RSJF-compliant schema translation Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> * Add ShortName field to TaskActionSpec and populate it through the queue and watch pipelines. Pass inputUri and runOutputBase as parameters to CreateRun instead of computing them after DB insert. Fix WatchActions to subscribe before Add ShortName field to TaskActionSpec and populate it through the queue and watch pipelines Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> * fix: Pass parent name to Subscribe() and add TODO Signed-off-by: machichima <nary12321@gmail.com> * fix: clean-up main and add StopWatching call Signed-off-by: machichima <nary12321@gmail.com> * build: update CR fields Signed-off-by: machichima <nary12321@gmail.com> * feat: build local env with k3d and run successfully Signed-off-by: machichima <nary12321@gmail.com> * docs: add TODO in CreateRun Signed-off-by: machichima <nary12321@gmail.com> * docs: add brief development guide Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com> Co-authored-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> Signed-off-by: Alex Wu <c.alexwu@gmail.com> Co-authored-by: Haytham Abuelfutuh <haytham@afutuh.com> Co-authored-by: Alex Wu <c.alexwu@gmail.com>

This was referenced Feb 26, 2026

Fix vet issues #6901

Merged

Flyte 2 #6583

Draft

Implement state service on top of etcd. #6902

Closed

popojk approved these changes Feb 26, 2026

View reviewed changes

EngHabu force-pushed the enghabu/state-etcd branch 3 times, most recently from 01d43dc to fcb40a8 Compare February 26, 2026 19:45

EngHabu and others added 12 commits February 27, 2026 08:17

Switch to pflags

2b585b8

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com>

Let it rain

b611db6

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com>

mostly working

01bb452

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com>

fix: Pass parent name to Subscribe() and add TODO

d264a05

Signed-off-by: machichima <nary12321@gmail.com>

fix: clean-up main and add StopWatching call

8785bac

Signed-off-by: machichima <nary12321@gmail.com>

build: update CR fields

0227b3e

Signed-off-by: machichima <nary12321@gmail.com>

feat: build local env with k3d and run successfully

24cff80

Signed-off-by: machichima <nary12321@gmail.com>

docs: add TODO in CreateRun

7fc1c90

Signed-off-by: machichima <nary12321@gmail.com>

docs: add brief development guide

71debcc

Signed-off-by: machichima <nary12321@gmail.com>

build: make gen

290474a

Signed-off-by: machichima <nary12321@gmail.com>

machichima force-pushed the cherry-pick-make-local-run-work branch from 391ceef to 290474a Compare February 27, 2026 00:18

machichima merged commit 763b977 into enghabu/state-etcd Feb 27, 2026
5 checks passed

machichima deleted the cherry-pick-make-local-run-work branch February 27, 2026 01:20

machichima mentioned this pull request Feb 27, 2026

Switch to pflags #6904

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry pick make local run work#6943

Cherry pick make local run work#6943
machichima merged 12 commits intoenghabu/state-etcdfrom
cherry-pick-make-local-run-work

machichima commented Feb 26, 2026 •

edited by github-actions Bot

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

machichima commented Feb 26, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Labels

Setup process

Screenshots

Check all the applicable boxes

Related PRs

Docs link

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

machichima commented Feb 26, 2026 •

edited by github-actions Bot

Loading