Skip to content

feat(proto): add task_type to TailLogsRequest for connector log routing#7290

Closed
pingsutw wants to merge 256 commits intoflyteorg:mainfrom
pingsutw:connector-tail-logs
Closed

feat(proto): add task_type to TailLogsRequest for connector log routing#7290
pingsutw wants to merge 256 commits intoflyteorg:mainfrom
pingsutw:connector-tail-logs

Conversation

@pingsutw
Copy link
Copy Markdown
Member

@pingsutw pingsutw commented Apr 27, 2026

Summary

  • Adds task_type and task_type_version fields to flyteidl2.dataproxy.TailLogsRequest and flyteidl2.workflow.TailLogsRequest.
  • Allows the dataplane operator to route a TailLogs request to a connector's AsyncConnectorService.GetTaskLogs RPC instead of pod logs when the action is a connector task.

Test Plan

  • buf generate regenerated Go and TS bindings; go build ./... clean.
  • Cloud-side integration is in unionai/cloud PR (link in description).

Notes

The cloud-side consumer lives in unionai/cloud and is what reads these fields off the request to decide whether to dial the connector or fall through to pod logs.

EngHabu and others added 30 commits September 30, 2025 14:27
* Fix cargo build and add CI check

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* lint lib.rs

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update Readme and contributing guides

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* signed?

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix build target

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix build-crate target

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix included ts files in npm

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* Add device class to `GpuAccelerator`

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* make gen

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Add HABANA_GAUDI as supported device class

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

---------

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
* Clean up triggers api

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

* make gen

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

---------

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Daniel Rammer <hamersaw@protonmail.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
* make state service apis unary

Signed-off-by: Daniel Rammer <hamersaw@protonmail.com>

* make buf && make gen

Signed-off-by: Daniel Rammer <hamersaw@protonmail.com>

---------

Signed-off-by: Daniel Rammer <hamersaw@protonmail.com>
…g#6663)

Signed-off-by: Alex Wu <c.alexwu@gmail.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
* Use Docker images locally and in CI

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix branch prefix tag

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* use github.head_ref instead

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* fix tag in comment

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* make local docker experience

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* try

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Faster python install?

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wait for image build wf

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Optimize image

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* cleanups

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* wip

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* regen

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* cleanup not needed commands

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update .github/workflows/regenerate-on-comment.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* test changes

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* PR Comments

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* PR Comments

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Trying with :v2 image

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Fix WF

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update guides

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update .github/workflows/regenerate-on-comment.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update .github/workflows/check-generate.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update .github/workflows/regenerate-on-comment.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update DOCKER_QUICK_START.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* PR Comments

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update README.md

Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update CONTRIBUTING.md

Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* PR Comments

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com>
* Don't publish SHA image when merging to v2 or main

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* regen

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

---------

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
* Pytorch proto v2 migration

Signed-off-by: WangWang0226 <eeha8834@gmail.com>

* Fix archived link and rebase branch to update Cargo.lock version change

Signed-off-by: WangWang0226 <eeha8834@gmail.com>

* Remove useless line in pytorch.proto

Signed-off-by: WangWang0226 <eeha8834@gmail.com>

---------

Signed-off-by: WangWang0226 <eeha8834@gmail.com>
* feat: add metadata to list tash response proto

Signed-off-by: machichima <nary12321@gmail.com>

* build: make gen

Signed-off-by: machichima <nary12321@gmail.com>

* build: make gen

Signed-off-by: machichima <nary12321@gmail.com>

---------

Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
* feat: add List versions request and response

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add ListVersions RPC

Signed-off-by: machichima <nary12321@gmail.com>

* fix: make gen

Signed-off-by: machichima <nary12321@gmail.com>

* fix: make gen

Signed-off-by: machichima <nary12321@gmail.com>

* feat: add common list request to list versions request

Signed-off-by: machichima <nary12321@gmail.com>

---------

Signed-off-by: machichima <nary12321@gmail.com>
pingsutw and others added 18 commits April 16, 2026 10:52
Signed-off-by: Kevin Su <pingsutw@apache.org>
…K8s cascade (flyteorg#7225)

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Kevin Su <pingsutw@apache.org>
…logKeys (flyteorg#7229)

* [V2] Expand contextutils with org, service_name, cluster_id keys and logKeys

The shared interceptor chain in Union Cloud sets context values using v1
contextutils.Key types, but v2 services (actions, leaseworker, leasor,
events-proxy, executorv2) use the v2 logger which reads v2 contextutils.Key.
Both are `type Key string` but are different Go types, so context.Value()
lookups fail silently and all context-propagated log fields are missing from
v2 service logs.

Add OrganizationKey, ServiceNameKey, and ClusterNameKey constants to v2
contextutils, and expand logKeys with ProjectKey, DomainKey, and the three
new keys so that GetLogFields() includes them in structured log output.

This is one half of the fix — the other half is a context bridge interceptor
in the cloud repo that copies v1 values onto v2 keys at request time.

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

* make gen

Signed-off-by: Sergey Vilgelm <sergey@union.ai>

---------

Signed-off-by: Sergey Vilgelm <sergey@union.ai>
* add: dep

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* restructure

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* wip

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* restruct

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* fix

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* move config

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* remove db.go

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* address comments

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* comments

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

---------

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
…inistic (flyteorg#7231)

* fix(runs): make TestWatchActionUpdates_OnlyStreamsTargetAction deterministic

Start watcher before creating actions so the subscriber is registered
when the creation NOTIFY arrives. Drain the creation notification before
the real assertions to eliminate the race that caused spurious
"unexpected update for action target" failures.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

* fix(runs): replace sleep with deterministic subscriber check

Replace time.Sleep with require.Eventually polling actionSubscribers
to ensure the watcher is registered before creating actions. This
removes all timing dependencies from the test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

---------

Signed-off-by: Sergey Vilgelm <sergey@union.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…org#7230)

* fix(Makefile): preserve exit code in timed macro

The timed macro previously lost the exit code of the wrapped command due to
chaining with semicolons — make always saw the exit code from the final echo
statement (0). Now the macro captures the command's exit code in a variable,
prints the timing, then exits with the original code, ensuring CI targets
like buf-lint properly fail when violations are detected.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

* fix(proto): resolve all buf lint violations

- Remove unused import of google/api/annotations.proto from cluster/service.proto
- Remove unused import of flyteidl2/task/common.proto from queue_service.proto
- Fix package mismatch in settings protos: change package from flyteidl2.org to
  flyteidl2.settings and update all field option references accordingly
- Remove unused buf dependency buf.build/grpc-ecosystem/grpc-gateway
- Regenerate all generated files (Go, Python, Rust, TypeScript)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

---------

Signed-off-by: Sergey Vilgelm <sergey@union.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
…g#7175)

* add: dep

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* restructure

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* wip

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* restruct

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* fix

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* move config

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* remove db.go

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* address comments

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* impl: internal service

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* add tests

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* address comments

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

---------

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
…al tarball steps (flyteorg#7242)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: pmahindrakar-oss <prafulla.mahindrakar@gmail.com>
…nding (flyteorg#7244)

Port of unionai/flyte#956 to v2. Fixes indefinite task hang when an init
container is stuck (e.g. ImagePullBackOff on a missing init image,
InvalidImageName, CreateContainerConfigError).

demystifyPendingHelper now inspects InitContainerStatuses before
ContainerStatuses, so the real init-container failure surfaces with the
correct reason and per-reason grace-period handling. Previously, the main
container's downstream Waiting{PodInitializing} masked the init-container
failure; only the coarse PodPendingTimeout would eventually rescue the task.

The waiting-reason switch is extracted into classifyWaitingContainer
(byte-equivalent logic move, preserving v2's existing classification for
CreateContainerError / CreateContainerConfigError past grace).

Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
…rg#7176)

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
…flyteorg#7188)

* add: watch reconnect

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

* fix: comments

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>

---------

Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
…es (flyteorg#7248)

* feat(proto): add PAUSED phase, Signal RPC, and condition action updates

- Add ACTION_PHASE_PAUSED = 9 to ActionPhase enum for condition actions
- Update ConditionAction: remove scope oneof (reserved fields 2-4),
  add prompt_type, timeout, webhook fields
- Add ConditionPromptType enum and ConditionWebhook message
- Add Signal RPC to ActionsService with SignalRequest/SignalResponse
- Add optional output field (core.Literal) to ActionUpdate

Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

* make gen

Signed-off-by: Sergey Vilgelm <sergey@union.ai>

* fix(tests): replace hand-written mockActionsClient with mockery-generated mock

The Signal RPC added to ActionsServiceClient broke the hand-written
mock in runs/service tests. Replace it with a mockery-generated mock
that automatically stays in sync with the interface.

- Add actionsconnect to .mockery.yaml
- Generate ActionsServiceClient mock via mockery
- Update run_service_test.go, abort_reconciler_test.go, and
  get_action_details_test.go to use the generated mock

Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

* refactor(proto): rename output to value in SignalRequest and ActionUpdate

Rename the field from 'output' to 'value' for clarity — conditions
carry a signal value, not an output in the task-action sense.

Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: Sergey Vilgelm <sergey@union.ai>

---------

Signed-off-by: Sergey Vilgelm <sergey@union.ai>
Adds task_type and task_type_version to flyteidl2.dataproxy.TailLogsRequest and
flyteidl2.workflow.TailLogsRequest so the dataplane operator can branch between
pod logs and connector GetTaskLogs based on the task category.

Signed-off-by: Kevin Su <pingsutw@apache.org>
@pingsutw pingsutw changed the base branch from master to v2 April 27, 2026 06:47
@pingsutw pingsutw marked this pull request as draft April 27, 2026 06:47
@pingsutw pingsutw changed the base branch from v2 to main April 28, 2026 18:35
@pingsutw pingsutw added flyte2 and removed flyte2 labels Apr 28, 2026
@pingsutw
Copy link
Copy Markdown
Member Author

Superseded by #7313 which targets main.

@pingsutw pingsutw closed this Apr 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.