feat(proto): add task_type to TailLogsRequest for connector log routing#7290
Closed
pingsutw wants to merge 256 commits intoflyteorg:mainfrom
Closed
feat(proto): add task_type to TailLogsRequest for connector log routing#7290pingsutw wants to merge 256 commits intoflyteorg:mainfrom
pingsutw wants to merge 256 commits intoflyteorg:mainfrom
Conversation
* Fix cargo build and add CI check Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * lint lib.rs Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update Readme and contributing guides Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * signed? Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix build target Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix build-crate target Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix included ts files in npm Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
* Add device class to `GpuAccelerator` Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * make gen Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add HABANA_GAUDI as supported device class Signed-off-by: Jeev B <jeevb@users.noreply.github.com> --------- Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
* Clean up triggers api Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * make gen Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> --------- Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Daniel Rammer <hamersaw@protonmail.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Alex Wu <c.alexwu@gmail.com>
* make state service apis unary Signed-off-by: Daniel Rammer <hamersaw@protonmail.com> * make buf && make gen Signed-off-by: Daniel Rammer <hamersaw@protonmail.com> --------- Signed-off-by: Daniel Rammer <hamersaw@protonmail.com>
…g#6663) Signed-off-by: Alex Wu <c.alexwu@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
* Use Docker images locally and in CI Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix branch prefix tag Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * use github.head_ref instead Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix tag in comment Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * make local docker experience Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * try Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Faster python install? Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wait for image build wf Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Optimize image Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * cleanups Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * regen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * cleanup not needed commands Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update .github/workflows/regenerate-on-comment.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * test changes Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * PR Comments Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * PR Comments Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Trying with :v2 image Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix WF Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update guides Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update .github/workflows/regenerate-on-comment.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update .github/workflows/check-generate.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update .github/workflows/regenerate-on-comment.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update DOCKER_QUICK_START.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * PR Comments Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update README.md Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update CONTRIBUTING.md Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com> Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * PR Comments Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Nary Yeh <60069744+machichima@users.noreply.github.com>
* Don't publish SHA image when merging to v2 or main Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * regen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
* Pytorch proto v2 migration Signed-off-by: WangWang0226 <eeha8834@gmail.com> * Fix archived link and rebase branch to update Cargo.lock version change Signed-off-by: WangWang0226 <eeha8834@gmail.com> * Remove useless line in pytorch.proto Signed-off-by: WangWang0226 <eeha8834@gmail.com> --------- Signed-off-by: WangWang0226 <eeha8834@gmail.com>
* feat: add metadata to list tash response proto Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>
* feat: add List versions request and response Signed-off-by: machichima <nary12321@gmail.com> * feat: add ListVersions RPC Signed-off-by: machichima <nary12321@gmail.com> * fix: make gen Signed-off-by: machichima <nary12321@gmail.com> * fix: make gen Signed-off-by: machichima <nary12321@gmail.com> * feat: add common list request to list versions request Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
…K8s cascade (flyteorg#7225) Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Kevin Su <pingsutw@apache.org>
…logKeys (flyteorg#7229) * [V2] Expand contextutils with org, service_name, cluster_id keys and logKeys The shared interceptor chain in Union Cloud sets context values using v1 contextutils.Key types, but v2 services (actions, leaseworker, leasor, events-proxy, executorv2) use the v2 logger which reads v2 contextutils.Key. Both are `type Key string` but are different Go types, so context.Value() lookups fail silently and all context-propagated log fields are missing from v2 service logs. Add OrganizationKey, ServiceNameKey, and ClusterNameKey constants to v2 contextutils, and expand logKeys with ProjectKey, DomainKey, and the three new keys so that GetLogFields() includes them in structured log output. This is one half of the fix — the other half is a context bridge interceptor in the cloud repo that copies v1 values onto v2 keys at request time. Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> * make gen Signed-off-by: Sergey Vilgelm <sergey@union.ai> --------- Signed-off-by: Sergey Vilgelm <sergey@union.ai>
* add: dep Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * restructure Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * wip Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * restruct Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * move config Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * remove db.go Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * address comments Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * comments Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> --------- Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
…inistic (flyteorg#7231) * fix(runs): make TestWatchActionUpdates_OnlyStreamsTargetAction deterministic Start watcher before creating actions so the subscriber is registered when the creation NOTIFY arrives. Drain the creation notification before the real assertions to eliminate the race that caused spurious "unexpected update for action target" failures. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> * fix(runs): replace sleep with deterministic subscriber check Replace time.Sleep with require.Eventually polling actionSubscribers to ensure the watcher is registered before creating actions. This removes all timing dependencies from the test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> --------- Signed-off-by: Sergey Vilgelm <sergey@union.ai> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…org#7230) * fix(Makefile): preserve exit code in timed macro The timed macro previously lost the exit code of the wrapped command due to chaining with semicolons — make always saw the exit code from the final echo statement (0). Now the macro captures the command's exit code in a variable, prints the timing, then exits with the original code, ensuring CI targets like buf-lint properly fail when violations are detected. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> * fix(proto): resolve all buf lint violations - Remove unused import of google/api/annotations.proto from cluster/service.proto - Remove unused import of flyteidl2/task/common.proto from queue_service.proto - Fix package mismatch in settings protos: change package from flyteidl2.org to flyteidl2.settings and update all field option references accordingly - Remove unused buf dependency buf.build/grpc-ecosystem/grpc-gateway - Regenerate all generated files (Go, Python, Rust, TypeScript) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> --------- Signed-off-by: Sergey Vilgelm <sergey@union.ai> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
…g#7175) * add: dep Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * restructure Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * wip Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * restruct Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * move config Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * remove db.go Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * address comments Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * impl: internal service Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * add tests Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * address comments Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> --------- Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
…al tarball steps (flyteorg#7242) Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: pmahindrakar-oss <prafulla.mahindrakar@gmail.com>
…nding (flyteorg#7244) Port of unionai/flyte#956 to v2. Fixes indefinite task hang when an init container is stuck (e.g. ImagePullBackOff on a missing init image, InvalidImageName, CreateContainerConfigError). demystifyPendingHelper now inspects InitContainerStatuses before ContainerStatuses, so the real init-container failure surfaces with the correct reason and per-reason grace-period handling. Previously, the main container's downstream Waiting{PodInitializing} masked the init-container failure; only the coarse PodPendingTimeout would eventually rescue the task. The waiting-reason switch is extracted into classifyWaitingContainer (byte-equivalent logic move, preserving v2's existing classification for CreateContainerError / CreateContainerConfigError past grace). Signed-off-by: Paul Dittamo <pvdittamo@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
…rg#7176) Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Kevin Su <pingsutw@apache.org>
…flyteorg#7188) * add: watch reconnect Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> * fix: comments Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com> --------- Signed-off-by: M. Adil Fayyaz <62440954+AdilFayyaz@users.noreply.github.com>
…es (flyteorg#7248) * feat(proto): add PAUSED phase, Signal RPC, and condition action updates - Add ACTION_PHASE_PAUSED = 9 to ActionPhase enum for condition actions - Update ConditionAction: remove scope oneof (reserved fields 2-4), add prompt_type, timeout, webhook fields - Add ConditionPromptType enum and ConditionWebhook message - Add Signal RPC to ActionsService with SignalRequest/SignalResponse - Add optional output field (core.Literal) to ActionUpdate Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> * make gen Signed-off-by: Sergey Vilgelm <sergey@union.ai> * fix(tests): replace hand-written mockActionsClient with mockery-generated mock The Signal RPC added to ActionsServiceClient broke the hand-written mock in runs/service tests. Replace it with a mockery-generated mock that automatically stays in sync with the interface. - Add actionsconnect to .mockery.yaml - Generate ActionsServiceClient mock via mockery - Update run_service_test.go, abort_reconciler_test.go, and get_action_details_test.go to use the generated mock Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> * refactor(proto): rename output to value in SignalRequest and ActionUpdate Rename the field from 'output' to 'value' for clarity — conditions carry a signal value, not an output in the task-action sense. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: Sergey Vilgelm <sergey@union.ai> --------- Signed-off-by: Sergey Vilgelm <sergey@union.ai>
Adds task_type and task_type_version to flyteidl2.dataproxy.TailLogsRequest and flyteidl2.workflow.TailLogsRequest so the dataplane operator can branch between pod logs and connector GetTaskLogs based on the task category. Signed-off-by: Kevin Su <pingsutw@apache.org>
Member
Author
|
Superseded by #7313 which targets main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
task_typeandtask_type_versionfields toflyteidl2.dataproxy.TailLogsRequestandflyteidl2.workflow.TailLogsRequest.TailLogsrequest to a connector'sAsyncConnectorService.GetTaskLogsRPC instead of pod logs when the action is a connector task.Test Plan
buf generateregenerated Go and TS bindings;go build ./...clean.Notes
The cloud-side consumer lives in
unionai/cloudand is what reads these fields off the request to decide whether to dial the connector or fall through to pod logs.main