-
Notifications
You must be signed in to change notification settings - Fork 654
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] Get*ExecutionData should optionally return data inline #419
Labels
enhancement
New feature or request
Comments
katrogan
added
enhancement
New feature or request
untriaged
This issues has not yet been looked at by the Maintainers
labels
Jul 20, 2020
This was referenced Jul 22, 2020
katrogan
removed
the
untriaged
This issues has not yet been looked at by the Maintainers
label
Aug 12, 2020
This was referenced Aug 19, 2020
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Dec 6, 2022
* change label selector op Signed-off-by: Lisa <aeioulisa@gmail.com> Signed-off-by: aeioulisa <aeioulisa@gmail.com> * lint error Signed-off-by: Lisa <aeioulisa@gmail.com> Signed-off-by: aeioulisa <aeioulisa@gmail.com> * test error Signed-off-by: Lisa <aeioulisa@gmail.com> Signed-off-by: aeioulisa <aeioulisa@gmail.com> * revert para name and keep old function Signed-off-by: Lisa <aeioulisa@gmail.com> Signed-off-by: aeioulisa <aeioulisa@gmail.com> * remove strconv.Itoa Signed-off-by: aeioulisa <aeioulisa@gmail.com> * two lebal Signed-off-by: aeioulisa <aeioulisa@gmail.com> * two lebal Signed-off-by: aeioulisa <aeioulisa@gmail.com> * reset config and add split function Signed-off-by: aeioulisa <aeioulisa@gmail.com>
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Dec 6, 2022
* add flytescheduler to the release archives Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update flytepropeller version Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Dec 20, 2022
Embedding a video to outline the Pytorch training process for single-node single GPUs.
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Dec 20, 2022
* feat: webpack config split * chore: import react from cdn * chore: update readmes * chore: add script to start prod build locally Signed-off-by: Carina Ursu <carina@union.ai>
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Dec 20, 2022
# [0.55.0](http://github.com/lyft/flyteconsole/compare/v0.54.2...v0.55.0) (2022-04-22) ### Features * webpack config changes ([flyteorg#419](http://github.com/lyft/flyteconsole/issues/419)) ([f93e94f](http://github.com/lyft/flyteconsole/commit/f93e94fd5b04c8da65b0bcd97ce1097451d83b28))
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Aug 9, 2023
* change label selector op Signed-off-by: Lisa <aeioulisa@gmail.com> Signed-off-by: aeioulisa <aeioulisa@gmail.com> * lint error Signed-off-by: Lisa <aeioulisa@gmail.com> Signed-off-by: aeioulisa <aeioulisa@gmail.com> * test error Signed-off-by: Lisa <aeioulisa@gmail.com> Signed-off-by: aeioulisa <aeioulisa@gmail.com> * revert para name and keep old function Signed-off-by: Lisa <aeioulisa@gmail.com> Signed-off-by: aeioulisa <aeioulisa@gmail.com> * remove strconv.Itoa Signed-off-by: aeioulisa <aeioulisa@gmail.com> * two lebal Signed-off-by: aeioulisa <aeioulisa@gmail.com> * two lebal Signed-off-by: aeioulisa <aeioulisa@gmail.com> * reset config and add split function Signed-off-by: aeioulisa <aeioulisa@gmail.com>
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Aug 21, 2023
* add flytescheduler to the release archives Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update flytepropeller version Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
eapolinario
pushed a commit
that referenced
this issue
Sep 26, 2023
* add field Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Pass task execution metadata from agent (#422) * Pass task execution metadata from agent Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com> * Add doc Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com> * Update protos/flyteidl/admin/agent.proto Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Honnix <honnix@users.noreply.github.com> * Regenerate --------- Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com> Signed-off-by: Honnix <honnix@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add tags to execution spec (#414) * add tags to execution spec Signed-off-by: Kevin Su <pingsutw@apache.org> * add tags to execution spec Signed-off-by: Kevin Su <pingsutw@apache.org> * add comment Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Correct comment for array job max parallelism (#431) Signed-off-by: Katrina Rogan <katroganGH@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add the scalar to the operand (#427) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add selector Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * move selectors from container to task metadata Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * drop only_preferred Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Updating boilerplate to lock golangci-lint version (#435) Signed-off-by: Daniel Rammer <daniel@union.ai> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add unpartitioned selector Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * refactor Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * refactor Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fix oneof names Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add build.os for read the docs Signed-off-by: Jeev B <jeevb@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com> Signed-off-by: Honnix <honnix@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Katrina Rogan <katroganGH@gmail.com> Signed-off-by: Daniel Rammer <daniel@union.ai> Co-authored-by: Honnix <honnix@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@gmail.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Katrina Rogan <katroganGH@gmail.com> Co-authored-by: Jeev B <jeevb@users.noreply.github.com> Co-authored-by: Dan Rammer <daniel@union.ai>
eapolinario
pushed a commit
that referenced
this issue
Sep 26, 2023
pvditt
pushed a commit
that referenced
this issue
Dec 29, 2023
* add field Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Pass task execution metadata from agent (#422) * Pass task execution metadata from agent Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com> * Add doc Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com> * Update protos/flyteidl/admin/agent.proto Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Honnix <honnix@users.noreply.github.com> * Regenerate --------- Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com> Signed-off-by: Honnix <honnix@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add tags to execution spec (#414) * add tags to execution spec Signed-off-by: Kevin Su <pingsutw@apache.org> * add tags to execution spec Signed-off-by: Kevin Su <pingsutw@apache.org> * add comment Signed-off-by: Kevin Su <pingsutw@apache.org> --------- Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Correct comment for array job max parallelism (#431) Signed-off-by: Katrina Rogan <katroganGH@gmail.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add the scalar to the operand (#427) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add selector Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * move selectors from container to task metadata Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * drop only_preferred Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Updating boilerplate to lock golangci-lint version (#435) Signed-off-by: Daniel Rammer <daniel@union.ai> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add unpartitioned selector Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * refactor Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * refactor Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * fix oneof names Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * add build.os for read the docs Signed-off-by: Jeev B <jeevb@users.noreply.github.com> --------- Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Jeev B <jeevb@users.noreply.github.com> Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com> Signed-off-by: Honnix <honnix@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Katrina Rogan <katroganGH@gmail.com> Signed-off-by: Daniel Rammer <daniel@union.ai> Co-authored-by: Honnix <honnix@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@gmail.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Katrina Rogan <katroganGH@gmail.com> Co-authored-by: Jeev B <jeevb@users.noreply.github.com> Co-authored-by: Dan Rammer <daniel@union.ai>
pvditt
pushed a commit
that referenced
this issue
Dec 29, 2023
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Apr 30, 2024
Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Apr 30, 2024
…rg#423) * WIP. Marked places where an acknowledgement before an update is needed. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * (1) Added error handling for methods fetching matchable attributes when attributes do not exist. (2) Added fetching data that is needed for diffing during updates. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Diff and ask for ack. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Fixed some of the TODOs. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Cleaned up error handling. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Updated tests. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * More tests. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Fix case for No string to no (flyteorg#419) Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com> Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Replaced diffing implementation. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Addressed pull request comments. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Fixed linter errors. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> --------- Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Future Outlier <eric901201@gmai.com>
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Apr 30, 2024
Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>
eapolinario
pushed a commit
to eapolinario/flyte
that referenced
this issue
Apr 30, 2024
…rg#423) * WIP. Marked places where an acknowledgement before an update is needed. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * (1) Added error handling for methods fetching matchable attributes when attributes do not exist. (2) Added fetching data that is needed for diffing during updates. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Diff and ask for ack. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Fixed some of the TODOs. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Cleaned up error handling. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Updated tests. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * More tests. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Fix case for No string to no (flyteorg#419) Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com> Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Replaced diffing implementation. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Addressed pull request comments. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Fixed linter errors. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> --------- Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Future Outlier <eric901201@gmai.com>
austin362667
pushed a commit
to austin362667/flyte
that referenced
this issue
May 7, 2024
Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>
austin362667
pushed a commit
to austin362667/flyte
that referenced
this issue
May 7, 2024
…rg#423) * WIP. Marked places where an acknowledgement before an update is needed. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * (1) Added error handling for methods fetching matchable attributes when attributes do not exist. (2) Added fetching data that is needed for diffing during updates. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Diff and ask for ack. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Fixed some of the TODOs. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Cleaned up error handling. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Updated tests. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * More tests. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Fix case for No string to no (flyteorg#419) Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com> Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Replaced diffing implementation. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Addressed pull request comments. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Fixed linter errors. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> --------- Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Future Outlier <eric901201@gmai.com>
robert-ulbrich-mercedes-benz
pushed a commit
to robert-ulbrich-mercedes-benz/flyte
that referenced
this issue
Jul 2, 2024
Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com>
robert-ulbrich-mercedes-benz
pushed a commit
to robert-ulbrich-mercedes-benz/flyte
that referenced
this issue
Jul 2, 2024
…rg#423) * WIP. Marked places where an acknowledgement before an update is needed. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * (1) Added error handling for methods fetching matchable attributes when attributes do not exist. (2) Added fetching data that is needed for diffing during updates. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Diff and ask for ack. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Fixed some of the TODOs. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Cleaned up error handling. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Updated tests. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * More tests. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Fix case for No string to no (flyteorg#419) Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future Outlier <eric901201@gmai.com> Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Replaced diffing implementation. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Addressed pull request comments. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> * Fixed linter errors. Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> --------- Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com> Signed-off-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Future Outlier <eric901201@gmai.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Motivation: Why do you think this is important?
The current model of fetching signed URL references to execution data [inputs + outputs] works fine in a hosted AWS environment but doesn't actually successfully return a usable link in local/sandbox mode. This is because "the url returned to flytekit is a presigned url, with the host name which is kubernetes cluster local - like - http://minio.flyte.svc.cluster.local:9000" (@kumare3) and makes for a frustrating and broken experience when standing up flyte locally.
Goal: What should the final outcome look like, ideally?
Ideally the Get*ExecutionData endpoints [Workflow, Node, Task] should allow callers to specify whether they want populated inputs & outputs in addition to signed URL blobs pointing to where the same data is stored.
The endpoints should remain separate from Get*Execution as is, but offer more customizability.
Describe alternatives you've considered
Oneof responses that include either the URL blob or the populated inputs. However URL blobs are cheap to generate and lightweight to pass around so we can keep them as is.
Optionally the inputs & outputs can be returned inline in the Get*Execution responses but this risks bloating them.
Flyte component
[Optional] Propose: Link/Inline
Additional context
First raised by @kumare3
Is this a blocker for you to adopt Flyte
N/A
The text was updated successfully, but these errors were encountered: