Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Get*ExecutionData should optionally return data inline #419

Closed
4 of 13 tasks
katrogan opened this issue Jul 20, 2020 · 1 comment
Closed
4 of 13 tasks

[Feature] Get*ExecutionData should optionally return data inline #419

katrogan opened this issue Jul 20, 2020 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@katrogan
Copy link
Contributor

Motivation: Why do you think this is important?
The current model of fetching signed URL references to execution data [inputs + outputs] works fine in a hosted AWS environment but doesn't actually successfully return a usable link in local/sandbox mode. This is because "the url returned to flytekit is a presigned url, with the host name which is kubernetes cluster local - like - http://minio.flyte.svc.cluster.local:9000" (@kumare3) and makes for a frustrating and broken experience when standing up flyte locally.

Goal: What should the final outcome look like, ideally?
Ideally the Get*ExecutionData endpoints [Workflow, Node, Task] should allow callers to specify whether they want populated inputs & outputs in addition to signed URL blobs pointing to where the same data is stored.

The endpoints should remain separate from Get*Execution as is, but offer more customizability.

Describe alternatives you've considered
Oneof responses that include either the URL blob or the populated inputs. However URL blobs are cheap to generate and lightweight to pass around so we can keep them as is.

Optionally the inputs & outputs can be returned inline in the Get*Execution responses but this risks bloating them.

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other

[Optional] Propose: Link/Inline

Additional context
First raised by @kumare3

Is this a blocker for you to adopt Flyte
N/A

@katrogan katrogan added enhancement New feature or request untriaged This issues has not yet been looked at by the Maintainers labels Jul 20, 2020
@katrogan
Copy link
Contributor Author

cc @kumare3 @wild-endeavor

@katrogan katrogan removed the untriaged This issues has not yet been looked at by the Maintainers label Aug 12, 2020
@katrogan katrogan self-assigned this Aug 12, 2020
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
* change label selector op

Signed-off-by: Lisa <aeioulisa@gmail.com>
Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* lint error

Signed-off-by: Lisa <aeioulisa@gmail.com>
Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* test error

Signed-off-by: Lisa <aeioulisa@gmail.com>
Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* revert para name and keep old function

Signed-off-by: Lisa <aeioulisa@gmail.com>
Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* remove strconv.Itoa

Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* two lebal

Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* two lebal

Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* reset config and add split function

Signed-off-by: aeioulisa <aeioulisa@gmail.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 6, 2022
* add flytescheduler to the release archives

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update flytepropeller version

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
Embedding a video to outline the Pytorch training process for single-node single GPUs.
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Dec 20, 2022
* feat: webpack config split
* chore: import react from cdn
* chore: update readmes
* chore: add script to start prod build locally

Signed-off-by: Carina Ursu <carina@union.ai>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 9, 2023
* change label selector op

Signed-off-by: Lisa <aeioulisa@gmail.com>
Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* lint error

Signed-off-by: Lisa <aeioulisa@gmail.com>
Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* test error

Signed-off-by: Lisa <aeioulisa@gmail.com>
Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* revert para name and keep old function

Signed-off-by: Lisa <aeioulisa@gmail.com>
Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* remove strconv.Itoa

Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* two lebal

Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* two lebal

Signed-off-by: aeioulisa <aeioulisa@gmail.com>

* reset config and add split function

Signed-off-by: aeioulisa <aeioulisa@gmail.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Aug 21, 2023
* add flytescheduler to the release archives

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

* Update flytepropeller version

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>
eapolinario pushed a commit that referenced this issue Sep 26, 2023
* add field

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Pass task execution metadata from agent (#422)

* Pass task execution metadata from agent

Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com>

* Add doc

Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com>

* Update protos/flyteidl/admin/agent.proto

Co-authored-by: Kevin Su <pingsutw@gmail.com>
Signed-off-by: Honnix <honnix@users.noreply.github.com>

* Regenerate

---------

Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com>
Signed-off-by: Honnix <honnix@users.noreply.github.com>
Co-authored-by: Kevin Su <pingsutw@gmail.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Add tags to execution spec (#414)

* add tags to execution spec

Signed-off-by: Kevin Su <pingsutw@apache.org>

* add tags to execution spec

Signed-off-by: Kevin Su <pingsutw@apache.org>

* add comment

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Correct comment for array job max parallelism (#431)

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Add the scalar to the operand (#427)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* add selector

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* move selectors from container to task metadata

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* drop only_preferred

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Updating boilerplate to lock golangci-lint version (#435)

Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* add unpartitioned selector

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* refactor

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* refactor

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* fix oneof names

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* add build.os for read the docs

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

---------

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com>
Signed-off-by: Honnix <honnix@users.noreply.github.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Co-authored-by: Honnix <honnix@users.noreply.github.com>
Co-authored-by: Kevin Su <pingsutw@gmail.com>
Co-authored-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Katrina Rogan <katroganGH@gmail.com>
Co-authored-by: Jeev B <jeevb@users.noreply.github.com>
Co-authored-by: Dan Rammer <daniel@union.ai>
eapolinario pushed a commit that referenced this issue Sep 26, 2023
* Revert "GPU Type (#419)"

This reverts commit 7bd98a9.

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Restore .readthedocs.yml

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

---------

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
pvditt pushed a commit that referenced this issue Dec 29, 2023
* add field

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Pass task execution metadata from agent (#422)

* Pass task execution metadata from agent

Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com>

* Add doc

Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com>

* Update protos/flyteidl/admin/agent.proto

Co-authored-by: Kevin Su <pingsutw@gmail.com>
Signed-off-by: Honnix <honnix@users.noreply.github.com>

* Regenerate

---------

Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com>
Signed-off-by: Honnix <honnix@users.noreply.github.com>
Co-authored-by: Kevin Su <pingsutw@gmail.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Add tags to execution spec (#414)

* add tags to execution spec

Signed-off-by: Kevin Su <pingsutw@apache.org>

* add tags to execution spec

Signed-off-by: Kevin Su <pingsutw@apache.org>

* add comment

Signed-off-by: Kevin Su <pingsutw@apache.org>

---------

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Correct comment for array job max parallelism (#431)

Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Add the scalar to the operand (#427)

Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* add selector

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* move selectors from container to task metadata

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* drop only_preferred

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Updating boilerplate to lock golangci-lint version (#435)

Signed-off-by: Daniel Rammer <daniel@union.ai>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* add unpartitioned selector

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* refactor

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* refactor

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* fix oneof names

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* add build.os for read the docs

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

---------

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com>
Signed-off-by: Honnix <honnix@users.noreply.github.com>
Signed-off-by: Kevin Su <pingsutw@apache.org>
Signed-off-by: Katrina Rogan <katroganGH@gmail.com>
Signed-off-by: Daniel Rammer <daniel@union.ai>
Co-authored-by: Honnix <honnix@users.noreply.github.com>
Co-authored-by: Kevin Su <pingsutw@gmail.com>
Co-authored-by: Kevin Su <pingsutw@apache.org>
Co-authored-by: Katrina Rogan <katroganGH@gmail.com>
Co-authored-by: Jeev B <jeevb@users.noreply.github.com>
Co-authored-by: Dan Rammer <daniel@union.ai>
pvditt pushed a commit that referenced this issue Dec 29, 2023
* Revert "GPU Type (#419)"

This reverts commit b92acad.

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

* Restore .readthedocs.yml

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

---------

Signed-off-by: Jeev B <jeevb@users.noreply.github.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
…rg#423)

* WIP. Marked places where an acknowledgement before an update is needed.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* (1) Added error handling for methods fetching matchable attributes when attributes do not exist. (2) Added fetching data that is needed for diffing during updates.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Diff and ask for ack.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Fixed some of the TODOs.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Cleaned up error handling.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Updated tests.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* More tests.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Fix case for No string to no (flyteorg#419)

Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Replaced diffing implementation.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Addressed pull request comments.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Fixed linter errors.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

---------

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>
Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future-Outlier <eric901201@gmail.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
eapolinario pushed a commit to eapolinario/flyte that referenced this issue Apr 30, 2024
…rg#423)

* WIP. Marked places where an acknowledgement before an update is needed.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* (1) Added error handling for methods fetching matchable attributes when attributes do not exist. (2) Added fetching data that is needed for diffing during updates.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Diff and ask for ack.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Fixed some of the TODOs.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Cleaned up error handling.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Updated tests.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* More tests.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Fix case for No string to no (flyteorg#419)

Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Replaced diffing implementation.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Addressed pull request comments.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Fixed linter errors.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

---------

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>
Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future-Outlier <eric901201@gmail.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
austin362667 pushed a commit to austin362667/flyte that referenced this issue May 7, 2024
Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
austin362667 pushed a commit to austin362667/flyte that referenced this issue May 7, 2024
…rg#423)

* WIP. Marked places where an acknowledgement before an update is needed.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* (1) Added error handling for methods fetching matchable attributes when attributes do not exist. (2) Added fetching data that is needed for diffing during updates.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Diff and ask for ack.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Fixed some of the TODOs.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Cleaned up error handling.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Updated tests.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* More tests.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Fix case for No string to no (flyteorg#419)

Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Replaced diffing implementation.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Addressed pull request comments.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Fixed linter errors.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

---------

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>
Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future-Outlier <eric901201@gmail.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
robert-ulbrich-mercedes-benz pushed a commit to robert-ulbrich-mercedes-benz/flyte that referenced this issue Jul 2, 2024
Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
robert-ulbrich-mercedes-benz pushed a commit to robert-ulbrich-mercedes-benz/flyte that referenced this issue Jul 2, 2024
…rg#423)

* WIP. Marked places where an acknowledgement before an update is needed.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* (1) Added error handling for methods fetching matchable attributes when attributes do not exist. (2) Added fetching data that is needed for diffing during updates.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Diff and ask for ack.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Fixed some of the TODOs.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Cleaned up error handling.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Updated tests.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* More tests.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Fix case for No string to no (flyteorg#419)

Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Replaced diffing implementation.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Addressed pull request comments.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

* Fixed linter errors.

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>

---------

Signed-off-by: Kamal Eybov <54046807+kamaleybov@users.noreply.github.com>
Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future-Outlier <eric901201@gmail.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant